Image anti-shake control method, electronic equipment, chip system and storage medium

Abstract

The invention provides an image anti-shake control method, electronic equipment, a chip system and a storage medium, and relates to the technical field of terminals. In a shooting scene of a camera night scene mode, a first anti-shake control parameter, such as a shake information filtering mode, an image framing mode and / or an image fusion mode, suitable for a current scene can be determined based on scene brightness information, and a second anti-shake control parameter suitable for the current scene can be determined based on shake information; if the second anti-shake control parameter comprises the exposure compensation amount, different pieces of shake information correspond to different exposure compensation amounts. According to the scheme, different degrees of exposure compensation can be carried out in the shooting stage for night scene shooting scenes with different jitter intensities, the image anti-jitter purpose is achieved, comprehensive anti-jitter control and image processing are achieved in combination with a jitter information filtering mode, an image output frame mode and / or an image fusion mode, and the image processing efficiency is improved. The image anti-shake effect of the scene shot in the night scene mode is improved, and the definition of the night scene photo is improved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an image stabilization control method, electronic equipment, a chip system and a storage medium. Background Art

[0002] With the widespread use of electronic devices, using electronic devices to take photos has become a daily behavior in people's lives. Taking mobile phones as an example, when people hold their mobile phones to take photos, their own hand shaking, walking vibrations, etc. will cause the phone to shake during the shooting process, resulting in blurred images.

[0003] In this regard, the related technology performs anti-shake by adding an optical image stabilization (OIS) controller to the electronic device. The anti-shake principle of the OIS controller is to use an OIS motor to drive the lens to move, thereby offsetting the displacement caused by the shake.

[0004] However, for night scene shooting scenes, the current optical image stabilization performance is relatively weak, and there is still blurry imaging, which cannot meet the needs of users who want to take clear night scene photos. Therefore, an image stabilization control method that is more suitable for night scene shooting scenes is urgently needed. Summary of the invention

[0005] The present application provides an image stabilization control method, an electronic device, a chip system and a storage medium, which can enhance the image stabilization effect of scenes shot in night scene mode and improve the clarity of night scene photos.

[0006] In a first aspect, the present application provides an image stabilization control method, the method comprising: opening a camera application; enabling a camera night scene mode; when the camera night scene mode is enabled, obtaining first scene brightness information and first jitter information; determining a first stabilization control parameter based on the first scene brightness information, the first stabilization control parameter including a jitter information filtering method, an image frame output mode and / or an image fusion method, and different scene brightness information corresponds to different stabilization control parameters; determining a second stabilization control parameter based on the first jitter information, the second stabilization control parameter including a first exposure compensation amount, and different jitter information corresponds to different exposure compensation amounts; receiving a first operation triggered by a user to take a photo; in response to the first operation, performing stabilization control and image processing based on the first stabilization control parameter, and performing stabilization control and image processing based on the second stabilization control parameter to obtain a target image.

[0007] Through the image anti-shake control method provided by the embodiment of the present application, in the shooting scene of the camera night scene mode, the first anti-shake control parameter applicable to the current scene can be determined based on the scene brightness information, such as the jitter information filtering method, the image frame mode and / or the image fusion method, and the second anti-shake control parameter applicable to the current scene can be determined based on the jitter information, such as the second anti-shake control parameter includes the exposure compensation amount, and different jitter information corresponds to different exposure compensation amounts. The present application scheme can perform comprehensive anti-shake control and image processing based on the first anti-shake control parameter and the second anti-shake control parameter during the shooting stage, and perform different degrees of exposure compensation during the shooting stage for night scene shooting scenes with different jitter intensities, thereby improving the image anti-shake effect of the night scene mode shooting scene and improving the clarity of the night scene photos.

[0008] It should be noted that when taking photos using the night scene mode of the camera of an electronic device, due to the insufficient scene lighting in the night scene, it is easy to cause blurred photos due to hand shaking. In this case, the image anti-shake control method provided in the embodiment of the present application is used to avoid or alleviate the problem of blurred night scene photos caused by shaking, and can effectively extend the shutter time, thereby achieving the purpose of obtaining more exposure information, reducing noise in night scene photos and improving clarity.

[0009] In the embodiment of the present application, the shooting process can be divided into a preview phase and a shooting phase. Specifically, after the camera application is turned on, the preview phase is entered. When the user triggers the photo shooting, the camera application switches from the preview phase to the shooting phase. After the photo shooting is completed, the camera application switches from the shooting phase to the preview phase.

[0010] The following first describes possible implementations of the image anti-shake control method in the preview stage.

[0011] In some possible implementations, after the camera application is started and before a first operation of the user triggering a photo-taking is received, the method further includes: in a preview phase, performing anti-shake control and image processing based on a fourth anti-shake control parameter.

[0012] In the preview stage, the electronic device may obtain the first jitter information through the gyroscope sensor.

[0013] For example, the first shake information may be a deviation angular velocity or a deviation angle of the electronic device.

[0014] In some possible implementations, the anti-shake control and image processing based on the fourth anti-shake control parameter include: (1) applying bandpass filtering to the first jitter information obtained in the preview stage, and controlling the movement of the anti-shake device of the electronic device based on the first jitter information after the bandpass filtering to offset the impact of the jitter on the image; (2) acquiring an image using a baseline exposure parameter to obtain a baseline exposure image; wherein the baseline exposure parameter includes a baseline exposure time and a baseline ISO value; and (3) outputting an image sequence of the preview stage using a ZSL mode, wherein the image sequence of the preview stage includes one or more baseline exposure images.

[0015] Among them, using bandpass filtering on jitter information and using ZSL mode to output frames in the preview stage has the beneficial effect of better preview real-time performance.

[0016] The image sequence in the preview stage includes images that have been subjected to anti-shake control. The image sequence in the preview stage can be used for scene recognition, thereby determining the intensity of scene illumination, determining whether the current scene meets the conditions for night scene shooting, determining the image dynamic range, and enabling variable exposure shooting in high dynamic range situations.

[0017] In some possible implementations, the image sequence in the preview phase may be fused with the image sequence in the shooting phase to obtain a night scene photo, thereby improving the image stabilization effect and enhancing the clarity of the photo.

[0018] Next, possible implementations of the image anti-shake control method during the shooting phase will be described.

[0019] In the shooting stage, anti-shake control and image processing can be performed based on the first anti-shake control parameter, and anti-shake control and image processing can be performed based on the second anti-shake control parameter to obtain a target image.

[0020] The first anti-shake control parameter includes a jitter information filtering method, an image frame output mode and an image fusion method.

[0021] In some possible implementations, the anti-shake control and image processing based on the first anti-shake control parameter may include: (1) filtering the second jitter information obtained in the shooting stage using a first filtering method, where the first filtering method is a bandpass filter or a low-pass filter; (2) outputting an image sequence using a first frame output mode, where the first frame output mode is a zero-delay shooting ZSL mode or a non-ZSL mode; (3) performing image fusion using a first image sequence, where the first image sequence is an image sequence in the preview stage and an image sequence in the shooting stage, or is an image sequence in the shooting stage.

[0022] Specifically, the possible implementation manner of performing anti-shake control and image processing based on the first anti-shake control parameter can be determined based on scene brightness information. Different scene brightness information corresponds to different anti-shake control parameters.

[0023] In some possible implementations, different scene brightness information corresponds to different illumination levels, and different illumination levels correspond to different anti-shake control parameters. For example, three illumination levels are divided as an example, which are respectively called the first illumination level, the second illumination level, and the third illumination level, and the illumination levels of the three illumination levels increase in sequence. That is, the illumination of the first illumination level is less than the illumination of the second illumination level, and the illumination of the second illumination level is less than the illumination of the third illumination level.

[0024] The anti-shake control and image processing based on the first anti-shake control parameter may be implemented in the following three ways:

[0025] Possible implementation method 1: When the brightness information of the first scene corresponds to the first illumination level (extremely low illumination scene), the second jitter information is filtered using a low-pass filter, and the ZSL mode is switched to the non-ZSL mode, and the image sequence of the shooting stage is used for image fusion.

[0026] Possible implementation method two: When the brightness information of the first scene corresponds to the second illumination level (low illumination scene), the second jitter information is filtered using a low-pass filter, and the ZSL mode is switched to the non-ZSL mode, and the image sequence of the preview stage and the image sequence of the shooting stage are used for image fusion.

[0027] Possible implementation method three: When the brightness information of the first scene corresponds to the third illumination level (medium illumination scene), the second jitter information is filtered using a bandpass filter, and the ZSL mode is used to output frames, and the image sequence of the preview stage and the image sequence of the shooting stage are used for image fusion.

[0028] The above scenes are divided based on the illumination level. In actual situations, scenes such as medium illumination can be divided into medium illumination, high dynamic range scenes, or medium illumination, non-high dynamic range scenes. Among them, the medium illumination, high dynamic range scenes and medium illumination, non-high dynamic range scenes have different anti-shake control and image processing methods based on the first anti-shake control parameter. In the medium illumination, high dynamic range scene, variable exposure parameters can be used to capture images to obtain variable exposure image sequences.

[0029] In some possible implementations, the method further includes: determining that the current scene is a high dynamic range scene based on the first scene brightness information; and setting the exposure parameter to a variable exposure parameter when a first operation of the user triggering a photo is received.

[0030] In some possible implementations, the variable exposure parameter includes a first exposure duration and a second exposure duration; wherein the image sequence in the shooting phase includes the first exposure image and the second exposure image. Alternatively, the variable exposure parameter includes a first exposure duration, a second exposure duration, and a reference exposure duration; wherein the image sequence in the shooting phase includes the first exposure image, the second exposure image, and the reference exposure image. wherein the first exposure duration is greater than the reference exposure duration, and the reference exposure duration is greater than the second exposure duration.

[0031] Through the above scheme, image fusion is performed based on a variable exposure image sequence, which can improve the image effect.

[0032] The above describes possible implementation methods of anti-shake control and image processing based on first anti-shake control parameters such as jitter information filtering method, image frame output mode and image fusion method. The following describes possible implementation methods of anti-shake control and image processing based on second anti-shake control parameters.

[0033] The second anti-shake control parameter includes an exposure compensation amount, and different shake information corresponds to different exposure compensation amounts.

[0034] In some possible implementations, the anti-shake control and image processing based on the second anti-shake control parameter includes: adjusting the baseline exposure parameter based on the first exposure compensation amount to obtain the first exposure parameter; and acquiring the image using the first exposure parameter to obtain an image sequence in the shooting stage.

[0035] In some possible implementations, the first exposure compensation amount includes a compensation amount for the exposure duration and a compensation amount for the ISO value, different jitter information corresponds to different jitter levels, and different jitter levels correspond to different exposure compensation amounts. For example, taking the division of the first jitter level (low jitter intensity) and the second jitter level (high jitter intensity) as an example for explanation, the jitter intensity of the first jitter level is less than the jitter intensity of the second jitter level.

[0036] Among them, the adjustment based on the first exposure compensation amount is performed on the basis of the baseline exposure parameter, including: when the first jitter information corresponds to the first jitter level (low jitter intensity), increasing the first duration based on the baseline exposure duration, and reducing the first sensitivity value based on the baseline sensitivity ISO value.

[0037] Through the above scheme, in scenes with low jitter intensity, the generated night scene images are less affected by jitter, so the exposure time can be appropriately increased and the sensitivity can be reduced, which can effectively extend the shutter time, thereby achieving the purpose of obtaining more exposure information, reducing night scene photo noise and improving clarity.

[0038] Among them, the adjustment based on the first exposure compensation amount is performed on the basis of the baseline exposure parameter, including: when the first jitter information corresponds to the second jitter level (high jitter intensity), reducing the second duration based on the baseline exposure duration, and increasing the second sensitivity value based on the baseline ISO value.

[0039] Through the above scheme, in scenes with high jitter intensity, the generated night scene images are greatly affected by the jitter, so the exposure time can be appropriately reduced and the sensitivity can be increased, which can avoid or alleviate the problem of blurred night scene photos caused by jitter, thereby improving the clarity and brightness of the night scene photos.

[0040] It should be noted that the first jitter level (low jitter intensity) can be further subdivided. For example, the first jitter level can be divided into a case where the jitter intensity is equal to 0 (tripod shooting scene), and a case where the jitter intensity is greater than 0 and less than the first jitter threshold (micro-jitter scene).

[0041] In some possible implementations, when the first jitter information corresponds to a first jitter level, increasing a first exposure duration based on a reference exposure duration, and reducing a first photosensitivity value based on a reference ISO value, includes:

[0042] (1) When the shake intensity indicated by the first shake information is equal to 0 (tripod shooting scene), the first shake information is within the first shake level range, the exposure time is increased by N levels based on the base exposure time, and the sensitivity value is reduced by N levels based on the base ISO value.

[0043] (2) When the jitter intensity indicated by the first jitter information is greater than zero and less than a first jitter threshold, the first jitter information is within the first jitter level range, the M-level exposure duration is increased on the basis of the reference exposure duration, and the M-level photosensitivity value is reduced on the basis of the reference ISO value.

[0044] Wherein, N is greater than M, and M is greater than or equal to zero.

[0045] Through the above scheme, taking into account the tripod shooting scene and the slight shaking scene, the exposure time and ISO value are adjusted according to actual needs and experience values, thereby improving the clarity and brightness of night scene photos.

[0046] It should be noted that the second jitter level (high jitter intensity) can also be further subdivided. For example, the second jitter level can be divided into a situation where the jitter intensity is greater than the first jitter threshold and less than the second jitter threshold (medium-intensity jitter scenario), and a situation where the jitter intensity is greater than or equal to the second jitter threshold (high-intensity jitter scenario).

[0047] In some possible implementations, when the first jitter information corresponds to a second jitter level, reducing a second exposure duration based on the reference exposure duration, and increasing a second photosensitivity value based on the reference ISO value, includes:

[0048] (1) When the jitter intensity indicated by the first jitter information is greater than or equal to a first jitter threshold and less than a second jitter threshold, the first jitter information is within the second jitter level range, the S-level exposure time is reduced based on the reference exposure time, and the S-level photosensitivity value is increased based on the reference ISO value.

[0049] (2) When the jitter intensity indicated by the first jitter information is greater than or equal to a second jitter threshold, the first jitter information is within the second jitter level range, the T-level exposure duration is reduced based on the reference exposure duration, and the T-level photosensitivity value is increased based on the reference ISO value.

[0050] Among them, S is less than T and S is greater than zero.

[0051] Through the above solution, taking into account the medium-intensity jitter scenes and the high-intensity jitter scenes, the exposure time and ISO value are adjusted according to actual needs and experience values, thereby improving the clarity and brightness of night scene photos.

[0052] It should be noted that the corresponding limit exposure time can be pre-calibrated for different anti-shake strengths. For example, the anti-shake limit exposure schedule can be obtained based on empirical values. The anti-shake limit exposure schedule is queried through the anti-shake strength value to determine the limit exposure time corresponding to the anti-shake strength value.

[0053] If the exposure time calculated by the exposure compensation amount is greater than the maximum exposure time found by looking up the table, the exposure time and ISO value need to be adjusted again.

[0054] In some possible implementations, the adjusting based on the first exposure compensation amount on the basis of the reference exposure parameter to obtain the first exposure parameter includes: adjusting based on the first exposure compensation amount on the basis of the reference exposure parameter to obtain the exposure duration after the first adjustment and the ISO value after the first adjustment; if the exposure duration after the first adjustment is greater than the preset limit exposure duration, then setting the exposure duration to the limit exposure duration, and performing a second adjustment based on the ISO value after the first adjustment according to the first increment. The first exposure parameter includes the limit exposure duration and the ISO value after the second adjustment.

[0055] In some possible implementations, the adjustment is performed according to a first increment based on the ISO value after the first adjustment, including: calculating a first ratio based on the exposure time after the first adjustment and the maximum exposure time, multiplying the first ratio by the ISO value after the first adjustment to obtain a first increment; and adding the first increment to the ISO value after the first adjustment to obtain a second adjusted ISO value.

[0056] For example, assuming that the preset limit exposure time is 500ms, the exposure compensation amount is determined according to the jitter intensity level and the first adjustment is made. For example, the exposure time after the first adjustment is 600ms, and the ISO value after the first adjustment is 100. The exposure time of 600ms after the first adjustment is greater than the preset limit exposure time of 500ms. In this case, according to the scheme of the present application, the exposure time and the ISO value are adjusted for the second time. On the one hand, the final exposure time is set to 500ms. On the other hand, the first ratio is calculated based on the exposure time after the first adjustment and the limit exposure time: (600-500) / 600=1 / 6, and the first ratio is 1 / 6. For the ISO value, it is adjusted according to the first ratio. By calculation, it can be determined that the adjustment amount is 100*(1 / 6)≈17, and the ISO value after the second adjustment is 100+17=117.

[0057] Through the above scheme, when the exposure time calculated by the exposure compensation amount is greater than the limit exposure time found by looking up the table, by making a secondary adjustment to the exposure time and ISO value, it is possible to avoid motion blur of the image caused by exceeding the optical image stabilization capability.

[0058] In addition, the embodiment of the present application can also perform exposure compensation in the shooting stage based on face detection information to improve the image anti-shake effect. The following is a possible implementation method of performing exposure compensation in the shooting stage based on face detection information.

[0059] In some possible implementations, the method also includes: obtaining face detection information when the camera night scene mode is enabled; determining a third anti-shake control parameter based on the face detection information, different face detection information corresponding to different anti-shake control parameters; and performing anti-shake control and image processing based on the third anti-shake control parameter when a first operation triggered by the user to take a photo is received.

[0060] In some possible implementations, the third anti-shake control parameter includes a second exposure compensation amount. In this case, the anti-shake control and image processing based on the third anti-shake control parameter include: adjusting the first exposure parameter based on the second exposure compensation amount to obtain a second exposure parameter; and acquiring an image using the second exposure parameter to obtain an image sequence in the shooting stage.

[0061] It should be noted that, since the second exposure parameter is adjusted based on the first exposure parameter, that is, the second exposure parameter replaces the first exposure parameter, when the second exposure parameter is used to capture an image, the first exposure parameter is not used to capture an image.

[0062] In some possible implementations, different face detection information corresponds to different exposure compensation amounts. The first exposure parameter includes a first exposure duration and a first ISO value. In this case, the second exposure compensation amount is adjusted based on the first exposure parameter, including: when the face detection information meets the portrait shooting condition, the R gear exposure duration is reduced based on the first exposure duration, and the R gear sensitivity value is increased based on the first ISO value. Wherein, R is greater than zero. When the face detection information does not meet the portrait shooting condition, the second exposure compensation amount is zero.

[0063] Through the above scheme, in the portrait shooting scene of the camera's night scene mode, not only can anti-shake control and image processing be comprehensively performed through the first anti-shake control parameter determined based on the scene brightness information and the second anti-shake control parameter determined based on the jitter information, but the third anti-shake control parameter can also be determined through face detection information, and the third anti-shake control parameter can be used to further perform anti-shake control and image processing, thereby improving the image anti-shake control effect of the portrait shooting scene in the night scene mode and improving the clarity of the night scene portrait photos.

[0064] In some possible implementations, enabling the camera night scene mode includes: acquiring an image sequence in a preview phase; performing scene recognition based on the image sequence in the preview phase; and automatically enabling the camera night scene mode when the scene recognition result meets the camera night scene mode triggering condition.

[0065] In some possible implementations, enabling the camera night mode includes: in response to a second operation of the user turning on the camera night mode, enabling the camera night mode.

[0066] Through the above scheme, whether the user triggers the activation of the camera night scene mode or the electronic device automatically triggers the activation of the camera night scene mode according to the scene recognition result, the image stabilization control method provided in the embodiment of the present application can be used to perform stabilization control and image processing to obtain clearer night scene photos and enhance the user experience.

[0067] In the process of optical image stabilization control and night scene photo generation in night scene shooting scenes, the night scene optical image stabilization control is performed through the application scheme, which can improve the clarity of night scene photos and reduce the noise of night scene photos.

[0068] In a second aspect, the present application provides an image stabilization control device, the device comprising a unit for executing the method in the first aspect. The device may correspond to executing the method described in the first aspect, and the relevant description of the unit in the device can refer to the description of the first aspect, which will not be repeated here for the sake of brevity.

[0069] The method described in the first aspect above can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a processing module or unit, a display module or unit, etc.

[0070] In a third aspect, the present application provides an electronic device, comprising a processor, a computer program or instructions stored in the processor and a memory, wherein the processor is configured to execute the computer program or instructions so that the method in the first aspect is executed.

[0071] In a fourth aspect, the present application provides a computer-readable storage medium on which a computer program (also referred to as instructions or codes) for implementing the method in the first aspect is stored. For example, when the computer program is executed by a computer, the computer can perform the method in the first aspect.

[0072] In a fifth aspect, the present application provides a chip, including a processor. The processor is used to read and execute a computer program stored in a memory to perform the method in the first aspect and any possible implementation thereof. Optionally, the chip also includes a memory, and the memory is connected to the processor via a circuit or wire.

[0073] In a sixth aspect, the present application provides a chip system, including a processor. The processor is used to read and execute a computer program stored in a memory to perform the method in the first aspect and any possible implementation thereof. Optionally, the chip system also includes a memory, and the memory is connected to the processor via a circuit or wire.

[0074] In a seventh aspect, the present application provides a computer program product, which includes a computer program (also referred to as instructions or codes). When the computer program is executed by an electronic device, the electronic device implements the method in the first aspect.

[0075] It can be understood that the beneficial effects of the second to seventh aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 A schematic diagram of an image sequence obtained after image stabilization is turned off and an image sequence obtained after image stabilization is turned on for the same shooting scene provided in an embodiment of the present application;

[0077] Figure 2 A schematic diagram of an image generated after anti-shake processing and an image generated without anti-shake processing for the same shooting scene provided in an embodiment of the present application;

[0078] Figure 3 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0079] Figure 4 A schematic diagram of a side view structure corresponding to a camera module provided in an embodiment of the present application;

[0080] Figure 5 A schematic diagram of the anti-shake system architecture provided in an embodiment of the present application;

[0081] Figure 6 A schematic diagram of the system architecture of an electronic device provided in an embodiment of the present application;

[0082] Figure 7 A flowchart of an image stabilization control method provided in an embodiment of the present application;

[0083] Figure 8 A schematic diagram of an optical image stabilization control loop provided in an embodiment of the present application;

[0084] Fig. 9 A schematic diagram of a flow chart of determining an optical image stabilization working mode of an electronic device provided in an embodiment of the present application;

[0085] Fig.10 A schematic diagram of a process for generating a night scene photo according to an image sequence output in a preview phase and an image sequence output in a shooting phase in a non-high dynamic scene;

[0086] Fig.11 A schematic diagram of a process for generating a night scene photo according to an image sequence output in a preview phase and an image sequence output in a shooting phase in a high dynamic scene;

[0087] Fig.12 A flowchart of an image stabilization control method provided in an embodiment of the present application;

[0088] Fig.13 A schematic diagram of exposure compensation amounts corresponding to various jitter levels provided in an embodiment of the present application;

[0089] Fig.14 A schematic diagram of a specific implementation method of image stabilization control in a night scene mode with medium illumination and non-high dynamic scenes provided by an embodiment of the present application;

[0090] Fig.15 A schematic diagram of a specific implementation method of image stabilization control in a medium illumination, high dynamic scene in night scene mode provided by an embodiment of the present application;

[0091] Fig.16 A schematic diagram of a specific implementation method of image stabilization control for low-illumination scenes in night scene mode provided in an embodiment of the present application;

[0092] Fig.17 A schematic diagram of a specific implementation method of image stabilization control for extremely low illumination scenes in night scene mode provided in an embodiment of the present application;

[0093] Fig.18 A schematic block diagram of an image stabilization control device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0094] The embodiments of the present application are further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0095] First, some terms used in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0096] 1. Sensitivity

[0097] Sensitivity, also known as ISO value, refers to the camera's sensitivity to light. The larger the ISO value, the higher the sensitivity, the brighter the photo will be, and vice versa. However, if the ISO value is too high, the photo noise will also increase.

[0098] 2. Exposure duration

[0099] The exposure time refers to the time it takes for the shutter to open and close. The exposure time will affect the clarity of the photo. The longer the exposure time, the brighter the image, but it is also more likely to be blurred.

[0100] 3. Exposure compensation

[0101] Exposure compensation refers to increasing or decreasing the exposure in the current metering data. In actual implementation, exposure compensation can be achieved by adjusting at least one of the sensitivity ISO, exposure time and aperture. EV (exposure values) is currently used to indicate the amount of exposure. For example, when the sensitivity ISO is 100, the aperture value is F1, and the exposure time is 1 second, the exposure obtained is EV0. +1EV exposure compensation means increasing the exposure by one level, for example, by doubling the exposure time alone, doubling the amount of light entering alone, or doubling the ISO sensitivity alone.

[0102] 4. Long exposure shooting

[0103] Long exposure shooting, also known as slow shutter speed shooting, can obtain more exposure information by extending the exposure time. When users hold the device for a long exposure, the stability of the device and the slight disturbance in the air may cause the photo to be blurred or vibrate, thus affecting the clarity of the photo. In addition, too long exposure time may also increase noise and affect the image quality.

[0104] 5. Variable exposure shooting

[0105] Variable exposure shooting means that the sensor outputs long exposure images and short exposure images at the same time, and forms a high dynamic range (HDR) image by fusing the long and short exposure images. In some shooting scenes with large differences in brightness levels, ordinary CMOS / CCD image sensors cannot fully present the real scene. The brighter areas in the scene will appear white due to overexposure, while the darker areas in the scene will appear dark due to underexposure. In this case, variable exposure shooting can be used, which can present the dark area information while ensuring that the bright area is not overexposed, thus retaining the brightness information.

[0106] 6. Zero shutter lag (ZSL) mode

[0107] When the electronic device turns on the zero shutter lag (ZSL) mode, the camera module will cache the captured image frames in the memory to facilitate the acquisition of the picture captured before the shutter button is pressed. When the shutter button is pressed, the camera can immediately save the previously cached picture, thereby reducing shutter delay. For example, in ZSL mode, the exposure time is relatively short, up to 1 / 10s, about 100ms. In non-ZSL mode, the exposure time is relatively long, which will be greater than 100ms. This makes it more convenient to shoot dynamic scenes or capture moments.

[0108] 7. Motion Blur

[0109] During the shooting process of electronic devices, the generation of each frame of the image relies on the accumulation of photons during the exposure time, and converting them into electrons through photoelectric conversion, and further transforming them into images recognizable to the human eye. During this time, if the electronic device has a large amplitude of movement, the movement information will also be accumulated and recorded, and the generated image will be accompanied by strong motion blur.

[0110] 8. Optical image stabilization (OIS)

[0111] Optical image stabilization can also be called optical image stabilization. OIS technology refers to the detection of the shake of the electronic device through a motion sensor (for example, a gyroscope, an accelerometer) during camera exposure. The OIS controller controls the motor that drives the OIS and moves the lens or image sensor based on the shake data detected by the motion sensor, so that the optical path remains as stable as possible during the entire exposure period, thereby obtaining a clearly exposed image.

[0112] Optical image stabilization includes two types of image stabilization, the first is lens-moving optical image stabilization, and the second is photosensitive element moving optical image stabilization. Among them, the principle of the first lens-moving optical image stabilization is to detect tiny movements through the gyroscope sensor in the lens, and then transmit the signal to the microprocessor. The microprocessor immediately calculates the displacement that needs to be compensated, and then compensates according to the lens shaking direction and displacement through the compensation lens group, thereby effectively overcoming the image blur caused by camera vibration. The second photosensitive element moving optical image stabilization uses image sensor offset to achieve image stabilization. The principle is: first, the CCD is placed on a bracket that can move up, down, left, and right. Then, when the gyroscope sensor detects shaking, the parameters such as the shaking direction, speed, and movement are processed to calculate the CCD movement amount sufficient to offset the shaking.

[0113] With the widespread use of electronic devices, using electronic devices to take photos has become a daily behavior in people's lives. When users use electronic devices to take photos, jitter is inevitable; for example, jitter can refer to hand-held jitter or touch jitter when the user is taking photos, or jitter can refer to the movement of the electronic device during the shooting process. Due to jitter during the shooting process, the video or image captured is blurred, affecting the clarity of the image. Taking the electronic device as a mobile phone as an example, when people hold the mobile phone to take photos, the mobile phone will shake during the shooting process due to their own hand shaking, walking vibration, etc., which will cause blurred images.

[0114] In this regard, the relevant technology implements anti-shake by adding an optical image stabilization OIS controller to the electronic device. The principle of OIS controller anti-shake is to use an OIS motor to drive the lens to move, thereby offsetting the displacement caused by the shake.

[0115] Figure 1 1 is a schematic diagram of an image sequence obtained after image stabilization is turned off and an image sequence obtained after image stabilization is turned on for the same shooting scene provided by an embodiment of the present application. Figure 1 As shown in (a) in the figure, the user holds an electronic device (such as a mobile phone) to take a photo. Figure 1 The image sequence shown in (b) may represent an image sequence acquired by the electronic device after image stabilization is turned off; Figure 1The image sequence shown in (c) in FIG. 1 may represent an image sequence obtained by the electronic device after image stabilization is turned on; Figure 1 (b) and Figure 1 As can be seen from (c) in FIG. 5 , after image stabilization is turned on in the electronic device, the electronic device performs optical stabilization processing during the shooting process, which can improve the stability of the image sequence, avoid large jitters between image frames in the image sequence, and generate a relatively stable image sequence.

[0116] Figure 2 It is a schematic diagram of an image generated after anti-shake processing and an image generated without anti-shake processing for the same shooting scene provided in an embodiment of the present application. Figure 2 (a) in the figure shows an image generated based on an image sequence acquired after image stabilization is turned on, and the image is an image generated after stabilization processing. Figure 2 (b) in FIG. 1 shows an image generated based on an image sequence obtained after image anti-shake is turned off, and the image has not been processed by anti-shake. In comparison, the clarity of the image obtained after the anti-shake processing is higher than that of the image without the anti-shake processing.

[0117] However, for night scene shooting scenes, the current optical image stabilization performance is relatively weak, and there is still blurry imaging, which cannot meet the needs of users who want to take clear night scene photos. The reason for the weak optical image stabilization performance of night scene shooting scenes is that in the current related technologies, due to the limitations of scene perception and optical image stabilization control capabilities, the anti-shake capability of the night scene mode is not enough to support longer exposure times, causing night scene photos to be greatly affected by hand shaking, resulting in image quality problems such as blurry and high noise in night scene photos, affecting the overall experience of the night scene mode.

[0118] Therefore, there is an urgent need for an image stabilization control method that is more suitable for night scene shooting scenes to improve the stabilization effect and enhance the image quality.

[0119] In response to the above problems, the present application combines scene perception, optical image stabilization control and night scene image exposure processing for improvement. In the shooting scene of the camera night scene mode, the first anti-shake control parameter applicable to the current scene can be determined based on the scene brightness information, such as the jitter information filtering method, the image frame output mode and / or the image fusion method, and the second anti-shake control parameter applicable to the current scene can be determined based on the jitter information. For example, the second anti-shake control parameter includes an exposure compensation amount, and different jitter information corresponds to different exposure compensation amounts. The present application scheme can perform comprehensive anti-shake control and image processing based on the first anti-shake control parameter and the second anti-shake control parameter during the shooting stage. For night scene shooting scenes with different jitter intensities, different degrees of exposure compensation are performed during the shooting stage, thereby improving the image anti-shake effect of the night scene mode shooting scene and improving the clarity of the night scene photos.

[0120] The implementation process of the image stabilization control method provided in the embodiment of the present application may include: performing scene perception according to the image sequence output in the preview stage to obtain scene information; then determining the optical stabilization working mode in the shooting stage according to the scene information; then responding to the user's operation of triggering a photo, switching to the optical stabilization working mode in the shooting stage, setting the jitter information filtering mode according to the optical stabilization working mode, setting the image frame output mode, adjusting the exposure parameters, and outputting the image sequence in the shooting stage; then performing image registration and fusion according to the image sequence in the preview stage and / or the image sequence in the shooting stage to obtain a night scene photo (target image).

[0121] When taking photos using the night scene mode of the camera of an electronic device, it is easy to cause blurry photos due to insufficient scene lighting in the night scene. In this case, the image anti-shake control method provided in the embodiment of the present application is used to avoid or alleviate the problem of blurry night scene photos caused by shaking, and the shutter time can be effectively extended, thereby achieving the purpose of obtaining more exposure information, reducing noise in night scene photos and improving clarity.

[0122] The present application scheme can be applied to night scene photo shooting scenes. Optionally, the present application scheme can be applied to shooting scenes where the user triggers the night scene mode to be turned on, and can also be applied to shooting scenes where the electronic device automatically turns on the night scene mode. For the sake of convenience, the shooting scenes where the user triggers the night scene mode to be turned on and the shooting scenes where the electronic device automatically turns on the night scene mode are collectively referred to as night scene shooting scenes.

[0123] Below we first combine the attached Figure 3 The hardware system of the electronic device provided in the embodiment of the present application is described.

[0124] Figure 3 FIG. 1 shows a hardware structure diagram of an electronic device provided in an embodiment of the present application. Figure 3 As shown, the electronic device 100 includes: a camera module 110 , an inertial measurement unit (IMU) 120 , a processor 130 , a memory 140 and a display screen 150 .

[0125] The processor 130 may include one or more processing units. For example, the processor 130 may include at least one of the following processing units: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and a neural-network processing unit (NPU). Among them, different processing units may be independent devices or integrated devices. The controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions.

[0126] The processor 130 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 130 is a cache memory. The memory may store instructions or data that the processor 130 has just used or cyclically used. If the processor 130 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 130, and thus improves the efficiency of the system.

[0127] The memory 140 may store a program, and the program may be executed by the processor 130, so that the processor 130 performs the method provided in the present application. The memory 140 may also store data. The processor 130 may read the data stored in the memory 140. The memory 140 and the processor 130 may be provided separately. Optionally, the memory 140 may also be integrated in the processor 130.

[0128] In the embodiment of the present application, the code for implementing the image stabilization control method described in the embodiment of the present application may be stored in a non-volatile memory. When running the camera application, the electronic device 100 may load the executable code stored in the non-volatile memory into the random access memory.

[0129] In an embodiment of the present application, the processor 130 can be used to open a camera application and enable a camera night scene mode; when the camera night scene mode is enabled, obtain first scene brightness information and first jitter information; determine a first anti-shake control parameter based on the first scene brightness information, the first anti-shake control parameter including a jitter information filtering method, an image frame output mode and / or an image fusion method, and different scene brightness information corresponds to different anti-shake control parameters; determine a second anti-shake control parameter based on the first jitter information, the second anti-shake control parameter including a first exposure compensation amount, and different jitter information corresponds to different exposure compensation amounts; receive a first operation triggered by a user to take a photo; in response to the first operation, perform anti-shake control and image processing based on the first anti-shake control parameter, and perform anti-shake control and image processing based on the second anti-shake control parameter to obtain a target image.

[0130] Among them, the camera module 110 can be used to capture images. Exemplarily, the camera module 110 may include a camera module 111 and an optical image stabilization (OIS) system 112. The camera module 111 can be used to generate an image at a certain moment, and the image can be composed of multiple pixels. The camera module 111 includes a lens and a photosensitive element. In the process of the camera module 111 generating an image, the optical image stabilization system 112 can dynamically adjust the relative position between the photosensitive element and the lens to achieve mechanical image stabilization.

[0131] The electronic device 100 can realize the shooting function through an image signal processor ISP, a camera module 111, a video codec, a GPU, a display screen 150 and an application processor.

[0132] In an embodiment of the present application, an image signal processor ISP is used to process data fed back by the camera module 111. For example, when shooting, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, and the light signal is converted into an electrical signal. The camera photosensitive element transmits the electrical signal to the image signal processor ISP for processing and converts it into an image visible to the naked eye. The image signal processor ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The image signal processor ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the image signal processor ISP can be set in the camera 193.

[0133] In an embodiment of the present application, the camera module 111 is used to capture static images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the image signal processor ISP to convert it into a digital image signal. The image signal processor ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0134] The inertial measurement module 120 can be used to record the rotation and translation of the body of the electronic device 100 in real time, and output the body's posture at a certain moment (i.e., the representation of rotation and translation data). The posture over a period of time is the movement path of the body during this period of time. Exemplarily, the inertial measurement module 120 may include a gyroscope and / or an accelerometer, etc.

[0135] A gyroscope sensor, also called an angular velocity sensor, can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., the x-axis, the y-axis, and the z-axis) can be determined by a gyroscope sensor. The gyroscope sensor can be used in scenes such as anti-shake shooting, navigation, and somatosensory games.

[0136] The acceleration sensor can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally the x-axis, y-axis and z-axis). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. The acceleration sensor can also be used to identify the posture of the electronic device 100 as an input parameter for applications such as horizontal and vertical screen switching and pedometers.

[0137] For ease of description, the following embodiments are described using a gyro sensor as an example.

[0138] In an embodiment of the present application, a gyroscope sensor can be used to collect jitter information, and the jitter information can be used to indicate the change in the posture of the electronic device during shooting. The jitter information can include the rotational angular velocity of the electronic device when it is deflected or tilted. Then, the angle is obtained by performing a discretized integral of the rotational angular velocity.

[0139] In the embodiment of the present application, the gyroscope sensor can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor detects the shaking angle of the electronic device 100, calculates the distance that the camera module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake.

[0140] The display screen 150 is used to display images or videos captured by the camera module 111. The display screen 150 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc.

[0141] The electronic device can realize the image display function through the GPU, the display screen 150, and the application processor. The GPU is a microprocessor for image processing, which connects the display screen 150 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0142] In addition, the electronic device 100 further includes an ambient light sensor. In the embodiment of the present application, the ambient light sensor can sense the ambient light brightness, and the processor 130 can determine whether the current scene is a night scene shooting scene based on the ambient light brightness.

[0143] In addition, the electronic device 100 further includes a touch sensor. In the embodiment of the present application, the electronic device 100 may use the touch sensor to detect the user's click, slide and other operations on the display screen 150. For example, the touch sensor may detect the user's touch operation on the shooting control, and the processor 130 may execute the corresponding anti-shake strategy and image shooting in response to the touch operation.

[0144] It is understandable that this plan Figure 3The illustrated structure does not constitute a specific limitation on the electronic device 100. In other embodiments of the present invention, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0145] It can be understood that in the embodiment of the present application, the electronic device 100 can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, as well as a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device and / or a smart city device, and the embodiment of the present application does not impose any special restrictions on the specific type of the electronic device.

[0146] The camera module of the electronic device is introduced in detail below. Figure 4 A side view structural schematic diagram corresponding to a camera module provided in an embodiment of the present application is shown.

[0147] Exemplarily, a camera module (or camera head) generally includes an optical lens and a photosensitive element (also called an image sensor), etc. The optical lens may include one or more lenses. The optical lens uses the refraction principle of the lens to transmit light so that the light forms an image on the photosensitive element.

[0148] It should be understood that the “lens” mentioned in the present application can be understood as an integral lens, which may include one or more lenses, and the “lens” can be understood as a lens in a lens structure or a lens or lens used to constitute a lens.

[0149] In addition, a filter (not shown in the figure) may be included between the lens and the photosensitive element. The filter is used to filter out unnecessary bands in the light to prevent the photosensitive element from generating false colors or ripples, so as to improve its effective resolution and color reproduction. Of course, this is only an example, and the camera may also include other structures, which are not limited in this application.

[0150] In conjunction with this example, in order to perform optical image stabilization, an OIS controller and an OIS motor can be added to the lens of the camera. The OIS controller is used to obtain the jitter data of the electronic device collected by the gyroscope sensor, such as angular velocity, and generate a control signal for controlling the movement of the OIS motor based on the jitter data collected by the gyroscope sensor. The OIS motor is used to push the lens to move under the control of the control signal, thereby offsetting the displacement caused by the jitter.

[0151] See also Figure 4 Explain how the optical image stabilization of the OIS motor is implemented. Figure 4 Taking the coordinate system shown in as an example, the OIS motor in the OIS controller can push the lens to move left and right along the x-axis, y-axis or z-axis to offset the displacement caused by the shake. It should be understood that the OIS controller can control the lens to move in various directions in combination with the compensation requirements to offset the displacement caused by the shake. The specific direction and distance of movement can be determined according to the requirements, and the embodiments of the present application are not limited thereto.

[0152] It should be noted that the OIS algorithm module can send an anti-shake instruction to the OIS controller, and the OIS controller sends an anti-shake instruction to the OIS motor, and the OIS motor controls the movement of the camera lens according to the anti-shake instruction. Among them, the OIS algorithm module and the OIS controller can be two independent modules or an integrated module.

[0153] Combination Figure 5 The anti-shake system architecture provided by the embodiment of the present application is described. Figure 5 A schematic structural diagram showing the relationship between the gyroscope sensor, OIS controller and lens.

[0154] like Figure 4 As shown, the gyro sensor can be connected to the OIS controller. The gyro sensor obtains the angular velocity of the electronic device shaking and transmits the angular velocity of the electronic device shaking to the OIS controller.

[0155] The OIS controller can be connected to the OIS motor (optical image stabilization motor), and the OIS motor can be connected to the lens. The OIS controller calculates the compensation angle according to the angular velocity of the electronic device shaking, and instructs the OIS motor to adjust the lens angle to stabilize the image.

[0156] The OIS controller can also be connected to an application processor AP / image signal processor ISP, which can be connected to a CMOS sensor, which can be connected to a lens. The OIS controller can call the application processor AP to calculate the compensation angle. The OIS controller can call the image processing signal of the image signal processor ISP to correct the compensation angle.

[0157] The OIS controller can also be connected to a Hall sensor, which is connected to the lens. The Hall sensor feeds back the position information of the lens to the OIS controller, forming a closed-loop control to accurately move the lens to the desired position.

[0158] Among them, the OIS controller can also be called an OIS control driver or an OIS driver chip.

[0159] refer to Figure 5 As shown in the structural diagram, the basic principle of optical image stabilization is: when the gyro sensor is vibrated by external factors, the vibration signal is fed back to the OIS controller, and the OIS controller drives the OIS motor to move the lens, thereby offsetting the image deviation caused by the shaking, ensuring that the camera can still maintain stable imaging in a shaking environment.

[0160] The hardware system of the electronic device 100 is described in detail above, and the software system of the electronic device 100 is described below. A software operating system runs on the hardware system, and the software operating system can be any one or more computer operating systems that implement business processing through processes. Applications can be installed and run on the software operating system.

[0161] Figure 6 Schematic diagram of the system architecture of the electronic device provided in the embodiment of the present application. Figure 6 As shown, the system architecture may include, from top to bottom, an application layer 210 , an application framework layer 220 , a hardware abstraction layer (HAL) 230 , a driver layer 240 , and a hardware layer 250 .

[0162] The application layer 210 may include a series of application packages. In the embodiment of the present application, the application package may include a camera application, a gallery, and the like.

[0163] The application framework layer 220 provides an application programming interface (API) and a programming framework for application programs of the application layer; the application framework layer may include some predefined functions.

[0164] In an embodiment of the present application, the application framework layer 220 may include a camera access interface; the camera access interface is used to provide an application programming interface and a programming framework for the camera application. The camera access interface may include camera management and camera devices. Among them, camera management can be used to provide an access interface for managing the camera; the camera device can be used to provide an interface for accessing the camera.

[0165] The hardware abstraction layer 230 is an interface layer located between the application framework layer and the driver layer, and provides a virtual hardware platform for the operating system.

[0166] In the embodiment of the present application, the hardware abstraction layer 230 includes a camera hardware abstraction layer and a camera algorithm. The camera hardware abstraction layer can call the camera algorithm; the camera algorithm can include a software algorithm for image processing. Among them, the camera hardware abstraction layer 230 can provide virtual hardware for the camera device.

[0167] Exemplarily, the camera algorithm library may include the operating code and data of the image stabilization control method (or referred to as OIS algorithm) provided in the embodiment of the present application.

[0168] Exemplarily, the algorithm in the camera algorithm may refer to a code that does not rely on specific hardware implementation; for example, a code that can generally be run in a CPU, etc.

[0169] The driver layer 240 is a layer between hardware and software. The driver layer includes various hardware drivers and is used to provide drivers for different hardware devices.

[0170] In an embodiment of the present application, the driver layer may include a camera device driver, a digital signal processor driver, and an image processor driver. Among them, the camera device driver (e.g., a camera driver) is used to drive the camera sensor to collect images and drive the image signal processor to pre-process the image. The digital signal processor driver is used to drive the digital signal processor to process images. The image processor driver is used to drive the graphics processor to process images.

[0171] The hardware layer 250 is located at the bottom layer of the operating system. Figure 6 As shown, the hardware layer 250 may include a camera, a display screen, and an OIS motor.

[0172] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to electronic devices based on operating systems such as iOS or Windows.

[0173] The following is an illustrative description of the workflow of the software system and the hardware system of the electronic device 100 in conjunction with a photo-taking scenario.

[0174] When a user performs a touch operation on the touch sensor, the corresponding hardware interrupt is sent to the kernel layer, and the kernel layer processes the touch operation into a raw input event, which includes information such as touch coordinates and the timestamp of the touch operation. The raw input event is stored in the kernel layer, and the application framework layer obtains the raw input event from the kernel layer, identifies the control corresponding to the raw input event, and notifies the application (APP) corresponding to the control. For example, the above touch operation is a single-click operation, and the APP corresponding to the above control is a camera APP. After the camera APP is awakened by the single-click operation, the camera driver of the kernel layer can be called through the API, and the camera module can be controlled by the camera driver to shoot.

[0175] Taking the photo scene as an example, in response to the user's operation of opening the camera application, such as clicking the camera application icon, the camera application calls the camera access interface of the application framework layer to start the camera application, and then sends an instruction to start the camera by calling the camera device (camera device 1 and / or other camera devices) in the camera hardware abstraction layer. The camera hardware abstraction layer sends the instruction to the camera device driver of the kernel layer. The camera device driver can start the corresponding camera sensor and collect image light signals through the sensor. A camera device in the camera hardware abstraction layer corresponds to a camera sensor in the hardware layer.

[0176] Then, the camera sensor can transmit the collected image light signal to the image signal processor for preprocessing to obtain the image electrical signal (raw image), and transmit the above raw image to the camera hardware abstraction layer through the camera device driver.

[0177] The camera hardware abstraction layer can send the original image to the camera algorithm library. The camera algorithm library stores the program code for implementing the image anti-shake control method provided in the embodiment of the present application. Based on the digital signal processor and the image processor, the camera algorithm library executes the above code. In the shooting scene of the camera night scene mode, the first anti-shake control parameter applicable to the current scene can be determined based on the scene brightness information, such as the jitter information filtering method, the image frame mode and / or the image fusion method, and the second anti-shake control parameter applicable to the current scene can be determined based on the jitter information. For example, the second anti-shake control parameter includes an exposure compensation amount, and different jitter information corresponds to different exposure compensation amounts. In the shooting stage, comprehensive anti-shake control and image processing are performed based on the first anti-shake control parameter and the second anti-shake control parameter. For night scene shooting scenes with different jitter intensities, different degrees of exposure compensation are performed in the shooting stage to generate night scene photos.

[0178] The camera algorithm library can send night scene photos to the camera hardware abstraction layer. Then, the camera hardware abstraction layer can display the night scene photos and store them in the gallery.

[0179] The embodiment of the present application provides an image stabilization control method and an electronic device, which improves the software of the electronic device. In the shooting scene of the camera night scene mode, comprehensive stabilization control and image processing are performed based on the first stabilization control parameter and the second stabilization control parameter during the shooting stage. For night scene shooting scenes with different tremor intensities, different degrees of exposure compensation are performed during the shooting stage, thereby improving the image stabilization effect of the night scene mode shooting scene and improving the clarity of the night scene photos.

[0180] The execution subject of the image stabilization control method provided in the embodiment of the present application can be the above-mentioned electronic device, or it can be a functional module and / or functional entity in the electronic device that can implement the image stabilization control method, and the present application scheme can be implemented by hardware and / or software, which can be determined according to actual use requirements and is not limited by the embodiment of the present application. The following takes an electronic device as an example and combines the accompanying drawings to exemplarily illustrate the image stabilization control method provided in the embodiment of the present application.

[0181] The following will be combined with the accompanying drawings to illustrate the embodiments of the present application through the following multiple exemplary embodiments. The methods in the following embodiments can be implemented in an electronic device with the above hardware structure and software architecture. The hardware structure diagram of the electronic device can be as follows: Figures 3 to 5 As shown, the software structure diagram of the electronic device can be as follows Figure 6 For the convenience of explanation, the embodiments of the present application are described with the electronic device as the execution subject.

[0182] The image stabilization control method provided by the embodiment of the present application is described below in conjunction with specific embodiments.

[0183] In the embodiment of the present application, the working phase of the camera module may include a preview phase and a shooting phase. After the user triggers shooting, the camera module switches from the preview phase to the shooting phase.

[0184] In the embodiment of the present application, the camera module can acquire a preview image sequence in ZSL mode during the preview phase, and can acquire an image sequence in ZSL mode or non-ZSL mode during the shooting phase.

[0185] For ease of explanation, the preview image sequence output by the camera module in the preview phase is collectively referred to as the front frame image or the image sequence in the preview phase, and the image sequence output by the camera module in the shooting phase is collectively referred to as the rear frame image or the image sequence in the shooting phase.

[0186] In the embodiment of the present application, since the user holds the electronic device to shoot, there are various possible shakes, whether in preview mode or shooting mode, which will affect the image clarity. In other words, anti-shake processing is required in both preview mode and shooting mode.

[0187] Figure 7 is a flow chart of the image anti-shake control method provided by the embodiment of the present application. Figure 7 As shown, the image anti-shake control method includes the following steps S301-S304.

[0188] S301 , performing scene recognition according to the image sequence output in the preview stage to obtain scene information.

[0189] In an embodiment of the present application, the scene information includes scene brightness information, jitter information and scene perception information.

[0190] In an embodiment of the present application, after the electronic device starts the camera program, the electronic device enters the preview stage. The image sensor outputs the original image sequence, which is processed by the hardware ISP to obtain the processed preview image sequence, and each frame image of the preview image sequence is downsampled, for example, the resolution of the image is reduced to VGA resolution, and the image sequence obtained after downsampling is output as a small-size image (tiny) stream. The electronic device uses statistical algorithms and perception algorithms to process the tiny stream to obtain scene brightness information, jitter information and scene perception information.

[0191] The scene brightness information (denoted as Lux_Idx) can be obtained based on automatic exposure statistics and automatic exposure algorithms.

[0192] Among them, the mobile phone jitter information (denoted as Gyro_Info) comes from the IMU sensor.

[0193] The scene perception information includes scene classification information. For example, it can be determined whether the current scene is a night scene shooting scene based on the scene classification information. For example, when the scene classification information is Scene_Idx, it indicates that the current scene is a night scene shooting scene.

[0194] In the embodiment of the present application, different scene perception information Scene_Idx may represent different types of shooting scenes. For example, different types of scenes may include sunny shooting scenes, cloudy shooting scenes, night scene shooting scenes, food shooting scenes, indoor shooting scenes, pet shooting scenes, etc.

[0195] The scene perception information may also include face detection information (denoted as Face_detect_info) obtained by processing the tiny stream with a face detection algorithm. For example, it may be determined whether the current scene is a portrait night scene based on the face detection information.

[0196] S302: Determine an optical image stabilization working mode during the shooting phase according to scene information.

[0197] It should be noted that since the optical image stabilization system has a good effect on reducing the shaking and blurring of handheld photography scenes, the optical image stabilization system is widely used in mobile phones. Among them, the optical image stabilization system has several forms, for example, the optical image stabilization system can adopt a lens movement method, or a photosensitive element movement method. The control of the optical image stabilization system also has many forms, for example, the optical image stabilization system can adopt a control method based on a driver chip (Driver IC) or a control method based on a coprocessor (Sensor Hub).

[0198] In practical applications, the most important thing for a high-quality optical image stabilization system, in addition to its optical and mechanical design, is the control system, that is, how to achieve optical image stabilization control through the OIS controller.

[0199] Figure 8 A schematic diagram of an optical image stabilization control loop implemented based on an OIS controller provided in an embodiment of the present application is shown.

[0200] On the one hand, see Figure 8 In ①, during the shooting process, the Hall element detects the lens position information, and after the lens position information is processed by the amplifier, it is fed back to the operator. It should be noted that the lens position information indicates the lens position when the electronic device is shaken. In some embodiments, the offset of the lens position relative to the initial lens position when the electronic device is shaken is defined as a positive offset (+).

[0201] On the other hand, see Figure 8 ②, during the shooting process, the gyroscope sensor detects the jitter information of the electronic device, and the jitter information is the angular velocity of rotation when the electronic device is deflected or tilted. The angle is obtained after the angular velocity of rotation is discretized and integrated by the integrator. The angle can be the offset angle around the X / Y axis. The gyroscope sensor feeds back the offset angle around the X / Y axis to the OIS controller, and then the OIS controller feeds back the offset angle around the X / Y axis to the operator. In some embodiments, when the electronic device shakes, the offset around the X / Y axis fed back by the OIS controller relative to the initial lens position is defined as a negative offset (-).

[0202] It can be understood that the operator obtains the current lens position and the offset angle of the lens around the X / Y axis when the electronic device is shaken. The operator can calculate the distance that the camera module needs to compensate based on the offset angle, and the OIS controller can control the lens to offset the compensation distance in the reverse direction based on the current lens position to offset the shake of the electronic device and achieve anti-shake.

[0203] Then, see Figure 8In ③, the OIS controller drives the actuator to drive the OIS motor to move the lens in the opposite direction, thereby offsetting the image deviation caused by the shake, ensuring that the camera can still maintain stable imaging in a shaking environment.

[0204] It should be noted that in different shooting scenes and different jitter conditions, there are different requirements for the signal processing and anti-shake control methods of the optical image stabilization system. For example, in a city night scene shooting scene with good illumination, the base exposure time is relatively short, and the requirements for optical image stabilization are relatively low, while there are higher requirements for preview real-time and hand tracking. For another example, in a low-illuminance night scene shooting scene, the base exposure time is relatively long, and the requirements for optical image stabilization are high, while there are lower requirements for preview real-time and hand tracking.

[0205] In the embodiment of the present application, according to different needs, the parameters of the optical image stabilization system can be adaptively adjusted to a certain extent in combination with scene brightness information, jitter information and scene perception information to better adapt to the image stabilization requirements of each scene.

[0206] It should be noted that the control signal of the optical image stabilization system is obtained by signal processing based on the detection signal of the gyro sensor. For the sake of convenience, the detection signal of the gyro sensor is called the gyro signal. Before using the gyro signal, it is necessary to perform a series of filtering processes on the gyro signal to retain the effective information of the gyro signal and eliminate noise.

[0207] For example, for photo preview or short exposure photography scenarios, the optical image stabilization system needs to perform bandpass filtering on the gyroscope signal to prevent the low-frequency response part from affecting the response speed and tracking performance of photo preview and short exposure photography.

[0208] For another example, for long exposure shooting scenes, the optical image stabilization system needs to retain the low-frequency components of the gyroscope signal to better compensate for low-frequency vibrations such as human breathing, partially giving up some of the hand-tracking experience.

[0209] Combining the above scene brightness information, jitter information and scene perception information, the optical image stabilization working mode during the shooting phase can be determined by pre-calibration combined with prior information or real-time judgment combined with an algorithm.

[0210] Fig. 9 FIG. 1 is a schematic diagram showing a flow chart of an electronic device determining an optical image stabilization working mode provided by an embodiment of the present application. Fig. 9As shown, the night scene algorithm module includes a night scene anti-shake decision module and a night scene shooting decision module. In the embodiment of the present application, the night scene anti-shake decision module is a decision module for controlling the anti-shake device. The night scene anti-shake decision module can determine the working mode of the anti-shake device and the filtering processing method of the jitter information in combination with the scene perception information, jitter information and scene brightness information. The night scene shooting decision module can determine the frame mode switching, exposure compensation algorithm and image fusion algorithm in combination with the scene brightness information and dynamic range information.

[0211] First, after the camera program is started, the electronic device obtains scene perception information, filtered jitter information and scene brightness information, and passes this information to the night scene anti-shake decision module. For example, the filtered jitter information Gyro_Info, scene brightness information Lux_Idx, scene perception information Scene_Idx and face detection information Face_detect_info are passed to the night scene anti-shake decision module.

[0212] Then, the night scene anti-shake decision module determines whether to switch the optical anti-shake working mode according to the scene perception information Scene_Idx, the jitter information Gyro Filter, the scene brightness information Lux_Idx and the face detection information Face_detect_info.

[0213] It should be noted that in an embodiment of the present application, the electronic device can determine whether the current scene is a night scene shooting scene based on the scene perception information Scene_Idx. For example, when the scene perception information Scene_Idx == Night_Scene, it is determined that the current scene is a night scene shooting scene. In the case where the current scene is determined to be a night scene shooting scene, based on the scene brightness information Lux_Idx, the filtered jitter information Gyro_Info and the face detection information Face_detect_info, optical image stabilization working mode switching and other processing are performed. It can be understood that in the case where the current scene is determined to be a non-night scene shooting scene, optical image stabilization working mode switching and other processing are not performed.

[0214] Finally, the night scene stabilization decision module passes the determination result of the optical stabilization working mode to the OIS HAL layer driver, which is then passed by the OIS HAL layer driver to the OIS bottom layer driver, and then by the OIS bottom layer driver to the OIS motor, and then the OIS motor switches the optical stabilization working mode.

[0215] In some embodiments, the optical image stabilization working mode can be switched in response to a switching event from the ZSL mode to the non-ZSL mode. The optical image stabilization working mode includes the working mode of the anti-shake device, whether the jitter information filtering mode is switched, whether the frame output mode is switched, the exposure compensation algorithm, the image fusion algorithm, etc. Exemplarily, in the shooting stage, when switching from the ZSL mode to the non-ZSL mode, the night scene anti-shake decision module switches the filtering processing method for the jitter information. For example, in the ZSL mode, the jitter information is processed by bandpass filtering, and in the non-ZSL mode, the jitter information is processed by low-pass filtering.

[0216] The following describes how to select a filtering method for jitter information in a ZSL mode or a non-ZSL mode in an embodiment of the present application.

[0217] In some embodiments, in ZSL mode, the exposure time is relatively short, up to 1 / 10s, about 100ms; at this time, a bandpass filter is used to filter the gyroscope signal to avoid excessive anti-shake of the preview image caused by the introduction of low-frequency and step signals, which can improve the tracking feel of the preview movement.

[0218] In some embodiments, in non-ZSL mode, the exposure time is relatively long, greater than 100ms. At this time, a low-pass filter is used to filter the gyroscope signal to introduce more low-frequency signals into the optical image stabilization control, mainly to solve the low-frequency jitter problem caused by human natural breathing during long exposure shooting, which can improve the clarity of long exposure imaging.

[0219] In some embodiments, in the special ZSL mode, if there is a need to enter the super optical image stabilization working mode that supports low frequency, the mode can be switched by combining relevant information and the night scene image stabilization decision module. It should be noted that 1 to 2 frames will be unavailable during the mode switching process, which will affect the preview effect and performance.

[0220] The following describes a process of switching between the ZSL mode and the non-ZSL mode according to lighting conditions in an embodiment of the present application.

[0221] In the embodiment of the present application, the electronic device can switch between the ZSL mode and the non-ZSL mode according to the illumination condition. In which, a lower threshold Thd_w1 and an upper threshold Thd_w2 are assumed to be used for judging the illumination condition.

[0222] When the illumination condition is Lux_Idx>Thd_w2, the night scene shooting decision module determines that the current scene is an extremely low illumination scene. The night scene shooting decision module can switch from the ZSL mode to the non-ZSL mode.

[0223] When the illumination condition is Thd_w1<Lux_Idx≤Thd_w2, the night scene shooting decision module determines that the current scene is a low illumination scene. The night scene shooting decision module can fuse the ZSL frame image and the non-ZSL frame image to obtain the target image.

[0224] When the lighting condition is Lux_Idx≤Thd_w1, the night scene shooting decision module determines that the current scene is a medium-high illumination scene. In some embodiments, if the current scene is a medium-high illumination scene but not a high dynamic scene, the night scene shooting decision module only performs ZSL frame output. In other embodiments, if the current scene is a medium-high illumination scene and a high dynamic scene, the night scene shooting decision module can use variable exposure shooting to improve the dynamic range.

[0225] It should be noted that the Lux_Idx value is inversely proportional to the scene brightness. For example, the larger the Lux_Idx value, the lower the scene brightness; the smaller the Lux_Idx value, the higher the scene brightness. Here, Lux_Idx is used as an example to represent the scene brightness. In actual implementation, other methods can also be used to represent the scene brightness, and the embodiments of the present application are not limited to this.

[0226] The following describes the image stabilization control implemented by the embodiment of the present application in the night scene shooting scene according to the scene brightness information. The image stabilization control here includes the control of the tremor information filtering method, the image frame output mode and the image fusion method.

[0227] First of all, it should be noted that the image framing mode includes a ZSL mode and a non-ZSL mode. Outputting an image in the ZSL mode can be called a ZSL framing mode, and outputting an image in the non-ZSL mode can be called a non-ZSL framing mode. An image output in the ZSL mode can be called a ZSL framing image, and an image output in the non-ZSL mode can be called a non-ZSL framing image.

[0228] In some embodiments, when Lux_Idx>Thd_w2, it is determined that the current scene is an extremely low illumination scene. In this case, all image sequences in the shooting phase are used to generate night scene photos, and the shooting phase adopts the non-ZSL mode. In addition, when switching from the ZSL mode to the non-ZSL mode, the electronic device sets the filtering mode for the jitter information to the low-pass filtering mode GYRO_FILTER_LOWPASS_MODE. When the non-ZSL frame is completed and the preview is restarted, the electronic device sets the filtering mode for the jitter information to the band-pass filtering mode GYRO_FILTER_BANDPASS_MODE.

[0229] In some embodiments, when Thd_w2≥Lux_Idx>Thd_w1, it is determined that the current scene is a low illumination scene. In this case, a ZSL frame-out image and a non-ZSL frame-out image are obtained, and the ZSL frame-out image and the non-ZSL frame-out image are fused to obtain a target image. That is, the ZSL mode is adopted in the preview stage and the non-ZSL mode is adopted in the shooting stage. Moreover, when switching from the ZSL mode to the non-ZSL mode, the electronic device sets the filtering mode for the jitter information to the low-pass filtering mode GYRO_FILTER_LOWPASS_MODE. When the non-ZSL frame is completed and the preview is restarted, the electronic device sets the filtering mode for the jitter information to the high-pass filtering mode GYRO_FILTER_BANDPASS_MODE.

[0230] In an embodiment of the present application, the night scene anti-shake decision module can determine the exposure compensation amount in the shooting stage according to the filtered jitter information Gyro_Info.

[0231] The exposure parameters involved in the embodiments of the present application are first introduced below.

[0232] In an embodiment of the present application, in a night scene shooting scenario, exposure parameters such as exposure time and / or sensitivity ISO can be adjusted to adjust the exposure during the shooting process.

[0233] For exposure time, multiple gears can be provided. Exemplarily, the gears of exposure time (also called shutter time) include 1 second, 1 / 2 second, 1 / 4 second, 1 / 8 second, 1 / 15 second, 1 / 30 second, 1 / 60 second, 1 / 125 second, 1 / 250 second, 1 / 500 second, 1 / 1000 second, etc. The exposure amount can be adjusted by raising or lowering the gear of exposure time, for example, if the exposure time is doubled, the exposure amount is doubled. It can be understood that the exposure time can also have other gear settings.

[0234] For the ISO sensitivity, multiple gears are also provided. For example, the ISO sensitivity gears may include 100, 200, 400, 800, 1600, 3200, 6400, etc. The exposure can be adjusted by increasing or decreasing the ISO sensitivity gear. For example, adjacent gears differ by one time in value and also by one time in exposure. It is understood that the ISO sensitivity may also have other gear settings.

[0235] It should be noted that the electronic device presets a reference exposure time and a reference sensitivity ISO. In the preview stage and the shooting stage, the electronic device can capture an image sequence according to the reference exposure time and the reference sensitivity ISO. In the embodiment of the present application, the exposure time and / or the sensitivity ISO value can be adjusted under different shaking intensities to improve the image anti-shake effect.

[0236] The following further describes possible implementations of determining the jitter intensity level according to the filtered jitter information and adjusting the exposure duration and / or the ISO value according to the jitter intensity level.

[0237] Exemplarily, the embodiments of the present application provide the following five possible situations.

[0238] Case 1: When the filtered shake information is greater than the second shake threshold, that is, Gyro_Info>Thd_gyro_shake_w2, it is determined that the current shake amplitude / intensity belongs to a large shake situation. In this case, according to the preset parameters corresponding to the large shake situation, the offset / adjustment amount of the reference exposure time in the shooting stage is set. For example, the exposure time is reduced by 2 stops, and the ISO value is increased by 2 stops.

[0239] Case 2: When the filtered jitter information is less than or equal to the second jitter threshold and greater than the first jitter threshold, that is, Thd_gyro_shake_w2≥Gyro_Info>Thd_gyro_shake_w1, it is determined that the current jitter amplitude / intensity belongs to the medium amplitude jitter situation. In this case, the exposure time offset of the subsequent frame is set according to the preset parameters corresponding to the medium jitter situation. For example, the exposure time is reduced by 1 stop and the ISO value is increased by 1 stop.

[0240] Case 3: When the filtered jitter information is less than or equal to the first jitter threshold, that is, Gyro_Info≤Thd_gyro_shake_w1, it is determined that the current jitter amplitude / intensity belongs to a small amplitude jitter situation. In this case, the post-frame reference exposure duration offset is set according to the preset parameters corresponding to the small amplitude jitter situation. For example, in some embodiments, the exposure duration is not reduced and the ISO value remains unchanged. In other embodiments, the exposure duration is increased by 1 to 2 stops, and the sensitivity ISO value is reduced by 1 to 2 stops.

[0241] Case 4: When the filtered jitter information is close to 0, that is, Gyro_Info≈0, it is determined that the current jitter amplitude / intensity belongs to the tripod situation. In this case, the post-frame reference exposure offset is set according to the tripod preset parameters. For example, in some embodiments, the exposure time is increased by 2 or more stops, and the ISO value is reduced by 2 or more stops. In other embodiments, when the filtered jitter information is close to 0, it is also possible to choose to continue to execute the default optical image stabilization working mode without switching the optical image stabilization working mode.

[0242] Case 5: When the face detection information meets the preset face detection conditions, such as is_face_detect = 1, the current scene is determined to be a night-time portrait shooting scene. In this case, based on the exposure compensation corrected based on the jitter situation, the exposure time of the later frame image is further reduced by 0.5 to 1 stop, and the ISO value is increased by 0.5 to 1 stop to better capture night-time portrait images and improve the success rate.

[0243] The first jitter threshold Thd_gyro_shake_w1 is smaller than the second jitter threshold Thd_gyro_shake_w2, and the first jitter threshold Thd_gyro_shake_w1 is greater than zero.

[0244] The following describes the process of how to perform secondary adjustment when the adjusted exposure time exceeds the limit exposure time.

[0245] In the embodiment of the present application, the corresponding limit exposure time can be pre-calibrated for different jitter strengths. In other words, a correspondence between the jitter strength level and the limit exposure time is established, for example, the correspondence between the jitter strength level and the limit exposure time is reflected by a limit exposure time table.

[0246] In the embodiment of the present application, by combining the anti-shake capability of the optical anti-shake system and the pre-calibrated maximum exposure time, the maximum exposure time at different anti-shake intensities can be obtained.

[0247] In some embodiments, after the above exposure time offset calculation is completed, the limit exposure time OIS_Max_Exposure_Time matching the jitter information Gyro_Info is searched in the limit exposure time table according to the jitter information Gyro_Info.

[0248] If the exposure time obtained after the above exposure time offset calculation is greater than the maximum exposure time OIS_Max_Exposure_Time, the exposure time is adjusted to the maximum exposure time, and the sensitivity ISO value is adjusted proportionally to avoid motion blur of the image due to exceeding the optical image stabilization capability.

[0249] For example, assuming that the maximum exposure time OIS_Max_Exposure_Time is 500ms, the exposure compensation determined according to the jitter intensity level this time is to increase the exposure time and reduce the ISO value. For example, the exposure time after the first adjustment is 600ms (for example, 300ms+300ms=600ms), and the ISO value after the first adjustment is 100. The exposure time of 600ms after the first adjustment is greater than the maximum exposure time of 500ms, which is equivalent to a reduction of 1 / 6 in the exposure time after the first adjustment. In this case, according to the scheme of the present application, the exposure time and ISO value are adjusted twice, for example, the exposure time finally adopted is set to the maximum exposure time of 500ms; accordingly, the adjustment amount of the sensitivity ISO is calculated proportionally, for example, since the exposure time is reduced by 1 / 6, then the ISO value after the first adjustment is increased by 1 / 6 accordingly, that is, 100+100*(1 / 6)≈117, so the sensitivity ISO value can be set to 117.

[0250] It should be noted that the above thresholds, gears and maximum exposure durations can be set according to actual usage requirements. For example, the above thresholds, gears and maximum exposure durations can be adjusted to preset empirical values ​​in combination with the sensor's photosensitivity and night scene algorithm capabilities, and can be flexibly adjusted according to actual needs to obtain the best effect.

[0251] S303, in response to the user triggering the photo-taking operation, switching to the optical image stabilization working mode of the shooting stage, setting the jitter information filtering mode according to the optical image stabilization working mode of the shooting stage, setting the image frame output mode, adjusting the exposure parameters, and outputting the image sequence of the shooting stage.

[0252] In some embodiments, the user triggering the photo taking operation may be the user clicking a photo taking button. In some embodiments, the user triggering the photo taking operation may be the user triggering the photo taking operation by voice. In some embodiments, the user triggering the photo taking operation may be the user triggering the photo taking operation by gesture. In some embodiments, the user triggering the photo taking operation may be the user triggering the photo taking operation by setting a countdown. In actual implementation, the user triggering the photo taking operation may also be other forms of operation, which is not limited in the embodiments of the present application.

[0253] It should be noted that in the embodiment of the present application, considering that mode switching will affect performance, for example, one or two frames of images will be lost when switching modes, and when switching from the preview stage to the long exposure shooting stage, one to two frames of time will be added on the basis of the frozen screen, which will affect the overall experience. In the embodiment of the present application, the mode switching is not performed during the night scene preview process, but the mode switching is performed when the night scene is taken.

[0254] In the embodiment of the present application, switching the optical image stabilization working mode and outputting images according to the optical image stabilization working mode includes: completing the setting of the filtering method and completing the setting of the exposure parameters, then performing image exposure acquisition according to the set exposure parameters, filtering the anti-shake information using the set filtering method during the image exposure process, and performing anti-shake processing according to the filtered shake information. Then, the image sequence is output in the shooting stage.

[0255] Specifically, in response to the user pressing the photo button, the electronic device reconfigures the camera preview data stream to complete the output of the post-frame data required by the night scene image fusion algorithm. In this process, the filtering processing mode used for the jitter information is set, such as the bandpass filtering mode GYRO_FILTER_BANDPASS_MODE or the low-pass filtering mode GYRO_FILTER_LOWPASS_MODE or other modes.

[0256] It should be noted that, for the 1-2 frame time required for mode switching, the previous image information can be used for frozen screen display, which can avoid the data in the switching process from being echoed to the preview interface.

[0257] At the same time, the automatic exposure algorithm sends the exposure time combination required for the shooting phase to the image sensor, preparing for the exposure processing of the subsequent frame image.

[0258] In some embodiments, the image sequence output during the shooting phase may include a long frame image with a long exposure time (referred to as a long frame L). The purpose of outputting the long frame L is to obtain more night scene exposure information.

[0259] Optionally, the image sequence outputted in the shooting stage may also include normal exposure frame images (referred to as normal exposure frames N for short) and short frame images with short exposure duration (referred to as short frames S for short).

[0260] The purpose of outputting the short frame S is to provide a better highlight suppression effect for highlight areas in scenes such as urban night scene shooting.

[0261] The purpose of outputting N frames is to perform multi-frame noise reduction fusion processing in conjunction with the ZSL N frames output in the preview stage.

[0262] It should be noted that the specific number of frames and exposure order of long frames L, short frames S and normal exposure frames N can be determined according to actual usage requirements, and are not limited in comparison with the embodiments of the present application.

[0263] It should also be noted that, for the sake of convenience, the image frame output in the preview stage is referred to as a normal exposure frame. The exposure time of the long frame N is greater than the exposure time of the normal exposure frame N. The exposure time of the short frame S is less than the exposure time of the normal exposure frame N.

[0264] During the shooting phase, the optical image stabilization working mode is switched first (i.e., the filter mode and exposure parameters are set), and then images are captured and image sequences are output according to the set exposure parameters. During the image exposure process, the set optical image stabilization working mode is used for image stabilization processing, thereby avoiding image blurring caused by the shaking of the mobile phone during the exposure time. This ensures that in the night scene mode, after the photo is triggered, the image sequence can be output according to the determined optical image stabilization working mode.

[0265] S304: Generate a target image according to the image sequence in the preview phase and / or the image sequence in the shooting phase.

[0266] In some embodiments, the image sequence output in the preview stage may be N, N, N, N, N.

[0267] In some embodiments, in a non-high dynamic scene, the image sequence output during the shooting phase may be N, L.

[0268] In some embodiments, under high dynamic conditions, the image sequence output during the shooting phase may be S, N, S, L.

[0269] It should be noted that the above image sequence is for exemplary purposes only. In actual implementation, the number of frames is not limited to the above scheme, but may be other numbers. Moreover, the combination of frames is not limited to the above scheme, but may be other combinations of long and short frames.

[0270] In an embodiment of the present application, according to the optical image stabilization working mode in the shooting stage, the image sequence in the preview stage and / or the image sequence in the shooting stage are image registered and fused to generate a night scene photo (target image).

[0271] In some embodiments, key frames (referred to as selected frames) are selected from the image sequence output in the preview stage (front frame images) and the image sequence output in the shooting stage (back frame images), and are processed through night scene image registration and fusion algorithms to obtain night scene photos (i.e., target images).

[0272] In the embodiment of the present application, key frames can be selected according to the image clarity in the image sequence and the time interval from the moment when the user presses the shutter button. For example, the clearer the image frame, the higher the corresponding score; the closer the image frame is to the moment when the shutter button is pressed, the higher the corresponding score.

[0273] In some embodiments, for each image in the image sequence, the scores corresponding to the image clarity and the time interval from the moment the shutter is pressed are weightedly summed, and the normally exposed frame N with the highest score in the image sequence is determined as the key frame.

[0274] In some embodiments, if due to jitter, the clarity of the normal exposure frame N output in the preview stage is lower than the normal exposure frame N output after anti-shake processing in the shooting stage, then the key frame can be replaced from the normal exposure frame N output in the preview stage to the normal exposure frame N output after anti-shake processing in the shooting stage to improve the clarity of the key frame of the night scene photo.

[0275] In an embodiment of the present application, after determining the key frame, the other image frames in the image sequence except the key frame are aligned with the key frame based on the key frame. After the image alignment is completed, the images in the image sequence are fused to obtain a night scene photo.

[0276] Fig.10 A schematic diagram of a process of generating a night scene photo according to an image sequence outputted in a preview phase and an image sequence outputted in a shooting phase in a non-high dynamic scene is shown.

[0277] like Fig.10 As shown, the image sequence output in the preview stage is N, N, N, N, N. The electronic device obtains scene perception information (the night scene mode can be enabled when the scene perception information meets the triggering conditions of the night scene mode), scene brightness information, dynamic range information, etc. according to the image sequence output in the preview stage, and then determines that the current scene is a night scene and a non-high dynamic scene based on this information. When the user triggers the photo, it switches from the preview stage to the shooting stage. In response to the user operation, the electronic device switches to the optical image stabilization working mode of the shooting stage, that is, the filtering mode is switched from the bandpass filtering mode to the low-pass filtering mode, and the exposure parameters such as the exposure time and the sensitivity OIS are adjusted according to the jitter intensity, and the image is captured according to the set exposure parameters and the image sequence is output. For example, the image sequence output in the shooting stage is N, L, ensuring that in the night scene and non-high dynamic scene, after the photo is triggered, a long exposure image sequence can be output according to the determined optical image stabilization working mode. Then, the night scene shooting decision module performs key frame selection, multi-frame image registration, multi-frame noise reduction and fusion based on the image sequence N, N, N, N, N output in the preview stage and the long exposure image sequence N, L output in the shooting stage to obtain the night scene photo.

[0278] like Fig.10 As shown, after the photo is taken, the shooting stage is switched to the preview stage, and then the optical image stabilization working mode of the preview stage is switched back to the optical image stabilization working mode of the preview stage: the filtering mode is switched from the low-pass filtering mode to the band-pass filtering mode, and the exposure parameters such as the exposure time and the sensitivity OIS are restored to the reference exposure parameters, and images are collected and output according to the reference exposure parameters. That is, the preview stage continues to output the image sequence of the normal exposure frame N.

[0279] Fig.11A schematic diagram of a process of generating a night scene photo according to an image sequence outputted in a preview phase and an image sequence outputted in a shooting phase in a high dynamic scene is shown.

[0280] like Fig.11 As shown, the image sequence output in the preview stage is N, N, N, N, N. The electronic device obtains scene perception information, scene brightness information, dynamic range information, etc. according to the image sequence output in the preview stage, and then determines that the current scene is a night scene or a high dynamic scene according to this information. When the user triggers the photo shooting, the preview stage is switched to the shooting stage. In response to the user operation, the electronic device switches to the optical image stabilization working mode in the shooting stage, that is, the filtering mode is switched from the bandpass filtering mode to the low-pass filtering mode, and the exposure parameters such as the exposure time and the sensitivity OIS are adjusted according to the jitter intensity, and the image is collected according to the set exposure parameters and the image sequence is output. For example, the image sequence output in the shooting stage is S, N, S, L, ensuring that the variable exposure image sequence can be output according to the determined optical image stabilization working mode after the photo shooting is triggered in the night scene or the high dynamic scene. Then, the night scene shooting decision module performs key frame selection, multi-frame image registration, multi-frame noise reduction fusion, and HDR fusion based on the image sequence N, N, N, N, N output in the preview stage and the variable exposure image sequence S, N, S, L output in the shooting stage to obtain a night scene photo.

[0281] The image stabilization control method provided in the embodiment of the present application optimizes the OIS algorithm part, determines the optical stabilization working mode in the shooting stage according to the scene perception information, jitter information and face detection information, and in the optical stabilization working mode in the shooting stage, adjusts the jitter information filtering method, the image frame output mode, the exposure parameters, and completes frame selection, image registration and fusion processing, so as to finally obtain high-quality night scene photos.

[0282] Fig.12 FIG. 2 is a flow chart of an image anti-shake control method provided by an embodiment of the present application. Fig.12 As shown, the image anti-shake control method includes the following steps S401-S412.

[0283] S401 . In the preview stage, anti-shake control is performed according to the shake information after the bandpass filtering process, and images are captured using reference exposure parameters during the anti-shake control period to output an image sequence.

[0284] S402: Perform scene recognition according to the image sequence output in the preview stage.

[0285] In some embodiments, the scene recognition result may include scene brightness information, where the scene brightness information is used to indicate the intensity of illumination of the current scene.

[0286] For example, the brightness represented by the scene brightness information is within a first brightness range, indicating that the current scene is an extremely low illumination scene.

[0287] For example, the brightness represented by the scene brightness information is within the second brightness range, indicating that the current scene is a low-illuminance scene.

[0288] For example, the brightness represented by the scene brightness information is within the third brightness range, indicating that the current scene is a medium illumination scene.

[0289] Among them, the brightness of the first brightness range is smaller than the brightness of the second brightness range, the brightness of the second brightness range is smaller than the brightness of the third brightness range, and the brightness of the third brightness range is smaller than the brightness of the fourth brightness range.

[0290] It should be noted that the scenes are divided here by way of example into extremely low illumination scenes, low illumination scenes, medium illumination scenes, and high illumination scenes according to the illumination levels. In actual implementation, other possible divisions of the scenes may be made according to actual needs, and the embodiments of the present application do not limit this.

[0291] In some embodiments, scene brightness information may be expressed in illuminance (in Lux). The greater the illuminance value, the greater the illuminance. The smaller the illuminance value, the smaller the illuminance.

[0292] In some other embodiments, the scene brightness information may be represented by an index value corresponding to the illumination (denoted as Lux_Idx). The larger the index value Lux_Idx is, the smaller the illumination is; the smaller the index value Lux_Idx is, the greater the illumination is.

[0293] In some embodiments, the scene recognition result may include dynamic range information, where the dynamic range information is used to represent the dynamic range of the current scene.

[0294] The electronic device can determine whether the current scene is a high dynamic scene or a non-high dynamic scene according to the dynamic range information.

[0295] For example, if the dynamic range information indicates that the dynamic range of the current scene is within the first dynamic range, that is, the image does not contain obvious light and dark areas, the electronic device determines that the current scene is a non-high dynamic scene.

[0296] For example, if the dynamic range information indicates that the dynamic range of the current scene exceeds the first dynamic range, that is, the image contains obvious light and dark areas, then the electronic device determines that the current scene is a high dynamic scene.

[0297] S403: Determine whether to enable the night scene mode.

[0298] In some embodiments, when the electronic device receives a user trigger to turn on the camera night mode, the electronic device enables the night mode.

[0299] In some embodiments, when the scene recognition result meets the night scene mode triggering condition, the electronic device determines to automatically enable the night scene mode during shooting.

[0300] For example, when the current scene is determined to be an extremely low illumination scene, a low illumination scene, or a medium illumination scene based on the scene brightness information, that is, the scene recognition result meets the night scene mode triggering condition, the electronic device determines to automatically enable the night scene mode during shooting.

[0301] When it is determined that the night scene mode is not enabled, the anti-shake control is performed according to the default anti-shake control strategy of the camera. For example, the jitter information is band-pass filtered in the preview stage and the shooting stage, and then the anti-shake control is performed based on the jitter information after the band-pass filtering.

[0302] S404: When it is determined that the night scene mode is enabled, the scene illumination level is determined according to the scene brightness information, and an anti-shake control strategy corresponding to the scene illumination level is determined.

[0303] Here, the scene brightness information is an index value Lux_Idx and three illumination levels. The electronic device compares the index value Lux_Idx with the index threshold to determine the scene illumination level. It should be noted that the larger the index value Lux_Idx, the smaller the illumination; the smaller the index value Lux_Idx, the greater the illumination.

[0304] When Lux_Idx≥Thd_w2, the electronic device determines that the scene illumination level is the first level, that is, an extremely low illumination scene.

[0305] When Thd_w2<Lux_Idx≤Thd_w1, the electronic device determines that the scene illumination level is the second level, that is, a low illumination scene.

[0306] When Lux_Idx<Thd_w1, the electronic device determines that the scene illumination level is the third level, that is, the medium illumination scene.

[0307] Wherein, Thd_w2 is greater than Thd_w1. A larger index threshold value indicates a smaller illumination; a smaller index threshold value indicates a larger illumination.

[0308] Exemplarily, three illumination levels (first level, second level and third level) and the image stabilization control strategy corresponding to each level are given as follows:

[0309] First illumination level - extremely low illumination scene: Image stabilization control strategy in the shooting stage: After triggering the photo, the jitter information is filtered using a low-pass filter, and the frame is switched to non-ZSL mode, and image fusion is performed using an image sequence based on the non-ZSL mode output (referred to as non-ZSL mode frame output).

[0310] Second illumination level - low illumination scene: Image stabilization control strategy in the shooting stage: After triggering the photo, the jitter information is filtered using a low-pass filter, and the frame is switched to non-ZSL mode, and the image sequence based on the ZSL mode output (referred to as ZSL mode frame) and non-ZSL mode frame are used for image fusion.

[0311] The third illumination level - medium illumination scene: Image stabilization control strategy in the shooting stage: after triggering the photo, the jitter information is still filtered using bandpass filtering, and the ZSL mode is used to output frames for image fusion.

[0312] S405: Determine a shake intensity according to the shake information, and determine a first exposure compensation amount corresponding to the shake intensity.

[0313] The jitter information is jitter information after bandpass filtering, and is represented as Gyro_Info.

[0314] For example, the jitter information may be an angle value, wherein a larger angle value indicates a stronger jitter, and a smaller angle value indicates a weaker jitter.

[0315] In the embodiment of the present application, the first exposure compensation amount includes an exposure duration adjustment amount and an ISO value adjustment amount.

[0316] Fig.13 Schematic diagram of exposure compensation amounts corresponding to various jitter levels provided in the embodiment of the present application. Fig.13 The various jitter levels provided in the embodiment of the present application and the first exposure compensation amount corresponding to each jitter level are exemplarily described.

[0317] First shake level - tripod-assisted shooting:

[0318] In some embodiments, when Gyro_Info≈0, it is determined that the current jitter amplitude / intensity is substantially zero, which is a tripod shooting situation. In this case, the exposure compensation amount can be set according to the preset parameters corresponding to the tripod situation during the shooting stage.

[0319] For example, when it is determined that the shot is taken with a tripod, the exposure time may be increased by 2 stops or more, and the ISO sensitivity may be reduced by 2 stops or more.

[0320] For another example, when it is determined that the shooting is done with a tripod, you can choose not to switch modes, that is, use the ZSL mode to output frames in both the preview stage and the shooting stage, and do not switch to the non-ZSL mode to output frames in the shooting stage.

[0321] Second jitter level - small jitter:

[0322] In some embodiments, when Gyro_Info≤Thd_gyro_shake_w1, it is determined that the current shaking amplitude / intensity belongs to a small shaking situation. In this case, the exposure duration adjustment amount can be set according to the preset parameters corresponding to the small shaking situation during the shooting stage.

[0323] For example, the exposure time is not reduced and the ISO value is kept unchanged, that is, the exposure time adjustment amount and the ISO value adjustment amount are both 0.

[0324] In other embodiments, the exposure duration is increased by 1 to 2 stops based on the baseline exposure duration, and the ISO value is reduced by 1 to 2 stops based on the baseline ISO value.

[0325] Among them, Thd_gyro_shake_w1 is greater than 0.

[0326] The third jitter level-medium jitter:

[0327] In some embodiments, when Thd_gyro_shake_w1≤Gyro_Info<Thd_gyro_shake_w2, it is determined that the current jitter amplitude / intensity belongs to the medium amplitude jitter situation. In this case, the exposure compensation amount can be set according to the preset parameters corresponding to the medium jitter situation during the shooting stage. For example, the exposure time is reduced by 1 level and the ISO value is increased by 1 level.

[0328] Among them, Thd_gyro_shake_w2 is greater than Thd_gyro_shake_w1.

[0329] Fourth jitter level-large jitter:

[0330] In some embodiments, when Gyro_Info≥Thd_gyro_shake_w2, it is determined that the current shaking amplitude / intensity belongs to a large shaking situation. In this case, the exposure compensation amount can be set according to the preset parameters corresponding to the large shaking situation during the shooting stage. For example, the exposure time is reduced by 2 stops and the ISO value is increased by 2 stops.

[0331] S406: Determine a second exposure compensation amount according to the face detection information.

[0332] In some embodiments, the current scene is determined to be a portrait night scene shooting scene based on face detection information. In this case, during the shooting stage, the exposure parameters are adjusted not only according to the jitter intensity, but also based on the portrait night scene shooting scene.

[0333] For example, in some embodiments, when the face detection information (such as is_face_detect=1) satisfies the preset face detection condition, the current scene is determined to be a portrait night scene shooting scene. In this case, during the shooting stage, based on the exposure compensation / adjustment based on the shake correction, the exposure time is further reduced by 0.5 to 1 stop, and the sensitivity ISO value is increased by 0.5 to 1 stop to better capture the night scene portrait image and improve the filming rate.

[0334] In some embodiments, the current scene is determined to be a night scene shooting scene without a portrait based on face detection information. In this case, the step of adjusting the exposure parameters based on the portrait night scene shooting scene is not performed, that is, the second exposure compensation amount is 0.

[0335] It should be noted that the embodiment of the present application does not limit the execution order of the above S405 and S406. For example, S405 may be executed first, and then S406; or S406 may be executed first, and then S405; or S405 and S406 may be executed simultaneously.

[0336] S407: Whether the electronic device receives a user-triggered photo-taking operation.

[0337] For example, the electronic device detects that the user clicks a photo button, that is, the electronic device receives the user's operation of triggering photo taking.

[0338] When the electronic device receives an operation of triggering a photo-taking operation by the user, it switches from the preview phase to the shooting phase.

[0339] S408: When the electronic device receives a user-triggered photo-taking operation, the electronic device filters the jitter information in a first filtering manner, and performs anti-shake control according to the filtered jitter information.

[0340] When the electronic device receives an operation of triggering a photo-taking operation by the user, the first filtering method is determined using step S404.

[0341] Exemplarily, in the first illumination level (extremely low illumination scene), the first filtering mode may be a low-pass filtering mode.

[0342] Exemplarily, in the second illumination level (low illumination scene), the first filtering mode may be a low-pass filtering mode.

[0343] Exemplarily, in the third illumination level (medium illumination scene), the first filtering mode may be a bandpass filtering mode.

[0344] S409: When the electronic device receives an operation triggered by the user to take a photo, it switches to a non-ZSL mode for framing or does not switch, and adjusts the exposure parameters based on the first exposure compensation amount and the second exposure compensation amount, and uses the adjusted exposure parameters to obtain an image sequence in the shooting stage.

[0345] In some embodiments, in an extremely low illumination scene or a low illumination scene, when the electronic device receives an operation triggered by the user to take a photo, the camera is controlled to switch to a non-ZSL mode.

[0346] In other embodiments, in a medium illumination scene, when the electronic device receives an operation of a user triggering a photo, the camera is controlled to still use the ZSL mode without switching modes.

[0347] It should be noted that the electronic device executes S408 and S409 simultaneously. It can be understood that while the anti-shake control is executed, the image is captured using the adjusted exposure parameters to achieve the purpose of image anti-shake.

[0348] In an embodiment of the present application, when the exposure parameters are set based on the jitter intensity and the image is exposed according to the exposure parameters, the jitter information is processed using a set low-pass filtering method and anti-shake control is performed based on the filtered jitter information, thereby avoiding image blurring caused by the jitter of the mobile phone during the exposure time.

[0349] S410: Perform image registration and image fusion according to the image sequence in the preview phase and / or the image sequence in the shooting phase to obtain a night scene photo.

[0350] In some embodiments, in extremely low illumination scenes, image registration and image fusion are performed according to an image sequence output in ZSL mode during the shooting phase to obtain a night scene photo.

[0351] In some embodiments, in low-light scenarios, the ZSL mode is used for frame output in the preview stage, and the non-ZSL mode is switched to frame output in the shooting stage. Accordingly, image registration and image fusion can be performed based on the image sequence output in the ZSL mode in the preview stage and the image sequence output in the non-ZSL mode in the shooting stage to obtain night scene photos.

[0352] In some embodiments, in a medium illumination scene, the ZSL mode is used for framing in both the preview stage and the shooting stage. Accordingly, image registration and image fusion can be performed based on the image sequence output in the ZSL mode in the preview stage and the image sequence output in the ZSL mode in the shooting stage to obtain a night scene photo.

[0353] In an embodiment of the present application, when an image sequence of the preview stage and an image sequence of the shooting stage are obtained, frame selection, multi-frame image registration, multi-frame noise reduction fusion, HDR fusion, and intelligent AI noise reduction can be performed based on the image sequence of the preview stage and the image sequence of the shooting stage to obtain night scene photos.

[0354] S411, determine whether the photo taking is finished.

[0355] For example, when the electronic device obtains a night scene photo, it determines that the photo taking is finished.

[0356] For another example, after the adjusted exposure parameters are used to acquire the image sequence in the shooting phase, it is determined that the shooting is finished.

[0357] S412: When it is determined that the photo shooting is finished, the shooting stage is switched to the preview stage, the frame output is switched to the ZSL mode, the exposure parameters are restored to the reference exposure parameters, and the filtering method for the jitter information is switched to the bandpass filter.

[0358] After S412, the process returns to and continues to execute the above-mentioned S401.

[0359] Through the image anti-shake control method provided by the embodiment of the present application, in the shooting scene of the camera night scene mode, the first anti-shake control parameter applicable to the current scene can be determined based on the scene brightness information, such as the jitter information filtering method, the image frame mode and / or the image fusion method, and the second anti-shake control parameter applicable to the current scene can be determined based on the jitter information, such as the second anti-shake control parameter includes the exposure compensation amount, and different jitter information corresponds to different exposure compensation amounts. The present application scheme can perform comprehensive anti-shake control and image processing based on the first anti-shake control parameter and the second anti-shake control parameter during the shooting stage, and perform different degrees of exposure compensation during the shooting stage for night scene shooting scenes with different jitter intensities, thereby improving the image anti-shake effect of the night scene mode shooting scene and improving the clarity of the night scene photos.

[0360] The above describes the implementation process of the image stabilization control method provided by the embodiment of the present application. The following describes the specific implementation methods of the image stabilization control method provided by the embodiment of the present application in different situations in conjunction with the accompanying drawings.

[0361] It should be noted that in a medium-illuminance scene, it may be a non-high-dynamic scene or a high-dynamic scene. Therefore, the image stabilization control of medium-illuminance and non-high-dynamic scenes in night scene mode and the image stabilization control of medium-illuminance and high-dynamic scenes in night scene mode are explained in combination with Case 1 and Case 2 respectively.

[0362] Case 1: Image stabilization control in night scene mode with medium illumination and non-high dynamic scenes

[0363] Fig.14 The specific implementation method of the image stabilization control in the night scene mode with medium illumination and non-high dynamic scenes provided by the embodiment of the present application is shown. After the camera application is turned on, it enters the preview stage. When the user triggers the photo shooting, the camera application switches from the preview stage to the shooting stage. After the photo shooting is completed, the camera application switches from the shooting stage to the preview stage.

[0364] like Fig.14 As shown in the figure, in the night scene mode with medium illumination and non-high dynamic scene, the image stabilization control strategy adopted by the electronic device includes the following four aspects:

[0365] 1) Filtering method: In both the preview stage and the shooting stage, the jitter information is band-pass filtered, and anti-shake control is performed based on the anti-shake information after band-pass filtering.

[0366] 2) Frame output mode: ZSL mode is used for frame output in both preview and shooting stages.

[0367] 3) Exposure parameters: In the preview stage, the baseline exposure parameters are used to capture images to obtain a baseline exposure image sequence. In the shooting stage, the exposure parameters are adjusted according to the jitter information, and the adjusted exposure parameters are used to capture images to obtain a compensated exposure image sequence.

[0368] 4) Image fusion: Image registration and fusion are performed based on the baseline exposure image sequence in the preview stage and the compensated exposure image sequence in the shooting stage to obtain night scene photos.

[0369] Case 2: Image stabilization control in medium illumination and high dynamic scenes in night scene mode

[0370] Fig.15 The specific implementation method of the image stabilization control in the night scene mode with medium illumination and high dynamic scene provided by the embodiment of the present application is shown. Fig.15 As shown in the figure, in the medium illumination and high dynamic scene in the night scene mode, the image stabilization control strategy adopted by the electronic device includes the following four aspects:

[0371] 1) Filtering method: In both the preview stage and the shooting stage, the jitter information is band-pass filtered, and anti-shake control is performed based on the anti-shake information after band-pass filtering.

[0372] 2) Frame output mode: ZSL mode is used for frame output in both preview and shooting stages.

[0373] 3) Exposure parameters: In the preview stage, the baseline exposure parameters are used to capture images to obtain a baseline exposure image sequence. In the shooting stage, the exposure parameters are adjusted according to the jitter information, and the adjusted exposure parameters are used to capture images to obtain a compensated exposure image sequence. The compensated exposure image sequence is a variable exposure image sequence.

[0374] It should be noted that the variable exposure image sequence may include a long exposure image and a short exposure image. Alternatively, the variable exposure image sequence includes a long exposure image, a short exposure image and a reference exposure image.

[0375] 4) Image fusion: Image registration and fusion are performed based on the baseline exposure image sequence in the preview stage and the compensated exposure image sequence in the shooting stage to obtain night scene photos.

[0376] The image stabilization control strategy adopted in case 2 is similar to that adopted in case 1 in that: when switching from the preview stage to the shooting stage, the filtering mode is not switched and the frame output mode is not switched.

[0377] The difference between the image stabilization control strategy adopted in case 2 and case 1 is that in terms of exposure parameters, since the scene in case 2 is a high-dynamic scene, in case 2, when adjusting the exposure parameters according to the shaking information during the shooting stage, it is necessary to set variable exposure parameters, which include long exposure time, short exposure time and ISO value. Images are collected using variable exposure parameters to obtain variable exposure image sequences.

[0378] That is, in case 2, the shooting stage outputs a variable exposure image sequence that has undergone exposure compensation; whereas in case 1, the shooting stage outputs a compensated exposure image sequence, but not a variable exposure image sequence.

[0379] Case 3: Image stabilization control in low-light scenes in night scene mode

[0380] Fig.16 The specific implementation method of image stabilization control in low-light scenes in night scene mode provided by the embodiment of the present application is shown. Fig.16 As shown in the figure, in the low-light scene in the night scene mode, the image stabilization control strategy adopted by the electronic device includes the following four aspects:

[0381] 1) Filtering method: In the preview stage, the jitter information is band-pass filtered by band-pass filtering, and anti-shake control is performed based on the anti-shake information after band-pass filtering. In the shooting stage, it is switched to low-pass filtering, and anti-shake control is performed based on the anti-shake information after low-pass filtering.

[0382] 2) Frame output mode: The preview stage uses the ZSL mode, and the shooting stage switches to the non-ZSL mode.

[0383] 3) Exposure parameters: In the preview stage, the baseline exposure parameters are used to capture images to obtain a baseline exposure image sequence. In the shooting stage, exposure compensation is performed according to the jitter information, that is, the exposure parameters are adjusted based on the degree of jitter, and the adjusted exposure parameters are used to capture images to obtain a compensated exposure image sequence.

[0384] 4) Image fusion: Image registration and fusion are performed based on the baseline exposure image sequence in the preview stage and the compensated exposure image sequence in the shooting stage to obtain night scene photos.

[0385] Compared with Case 1 and Case 2, the differences and similarities of the image stabilization control strategies used in Case 3 include:

[0386] When switching from the preview stage to the shooting stage, the filtering mode is switched in case 3, and the frame output mode is switched. In case 1 and case 2, the filtering mode and the frame output mode are not switched.

[0387] In terms of exposure parameters and image fusion, Case 3 is basically the same as Cases 1 and 2, that is, the baseline exposure parameters are used to capture images in the preview stage, and exposure compensation is performed according to the jitter information in the shooting stage, that is, the exposure parameters are adjusted based on the degree of jitter, and the adjusted exposure parameters are used to capture images, and image registration and fusion are performed based on the baseline exposure image sequence in the preview stage and the compensated exposure image sequence in the shooting stage.

[0388] Case 4: Image stabilization control in extremely low-light scenes in night scene mode

[0389] Fig.17 The specific implementation method of image stabilization control in extremely low illumination scenes in night scene mode provided by the embodiment of the present application is shown. Fig.17 As shown in the figure, in extremely low illumination scenes in night scene mode, the image stabilization control strategy adopted by electronic devices includes the following four aspects:

[0390] 1) Filtering method: In the preview stage, the jitter information is band-pass filtered by band-pass filtering, and anti-shake control is performed based on the anti-shake information after band-pass filtering. In the shooting stage, it is switched to low-pass filtering, and anti-shake control is performed based on the anti-shake information after low-pass filtering.

[0391] 2) Frame output mode: The preview stage uses the ZSL mode, and the shooting stage switches to the non-ZSL mode.

[0392] 3) Exposure parameters: In the preview stage, the baseline exposure parameters are used to capture images to obtain a baseline exposure image sequence. In the shooting stage, exposure compensation is performed according to the jitter information, that is, the exposure parameters are adjusted based on the degree of jitter, and the adjusted exposure parameters are used to capture images to obtain a compensated exposure image sequence.

[0393] 4) Image fusion: Image registration and fusion are performed based on the compensated exposure image sequence in the shooting stage to obtain night scene photos.

[0394] Compared with case 3, the differences and similarities of the image stabilization control strategies used in case 4 include:

[0395] Case 4 and Case 3 use the same image stabilization control strategy in terms of filtering method, frame output mode, and exposure parameters. That is, when switching from the preview stage to the shooting stage, the bandpass filter is switched to the low-pass filter, and the frame output in the ZSL mode is switched to the non-ZSL mode. The exposure parameters are adjusted based on the baseline exposure parameters.

[0396] In terms of image fusion, case four is different from case three. In case three, image registration and fusion are performed based on the baseline exposure image sequence in the preview stage and the compensation exposure image sequence in the shooting stage, while in case four, image registration and fusion are performed based on the compensation exposure image sequence in the shooting stage.

[0397] Through the solution of the present application, when taking photos using the night scene mode of the camera of an electronic device, due to the insufficient scene lighting in the night scene, it is easy for the photos to be blurred due to hand shaking. In this case, the image anti-shake control method provided in the embodiment of the present application is used to avoid or alleviate the problem of blurred night scene photos caused by shaking, and the shutter time can be effectively extended, thereby achieving the purpose of obtaining more exposure information, reducing the noise of night scene photos and improving clarity.

[0398] It should be noted that, in the embodiments of the present application, "greater than" can be replaced by "greater than or equal to", "less than or equal to" can be replaced by "less than", or "greater than or equal to" can be replaced by "greater than", and "less than" can be replaced by "less than or equal to".

[0399] The various embodiments described herein may be independent solutions or may be combined according to internal logic, and all of these solutions fall within the protection scope of this application.

[0400] The above mainly describes the solution provided by the embodiment of the present application from the perspective of method steps. It is understandable that, in order to achieve the above functions, the electronic device implementing the method includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should be aware that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of protection of the present application.

[0401] The embodiment of the present application can divide the functional modules of the electronic device according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other feasible division methods in actual implementation. The following is an example of dividing each functional module corresponding to each function.

[0402] Fig.18 The present invention provides a schematic block diagram of an image stabilization control device 500 provided in an embodiment of the present invention. The device 500 can be used to perform the actions performed by the electronic device in the above method embodiment. The device 500 includes a camera unit 510, a processing unit 520 and an image stabilization control unit 530.

[0403] The camera unit 510 is used to start a camera application and enable a camera night scene mode.

[0404] The processing unit 520 is used to obtain first scene brightness information and first jitter information when the camera night scene mode is enabled; determine a first anti-shake control parameter based on the first scene brightness information, the first anti-shake control parameter including a jitter information filtering method, an image frame output mode and / or an image fusion method, and different scene brightness information corresponds to different anti-shake control parameters; determine a second anti-shake control parameter based on the first jitter information, the second anti-shake control parameter including a first exposure compensation amount, and different jitter information corresponds to different exposure compensation amounts.

[0405] The camera unit 510 is further configured to receive a first operation of a user triggering a photo-taking operation, and in response to the first operation, perform image acquisition and frame output based on a first anti-shake control parameter and a second anti-shake control parameter.

[0406] The image anti-shake control unit 530 is configured to perform anti-shake control based on the first anti-shake control parameter and the second anti-shake control parameter in response to the first operation.

[0407] The processing unit 520 is further configured to, in response to the first operation, perform image processing based on the first anti-shake control parameter and the second anti-shake control parameter to obtain a target image.

[0408] Through the image anti-shake control device provided by the embodiment of the present application, in the shooting scene of the camera night scene mode, the first anti-shake control parameter applicable to the current scene can be determined based on the scene brightness information, such as the jitter information filtering method, the image frame output mode and / or the image fusion method, and the second anti-shake control parameter applicable to the current scene can be determined based on the jitter information, such as the second anti-shake control parameter includes the exposure compensation amount, and different jitter information corresponds to different exposure compensation amounts. The present application scheme can perform comprehensive anti-shake control and image processing based on the first anti-shake control parameter and the second anti-shake control parameter during the shooting stage, and perform different degrees of exposure compensation during the shooting stage for night scene shooting scenes with different jitter intensities, thereby improving the image anti-shake effect of the night scene mode shooting scene and improving the clarity of the night scene photos.

[0409] According to the embodiment of the present application, the device 500 may correspond to executing the method described in the embodiment of the present application, and the above and other operations and / or functions of the units in the device 500 are respectively for implementing the corresponding processes of the method, which will not be repeated here for the sake of brevity.

[0410] The present application also provides a chip, which is coupled to a memory and is used to read and execute computer programs or instructions stored in the memory to execute the methods in the above embodiments.

[0411] The present application also provides an electronic device, which includes a chip, and the chip is used to read and execute a computer program or instruction stored in a memory, so that the method in each embodiment is executed.

[0412] This embodiment also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the image stabilization control method in the above-mentioned embodiment.

[0413] This embodiment further provides a computer program product, wherein the computer-readable storage medium stores program codes. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the image stabilization control method in the above-mentioned embodiment.

[0414] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0415] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0416] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0417] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of objects. In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two, for example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0418] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image stabilization control method, applied to an electronic device having a camera application, characterized in that: The method comprises: Open the camera application; Enable camera night mode; When the camera night scene mode is enabled, obtaining first scene brightness information and first jitter information; determining a first anti-shake control parameter according to the first scene brightness information, the first anti-shake control parameter including a jitter information filtering mode, an image frame output mode and / or an image fusion mode; different scene brightness information corresponds to different anti-shake control parameters; determining a second anti-shake control parameter according to the first jitter information, wherein the second anti-shake control parameter includes a first exposure compensation amount; different jitter information corresponds to different exposure compensation amounts; Receiving a first operation of the user triggering a photo-taking operation; In response to the first operation, anti-shake control and image processing are performed based on the first anti-shake control parameter, and anti-shake control and image processing are performed based on the second anti-shake control parameter to obtain a target image.

2. The method according to claim 1, characterized in that The performing anti-shake control and image processing based on the second anti-shake control parameter includes: Adjusting the first exposure compensation amount on the basis of the reference exposure parameter to obtain a first exposure parameter; The first exposure parameter is used to capture images to obtain an image sequence in the shooting phase.

3. The method according to claim 2, characterized in that The first exposure compensation amount includes a compensation amount for exposure time and a compensation amount for ISO value, different jitter information corresponds to different jitter levels, and different jitter levels correspond to different exposure compensation amounts; The adjusting based on the first exposure compensation amount on the basis of the reference exposure parameter includes: When the first jitter information corresponds to a first jitter level, increasing the first exposure duration on the basis of the reference exposure duration, and reducing the first photosensitivity value on the basis of the reference photosensitivity ISO value; When the first jitter information corresponds to a second jitter level, reducing a second exposure duration based on the reference exposure duration, and increasing a second photosensitivity value based on the reference ISO value; The jitter intensity of the first jitter level is smaller than the jitter intensity of the second jitter level.

4. The method according to claim 3, characterized in that When the first jitter information corresponds to a first jitter level, increasing a first exposure duration based on a reference exposure duration, and reducing a first photosensitivity value based on a reference ISO value, comprises: When the jitter intensity indicated by the first jitter information is equal to 0, the first jitter information is within the first jitter level range, the exposure duration is increased by N levels based on the reference exposure duration, and the sensitivity value is reduced by N levels based on the reference ISO value; When the jitter intensity indicated by the first jitter information is greater than zero and less than a first jitter threshold, the first jitter information is within the first jitter level range, the M-level exposure duration is increased based on the reference exposure duration, and the M-level photosensitivity value is reduced based on the reference ISO value; Wherein, N is greater than M, and M is greater than or equal to zero.

5. The method according to claim 3, characterized in that: When the first jitter information corresponds to a second jitter level, reducing a second exposure time on the basis of the reference exposure time, and increasing a second photosensitivity value on the basis of the reference ISO value, comprises: When the jitter intensity indicated by the first jitter information is greater than or equal to a first jitter threshold and less than a second jitter threshold, the first jitter information is within the second jitter level range, the S-level exposure duration is reduced based on the reference exposure duration, and the S-level photosensitivity value is increased based on the reference ISO value; When the jitter intensity indicated by the first jitter information is greater than or equal to a second jitter threshold, the first jitter information is within the second jitter level range, the T-level exposure duration is reduced based on the reference exposure duration, and the T-level photosensitivity value is increased based on the reference ISO value; Among them, S is less than T and S is greater than zero.

6. The method according to any one of claims 2 to 5, characterized in that The adjusting the first exposure compensation amount based on the reference exposure parameter to obtain the first exposure parameter includes: Adjusting the reference exposure parameter based on the first exposure compensation amount to obtain a first adjusted exposure time and a first adjusted ISO value; If the exposure time after the first adjustment is greater than the preset limit exposure time, the exposure time is set to the limit exposure time, and a second adjustment is performed according to the first increment based on the ISO value after the first adjustment; The first exposure parameter includes the maximum exposure time and the ISO value after secondary adjustment.

7. The method according to claim 6, characterized in that The adjusting according to the first increment based on the first adjusted ISO value includes: Calculating a first ratio according to the exposure duration after the first adjustment and the limit exposure duration; multiplying the first ratio by the ISO value adjusted for the first time to obtain the first increment; The first increment is added to the first adjusted ISO value to obtain the second adjusted ISO value.

8. The method according to any one of claims 1 to 7, characterized in that The performing anti-shake control and image processing based on the first anti-shake control parameter includes: Using a first filtering method to filter the second jitter information obtained in the shooting stage, wherein the first filtering method is a bandpass filter or a low-pass filter; Outputting an image sequence using a first frame output mode, wherein the first frame output mode is a zero-delay shooting ZSL mode or a non-ZSL mode; The first image sequence is used for image fusion, where the first image sequence is an image sequence in a preview phase and an image sequence in a shooting phase, or is an image sequence in the shooting phase.

9. The method according to claim 8, characterized in that Different scene brightness information corresponds to different illumination levels, and different illumination levels correspond to different anti-shake control parameters; The performing anti-shake control and image processing based on the first anti-shake control parameter includes: When the first scene brightness information corresponds to a first illumination level, filtering the second jitter information by a low-pass filtering method, switching from the ZSL mode to the non-ZSL mode, and performing image fusion by using the image sequence in the shooting phase; When the first scene brightness information corresponds to a second illumination level, filtering the second jitter information by a low-pass filtering method, switching from the ZSL mode to the non-ZSL mode, and performing image fusion by using the image sequence of the preview stage and the image sequence of the shooting stage; When the first scene brightness information corresponds to a third illumination level, the second jitter information is filtered by a bandpass filter, the frame is output by the ZSL mode, and the image sequence in the preview phase and the image sequence in the shooting phase are used for image fusion; The illumination of the first illumination level is smaller than that of the second illumination level, and the illumination of the second illumination level is smaller than that of the third illumination level.

10. The method according to any one of claims 1 to 9, characterized in that The method further comprises: Determining, according to the first scene brightness information, that the current scene is a high dynamic range scene; When a first operation of the user triggering a photo-taking operation is received, the exposure parameter is set as a variable exposure parameter.

11. The method according to claim 10, characterized in that The variable exposure parameter includes a first exposure duration and a second exposure duration; wherein the image sequence in the shooting phase includes a first exposure image and a second exposure image; or, The variable exposure parameters include a first exposure duration, a second exposure duration, and a reference exposure duration; wherein the image sequence in the shooting phase includes a first exposure image, a second exposure image, and a reference exposure image; The first exposure duration is greater than the reference exposure duration, and the reference exposure duration is greater than the second exposure duration.

12. The method according to any one of claims 1 to 11, characterized in that The method further comprises: When the night scene mode of the camera is enabled, obtaining face detection information; Determining a third anti-shake control parameter according to the face detection information, where different face detection information corresponds to different anti-shake control parameters; When the first operation of the user triggering the photo-taking is received, anti-shake control and image processing are performed based on the third anti-shake control parameter.

13. The method according to claim 12, characterized in that The third anti-shake control parameter includes a second exposure compensation amount; and the anti-shake control and image processing based on the third anti-shake control parameter include: Adjusting the first exposure parameter based on the second exposure compensation amount to obtain a second exposure parameter; The second exposure parameter is used to capture images to obtain an image sequence in the shooting phase.

14. The method according to claim 13, characterized in that Different face detection information corresponds to different exposure compensation amounts; the first exposure parameter includes a first exposure time and a first ISO value; The adjusting based on the second exposure compensation amount on the basis of the first exposure parameter includes: When the face detection information satisfies the portrait shooting condition, reducing the exposure time of the R gear on the basis of the first exposure time, and increasing the photosensitivity value of the R gear on the basis of the first ISO value; wherein R is greater than zero; When the face detection information does not meet the portrait shooting condition, the second exposure compensation amount is zero.

15. The method according to any one of claims 1 to 14, characterized in that After starting the camera application and before receiving a first operation of a user triggering a photo-taking operation, the method further includes: In the preview stage, anti-shake control and image processing are performed based on the fourth anti-shake control parameter.

16. The method according to claim 15, characterized in that The performing anti-shake control and image processing based on the fourth anti-shake control parameter includes: Using bandpass filtering on the first jitter information obtained in the preview phase, and controlling the movement of the anti-shake device of the electronic device based on the first jitter information after the bandpass filtering; Acquiring an image using a reference exposure parameter to obtain a reference exposure image; wherein the reference exposure parameter includes a reference exposure time and a reference ISO value; The image sequence of the preview stage is outputted in ZSL mode, and the image sequence of the preview stage includes one or more of the reference exposure images.

17. The method according to any one of claims 1 to 16, characterized in that The enabling of the camera night scene mode includes: Get the image sequence of the preview stage; Performing scene recognition based on the image sequence in the preview stage; When the scene recognition result meets the camera night scene mode triggering condition, the camera night scene mode is automatically enabled.

18. The method according to any one of claims 1 to 16, characterized in that The enabling of the camera night scene mode includes: In response to a second operation of the user turning on the camera night scene mode, the camera night scene mode is enabled.

19. The method according to any one of claims 11 to 18, characterized in that The first jitter information is a deviation angular velocity or a deviation angle of the electronic device, and the first jitter information is obtained through a gyroscope sensor.

20. An electronic device, characterized in that: The electronic device includes: one or more processors, and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to perform the method as claimed in any one of claims 1 to 19.

21. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 19.

22. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises instructions, which, when executed on an electronic device, cause the electronic device to perform the method as claimed in any one of claims 1 to 19.

Images