Image processing method and device and storage medium

Abstract

The invention provides an image processing method and device and a storage medium. According to the method, the amplitude of an intermediate frequency flash source in a shooting scene is processed, the processed stroboscopic intensity is compared with a preset intensity threshold value, whether stroboscopic stripes appear in a preview image of a camera application or not is judged, under the condition that the judgment result is yes, the dynamic range expansion coefficient of a long-frame image is calculated, and the dynamic range expansion coefficient of the long-frame image is calculated. The method is used for correcting the target brightness of the long-frame image, so that when the electronic equipment regenerates the exposure parameters, stroboscopic stripe removal processing is carried out, and the photographing experience of a user is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus and storage medium. Background Technology

[0002] Taking photos and videos generally refers to the process of exposing a photosensitive medium to light reflected from an object. With the widespread use of smartphones, recording life's moments with photos has become increasingly popular.

[0003] However, in some artificial light source scenarios, when users take photos with their mobile phones, fluctuations in bright and dark stripes appear in the preview screen of the camera app on the phone and in the final generated photo, which greatly reduces the user's photography experience. Summary of the Invention

[0004] This application provides an image processing method, device, and storage medium, aiming to solve the problem of bright and dark stripes appearing in the preview screen and photos of the camera application on the mobile terminal in artificial light source scenarios, thereby improving the user's photography experience.

[0005] In a first aspect, embodiments of this application provide an image processing method applied to an electronic device. The shooting scene of the electronic device is illuminated by a strobe light source, the strobe light source including frequency and amplitude, the frequency being used to indicate the strobe period of the strobe light source. The image processing method includes: responding to a first operation by a user, displaying a first preview image on a camera interface, the first preview image being obtained by fusing a first long frame image and a first short frame image, the camera interface being the interface of a camera application in the electronic device; acquiring long frame statistical parameters and long frame exposure parameters of the first long frame image, and short frame statistical parameters of the first short frame image, the long frame statistical parameters including a first long frame target brightness and a first long frame current brightness, the long frame exposure parameters including a first long frame target exposure time and a first long frame current exposure time. The short-frame statistical parameters include the target brightness of the first short frame and the current brightness of the first short frame; the amplitude of the stroboscopic source is dimensionlessly processed to obtain the stroboscopic intensity; when the stroboscopic intensity is greater than a preset intensity threshold, the dynamic range expansion coefficient of the first long frame image is calculated based on the target exposure time of the first long frame, the current exposure time of the first long frame, the gain, and the sensitivity; the target brightness of the first long frame is corrected based on the dynamic range expansion coefficient; the long frame exposure parameters are regenerated based on the corrected target brightness of the first long frame and the current brightness of the first long frame, and the short frame exposure parameters are regenerated based on the statistical parameters of the first short frame image; a second preview image is generated based on the long frame exposure parameters and the short frame exposure parameters and displayed on the camera interface.

[0006] Understandably, a user's first action might be to enter HDR mode. For example, such as... Figure 4 (1) After the user clicks control 10c-1, they enter HDR mode and the display is as follows: Figure 4 The HDR photo-taking interface of (2) in the middle.

[0007] For example, the first preview image mentioned above can be a photo preview interface with flickering stripes as shown in the following embodiments, such as... Figure 5 The camera preview area 10e-1 is shown.

[0008] For example, the aforementioned strobe light source can be a light source driven by alternating current, such as... Figure 5 The light source is 200. The frequencies of stroboscopic light sources typically include 50Hz and 60Hz. At a frequency of 50Hz, meaning a period of 10ms, the changing pattern is as follows... Figure 6 The waveform of the light source is shown in the figure.

[0009] For example, the frequency and amplitude of the above-mentioned strobe light source can be directly obtained by the strobe detection module in the following embodiments.

[0010] For example, in HDR mode, the first preview image in the aforementioned photo preview area is obtained by fusing two images, including a long frame image and a short frame image. The AE algorithm module in HDR mode calculates the exposure parameters of the long frame image and the short frame image respectively, and sends them to the image sensor in the following embodiment, so that the image sensor generates the long frame image and the short frame image based on the exposure parameters until the user exits the camera application.

[0011] For example, in the image signal processing algorithm of the following embodiments, the statistics module is used to perform statistics on the brightness information of the long frame image and the short frame image generated by the image sensor to obtain the target brightness and current brightness of the long frame image, and the target brightness and current brightness of the short frame image.

[0012] For example, the step of dimensionless processing of the amplitude of the stroboscopic light source can be a step of scaling it to a specified numerical range and then rounding it down, with the rounded result being the stroboscopic intensity. The aforementioned numerical range can be [0, 10] as in the following embodiments.

[0013] For example, the aforementioned preset intensity threshold can be set by each electronic device manufacturer based on their own experience. In this embodiment, the set value can be 3. That is, when the flicker intensity exceeds 3, it can be determined that flicker stripes appear in the camera's preview image, and flicker stripe removal processing is required.

[0014] For example, the process of regenerating the long frame exposure parameters described above can be that the AE algorithm module calculates the exposure of the next long frame image based on the target brightness of the long frame, the current brightness of the long frame, and the target exposure of the long frame, and decomposes it into exposure time and gain and sends it to the image sensor described below.

[0015] For example, the second preview image mentioned above is a preview image after removing flicker stripes, and its effect is as follows: Figure 13 As shown.

[0016] Therefore, this embodiment of the application obtains the flicker intensity by processing the amplitude of the stroboscopic light source. When the flicker intensity is greater than a preset intensity threshold, the dynamic range expansion coefficient of the long frame image is calculated and applied to the target brightness of the long frame image. This allows the AE algorithm module to perform flicker stripe removal processing on the preview image when recalculating the exposure parameters of the long frame image. This solves the problem of bright and dark stripes appearing in the preview image and photos in the camera application of electronic devices in the shooting scenario of stroboscopic light source, thus improving the user's shooting experience.

[0017] According to the first aspect, before displaying the first preview image on the camera interface, the process includes: generating initial exposure parameters based on the obtained light intensity of the current environment of the shooting scene, and generating the first preview image based on the initial exposure parameters.

[0018] For example, the ambient light intensity can be directly obtained through sensors built into electronic devices.

[0019] Therefore, when the camera app first enters the shooting interface, the phone estimates an initial exposure parameter based on the aforementioned light intensity. This parameter enables the image sensor to operate and generate a first long frame image and a first short frame image. The image signal processor then fuses the first long frame image and the first short frame image to obtain a first preview image. This avoids the situation where a blank interface is displayed after the user enters the shooting interface, thus improving the user's shooting experience.

[0020] According to the first aspect, or any implementation of the first aspect above, the step of performing dimensionless processing on the amplitude of the stroboscopic light source to obtain the stroboscopic intensity further includes: obtaining the exposure ratio of the first long frame image based on the ratio of the first long frame target exposure time to the current exposure time of the first long frame; calculating the stroboscopic intensity adjustment coefficient using linear interpolation based on the exposure ratio and the mapping relationship between the preset exposure ratio and the adjustment coefficient; and correcting the stroboscopic intensity based on the stroboscopic intensity adjustment coefficient.

[0021] As is understandable, flicker intensity is used to indicate the probability of flickering during photography in the current scene. When the exposure time is close to an integer multiple of the light source period, the flicker intensity is less than a preset flicker threshold, meaning that the image based on the current exposure parameters will not produce flickering. When the difference between the exposure time and the light source period is large, the flicker intensity is greater than the preset threshold, meaning that the image based on the current exposure parameters will produce flickering. Since flicker intensity is affected by exposure time, to obtain the flicker intensity of the current scene more accurately, it is necessary to correct it based on the difference between the current exposure time and the target exposure time.

[0022] The formula for calculating the exposure ratio of the first long frame image is as follows, based on the ratio of the exposure time of the first long frame target to the current exposure time of the first long frame:

[0023] expoRatio=targetExpoTime / currentExpoTime;

[0024] Where targetExpoTime is the target exposure time, which is an integer multiple of the current light source cycle, banding Step, and currentExpoTime is the current exposure time.

[0025] For example, when the preset exposure ratio and adjustment coefficient mapping relationship is represented as an array, the exposure ratio is: expoRatioZone

[10] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}, and the adjustment coefficient is: adjustRatioZone

[10] = {0.1, 0.5, 1.5, 3, 4, 5, 6, 7, 8, 9}.

[0026] For example, the calculation formula corresponding to the above linear interpolation method is as follows:

[0027] adjustRatio=adjustRatioZone[i-1]+(expoRatio-expoRatioZone[i-1])*(adjustRatioZone[i]-adjustRatioZone[i-1]) / (expoRatioZone[i]-expoRatioZone[i-1]).

[0028] Where i is the position of the corrected exposure ratio in the array expoRatioZone, and expoRatio is the original exposure ratio.

[0029] Therefore, by adjusting the flicker intensity, a flicker intensity that accurately reflects the probability of flickering can be obtained.

[0030] According to the first aspect, or any implementation of the first aspect above, after calculating the dynamic range expansion coefficient of the first long frame image based on the first long frame target exposure time, the first long frame current exposure time, gain, and sensitivity, the method includes: correcting the dynamic range expansion coefficient when the dynamic range expansion coefficient exceeds a preset limit value.

[0031] Understandably, an excessively high dynamic range can lead to excessively high brightness in long-frame images, which can cause abnormal fusion results when the ISP fuses long-frame and short-frame images. Therefore, it is necessary to limit the dynamic range expansion coefficient.

[0032] For example, since the minimum gain coefficient for long frame images needs to be set to no less than ISO 4, the maximum value of the dynamic range extension coefficient can usually be set to the ISO 4 of the image sensor.

[0033] According to the first aspect, or any implementation of the first aspect above, the dynamic range expansion coefficient is corrected, including: calculating the ambient brightness of the shooting scene based on the shutter speed, aperture, ISO of the camera on the electronic device and the long frame statistical parameters; calculating the maximum expansion coefficient corresponding to the shooting scene using linear interpolation based on the ambient brightness and the mapping relationship between the preset ambient brightness and the maximum expansion coefficient; and correcting the dynamic range expansion coefficient based on the maximum expansion coefficient.

[0034] For example, when the mapping relationship between ambient brightness and maximum expansion coefficient is represented as an array, the ambient brightness is: LightValueZone[6]={90, 100, 110, 120, 130, 140}, and the maximum expansion coefficient is: maxRatioZone[6]={4, 3.5, 3, 2.5, 2, 1}.

[0035] For example, the calculation formula corresponding to the above linear interpolation method is as follows:

[0036] maxRatio=maxRatioZone[i-1]+(LightValue-LightValueZone[i-1])*(maxRatioZone[i]-maxRati oZone[i-1]) / (LightValueZone[i]-LightValueZone[i-1]);

[0037] Where i is the position of the ambient light value in the array LightValueZone, and LightValue is the ambient light value.

[0038] According to the first aspect, or any implementation of the first aspect above, the dynamic range expansion coefficient is corrected based on the maximum expansion coefficient, including: when the maximum expansion coefficient is less than the dynamic range expansion coefficient, the dynamic range expansion coefficient is updated to the maximum expansion coefficient.

[0039] For example, the formula for correcting the dynamic range extension factor based on the maximum extension factor is as follows:

[0040] drExtensionRatio=min(drExtensionRatio,maxRatio).

[0041] Where drExtensionRatio is the dynamic range extension factor and maxRatio is the maximum extension factor.

[0042] According to the first aspect, or any implementation of the first aspect above, the brightness of the first long frame target is corrected based on the dynamic range expansion coefficient, including: using the dynamic range expansion coefficient to amplify the brightness of the first long frame target.

[0043] According to the first aspect, or any implementation thereof, the long frame exposure parameters are regenerated based on the corrected first long frame target brightness and the first long frame current brightness, including: calculating the current exposure of the first long frame image based on the exposure time and the gain; calculating the target exposure of the next long frame image based on the current exposure, the first long frame target brightness and the first long frame current brightness; calculating the exposure of the next long frame image based on the current exposure and the target exposure; and regenerating the long frame exposure parameters based on the exposure.

[0044] According to the first aspect, or any implementation of the first aspect above, the exposure of the next long frame image is calculated based on the current exposure and the target exposure, and further includes: adjusting the exposure based on a preset exposure adjustment coefficient.

[0045] According to the first aspect, or any implementation of the first aspect above, the amplitude of the stroboscopic light source is dedimensified to obtain the stroboscopic intensity, including: taking the amplitude scaling value of the stroboscopic light source within a preset numerical range and then rounding it to obtain the stroboscopic intensity.

[0046] Secondly, embodiments of this application provide an electronic device. The electronic device includes: a memory and a processor, the memory and the processor being coupled; the memory stores program instructions, which, when executed by the processor, cause the electronic device to perform the method of the first aspect or any possible implementation thereof.

[0047] Thirdly, embodiments of this application provide a computer-readable medium for storing a computer program, the computer program including instructions for performing the method in the first aspect or any possible implementation of the first aspect.

[0048] Fourthly, embodiments of this application provide a computer program including instructions for performing the method in the first aspect or any possible implementation thereof.

[0049] Fifthly, embodiments of this application provide a chip including a processing circuit and transceiver pins. The transceiver pins and the processing circuit communicate with each other via an internal connection path. The processing circuit executes the method in the first aspect or any possible implementation of the first aspect to control the receiving pin to receive signals and to control the transmitting pin to transmit signals. Attached Figure Description

[0050] Figure 1 A schematic diagram of the hardware structure of an electronic device as an example;

[0051] Figure 2 A schematic diagram of the software structure of an electronic device as an example;

[0052] Figure 3 This is an example illustration of a user accessing the camera application.

[0053] Figure 4 This is an example illustration of a user entering HDR mode;

[0054] Figure 5 This is an example illustration of a camera preview interface exhibiting flickering.

[0055] Figure 6 The light source waveform and light source energy wave diagram are shown as examples.

[0056] Figure 7 This is an illustrative diagram of a camera's rolling exposure pattern, as shown by example.

[0057] Figure 8 This is a schematic diagram illustrating the comparison between exposure time and light source energy cycle in the context of stroboscopic phenomenon.

[0058] Figure 9 This is a schematic diagram of a typical AE algorithm flow, as shown by example.

[0059] Figure 10 This is a schematic diagram illustrating the HDR mode AE ​​algorithm flow as an example.

[0060] Figure 11This is a schematic diagram illustrating module interaction as an example.

[0061] Figure 12 This is an example of a module interaction timing diagram;

[0062] Figure 13 This is an example illustration of the camera preview interface after flicker removal. Detailed Implementation

[0063] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0064] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0065] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0066] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0067] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0068] To better understand the technical solutions provided in the embodiments of this application, before describing the technical solutions of the embodiments of this application, the hardware structure of the terminal devices (e.g., mobile phones, tablets, touch-screen PCs, etc.) to which the embodiments of this application are applicable will first be described with reference to the accompanying drawings. For ease of explanation, Figure 1 Let's take a mobile phone as an example.

[0069] See Figure 1The mobile phone 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0070] The processor 110 may include one or more processing units, such as an application processor (AP), a modem, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU), etc., which will not be listed here and this application does not limit them.

[0071] The controller mentioned above, which serves as the processing unit, can be the central nervous system and command center of the mobile phone 100. In practical applications, the controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0072] The aforementioned modulation and demodulation processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal and transmits the demodulated low-frequency baseband signal to the baseband processor for processing.

[0073] The aforementioned baseband processor is used to process the low-frequency baseband signal transmitted by the regulator and then transmit the processed low-frequency baseband signal to the application processor.

[0074] It should be noted that in some implementations, the baseband processor can be integrated into the modem, meaning the modem can have the functionality of a baseband processor.

[0075] The aforementioned application processor is used to output sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.) or to display images or videos through display screen 194.

[0076] The aforementioned digital signal processor is used to process digital signals. Specifically, in addition to processing digital image signals, a digital signal processor can also process other digital signals. For example, when the mobile phone 100 is selecting a frequency, the digital signal processor can be used to perform Fourier transforms on the frequency energy, etc.

[0077] The aforementioned video codecs are used for compressing or decompressing digital video. For example, mobile phone 100 may support one or more video codecs. Thus, mobile phone 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0078] The aforementioned ISP (Image Signal Processor) is used to output digital image signals to the DSP (Digital Signal Processor) for processing. Specifically, the ISP processes data fed back from the camera 193. For example, when taking a photo or recording video, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into a visible image. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some implementations, the ISP can be integrated into the camera 193.

[0079] The DSP mentioned above is used to convert digital image signals into standard RGB, YUV, and other image signal formats.

[0080] Furthermore, it should be noted that, regarding the processor 110 including the aforementioned processing units, in some implementations, the different processing units can be independent devices. That is, each processing unit can be considered as a processor. In other implementations, the different processing units can also be integrated into one or more processors. For example, in some implementations, the modem processor can be an independent device. In other implementations, the modem processor can be independent of the processor 110 and housed in the same device as the mobile communication module 150 or other functional modules.

[0081] It should be understood that the above description is merely an example provided to better understand the technical solution of this embodiment, and is not intended as the only limitation on this embodiment.

[0082] In addition, the processor 110 may also include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc., which will not be listed here, and this application does not impose any limitations on them.

[0083] In addition, processor 110 may also include memory for storing instructions and data. In some implementations, the memory in processor 110 is a cache memory. This memory can store instructions or data that processor 110 has just used or is recurring. If processor 110 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of processor 110, and thus improves system efficiency.

[0084] See also Figure 1 The external storage interface 120 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the mobile phone 100. The external storage card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external storage card.

[0085] See also Figure 1The internal memory 121 can be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the mobile phone 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, and stereo recording function as described in the embodiments of this application), etc. The data storage area may store data created during the use of the mobile phone 100 (such as stereo audio data recorded based on the technical solution provided in the embodiments of this application), etc. In addition, the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0086] See also Figure 1 The charging management module 140 is used to receive charging input from the charger. The charger can be a wireless charger or a wired charger.

[0087] See also Figure 1 The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, internal memory 121, external memory, display 194, camera 193, and wireless communication module 160, etc.

[0088] See also Figure 1 The wireless communication function of mobile phone 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0089] It should be noted that antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. Each antenna in mobile phone 100 can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other implementations, the antennas can be used in conjunction with a tuning switch.

[0090] See also Figure 1 The mobile communication module 150 can provide solutions for wireless communication applications including 2G / 3G / 4G / 5G for mobile phones 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc.

[0091] See also Figure 1 The wireless communication module 160 can provide solutions for wireless communication applications on the mobile phone 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc.

[0092] See also Figure 1 The audio module 170 may include a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, etc. For example, the mobile phone 100 can implement audio functions through the application processor and the speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, etc. in the audio module 170. Examples include recording and video recording functions.

[0093] In the process of implementing audio functions through the application processor and audio module 170, the audio module 170 can be used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. The audio module 170 can also be used for encoding and decoding audio signals. In some implementations, the audio module 170 can be located in the processor 110, or some functional modules of the audio module 170 can be located in the processor 110.

[0094] See also Figure 1 The sensor module 180 may include pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, proximity sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, bone conduction sensors, etc., which will not be listed here, and this application does not limit them.

[0095] See also Figure 1 The buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch buttons. The mobile phone 100 can receive button input and generate button signal inputs related to the user settings and function control of the mobile phone 100.

[0096] See also Figure 1Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback.

[0097] See also Figure 1 The indicator 192 can be an indicator light, which can be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0098] See also Figure 1 The camera 193 is used to capture still images or videos. The mobile phone 100 can achieve its shooting function through an ISP, camera 193, video codec, GPU, display 194, and application processor. Specifically, an object generates an optical image through a lens and projects it onto a photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some implementations, the mobile phone 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0099] See also Figure 1 The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some implementations, the mobile phone 100 may include one or N displays 194, where N is a positive integer greater than 1. The mobile phone 100 can implement display functions through a GPU, the display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0100] That concludes the introduction to the hardware structure of the Mobile 100. It should be understood that... Figure 1 The mobile phone 100 shown is just an example. In a specific implementation, the mobile phone 100 may have more or fewer components than shown in the figure, may combine two or more components, or may have different component configurations. Figure 1 The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0101] To better understand Figure 1The software structure of the mobile phone 100 shown is described below. Before describing the software structure of the mobile phone 100, the possible architectures for the software system of the mobile phone 100 will be explained first.

[0102] Specifically, in practical applications, the software system of Mobile 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture.

[0103] Furthermore, it is understood that the software systems used by mainstream terminal devices currently include, but are not limited to, Windows, Android, and iOS systems. For ease of explanation, this application embodiment uses the layered architecture of the Android system as an example to exemplify the software structure of the terminal device 100.

[0104] Furthermore, the image processing solutions provided in the embodiments of this application are also applicable to other systems in specific implementations.

[0105] See Figure 2 This is a software structure block diagram of the mobile phone 100 according to an embodiment of this application.

[0106] like Figure 2 As shown, the layered architecture of the mobile phone 100 divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some implementations, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0107] The application layer can include a series of application packages. For example... Figure 2 As shown, the application package may include applications such as camera, app store, video, shopping, permission management, Bluetooth, Wi-Fi, settings, etc., which will not be listed here, and this application does not impose any restrictions on them.

[0108] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications within the application layer. In some implementations, these APIs and frameworks can be described as functions. For example... Figure 2 As shown, the application framework layer may include functions such as camera service, view system, content provider, speech recognition module, wake word / command word detection module, scene recognition module, speech data fusion module, speech activity detection module, voiceprint verification module, and wake word voiceprint template management module, etc., which will not be listed here, and this application does not impose any restrictions on them.

[0109] It should be understood that the above description is merely an example provided to better understand the technical solution of this embodiment, and is not intended as the only limitation on this embodiment.

[0110] Furthermore, it is understood that the above division of functional modules is merely an example provided to better understand the technical solution of this embodiment, and is not intended to be the sole limitation of this embodiment. In practical applications, the above functions can also be integrated into a single functional module, and this embodiment does not impose any restrictions on this.

[0111] Furthermore, in practical applications, the aforementioned functional modules can also be represented as services or frameworks. For example, a speech recognition module can be represented as a speech recognition service or a speech recognition framework. This embodiment does not impose any restrictions on this.

[0112] In addition, it should be noted that the window manager, located in the application framework layer, is used to manage window applications. The window manager can obtain the screen size, determine whether there is a status bar, lock the screen, and capture the screen, etc.

[0113] Furthermore, it should be noted that the content provider located in the application framework layer is used to store and retrieve data, and to make this data accessible to the application. The data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc., which will not be listed here, and this application does not impose any limitations on this.

[0114] Furthermore, it should be noted that the view system described above, located in the application framework layer, includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0115] Furthermore, it should be noted that the phone manager located in the application framework layer is used to provide communication functions for mobile phone 100, such as call status management (including call connection, call termination, etc.).

[0116] The resource manager provides various resources for the application, such as localized strings, icons, images, layout files, video files, etc., which will not be listed here, and this application does not impose any restrictions on them.

[0117] In addition, it should be noted that the notification manager located in the application framework layer allows the application to display notification information in the status bar. It can be used to convey informational messages and can disappear automatically after a short time without user interaction.

[0118] The Android Runtime consists of core libraries and a virtual machine. The Android Runtime is responsible for the scheduling and management of the Android system.

[0119] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0120] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0121] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0122] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0123] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0124] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0125] Understandably, the 2D graphics engine mentioned above is a 2D drawing engine.

[0126] Furthermore, it is understandable that the kernel layer in the Android system is the layer between hardware and software. The kernel layer includes at least display drivers, camera drivers, audio drivers, sensor drivers, etc. For example, a sensor driver can be used to output detection signals from sensors (such as touch sensors) to the view system, so that the view system responds to the detection signals and displays the corresponding application interface.

[0127] That concludes the introduction to the software structure of the mobile phone 100. As you can understand, Figure 2 The layers in the illustrated software structure and the components contained in each layer do not constitute a specific limitation on the mobile phone 100. In other embodiments of this application, the mobile phone 100 may include more or fewer layers than illustrated, and each layer may include more or fewer components; this application does not impose any limitations.

[0128] Flicker is an optical phenomenon that refers to the periodic changes in brightness or intensity of a light source during emission. This phenomenon can be caused by a variety of factors, including but not limited to power fluctuations, equipment design flaws, or the influence of the operating environment.

[0129] For example, in an artificial light source scenario, an AC light source operates under the drive of AC power, and the energy of the AC power changes periodically over time. Therefore, the energy of the AC light source is not uniformly distributed over time, but changes with the same period as the AC power. Consequently, due to the influence of AC power, AC light sources also exhibit flickering.

[0130] Alternating current (AC) light sources are commonly used in indoor and nighttime venues, such as indoor badminton courts, basketball courts, dance studios, and exhibition halls. In these spaces, users may use their smartphones' cameras to capture people and objects, recording memorable moments. However, the flickering effect of AC light sources can cause the camera sensor to integrate different amounts of energy onto each line of light source during exposure. This results in fluctuations in bright and dark stripes on the image, a phenomenon known as flickering or banding, significantly degrading the user's photography experience.

[0131] To better understand the photo-taking process in the above scenarios, the following will combine... Figures 3 to 5 Taking a mobile phone as an example, this paper describes the photography scenario in an indoor exhibition hall.

[0132] See Figure 3 Middle (1), Figure 3 (1) Example shows the interface 10a of a mobile phone. The interface 10a displays icons for multiple applications, such as: camera 10a-1, contacts, phone, messages, clock, calendar, gallery, memo, file manager, email, music, calculator, video, recorder, weather, browser, settings and other applications.

[0133] It should be noted that, among some possible implementation methods, Figure 3 The interface 10a shown in (1) can be called the main interface. When the user clicks the icon 10a-1 in the interface 10a, he / she can use the camera's functions such as taking pictures and recording videos.

[0134] See also Figure 3In example (1), when a user clicks the camera application icon 10a-1, the phone responds to the user's operation, recognizes the icon corresponding to the user's click as the camera application icon, and then calls the corresponding interface in the application framework layer to start the camera application. It also calls the kernel layer to start the camera driver and acquires an image stream (in this case, a preview stream) through the camera driver. At this time, the phone displays the corresponding interface of the camera application, for example... Figure 3 Interface 10b is shown in (2).

[0135] As mobile phone camera functions have improved, camera apps support an increasing number of shooting modes. For example, shooting modes include, but are not limited to, aperture mode, night mode, portrait mode, photo mode, video mode, smile mode, and professional mode. (See also...) Figure 3 The mode options are shown in the shooting mode list 10b-1 in (2).

[0136] For example, when a user taps the image option corresponding to a certain shooting mode, the phone displays the camera application interface for that shooting mode. For instance, if the user taps the photo mode icon, the phone displays the camera application interface when using photo mode, which can be referenced here. Figure 3 Interface 10b is shown in (2).

[0137] It should be noted that in some possible implementations, after the phone launches the camera app, the camera app defaults to the default shooting mode. However, to enhance image contrast, make the image more three-dimensional, and better reproduce details in the real scene, users may also select the HDR mode provided by the camera app. For example, by clicking the "More" option in the shooting mode list 10b-1, the following will be displayed: Figure 4 The more shooting mode selection interface 10c shown in (1) is shown in the middle.

[0138] See Figure 4 Middle (1), Figure 4 Example (1) shows an interface 10c with a list of more shooting modes. This interface 10c displays multiple shooting mode controls, including: professional mode, slow motion mode, panorama mode, HDR shooting mode 10c-1, time-lapse mode, watermark mode, high-resolution mode, short film mode, and document scanning mode. Users can enter the corresponding shooting mode interface by clicking on a shooting mode control. For example, after a user clicks on the HDR shooting mode control 10c-1, the phone responds to the operation, enters HDR shooting mode, and displays the following... Figure 4 The photo-taking interface 10d is shown in (2).

[0139] See Figure 4 (2) Figure 4(2) Exemplarily illustrates a preview interface 10d for HDR shooting mode. This interface 10d includes a top operation bar, an image preview window, and a bottom operation bar; the top operation bar includes a flash on / off control and a settings control; the bottom operation bar includes a shooting mode exit control 10d-1, a gallery quick access control, and a shooting control. The user can exit the current HDR shooting mode by clicking the control 10d-1.

[0140] After the user selects HDR shooting mode and points the camera at the subject, the camera app's image preview area displays the current image of the subject. For example, if the user points the camera at a three-legged ding (a type of ancient Chinese cooking vessel) in an indoor exhibition hall, the camera app's image preview area will display the image of the ding, showing something like... Figure 5 The interface 10e is shown. In this interface 10e, the camera's image preview area 10e-1 displays the image of the three-legged ding (a type of ancient Chinese cooking vessel).

[0141] However, in Figure 5 In the indoor exhibition hall photography scene shown, since the indoor exhibition hall uses artificial alternating current light source, the flickering phenomenon of artificial alternating current light source may cause flickering stripes to appear in the image preview area 10e-1 of the camera application and in the final image.

[0142] For example, see [link to previous article] Figure 5 , Figure 5 In the indoor exhibition hall scene shown, light source 200 is an artificial alternating current light source, and flickering stripes appear in the image preview area 10e-1 of the camera application. The appearance of flickering stripes greatly reduces the user's photography experience.

[0143] In view of this, embodiments of this application provide an image processing method to remove flicker stripes in artificial alternating current light source scenes, thereby improving the user's photography experience.

[0144] Let's continue with the example of taking photos in an indoor exhibition hall using a camera app, analyzing the mobile phone 100 and the light source 200 in the scene. The mobile phone 100 uses its CMOS image sensor to perform line-by-line exposure to generate the image. The light source 200 is driven by alternating current (AC). AC is a sine wave with two frequencies: 50Hz and 60Hz. In China, 50Hz is commonly used, so we will use 50Hz AC as an example here.

[0145] The period of a 50Hz alternating current is 1 / 50 of a second (s), or 20 milliseconds (ms), and its variation pattern is as follows: Figure 6The waveform of the light source is shown in the figure. As for energy, there is no positive or negative value; therefore, the energy cycle of a light source driven by alternating current is 1 / 100s, or 10ms, and its variation pattern is as follows: Figure 6 The energy waveform of the light source is shown. Therefore, the brightness of a light source powered by alternating current actually changes with the alternating current in a period of 10ms. Of course, this change is imperceptible to the human eye. After each exposure, the CMOS image sensor of a mobile phone generates an image frame, such as... Figure 6 The timing of the outgoing frames is shown.

[0146] However, when a mobile phone's CMOS image sensor uses a rolling shutter to generate images line by line, not all lines are exposed simultaneously. Therefore, each line in an image frame has a certain line interval during exposure, such as... Figure 7 As shown. This situation results in different exposure starting points for each row. The exposure process can be viewed as integrating brightness, and the magnitude of the integral directly determines the image brightness. When the exposure time is an integer multiple of the light source energy period, each row receives the same amount of energy, and there is no flickering. However, when the exposure time is not an integer multiple of the light source energy period, the energy received by each row is different, leading to differences in the exposure brightness of different rows within a single frame, causing flickering in the camera's image preview interface. Figure 5 and Figure 8 The flickering phenomenon shown.

[0147] See Figure 8 , Figure 8 This example illustrates the brightness relationship between the light source energy cycle and exposure time. In the scenario of taking photos in an indoor badminton court, the exposure time t of the phone's CMOS image sensor is less than the light source energy cycle T. Therefore, when the CMOS image sensor exposes line by line, it is at the peak of the light source energy cycle when exposing the first line, resulting in the first line appearing bright in the image frame. When exposing the second line, the light source energy cycle is decreasing, and for the same exposure time t, the energy integrated by the CMOS image sensor is weaker, leading to a decrease in brightness for the second line in the image frame. When exposing the third line, the light source energy cycle is at its lowest, and for the same exposure time t, the energy integrated by the CMOS image sensor is weakest, resulting in the lowest brightness for the third line in the image frame. Therefore, flickering occurs in the camera's image preview interface.

[0148] Currently, to eliminate the aforementioned flickering phenomenon, the phone 100 incorporates an Auto Exposure (AE) algorithm. This AE algorithm senses the intensity of external light and uses an exposure table to allocate exposure, ensuring that the exposure time is an integer multiple of the light source's energy cycle. For details on the principle of the AE algorithm, please refer to [link to relevant documentation]. Figure 9 .

[0149] See Figure 9 , Figure 9 The working principle of the AE algorithm is illustrated by example. Figure 9 In this process, the sensor (i.e., the aforementioned CMOS image sensor) first acquires the ambient light intensity and estimates an initial exposure parameter based on it. Then, it generates an image frame based on this initial exposure parameter. The statistical module (Stat3A) in the image signal processor (ISP) collects statistical information from this image frame, including but not limited to a brightness histogram and brightness region statistics. The statistical module sends this image frame's statistical information to the AE algorithm module. The AE algorithm module calculates the target brightness (targetLuma) and the current measured brightness (frameLuma) of the current scene based on the statistical information and compares them. If the current brightness is not equal to the target brightness, a new set of exposure parameters is obtained based on the target brightness. These exposure parameters include shutter speed and gain coefficient. The new exposure parameters are then sent to the sensor, enabling it to generate a new image frame based on these new exposure parameters, until the current measured brightness of the new image frame equals the target brightness.

[0150] However, in the aforementioned indoor exhibition hall photography scenario, users selected HDR mode to obtain superior image quality. Compared to traditional shooting modes, HDR mode has a much higher sensitivity, sometimes more than four times higher. However, at the same brightness, the higher sensitivity means the camera needs to reduce the exposure time / shutter speed to meet the required sensitivity. This makes it more difficult for the camera to achieve an exposure time that is an integer multiple of the light source's energy cycle in artificial light environments. When the exposure time is less than an integer multiple of the light source's energy cycle, the aforementioned after-effects (AE) algorithm is insufficient to eliminate flicker.

[0151] For example, consider the Dual Conversion Gain (DCG) mode in HDR. The principle of DCG mode is that the sensor, under the same exposure, uses different gain circuits (i.e., conversion gain) to read out two images with different brightness levels: a high-gain image (HCG) and a low-gain image (LCG). By fusing the high-gain and low-gain images, high dynamic range imaging can be achieved.

[0152] In DCG mode, the exposure time of two frames read from different gain circuits is the same. The sensitivity of the long frame image is 4 times that of the short frame image. Therefore, the sensitivity of the long frame image (corresponding to the high gain image) in DCG mode cannot be lower than 4 times the gain (referred to as gain, which refers to the signal value added by the camera to increase the brightness of the image). If the sensitivity of the long frame image is lower than 4 times the gain, it will cause abnormal sensor exposure and ultimately abnormal imaging.

[0153] Typically, the dynamic range of a scene can be calculated using the exposure of a long-frame image and the exposure of a short-frame image (corresponding to a low-gain image), with the formula: Dynamic Range = Long-frame image exposure / Short-frame image exposure. In low dynamic range scenes, with the same exposure time, the dynamic range of the image is reflected in the gain; that is, the ISO of the long-frame image cannot be less than 4 times the gain. Since the exposure parameters output by the After Effects (AE) algorithm need to reflect ISO, the exposure time needs to be reduced to increase the gain. However, as the exposure time decreases, the possibility of flickering increases.

[0154] For example, in low dynamic range scenes, the exposure parameters for short-frame images are 10ms exposure with 1x gain, and for long-frame images, 10ms exposure with 2x gain, resulting in a dynamic range of 2. However, the sensitivity of long-frame images is less than 4. If these exposure parameters are sent to the sensor, the sensor will exhibit exposure anomalies. Therefore, while maintaining the overall exposure, it is necessary to reduce the exposure time and increase the gain to meet the sensitivity requirements of long-frame images. Specifically, the exposure parameters for short-frame images are set to 5ms exposure with 2x gain, and for long-frame images, to 5ms exposure with 4x gain. However, because the exposure time is reduced, the likelihood of it being less than an integer multiple of the light source's energy period increases, increasing the possibility of flickering.

[0155] Therefore, the image processing method provided in this application, after obtaining the light source frequency and amplitude of the target scene, calculates the flicker intensity based on the light source frequency and amplitude, corrects the flicker intensity by exposure time, and when it is determined that the flicker phenomenon still cannot be eliminated after correction, calculates the dynamic expansion coefficient of the image, and applies the dynamic expansion coefficient to the target brightness of the long frame image, thereby expanding the dynamic range of the image, increasing the exposure of the long frame image, so that the gain of the long frame image meets the sensitivity requirements, and at the same time increasing the exposure time so that the exposure time can be kept as an integer multiple of the light source energy cycle, thereby achieving the purpose of reducing and eliminating the flicker phenomenon.

[0156] It should be understood that the above description is merely an example provided to better understand the technical solution of this embodiment, and is not intended as the only limitation of this embodiment.

[0157] To illustrate the image processing method provided in the embodiments of this application in detail, the following embodiments still use a scenario of taking photos in an indoor exhibition hall using a camera application, with the mobile device being a mobile phone. Figure 1 The hardware structure shown is as follows. Figure 2 Taking the software structure shown as an example, combined with... Figures 10 to 13 The image processing methods are explained in detail.

[0158] To make it easier to understand, let's first combine... Figure 10 The working principle of the AE algorithm in the embodiments of this application is explained. See [link to relevant documentation]. Figure 10 , Figure 10 This example illustrates the working principle of the AE algorithm in HDR mode of this application embodiment. In DCG mode of this application embodiment, the AE algorithm module needs to control the calculation and allocation of exposure parameters for long-frame and short-frame images respectively, thereby controlling the dynamic range capability of the camera. Specifically, in Figure 10In this process, the sensor first acquires the ambient light intensity and estimates an initial exposure parameter based on it. Then, based on this initial exposure parameter, it generates long-frame and short-frame images. The statistical module in the image signal processor collects statistical information from both the long-frame and short-frame images and sends it to the AE algorithm module. The AE algorithm module calculates the current brightness (weighted average brightness of the short-frame image) and target brightness for both the long-frame and short-frame images, and compares the current brightness of the short-frame image with the target brightness, and vice versa. If the current brightness of the long-frame image does not equal the target brightness, the exposure parameters for the long-frame image are calculated, along with a dynamic range expansion factor. This dynamic range expansion factor is then applied to the target brightness of the long-frame image to expand the dynamic range, increase the exposure of the long-frame image, and make it easier for the long-frame gain to meet the sensitivity requirements. This increases the exposure time and reduces or eliminates flicker.

[0159] After understanding the working principle of the AE algorithm, then combine it with Figure 11 The interaction flow of each functional module involved in the embodiments of this application is described. See [link to relevant documentation]. Figure 11 , Figure 11 This example illustrates the interaction flow of each functional module in the HDR mode within an indoor exhibition hall photography scene, as exemplified in an embodiment of this application. Figure 11 In this process, when the phone enters HDR mode, the flicker detection module starts working, acquiring the frequency and amplitude of light sources in the current scene in real time and outputting it to the AE algorithm module. Upon receiving the frequency and amplitude of the light sources, the AE algorithm module processes them to obtain the flicker intensity of the current scene, and then corrects the flicker intensity using exposure time to obtain the corrected flicker intensity. If it determines that the corrected flicker intensity might cause flickering in the current HDR mode, it calculates the dynamic range expansion coefficient, constrains the dynamic range expansion coefficient, and finally applies it to the target brightness of the long frame image. Simultaneously, the AE algorithm module recalculates the exposure parameters for both the long and short frame images and sends them to the sensor. Upon receiving the exposure parameters from the AE algorithm module, the sensor regenerates the long and short frame images. The image information processor's statistics module parses the long and short frame images output by the sensor and sends the statistical results to the AE algorithm module. The AE algorithm module recalculates based on the newly received statistical results until the flickering phenomenon is eliminated. Meanwhile, the image information processor's merging module fuses the long-frame and short-frame images output by each sensor and outputs them to the camera's preview interface for display.

[0160] After understanding the working principle of the AE algorithm and the interaction flow of each functional module involved in the embodiments of this application, then combining... Figure 12 The timing sequence of the image processing method according to the embodiments of this application will be described.

[0161] Step S101: The camera application enters HRD mode.

[0162] For example, the aforementioned camera application can be a system application provided by mobile phone 100, or it can be a third-party application that calls the interface of the camera application to implement camera functions. This application embodiment does not impose any limitations on this. When the aforementioned camera application is a system application, its access method can be through... Figure 3 Clicking the 10a-1 icon in (1) leads to the camera application. The camera application interface displayed after entering the application can be... Figure 3 Interface 10b is shown in (2).

[0163] For example, after entering interface 10b, the user can also access the camera's shooting mode selection by clicking "More Options". Figure 4 The interface 10c shown in (1) is displayed. In interface 10c, the HDR shooting mode is selected, causing the camera application to display as shown. Figure 4 Interface 10d is shown in (2).

[0164] It is understood that the above-mentioned HRD modes include, but are not limited to, single-needle high dynamic mode (Dual Analog Gain, DAG), dual conversion gain mode (Dual Conversion Gain, DCG), lateral overflow integration Capcity mode (LOFIC), etc.

[0165] It should be understood that after the camera application enters HDR mode, the phone's image sensor will expose at a certain frequency to capture long-frame and short-frame images of the current scene facing the camera. The image signal processor will fuse the long-frame and short-frame images in real time to obtain a preview image (the final captured image is obtained when the user clicks the shutter button) and display the preview image in the image preview area 10e-1 of the camera application.

[0166] In step S102, the flicker detection module is activated and collects the frequency and amplitude of the light source in the current scene in real time.

[0167] Understandably, the aforementioned flicker detection module is used to detect flicker in the current scene being faced by the camera. When the user opens the camera application and enters HDR mode, this module will perform flicker detection on the current scene in real time, obtaining the frequency and amplitude of light sources in the current scene, until the user exits the HDR mode.

[0168] For example, the current scenario described above could be Figure 5 The image shows a scene inside an indoor exhibition hall. In this scene, the flicker detection module collects the frequency and amplitude of the AC light source 200 in real time. Since the frequency of the AC power driving the light source 200 is 50Hz, the frequency of the light source is 50Hz, its energy period is 10ms, and its variation pattern is as follows: Figure 6 The energy waveform of the light source is shown in the figure.

[0169] In step S103, the flicker detection module sends the collected light source frequency and amplitude to the automatic exposure module.

[0170] For example, the automatic exposure module mentioned above integrates an automatic exposure (AE) algorithm for calculating and allocating exposure parameters for long-frame and short-frame images in HDR mode.

[0171] It should be noted that after the camera application enters HDR mode, the automatic exposure module first estimates an initial exposure parameter based on the ambient light intensity of the current scene, and then sends this initial exposure parameter to the image sensor so that the image sensor can produce initial long-frame and short-frame images based on this initial exposure parameter. This initial exposure parameter includes, but is not limited to, shutter speed (exposure time) and gain.

[0172] In step S104, the automatic exposure module calculates and corrects the flicker intensity of the current scene based on the frequency and amplitude of the light source.

[0173] For example, after obtaining the frequency and amplitude of the light source, the automatic exposure module determines the light source in the current scene. If the frequency of the light source changes periodically and the amplitude is not zero, the light source in the current scene is determined to be a stroboscopic light source, and its stroboscopic intensity needs to be calculated; otherwise, the process of this embodiment ends.

[0174] As is understandable, flicker intensity is used to indicate the probability of flickering when taking photos in HDR mode in the current scene. When the exposure time is close to an integer multiple of the light source period, the flicker intensity is less than the preset flicker threshold, meaning that the image based on the current exposure parameters will not produce flickering; when the difference between the exposure time and the light source period is large, the flicker intensity is greater than the preset threshold, meaning that the image based on the current exposure parameters will produce flickering.

[0175] It should be understood that the amplitude of a light source refers to the magnitude of the change in its brightness. The larger the amplitude, the greater the range of brightness variation, and the more pronounced the flicker phenomenon. Therefore, the amplitude represents the flicker intensity of the current scene. Typically, AC light sources use 220 volts (V) AC power, so their amplitude is 311V. The flicker intensity is obtained by dimensionless processing. This dimensionless processing can be achieved by scaling the amplitude to the range [0, 10] and then rounding it. For example, scaling the amplitude 311V to the range [0, 10] and then rounding it yields a flicker intensity of 3.

[0176] For example, since flicker intensity is affected by exposure time, in order to obtain the flicker intensity of the current scene more accurately, it is necessary to correct it by the difference between the current exposure time and the target exposure time. The correction process is as follows:

[0177] Step S1041: Calculate the exposure ratio based on the current exposure time and the target exposure time.

[0178] It is understandable that the aforementioned current exposure time is also the current exposure time used in the image sensor. The aforementioned target exposure time is an integer multiple of the light source energy cycle in the current scene, and the target exposure time is taken as the light source energy cycle that is closest to the current exposure time as an integer multiple.

[0179] For example, in a 50Hz AC light source scenario, when the current exposure time is 5ms, the target exposure time is 10ms, which is one time the energy cycle of the light source; when the current exposure time is 17ms, the target exposure time is 20ms, which is two times the energy cycle of the light source.

[0180] For example, the formula for calculating the exposure ratio (expoRatio) is:

[0181] expoRatio=targetExpoTime / currentExpoTime;

[0182] Where targetExpoTime is the target exposure time, which is an integer multiple of the current light source cycle, banding Step, and currentExpoTime is the current exposure time.

[0183] Step S1042: Calculate the flicker intensity adjustment coefficient based on the mapping relationship between the preset exposure ratio and the adjustment coefficient.

[0184] It is understandable that the above-mentioned preset exposure ratio and adjustment coefficient mapping relationship can be built into the phone's memory, used to calculate the flicker intensity through the exposure ratio, and its array form is as follows: exposure ratio: expoRatioZone

[10] ={0,1,2,3,4,5,6,7,8,9}, adjustment coefficient: adjustRatioZone

[10] ={0.1,0.5,1.5,3,4,5,6,7,8,9}; its list form is shown in Table 1.

[0185] Table 1. Mapping Table of Exposure Ratio and Adjustment Factor

[0186] Exposure ratio 0 1 2 3 4 5 6 7 8 9 Adjustment coefficient 0.1 0.5 1.5 3 4 5 6 7 8 9

[0187] It should be understood that, according to the exposure ratio calculation formula above, the exposure ratio may not be an integer. When it contains a decimal, the decimal part needs to be corrected by rounding up. For example, when the exposure ratio is 1.1, it becomes 2 after rounding up.

[0188] Specifically, after obtaining the exposure ratio and the mapping relationship between the exposure ratio and the adjustment coefficient, the final flicker intensity adjustment coefficient, adjustRatio, is calculated through linear interpolation. The calculation formula is as follows:

[0189] adjustRatio=adjustRatioZone[i-1]+(expoRatio-expoRatioZone[i-1])*(adjustRatioZone[i]-adjustRatioZone[i-1]) / (expoRatioZone[i]-expoRatioZone[i-1]).

[0190] Where i is the position of the corrected exposure ratio in the array expoRatioZone, and expoRatio is the original exposure ratio.

[0191] Step S1043: Correct the flicker intensity according to the flicker intensity adjustment coefficient.

[0192] Specifically, after calculating the flicker intensity adjustment coefficient, the flicker intensity waveDepth of the current scene is corrected based on the flicker intensity adjustment coefficient. The calculation formula is as follows:

[0193] waveDepth=waveDepth*adjustRatio.

[0194] In step S105, the automatic exposure module determines that the flicker intensity of the current scene exceeds the preset intensity threshold.

[0195] Specifically, the automatic exposure module performs a threshold judgment on the corrected strobe intensity. If the strobe intensity is greater than the preset intensity threshold, step S106 is executed; otherwise, the process ends.

[0196] It is understood that the aforementioned preset intensity threshold can be a set value that can be set according to actual needs. Typically, the preset intensity threshold can be set to 3. When the flicker intensity threshold is greater than 3, flicker removal processing is required, and step S106 is executed.

[0197] Step S106: When the strobe intensity exceeds the preset intensity threshold, the automatic exposure module calculates the dynamic range expansion coefficient.

[0198] Specifically, the dynamic range extension factor drExtensionRatio is calculated as the ratio of the target exposure in the long frame to the current exposure in the long frame. The calculation formula is as follows:

[0199] drExtensionRatio=(targetExpoValue*sensitivity) / curExpoValue;

[0200] Wherein, targetExpoValue is the target exposure value of the long frame image, targetExpoValue = targetExpoTime * 1 x gain (i.e., the product of target exposure time and 1x gain), sensitivity is the sensitivity of the current mode, and curExpoValue is the current exposure value of the long frame image, curExpoValue = currentExpoTime * 1 x gain (i.e., the product of current exposure time and current gain).

[0201] It is understandable that the aforementioned sensitivity is one of the fixed parameters of the image sensor, which the AE algorithm module can directly read. The sensitivity value mentioned above is typically 4.

[0202] Step S107: The automatic exposure module determines whether the dynamic range expansion factor exceeds the limit value.

[0203] In step S108, the automatic exposure module constrains the dynamic range expansion coefficient when the dynamic range expansion coefficient exceeds the limit value.

[0204] Understandably, excessive dynamic range expansion can lead to overly high brightness in long-frame images, potentially causing abnormal fusion results when the ISP fuses long and short frames. Therefore, it's necessary to limit the dynamic range expansion coefficient. Since the minimum gain coefficient for long-frame images needs to be set to a sensitivity of 4, the maximum value of the dynamic range expansion coefficient is set to the image sensor's sensitivity, i.e., sensitivity = 4.

[0205] Furthermore, when the ambient brightness is high, the brightness of long-frame images is also very high. If the sensitivity is still used as the maximum expansion factor in this case, the ISP fusion anomaly problem will still occur. Therefore, it is also necessary to attenuate the maximum expansion factor according to the ambient brightness and interpolate the maximum expansion factor by looking up a table for the current ambient brightness.

[0206] It should be understood that the Ambient LightValue measures the brightness of the current scene, and it can be calculated from shutter speed, aperture, ISO, and statistical information. The smaller the value, the darker the current scene, and vice versa.

[0207] For example, the mapping relationship between the ambient brightness and the maximum expansion coefficient can be built into the phone's memory. Its array representation is as follows: Ambient brightness: LightValueZone[6] = {90, 100, 110, 120, 130, 140}, Maximum expansion coefficient: maxRatioZone[6] = {4, 3.5, 3, 2.5, 2, 1}. Its list representation is shown in Table 2.

[0208] Table 2. Mapping Table Between Ambient Brightness and Maximum Spread Coefficient

[0209] Ambient brightness 90 100 110 120 130 140 Maximum expansion coefficient 4 3.5 3 2.5 2 1

[0210] Specifically, after obtaining the mapping relationship between ambient brightness and the maximum spread factor, the maximum spread factor maxRatio is calculated using linear interpolation. The calculation formula is as follows:

[0211] maxRatio=maxRatioZone[i-1]+(LightValue-LightValueZone[i-1])*(maxRatioZone[i]-maxRati oZone[i-1]) / (LightValueZone[i]-LightValueZone[i-1]);

[0212] Where i is the position of the ambient brightness LightValue in the array LightValueZone (for example, when the ambient brightness is 115, LightValueZone[3]>LightValue>LightValueZone[2], i equals 3), and LightValue is the ambient brightness.

[0213] After calculating the maximum extension coefficient, the dynamic range extension coefficient is restricted based on the maximum extension coefficient. The restriction calculation formula is: drExtensionRatio=min(drExtensionRatio,maxRatio).

[0214] In step S109, the automatic exposure module uses a dynamic expansion coefficient to correct the target brightness of the long frame image and regenerates the exposure parameters of the long frame image and the short frame image.

[0215] Specifically, the automatic exposure module applies the limited dynamic expansion coefficient to the target brightness of the long frame image for correction. The correction formula is as follows:

[0216] longTargetLuma=longTargetLuma*drExtensionRatio;

[0217] Simultaneously, the automatic exposure module recalculates the exposure parameters for the next long frame image, including the target exposure value (targetExpoValue) and the next exposure value (nextExpoValue) for the next long frame image. The calculation formula is as follows:

[0218] targetExpoValue=curExpoValue*longTargetLuma / curLuma;

[0219] Where, curExpoValue is the current exposure of the long frame image, longTargetLuma is the corrected target brightness of the long frame image, and curLuma is the current brightness of the long frame image (the brightness value output by the image sensor after weighted averaging of the current long frame image).

[0220] nextExpoValue=lastExpoValue+(targetExpoValue-lastExpoValue)*step;

[0221] Where lastExpoValue is the exposure of the previous long frame image (i.e., the curExpoValue mentioned above), and step is the exposure adjustment coefficient.

[0222] Understandably, the step can be set according to the need for a smooth transition of the image when removing stripes. By controlling the exposure to be adjusted next time, the preview images of adjacent frames will not have obvious flickering, thus improving the user experience.

[0223] The automatic exposure module decomposes the recalculated target exposure and the exposure of the next frame image into exposure parameters, and sends the obtained exposure parameters to the image sensor. These exposure parameters include, but are not limited to, exposure time and gain.

[0224] In step S110, the image sensor generates long-frame and short-frame images based on the exposure parameters reissued by the automatic exposure module.

[0225] In step S111, the image signal processing module calculates the exposure of the long frame image and the short frame image, and sends the calculation results to the automatic exposure module.

[0226] Step S112: The automatic exposure module performs debanding processing.

[0227] Specifically, debanding is implemented to obtain the preview area display of the camera application interface, as shown below. Figure 13 As shown.

[0228] Therefore, the image processing method of this application embodiment, after the mobile phone enters HDR mode, the flicker detection module starts working, real-time acquiring the frequency and amplitude of the light source in the current scene, and outputting it to the AE algorithm module. Upon receiving the frequency and amplitude of the light source, the AE algorithm module processes the frequency and amplitude to obtain the flicker intensity of the current scene, and corrects the flicker intensity using exposure time to obtain the corrected flicker intensity. After determining that the corrected flicker intensity might cause flicker in the current HDR mode, it calculates the dynamic range expansion coefficient, constrains the dynamic range expansion coefficient, and finally applies it to the target brightness of the long frame image. Simultaneously, the AE algorithm module recalculates the exposure parameters of the long frame image and the short frame image respectively and sends them to the sensor. The sensor, upon receiving the exposure parameters from the AE algorithm module, regenerates the long frame image and the short frame image. The statistical module of the image information processor parses the long frame image and the short frame image output by the sensor and sends the statistical results to the AE algorithm module. The AE algorithm module recalculates based on the newly received statistical results until the flicker phenomenon is eliminated. This solves the problem of bright and dark stripes appearing in the preview screen and photos of mobile terminal camera applications in artificial light source scenarios, thus improving the user's photography experience.

[0229] Furthermore, it is understood that, in order to achieve the aforementioned functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. Based on the algorithmic steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in a hardware-driven or software-driven manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0230] Furthermore, it should be noted that, in practical application scenarios, the image processing methods provided in the above embodiments, implemented by electronic devices, can also be executed by a chip system included in the electronic device. This chip system may include a processor. The chip system can be coupled to a memory, enabling it to call computer programs stored in the memory during runtime to implement the steps executed by the electronic device. The processor in this chip system can be an application processor or a non-application processor.

[0231] In addition, this application embodiment also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the above-described related method steps to implement the image processing method in the above embodiment.

[0232] In addition, this application also provides a computer program product that, when run on an electronic device, causes the electronic device to perform the above-mentioned related steps to implement the image processing method in the above embodiments.

[0233] In addition, embodiments of this application also provide a chip (which may also be a component or module), which may include one or more processing circuits and one or more transceiver pins; wherein the transceiver pins and the processing circuits communicate with each other through internal connection paths, and the processing circuits execute the above-mentioned related method steps to implement the image processing method in the above embodiments, so as to control the receiving pin to receive signals and control the transmitting pin to transmit signals.

[0234] Furthermore, as can be seen from the above description, the electronic devices, computer-readable storage media, computer program products, or chips provided in the embodiments of this application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0235] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized in that, An image processing method is applied to an electronic device, wherein the scene being captured by the electronic device is illuminated by a strobe light source, the strobe light source comprising a frequency and an amplitude, the frequency being used to indicate the strobe period of the strobe light source, and the image processing method comprising: In response to the user's first operation, a first preview image is displayed on the camera interface. The first preview image is obtained by fusing a first long frame image and a first short frame image. The camera interface is the interface of the camera application in the electronic device. Obtain long-frame statistical parameters and long-frame exposure parameters of the first long-frame image, and short-frame statistical parameters of the first short-frame image. The long-frame statistical parameters include the target brightness of the first long-frame and the current brightness of the first long-frame. The long-frame exposure parameters include the target exposure time of the first long-frame, the current exposure time of the first long-frame, gain, and sensitivity. The short-frame statistical parameters include the target brightness of the first short-frame and the current brightness of the first short-frame. The amplitude of the stroboscopic light source is dimensionless to obtain the stroboscopic intensity; When the flicker intensity is greater than a preset intensity threshold, the dynamic range expansion coefficient of the first long frame image is calculated based on the first long frame target exposure time, the first long frame current exposure time, gain, and sensitivity. The brightness of the first long frame target is corrected based on the dynamic range expansion coefficient. Based on the corrected target brightness of the first long frame and the current brightness of the first long frame, the long frame exposure parameters are regenerated, and the short frame exposure parameters are regenerated based on the statistical parameters of the first short frame image. A second preview image is generated based on the long frame exposure parameters and the short frame exposure parameters and displayed on the camera interface.

2. The method according to claim 1, characterized in that, Before displaying the first preview image on the camera interface, the following steps are included: Based on the obtained light intensity of the current environment of the shooting scene, initial exposure parameters are generated, and a first preview image is generated based on the initial exposure parameters.

3. The method according to claim 1, characterized in that, The step of dimensionless processing of the amplitude of the stroboscopic light source to obtain the stroboscopic intensity further includes: The exposure ratio of the first long frame image is obtained based on the ratio of the exposure time of the first long frame target to the current exposure time of the first long frame. Based on the exposure ratio and the mapping relationship between the preset exposure ratio and the adjustment coefficient, the flicker intensity adjustment coefficient is calculated using linear interpolation. The flicker intensity is corrected based on the flicker intensity adjustment coefficient.

4. The method according to claim 1, characterized in that, After calculating the dynamic range expansion coefficient of the first long frame image based on the first long frame target exposure time, the first long frame current exposure time, gain, and sensitivity, the process includes: If the dynamic range expansion coefficient exceeds a preset limit, the dynamic range expansion coefficient is corrected.

5. The method according to claim 4, characterized in that, The correction of the dynamic range extension coefficient includes: The ambient brightness of the shooting scene is calculated based on the shutter speed, aperture, ISO of the camera on the electronic device and the long frame statistical parameters. Based on the mapping relationship between the ambient brightness and the preset ambient brightness and the maximum expansion coefficient, the maximum expansion coefficient corresponding to the shooting scene is calculated using linear interpolation. The dynamic range expansion coefficient is corrected based on the maximum expansion coefficient.

6. The method according to claim 5, characterized in that, The dynamic range expansion coefficient is corrected based on the maximum expansion coefficient, including: If the maximum expansion factor is less than the dynamic range expansion factor, the dynamic range expansion factor is updated to the maximum expansion factor.

7. The method according to claim 1, characterized in that, The brightness of the first long frame target is corrected based on the dynamic range expansion coefficient, including: The brightness of the first long frame target is amplified using the dynamic range expansion coefficient.

8. The method according to claim 1, characterized in that, Based on the corrected target brightness of the first long frame and the current brightness of the first long frame, the long frame exposure parameters are regenerated, including: The current exposure of the first long frame image is calculated based on the exposure time and the gain. Based on the current exposure, the target brightness of the first long frame, and the current brightness of the first long frame, the target exposure of the next long frame image is calculated. Calculate the exposure of the next long frame image based on the current exposure and the target exposure. Based on the exposure amount, long frame exposure parameters are regenerated.

9. The method according to claim 8, characterized in that, Calculating the exposure of the next long frame image based on the current exposure and the target exposure also includes: The exposure is adjusted based on a preset exposure adjustment coefficient.

10. The method according to claim 1, characterized in that, The amplitude of the stroboscopic source is dimensionlessly reduced to obtain the stroboscopic intensity, including: The stroboscopic intensity is obtained by rounding down the amplitude scaling value of the stroboscopic light source within a preset range.

11. An electronic device, characterized in that, The electronic device includes: a memory and a processor, the memory and the processor being coupled; the memory stores program instructions, which, when executed by the processor, cause the electronic device to perform the image processing method as described in any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, The method includes a computer program that, when run on an electronic device, causes the electronic device to perform the image processing method as described in any one of claims 1 to 10.

Images