Flickering light source detection method and electronic equipment

Abstract

The invention provides a flicker light source detection method and electronic equipment, and in the method, the electronic equipment sets the exposure time of a first camera as an integral multiple of a first light energy period, sets the exposure time of a second camera as an integral multiple of a second light energy period, and carries out flicker detection on images collected by the two cameras. And if the first image contains the water ripple characteristics, determining that a flickering light source of a second light energy period exists in the environment, and exchanging the light energy periods corresponding to the two cameras. And if the second image contains the water ripples, determining that the flickering light source of the first light energy period exists in the environment, and keeping the setting of the current light energy period. If the two images do not contain the water ripples, the flickering light source does not exist in the environment. In this way, whether the environment is a flickering light source with the first light energy period or a flickering light source with the second light energy period or not can be detected at a time; the exposure time is adjusted according to the detection result, and the water ripple phenomenon is avoided.

Description

Technical Field

[0001] The present application relates to the field of terminals, and in particular to a method and electronic device for detecting a flickering light source. Background Art

[0002] With the development of mobile terminals and the maturity of image processing technology, people's requirements for terminal photography are gradually increasing. In practical applications, using a rolling shutter camera to take pictures under an AC light source (such as fluorescent lamps, TVs, computer screens or household light sources) may cause "water ripples" to appear in the image, seriously affecting the image quality.

[0003] In order to improve the image quality, the device is usually equipped with a dedicated flicker detector to detect whether there is a flickering light source in the shooting environment, and the exposure time is set to an integer multiple of the corresponding light energy cycle with the camera to solve the water ripple problem. However, such devices require additional costs and design space. Summary of the invention

[0004] The embodiments of the present application provide a method and electronic device for detecting flickering light sources. Without introducing additional dedicated detection devices, it is possible to detect at one time whether there is a flickering light source in the environment and the specific type of the flickering light source. The exposure time of the camera can then be adjusted according to the specific type of the flickering light source, thereby preventing users from seeing water ripples caused by various types of flickering light sources in the images sent by the camera, thereby effectively improving the user experience.

[0005] In a first aspect, the present application provides a method for detecting a flickering light source, which is applied to an electronic device including a first camera and a second camera, the method comprising: setting an exposure time of the first camera to an integer multiple of a first light energy cycle, and the first camera collecting a first image; the electronic device displays a preview image on a display screen, the preview image being an image collected by the first camera; setting the exposure time of the second camera to an integer multiple of a second light energy cycle, and the second camera collecting a second image; wherein the shutters of the first camera and the second camera are both rolling shutters; the electronic device performs water ripple detection according to the first image and the second image; when the first image contains a water ripple feature, there is a flickering light source corresponding to the second light energy cycle in the environment, and the electronic device sets the exposure time of the first camera to an integer multiple of the second light energy cycle, and sets the exposure time of the second camera to an integer multiple of the first light energy cycle; when the first image does not contain a water ripple feature and the second image contains a water ripple feature, there is a flickering light source corresponding to the first light energy cycle in the environment, and the exposure time of the first camera is kept as an integer multiple of the first light energy cycle, and the exposure time of the second camera is kept as an integer multiple of the second light energy cycle; when both the first image and the second image do not contain a water ripple feature, there is no flickering light source in the environment.

[0006] In implementing the method provided in the first aspect, the electronic device can respectively set the exposure time of the first camera to an integer multiple of the first light energy cycle, and set the second camera to an integer multiple of the second light energy cycle. According to the first image collected by the first camera and the second image collected by the second camera, the electronic device can obtain the flickering light source information in the current shooting environment, and then perform exposure control on the first camera and the second camera. If the first image contains a water ripple feature, the electronic device can determine that there is a flickering light source corresponding to the second light energy cycle in the environment, and the electronic device can set the exposure time of the first camera to an integer multiple of the second light energy cycle, thereby avoiding the appearance of water ripples in the preview image provided by the first camera. In order to continue to detect the flickering light source in the shooting environment, the exposure time of the second camera can also be changed to an integer multiple of the first light energy cycle. If the first image does not contain a water ripple feature and the second image contains a water ripple feature, the electronic device can determine that there is a flickering light source corresponding to the first light energy cycle in the environment, and the electronic device can maintain the current settings of the light energy cycles corresponding to the first camera and the second camera. If the images collected by both cameras do not contain water ripple features, the electronic device can determine that there is no flickering light source in the environment.

[0007] Compared with the existing solution of using a dedicated flicker detector to detect whether there is a flickering light source in the shooting environment, this solution of using two existing cameras of the electronic device to detect the flickering light source information in the shooting environment can effectively reduce the solution cost and save design space on the premise of detecting the flickering light source in the shooting environment. The two cameras cooperate in detection, and can also help the electronic device quickly obtain the flickering light source information in the current shooting environment in one detection, so as to eliminate water ripples faster and give users a better user experience.

[0008] In combination with the first aspect, in some embodiments, when the first image includes water ripple features, the method also includes: the electronic device adjusts the exposure time of the first camera, and the minimum adjustment unit and minimum value of the exposure time of the first camera are both the second light energy period currently corresponding to the first camera.

[0009] By implementing the method provided in the above embodiment, when the first image contains water ripples, the electronic device can determine that there is a flickering light source corresponding to the second light energy cycle in the environment. In the case where there is a flickering light source corresponding to the second light energy cycle in the shooting environment, the electronic device can further use the current second light energy cycle of the first camera as the adjustment accuracy (i.e., the minimum adjustment unit) to fine-tune the exposure time of the first camera when there is no water ripple in the first image currently captured by the first camera for display (i.e., after swapping the light energy cycles corresponding to the first camera and the second camera by setting the exposure time), thereby further improving the clarity and exposure of the image and improving the imaging quality.

[0010] In combination with the first aspect, in some embodiments, when the first image does not include water ripple features and the second image includes water ripple features, the method also includes: the electronic device adjusts the exposure time of the first camera, and the minimum adjustment unit and minimum value of the exposure time of the first camera are both the first light energy period currently corresponding to the first camera.

[0011] By implementing the method provided in the above embodiment, when the first image does not contain water ripple features and the second image contains water ripple features, the electronic device can determine that there is a flickering light source corresponding to the first light energy period in the environment. In the case where there is a flickering light source corresponding to the first light energy period in the shooting environment, the electronic device can further use the current first light energy period of the first camera as the adjustment accuracy (i.e., the minimum adjustment unit) to fine-tune the exposure time of the first camera after flicker detection, thereby further improving the clarity and exposure of the image and improving the imaging quality.

[0012] In combination with the first aspect, in some embodiments, when the first image and the second image do not contain water ripple features, the method also includes: the electronic device adjusts the exposure time of the first camera, and the minimum adjustment unit and minimum value of the exposure time of the first camera are both smaller than the first light energy period and the second light energy period.

[0013] By implementing the method provided in the above embodiment, when there is no flickering light source in the shooting environment, the electronic device is not limited to using the light energy period as the adjustment accuracy of the exposure time of the first camera (i.e., the minimum adjustment unit), so that the electronic device can adjust the exposure time according to other factors that actually affect the imaging quality, such as the actual light conditions of the current scene or the actual movement speed of the object being shot, thereby obtaining better shooting effects.

[0014] In combination with the first aspect, in some embodiments, the electronic device adjusts the exposure time of the first camera, specifically including: the electronic device performs image recognition on the first image to determine whether the content of the first image contains motion blur features; when the first image contains motion blur features, the electronic device reduces the exposure time of the first camera.

[0015] By implementing the method provided in the above embodiment, the electronic device can perform image recognition on the first image when the image captured by the first camera does not have water ripples, or after the water ripples are eliminated. If there are motion blur features in the content of the first image, it means that the current exposure time of the first camera is too long. At this time, the electronic device can reduce the exposure time of the first camera, thereby reducing the motion blur in the first image caused by the long exposure time set during flicker detection, and improving the clarity of the image.

[0016] In combination with the first aspect, in some embodiments, the electronic device adjusts the exposure time of the first camera, specifically including: the electronic device reduces the exposure time of the first camera through automatic exposure control AEC.

[0017] By implementing the method provided in the above embodiment, the electronic device can use AEC to further reduce the exposure time of the first camera when there is no water ripple in the image captured by the first camera, or after the water ripple is eliminated, so that the first camera for display can set the accurate exposure time in real time in the changing ambient light, thereby improving the imaging quality.

[0018] In combination with the first aspect, in some embodiments, the exposure time of the first camera is set to an integer multiple of the first light energy cycle, specifically including: when the electronic device starts the first camera, the exposure time of the first camera is set to an integer multiple of the first light energy cycle; the method also includes: when the electronic device starts the first camera, the second camera is started; the exposure time of the second camera is set to an integer multiple of the second light energy cycle, specifically including: when the second camera is started, the exposure time of the second camera is set to an integer multiple of the second light energy cycle.

[0019] By implementing the method provided in the above embodiment, the electronic device can set the initial exposure time when starting the first camera, that is, when starting the first camera, the exposure time of the first camera is set to an integer multiple of the first light energy cycle. When the first camera is started, the second camera can be started accordingly, and its exposure time is set to an integer multiple of the second light energy cycle. Since the start of the second camera is triggered by the start of the first camera, the user does not perform any operation, and therefore, the user is not aware of the start of the second camera, which simplifies the user operation.

[0020] In combination with the first aspect, in some embodiments, before the electronic device starts the first camera, the method further includes: the electronic device detects a user operation of opening a camera application.

[0021] In combination with the first aspect, in some embodiments, the exposure time of the first camera is set to an integer multiple of the first light energy cycle, specifically including: the electronic device determines the automatic exposure time of the first camera through the automatic exposure control AEC; the electronic device determines the value of the integer multiple based on the automatic exposure time and the first light energy cycle; the electronic device sets the exposure time of the first camera to an integer multiple of the first light energy cycle.

[0022] In combination with the first aspect, in some embodiments, after the electronic device displays a preview image on a display screen, the method also includes: the electronic device detects a first zoom operation; the preview image displayed by the electronic device is switched to an image captured by a second camera; and the first zoom operation is used to adjust the zoom ratio from the zoom range of the first camera to the zoom range of the second camera.

[0023] By implementing the method provided in the above embodiment, the electronic device can switch the camera for display to the second camera when the zoom ratio is adjusted from the zoom range of the first camera to the zoom range of the second camera.

[0024] Among them, adjusting the zoom ratio from the zoom range of the first camera to the zoom range of the second camera includes reducing the zoom ratio to the zoom range of the second camera, and also includes increasing the zoom ratio to the zoom range of the second camera.

[0025] In combination with the first aspect, in some embodiments, in response to the first zoom operation, before switching the preview image to the image captured by the second camera, the method also includes: if the exposure time of the first camera is currently constrained to be an integer multiple of the first light energy period, setting the exposure time of the second camera to an integer multiple of the first light energy period, and the second camera captures a fourth image; setting the exposure time of the first camera to an integer multiple of the second light energy period, and the second camera captures a third image; if the exposure time of the first camera is currently constrained to be an integer multiple of the second light energy period, setting the exposure time of the second camera to an integer multiple of the second light energy period, and the second camera captures a fourth image; setting the exposure time of the first camera to an integer multiple of the first light energy period An integer multiple of the first light energy period or the second light energy period is set, and the second camera captures the third image; if the exposure time of the first camera is not currently constrained to be an integer multiple of the first light energy period or the second light energy period, the exposure time of the second camera is set to be an integer multiple of the second light energy period, and the second camera captures the fourth image; the exposure time of the first camera is set to be an integer multiple of the first light energy period, and the first camera captures the third image; or, when the exposure time of the first camera is not currently constrained to be an integer multiple of the first light energy period or the second light energy period, the exposure time of the second camera is set to be an integer multiple of the first light energy period, and the second camera captures the fourth image; the exposure time of the first camera is set to be an integer multiple of the second light energy period, and the first camera captures the third image.

[0026] When the method provided in the above embodiment is implemented, if the exposure time of the first camera is constrained to be an integer multiple of the first light energy cycle or the second light energy cycle, when the zoom ratio of the electronic device is adjusted to the zoom ratio of the second camera, the electronic device can reset the exposure time of the two cameras to swap the light energy cycles corresponding to the two cameras. In this way, after the electronic device performs the water ripple elimination, it can perform the zoom operation so that when the display camera is switched from the first camera to the second camera, the first frame of the fourth image displayed by the second camera does not have water ripples, thereby achieving smooth zooming.

[0027] If the exposure time of the first camera is not limited to an integer multiple of the first light energy period or the second light energy period at this time, then when the zoom magnification of the electronic device is adjusted to the zoom magnification of the second camera, the electronic device can first set the first camera and the second camera to integer multiples of the light energy period, and then detect and eliminate water ripples for the third image and the fourth image captured after the set exposure time, thereby ensuring that there are no water ripples in the first frame of the fourth image sent by the second camera, thereby achieving smooth zoom.

[0028] In combination with the first aspect, in some embodiments, in response to the first zoom operation, before switching the preview image to the image captured by the second camera, the method also includes: the electronic device performs water ripple detection based on the third image and the fourth image; when the fourth image contains water ripple features, the electronic device sets the exposure time of the second camera to the first exposure time, and the first exposure time is an integer multiple of a light energy period different from a light energy period currently corresponding to the second camera; when the fourth image does not contain water ripple features and the third image contains water ripple features, the exposure time of the second camera is maintained at an integer multiple of the light energy period currently corresponding to the second camera; when both the fourth image and the third image do not contain water ripple features, there is no flickering light source in the environment.

[0029] When implementing the method provided in the above embodiment, the zoom operation can occur after any link of the above flickering light source detection method. Therefore, the electronic device cannot guarantee that when performing zoom operation 1, the electronic device has completed at least one round of water ripple detection and elimination. If the electronic device has not completed at least one round of water ripple detection and elimination, that is, the light energy cycle set by the first camera is different from the flickering light source in the current shooting environment, there is no water ripple in the image captured by the second camera. At this time, since the electronic device swaps the light energy cycles corresponding to the first camera and the second camera after performing zoom operation 1, water ripple features exist in the image captured by the second camera after the swap. In this case, after the electronic device performs zoom operation 1 and before the fourth image captured by the second camera is sent for display, the third image captured by the first camera and the fourth image captured by the second camera are detected and eliminated for water ripples, so that when the zoom operation occurs, even if the electronic device has not yet performed water ripple detection and elimination, it can be ensured through this step that the fourth image sent for display does not contain water ripple features.

[0030] In combination with the first aspect, in some embodiments, before the electronic device detects the first zoom operation, the method also includes: when the zoom ratio is adjusted to the first zoom range, the exposure time of the second camera is constrained to a third light energy cycle, and the electronic device collects the fifth image of the first camera and the sixth image of the second camera respectively; the third light energy cycle is the first light energy cycle or the second light energy cycle; wherein, when the first zoom range is a zoom range between a first focal length threshold and a zoom range of the second camera, the first focal length threshold belongs to the zoom range of the first camera, and the minimum difference between the first focal length threshold and the focal length within the zoom range of the second camera is a preset value.

[0031] By implementing the method provided in the above embodiment, before switching the main camera for display to the second camera, when the zoom ratio is within the first zoom range (i.e., the transition zoom range), the exposure time of the second camera is constrained to the third light energy cycle (i.e., light energy cycle 3), so that the exposure time of the second camera is matched with the flickering light source in the environment in advance, so that the first image displayed by the second camera after switching the camera will not have water ripple features, thereby ensuring a smooth zoom user experience.

[0032] In combination with the first aspect, in some embodiments, before the electronic device detects the first zoom operation, the method also includes: the electronic device performs water ripple detection based on the fifth image and the sixth image; when the sixth image contains water ripple features, setting the exposure time of the second camera and the current exposure time of the first camera to integer multiples of a fourth light energy cycle, and the fourth light energy cycle is a light energy cycle different from the third light energy cycle; when the sixth image does not contain water ripple features and the fifth image contains water ripple features, setting the current exposure time of the first camera to an integer multiple of the third light energy cycle; when both the fifth image and the sixth image do not contain water ripple features, there is no flickering light source in the environment.

[0033] When implementing the method provided in the above embodiment, the zoom operation may occur after any step of the above flickering light source detection method. Therefore, the electronic device cannot guarantee that when performing zoom operation 1, the electronic device has completed at least one round of water ripple detection and elimination. At this time, the electronic device can perform water ripple detection and elimination based on the fifth image and the sixth image, so that when the zoom operation occurs, even if the electronic device has not yet performed water ripple detection and elimination, it can be ensured through this step that the fourth image sent for display does not contain water ripple features.

[0034] In combination with the first aspect, in some embodiments, the method also includes: when the sixth image includes water ripple features, in response to the first zoom operation, setting the exposure time of the first camera to an integer multiple of the third light energy period; when the sixth image does not include water ripple features and the fifth image includes water ripple features, in response to the first zoom operation, setting the current exposure time of the first camera to an integer multiple of the fourth light energy period.

[0035] By implementing the method provided in the above embodiment, the electronic device can adaptively set the exposure time of the first camera after sending and displaying the fourth image, so that the light energy cycle corresponding to the first camera is different from the light energy cycle corresponding to the second camera, so that the electronic device can continue to detect the flickering light source in the shooting environment through the first camera and the second camera after performing a zoom operation and switching the sending and display camera from the first camera to the second camera.

[0036] In combination with the first aspect, in some embodiments, the first camera is a wide-angle camera, and the second camera is an ultra-wide-angle camera; or the first camera is an ultra-wide-angle camera, and the second camera is a wide-angle camera.

[0037] In a second aspect, the present application provides an electronic device, comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the method described in the first aspect and any possible implementation method of the first aspect.

[0038] In a third aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, which are used to call computer instructions so that the electronic device executes the method described in the first aspect and any possible implementation method of the first aspect.

[0039] In a fourth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on an electronic device, causes the electronic device to execute the method described in the first aspect and any possible implementation of the first aspect.

[0040] It is understandable that the electronic device provided in the second aspect, the chip system provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the method provided in the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1The working principle of the rolling shutter is shown by way of example;

[0042] Figure 2 This example shows that the preview image provided by the rolling shutter camera has a water ripple problem;

[0043] Figure 3 An example is shown in which the exposure time of a rolling shutter camera is set to be twice as long as the light energy period to solve the water ripple problem;

[0044] Figure 4 The overall process of a flashing light source detection method provided by an embodiment of the present application is shown;

[0045] Figure 5A-5C A schematic diagram showing changes in exposure time in a group of different shooting environments provided by an embodiment of the present application is exemplified;

[0046] Figure 6 A schematic diagram of a process of detecting a flickering light source in a smooth zooming scenario provided by an embodiment of the present application is exemplarily shown;

[0047] Fig. 7A A schematic diagram of a flow chart of another method for detecting a flickering light source in a smooth zoom scenario provided by an embodiment of the present application is exemplified;

[0048] Figure 7B-Figure 7C A set of zoom range schematic diagrams provided in an embodiment of the present application is shown;

[0049] Figure 8 The hardware structure of the electronic device of the embodiment of the present application is exemplarily shown;

[0050] Fig. 9 The software architecture of the electronic device of the embodiment of the present application is exemplarily shown;

[0051] Fig.10 The detailed implementation process of a flashing light source method provided by an embodiment of the present application based on the internal architecture of an electronic device is shown;

[0052] Fig.11 The detailed implementation process of a flashing light source method in a smooth zoom scenario provided by an embodiment of the present application based on the internal architecture of an electronic device is shown. DETAILED DESCRIPTION

[0053] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0054] The shutter is an important part of the camera, which is used to control the exposure time of the photosensitive film. The shutter can be divided into a global shutter and a rolling shutter, and the exposure methods of the two are different. Among them, the exposure method of the global shutter is: all pixels on the photosensitive film start to be exposed to light at the same time, and after experiencing the same exposure time, they end at the same time; the exposure method of the rolling shutter is: each row of pixels on the photosensitive film is exposed row by row in a rolling manner, and the exposure time of each row is the same until all pixels are exposed. The exposure start time and end time of different rows are different.

[0055] Taking pictures with a rolling shutter camera under certain light sources may cause "banding" to appear in the image. These light sources are often driven by alternating current. The energy transmitted by alternating current is not constant, but changes with a fixed frequency, which causes periodic fluctuations in the light intensity of the light source. At present, the fixed frequency of electrical appliances in some countries is 50Hz, and the fixed frequency of electrical appliances in some countries is 60Hz. The period followed is the light energy period corresponding to the fixed frequency (also called the industrial frequency). For example, the light intensity fluctuation frequency of a 50Hz AC light source is 100Hz, and the light energy period is 10 milliseconds, that is, the light source flickers 100 times in 1 second. In this article, this light source is also called a flickering light source.

[0056] For rolling shutter cameras, when the exposure time is less than the light energy cycle of the light source, since the exposure start and end time of each row of pixels on the photosensitive film are different, the accumulated light energy received by each row of pixels within the same exposure time is different, and light and dark stripes will appear on the image. Figure 1 As shown in FIG. 1 , when imaging the first frame of the image, the light energy received by the first row of pixels on the photosensitive film is the light energy accumulated from time 0 to time t2, while the light energy received by the nth row of pixels on the photosensitive film is the light energy accumulated from time t1 to 5 milliseconds. The light energy received by the nth row is less than the light energy received by the first row. For another example, Figure 1 As shown in the figure, when imaging the N+1th frame, the light energy received by the pixels in the first row on the photosensitive film is the light energy accumulated from time t3 to time t5, while the light energy received by the pixels in the nth row on the photosensitive film is the light energy accumulated from time t4 to time t6. The light energy received by the first row is less than the light energy received by the nth row. In a frame of imaging, the rows with large light energy appear as bright stripes, while the rows with small light energy appear as dark stripes, which can be shown as Figure 2 As shown in the figure, the light and dark stripes are water ripples, which is also called flicker. Moreover, when shooting preview or recording, users can also observe the rolling of water ripples.

[0057] In shooting scenes with fast shutter speeds, such as motion capture, high dynamic range (HDR) and other shooting modes, water ripples are more likely to appear in the image. After detecting that the light source in the shooting environment is a flickering light source, one way to avoid this problem is to set the exposure time to an integer multiple of the light energy cycle. Figure 3 As shown, the light energy cycle is 10 milliseconds, and the exposure time can also be set to 10 milliseconds, that is, 1 times the light energy cycle.

[0058] The detection of flickering light sources is the key to solving the water ripple problem. The existing detection method is to use a high-sampling-rate ambient light sensor, a multi-spectral flicker detector or other dedicated flicker detector to detect whether there is a flickering light source in the shooting environment, and cooperate with the camera to solve the water ripple problem. However, such devices require additional costs and design space.

[0059] The embodiment of the present application provides a detection method, in which, for an electronic device with dual cameras, one of the cameras is used as the main camera, which is only used for preview display, and the other camera is used as the secondary camera, which is only used to detect whether there is a flickering light source with a specific power frequency in the shooting environment, for example, whether there is a flickering light source of 50Hz; and then the water ripples of the above-mentioned specific power frequency in the image captured by the main camera are eliminated according to the detection result of the secondary camera. However, with globalization, electrical products of various countries are circulated with each other, and the environment where the user is located may have a flickering light source of 50Hz or a flickering light source of 60Hz. The above method can only detect flickering light sources of specific power frequencies, and cannot detect at one time whether there is a flickering light source of 50Hz, a flickering light source of 60Hz, or no flickering light source in the shooting environment, and thus cannot make subsequent adjustments according to the specific situation of the flickering light source in the environment.

[0060] An embodiment of the present application also provides a flickering light source detection method, which can quickly detect the flickering light source information in the environment at one time without introducing additional dedicated detection devices, that is, the presence of a 50Hz flickering light source, or a 60Hz flickering light source, or the absence of a flickering light source.

[0061] The electronic device 100 implementing the method has at least two rolling shutter cameras, which are respectively recorded as a first camera and a second camera. The first camera and the second camera respectively detect flickering light sources of different working frequencies in the shooting environment. The first camera or the second camera can also be used for preview display. The first camera and the second camera are not specially introduced for flickering light source detection, and they can be existing cameras on the electronic device, such as an ultra-wide-angle camera or a macro camera.

[0062] Figure 4The overall process of a flickering light source detection method provided by an embodiment of the present application is shown and expanded as follows.

[0063] S101. Set the exposure time of the first camera to N1 times of the first light energy period, collect the first image and send it for display.

[0064] In some embodiments, the operation of setting the exposure time of the first camera may be triggered by an event in which a user opens a camera application and starts the first camera.

[0065] Specifically, when starting the first camera, the exposure time of the first camera can be set to N1 times the first light energy cycle. For example, when N1=1, the exposure time of the first camera is 10 milliseconds. In this way, if the flashing light source frequency contained in the current shooting environment is 50Hz, the user can be prevented from seeing the generation of water ripples at the beginning of the camera startup. It should be noted that the integer multiples N1, M1, N2, and M2 mentioned in this article are all positive integers.

[0066] The value of N1 may be determined by the ISP for the first camera according to an output result of an automatic exposure control (AEC) algorithm.

[0067] Specifically, the photosensitive element (such as an image sensor) in the first camera can collect light information in the environment. The above light information may include light intensity, brightness distribution, etc. The AEC algorithm can generate an appropriate exposure time based on the above light information collected by the first camera, which is called automatic exposure duration. The electronic device 100 can obtain the automatic exposure duration generated by AEC, and then determine the integer multiple of the first light energy cycle closest to the above automatic exposure duration, and then determine the value of N1. Optionally, N1 times the above first light energy cycle should be greater than or equal to the automatic exposure duration. Exemplarily, the automatic exposure duration of the first camera generated by ISP according to AEC is 1 / 60 second (about 16.6 milliseconds), then at this time, N1 can be set to 2, so that 2×1 / 100=1 / 50 second (ie 20 milliseconds) is the minimum integer multiple of the first light energy cycle greater than 1 / 60.

[0068] The scenario of starting the first camera is not limited to the scenario where the user opens the camera application. The scenario of starting the first camera can also be other scenarios that require starting the camera to capture images, such as video call scenarios, scan code payment, identity verification and other scenarios. For example, in the scan code payment scenario, when it is detected that the user turns on the scan code payment, the electronic device can start the rear main camera to scan the payment QR code. At this time, the first camera is the rear main camera. For another example, in the identity verification scenario, the electronic device can start the front main camera and the front depth of field camera to scan and obtain facial feature data for user identity verification. At this time, the first camera can be the front main camera. For another example, in the video call scenario, the electronic device can start the front main camera to capture the portrait of the user in front of the screen. The main camera mentioned in this article is the camera for display.

[0069] After the exposure time is set, the image captured by the first camera is recorded as the first image. The first image can be used for preview display, i.e., sent display, so as to be used as a preview image displayed on the display screen by the camera application. It can be understood that as time passes, the sent display can be continued to display the preview stream of the first camera continuously to continuously provide a preview display to the user.

[0070] S102: Set the exposure time of the second camera to M1 times the second light energy period to capture a second image.

[0071] In some embodiments, when the user opens the camera application and starts the first camera to capture preview images and detect flickering light sources, the second camera is also started to detect flickering light sources; the operation of setting the exposure time of the second camera can also be triggered by the above-mentioned event of starting the second camera.

[0072] In some embodiments, the process of setting the M1 value in the exposure time of the second camera is the same as the process of setting the N1 value in the exposure time of the first camera. In some embodiments, M1=N1. In some embodiments, since the second camera is only used for detection, M1 can be a preset fixed value, such as 1.

[0073] It should be noted that the second light energy cycle corresponding to the exposure time of the second camera should be set to a light energy cycle different from the first light energy cycle corresponding to the exposure time of the first camera, so that as long as there is a flickering light source with an industrial frequency of 50Hz or 60Hz in the shooting environment, it can be detected by the first camera and the second camera. The first light energy cycle and the second light energy cycle are set according to the flickering light sources that may exist in the known environment; in the subsequent embodiments, "the first light energy cycle is the light energy cycle of the flickering light source with a 50Hz first light energy cycle, and the second light energy cycle is the light energy cycle of the flickering light source with a 60Hz second light energy cycle" is used as an example for illustrative description. It can be understood that in another implementation, the first light energy cycle can be the light energy cycle of the flickering light source with a 60Hz first light energy cycle, and the second light energy cycle can be the light energy cycle of the flickering light source with a 50Hz first light energy cycle. Not limited to flickering light sources of 50Hz and 60Hz, flickering light sources with other industrial frequencies may also exist in the environment, which is not specifically limited here.

[0074] After the exposure time is set, the image captured by the second camera is recorded as the second image. The second image is not used for display but only for flicker detection, so the user does not perceive it.

[0075] S103 , performing flicker detection on the first image and the second image respectively.

[0076] The result of the flicker detection may include: detecting a flickering light source and determining flickering light source information, and not detecting a flickering light source.

[0077] It can be understood that performing flicker detection on an image is essentially to identify whether there are water ripple features in the image, so the above flicker detection is also called water ripple detection.

[0078] Taking flicker detection on the first image as an example, optionally, the electronic device 100 may perform flicker detection by performing a method of subtracting two consecutive frame images.

[0079] Specifically, the electronic device 100 can capture a continuous sequence of image frames through a first camera. Then, two consecutive first images are selected from the image sequence captured by the first camera to perform pixel-level difference calculation. The pixel-level difference calculation refers to subtracting the corresponding pixels between the two consecutive first images. The result of the difference calculation is a difference image, which reflects the brightness difference between the two images. If there are alternating light and dark strips in the difference image, it is determined that the first image has water ripple features, which further indicates that there is a flickering light source in the shooting environment of the first camera. The same is true for flicker detection of the second image.

[0080] Not limited to the above method, the electronic device may also implement flicker detection for the first image and the second image through other algorithms such as neural networks. The embodiment of the present application does not impose any particular limitation on the flicker detection algorithm used by the electronic device 100.

[0081] S104: Determine whether water ripples exist in the first image.

[0082] It can be understood that when there is a flickering light source (e.g., a 50HZ flickering light source) in the environment, if the exposure time of the first camera is not an integer multiple of the light energy cycle corresponding to the flickering light source, the image captured by the camera will have water ripples. Therefore, if the first image has water ripple features, the electronic device 100 can determine that there is a flickering light source in the current shooting environment, and the light energy cycle corresponding to the flickering light source is different from the first light energy cycle set by the first camera.

[0083] In S101, the first camera is set to an integer multiple of the first light energy cycle (i.e., the light energy cycle corresponding to 50 Hz). Therefore, if the first image has water ripple features, the electronic device can exclude the fact that the power frequency of the flickering light source in the current shooting environment is 50 Hz. Since the power frequencies of the AC light sources used at home and abroad are 50 Hz or 60 Hz, the electronic device can further determine whether there is a flickering light source with a light energy cycle of the above-mentioned second light energy cycle (a flickering light source with a power frequency of 60 Hz) in the current shooting environment.

[0084] If the first image does not have the above-mentioned water ripple feature, the electronic device 100 can determine that there is no flickering light source in the current shooting environment, or that the flickering light source in the current shooting environment has an operating frequency of 50 Hz.

[0085] S105: Determine whether water ripples exist in the second image.

[0086] Correspondingly, if the second image has water ripple features, the electronic device 100 can determine that there is a 50Hz flickering light source in the current shooting environment. If the second image does not have water ripple features, the electronic device 100 can determine that there is no flickering light source in the current shooting environment, or that the flickering light source in the current shooting environment has a power frequency of 60Hz. The judgment logic is the same as the judgment logic in step S104, and will not be repeated here.

[0087] In combination with the above steps S104 and S105, the electronic device 100 can determine the flickering light source information of the current shooting environment: that is, there is a 50 Hz flickering light source, or there is a 60 Hz flickering light source, or there is no flickering light source.

[0088] S106. When the first image has water ripple features but the second image does not have water ripple features, setting the exposure time of the first camera to an integer multiple of the second light energy period, and setting the exposure time of the second camera to an integer multiple of the first light energy period.

[0089] In some embodiments, when it is detected that the first image has water ripple features, but the second image does not have water ripple features, it can be determined that the flickering light source in the environment matches the second light energy cycle currently corresponding to the second camera, that is, the light energy cycle corresponding to the flickering light source in the environment is the same as the second light energy cycle currently corresponding to the second camera, that is, there is a 60Hz flickering light source. The electronic device 100 sets the exposure time of the first camera to an integer multiple of the second light energy cycle, and sets the exposure time of the second camera to an integer multiple of the first light energy cycle. In this way, the light energy cycle corresponding to the first camera matches the flickering light source in the environment, eliminating the water ripples in the image sent by the first camera.

[0090] In some embodiments, before step S106, the electronic device 100 can obtain the light energy cycle currently corresponding to the first camera (for example, the first light energy cycle) and the light energy cycle currently corresponding to the second camera (for example, the second light energy cycle), and then reset the exposure time of each camera.

[0091] Taking the first camera as an example, in one implementation, the electronic device 100 obtains the current exposure time 1 of the first camera, and then determines the light energy cycle currently corresponding to the first camera according to the exposure time 1. In another implementation, the electronic device 100 stores the light energy cycle currently corresponding to the first camera in field 1, and the electronic device 100 can directly obtain the light energy cycle currently corresponding to the first camera from field 1.

[0092] It can be understood that after obtaining the light energy cycles corresponding to the first camera and the second camera in step S106, the electronic device 100 swaps the corresponding light energy cycles by setting the exposure time of the two cameras, and then swaps the flickering light sources that the two cameras can detect, thereby eliminating the water ripple features in the image sent by the first camera while achieving the effect of continuing to detect the flickering light source information at one time through the two cameras. Therefore, how to set the exposure time of the first camera and the second camera in step S106 is not fixed, but is related to the light energy cycle when the first camera and the second camera perform flicker detection.

[0093] The following describes how to determine the value of the integer multiple corresponding to the light energy period when setting the exposure time of the first camera. In one implementation, the exposure time of the second camera before step S106 is M1 times the second light energy period, and the electronic device 100 sets the exposure time of the first camera to M1 times the second light energy period in step S106. In one implementation, determine the integer multiple of the second light energy period that is closest to the exposure time 1 of the first camera, that is, the value of M1'; update the exposure time of the first camera to M1' times the second light energy period. In one implementation, the AEC algorithm is used to determine the value of the integer multiple of the second light energy period corresponding to the exposure time of the first camera. In the embodiment of the present application, the value of the integer multiple can also be determined in combination with one or more of the above-mentioned implementations, and the embodiment of the present application does not specifically limit this.

[0094] The following describes how to determine the value of the integer multiple of the light energy cycle when setting the exposure time of the second camera. In step S106, the exposure time of the second camera can be set to N1 times the first light energy cycle; or, since the second camera is only used for detection, the value of the integer multiple of the first light energy cycle corresponding to the exposure time of the second camera can be set to a preset fixed value, such as 1; or, the setting method of the first camera can be referred to, and the value of the integer multiple of the first light energy cycle corresponding to the exposure time of the second camera can be set in the same way.

[0095] S107: When the first image does not have water ripple features but the second image does have water ripple features, determine to maintain the light energy cycle corresponding to the current camera.

[0096] Specifically, when the first image does not have water ripple features, and the second image has water ripple features, it can be determined that the flickering light source in the environment matches the light energy cycle currently corresponding to the first camera, that is, there is a 50Hz flickering light source; the exposure time of the first camera can continue to be an integer multiple of the first light energy cycle, and the exposure time of the second camera can continue to be an integer multiple of the second light energy cycle. In one implementation, the exposure time of the first camera is maintained at N1 times the first light energy cycle, and the exposure time of the second camera is maintained at M1 times the second light energy cycle.

[0097] S108: When neither the first image nor the second image has water ripple features, adjust the exposure time of the first camera according to the second exposure strategy.

[0098] The above-mentioned second exposure strategy means that: the adjustment accuracy of the exposure time of the above-mentioned first camera is less than the first light energy cycle and the second light energy cycle, and the minimum value of the exposure time is less than the first light energy cycle and the second light energy cycle, for example, 1 millisecond. The above-mentioned adjustment accuracy is also called the minimum adjustment unit. It can be understood that when the exposure time of the first camera is adjusted according to the above-mentioned second exposure strategy, the exposure time of the first camera can be a non-integer multiple of the first light energy cycle or the second light energy cycle, or it can be an integer multiple of the first light energy cycle or the second light energy cycle.

[0099] When neither the first image nor the second image has water ripple features, it represents that there is no flickering light source in the shooting environment where the electronic device 100 is located. Since the exposure time set for the first camera in S101 and S102 is relatively long, it is easy to cause blurred shooting images for some fast-moving shooting objects. At this time, reducing the exposure time of the camera will not only prevent the appearance of water ripple features, but also obtain a clearer image. Therefore, in some embodiments, under the premise of following the adjustment accuracy and minimum value of the second exposure strategy, the electronic device 100 can adjust the exposure time of the first camera based on motion detection to avoid motion blur in the shot.

[0100] Specifically, the electronic device 100 can obtain two consecutive frames of the first image from the first image captured by the above-mentioned first camera. Then, the electronic device 100 can estimate the movement direction and speed of the pixel by analyzing the pixel displacement between the two consecutive frames of the first image, and then calculate the movement speed of the moving object. Optionally, if the electronic device 100 determines that there is a motion blur feature in the first image through the above-mentioned motion detection, the electronic device 100 can further obtain the movement speed of the moving object according to the motion blur feature, and determine the exposure time of the first camera according to the motion speed reference mapping table. The above-mentioned mapping table shows the corresponding relationship between the object movement speed interval and the exposure time; the greater the movement speed indicated by the movement speed interval, the smaller the exposure time corresponding to the movement speed interval. The mapping table can be established in advance and built into the electronic device. It can be understood that when the object is moving and the movement speed increases, the exposure time of the first camera can be reduced with the adjustment accuracy of the second exposure strategy.

[0101] Exemplarily, the current exposure time of the first camera is 10 milliseconds. If the electronic device 100 determines through motion detection that there is a motion blur feature in the first image, and further concludes that the moving speed of the moving object in the first image is 60 kilometers per hour, the exposure time of the first camera can be reduced to 1 millisecond according to the mapping table.

[0102] If the electronic device 100 determines through the above motion detection that there is no motion blur feature in the first image, the electronic device can maintain the current exposure time.

[0103] Not limited to motion detection, under the premise of following the second exposure strategy, the exposure time can also be adjusted by one or more other methods such as AEC, and the embodiments of the present application do not make specific limitations on this.

[0104] In the embodiment of the present application, S109 may be further executed after steps S106 and S107.

[0105] S109: Adjust the exposure time of the first camera according to the first exposure strategy.

[0106] The above-mentioned first exposure strategy means: adjusting the exposure time of the first camera by adjusting the value of an integer multiple (such as N1 or M1), that is, the adjustment accuracy of the above-mentioned first exposure strategy is equal to the current light energy cycle of the first camera, and the minimum value of the exposure time is equal to the current light energy cycle.

[0107] In some embodiments, before executing the first exposure strategy, the electronic device 100 should first obtain the light energy cycle currently corresponding to the first camera, so as to determine the adjustment accuracy of the subsequent exposure time.

[0108] Optionally, the exposure time of the first camera can be adjusted using the AEC algorithm under the premise of following the adjustment accuracy and minimum value of the first exposure strategy. It is understandable that during the execution of this method by the electronic device 100, the ambient light may have changed to a certain extent, and the electronic device can reset the exposure time of the first camera based on the AEC algorithm, that is, readjust the value of the integer multiple N1 or M1. For details, please refer to the relevant description in step S101, which will not be repeated here.

[0109] Optionally, under the premise of following the first exposure strategy, the exposure time of the first camera may also be adjusted by using motion detection. For details, reference may be made to the relevant description in step S108, which will not be repeated here.

[0110] Not limited to the AEC algorithm or motion detection, under the premise of following the first exposure strategy, the exposure time can also be adjusted in combination with one or more other algorithms, and the embodiments of the present application do not make specific limitations on this.

[0111] In the embodiment of the present application, when the exposure time of the first camera is adjusted according to the first exposure strategy (or the second exposure strategy), the exposure time of the second camera can be maintained without adjustment, or the exposure time of the second camera can be adjusted according to the first exposure strategy (or the second exposure strategy) simultaneously. Preferably, the exposure time of the second camera can be maintained without adjustment, that is, the exposure time of the second camera can be maintained as an integer multiple of the light energy period.

[0112] In the above method, S108 and S109 are optional, and in some embodiments, only S101-S107 may be executed. In addition, in the above method, the execution order of S101-S102 and S104-S105 is not particularly limited. The electronic device 100 may first execute S102 to set the exposure time of the second camera, and then execute S101 to set the exposure time of the first camera. Similarly, S105 may also be executed before S104. However, preferably, the electronic device 100 executes S104 first and then S105. In step S103, flicker detection may be performed on the first image first, and when it is determined that water ripples exist in the first image, flicker light source detection may not be performed on the second image. It can be understood that Figure 4 The flowchart shown only shows a possible schematic flow of the electronic device 100 executing the above method. In the above method, for steps S101 and S102, it is sufficient as long as the light energy cycles corresponding to the first camera and the second camera are different. That is to say, when the exposure time of the first camera is initially set, it can also be set to the exposure time corresponding to the second light energy cycle. At this time, the exposure time of the second camera can be set to the exposure time corresponding to the first light energy cycle. If based on the initial setting of the camera exposure time, the subsequent step S106 should also be adaptively modified to "when the first image has water ripple features and the second image does not have water ripple features, set the exposure time of the first camera to N1 times the first light energy cycle; set the exposure time of the second camera to M1 times the second light energy cycle".

[0113] In some embodiments, the user can carry the electronic device 100 and move around. During the movement of the user, the shooting environment in which the electronic device 100 is located may be constantly changing. For example, when the user carries the electronic device 100 from room A to room B, the power frequency of the flashing light source in the shooting environment may change from 50Hz in room A to 60Hz in room B. Therefore, after starting the first camera to shoot and triggering the electronic device 100 to execute the above method, the electronic device 100 can also execute the above method provided in the embodiment of the present application at a certain frequency, thereby avoiding the reappearance of water ripple features in the image captured by the camera due to changes in the shooting environment. The above frequency is determined by the developer based on experience.

[0114] In some embodiments, if the electronic device 100 can confirm that there is a flickering light source in the shooting environment through the above flicker detection, the electronic device 100 can jump to step S103 after executing step S106, S107 or S109 to start looping the above method. If the electronic device 100 confirms that there is no flickering light source in the shooting environment through the above flicker detection, the electronic device can loop the above method from S101 after executing S108.

[0115] In some embodiments, the electronic device periodically executes Figure 4 In the method flow shown (including steps S101 and S102), the second camera is only used for flicker detection, and there is no need to adjust the exposure time of the second camera after step S102. For example, there is no need to set the exposure time of the second camera in step S106.

[0116] Figure 5A-5C A schematic diagram showing changes in exposure time in a group of different shooting environments provided by an embodiment of the present application is shown.

[0117] For example, if there is a 50Hz flickering light source in the current ambient light source, according to Figure 4 In the method flow shown, at this time, the first image does not have water ripple features, but the second image does have water ripple features.

[0118] When the electronic device 100 executes this method, the initial exposure time of the first camera is set to N1 / 100 seconds, and the initial exposure time of the second camera is set to M1 / 100 seconds. After the execution of this method, both cameras maintain the current light energy cycle, the exposure time of the first camera can also be N1 / 100 seconds, and the exposure time of the second camera can also be M1 / 120.

[0119] If the user moves the electronic device from a 50Hz flickering light environment to a 60Hz flickering light environment, the first image will have water ripple features, but the second image will not have water ripple features. Since the electronic device executes this method at a certain frequency, the exposure time of the first camera and the second camera will change from Figure 5A Switch to Figure 5B .like Figure 5B As shown, the exposure time of the first camera can be switched to M1 / 120 seconds, and the exposure time of the second camera can be switched to N1 / 100 seconds.

[0120] If the user carries the electronic device from a 50Hz flickering light environment to an environment without flickering light, the exposure time of the first camera and the second camera will be changed from Figure 5A Switch to Figure 5C , at this time, neither the first image nor the second image has water ripple features. Figure 5C As shown, the electronic device 100 can reduce the exposure time of the first camera to 1 millisecond by further executing the second exposure strategy (step S108), and the exposure time of the second camera can be maintained at M1 / 120 seconds.

[0121] In some embodiments, the above method can also be applied to zoom scenarios.

[0122] During the zooming process, due to the change in focal length, the electronic device 100 may switch the camera to be displayed. Figure 4 and Figure 5A-5C If there is a flashing light source in the environment, Figure 4 After the process shown is processed, the exposure time of the first camera used for display can match the flickering light source in the environment, and the image displayed does not have water ripple features, and the exposure time of the second camera does not match the flickering light source in the environment. After switching the camera for display to the second camera, the image displayed will have water ripple features again. At this time, the above method needs to further solve the water ripple features that may exist when switching cameras, so that the picture will not abruptly produce water ripples during the zoom process, thereby achieving a smooth zoom effect and improving the user experience.

[0123] It is understandable that the above zoom operation can occur at Figure 4 Any link after S101 and S102 in the process shown. For example, the above zoom operation can occur Figure 4 After S106 and S107 of the method shown, the electronic device may first complete a round of flicker detection and water ripple elimination processing, and then perform a zoom operation, and then trigger subsequent steps S201-S210.

[0124] Figure 6 FIG. 1 is a flow chart showing a method for detecting a flickering light source in a zooming scene provided by an embodiment of the present application. Figure 6 As shown, the method may occur after the zoom ratio of the electronic device 100 is adjusted from the zoom range of the first camera to the zoom range of the second camera.

[0125] S201. When zoom operation 1 is detected, if the current exposure time of the first camera is constrained to be an integer multiple of the light energy period, execute S202A; if the current exposure time of the first camera is not constrained to be an integer multiple of the light energy period, execute S202B; zoom operation 1 is used to adjust the zoom ratio from the zoom range of the first camera to the zoom range of the second camera.

[0126] The zoom operation 1 mentioned above is also referred to as a first zoom operation.

[0127] If the current exposure time of the first camera is constrained to be an integer multiple of the light energy cycle, execute S202A to swap the light energy cycles corresponding to the two cameras. If the current exposure time of the first camera is not constrained to be an integer multiple of the light energy cycle, execute S202B to reset the exposure time corresponding to the two cameras and constrain them to be integer multiples of different light energy cycles.

[0128] It can be understood that if the electronic device 100 executes the second exposure strategy in the aforementioned step S108 before step S201, the exposure time of the first camera is not constrained to an integer multiple of the light energy cycle. If the electronic device 100 executes steps S106, S107 or S109 (i.e., executes the first exposure strategy) before step S201, the exposure time of the first camera can be constrained to an integer multiple of the light energy cycle.

[0129] In some embodiments, it is possible to determine whether the exposure time of the first camera is constrained to be an integer multiple of the light energy period based on the current exposure time of the first camera; or, the electronic device 100 stores field 2, which indicates whether the exposure time of the first camera is constrained to be an integer multiple of the light energy period, and field 2 and the aforementioned field 1 may be the same field or different fields. The embodiment of the present application does not specifically limit the implementation method of "determining whether the exposure time of the camera is constrained to be an integer multiple of the light energy period".

[0130] For ease of description, when the zoom ratio is adjusted to the zoom range of the second camera, before step S202A or S202B, the exposure time of the first camera may also be referred to as the first exposure time, and the exposure time of the second camera may also be referred to as the second exposure time.

[0131] S202A, determine the light energy cycle 1 currently corresponding to the first camera and the light energy cycle 2 currently corresponding to the second camera; set the exposure time of the first camera to an integer multiple of the light energy cycle 2, and collect the third image; set the exposure time of the second camera to an integer multiple of the light energy cycle 1, and collect the fourth image.

[0132] After detecting zoom operation 1, the electronic device 100 can switch the main camera used for display from the first camera to the second camera. In the embodiment of the present application, after switching the camera, the image captured by the first camera is called the third image; the image captured by the second camera is called the fourth image.

[0133] It can be understood that in step S202A, the light energy cycles currently corresponding to the first camera and the second camera are swapped. How to determine the light energy cycles currently corresponding to the first camera and the second camera, and how to determine the integer multiples of the light energy cycles of each camera when swapping the light energy cycles, can be referred to the relevant description in step S106, which will not be repeated here.

[0134] In some embodiments, the exposure time of the second camera is set to the first exposure time in step S202A. In this way, after the main camera is switched to the second camera, the difference between the exposure time of the second camera and the first exposure time of the first camera is reduced, thereby avoiding excessive brightness changes between the fourth image sent by the second camera and the first image sent by the first camera, which results in poor user experience.

[0135] In one usage scenario, there is a flickering light source in the shooting environment. When the electronic device detects that the user performs a zoom operation 1, the electronic device 100 performs Figure 4 The detection method shown in the figure, the current exposure time of the first camera matches the flickering light source in the environment, so that the image sent by the first camera will not produce water ripple characteristics. At this time, after the user switches the camera sent to the second camera through zoom operation 1, the image collected by the second camera will contain water ripple characteristics. Therefore, when executing Figure 6 In the process shown, the light energy cycles corresponding to the first camera and the second camera are swapped first, which can ensure with a higher probability that the first frame fourth image sent by the second camera does not have water ripple features.

[0136] S202B, setting the exposure time of the first camera to an integer multiple of the first light energy period, and collecting the third image; setting the exposure time of the second camera to an integer multiple of the second light energy period, and collecting the fourth image.

[0137] When zoom operation 1 is detected, if the exposure time of the first camera is not constrained to be an integer multiple of the light energy cycle, the exposure time of the first camera can be reset to an integer multiple of the first light energy cycle, and the exposure time of the second camera can be reset to an integer multiple of the second light energy cycle.

[0138] In some embodiments, how to set step S202B specifically can refer to the relevant description of steps S101 and S102. In some embodiments, the value of the integer multiple of the first light energy period closest to the second exposure time of the second camera is determined, such as N2, and then the exposure time of the first camera is set to N2 times the first light energy period. In some embodiments, the value of the integer multiple of the second light energy period closest to the first exposure time of the first camera is determined, such as M2, and then the exposure time of the first camera is set to M2 times the second light energy period.

[0139] It can be understood that in step S202B, the exposure time of the first camera can also be set to an integer multiple of the second light energy period, and the exposure time of the second camera can also be set to an integer multiple of the first light energy period. The embodiment of the present application does not make any special limitation on this.

[0140] After executing S202A or S202B, the electronic device 100 continues to execute steps S203 to S209 in the above method (i.e., detecting and eliminating flickers on the third image and the fourth image captured by the two cameras). The specific implementation of steps S203 to S209 can refer to the relevant description of the aforementioned steps S103 to S109 (i.e., detecting and eliminating flickers on the first image and the second image captured by the two cameras), which will not be repeated later.

[0141] S203: Perform flicker detection on the third image and the fourth image respectively.

[0142] S204: Whether there are water ripples in the fourth image.

[0143] S205: Whether there are water ripples in the third image.

[0144] S206: When the fourth image has water ripple features, setting the exposure time of the first camera and the second camera to swap the light energy cycles currently corresponding to the two cameras.

[0145] If S202A is executed after step S201, the current light energy cycle of the second camera in step S206 is light energy cycle 1, and step S206 includes: "setting the exposure time of the second camera to an integer multiple of light energy cycle 1; setting the exposure time of the first camera to an integer multiple of light energy cycle 2".

[0146] If S202B is executed after step S201, the current light energy cycle of the second camera in step S206 is the second light energy cycle, and step S206 includes: "setting the exposure time of the first camera to an integer multiple of the second light energy cycle, and setting the exposure time of the second camera to an integer multiple of the first light energy cycle."

[0147] S207: When the fourth image does not have water ripple features and the third image does have water ripple features, maintain the current light energy cycle.

[0148] S208: When neither the third image nor the fourth image has water ripple features, adjust the exposure time of the first camera according to the second exposure strategy.

[0149] S209. After resetting the camera exposure time in step S206, or when maintaining the current light energy cycle in step S207, the electronic device 100 may also adjust the exposure time of the first camera according to the first exposure strategy.

[0150] S210, sending and displaying the fourth image.

[0151] It can be understood that in the above method, after the zoom ratio is adjusted from the zoom range of the first camera to the zoom range of the second camera, the image captured by the second camera will be displayed and used as a preview image displayed on the display screen by the camera application. In some embodiments, for the sake of smooth zoom, the fourth image captured by the second camera is not displayed until step S210. That is, after the above zoom operation 1 of the user is detected, the first image captured by the first camera is still displayed continuously until the electronic device 100 performs the above steps S201-S209 (that is, the first water ripple detection and water ripple elimination after the zoom operation 1 is completed) before the fourth image captured by the second camera is displayed, thereby ensuring that the image displayed by the electronic device 100 has no water ripple features before and after switching the camera to display during the zoom process, thereby achieving smooth zoom. In some embodiments, after the zoom operation 1 of the user is detected, it is immediately switched to display the fourth image captured by the second camera; the execution speed of the electronic device 100 is fast enough to complete the water ripple detection and elimination before displaying the fourth image captured by the second camera, thereby achieving smooth zoom.

[0152] Similarly, in the above method, S208 and S209 are optional. In some embodiments, step S210 can be directly executed after executing step S205, S207 or S206. In addition, the execution order of S205-S206 is not particularly limited. Preferably, the electronic device 100 executes S205 first and then executes S206. In step S203, flicker detection can be performed on the fourth image first. When it is determined that water ripples exist in the fourth image, flicker light source detection can be omitted for the third image. Figure 6 The flowchart shown only illustrates a possible process of the electronic device 100 executing the above method.

[0153] In some usage scenarios, after the zoom operation 1 is detected, the main camera used for display is switched from the first camera to the second camera. Due to insufficient computing speed of the electronic device 100 or hardware and software jams, the display may not be completed within the gap between adjacent frames (i.e., from the time when the first camera displays the last frame of the first image to the time when the second camera displays the first frame of the fourth image). Figure 6 In view of this, the embodiment of the present application also proposes another flicker detection method in a smooth zoom scenario, which can enable the electronic device 100 to still complete flicker detection and water ripple elimination in the above situation.

[0154] Fig. 7A A schematic flow chart of another method for detecting a flickering light source in a zoom scenario provided in an embodiment of the present application is shown.

[0155] S301: When a zoom ratio change is detected, adjust the exposure time of the first camera.

[0156] In some embodiments, when the electronic device detects the above-mentioned change in zoom ratio, the electronic device 100 may adjust the exposure time of the first camera according to the zoom ratio.

[0157] In one implementation, if the current exposure time of the first camera is constrained to be an integer multiple of the light energy cycle, the electronic device can use the first exposure adjustment strategy and the AEC algorithm to adjust the exposure time of the first camera, that is, adjust the value of the integer multiple of the light energy cycle. If the current exposure time of the first camera is not constrained to be an integer multiple of the light energy cycle, the electronic device can use the second exposure adjustment strategy and the AEC algorithm to adjust the exposure time of the first camera.

[0158] S302: When a change in zoom ratio is detected, adjust the exposure time of the second camera.

[0159] In some embodiments, the adjustment scheme of the exposure time of the second camera is the same as S301 and will not be described in detail.

[0160] Steps S301 and / or S302 are optional. In some embodiments, before the zoom ratio is adjusted to the transition zoom range, the exposure time of the first camera and the second camera does not need to be adjusted. The above transition zoom range is also referred to as the first zoom range. In some embodiments, before the zoom ratio is adjusted to the transition zoom range, when S301 is executed to adjust the exposure time of the first camera, the exposure time of the second camera does not need to be adjusted.

[0161] S303, detecting whether the focal length reaches the transition zoom range of the first camera; if so, executing S304.

[0162] Exemplary, reference Figure 7B and Figure 7C As shown, the zoom range of the first camera includes a transition zoom range, the transition zoom range is adjacent to the zoom range of the second camera, the transition zoom range is a zoom range between a first focal length threshold and a second focal length threshold, the second focal length threshold is a minimum value or a maximum value within the zoom range of the second camera, and the difference between the first focal length threshold and the second focal length threshold is a smaller preset value. The preset value is determined by the developer based on experience.

[0163] refer to Figure 7B , the focal length within the zoom range of the second camera is greater than the focal length within the zoom range of the first camera, the current zoom ratio belongs to the zoom range of the first camera, and the first camera is used for display; at this time, the user can increase the zoom ratio so that the focal length reaches the transition zoom range of the first camera; in step S303, it can be continuously detected whether the focal length increases to the first focal length threshold. Reference Figure 7C , the focal length within the zoom range of the second camera is smaller than the focal length within the zoom range of the first camera, the current zoom ratio belongs to the zoom range of the first camera, and the first camera is used for display; at this time, the user can reduce the zoom ratio so that the focal length reaches the transition zoom range of the first camera; in step S303, it can be continuously detected whether the focal length is reduced to the first focal length threshold.

[0164] The electronic device 100 continuously detects whether the zoom ratio reaches the transition zoom range of the first camera; when the zoom ratio reaches the transition zoom range, the zoom ratio is close to the zoom range of the second camera. At this time, the electronic device 100 performs flicker detection in advance and executes steps S304 to S307, so that the light energy cycle corresponding to the second camera matches the flickering light source in the environment in advance.

[0165] S304 , constraining the exposure time of the second camera to be an integer multiple of the light energy period 3, and performing flicker detection on the fifth image captured by the first camera and the sixth image captured by the second camera.

[0166] In some embodiments, when the electronic device 100 detects that the zoom ratio is adjusted from the non-transition zoom range of the first camera to the transition zoom range, if the exposure time of the first camera is constrained to an integer multiple of the light energy cycle, the electronic device 100 can obtain the light energy cycle 3 currently corresponding to the first camera, and set the exposure time of the second camera to an integer multiple of the light energy cycle 3; if the exposure time of the first camera is not constrained to an integer multiple of the light energy cycle, the electronic device 100 can directly set the exposure time of the second camera to an integer multiple of the light energy cycle 3, and the light energy cycle 3 can be any one of the first light energy cycle and the second light energy cycle. Among them, how to determine the light energy cycle currently corresponding to the first camera, and how to determine the value of the integer multiple when setting the exposure time of the second camera to an integer multiple of the light energy cycle 3, can refer to the relevant descriptions in step S106 and step S101, which will not be repeated here. The above-mentioned light energy cycle 3 is also called the third light energy cycle.

[0167] For the convenience of description, when the zoom ratio is within the transition zoom range, the image captured by the first camera is called the fifth image, and the image captured by the second camera is called the sixth image.

[0168] S305, determining whether the sixth image captured by the second camera has water ripples; if so, executing S306; otherwise, executing S311.

[0169] In the embodiment of the present application, when water ripples exist in the sixth image, the electronic device 100 may execute steps S306 and S307.

[0170] S306. When water ripples exist in the sixth image, the electronic device 100 sets the exposure time of the second camera to an integer multiple of the light energy cycle 4; constrains the exposure time of the first camera to an integer multiple of the light energy cycle 4; and the light energy cycle 4 is different from the light energy cycle 3.

[0171] The above-mentioned light energy cycle 4 is also called the fourth light energy cycle.

[0172] Among them, when light energy cycle 3 is the first light energy cycle, light energy cycle 4 is the second light energy cycle; or, when light energy cycle 4 is the second light energy cycle, light energy cycle 3 is the first light energy cycle.

[0173] When there are water ripples in the sixth image, there is a flickering light source in the environment, and it does not match the light energy cycle 3 currently corresponding to the second camera, that is, it matches the light energy cycle 4; the exposure time of the second camera is set to an integer multiple of the light energy cycle 4, so that when the zoom ratio is within the transition zoom range, the light energy cycle corresponding to the second camera matches the flickering light source in the environment in advance.

[0174] It should be noted that if it is determined in step S304 that the exposure time of the first camera is currently constrained to be an integer multiple of the light energy cycle 4, then step S306 continues to maintain the exposure time of the first camera as an integer multiple of the light energy cycle 4; if it is determined in step S304 that the exposure time of the first camera is currently not constrained to be an integer multiple of the light energy cycle, then step S306 sets the exposure time of the first camera to an integer multiple of the light energy cycle 4, so that the exposure time of the first camera currently sent for display matches the flickering light source in the environment, thereby eliminating water ripples in the image currently sent for display by the first camera.

[0175] S307, determining whether the zoom ratio is adjusted to the zoom range of the second camera; if so, executing S308 to S310.

[0176] S308: Switch the main camera for display to the second camera, and display the fourth image captured by the second camera.

[0177] S309, setting the exposure time of the first camera to an integer multiple of the light energy period 3.

[0178] With reference to the related description of step S202A, for the convenience of description, after the camera is switched, the image captured by the first camera is referred to as the third image; and the image captured by the second camera is referred to as the fourth image.

[0179] It can be seen from step S306 that there is a flickering light source in the current environment, and the light energy cycle corresponding to the flickering light source is light energy cycle 4. When the zoom ratio is adjusted to the zoom range of the second camera, the main camera for display is switched from the first camera to the second camera; at this time, the second camera is used for display and flicker detection, while the first camera is only used for flicker detection. In some embodiments, the exposure time of the second camera has been constrained in advance in step S306 to be an integer multiple of light energy cycle 4, in order to continue to regularly complete flicker detection (i.e., perform flicker detection) in the future. Figure 4 The flickering light source detection method shown in FIG. 1 ) requires constraining the exposure time of the first camera to be an integer multiple of another light energy cycle (ie, light energy cycle 3).

[0180] In an embodiment of the present application, before switching the main camera for display to the second camera, when the zoom ratio is within the transition zoom range, the exposure time of the second camera is matched in advance with the flickering light source in the environment, so that the first image displayed by the second camera after switching the camera will not have water ripple features, thereby ensuring a smooth zoom user experience.

[0181] Step S309 is optional. In some embodiments, it is not necessary to execute S309, and the execution is performed periodically. Figure 4 In the flickering light source detection method shown in FIG. 1 , the exposure time of the first camera is reset in step S101 .

[0182] The embodiment of the present application does not specifically limit the execution order of steps S308 and S309.

[0183] S310: Adjust the exposure time of the second camera currently sending the display according to the first exposure strategy.

[0184] For details, please refer to the relevant description of step S109, which will not be repeated here.

[0185] S311. When there are no water ripples in the sixth image, the electronic device 100 determines whether there are water ripples in the fifth image captured by the first camera; if so, execute S312; otherwise, execute S317.

[0186] S312, setting the exposure time of the first camera to an integer multiple of the light energy period 3.

[0187] Referring to the relevant contents of step S304, it can be known that when adjusting the zoom ratio to the transition zoom range of the first camera, the first camera may be constrained to an integer multiple of the light energy cycle 3, or may not be constrained to an integer multiple of the light energy cycle. In step S311, the exposure time of the second camera is an integer multiple of the light energy cycle 3. If the sixth image captured by the second camera does not include water ripples, and the fifth image captured by the first camera includes water ripples, it means that there is a flickering light source in the environment, and the light energy cycle corresponding to the flickering light source is the light energy cycle 3; in step S312, the exposure time of the first camera is set to an integer multiple of the light energy cycle 3, so that the exposure time of the first camera currently sent for display matches the flickering light source in the environment, thereby eliminating the water ripples in the sent image.

[0188] S313, determining whether the zoom ratio is adjusted to the zoom range of the second camera; if so, executing S314 to S316.

[0189] S314: Switch the main camera for display to the second camera, and display the fourth image captured by the second camera.

[0190] S315, setting the exposure time of the first camera to an integer multiple of the light energy period 4.

[0191] It can be seen from step S312 that there is a flickering light source in the current environment, and the light energy cycle corresponding to the flickering light source is light energy cycle 3. When the zoom ratio is adjusted to the zoom range of the second camera, the main camera for display is switched from the first camera to the second camera; at this time, the second camera is used for display and flicker detection, while the first camera is only used for flicker detection; the exposure time of the second camera has been constrained in advance to an integer multiple of light energy cycle 3 in step S304, in order to continue to regularly complete flicker detection and water ripple elimination (i.e., perform Figure 4 The flickering light source detection method shown in FIG. 4 ) requires constraining the exposure time of the first camera to be an integer multiple of another light energy cycle (ie, light energy cycle 4).

[0192] Step S315 is optional. In some embodiments, it is not necessary to execute S315, and the execution is performed periodically. Figure 4 In the flickering light source detection method shown in FIG. 1 , the exposure time of the first camera is reset in step S101 .

[0193] The embodiment of the present application does not specifically limit the execution order of steps S314 and S316.

[0194] S316: Adjust the exposure time of the second camera currently sending the display according to the first exposure strategy.

[0195] For details, please refer to the relevant description of step S109, which will not be repeated here.

[0196] S317: When there are no water ripples in the fifth image, the electronic device 100 determines whether the zoom ratio is adjusted to the zoom range of the second camera; if so, execute S318 to S319.

[0197] In step S317, the exposure time of the second camera is an integer multiple of the light energy period 3, and the first camera is constrained to be an integer multiple of the light energy period 3, or is not constrained to be an integer multiple of the light energy period; at this time, if the sixth image captured by the second camera does not include water ripples, the fifth image captured by the first camera also does not include water ripples, indicating that there is no flickering light source in the environment.

[0198] S318: Switch the main camera for display to the second camera, and display the fourth image captured by the second camera.

[0199] Furthermore, the electronic device may also fine-tune the exposure time of the second camera, and the method further includes:

[0200] S319: The electronic device 100 may adjust the exposure time of the second camera according to the second exposure strategy.

[0201] For details, please refer to the relevant description of step S108, which will not be repeated here.

[0202] In some embodiments, in the above method, S310, S316, and S319 are optional, and the electronic device 100 not executing the above steps does not affect the implementation of the smooth zoom solution. Fig. 7A The flowchart shown only illustrates a possible process of the electronic device 100 executing the above method.

[0203] In the above method, the first camera and the second camera are not fixed.

[0204] When the user opens the camera application, the electronic device can default to the normal shooting mode with a zoom ratio of 1. At this time, the first camera can be a wide-angle camera for display and flicker detection; the second camera can be selected from the remaining cameras and used only for flicker detection. However, when the shooting mode changes or the zoom ratio changes, the main camera used for display can switch from the wide-angle camera to other cameras.

[0205] For example, during photo preview, video preview or video recording, if it is detected that the user increases the zoom ratio to the zoom range of the telephoto camera, the electronic device can switch the main camera from the wide-angle camera to the telephoto camera, and the telephoto camera is used for display and flicker detection. For another example, during photo preview, video preview or video recording, if it is detected that the user reduces the zoom ratio to the zoom range of the ultra-wide-angle camera, the electronic device can switch the main camera from the wide camera to the ultra-wide-angle camera, and the ultra-wide-angle camera is used for display and flicker detection. For another example, if it is detected that the photo mode is switched from normal mode to macro shooting mode, the electronic device can switch the main camera from the main camera to the macro camera, and the macro camera is used for display and flicker detection. For another example, the electronic device detects the user operation of flipping the camera, switches the first camera from the rear main camera to the front main camera, or switches the main camera from the front main camera to the rear main camera, and the rear main camera is used for display and flicker detection.

[0206] The original first camera can be switched to an auxiliary camera used only for flicker detection, especially when the electronic device has only two cameras, so there is no need to start a new camera, and the response speed is faster. Of course, in actual applications, other cameras with lower power consumption can also be considered to be switched to auxiliary cameras based on power consumption and other issues, and the embodiments of the present application do not limit this.

[0207] Preferably, the first camera is a wide-angle camera, and the second camera is an ultra-wide-angle camera.

[0208] Next, the software and hardware architecture of the electronic device provided in the embodiment of the present application is introduced.

[0209] Electronic equipment can be equipped Or portable terminal devices with other operating systems, such as mobile phones, tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPC), netbooks, as well as cellular phones, personal digital assistants (PDA), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, vehicle-mounted devices, smart home devices and / or smart city devices, etc.

[0210] Figure 8The electronic device 100 provided by the embodiment of the present application is exemplarily shown. The electronic device 100 can detect whether there is a flickering light source in the shooting environment, and does not require an additional dedicated flicker detection device, and can also prevent the user from seeing the generation of water ripples.

[0211] like Figure 8 As shown, the electronic device 100 may include: a processor 110 , a memory 120 , a camera 130 , and a display screen 140 .

[0212] in:

[0213] The processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices or integrated into one or more processors. The processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc. These interfaces are used for data exchange between the processor 110 and peripherals.

[0214] The memory 120 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data, etc. The non-volatile memory can also store executable programs and store user and application data, etc., and can be loaded into the random access memory in advance for direct reading and writing by the processor 110. A storage unit can also be set in the processor 110, and the storage unit can be a high-speed cache storage unit, which can be used to save instructions or data that the processor 110 has just used or circulated. The implementation code of the flickering light source detection method provided in the embodiment of the present application can be stored in the NVM. When the camera application is started, the code can be loaded into the RAM. In this way, the processor 110 can directly read the program code from the RAM to implement the flickering light source detection method provided in the embodiment of the present application. In addition, image files such as photos and videos obtained by the user using the camera application can be written into the NVM for storage for the user to browse.

[0215] The camera 130 may include: a lens, a photosensitive sensor and a flexible printed circuit board (FPCB). The FPCB is responsible for connecting other components of the camera 130 to the processor 110, for example, transmitting the raw data output by the photosensitive sensor to the processor 110. When taking a photo, the shutter of the camera 130 is opened, and light enters and shines on the photosensitive sensor. The photosensitive sensor converts the light signal into an electrical signal, and then further converts the electrical signal into a digital signal through analog-digital conversion (ADC) to pass it to the ISP for processing. The ISP can perform the following processing on the output data of the photosensitive sensor: automatic exposure control (AEC), automatic gain control (AGC), automatic white balance (AWB), color correction, removal of bad pixels, etc. The ISP can also be integrated into the camera 130.

[0216] The display screen 140 can be used to display the image captured by the camera 130. The image processed by the ISP will be sent to the display screen 140 to show the user a preview of the image captured by the camera. Here, sending to the display screen means pushing the image captured by the camera to the frame buffer (FB) for storage. The frame buffer is a storage space that can be located in the video memory or the internal memory, and is used to store the rendering data processed by the graphics card chip or to be extracted. The content of the frame buffer corresponds to the interface display on the display screen 140, and it can be simply understood as the cache corresponding to the content displayed on the display screen 140. Modifying the content in the frame buffer is to modify the content on the display screen 140.

[0217] The electronic device 100 has multiple cameras 130, for example, a rear main camera, a front main camera, a telephoto camera, a wide-angle camera, a depth of field camera, a macro camera, etc. In terms of use, these multiple cameras 130 may include at least two cameras 130 (for example, a first camera and a second camera) that can be used for preview display and flicker detection. Among them, when the flicker light source detection method provided in the embodiment of the present application starts to be executed, the image captured by the first camera will be displayed, so as to be used as a preview image displayed by the camera application on the display screen 140; the image captured by the second camera will not be displayed, but only used for flicker detection, so the user does not perceive it. During the zooming process, when the electronic device 100 detects that the zoom ratio reaches the zoom range of the second camera, the image captured by the second camera will be displayed, so as to be used as a preview image displayed by the camera application on the display screen 140; the image captured by the second camera will not be displayed, but only used for flicker detection. During the execution of the above flicker light source detection method, the image captured by the camera that is displayed is also used for flicker detection. The shutters of the first camera and the second camera are both rolling shutters.

[0218] When the electronic device needs to perform flicker detection, for example, when the user opens the camera application to trigger the first camera to turn on, or when the user performs a zoom operation, the first camera can be set to an integer multiple of a certain light energy cycle, and the second camera can be set to an integer multiple of another light energy cycle, so that as long as there is a flickering light source in the shooting environment, the image collected by the first camera or the image collected by the second camera must contain water ripple features, and the water ripple features can be detected by image recognition technology. Based on the flicker detection result, the electronic device 100 can effectively avoid the problem of water ripples in the imaging picture of the first camera by setting the exposure time of the first camera for preview display. When avoiding the problem of water ripples in the imaging picture of the first camera, the electronic device 100 can adaptively adjust the exposure time of the second camera so that the light energy cycles corresponding to the first camera and the second camera are different, so that after adjusting the exposure time, the first camera and the second camera can detect at one time whether there is a 50Hz flickering light source, a 60Hz flickering light source, or no flickering light source in the shooting environment.

[0219] The first camera and the second camera are not specially introduced for flicker detection. Usually, they can be ultra-wide-angle cameras, wide-angle cameras or macro cameras, etc. The embodiments of the present application give them new functions.

[0220] Taking the first camera for preview display as an example, in addition to the flicker detection results provided by the first camera and the second camera, the exposure time of the main camera can also be controlled by the automatic exposure control (AEC) function. In short, AEC automatically adjusts the exposure time according to the light intensity. In different environments, the intensity of light varies greatly, and the exposure time of the photosensor in the camera 130 also needs to be adapted accordingly to ensure normal exposure of the image. The automatic exposure control (AEC) provided by the ISP provides this adaptive capability. Therefore, the exposure time of the photosensor can reflect the brightness of the shooting environment to a certain extent.

[0221] In addition, the electronic device 100 may also store an exposure table to be used by the AEC. The exposure table of a camera sets the exposure parameters (such as exposure time, exposure gain, aperture size) to be used by the camera at different exposure values ​​(EV), which may be debugged and recorded by engineers.

[0222] In addition, the exposure time of the main camera can also be controlled by the motion detection algorithm. Taking the main camera as the first camera as an example, the motion detection algorithm can identify whether there is a motion blur feature in the first image captured by the first camera. If so, the movement speed of the moving object can be further determined based on the above-mentioned motion blur feature, and then the exposure time of the first camera can be adjusted. For objects with different movement speeds, the corresponding clear imaging time is also different, and the exposure time of the photosensitive sensor in the camera 130 also needs to be adapted accordingly to ensure clear image imaging. The motion detection algorithm can be provided by the processor 110. Therefore, if the final image is clear, the exposure time of the photosensitive sensor can reflect the movement speed of the object in the image to a certain extent.

[0223] The electronic device 100 may also store the corresponding relationship between the moving object speed interval and the camera exposure time required for the above-mentioned motion detection algorithm. The corresponding relationship may be obtained by simulation or actual measurement by engineers.

[0224] like Figure 8 As shown, the electronic device 100 may further include: an audio module 150, a speaker 150A, a receiver 150B, a microphone 150C, and an earphone interface 150D.

[0225] Among them, the electronic device can realize audio functions through the audio module 150, the speaker 150A, the receiver 150B, the microphone 150C, the headphone interface 150D, and the application processor (AP). For example, music playing, recording, etc. The audio module 150 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 150 can also be used to encode and decode audio signals. In some embodiments, the audio module 150 can be arranged in the processor 110, or some functional modules of the audio module 150 are arranged in the processor 110. The speaker 150A, also known as the "speaker", is used to convert audio electrical signals into sound signals. The electronic device can listen to music or listen to hands-free calls through the speaker 150A. The receiver 150B, also known as the "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device answers a call or voice message, the voice can be answered by placing the receiver 150B close to the human ear. Microphone 150C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by putting his mouth close to the microphone 150C to input the sound signal into the microphone 150C. The electronic device can be provided with at least one microphone 150C. In other embodiments, the electronic device can be provided with two microphones 150C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device can also be provided with three, four or more microphones 150C to realize the collection of sound signals, noise reduction, identification of sound sources, realization of directional recording functions, etc. The headphone jack 150D is used to connect wired headphones. The headphone jack 150D can be a USB interface, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0226] like Figure 8 As shown, the electronic device 100 may further include a sensor module 160, which may specifically include a pressure sensor 160A, a distance sensor 160F, a proximity light sensor 160G, a touch sensor 160K, an ambient light sensor 160L, and the like.

[0227] Figure 8 The illustrated structure does not constitute a specific limitation on the electronic device, and the electronic device may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0228] Fig. 9 The main software architecture involved in the flickering light source detection method provided by the embodiment of the present application on the electronic device 100 is shown.

[0229] like Fig. 9 As shown in the figure, the software architecture may include the following functions or modules: camera application (camera apk), camera service (camera service), camera hardware abstraction layer (camera HAL), flicker detection (CameraFlickerDetector), camera vendor native development kit (camera VNDK) and Kernel part. Among them,

[0230] The camera application (camera apk) can be located in the application layer. In addition to the camera application, the application layer can include a series of other applications, such as the gallery application. The camera application can be mainly responsible for human-computer interaction, such as monitoring and responding to user operations such as selecting a shooting mode, adjusting the zoom ratio, and selecting a focus.

[0231] The camera service can be located in the application framework layer. The camera service can include interfaces such as camera management and camera devices, which can interact with the camera application through the application programming interface (API) and with the camera HAL through the HAL interface definition language (HIDL). In addition, the camera service can also interact with the vendor native development kit (VNDK).

[0232] The camera hardware abstraction layer (camera HAL) can be the part of the virtual hardware function about the camera in the hardware abstraction layer (HAL). The hardware abstraction layer is located in the interface layer between the application framework layer and the driver layer, providing a virtual hardware platform for the operating system, and the upper layer does not need to care about the underlying hardware implementation. The camera HAL can communicate with the camera service through the HIDL interface, and can issue control operations for the camera through the standard hardware abstraction layer interface. The camera HAL can include a camera startup (CamEntry) module and an automatic exposure control processor (AECProcessor) module, wherein the CamEntry module can be used as the entrance of the HAL, receiving instructions such as open, close, stream configuration (configurestream), and request from the camera service, and is a general camera HAL function entrance. The AECProcessor module can provide ISP's AEC-related functions, such as providing the AEC parameters (such as exposure time) of the first camera and the second camera to the CameraFlickerDetector, so that the CameraFlickerDetector can determine whether the current shooting environment requires flicker detection.

[0233] The flicker detection (CameraFlickerDetector) module can be located in the application framework layer (framework) or the application layer. The CameraFlickerDetector module can obtain the AEC parameters (such as exposure time) of the first camera and the second camera from the camera HAL through the HIDL interface, so as to decide whether to start the first camera and the second camera for flicker detection. When it is determined that flicker detection is to be performed, the CameraFlickerDetector module can issue the control operation of starting the second camera and setting the exposure time of the first camera and the second camera through the interface of the cameraVNDK. The CameraFlickerDetector module may include a CameraHelper module and an ImageReader module, wherein the CameraHelper module can be used to implement the call of functions such as opening (open), closing (close), stream configuration (configurestream), and request (request) of the camera service of the second camera through the call of the VNDK interface, and the ImageReader module is mainly used to receive and further process the images sent by the first camera and the second camera when executing the flicker detection method.

[0234] The Vendor Native Development Kit (VNDK) is a collection of libraries for vendors to develop their own HALs (vendors). The camera VNDK is the "own camera HAL" developed by the vendor of the second camera. The camera VNDK can be used to receive the control operation of starting the second camera issued by the CameraFlickerDetector, and forward the control operation to the cameraservice. The camera VNDK may include the ACameraManager module, the ACameraDevice module, the ACameraReader module, the ACaptureSession module, and the ACaptureRequest module. Among them, the ACameraManager module provides the function of accessing the camera service from the vendor layer, the ACameraDevice module provides the function of accessing the camera device from the vendor layer, the ACameraReader module is responsible for reading image data, such as the data captured by the first camera or the second camera, the ACaptureSession module provides the function of allowing the vendor to manage the frames captured by the camera, and the ACaptureRequest module is responsible for the settings and output targets required for the camera to capture a single image.

[0235] The kernel part may include various hardware driver control modules, such as ISP, photosensor, DSP, AP and other driver controls.

[0236] Fig. 9 Only the main software architecture involved in the flickering light source detection method provided in the embodiment of the present application is shown, which is essentially only a part of the software architecture of the electronic device 100.

[0237] Fig. 9 The implementation process of the flashing light source detection method is also shown, including:

[0238] 1. First camera startup process

[0239] The camera application calls the camera service, which authenticates the camera application. Authentication is used to confirm whether the camera application has the authority to access the hardware camera, such as turning the camera on and off, to prevent illegal applications from turning the camera on and off without user authorization, thereby effectively preventing user privacy from being violated. It can also prevent the camera application from conflicting with other applications that are accessing the camera (such as video call applications) in the use of camera resources. After authentication, the camera service calls the camera HAL to start the first camera through the camera HAL, and control the photosensor and ISP to output images. The image frame will be sent to the display to implement image preview, photo taking and other functions.

[0240] On this basis, in order to perform flicker detection without letting the user know, the electronic device 100 can start the second camera after starting the first camera to perform detection together. After the second camera is started, the image frame is not sent for display. The first camera and the second camera are not specially introduced for flicker detection, where the first camera can be a wide-angle camera and the second camera can be an ultra-wide-angle camera. The embodiment of the present application gives it a new function.

[0241] 2. Flicker detection process

[0242] First, when the preview of the first camera is turned on, the AECProcessor module in the camera HAL can pass the exposure time and other parameters of the first camera to the CameraFlickerDetector module through the HIDL interface. The exposure time set by the AEC can reflect the brightness of the shooting environment. Therefore, the CameraFlickerDetector module can determine whether to perform flicker detection based on the exposure time. The environment for flicker detection is usually an indoor scene. Flicker detection is generally not required in a shooting environment that is too bright or too dark, because in a too bright environment, the main consideration is the overexposure problem and the exposure time needs to be significantly reduced, which makes it difficult to take into account the problem of avoiding water ripples; in a too dark environment, there is generally no need to reduce the exposure time to less than the light energy cycle, otherwise it is easy to cause underexposure. Here, too bright or too dark can be defined with reference to the exposure table on the electronic device 100. For example, the shortest exposure times corresponding to the high EV in the exposure table are determined as the exposure time of the too bright environment. Once the actual exposure time belongs to these shortest exposure times, it is considered that the environment is too bright. For another example, the longest exposure times corresponding to the low EV in the exposure table are determined as the exposure time of the too dark environment. Once the actual exposure time falls within these longest exposure times, it is considered that the environment is too dark.

[0243] Then, after determining that flicker detection is to be performed, the CameraFlickerDetector module transmits a control command to start the second camera to cameraVNDK.

[0244] The camera VNDK then passes the CameraFlickerDetector module to the camera service to start the control operation of the second camera.

[0245] After authenticating the CameraFlickerDetector module, the camera service calls the camera HAL again to start the second camera through the camera HAL, so that the second camera can be started from the camera HAL side without involving the application layer, so the user does not perceive it. At the same time, the CameraFlickerDetector module controls the exposure of the first camera and the second camera, so that the exposure time of the first camera is an integer multiple of the first light energy cycle, and the exposure time of the second camera is an integer multiple of the second light energy cycle. For example, the exposure time of the first camera is 20 milliseconds, and the exposure time of the second camera is 8.3 milliseconds (i.e., 1 / 120 seconds), so that the image collected by the first camera under a 60Hz flickering light source will inevitably produce water ripples, and the image collected by the second camera under a 50Hz flickering light source will inevitably produce water ripples, and thus as long as there is a flickering light source in the shooting environment, it can be detected. The CameraFlickerDetector module can also call the camera HAL to read the output of the first camera and the second camera, detect whether the image content has water ripple features through image recognition technology, and pass the detection results to the first camera and the second camera. The output of the second camera is not sent for display, but is only used for flicker detection.

[0246] After the first camera and the second camera learn the flicker detection result, they can perform corresponding exposure operations. If the CameraFlickerDetector module detects that there are water ripple features in the image content of the first image, the exposure time of the first camera can be set to an integer multiple of the second light energy cycle, and the exposure time of the second camera can be set to an integer multiple of the first light energy cycle, thereby solving the problem of water ripples in the image under the flickering light source.

[0247] Furthermore, or after the CameraFlickerDetector module detects the presence of water ripple features in the image content of the first image or the second image, the first camera can also adjust the exposure time of the first camera based on motion detection, for example, reducing the exposure time of the first camera to 10ms, while following the adjustment accuracy and minimum value of the first exposure strategy, so as to avoid motion blur in the captured image. If the CameraFlickerDetector module detects that there are no water ripple features in the image content of both the first image and the second image, the exposure time of the first camera can be further reduced to less than the current corresponding light energy cycle, for example, 1ms, so as to solve the problem of water ripples in the image under the flickering light source.

[0248] The above software architecture newly introduces the CameraFlickerDetector module, and implements the interaction between the CameraFlickerDetector module and the camera HAL and camera service through VNDK and HIDL. It does not involve modifications to the camera application and various layers of the architecture, and has good portability, scalability, and maintainability.

[0249] Combine the following Fig.10 The internal structure of the electronic device 100 is further described to explain the flickering light source detection method of the embodiment of the present application. Fig.10 The method is described by taking the photo or video preview provided by the camera application as an example, so that those skilled in the art can have a deeper understanding of the embodiments of the present application, and its details should not limit the scope of protection of the present application. The following is an example of an electronic device 100 having a wide-angle camera and an ultra-wide-angle camera.

[0250] Normal photo mode

[0251] S11. When a user operation of opening a camera application is detected, an application processor (AP) may start a wide-angle camera.

[0252] For example, taking a wide-angle camera as the first camera as an example, the photographing mode may be a normal photographing mode, and the exposure time of the wide-angle camera may be set by the AEC of the ISP.

[0253] Prior to S11, the display screen may display a system desktop, including a desktop icon of a camera application. The user operation of opening the camera application may refer to the user clicking the desktop icon of the camera application.

[0254] S12. The wide-angle camera transmits the acquired raw image frame (RAW) to the ISP.

[0255] The ISP can perform the following processing on the RAW image output by the wide-angle camera: automatic exposure control (AEC), automatic gain control (AGC), automatic white balance (AWB), color correction, removal of bad pixels, etc. The image processed by the ISP can be a color image, such as a YUV image.

[0256] S13, ISP can transmit the YUV image frame output by the wide-angle camera to the display screen.

[0257] Here, S13 only illustrates the flow path of image data, and does not mean that the ISP directly sends data to the display screen. Its implementation includes the process of sending and displaying. The YUV image frame of the wide-angle camera can be cached in the frame buffer first. When the GPU renders the preview image of the camera, it reads the YUV image frame of the wide-angle camera from the frame buffer and displays it on the display screen. The understanding of all the following YUV format images transmitted to the display screen is the same.

[0258] S14. The display screen displays the YUV image frame of the wide-angle camera.

[0259] From then on, the wide-angle camera began to provide preview images to realize the preview function in normal photo mode.

[0260] In normal camera mode, if there is a flickering light source in the shooting environment, water ripples may appear in the preview image because the exposure time of the wide-angle camera is not necessarily an integer multiple of the light energy cycle corresponding to the flickering light source. Therefore, it is very necessary to detect and process the flickering light source as soon as possible. How to detect the flickering light source information in the shooting environment as soon as possible under these shooting modes and set the exposure time in time to avoid the appearance of water ripples is a problem that the embodiments of the present application focus on solving.

[0261] S15. The AP may obtain the exposure time set by the AEC for the wide-angle camera from the ISP. The exposure time set by the AEC may reflect the brightness of the shooting environment.

[0262] S16. The AP may determine whether flicker detection is required according to the exposure time set by the AEC for the wide-angle camera.

[0263] The environment for flicker detection is usually an indoor scene, and flicker detection is generally not required in a shooting environment that is too bright or too dark. Because, in a too bright environment, the main consideration is the overexposure problem, and the exposure time needs to be significantly reduced, and it is difficult to take into account the water ripple problem; in a too dark environment, there is generally no need to reduce the exposure time to less than the light energy cycle, otherwise the problem of underexposure is likely to occur. Here, too bright or too dark can be defined with reference to the exposure table on the electronic device 100. For example, the shortest exposure times corresponding to the high EV in the exposure table are determined as the exposure time of the too bright environment. Once the AEC exposure time belongs to these shortest exposure times, the environment is considered to be too bright. For another example, the longest exposure times corresponding to the low EV in the exposure table are determined as the exposure time of the too dark environment. Once the AEC exposure time belongs to these longest exposure times, the environment is considered to be too dark. That is, the reference exposure value of too bright or too dark can be derived from the exposure table. Then, if the AEC exposure time belongs to the remaining exposure time in the exposure table, it can reflect that the ambient brightness is neither too bright nor too dark. The electronic device is not limited to analyzing the ambient brightness by the exposure time set by the AEC for the first camera. The electronic device can also collect ambient light through an ambient light brightness sensor to determine whether the shooting environment is too bright or too dark. This application does not impose any restrictions on its technical implementation.

[0264] Flicker detection and water ripple elimination

[0265] S17. After determining to perform flicker detection, the application processor (AP) may set the exposure time of the wide-angle camera to N1 times the first light energy period, such as 10 milliseconds.

[0266] Since the exposure time of the first camera is an integer multiple of the first light energy cycle, the image captured by the first camera under a flickering light source with an operating frequency other than 50 Hz will inevitably produce water ripples. That is, in this case, a flickering light source with a frequency of 60 Hz can be detected.

[0267] After starting the first camera, the AP may also directly set the exposure time of the wide-angle camera without determining whether to perform flicker detection, that is, S15-S16 may not be executed.

[0268] S18. After being activated, the wide-angle camera may transmit the captured first image in a raw image frame (RAW) format to the ISP.

[0269] The ISP may perform a series of processes on the RAW image of the first image and output a YUV image frame.

[0270] S19. The AP may obtain a first image in a YUV format from the ISP.

[0271] S20. The AP may perform image recognition on the first image in the YUV format to detect whether the image content contains water ripple features to obtain a flicker detection result. If the image content contains water ripple features, it is considered that there is a 60 Hz flickering light source in the shooting environment; otherwise, it is considered that there is no flickering light source in the shooting environment, or there is a 50 Hz flickering light source.

[0272] Among them, AP can run Fig. 9 The software architecture shown, especially the CameraFlickerDetector module, is used to implement flicker detection for the first image. As time passes, flicker detection can continue to be performed, especially during the period when the first camera is turned on.

[0273] S21. After determining to perform flicker detection, the application processor (AP) can start the ultra-wide-angle camera as the second camera, and set the exposure time of the ultra-wide-angle camera to M1 times the second light energy period, such as 8.3 milliseconds (1 / 120 seconds).

[0274] Since the exposure time of the second camera is an integer multiple of the second light energy cycle, the image captured by the second camera under a flickering light source other than 60 Hz will inevitably produce water ripples. That is, in this case, a 50 Hz flickering light source can be detected.

[0275] After starting the first camera, the AP may also directly start the ultra-wide-angle camera without determining whether to perform flicker detection, that is, S15-S16 may not be executed.

[0276] S22. After the ultra-wide-angle camera is started, the captured second image in the raw image frame (RAW) format may be transmitted to the ISP.

[0277] The ISP can perform a series of processing on the RAW image of the second image and output a YUV image frame, wherein the YUV image frame of the second camera is not sent for display, that is, the second image in YUV format will not be transmitted to the display screen for display.

[0278] S23. The AP may obtain a second image in a YUV format from the ISP.

[0279] S24. The AP may perform image recognition on the second image in the YUV format to detect whether the image content contains water ripple features to obtain a flicker detection result. If the image content contains water ripple features, it is considered that there is a 50 Hz flickering light source in the shooting environment; otherwise, it is considered that there is no flickering light source in the shooting environment, or there is a 60 Hz flickering light source.

[0280] Among them, AP can run Fig. 9The software architecture shown, especially the CameraFlickerDetector module, is used to implement flicker detection for the second image. Flicker detection can continue to be performed over time, especially during the period when the first camera is turned on.

[0281] Among them, S17-S20 and S21-S24 can be executed in sequence, or S21-S24 can be executed first and then S17-S20. This application does not impose any special restrictions on the execution order here.

[0282] S25. The AP determines whether the first image has a water ripple feature.

[0283] If the AP determines that the first image has water ripple features, S26 is skipped and the following S27-S31 are executed. If the AP determines that the first image does not have water ripple features, S26 is executed to further determine the flickering light source information in the shooting environment.

[0284] S26. The AP determines whether the second image has water ripple features.

[0285] If the AP determines that the first image does not have water ripple features but the second image has water ripple features, the current light energy cycle setting can be maintained and subsequent steps of adjusting the exposure time of the first camera (S32-S37) can be continued.

[0286] S27. Set the exposure time of the wide-angle camera to M1 times the second light energy period.

[0287] Since the AP determines that the first image has water ripple features, it means that there is a 60Hz flickering light source in the current shooting environment. Therefore, the water ripple features in the first image can be eliminated by adjusting the exposure time of the first camera to an integer multiple of the second light energy period corresponding to 60Hz.

[0288] S28. After resetting the exposure time, the wide-angle camera may transmit the acquired first image in a raw image frame (RAW) format to the ISP.

[0289] S29. The ISP may transmit the first image in a YUV format output by the wide-angle camera to the display screen.

[0290] S30. The display screen displays the first image in YUV format from the wide-angle camera.

[0291] From then on, the wide-angle camera began to provide a preview image with the water ripple features removed.

[0292] S31. Set the exposure time of the ultra-wide-angle camera to N1 times the first light energy period.

[0293] In step S27, the AP sets the exposure time of the wide-angle camera to M1 times the second light energy cycle, which not only eliminates the water ripple features in the first image captured by the wide-angle camera, but also enables the wide-angle camera to continue to detect whether there are flickering light sources other than 60Hz in the environment when the shooting environment changes.

[0294] In order to continuously detect the shooting environment where the electronic device 100 is located, the exposure time of the ultra-wide-angle camera can be set to an integer multiple of another light energy cycle (i.e., the first light energy cycle), so that once there is a flashing light source in the environment, it will be detected by the first camera or the second camera.

[0295] S31 is not limited to being executed sequentially after S30, and may also occur after any step of S27-S29. The present application does not impose any special limitation on the execution order of S31.

[0296] The exposure time of the first camera is not fixed after the flicker detection and water ripple elimination steps. The subsequent steps show a possible solution for adjusting the exposure time of the first camera based on motion detection.

[0297] Motion Detection

[0298] S32: The AP may obtain a first image in a YUV format from the ISP.

[0299] S33, the AP may perform image recognition on the first image in the YUV format to detect whether the image content contains motion blur features to obtain a motion detection result. If the image content contains motion blur features, it is considered that the exposure time is set too long, and then the exposure strategy is executed to reduce the exposure time of the camera. Otherwise, it is considered that the exposure time is set correctly and the current exposure time setting is maintained.

[0300] Thereafter, if the content of the first image contains motion blur features, the electronic device 100 can determine whether there is a flickering light source in the current shooting environment based on the aforementioned S25 and S26, and then can execute the following steps S34A or S34B to reduce the exposure time of the camera.

[0301] S34A: When the content of the first image includes a motion blur feature, if the AP determines that there is a flickering light source in the current shooting environment according to the aforementioned step, the AP reduces the exposure time of the wide-angle camera according to the first exposure strategy.

[0302] The AP determines that there is a flickering light source in the current shooting environment, including: determining that there is a 60 Hz flickering light source in step S25; or determining that there is a 50 Hz flickering light source in step S26.

[0303] When the AP adjusts the exposure time of the wide-angle camera according to the first exposure strategy, the light energy period corresponding to the wide-angle camera at that time is used as the adjustment accuracy.

[0304] S34B: When the content of the first image includes a motion blur feature, if the AP determines according to the aforementioned step that there is no flickering light source in the current shooting environment, the AP reduces the exposure time of the wide-angle camera according to the second exposure strategy.

[0305] The AP determines that there is no flickering light source in the current shooting environment, specifically including: the AP does not detect the flickering light source in step S25 and step S26. Optionally, the AP can further analyze the motion blur feature in the first image to obtain motion blur information such as motion speed to determine the reduced exposure time.

[0306] S35. After reducing the exposure time, the wide-angle camera may transmit the collected first image in a raw image frame (RAW) format to the ISP.

[0307] S36. The ISP may transmit the first image in a YUV format output by the wide-angle camera to the display screen.

[0308] S37. The display screen displays the first image in YUV format from the wide-angle camera.

[0309] From now on, the wide-angle camera starts providing preview images with motion blur removed.

[0310] During the execution of the above method, the camera used for display is always a wide-angle camera, but the camera used for display is not fixed. Fig.11 Shows Fig. 7A The switching of the camera in the zoom scene shown, as well as the detection of flickers and the elimination of water ripple features in the process.

[0311] Smooth zoom

[0312] In the zoom scene, you can change the camera that provides the preview image by increasing the zoom ratio to the zoom range of the rear main camera or telephoto camera, or reducing the zoom ratio to the zoom range of the ultra-wide-angle camera. The following is an explanation of reducing the zoom ratio to the zoom range of the ultra-wide-angle camera.

[0313] S38. AP detects that the zoom ratio is reduced.

[0314] S39. The AP determines whether the zoom ratio is reduced to the first focal length threshold of the transition zoom range of the wide-angle camera.

[0315] S40: If the AP determines that the zoom ratio is reduced to the first focal length threshold, the exposure time of the ultra-wide-angle camera is set to an integer multiple of the light energy period 3.

[0316] S41. After setting the exposure time of the ultra-wide-angle camera to an integer multiple of the light energy cycle 3, the ultra-wide-angle camera may transmit the collected sixth image in a raw image frame (RAW) format to the ISP.

[0317] S42. The AP may obtain a sixth image in a YUV format from the ISP.

[0318] S43. The AP may perform image recognition on the sixth image in the YUV format to detect whether the image content contains water ripple features to obtain a flicker detection result.

[0319] S44. After setting the exposure time of the ultra-wide-angle camera to an integer multiple of the light energy cycle 3, the wide-angle camera can transmit the fifth image in the acquired raw image frame (RAW) format to the ISP.

[0320] S45. The AP may obtain the fifth image in YUV format from the ISP.

[0321] S46. The AP may perform image recognition on the fifth image in the YUV format to detect whether the image content contains water ripple features to obtain a flicker detection result.

[0322] S47. AP determines whether the sixth image has water ripple features.

[0323] If the AP determines that the sixth image has water ripple features, it means that the light energy cycle of the flickering light source in the current shooting environment does not match the light energy cycle 3, and then the subsequent steps S48-S49 are executed. If the AP determines that the sixth image does not have water ripples, it jumps to step S50 to further determine the flickering light source information in the current shooting environment through the image recognition result of the fifth image.

[0324] S48. If the AP determines that the sixth image has water ripple features, the exposure time of the ultra-wide-angle camera is set to an integer multiple of the light energy period 4.

[0325] For example, in the above example, if the AP determines that the sixth image has water ripple features, it means that the light energy cycle of the flickering light source in the current shooting environment is not light energy cycle 3, and the light energy cycle of the flickering light source is light energy cycle 4. At this time, the AP can set the exposure time of the ultra-wide-angle camera to an integer multiple of light energy cycle 4, so as to avoid the generation of water ripple features in the image captured by the ultra-wide-angle camera.

[0326] S49. If the AP determines that the sixth image has water ripple features, and if the exposure time of the wide-angle camera is not constrained to be an integer multiple of the light energy period, the exposure time of the wide-angle camera is set to be an integer multiple of the light energy period 4.

[0327] After executing S49, the electronic device may jump to execute S52. It is understandable that if the AP determines that the sixth image has water ripple features, the exposure time of the wide-angle camera has been constrained to an integer multiple of the light energy cycle, then after executing S48, the electronic device may directly jump to execute S52.

[0328] S50. AP determines whether the fifth image has water ripple features; if yes, execute S51; otherwise, execute S52.

[0329] It can be understood that if the AP determines that there is a water ripple feature in the fifth image, then there is a flickering light source with a light energy period corresponding to light energy period 3 in the current shooting environment; if the AP determines that there is no water ripple feature in the fifth image, then in combination with step S47, it can be determined that there is no flickering light source in the current shooting environment.

[0330] S51. If the AP determines that the fifth image has water ripple features, the exposure time of the wide-angle camera is set to an integer multiple of the light energy period 3.

[0331] S52: The AP determines whether the zoom ratio reaches the zoom range of the ultra-wide-angle camera.

[0332] S53: When the zoom range of the ultra-wide-angle camera is reached, the ultra-wide-angle camera may transmit the fourth image in the acquired raw image frame (RAW) format to the ISP.

[0333] S54, the AP notifies the ISP to switch the display camera.

[0334] S55. The ISP may transmit the fourth image in a YUV format output by the ultra-wide-angle camera to the display screen.

[0335] S56. The display screen displays the fourth image in YUV format from the ultra-wide-angle camera.

[0336] It is understandable that when executing S48, since the light energy cycles set by the wide-angle camera and the ultra-wide-angle camera are different, and the sixth image has water ripple features, the fifth image does not have water ripple features. After executing S54, the camera sent for display is changed from the original wide-angle camera to the ultra-wide-angle camera, and the fourth image sent for display by the ultra-wide-angle camera has eliminated the water ripples after the above step S48. Therefore, after the display is sent, even if the display camera is switched, from the user's perspective, there is no abrupt water ripple feature, thereby achieving smooth zoom.

[0337] After switching the display camera, the exposure time of the wide-angle camera can be modified accordingly. If the AP determines that the sixth image has water ripple features before, S57A is executed; if the AP determines that the fifth image has water ripple features before, S57B is executed.

[0338] S57A, after the fourth image is sent for display, the AP sets the exposure time of the wide-angle camera to an integer multiple of the light energy cycle 3.

[0339] S57B, after the fourth image is sent for display, the AP sets the exposure time of the wide-angle camera to an integer multiple of the light energy cycle 4.

[0340] Further, in the smooth zoom solution, after the water ripples are detected and eliminated, the exposure time of the display camera can be further adjusted, that is, the exposure time of the ultra-wide-angle camera can be further adjusted. The above method also includes: the AP can adjust the exposure time of the ultra-wide-angle camera according to the first exposure strategy or the second exposure strategy.

[0341] The above "smooth zoom" illustrates that the camera to be displayed can be changed according to the specific application scenario. Not limited to this, the embodiment of the present application does not limit the application scenario of the camera switching. For example, for an electronic device with only two cameras (front camera and rear camera), when the camera is flipped, the first camera switches to the front camera, and the second camera can be switched to the rear camera.

[0342] Fig.10 and Fig.11 In the embodiment, each camera is coupled to one ISP. Not limited thereto, the electronic device may be configured with multiple ISPs, and multiple cameras may be coupled to different ISPs.

[0343] The term "user interface (UI)" in the specification, claims and drawings of this application refers to the medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on the terminal device, and finally presented as content that the user can recognize, such as pictures, text, buttons and other controls. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, pictures and text. The properties and contents of controls in the interface are defined by tags or nodes, such as XML through <textview> 、 <imgview> 、 <videoview>The controls contained in the interface are specified by nodes such as . A node corresponds to a control or attribute in the interface. After parsing and rendering, the node is presented as user-visible content. In addition, many applications, such as hybrid applications, usually also contain web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is source code written in a specific computer language, such as hypertext markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code. For example, HTML is defined by 、 、 <video> 、 <canvas>To define the elements and attributes of a web page.

[0344] The most common form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operation that is displayed in a graphical manner. It can be an icon, window, control or other interface element displayed on the display screen of an electronic device, where a control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets and other visual interface elements.

[0345] Each step in the above method embodiment provided by the present application can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0346] The present application also provides an electronic device, which may include: a memory and a processor, wherein the memory may be used to store a computer program; and the processor may be used to call the computer program in the memory so that the electronic device executes the method in any one of the above embodiments.

[0347] The present application also provides a chip system, which includes at least one processor for implementing the functions involved in the method executed by the electronic device in any of the above embodiments.

[0348] In one possible design, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0349] The chip system may be composed of the chip, or may include the chip and other discrete devices.

[0350] Optionally, the processor in the chip system may be one or more. The processor may be implemented by hardware or by software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented by software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0351] Optionally, the memory in the chip system may also be one or more. The memory may be integrated with the processor or may be separately arranged with the processor, which is not limited in the embodiments of the present application. Exemplarily, the memory may be a non-transient processor, such as a read-only memory ROM, which may be integrated with the processor on the same chip or may be arranged on different chips respectively. The embodiments of the present application do not specifically limit the type of memory and the arrangement of the memory and the processor.

[0352] Exemplarily, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.

[0353] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which enables a computer to execute the method executed by the electronic device in any of the above embodiments when the computer program is executed.

[0354] The present application also provides a computer-readable storage medium, which stores a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method executed by the electronic device in any of the above embodiments.

[0355] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that the computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. Available media can be magnetic media, (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state drive Solid State Disk), etc.

[0356] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0357] In summary, the above are only embodiments of the technical solution of the present invention, and are not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present invention shall be included in the protection scope of the present invention.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. A method for detecting a flickering light source, applied to an electronic device including a first camera and a second camera, characterized in that: The method comprises: Setting the exposure time of the first camera to an integer multiple of the first light energy period, and the first camera capturing a first image; The electronic device displays a preview image on a display screen, where the preview image is an image captured by the first camera; Setting the exposure time of the second camera to an integer multiple of the second light energy period, so that the second camera collects a second image; Wherein, the shutters of the first camera and the second camera are both rolling shutters; The electronic device performs water ripple detection according to the first image and the second image; When the first image contains water ripple features, there is a flashing light source corresponding to the second light energy period in the environment, and the electronic device sets the exposure time of the first camera to an integer multiple of the second light energy period, and sets the exposure time of the second camera to an integer multiple of the first light energy period; When the first image does not contain water ripple features and the second image contains water ripple features, there is a flashing light source corresponding to the first light energy period in the environment, and the exposure time of the first camera is kept to be an integer multiple of the first light energy period, and the exposure time of the second camera is kept to be an integer multiple of the second light energy period; When neither the first image nor the second image includes water ripple features, there is no flickering light source in the environment.

2. The method according to claim 1, characterized in that When the first image contains water ripple features, the method further includes: The electronic device adjusts the exposure time of the first camera, and the minimum adjustment unit and the minimum value of the exposure time of the first camera are both the second light energy period currently corresponding to the first camera.

3. The method according to claim 1 or 2, characterized in that: When the first image does not include a water ripple feature and the second image includes a water ripple feature, the method further includes: The electronic device adjusts the exposure time of the first camera, and the minimum adjustment unit and the minimum value of the exposure time of the first camera are both the first light energy cycle currently corresponding to the first camera.

4. The method according to any one of claims 1 to 3, characterized in that: When neither the first image nor the second image contains water ripple features, the method further includes: The electronic device adjusts the exposure time of the first camera, and the minimum adjustment unit and the minimum value of the exposure time of the first camera are both smaller than the first light energy period and the second light energy period.

5. The method according to any one of claims 2 to 4, characterized in that: The electronic device adjusting the exposure time of the first camera specifically includes: The electronic device performs image recognition on the first image to determine whether the first image content contains a motion blur feature; When the first image includes the motion blur feature, the electronic device reduces the exposure time of the first camera.

6. The method according to any one of claims 2 to 4, characterized in that: The electronic device adjusting the exposure time of the first camera specifically includes: The electronic device reduces the exposure time of the first camera through automatic exposure control AEC.

7. The method according to any one of claims 1 to 6, characterized in that: The step of setting the exposure time of the first camera to an integer multiple of the first light energy period specifically includes: When the electronic device starts the first camera, the exposure time of the first camera is set to an integer multiple of the first light energy period; The method further includes: when the electronic device starts the first camera, starting the second camera; The step of setting the exposure time of the second camera to an integer multiple of the second light energy period specifically includes: When the second camera is started, the exposure time of the second camera is set to an integer multiple of the second light energy period.

8. The method according to claim 7, characterized in that Before the electronic device starts the first camera, the method further includes: the electronic device detecting a user operation of opening a camera application.

9. The method according to any one of claims 1 to 8, characterized in that: The step of setting the exposure time of the first camera to an integer multiple of the first light energy period specifically includes: The electronic device determines the automatic exposure duration of the first camera through automatic exposure control AEC; The electronic device determines the value of the integer multiple according to the automatic exposure duration and the first light energy period; The electronic device sets the exposure time of the first camera to be the integer multiple of the first light energy period.

10. The method according to any one of claims 1 to 9, characterized in that: After the electronic device displays the preview image on the display screen, the method further includes: The electronic device detects a first zoom operation; The preview image displayed by the electronic device is switched to the image captured by the second camera; the first zoom operation is used to adjust the zoom ratio from the zoom range of the first camera to the zoom range of the second camera.

11. The method according to claim 10, characterized in that In response to the first zoom operation, before switching the preview image to the image captured by the second camera, the method further includes: If the exposure time of the first camera is currently constrained to be an integer multiple of the first light energy period, the exposure time of the second camera is set to be an integer multiple of the first light energy period, and the second camera collects the fourth image; the exposure time of the first camera is set to be an integer multiple of the second light energy period, and the second camera collects the third image; If the exposure time of the first camera is currently constrained to be an integer multiple of the second light energy period, the exposure time of the second camera is set to be an integer multiple of the second light energy period, and the second camera collects the fourth image; the exposure time of the first camera is set to be an integer multiple of the first light energy period, and the second camera collects the third image; If the exposure time of the first camera is not currently constrained to be an integer multiple of the first light energy period or the second light energy period, setting the exposure time of the second camera to be an integer multiple of the second light energy period, and the second camera collects the fourth image; setting the exposure time of the first camera to be an integer multiple of the first light energy period, and the first camera collects the third image; Alternatively, when the exposure time of the first camera is not currently constrained to be an integer multiple of the first light energy period or the second light energy period, the exposure time of the second camera is set to an integer multiple of the first light energy period, and the second camera captures the fourth image; the exposure time of the first camera is set to an integer multiple of the second light energy period, and the first camera captures the third image.

12. The method according to claim 11, characterized in that In response to the first zoom operation, before switching the preview image to the image captured by the second camera, the method further includes: The electronic device performs water ripple detection according to the third image and the fourth image; When the fourth image includes a water ripple feature, the electronic device sets the exposure time of the second camera to a first exposure time, where the first exposure time is an integer multiple of a light energy period different from a light energy period currently corresponding to the second camera; When the fourth image does not include the water ripple feature and the third image includes the water ripple feature, maintaining the exposure time of the second camera to be an integer multiple of the light energy period currently corresponding to the second camera; When neither the fourth image nor the third image includes water ripple features, there is no flickering light source in the environment.

13. The method according to claim 10, characterized in that Before the electronic device detects the first zoom operation, the method further includes: When the zoom ratio is adjusted to the first zoom range, the exposure time of the second camera is constrained to the third light energy period, the electronic device collects the fifth image from the first camera and the sixth image from the second camera respectively; the third light energy period is the first light energy period or the second light energy period; Among them, when the first zoom range is a zoom range between a first focal length threshold and a zoom range of the second camera, the first focal length threshold belongs to the zoom range of the first camera, and the minimum difference between the first focal length threshold and the focal length within the zoom range of the second camera is a preset value.

14. The method according to claim 13, characterized in that Before the electronic device detects the first zoom operation, the method further includes: The electronic device performs water ripple detection according to the fifth image and the sixth image; When the sixth image includes water ripple features, setting the exposure time of the second camera and the current exposure time of the first camera to be integer multiples of a fourth light energy cycle, wherein the fourth light energy cycle is a light energy cycle different from the third light energy cycle; When the sixth image does not include water ripple features and the fifth image includes water ripple features, setting the current exposure time of the first camera to an integer multiple of the third light energy period; When neither the fifth image nor the sixth image includes water ripple features, there is no flickering light source in the environment.

15. The method according to claim 14, characterized in that The method further comprises: When the sixth image includes a water ripple feature, in response to the first zoom operation, setting the exposure time of the first camera to an integer multiple of the third light energy period; When the sixth image does not include water ripple features and the fifth image includes water ripple features, in response to the first zoom operation, the current exposure time of the first camera is set to an integer multiple of the fourth light energy period.

16. The method according to any one of claims 1 to 15, characterized in that: The first camera is a wide-angle camera, and the second camera is an ultra-wide-angle camera; Or the first camera is an ultra-wide-angle camera, and the second camera is a wide-angle camera.

17. An electronic device, characterized in that: The method comprises one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program codes, wherein the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the method according to any one of claims 1 to 16 is executed.

18. A chip system, the chip system is applied to electronic equipment, the chip system comprises one or more processors, characterized in that: The processor is configured to call computer instructions so as to execute the method according to any one of claims 1 to 16.

19. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the method according to any one of claims 1 to 16 is executed.

Images