Flickering light source detection method and electronic equipment

Abstract

The embodiment of the invention provides a flicker light source detection method and electronic equipment, which can amplify water ripple characteristics by adjusting exposure time (integral time of accumulated brightness) to generate an image, and further perform image recognition on the image to locate whether water ripples are contained or not, thereby realizing detection of a flicker light source.

Description

Technical Field

[0001] This application relates to the field of electronic technology, and in particular to a method and electronic device for detecting flickering light sources. Background Technology

[0002] With the development of electronic technology, mobile phones, tablets, and other electronic devices are generally equipped with cameras. Most cameras use rolling shutters, which cause water ripples in the image when taking pictures under AC light sources (such as fluorescent lamps, televisions, computer screens, or household light sources), severely affecting image quality. Summary of the Invention

[0003] In a first aspect, this application provides a flickering light source detection method and an electronic device, which can be applied to electronic devices such as mobile phones and tablets. The method may include: the electronic device activating a camera and acquiring an image through the camera; the image acquired by the camera may include a first image and a second image, wherein the exposure time of the first image is shorter than the exposure time of the second image.

[0004] The electronic device then displays a preview image, which may include a second image but does not include the first image. Based on the first image, the electronic device determines that a flickering light source exists in the shooting environment.

[0005] Using the first method, the electronic device can control the exposure time during image acquisition to produce a first image with a shorter exposure time. In the presence of flickering light sources in the environment, images with shorter exposure times are more likely to exhibit bright and dark stripes (i.e., ripples) under flickering light. Therefore, flickering light sources can be detected by checking whether the first image contains ripples. The second image, with a longer exposure time, is then used for display. This way, the user will not observe the ripples and will hardly perceive any noticeable frame drops.

[0006] In conjunction with the first aspect, in some embodiments, when acquiring the first image, the electronic device can control the camera's exposure time to be less than the camera's exposure time threshold; when acquiring the second image, the camera's exposure time is controlled to be greater than or equal to the camera's exposure time threshold. The threshold can be determined by the dynamic range supported by the camera. That is, the camera's exposure time can be reduced when acquiring the first image, and the reduced exposure time can be lower than the camera's exposure time threshold, causing bright and dark stripes to appear in the first image when captured under flickering light sources. This supports image recognition of the first image to determine whether the first image contains water ripples, thus achieving flickering light source detection.

[0007] In conjunction with the first aspect, in some embodiments, the first image can be a single frame, and the second image can be the next frame after the first image. Thus, at a normal frame rate of 30 FPS or even higher, the first image is not displayed, and the user is unlikely to notice any frame drop, thus not affecting the continuity of the image. Furthermore, the camera's exposure time is immediately restored in the next frame after the first image, without affecting the normal exposure of subsequent captured images.

[0008] In conjunction with the first aspect, in some embodiments, when acquiring the first image, the electronic device can also increase the ISO sensitivity of the camera through automatic exposure, so that the exposure of the first image is close to or equal to the exposure of the previous frame, avoiding the first image being too dark due to underexposure. In the next frame of the first image, as the camera's exposure time recovers, the electronic device can restore the camera's ISO sensitivity through the AE function.

[0009] Specifically, the exposure time threshold of the camera can be calculated and determined using either Formula 1 or Formula 2:

[0010] Formula 1:

[0011] Formula 2:

[0012] The exposure time threshold of the camera can be determined as follows: DR(s) in Formula 1 or Formula 2 above is assigned the calculated value of the exposure time s when the dynamic range supported by the camera is used. In this way, the exposure time threshold can be predetermined based on the known parameter of the camera, the dynamic range supported by the camera, and thus the exposure time of the first image can be controlled according to the threshold when acquiring the image, so that the first image captured under flickering light source can show water ripples.

[0013] In conjunction with the first aspect, in some embodiments, determining the presence of a flickering light source in the shooting environment based on the first image may specifically include: performing image recognition on the first image to determine whether water ripples exist in the first image; if so, determining that a flickering light source exists in the shooting environment. In this way, flickering light source detection can be achieved by performing image recognition on the first image (frames extracted from the image), and the presence or absence of a flickering light source in the shooting environment can be determined based on the presence or absence of water ripples in the first image, without the need for additional detection devices.

[0014] In conjunction with the first aspect, in some embodiments, if it is determined that water ripples exist in the first image, the electronic device can also determine the frequency of the flashing light source based on the location of the dark stripes in the first image. Because the recurring appearance of the dark stripes in the first image under the flashing light source reflects the energy periodicity of the flashing light source, the position of the dark stripes is related to the period of the flashing light source. Therefore, the period of the flashing light source can be determined by the position of the dark stripes, thereby obtaining the frequency of the flashing light source.

[0015] Specifically, the frequency of the flashing light source can be determined by the energy frequency of the light source, the image height of the first image, the spacing between adjacent dark fringes in the first image, and the shutter time. A larger spacing between adjacent dark fringes indicates a longer energy period of the flashing light source; conversely, a smaller spacing indicates a shorter energy period. The electronic device can determine the frequency f of the flashing light source using the following formula: L This formula fully reflects the correlation between the spacing of the dark stripes and the period of the flickering light source:

[0016] f L =f E / 2;

[0017]

[0018] Among them, f E v_diff represents the energy frequency of the flashing light source, v_diff represents the spacing between adjacent dark stripes in the first image, height represents the image height of the first image, and rst represents the shutter speed of the camera.

[0019] In conjunction with the first aspect, in some embodiments, an electronic device can determine whether water ripples exist in a first image by inputting the first image into a first neural network to identify whether water ripples exist in the first image. The input to the first neural network is an image, and the output is a recognition result indicating whether the image contains water ripples.

[0020] In conjunction with the first aspect, in some embodiments, the electronic device can determine whether water ripples exist in the first image by: if a display screen is detected in the first image, the electronic device extracts an image of the display screen area from the first image and inputs the image of the display screen area into a first neural network to identify whether water ripples exist in the image of the display screen area. This allows for flicker light source detection in scenarios where the screen is both a light source and the subject of the image, covering more detection scenarios.

[0021] A display screen can be any object used to display images, such as a mobile phone screen, computer screen, television screen, projection screen, or watch screen.

[0022] In conjunction with the first aspect, in some embodiments, if it is determined that there is a flickering light source in the shooting environment, the electronic device can also control the camera's exposure time to be an integer multiple of the light energy cycle in order to avoid shooting images with water ripples under flickering light sources.

[0023] In a second aspect, this application provides an electronic device comprising: a camera, a display screen, one or more processors, and one or more memories; wherein the camera is used to acquire images, and the display screen is used to display images; the camera, the display screen, and the one or more memories are coupled to one or more processors, and the one or more memories are used to store a computer program, which, when executed by the processor, causes the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0024] Thirdly, embodiments of this application provide a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to perform the methods described in the first aspect and any possible implementation thereof.

[0025] Fourthly, this application provides a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0026] Fifthly, this application provides a computer program product containing instructions that, when the computer program product is run on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0027] Understandably, the electronic device provided in the second aspect, the chip system provided in the third aspect, the computer storage medium provided in the fourth aspect, and the computer program product provided in the fifth aspect are all used to execute the method provided in this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0028] Figure 1 The working principle of a rolling shutter is illustrated by example;

[0029] Figure 2 An example is shown where the preview image provided by the rolling shutter camera has a water ripple effect;

[0030] Figure 3 An example is shown where the exposure time of a rolling shutter camera is set to be twice the length of the light energy cycle to address the water ripple problem;

[0031] Figure 4 The linear segment of the camera response curve is shown;

[0032] Figure 5 The instantaneous and cumulative brightness of the power frequency light source are shown in comparison;

[0033] Figure 6 The comparison shows the periodic variation of the cumulative brightness of the shooting light source and the scene dynamic range within a period;

[0034] Figure 7 The following examples illustrate the dynamic range exhibited by images with different exposure times;

[0035] Figure 8 The comparison shows images with and without global stripes;

[0036] Figure 9 The image height of the first image, the global stripe spacing v_diff in the first image, and the camera shutter time rst are shown.

[0037] Figure 10 The main process of flicker light source detection in the embodiments of this application is shown;

[0038] Figure 11 The hardware structure of an electronic device according to an embodiment of this application is illustrated. Detailed Implementation

[0039] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be a limitation of this application.

[0040] The shutter is a crucial component of a camera, used to control the exposure time of the image sensor. Shutters can be divided into global shutters and rolling shutters, each with different exposure methods. With a global shutter, all pixels on the image sensor begin exposing to light simultaneously, and after the same exposure time, they all cease exposure simultaneously. With a rolling shutter, pixels on the image sensor are exposed row by row in a sequential manner, with each row having the same exposure time, until all pixels have been exposed. However, the start and end times of exposure differ between different rows.

[0041] Taking photos with a rolling shutter camera under certain light sources may result in "banding" in the image. These light sources are often driven by alternating current (AC). The energy transmitted by AC is not constant but varies with a fixed frequency (e.g., 50Hz or 60Hz), causing periodic fluctuations in the light intensity. The period is the light energy cycle corresponding to that fixed frequency (also known as the power frequency). For example, the light intensity fluctuation frequency of a 50Hz AC light source is 100Hz, and the light energy cycle is 10 milliseconds, meaning the light source flickers 100 times per second. In this article, this type of light source is referred to as a flickering light source.

[0042] For rolling shutter cameras, when the exposure time is less than the light energy cycle of the light source, because the start and end times of exposure for each row of pixels on the sensor are different, the accumulated light energy received by each row of pixels within the same exposure time is different, resulting in alternating bright and dark stripes in the image. For example, as... Figure 1 As shown, when imaging the first frame of the image, the light energy received by the pixels in the first row of the film is the light energy accumulated from time 0 to time t2, while the light energy received by the pixels in the nth row of the film is the light energy accumulated from time t1 to 5 milliseconds. The light energy received by the nth row is less than the light energy received by the first row. For example, as... Figure 1 As shown, when imaging the (N+1)th frame of the image, the light energy received by the pixels in the first row of the film is the light energy accumulated from time t3 to t5, while the light energy received by the pixels in the nth row of the film is the light energy accumulated from time t4 to t6. The light energy received by the first row is less than that received by the nth row. In a frame of the image, rows with high light energy appear as bright stripes, while rows with low light energy appear as dark stripes, as shown in the diagram. Figure 2 As shown, the alternating light and dark stripes are water ripples, a phenomenon also known as flickering. Furthermore, users can observe the ripples rolling during previewing or recording.

[0043] In high-shutter-speed shooting scenarios, such as motion capture and high dynamic range (HDR) modes, water ripples are more likely to appear in the image. After detecting flickering light sources in the shooting environment, one way to avoid this problem is to set the exposure time to an integer multiple of the light energy cycle. For example... Figure 3 As shown, the light energy period is 10 milliseconds. The exposure time can also be set to 10 milliseconds, which is twice the light energy period. In this way, each row of pixels on the film receives the same amount of light energy during the same exposure time, and there will be no alternating bright and dark stripes in the image.

[0044] Detecting flickering light sources is key to solving the water ripple problem. One detection method involves using image recognition algorithms to identify water ripple features in the image content. Once the presence of water ripples is confirmed in the image, the exposure time is adjusted to avoid them. However, until the water ripples are eliminated, the user will still see them, resulting in a poor user experience. Another method is to use dedicated flicker detectors, such as high-sampling-rate ambient light sensors or multispectral flicker detectors, to detect the presence of flickering light sources in the shooting environment, working in conjunction with the camera to solve the water ripple problem. However, these devices require additional cost and design space.

[0045] This application provides a method for detecting flickering light sources. An image is generated by adjusting the exposure time (integration time of cumulative brightness) to amplify the banding features of water ripples. The image is then used for image recognition to locate whether it contains water ripples, thereby detecting flickering light sources.

[0046] First, some theoretical analysis related to the embodiments of this application will be introduced.

[0047] 1. Linear model of image brightness and exposure

[0048] When ambient lighting is stable, calibration is performed on a large number of camera systems, such as... Figure 4 As shown, within most exposure value ranges, such as 50-200 EV, the camera response function (CRF) curve is basically linear, meaning that the image brightness B is linearly positively correlated with the exposure EV. This relationship can be expressed as: B = p1·EV, where p1 is a coefficient with a value greater than 0, and "·" represents the multiplication operator.

[0049] When the exposure EV is constant, the image brightness B and the ambient light (subject brightness) L are positively correlated. Here, we first assume that this positive correlation is linear, which can be expressed as: B = p2·EV·L, where p2 is a coefficient with a value greater than 0, and p2 is different from p1 in the previous formula.

[0050] 2. Brightness Model of Power Frequency Electric Light Source

[0051] The instantaneous brightness of a power frequency electric light source with a period of π / θ can be expressed as: f(t) = abs(W·sinθt). Here, W represents the peak brightness, and abs represents the operation of taking the absolute value. The waveform of the instantaneous brightness f(t) can be shown as follows: Figure 5 The waveform diagram above is an example.

[0052] Therefore, the cumulative luminance L(t) of the power frequency electric light source during the period from time t to time t+s can be expressed as:

[0053]

[0054] The cumulative brightness is obtained by integrating the brightness f(t) of the power frequency electric light source over a duration s. The cumulative brightness L(t) can be expressed as follows: Figure 5 The waveform diagram below is an example, specifically the integral area of ​​the waveform. The cumulative brightness over a period of integration time is the area enclosed by the waveform diagram during that integration time.

[0055] Since the absolute value function f(t) is discontinuous, L(t) needs to be integrated piecewise over t:

[0056] when hour, Among them, when When L(t) reaches its maximum value:

[0057] when hour, Among them, when When L(t) reaches its minimum value:

[0058] when When the integration time is an integer multiple of the light energy cycle, the cumulative brightness generated by integrating at any given time is a constant value: 2W / θ. However, when... At that time, the cumulative brightness L(t) fluctuates periodically, and the period is also π / θ.

[0059] Substituting the formula for calculating cumulative brightness L(t) into the formula for calculating image brightness B, we can obtain the brightness of one line of image produced by an exposure of duration s starting from a certain time t under line-by-line exposure:

[0060]

[0061] The period of the imaging brightness B(t) is π / θ.

[0062] In this embodiment of the application, the scene dynamic range DR(s) caused by the energy difference perceived by different exposure rows on the film within a single frame of an image can be calculated and determined by the following formula:

[0063] DR(s) = 20log(L_max / L_min)

[0064] Where L_max represents the brightness of the bright areas in a frame of an image, and L_min represents the brightness of the dark areas in a frame of an image.

[0065] L_max can be calculated as follows:

[0066] L_min can be calculated as follows:

[0067] The formula for calculating the dynamic range of the aforementioned scenario can be transformed into:

[0068] In the above formula for calculating DR(s), L_max can be simply replaced by the average brightness L_avg of a frame of image. L_avg represents the brightness of the normal bright and dark areas of a frame of image, and L_avg can be calculated as follows:

[0069]

[0070] The formula for calculating the dynamic range of the aforementioned scenario can also be transformed into:

[0071] The aforementioned scene dynamic range DR(s) is a function of exposure time s, which reflects the dynamic range of ambient light. Figure 6 The comparison shows the periodic variation of the cumulative brightness of the shooting light source and the scene dynamic range (DR) (s) within one period. For example... Figure 6 As shown, the scene dynamic range DR(s) within one period π / θ is as follows: when s approaches 0, DR(s) approaches infinity; when s equals one period π / θ, DR(s) equals 0, that is, the brightness of the dark area is the same as the brightness of the bright area (or normal area), and there is no phenomenon of bright and dark stripes due to uneven exposure; when s is greater than one period π / θ, DR(s) stabilizes at a small value, and the ratio of the same length of integration starting at any time t does not exceed 2 when the integration time s is fixed.

[0072] The shorter the exposure time s, the larger the DR(s), and the greater the difference between light and dark areas in an image, which is more conducive to observing water ripples under flickering light sources. The embodiments of this application can amplify the difference between light and dark areas in a specific frame by reducing the exposure time s in that frame, thereby magnifying the features of water ripples under flickering light sources.

[0073] Cameras in mobile phones and other electronic devices have a limited dynamic range. When the exposure time *s* is reduced so that the dynamic range the image needs to represent (i.e., the scene dynamic range) exceeds the camera's supported dynamic range (e.g., 5 dB), the captured image will inevitably be underexposed, resulting in almost pure black dark stripes. This significantly improves the accuracy of water ripple recognition. dB is a unit called decibel.

[0074] Different manufacturers and models of cameras may support different dynamic ranges. The dynamic range threshold supported by a camera is a known parameter of the camera. By substituting the dynamic range supported by the camera into the above formula for calculating DR(s), the exposure time threshold of the camera can be solved. In practical applications, this exposure time threshold may not be equal to the solution value of the above formula for calculating DR(s), but rather a value close to that solution value. Once the exposure time s is less than this exposure time threshold, observable water ripples with obvious differences in brightness will appear in the image captured under flickering light sources.

[0075] Figure 7 Examples illustrate the dynamic range exhibited by images with different exposure times. For instance... Figure 7 As shown, the exposure times for images a to e are 9 milliseconds, 7 milliseconds, 5 milliseconds, 3 milliseconds, and 1 millisecond, respectively. Observable bright and dark stripes begin to appear in image a. As the exposure time is further shortened, the bright and dark stripes in images b, c, d, and e become more and more obvious.

[0076] By reducing the exposure time s, the dynamic range DR(s) of the shooting scene can exceed the dynamic range threshold supported by the camera. In this case, the image captured under flickering light sources will inevitably be underexposed, resulting in a water ripple effect.

[0077] In addition to amplifying the water ripple features by reducing the exposure time, the auto exposure (AE) function can be enabled to ensure clear imaging even with reduced exposure time, guaranteeing water ripple recognition. The AE function maintains the brightness of consecutive image frames within the target range, thus keeping the exposure stable. The formula for calculating exposure EV is as follows: EV = Exposure time (s) * Aperture size * ISO sensitivity, where "*" represents the multiplication operator. The AE function can maintain stable exposure by changing the shutter speed (s) and ISO sensitivity. When reducing the exposure time (s) in a specific frame, the camera's ISO sensitivity can be increased simultaneously, making the exposure of that frame close to or equal to the exposure of the previous frame, preventing the image from being too dark due to underexposure.

[0078] Next, we will introduce the flickering light source detection method provided in the embodiments of this application. This method can be applied to electronic devices such as mobile phones and tablets.

[0079] An electronic device implementing this method may have one or more rolling shutter cameras. In the presence of flickering light sources in the shooting environment, if the exposure time is not set to an integer multiple of the light energy period, water ripples will appear in the image captured by the rolling shutter camera.

[0080] The following is a general flow chart of the flickering light source detection method provided in the embodiments of this application.

[0081] S101, Electronic device activates camera.

[0082] The camera can be activated by the user opening the camera app. The camera can be either a rear camera or a front camera.

[0083] When the camera is turned on, the electronic device can set the camera's exposure time to a first duration, which can be greater than the light energy cycle and an integer multiple of the light energy cycle, such as 50ms. This will prevent users from seeing water ripples when the camera is turned on.

[0084] Furthermore, when activating the camera, the camera application can be in normal shooting mode. Unlike motion capture or HDR shooting modes, which require reduced exposure time, the camera's exposure time in normal shooting mode can be set to a relatively long value, such as 50ms or 100ms, which are high multiples of the light energy cycle. This initial exposure time can be set by the ISP for the camera through the AE. The light energy cycle is not limited to the 50Hz power frequency; it can also be the 60Hz power frequency or other power frequencies.

[0085] The scenario isn't limited to a user opening the camera app; camera activation can occur in other situations requiring image capture, such as video calls, QR code payments, and identity verification. For example, in a QR code payment scenario, when the device detects a user initiating payment, it can activate the rear main camera to scan the payment QR code. In this case, the camera is the rear main camera. The rear main camera could be, for example, a rear wide-angle camera. Similarly, in an identity verification scenario, the device can activate the front camera, along with a front-facing time-of-flight (ToF) or structured light camera, to scan and obtain facial feature data for user authentication. In this case, the camera can be the front main camera. Furthermore, in a video call scenario, the device can activate the front main camera to capture the user's image on the screen.

[0086] S102, the electronic device acquires an image through a camera, wherein the acquired image may include a first image and a second image, and the exposure time of the first image is shorter than the exposure time of the second image.

[0087] Specifically, when acquiring the first image, the electronic device can control the camera's exposure time to be less than the camera's exposure time threshold; when acquiring the second image, the camera's exposure time can be controlled to be greater than or equal to the camera's exposure time threshold.

[0088] Specifically, the electronic device can reduce the camera's exposure time to below the camera's exposure time threshold when capturing the first image, and then restore the camera's exposure time in the next frame of the first image.

[0089] As mentioned earlier, once the exposure time is less than the exposure time threshold, observable water ripples with significant differences in brightness will appear in the image captured under a flickering light source. Therefore, if a flickering light source exists in the shooting environment, water ripples will inevitably appear in the first image, and the flickering light source can be detected by analyzing the image content of the first image.

[0090] The camera's exposure time threshold can be set when entering... Figure 8The method flow shown is determined beforehand. The dynamic range supported by the camera is a known parameter of the camera. An exposure time can be calculated using the aforementioned formula for scene dynamic range DR(s) such that the scene dynamic range DR(s) exceeds or approaches the dynamic range supported by the camera. This exposure time can be used as the camera's exposure time threshold.

[0091] The first image can be an image frame captured at any time during the camera's operation. The first image can be further specified as an image frame captured shortly after the camera is turned on, in order to capture an image frame containing water ripples as early as possible and minimize the likelihood of the user observing the water ripples.

[0092] S103, the electronic device displays the image captured by the camera, wherein the first image is not displayed.

[0093] Images captured by a camera can be used as preview images displayed on a screen by a camera application. The preview image displayed on an electronic device includes a second image but does not include the first image.

[0094] The first image can be a small number of frames, such as just one frame. In this way, at a normal frame rate of 30 FPS or even higher, the first image is not displayed, and the user will hardly notice the frame drop, thus not affecting the continuity of the picture.

[0095] The camera can be a single camera, in which case the preview image displayed on the screen by the camera app comes only from that camera. Alternatively, the camera can be multiple cameras, in which case the preview image displayed on the screen is a composite of images captured simultaneously by these multiple cameras, providing the user with multi-camera photo or video recording capabilities.

[0096] As time passes, S103 can continue to be executed, continuing to display the preview stream from the camera to continuously provide a preview display to the user.

[0097] S104, the electronic device can determine from the first image that there is a flickering light source in the shooting environment.

[0098] Specifically, the electronic device performs image recognition on the first image to determine whether there are water ripples in the first image. If there are, it is determined that there is a flickering light source in the shooting environment.

[0099] The first image is not displayed; it is only used for flicker light source detection. Therefore, the user cannot see the first image or any water ripples that may appear within it. The detection result for the flicker light source can include: flicker light source detected, or flicker light source not detected. Subsequently, the electronic device can control the camera's exposure based on this detection result, ensuring that the camera's exposure time is an integer multiple of the light energy cycle to avoid water ripples appearing in the preview image provided by the camera.

[0100] For example, when switching from normal shooting mode to motion capture mode, in order to reduce the exposure time to mitigate motion blur, the electronic device can specifically reduce the camera's exposure time to twice the light energy cycle. That is, the reduced exposure time is controlled to be an integer multiple of the light energy cycle, so as to avoid the problem of water ripples appearing in the image under flickering light sources.

[0101] In addition, after the camera is activated, the electronic device can also enable the automatic exposure (AE) function to ensure sufficient exposure in the first image, preventing it from being too dark. Specifically, in the first image, the electronic device can increase the camera's ISO sensitivity through the AE function, making the exposure of the first image close to or equal to the exposure of the previous frame, thus preventing the first image from being too dark due to underexposure. In the next frame, as the camera's exposure time recovers, the electronic device can restore the camera's ISO sensitivity through the AE function. That is, after capturing the first image, the camera's exposure time can be increased to restore the exposure time used before the first image, and the camera's ISO sensitivity can be decreased to restore the ISO sensitivity used before the first image.

[0102] In this embodiment, the electronic device can input a first image into a first neural network to identify whether there are water ripples in the first image, thereby determining whether there is a flickering light source in the shooting environment. The input of the first neural network is an image, and the output is the recognition result of whether the image contains water ripples.

[0103] If the first image contains an object such as a display screen, the electronic device can extract the image of the display screen from the first image, and then input the image of the display screen into a first neural network to identify whether there are water ripples in the image of the display screen, thereby determining whether there is a flickering light source in the shooting environment. In this scenario, the flickering light source is the display screen. In addition to being an image display device, the display screen is also a light-emitting device.

[0104] A display screen can be any object used to display images, such as a mobile phone screen, computer screen, television screen, projection screen, or watch screen.

[0105] A specific implementation for recognizing water ripples and further determining the frequency of the flickering light source may include:

[0106] S201, determine whether there are global stripes in the first image. If there are, there are water ripples, and S202 can be executed. If not, S203 can be executed.

[0107] Figure 8 The comparison shows images with and without global stripes. (Example) Figure 8 As shown, global stripes refer to dark stripes in the first image. Global stripes extend in their own direction to the two image boundaries of the first image.

[0108] The electronic device can further determine whether global stripes exist in the first image using a first neural network. The input of the first neural network is an image, and the output is the recognition result of whether global stripes exist in the image. The electronic device is not limited to using a neural network; it can also determine whether global stripes exist in the first image using an image recognition algorithm. This application embodiment does not limit the method for recognizing whether global stripes exist in the first image.

[0109] As analyzed above, when the dynamic range of the first image (the dynamic range of the shooting scene) exceeds the dynamic range supported by the camera by reducing the camera's exposure time, underexposed dark stripes will inevitably appear in the first image captured under flickering light sources. Furthermore, the dark stripes are almost pure black, making the contrast between the dark stripes and the bright areas in the image sufficiently large, which greatly improves the accuracy of water ripple recognition.

[0110] S202, locate global stripes in the first image, and determine the frequency of the flashing light source based on the spacing between adjacent global stripes.

[0111] Specifically, the electronic device can locate global stripes in the first image using a second neural network. The input to the second neural network can be an image containing global stripes, and the output can be the position of the global stripes in that image. Here, the position of the stripes in the image can be described in pixels. The electronic device is not limited to using a neural network; it can also locate global stripes in the first image using an image recognition algorithm. This application embodiment does not limit the method of locating global stripes in the first image.

[0112] After locating the global stripes, the electronic device can determine the frequency f of the flashing light source using the following method. L :

[0113] f L =f E / 2;

[0114]

[0115] Where " / " represents division and "*" represents multiplication, f E The energy frequency of the flashing light source is represented by v_diff, the spacing between adjacent global stripes is represented by height, the image height of the first image is represented by rst, and the shutter speed of the camera is represented by rst.

[0116] rst = height * tline, where tline is the line readout time of the camera. V_diff, height, and rst can be defined as follows: Figure 9 The diagram is shown in the image.

[0117] S203, detect whether there is a display screen in the first image. If there is, first extract the display screen area from the first image, then determine whether there are global stripes in the display screen area. If there are global stripes in the display screen area, it is determined that there are water ripples. Then locate the global stripes in the display screen area and determine the frequency of the flashing light source based on the spacing between adjacent global stripes.

[0118] The first image containing an object such as a display screen can be referenced. Figure 8 The image on the right in the first image. An electronic device can use a third neural network to identify whether a display screen exists in the first image. The input to the third neural network can be an image, and the output can be the identification result of whether a display screen exists in that image. The electronic device is not limited to using a neural network; it can also use image recognition algorithms to identify whether a display screen exists in the first image. This application does not limit how the first image is identified.

[0119] The global stripes in the display area will extend in their own direction to both image boundaries of the display area.

[0120] The method for locating global stripes in this display area is the same as that for locating global stripes in the first image, as described in S202, and will not be repeated here. However, in the formula for calculating frequency fL, height represents the image height of the display area.

[0121] In the embodiments of this application, such as Figure 10 As shown, the flashing light source detection process mainly includes two steps:

[0122] (I) Magnification of Water Ripples

[0123] The automatic exposure method finds a short exposure time for the camera, which is the aforementioned exposure time threshold of the camera. If the camera's exposure time is shorter than this short exposure time, the dynamic range of the image captured by the camera will exceed the dynamic range supported by the camera, resulting in bright and dark stripes in the image.

[0124] During image acquisition by the camera, the electronic device can reduce the camera's exposure time when acquiring the first image. The reduced exposure time is shorter than the camera's exposure time threshold, and the exposure time is restored in the next frame of the first image. This results in bright and dark stripes appearing in the first image when shooting under flickering light. The first image is not displayed, so the user will not observe the water ripples and will hardly notice any significant frame drop. This process is described in S102 above.

[0125] (II) Global Stripe Recognition

[0126] Step 1: Use a classification network to identify whether there are global stripes in the first image. If there are global stripes, then use a global stripe localization network to locate the position of the global stripes in the first image.

[0127] The classification network, also known as the first neural network mentioned above, can be used to classify images based on whether they contain global stripes, including images with global stripes and images without global stripes. The global stripe localization network, also known as the second neural network mentioned above, can be used to locate the positions of global stripes in images containing water ripples.

[0128] Step 2: Determine the frequency of the flashing light source based on the spacing between adjacent global stripes.

[0129] Step 3: If the classification network identifies no global stripes in the first image in Step 1, the object detection network can be used to identify whether the first image contains a display screen. If it does, the display screen area can be cropped from the first image. Then, the classification network can be used to identify whether there are global stripes in the display screen area. If there are global stripes, the global stripe localization network can be used to locate the position of the global stripes in the display screen area.

[0130] The object detection network, also known as the aforementioned third neural network, can be used to detect whether an image contains a target object such as a display screen.

[0131] For the specific implementation of the global stripe recognition process, please refer to the aforementioned S201-S203, which will not be repeated here.

[0132] The two processes described above can be implemented as two software modules in an electronic device: a water ripple amplification module and a global stripe recognition module.

[0133] Next, the hardware and software architecture of the electronic device provided in the embodiments of this application will be introduced.

[0134] Electronic devices can be equipped with Or other portable terminal devices with different operating systems, such as mobile phones, tablets, desktop computers, laptops, handheld computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, in-vehicle devices, smart home devices and / or smart city devices, etc.

[0135] Figure 11 The hardware structure of the electronic device 100 provided in this application embodiment is illustrated by way of example. The electronic device 100 can detect the presence of flickering light sources in the shooting environment, and it does not require additional dedicated flicker detection devices, nor does it prevent the user from seeing the generation of water ripples.

[0136] like Figure 11 As shown, the electronic device 100 may include: a processor 110, a memory 120, a camera 130, and a display screen 140. Wherein:

[0137] Processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. Processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface. These interfaces are used by the processor 110 to interact with peripherals.

[0138] The memory 120 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM). The RAM can be directly read and written by the processor 110 and can be used to store executable programs (e.g., machine instructions) of the operating system or other running programs, as well as user and application data. The NVM can also store executable programs and user and application data, and can be pre-loaded into the RAM for direct reading and writing by the processor 110. The processor 110 may also include a storage unit, which can be a cache storage unit, used to store recently used or repeatedly used instructions or data. The implementation code of the flicker light source detection method provided in this embodiment can be stored in the NVM. When the camera application is started, this code can be loaded into RAM. Thus, the processor 110 can directly read the program code from RAM to implement the flicker light source detection method provided in this embodiment. Furthermore, photos, videos, and other image files obtained by the user using the camera application can be written to the NVM for storage and browsing by the user.

[0139] Camera 130 may include a lens, an image sensor, and a flexible printed circuit board (FPCB). The FPCB is responsible for connecting other components of camera 130 to processor 110, such as transmitting raw data output from the image sensor to processor 110. When taking a picture, the shutter of camera 130 is opened, allowing light to enter and illuminate the image sensor. The image sensor converts the light signal into an electrical signal, which is then further converted into a digital signal via analog-to-digital converter (ADC) for transmission to the image sensor's ISP. The ISP can perform the following processing on the image sensor's output data: auto exposure control (AEC), auto gain control (AGC), auto white balance (AWB), color correction, dead pixel removal, etc. The ISP can also be integrated within camera 130.

[0140] Display screen 140 can be used to display images captured by camera 130. The image processed by the ISP is sent to the display screen 140 to show the user a preview of the image captured by the camera. Here, "sending to the display screen" means pushing the image captured by the camera into the frame buffer (FB) for storage. The frame buffer is a storage space, which can be located in video memory or main memory, used to store rendering data processed or to be extracted by the graphics card chip. The contents of the frame buffer correspond to the interface display on display screen 140; it can be simply understood as a cache corresponding to the content displayed on display screen 140. Modifying the contents of the frame buffer is equivalent to modifying the content displayed on display screen 140.

[0141] Camera 130 may include one or more rolling shutter cameras.

[0142] When the camera is activated (e.g., when a user opens a camera app, triggering activation), the electronic device 100 can capture images through the camera. During image capture, in the first frame, the electronic device 100 can reduce the camera's exposure time to below the camera's exposure time threshold and restore the camera's exposure time in the next frame. The first image is not displayed. Thus, detecting flickering light sources based on the first image will not make the water ripples perceptible to the user, and will not cause the user to perceive a noticeable frame drop.

[0143] Based on the detection results of the flickering light source, the electronic device 100 can effectively avoid the water ripple problem in the camera's image. For example, when switching from normal shooting mode to sports shooting mode, the camera's exposure time needs to be reduced to alleviate motion blur. At the same time, in order to also solve the problem of water ripples in the image under flickering light source, the camera's exposure time can be set to a minimum of 10ms, which is the length of the light energy cycle.

[0144] If water ripples are detected in the first image, the electronic device 100 can also locate the position of the dark stripe in the first image and determine the frequency of the flashing light source based on the position of the dark stripe in the first image.

[0145] The camera can be a single camera, in which case the preview image displayed on the display screen 140 by the camera app comes only from that camera. Alternatively, the camera can be multiple cameras, in which case the preview image displayed on the display screen 140 is composed of stitched images captured simultaneously by these multiple cameras, providing the user with the ability to take photos or record videos using multiple cameras.

[0146] The camera's exposure time can also be controlled by the Automatic Exposure (AE) function. AE, simply put, automatically adjusts the exposure time based on light intensity. Light intensity varies greatly in different environments, and the exposure time of the image sensor in the camera 130 needs to adapt accordingly to ensure proper image exposure. Automatic Exposure (AE) provides this adaptive capability. Therefore, the image sensor's exposure time can reflect the brightness of the shooting environment to a certain extent.

[0147] In addition, the electronic device 100 can also store the exposure table to be used by the AE. The exposure table of a camera sets the exposure parameters (such as exposure time, exposure gain, aperture size) to be used by the camera at different exposure values ​​(EV), and can be calibrated and recorded by the engineer.

[0148] like Figure 11 As shown, the electronic device 100 may further include: a charging management module 143, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, and a wireless communication module 170.

[0149] The charging management module 143 receives charging input from the charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 143 receives charging input from the wired charger via a USB interface. In some wireless charging embodiments, the charging management module 143 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 143 can also supply power to the electronic device via the power management module 141.

[0150] The power management module 141 connects the battery 142, the charging management module 143, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 143, supplying power to the processor 110, memory 120, display screen 140, camera 130, mobile communication module 150, wireless communication module 170, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 143 may be located in the same device.

[0151] The wireless communication function of electronic device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 170, modem processor, and baseband processor.

[0152] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0153] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0154] The wireless communication module 170 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 170 can be one or more devices integrating at least one communication processing module. The wireless communication module 170 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 170 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2. For example, the wireless communication module 170 may include a Bluetooth module, a Wi-Fi module, etc.

[0155] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 170, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0156] like Figure 11 As shown, the electronic device 100 may also include a sensor module 160, which may specifically include a pressure sensor 160A, a distance sensor 160F, a proximity light sensor 160G, a touch sensor 160K, an ambient light sensor 160L, etc.

[0157] Figure 11 The illustrated structure does not constitute a specific limitation on the electronic device. An electronic device may include more or fewer components than illustrated, or combine or separate certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of both.

[0158] The term "user interface (UI)" used in the specification, claims, and drawings of this application refers to the medium through which an application or operating system interacts and exchanges information with the user. It converts information from its internal form to a form acceptable to the user. The user interface of an application is source code written in a specific computer language such as Java or Extensible Markup Language (XML). This source code is parsed and rendered on the terminal device, ultimately presenting user-recognizable content such as images, text, and buttons. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, images, and text. The attributes and content of controls in the interface are defined using tags or nodes, such as XML tags. <textview> 、 <imgview> 、

[0159] <videoview>Nodes define the controls contained in the interface. A node corresponds to a control or property in the interface, and after parsing and rendering, the node is presented as the content visible to the user. In addition, many applications, such as hybrid applications, often contain web pages within their interfaces. A web page, also known as a webpage, can be understood as a special control embedded in the application interface. Web pages are source code written in a specific computer language, such as Hypertext Markup Language (HTML), Cascading Style Sheets (CSS), JavaScript (JS), etc. Web page source code can be loaded and displayed as user-readable content by a browser or a web page display component with browser-like functionality. The specific content contained in a webpage is also defined through tags or nodes in the webpage source code; for example, HTML uses tags or nodes to define the content. 、 、 <video> 、 <canvas>Used to define the elements and attributes of a webpage.

[0160] The most common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0161] The steps in the above-described method embodiments provided in this application can be implemented by integrated logic circuits in the processor or by instructions in software form. The method steps disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules within the processor.

[0162] This application also provides an electronic device that may include a memory and a processor. The memory may be used to store a computer program; the processor may be used to invoke the computer program in the memory to cause the electronic device to perform the methods in any of the above embodiments.

[0163] This application also provides a chip system including at least one processor for implementing the functions involved in the methods performed by the electronic device in any of the above embodiments.

[0164] In one possible design, the chip system also includes a memory for storing program instructions and data, which may be located inside or outside the processor.

[0165] The chip system can consist of chips or include chips and other discrete components.

[0166] Optionally, the chip system may contain one or more processors. These processors can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.

[0167] Optionally, the chip system may contain one or more memories. The memory may be integrated with the processor or disposed separately from it; this application embodiment does not limit this. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or disposed separately on different chips. This application embodiment does not specifically limit the type of memory or the arrangement of the memory and processor.

[0168] For example, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0169] This application also provides a computer program product, which includes a computer program (also referred to as code or instructions) that, when run, causes a computer to perform the method executed by the electronic device in any of the above embodiments.

[0170] This application also provides a computer-readable storage medium storing a computer program (also referred to as code or instructions). When the computer program is run, it causes the computer to perform the method executed by the electronic device in any of the above embodiments.

[0171] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, they produce, in whole or in part, the processes or functions according to this application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any usable medium that a computer can access or a data storage device such as a server or data center that integrates one or more usable media. The usable medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive).

[0172] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0173] In summary, the above are merely some embodiments provided in this application, intended for illustrative and exemplary purposes, and not for limiting the scope of protection claimed by this application. For those skilled in the art, many modifications, variations, and improvements are possible given the teachings of the above embodiments. These modifications, variations, and improvements may fall within the scope of protection defined by the claims of this application.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. A scintillation light source detection method characterized by, The method is applied to an electronic device, and the method comprises: starting a camera; capturing an image by the camera, wherein the image captured by the camera comprises a first image and a second image, and an exposure time of the first image is less than an exposure time of the second image; displaying a preview image, wherein the preview image comprises the second image and does not comprise the first image; determining, according to the first image, that a flickering light source exists in a shooting environment.

2. The method of claim 1, wherein the exposure time of the first image is less than a threshold value, and the exposure time of the second image is greater than or equal to the threshold value, wherein the threshold value is determined by a dynamic range supported by the camera.

3. The method of claim 2, wherein, the first image is a frame image, and the second image is a next frame image of the first image.

4. The method of any one of claims 1-3, wherein, when the first image is captured, the electronic device further increases an ISO sensitivity of the camera through automatic exposure.

5. The method of any one of claims 2-4, wherein, the exposure time threshold value of the camera is determined by Formula 1 or Formula 2: Equation One: Equation Two: wherein the exposure time threshold value of the camera is a calculated value of exposure time s when DR(s) in the Formula 1 or the Formula 2 is assigned to the dynamic range supported by the camera.

6. The method of any one of claims 1-5, wherein, The determination, according to the first image, that a flickering light source exists in a shooting environment specifically comprises: performing image recognition on the first image to determine whether water ripples exist in the first image, and if so, determining that a flickering light source exists in the shooting environment.

7. The method of claim 6, wherein, Further comprising: if it is determined that water ripples exist in the first image, the electronic device determines a frequency of the flickering light source according to positioning of dark stripes in the first image.

8. The method of claim 7, wherein, The frequency of the flickering light source is determined by an energy frequency of a light source, an image height of the first image, a distance between adjacent dark stripes in the first image, and a shutter time.

9. The method of claim 7 or 8, wherein, The electronic device determines the frequency of the flickering light source according to positioning of dark stripes in the first image, specifically comprising: The electronic device determines the frequency f of the flickering light source by the following equation L : f L = f E / 2; wherein f E represents the energy frequency of the flicker light source, v_diff represents the spacing between adjacent dark fringes in the first image, height represents the image height of the first image, and rst represents the shutter time of the camera.

10. The method of any one of claims 6-9, wherein, The determination of whether water ripples exist in the first image specifically comprises: inputting the first image into a first neural network to identify whether water ripples exist in the first image.

11. The method of any one of claims 6-9, wherein, The determination of whether water ripples exist in the first image specifically comprises: if a display screen is detected in the first image, the electronic device crops an image of the display screen area from the first image and inputs the image of the display screen area into a first neural network to identify whether water ripples exist in the image of the display screen area.

12. The method of any one of claims 1-11, wherein, Further comprising: if it is determined that a flickering light source exists in the shooting environment, the electronic device controls the exposure time of the camera to be an integer multiple of a light energy period.

13. An electronic device, comprising: Comprising: a camera, a display screen, one or more processors, and one or more memories; wherein the camera is configured to capture an image, and the display screen is configured to display an image; the camera, the display screen, the one or more memories, and the one or more processors are coupled, the one or more memories are configured to store a computer program, and the computer program is configured to be executed by the processor to implement the method of any one of claims 1-12.

14. A computer readable storage medium having stored thereon a computer program, characterized in that The computer program, when executed by a processor, implements the method of any one of claims 1-12.

15. A computer program product, characterised in that, comprising instructions, which when executed by a processor, implement the method of any one of claims 1-12.

16. A chip system, characterized by comprising one or more processors configured to invoke computer instructions to implement the method of any one of claims 1-12.

Images