System and method for detecting and mitigating roll striping using a secondary camera

Abstract

An example method includes displaying, through a viewfinder, a preview of a video of a scene captured by a primary camera. The method also includes receiving pixel data corresponding to the scene from an image sensor of a secondary camera, where the primary camera and the secondary camera share a common field of view. The method additionally includes determining, based on the pixel data from the secondary camera, a presence or absence of a rolling banding in the preview of the video captured by the primary camera.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application No. 18 / 615,186, filed March 25, 2024, the contents of which are incorporated herein by reference in their entirety. Background Technology

[0003] Many modern computing devices, including mobile phones, personal computers, and tablets, incorporate image capture devices, such as still cameras and / or video cameras. Image capture devices can capture images, such as images of people, animals, landscapes, and / or objects. Image capture devices may include image sensors. Image sensors include multiple light-sensing pixels that measure the intensity of light incident upon them and thereby collectively capture an image of the environment. The light intensity can have a high dynamic range depending on the environment. Summary of the Invention

[0004] This application generally relates to the detection and correction of scrolling stripes in real-time captured video. Scrolling stripes can occur when recording video on a smartphone. It is typically caused by a mismatch between the video's frame rate and the frequency of the ambient lighting being used. This can cause horizontal lines or stripes to appear to move within the video. Existing techniques identify the presence of scrolling stripes and then increase the camera's exposure time to remove and / or reduce them. However, this can reduce dynamic range (e.g., in bright indoor scenes), and users may see bright areas of the image being overexposed. Furthermore, stripe detection based solely on line sums can be unreliable. For example, changes in line sums could be caused by moving objects. However, the frequency of the stripes can be compared to the frequency of ambient light to eliminate false detections of stripes, such as those caused by motion. Some advanced devices (e.g., smartphones) may be equipped with flicker sensors that detect ambient flicker and / or ambient light sensors (ALS) that detect the amount of ambient light. However, ALSs can be defective, resulting in poor image quality. Moreover, many devices may not have an ALS. When a device is configured with an ALS (Alternating Range Sight) to measure ambient light intensity, the ALS exposure time is typically long (e.g., 250 ms). When a device is configured with a scintillation sensor to measure changes in brightness levels caused by flickering light sources, the scintillation sensor exposure time may be short (e.g., less than 1 ms). In some devices, the operation of both the ALS and scintillation sensors can be performed by a single sensor. For example, the ALS sensor and the scintillation sensor can operate on the same hardware but with different exposure time settings.

[0005] In one aspect, a computer-implemented method is provided. The method includes displaying a preview of a video of a scene captured by a main camera through a viewfinder. The method also includes receiving pixel data corresponding to the scene from the image sensor of a secondary camera, wherein the main camera and the secondary camera share a common field of view. The method further includes determining, based on the pixel data from the secondary camera, whether scrolling stripes are present or absent in the preview of the video captured by the main camera.

[0006] In another aspect, a system is provided. The system may include one or more processors. The system may also include a data storage device storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations. These operations may include displaying a preview of a video of a scene captured by a main camera through a viewfinder. These operations may further include receiving pixel data corresponding to the scene from the image sensor of a secondary camera, wherein the main camera and the secondary camera share a common field of view. These operations may further include determining, based on the pixel data from the secondary camera, whether scrolling stripes are present or absent in the preview of the video captured by the main camera.

[0007] In another aspect, a computing device is provided. The device includes a main camera and a secondary camera sharing a common field of view. The device also includes one or more processors and a data storage device storing computer-executable instructions that, when executed by the one or more processors, cause the computing device to perform operations. These operations may include displaying a preview of a video of a scene captured by the main camera through a viewfinder of the computing device. These operations may also include receiving pixel data corresponding to the scene from the image sensor of the secondary camera. These operations may further include determining, based on the pixel data from the secondary camera, whether scrolling stripes are present or absent in the preview of the video captured by the main camera.

[0008] In another aspect, an article of manufacture is provided. The article of manufacture may include a non-transitory computer-readable medium storing program instructions that, when executed by one or more processors of a computing device, cause the computing device to perform operations. These operations may include displaying a preview of a video of a scene captured by a main camera through a viewfinder. These operations may further include receiving pixel data corresponding to the scene from an image sensor of a secondary camera, wherein the main camera and the secondary camera share a common field of view. These operations may further include determining, based on the pixel data from the secondary camera, whether scrolling stripes are present or absent in the preview of the video captured by the main camera.

[0009] On the other hand, a program is provided. When executed by one or more processors of a computing device, the program causes the computing device to perform operations. These operations may include displaying a preview of a video of a scene captured by a main camera through a viewfinder. These operations may also include receiving pixel data corresponding to the scene from the image sensor of a secondary camera, wherein the main camera and the secondary camera share a common field of view. These operations may further include determining, based on the pixel data from the secondary camera, whether scrolling stripes are present or absent in the preview of the video captured by the main camera.

[0010] The foregoing description of the invention is merely illustrative and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0011] Figure 1 These are illustrations of a front view, a right side view, and a rear view of a digital camera device according to an example embodiment.

[0012] Figure 2 This is an example illustration of rolling stripe detection and correction according to an example embodiment.

[0013] Figure 3 This is an example illustration of scrolling stripe detection according to an example embodiment.

[0014] Figure 4 This is another example illustration of scrolling stripe detection according to an example embodiment.

[0015] Figure 5 This is another example illustration of scrolling stripe detection according to an example embodiment.

[0016] Figure 6 This is a block diagram of an example computing device according to an example embodiment.

[0017] Figure 7 This is a flowchart of a method according to an example embodiment. Detailed Implementation

[0018] This document describes exemplary methods, apparatus, and systems. It should be understood that the terms "exemplary" and "illustrated" are used herein to mean "serving as an example, instance, or illustration." Any embodiment or feature described herein as "exemplary" or "illustrated" is not necessarily to be construed as preferred or superior to other embodiments or features. Other embodiments may be utilized and other changes may be made without departing from the scope of the subject matter presented herein.

[0019] Therefore, the exemplary embodiments described herein are not intended to be limiting. As generally described herein and illustrated in the accompanying drawings, aspects of this disclosure can be arranged, replaced, combined, separated, and designed in a wide variety of different configurations, all of which are contemplated herein.

[0020] Furthermore, unless the context otherwise suggests, the features shown in each figure can be used in combination with each other. Therefore, the figures should generally be considered as aspects of one or more overall embodiments, but it should be understood that not all features shown are necessary for every embodiment.

[0021] Overview

[0022] Anti-striping features in telephone cameras require knowledge of the active mains / power line frequency, which can be accurately and reliably calculated from time-series data obtained from an ambient light sensor (ALS). Typically, an ALS is capable of sampling ambient light intensity at extremely high rates. However, some devices may not be equipped with such a sensor. Therefore, image-based methods can be used to obtain the utility frequency. Existing image-based utility frequency detection systems utilize images captured from a currently active main camera (e.g., for previewing and direct capture of image data). In such cases, it may be necessary to additionally configure the exposure time and gain of the main camera to ensure that stripes are not visible in the viewfinder, in order to display acceptable preview quality and acceptable final image quality in a zero-shutter-hysteresis system. Some other methods involve using images from both the main and secondary cameras to determine the incremental difference between the corresponding image data.

[0023] Both single-camera and dual-camera methods with comparisons have significant drawbacks. For example, exposure time and gain must prioritize an acceptable preview of the image from the main camera over successful detection of stripes and mains frequencies. These two objectives often conflict with each other. For instance, mains frequencies can usually be successfully detected when stripes are prominently visible at the camera. However, prominent stripes may not be the desired end result for image capture purposes. Moreover, for example, when mains frequencies are detected due to stripe artifacts in the image captured from the main camera, the stripe artifacts may no longer be visible once a corrected exposure time is set. Therefore, the light source causing the stripe artifacts may no longer be discernible to the camera, resulting in an image that has not yet been corrected for stripes. Thus, determining whether the exposure time constraint used to prevent stripe artifacts can be removed can be challenging. One approach could involve instantaneously releasing the exposure time constraint and checking if stripes are detected again. However, this could degrade the user experience. Furthermore, using incremental comparisons between image data from both cameras could lead to higher computational resources, for example.

[0024] In a dual-camera system, typically one camera is the active streaming camera (called the master camera), while the other is the passive streaming camera (called the secondary camera). The image from the master camera can be displayed as a preview image in the viewfinder. However, the secondary camera image can be used to assist in exposure decisions and image quality assessments based on the master camera. The secondary camera is also less constrained to meet acceptable preview quality and can therefore be set to such low exposure times and gains to prioritize fringe artifact detection. Generally, sacrificing the image quality of the master camera is not a preferred option, but the image from the secondary camera allows for greater flexibility.

[0025] Detecting scrolling stripes involves determining the row sum of pixel intensities across consecutive frames of a video. Row-by-row variations in pixel intensity can be used to detect the presence of stripes. For example, a sine curve can indicate stripes, while a generally linear function indicates the absence of stripes. Furthermore, if stripes are present, their frequency can be determined from the sine curve. Image data from a secondary camera can be used to both detect stripes and determine the frequency of flickering. The frequency of flickering and the frequency of stripes can be compared to determine the amount of stripes and whether mitigation is needed to address visible stripes.

[0026] The severity of observed stripes is inversely proportional to the exposure time. At exposure times close enough to the specific amount required to eliminate the stripes, they may be barely visible in the viewfinder. At lower exposure times (e.g., ~2 / 3 ms), stripes in the flicker statistics may become more pronounced, and this can greatly facilitate the task of detecting rolling stripes. When the ambient brightness level remains constant, the average luminance value in the image will remain constant. However, in the presence of flicker, i.e., when the frequency of the ambient brightness level changes, the period of the sine curve changes.

[0027] As described in this paper, the auxiliary camera can operate at low exposure times (e.g., 3 ms or less) and can detect stripes directly from image data. This approach eliminates the need to compromise and / or adjust the exposure time of the main camera for a few frames of images solely for the purpose of detecting flickering light sources. The lower exposure time in the auxiliary camera also facilitates the detection of flicker / stripes.

[0028] Some methods rely on acquiring images from a primary and secondary camera and determining the incremental difference between the images. For example, an image capture device can capture multiple images of a scene using different exposure times. Frequency analysis can be performed on the difference between the captured images to automatically detect flicker. Based on the detected flicker, the image capture device can adjust the exposure time to capture subsequent images. However, as described in the method presented herein, the primary camera is typically not required to participate in flicker detection. The secondary camera can typically be set to capture stripes at a low exposure time (e.g., approximately 2 ms). This eliminates the need for multiple images at different exposure times and also keeps the preview image unchanged due to ongoing flicker measurements and / or adjustments.

[0029] Once the secondary camera determines that the scrolling stripes are sufficiently significant and / or visible to be mitigated, techniques for removing / reducing the scrolling stripes are applied to the primary camera. The secondary camera can continue to monitor ambient light (e.g., at regular time intervals, or by detecting changes in the determined ambient light frequency) and adjust the primary camera settings as needed. Therefore, users viewing the preview from the primary camera will not see the stripes, and the preview image quality can be of a high standard that can be captured by the primary camera.

[0030] Devices equipped with ALS (Ambient Light Detection and Reduction) can also use ALS data to determine ambient light frequencies. In such cases, the ALS data can be compared with image data from a secondary camera to determine the confidence level of the measurement. Typically, the secondary camera can have a wider field of view than the primary camera, and therefore can be able to detect light sources that contribute to the ambient light brightness of the preview from the primary camera. Therefore, in cases where there is a discrepancy between the ALS data measurement and the image data from the secondary camera, the image data from the secondary camera can be prioritized and relied upon. Furthermore, in such cases, a notification indicating potential errors in the ALS can be generated.

[0031] Example camera system

[0032] As image capture devices such as cameras become increasingly popular, they can be used as standalone hardware devices or integrated into a wide variety of other types of devices. For example, still cameras and video cameras are now commonly included in wireless computing devices (e.g., mobile devices, such as mobile phones), tablet computers, laptop computers, video game interfaces, home automation devices, and even cars and other types of vehicles.

[0033] The physical components of a camera may include one or more apertures through which light enters, one or more recording surfaces for capturing an image represented by light, and lenses positioned in front of each aperture to focus at least a portion of the image onto the recording surface. The apertures may be fixed in size or adjustable. In analog cameras, the recording surface may be photographic film. In digital cameras, the recording surface may include an electronic image sensor (e.g., a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) sensor) to transfer the captured image and / or store it in a data storage unit (e.g., memory).

[0034] One or more shutters may be coupled to or positioned near a lens or recording surface. Each shutter can be in a closed or open position, where it blocks light from reaching the recording surface and allows light to reach the recording surface. The position of each shutter can be controlled by a shutter button. For example, the shutter may be in the closed position by default. When the shutter button is triggered (e.g., pressed), the shutter can change from the closed to the open position over a period of time, which is called a shutter cycle. During the shutter cycle, an image can be captured on the recording surface. At the end of the shutter cycle, the shutter can return to the closed position.

[0035] Alternatively, the shuttering process can be electronic. For example, before the electronic shutter of a CCD image sensor “opens,” the sensor can be reset to remove any residual signal from its photodiodes. While the electronic shutter remains open, the photodiodes can accumulate charge. When the shutter closes or after it closes, this charge can be transferred to a longer-term data storage device. A combination of mechanical and electronic shutter opening and closing is also possible.

[0036] Regardless of type, a shutter can be activated and / or controlled by something other than a shutter button. For example, a shutter can be activated by a soft key, a timer, or some other trigger. In this article, the term "capture" can refer to any mechanical and / or electronic shutter opening and closing process that results in the recording of one or more images, regardless of how the shutter opening and closing process is triggered or controlled.

[0037] The exposure of the captured image can be determined by a combination of the aperture size, the brightness of the light entering the aperture, and the duration of the shutter cycle (also referred to as shutter speed, exposure duration, or exposure time). Additionally, digital and / or analog gains (e.g., based on ISO settings) can be applied to the image, thereby affecting the exposure. In some embodiments, the terms "exposure duration," "exposure time," or "exposure interval" can refer to the shutter speed multiplied by a gain for a given aperture size. Therefore, these terms can be used interchangeably to some extent and should be interpreted as shutter speed, exposure time, and / or any other indicator controlling the amount of signal response resulting from light reaching the recording surface.

[0038] In some implementations or operating modes, a camera can capture one or more still images whenever image capture is triggered. In other implementations or operating modes, a camera can capture video images by continuously capturing images at a specific rate (e.g., 24 frames per second) as long as image capture remains triggered (e.g., when the shutter button is held down). Some cameras, when operating in still image capture mode, can open the shutter when the camera device or application is activated, and the shutter can remain in that position until the camera device or application is deactivated. When the shutter is open, the camera device or application can capture the scene and display a representation of the scene on the viewfinder (sometimes referred to as displaying a "preview frame"). When image capture is triggered, one or more different payload images of the current scene can be captured.

[0039] Cameras (including digital and analog cameras) may include software for controlling one or more camera functions and / or settings, such as aperture size, exposure time, gain, etc. Additionally, some cameras may include software for digitally processing images during or after image capture. While the above description generally refers to cameras, it may be particularly relevant to digital cameras. Digital cameras can be standalone devices (e.g., DSLR cameras) or integrated with other devices.

[0040] Either or both of the front-facing and rear-facing cameras may include an ALS (Ambient Light Sensor) capable of continuously or intermittently determining the ambient brightness of the scene that the camera can capture, or associated with such an ALS. In some devices, the ALS may be used to adjust the display brightness of a screen (e.g., a viewfinder) associated with the camera. When the determined ambient brightness is high, the screen brightness level may be increased to make the screen easier to see. When the determined ambient brightness is low, the screen brightness level may be decreased to similarly make the screen easier to see and potentially save power. Additionally, input from an ambient light sensor may be used to determine the exposure time of the associated camera, or may contribute to this determination.

[0041] Figure 1These are illustrations of a front view, a right side view, and a rear view of a digital camera device 100 according to an example embodiment. The digital camera device 100 may be, for example, a mobile device (e.g., a mobile phone), a tablet computer, or a wearable computing device. However, other embodiments are also possible. The digital camera device 100 may include various elements such as a body 102, a front-facing camera 104, a multi-element display 106, a shutter button 108, and other buttons 110. The digital camera device 100 may further include one or more rear-facing cameras 112, 114. The front-facing camera 104 may be positioned on the side of the body 102 that is typically user-facing during operation or on the same side as the multi-element display 106. The rear-facing cameras 112, 114 may be positioned on the side of the body 102 opposite to the front-facing camera 104. Referring to the cameras as front-facing and rear-facing cameras is arbitrary, and the digital camera device 100 may include multiple cameras positioned on various sides of the body 102.

[0042] The multi-element display 106 may represent a cathode ray tube (CRT) display, a light-emitting diode (LED) display, a liquid crystal display (LCD), a plasma display, or any other type of display known in the art. In some embodiments, the multi-element display 106 may display a digital representation of a current image being captured by the front-facing camera 104 and / or the rear-facing cameras 112, 114, or an image that may be captured by any or both of these cameras, or an image recently captured. Thus, the multi-element display 106 can be used as a viewfinder for any of the cameras. The multi-element display 106 may also support a touchscreen and / or presence-sensitive function that can enable adjustments to settings and / or configurations of any aspect of the digital camera device 100.

[0043] The multi-element display 106 may include additional features related to the camera application. For example, multiple modes may be available to the user, including sports mode, portrait mode, video mode, video bokeh mode, and so on. The camera application may be in camera mode and provide additional features such as a reverse icon for activating a reverse camera view, a trigger button for capturing a preview image, and a photo stream icon for accessing a database of captured images. Furthermore, for example, a magnification slider may be displayed, and the user may move a virtual object along the zoom level slider to select the magnification. In some embodiments, the user may use the multi-element display 106 (also referred to herein as a display screen) to adjust the magnification (e.g., by moving two fingers away from each other in an outward motion on the display screen), and the magnification slider may automatically display the magnification level.

[0044] The forward-facing camera 104 may include an image sensor and associated optical elements, such as lenses. The forward-facing camera 104 may provide zoom capability or may have a fixed focal length. In other embodiments, interchangeable lenses may be used with the forward-facing camera 104. The forward-facing camera 104 may have a variable mechanical aperture and a mechanical and / or electronic shutter. The forward-facing camera 104 may also be configured to capture still images, video images, or both. Further, the forward-facing camera 104 may represent a monocular, stereo, or multi-view camera. Rear-facing cameras 112, 114 may be arranged similarly or differently. Additionally, the forward-facing camera 104, rear-facing cameras 112, 114, or both may be an array of one or more cameras.

[0045] Either or both of the forward camera 104 and the rear cameras 112, 114 may include an illumination component that provides a light field for illuminating a target object or be associated with such an illumination component. For example, the illumination component may (e.g., using one or more LEDs) provide flash or constant illumination of the target object. The illumination component may also be configured to provide a light field that includes one or more of structured light, polarized light, and light with specific spectral content. Other types of light fields known for reconstructing three-dimensional (3D) models from objects are also possible in the context of the embodiments herein.

[0046] In some digital camera devices 100, either or both of the front-facing camera 104 and the rear-facing cameras 112, 114 may include an ambient light sensor or be associated with an ambient light sensor capable of continuously or intermittently determining the ambient brightness of the scene that the camera can capture. In some devices, the ambient light sensor may be used to adjust the display brightness of a screen (e.g., viewfinder) associated with the camera. When the determined ambient brightness is high, the screen brightness level may be increased to make the screen easier to see. When the determined ambient brightness is low, the screen brightness level may be decreased to similarly make the screen easier to see and potentially save power. Additionally, the input from the ambient light sensor may be used to determine the exposure time of the associated camera, or may contribute to this determination.

[0047] Digital camera device 100 can be configured to capture images of a target object (e.g., a subject within a scene) using multi-element display 106 and a front-facing camera 104 or a rear-facing camera 112, 114. The captured images can be multiple still images or video images (e.g., a series of still images captured rapidly in succession, with or without accompanying audio captured by a microphone). Image capture can be triggered by activating shutter button 108, pressing a soft key on multi-element display 106, or by some other mechanism. Depending on the implementation, images can be captured automatically at specific time intervals (e.g., after pressing shutter button 108, after appropriate lighting conditions on the target object, after moving the digital camera device 100 a predetermined distance, or according to a predetermined capture schedule).

[0048] As described above, the functionality of the digital camera device 100 (or another type of digital camera) can be integrated into a computing device (such as a wireless computing device, mobile phone, tablet computer, laptop computer, etc.). For example, a camera controller can be integrated with the digital camera device 100 to control one or more functions of the digital camera device 100.

[0049] Rolling stripe detection and correction

[0050] Figure 2 This is an example illustration of scrolling stripe detection and correction according to an example embodiment. In a dual-camera system, one camera (referred to herein as the primary camera) can be configured for active streaming, while the other camera (referred to herein as the secondary camera) can be configured for passive streaming. As described herein, images from the secondary camera can be used to adjust the exposure and image quality settings of the primary camera. Images from the primary camera can be displayed as a preview. Therefore, image quality degradation in the primary camera's images is undesirable, as it degrades the user experience and may reduce the quality of the captured images. However, images from the secondary camera may not be displayed to the user and can therefore be used to detect scrolling stripes and apply adjustments to the primary camera's settings.

[0051] The computing device 205 is shown having a main camera 215 and an auxiliary camera 220. The main camera 215 can capture video of scene 210, and the auxiliary camera 220 can be used to detect scrolling stripes. A preview of the video of scene 210 captured by the main camera 215 can be displayed in the viewfinder of the computing device 205. Typically, the auxiliary camera 220 can view the same scene as the scene captured by the main camera 215 (e.g., scene 210). For example, the main camera 215 and the auxiliary camera 220 can share a common field of view. As shown, the main camera 215 can be associated with a main field of view 215A, and the auxiliary camera 220 can be associated with a secondary field of view 220A. The shared common field of view can then correspond to the intersection of the main field of view 215A and the secondary field of view 220A. The shared scene 210 can be included within the intersection of the main field of view 215A and the secondary field of view 220A.

[0052] Light source 220B is shown within the secondary field of view 220A but outside the primary field of view 215A. Typically, light source 220B can illuminate scene 210 and cause scrolling stripes. In some embodiments, the secondary field of view 220A can be wider than the primary field of view 215A. This allows the secondary camera 220 to detect ambient light sources (e.g., light source 220B) that may be present in the secondary field of view 220A but may not be in the primary field of view 215A.

[0053] In some embodiments, the secondary camera 220 can typically be set at a low exposure time (e.g., about 2 ms) to capture the rolling stripes. This eliminates the need for multiple images at different exposure times and also ensures that the preview image in the main camera 215 is not altered due to ongoing flicker measurements and / or adjustments.

[0054] At step 1, the rolling stripe detection component 225 of the computing device 205 may be configured to receive pixel data corresponding to scene 210 from the image sensor of the auxiliary camera 220. As used herein, the term "pixel data" generally refers to light intensity data from an image sensor. In some embodiments, pixel data may be aggregated light intensity data (e.g., the row sum of light intensity in rows of a pixel array, the row-by-row average of light intensity, and other such statistical measurements).

[0055] In some embodiments, the scrolling stripe detection component 225 may be configured to determine whether scrolling stripes are present or absent in a preview of a video captured by the main camera 215 based on pixel data from the secondary camera 220.

[0056] In step 2, once it is determined that scrolling stripes exist in the video preview, the scrolling stripe correction component 230 of the computing device 205 can be configured to adjust the exposure time of the main camera 215 to reduce the scrolling stripes in the video preview.

[0057] At step 3, the adjusted parameters of the exposure time can be transmitted to the main camera 215 to reduce and / or eliminate scrolling stripes in future captured frames of the video.

[0058] Rolling stripe detection

[0059] The severity of observed fringes is generally inversely proportional to the exposure time. Moreover, at exposure times close to a specific threshold where fringing correction may be important, the fringes may be barely visible in the viewfinder. For example, this could occur with an exposure time of approximately 7.5 milliseconds (ms) under 120 Hz illumination. Generally, flicker statistics under such conditions (e.g., from a single frame) may be difficult to distinguish from a non-fringed scene. This can also occur when the fringes are reasonably visible (e.g., at an exposure time of 6 ms under 120 Hz illumination).

[0060] Figure 3 This is an example illustration of scrolling stripe detection according to an example embodiment. A screenshot of preview 305 is shown with example bright areas 310 and example dark areas 315. Preview 305 can correspond to an exposure time setting of 7 ms. Graphical representation 320 shows the relationship between brightness values ​​(values ​​displayed along the vertical axis) and positions in preview 305 (values ​​displayed along the horizontal axis). Peaks 325 indicating high brightness values ​​can correspond to example bright areas 310 in preview 305. Similarly, valleys 330 indicating low brightness values ​​can correspond to example dark areas 315 in preview 305. Scrolling stripes can be detected based on the occurrence of multiple peaks and valleys in the graphical representation of the scene's brightness values.

[0061] Figure 4 This is another example illustration of scrolling stripe detection according to an example embodiment. A screenshot of preview 405 is shown with example bright areas 410 and example dark areas 415. Preview 405 can correspond to an exposure time setting of 6 ms. Graphical representation 420 shows the relationship between brightness values ​​(values ​​displayed along the vertical axis) and positions in preview 405 (values ​​displayed along the horizontal axis). Peaks 425 indicating high brightness values ​​can correspond to example bright areas 410 in preview 405. Similarly, valleys 430 indicating low brightness values ​​can correspond to example dark areas 415 in preview 405. Scrolling stripes can be detected based on the occurrence of multiple peaks and valleys in the graphical representation of the scene's brightness values.

[0062] Stripes may be more easily detected at lower exposure time settings (e.g., 2-3 ms). However, while these settings are suitable for stripe detection, they may not be optimal for image capture. Therefore, a secondary camera can be configured to capture the scene at lower exposure time settings to detect stripes. This reduces and / or eliminates the negative impact of stripes on the preview from the primary camera, while allowing the image processor to analyze the image data from the secondary camera and adjust the primary camera settings if stripes are detected.

[0063] Figure 5 This is another example illustration of scrolling stripe detection according to an example embodiment. Graphical representation 500 shows the relationship between brightness values ​​(displayed along the vertical axis) and their positions in a video preview (displayed along the horizontal axis). Preview ( Figure 5 (Not shown) This corresponds to a 2 ms exposure time setting. Multiple peaks of 525 indicate high brightness values ​​corresponding to bright areas in the preview. Similarly, multiple valleys of 530 indicate low brightness values ​​corresponding to dark areas in the preview. The abundance of peaks and valleys in the graphical representation of scene brightness values ​​indicates the presence of scrolling stripes. As shown in the figure, the stripes are more clearly visible in the flicker statistics for a 2 ms exposure time.

[0064] The graphical representation of 300, 400, and 500 indicates that the "average" of the scrolling stripes remains constant. This is because the total brightness value of the scene remains constant. However, the maximum amplitude of each stripe within the stripes is altered. For example, in graphical representation 500, the maximum brightness (e.g., at point A) could be approximately 350,000 units (e.g., lumens), and the minimum brightness (e.g., at point B) could be approximately 200,000 units, while the average brightness could be approximately 270,000 units. Note that the actual units used to measure brightness are irrelevant; however, the relative amplitudes of the different areas are relevant.

[0065] Furthermore, for example, in graphic representation 400, the maximum and minimum values ​​of brightness (e.g., at points C and D, respectively) are lower than Figure 5 The values ​​corresponding to A and B are shown below. However, the average brightness is approximately 270,000 units, which is... Figure 5 The same as in.

[0066] In some embodiments, the graphic representations 300, 400, and 500 can be determined based on sensor row summation input. For example, sensor row summation input can include aggregated light intensity data from two consecutive frames of video. Typically, the aggregated light intensity data can be any type of statistical data (e.g., mean, median, statistical distribution, etc.) indicating row-by-row light intensity. In some embodiments, the aggregated light intensity data can include the corresponding row sums of pixel intensities in two consecutive frames. For example, sensor row summation is a technique for measuring the amount of light received by each row of pixels on an image sensor.

[0067] (For example,) an image sensor for a secondary camera may include multiple light-sensing pixels that measure the intensity of light incident upon them and thereby collectively capture an image of the environment. The light-sensing pixels may be arranged in an array and grouped by position into pixel sensor groups. For example, arrays of pixel sensors may be grouped into pixel sensor groups, where each pixel sensor group includes an array of pixel sensors. Such pixel sensor groups may be further grouped to define regions of interest (ROIs). An image processing circuitry group may be configured so that each image processing circuitry receives pixel information from its corresponding pixel sensor group and is further configured to perform image processing operations on the pixel information to provide processed pixel information during operation of the image sensor.

[0068] In image processing, the input is typically a single-channel image or a set of three-channel images consisting of red, green, and blue (RGB) color channels. Such color images are usually captured using an array of light sensors covered with different color filters (called a color filter array), which are typically arranged in a repeating mosaic pattern. Examples of color filters can be, for example, Bayer filters, modified Bayer filters (such as RGBE, where the green filter is modified to a "turquoise" filter), red-yellow-yellow-blue (RYYB) filters, cyan-yellow-yellow-magenta (CYYM) filters, cyan-yellow-green-magenta (CYGM) filters, various modifications of Bayer filters (e.g., RGBW, where the green filter is modified to a "white" filter), and quad Bayer filters (including 4...). Color filters include 4x blue, 4x red, and 8x green filters; RYYB quad Bayer filters (including 4x blue, 4x red, and 8x yellow filters); nine-in-one pixel filters (including 9x blue, 9x red, and 18x green filters); RCCC filters (monochrome sensors with a red channel for detecting traffic lights, stop signs, etc.); RCCB filters (where the green pixels are transparent), and so on. For example, color filters can also correspond to those used in multispectral sensors. For low-latency and memory-constrained applications, it is sometimes desirable to process the original color mosaic image rather than some RGB equivalent reconstructed from the original image.

[0069] Some embodiments involve determining a pattern of light intensity for two consecutive frames of a video based on aggregated light intensity data (e.g., from a secondary camera). For example, the pattern of light intensity can be based on the difference between corresponding row-by-row sums of light intensity in the respective pixel arrays. For instance, a first row-by-row increment difference can be determined based on the difference between a first sum of pixel intensity in a first row of a first frame and a second sum of pixel intensity in a first row of a second frame (e.g., a frame immediately following the first frame). Similarly, a second row-by-row increment difference can be determined based on the difference between a third sum of pixel intensity in a second row of a first frame and a fourth sum of pixel intensity in a second row of a second frame (e.g., a frame immediately following the first frame). And so on, the row-by-row increment difference for each row of the pixel array can be determined.

[0070] In some embodiments, determining the pattern of light intensity may involve applying a Fourier transform (e.g., a Fast Fourier Transform (FFT)) to such row-by-row incremental differences across consecutive frames. This allows for analysis of aggregated light intensity data against a signal in the frequency domain. Generally, this indicates the absence of scrolling stripes when the incremental differences are substantially similar (e.g., the signal is substantially linear or flat). However, this can indicate the presence of scrolling stripes when the incremental differences are decomposed into curves (e.g., the signal is substantially sinusoidal). In some embodiments, scrolling stripes can be identified by signal processing through a comparison of row sums in two consecutive frames. For example, the pattern of light intensity in two consecutive frames can be determined based on the difference between the corresponding row sums of light intensity in the respective pixel arrays.

[0071] Typically, patterns of light intensity can be used to determine fringe frequencies. For example, as described herein, an FFT can be applied to the incremental differences between the corresponding row-by-row sums of light intensity in the respective pixel array to obtain the signal in the frequency domain. In the frequency domain, the frequency of the signal can be obtained to determine the fringe frequency, and the peak value can be used to determine the fringe amplitude.

[0072] Rolling stripe correction

[0073] Some embodiments involve determining the presence of scrolling bars in a video preview and adjusting the exposure time of the main camera to reduce the scrolling bars in the video preview. In some embodiments, adjusting the exposure time involves determining whether to increase the exposure time of the image capture device. Typically, the exposure time adjustment can be applied frame-by-frame during image capture (e.g., near real-time).

[0074] While existing methods increase exposure time upon detecting the presence of rolling stripes, such automatic correction may not be desirable. For example, such automatic correction could negatively impact the high dynamic range (HDR) of a video. HDR display of an image can expand the range of the user experience when viewing an image. For example, an image of a person in the dark may have a lot of dark colors and low luminance values. Similarly, an image of a sunlit sky may have a lot of bright colors and high luminance values. In some aspects, the ratio of the corresponding luminance values ​​can have a high dynamic range, such as 1:1,000,000. Typically, camera parameters can be adjusted to expand the exposure range to capture more detail in a scene (e.g., capturing detail in both bright and dark areas). Increasing the exposure time may overexpose brighter areas. Therefore, in some embodiments, the determination of whether to increase the exposure time can be based on the determination of whether increasing the exposure time reduces the dynamic range of the video. In some embodiments, once it is determined that increasing the exposure time does not reduce the dynamic range of the video, rolling stripe correction can be applied and the exposure time of the main camera can be increased. Moreover, once it is determined that increasing the exposure time reduces the dynamic range of the video, rolling stripe correction can be omitted, and the exposure time of the main camera can be maintained (e.g., without increasing it).

[0075] For example, a video may include objects of interest, regions of interest, etc. One or more trained machine learning algorithms (e.g., object detection algorithms, face detection algorithms, segmentation algorithms, etc.) can be used to identify one or more regions of interest in an image. In some implementations, user-approved facial recognition algorithms can be applied to identify one or more individuals in an image as potential objects of interest. For example, a user may be able to access their history of preferences, and objects and / or individuals may be identified as of high user interest. Therefore, the determination of whether to increase the exposure time of the image capture device can be based on whether increasing the exposure time negatively impacts the visibility and / or image characteristics of the objects of interest, regions of interest, etc. For example, increasing the exposure time may cause some objects and / or regions of interest to become bright enough, degrading their image quality. Similarly, decreasing the exposure time may cause some objects and / or regions of interest to become dark enough, degrading their image quality.

[0076] In some embodiments, the secondary camera may maintain an exposure time that does not exceed a time threshold. For example, the time threshold may be 3 ms, and the exposure time of the secondary camera may be maintained at 3 ms or less. (See reference...) Figure 5 As described, a lower exposure time can facilitate the detection of rolling stripes that might otherwise be undetectable at higher exposure time settings.

[0077] Scrolling stripe detection and / or correction can be performed at various times during video capture (e.g., at periodic intervals). For example, ambient light measurements from the auxiliary camera can be analyzed at various times to determine if there are changes in the amount of ambient light and / or changes in ambient light sources (e.g., the addition or removal of a light source). Furthermore, the camera system can determine, for example, whether such changes are caused by movement (e.g., device movement, scene changes, etc.). In some embodiments, the camera system can determine whether perceived changes in ambient light measurements are caused by ambient reflections of light from the device. These and other factors can be considered to determine the presence of scrolling stripes and / or to mitigate their presence.

[0078] ALS sensor reliability

[0079] Some devices may be equipped with a flicker sensor and / or an ambient light sensor (ALS) that can continuously or intermittently determine the ambient brightness of the scene that the camera can capture. In some devices, the ALS can be used to adjust the display brightness of a screen associated with the camera (e.g., a viewfinder). When the determined ambient brightness is high, the screen brightness level can be increased to make the screen easier to see. When the determined ambient brightness is low, the screen brightness level can be decreased to similarly make the screen easier to see and potentially save power. Additionally, the input from the ambient light sensor can be used to determine the exposure time of the associated camera, or contribute to this determination. Therefore, determining the reliability of data from the ALS can be useful.

[0080] Some embodiments involve receiving sensor data indicating ambient light from an ALS. Such embodiments involve determining a first frequency of ambient light based on the sensor data. For example, aggregated statistics related to the brightness level of pixels can be determined, and an FFT method can be applied to determine the first frequency of ambient light. Such embodiments also involve determining a second frequency of ambient light based on pixel data from an image sensor of a secondary camera. For example, pixel data from the secondary camera can be analyzed in a manner similar to that from the sensor data from the ALS to determine the second frequency of ambient light. Assuming that the ALS and the secondary camera typically share similar fields of view, the first and second frequencies should be close to each other. Such embodiments further involve comparing the first and second frequencies to determine if a mismatch exists.

[0081] Some embodiments involve determining a mismatch between a first frequency and a second frequency. For example, a comparison of the first and second frequencies can indicate a perceptible difference in values. This can indicate that the measurement of ambient light is inaccurate. Such embodiments involve reducing the confidence level of the ambient light measurement. In some embodiments, an image processing pipeline can be configured to use the confidence level of the ambient light measurement to determine whether certain adjustments (e.g., brightness level, exposure time, etc.) should be applied.

[0082] Some embodiments involve determining that the mismatch between a first frequency and a second frequency exceeds a threshold mismatch value. For example, a comparison of the first and second frequencies can indicate a significant difference in values. Typically, pixel data from a secondary camera may be more reliable than sensor data from the ALS. Therefore, a significant difference in values ​​may indicate a potential malfunction in the ALS. Such embodiments involve providing notification that the ALS may have a potential malfunction.

[0083] Some embodiments involve determining that the mismatch between a first frequency and a second frequency is within a threshold mismatch value. For example, a comparison of the first and second frequencies can indicate a negligible difference in the values. This can indicate that the ambient light measurement is reliable. Such embodiments involve increasing the confidence level of the ambient light measurement.

[0084] Computing device architecture

[0085] Figure 6 This is a block diagram of an example computing device 600 according to an example embodiment. Specifically, Figure 6 The computing device 600 shown can be configured to perform at least one function described herein, including method 700.

[0086] The computing device 600 may include a user interface module 601, a network communication module 602, one or more processors 603, a data storage device 604, one or more cameras 618, one or more sensors 620, and a power system 622, all of which may be linked together via a system bus, network, or other connection mechanism 605.

[0087] User interface module 601 may be operable to send data to and / or receive data from external user input / output devices. For example, user interface module 601 may be configured to send data to and / or receive data from user input devices such as touchscreens, computer mice, keyboards, keypads, touchpads, trackballs, joysticks, voice recognition modules, and / or other similar devices. User interface module 601 may also be configured to provide output to user display devices, such as one or more cathode ray tubes (CRTs), liquid crystal displays, light-emitting diodes (LEDs), displays using digital light processing (DLP) technology, printers, light bulbs, and / or other similar devices, whether now known or later developed. User interface module 601 may also be configured to generate audible output using means such as speakers, speaker jacks, audio output ports, audio output devices, headphones, and / or other similar devices. User interface module 601 may also be configured with one or more haptic devices that can generate haptic output, such as vibrations and / or other outputs detectable by touch and / or physical contact with computing device 600. In some examples, the user interface module 601 may be used to provide a graphical user interface (GUI) to utilize the computing device 600.

[0088] The network communication module 602 may include one or more devices that provide one or more wireless interfaces 607 and / or one or more wired interfaces 608 that can be configured to communicate via a network. The wireless interface 607 may include one or more wireless transmitters, receivers, and / or transceivers, such as Bluetooth™ transceivers, Zigbee® transceivers, Wi-Fi™ transceivers, WiMAX™ transceivers, LTE™ transceivers, and / or other types of wireless transceivers that can be configured to communicate via a wireless network. The wired interface 608 may include one or more wired transmitters, receivers, and / or transceivers, such as Ethernet transceivers, Universal Serial Bus (USB) transceivers, or similar transceivers that can be configured to communicate via twisted-pair, coaxial cable, fiber optic links, or similar physical connections to a wired network.

[0089] In some examples, network communication module 602 may be configured to provide reliable, protected, and / or authenticated communication. For each communication described herein, information may be provided to facilitate reliable communication (e.g., guarantee message delivery), which may be part of message headers and / or message trailers (e.g., packet / message ordering information, encapsulation headers and / or encapsulation trailers, size / time information, and transmission verification information (such as Cyclic Redundancy Check (CRC) and / or parity values)). One or more cryptographic protocols and / or algorithms may be used to ensure communication security (e.g., encode or encrypt communication) and / or decrypt / decode communication, such as, but not limited to, Data Encryption Standard (DES), Advanced Encryption Standard (AES), Rivest-Shamir-Adelman (RSA) algorithm, Diffie-Hellman algorithm, Secure Sockets Protocols (such as Secure Sockets Layer (SSL) or Transport Layer Security (TLS)), and / or Digital Signature Algorithm (DSA). In addition to the protocols and / or algorithms listed in this article, other cryptographic protocols and / or algorithms may be used to secure (and then decrypt / decode) communications.

[0090] One or more processors 603 may include one or more general-purpose processors (e.g., a central processing unit (CPU) and / or one or more special-purpose processors (e.g., a digital signal processor, a tensor processing unit (TPU), a graphics processing unit (GPU), an application-specific integrated circuit, etc.). One or more processors 603 may be configured to execute computer-readable instructions 606 and / or other instructions as described herein contained in data storage device 604.

[0091] Data storage device 604 may include one or more non-transitory computer-readable storage media that can be read and / or accessed by at least one of one or more processors 603. The one or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, or other memory or disk storage devices, that can be integrated wholly or partially with at least one of the one or more processors 603. In some examples, data storage device 604 may be implemented using a single physical device (e.g., a single optical, magnetic, organic, or other memory or disk storage unit), while in other examples, data storage device 604 may be implemented using two or more physical devices.

[0092] Data storage device 604 may include computer-readable instructions 606 and possible additional data. In some examples, data storage device 604 may include storage means required to perform at least a portion of the methods, scenarios, and techniques described herein and / or at least a portion of the functionality of the devices and networks described herein. Specifically, computer-readable instructions 606 may include instructions that, when executed by processor 603, enable computing device 600 to provide some or all of the functions described herein.

[0093] In some embodiments, the computer-readable instructions 606 may include instructions that, when executed by the processor 603, enable the computing device 600 to perform operations. These operations may include displaying a preview of a video of a scene captured by the main camera through a viewfinder. These operations may also include receiving pixel data corresponding to the scene from the image sensor of a secondary camera, wherein the main and secondary cameras share a common field of view. These operations may further include determining, based on the pixel data from the secondary camera, whether scrolling stripes are present or absent in the preview of the video captured by the main camera.

[0094] In some examples, computing device 600 may include a scrolling stripe module 612. The scrolling stripe module 612 may be configured to detect and / or correct scrolling stripes in a video. For example, the scrolling stripe module 612 may be configured to correlate a pattern of light intensity (e.g., based on image data from one or more auxiliary cameras 618) and the amount of flickering (e.g., based on image data from the auxiliary cameras) to determine whether scrolling stripes are present or absent in the video.

[0095] In some examples, computing device 600 may include one or more cameras 618. Camera 618 may include one or more image capturing devices, such as still cameras and / or video cameras, equipped to capture light and record the captured light in one or more images; that is, camera 618 may generate an image of the captured light. The one or more images may be one or more still images and / or one or more images used in video images. Camera 618 may capture light and / or electromagnetic radiation emitted as visible light, infrared radiation, ultraviolet light, and / or as light of one or more other frequencies. Camera 618 may include wide-angle cameras, telephoto cameras, ultra-wide-angle cameras, etc. Furthermore, for example, camera 618 may be a forward-facing or rear-facing camera of the reference computing device 600. Camera 618 may include camera components such as, but not limited to, aperture, shutter, recording surface (e.g., photographic film and / or image sensor), lens, and / or shutter button. The camera components may be at least partially controlled by software executed by one or more processors 603. For illustrative purposes, one of the one or more cameras 618 may be a primary camera capturing video of the scene, and a second camera of the one or more cameras 618 may be an auxiliary camera for detecting scrolling stripes. Typically, the auxiliary camera can view the same scene as the primary camera. In some embodiments, the field of view of the auxiliary camera may be wider than that of the primary camera. This allows the auxiliary camera to detect ambient light sources that may not be within the field of view of the primary camera.

[0096] In some examples, computing device 600 may include one or more sensors 620. Sensors 620 may be configured to measure conditions within computing device 600 and / or conditions in the environment of computing device 600 and provide data about these conditions. For example, sensors 620 may include one or more of the following: (i) sensors for obtaining data about computing device 600, such as, but not limited to, thermometers for measuring the temperature of computing device 600, battery sensors for measuring the charge of one or more batteries in power system 622, and / or other sensors for measuring the condition of computing device 600; (ii) identification sensors for identifying other objects and / or devices, such as, but not limited to, radio frequency identification (RFID) readers, proximity sensors, one-dimensional barcode readers, two-dimensional barcode (e.g., quick response (QR) code) readers, and laser trackers, wherein the identification sensors may be configured to read identifiers, such as RFID tags, barcodes, QR codes, and / or other devices and / or objects configured to be read and at least provide identification information; (iii) sensors for measuring the position and / or movement of computing device 600, etc. Such as, but not limited to, tilt sensors, gyroscopes, accelerometers, Doppler sensors, GPS devices, sonar sensors, radar devices, laser displacement sensors, and compasses; (iv) environmental sensors for obtaining data indicating the environment of computing device 600, such as, but not limited to, infrared sensors, optical sensors, light sensors (e.g., ambient light sensors), biosensors, capacitive sensors, touch sensors, temperature sensors, wireless sensors, radio sensors, motion sensors, microphones, sound sensors, ultrasonic sensors, and / or smoke sensors; and / or (v) force sensors for measuring one or more forces (e.g., inertial forces and / or G-forces) acting on computing device 600, such as, but not limited to, sensors that measure one or more of the following: forces in one or more dimensions, torque, ground forces, frictional forces, and / or zero-moment point (ZMP) sensors that identify the ZMP and / or the location of the ZMP. Many other examples of sensor 620 are also possible.

[0097] Power system 622 may include one or more batteries 624 and / or one or more external power interfaces 626 for supplying power to computing device 600. Each of the one or more batteries 624 may act as a stored power source for computing device 600 when electrically coupled to computing device 600. The one or more batteries 624 of power system 622 may be configured to be portable. Some or all of the one or more batteries 624 may be easily removable from computing device 600. In other examples, some or all of the one or more batteries 624 may be located inside computing device 600 and therefore may not be easily removable from computing device 600. Some or all of the one or more batteries 624 may be rechargeable. For example, rechargeable batteries may be charged via a wired connection between the battery and another power supply device (such as through one or more power supplies located outside computing device 600 and connected to computing device 600 via one or more external power interfaces). In other examples, some or all of the one or more batteries 624 may be non-rechargeable batteries.

[0098] One or more external power interfaces 626 of the power system 622 may include one or more wired power interfaces, such as USB cables and / or power cords, enabling a wired power connection to one or more power supply devices located outside the computing device 600. One or more external power interfaces 626 may include one or more wireless power interfaces, such as Qi wireless chargers, enabling a wireless power connection to one or more external power supply devices. Once a power connection to an external power source is established using one or more external power interfaces 626, the computing device 600 can draw power from the external power source through the established power connection. In some examples, the power system 622 may include associated sensors, such as battery sensors associated with one or more batteries or other types of power sensors.

[0099] One or more external power interfaces 626 of the power system 622 may include one or more wired power interfaces, such as USB cables and / or power cords, enabling a wired power connection to one or more power supply devices located outside the computing device 600. One or more external power interfaces 626 may include one or more wireless power interfaces, such as Qi wireless chargers, enabling a wireless power connection to one or more external power supply devices. Once a power connection to an external power source is established using one or more external power interfaces 626, the computing device 600 can draw power from the external power source through the established power connection. In some examples, the power system 622 may include associated sensors, such as battery sensors associated with one or more batteries or other types of power sensors.

[0100] Example operation method

[0101] Figure 7 This is a flowchart of a method according to an example embodiment. Method 700 may include various blocks or steps. These blocks or steps may be performed individually or in combination. These blocks or steps may be performed in any order and / or in series or in parallel. Further, blocks or steps may be omitted or added to method 700.

[0102] The block of method 700 can be executed by various elements of computing device 600, as referenced. Figure 6 As shown and described.

[0103] Box 710 relates to a preview of the video of the scene captured by the main camera, displayed through the viewfinder.

[0104] Box 720 relates to receiving scene-corresponding pixel data from the image sensor of the secondary camera, where the primary and secondary cameras share a common field of view.

[0105] Box 730 relates to determining whether scrolling stripes are present or absent in a preview of video captured by the main camera, based on pixel data from the secondary camera.

[0106] Some embodiments involve determining the presence of scrolling stripes in a video preview. Such embodiments involve adjusting the exposure time of the main camera to reduce scrolling stripes in the video preview.

[0107] In some embodiments, adjusting the exposure time involves determining whether to increase the exposure time of the main camera.

[0108] In some embodiments, the determination of whether to increase the exposure time can be based on whether increasing the exposure time reduces the dynamic range of the video.

[0109] Some embodiments involve maintaining the exposure time of the main camera when it is determined that increasing the exposure time would reduce the dynamic range of the video.

[0110] Some embodiments involve increasing the exposure time of the main camera once it is determined that increasing the exposure time does not reduce the dynamic range of the video.

[0111] In some embodiments, the auxiliary camera may maintain an exposure time that does not exceed a time threshold.

[0112] Some embodiments involve receiving sensor data indicating ambient light from an ambient light sensor (ALS). Such embodiments involve determining a first frequency of ambient light based on the sensor data. Such embodiments also involve determining a second frequency of ambient light based on pixel data from an image sensor of a secondary camera. Such embodiments further involve comparing the first and second frequencies to determine if a mismatch exists.

[0113] Some embodiments involve determining that a mismatch exists between a first frequency and a second frequency. Such embodiments involve reducing the confidence level of ambient light measurements.

[0114] Some embodiments involve determining that the mismatch between a first frequency and a second frequency is within a threshold mismatch value. Such embodiments involve increasing the confidence level of the measurement of ambient light.

[0115] Some embodiments involve determining that a mismatch between a first frequency and a second frequency exceeds a threshold mismatch value. Such embodiments involve providing notification that the ALS may have a potential failure.

[0116] In some embodiments, the main camera and the secondary camera may be components of a mobile device.

[0117] In some embodiments, the main camera and the auxiliary camera may be located on the same side of the mobile device.

[0118] The specific arrangements shown in the accompanying drawings should not be considered limiting. It should be understood that other embodiments may include more or fewer of each element shown in the given drawings. Furthermore, some of the elements shown may be combined or omitted. Even further, illustrative embodiments may include elements not shown in the figures.

[0119] The steps or blocks representing the processing of information may correspond to a circuit system that can be configured to perform a specific logical function of the methods or techniques described herein. Alternatively or additionally, the steps or blocks representing the processing of information may correspond to a module, segment, or portion of program code (including associated data). The program code may include one or more instructions that can be executed by a processor to implement a specific logical function or action in the method or technique. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device including a disk, hard disk drive, or other storage media.

[0120] Computer-readable media may also include non-transitory computer-readable media, such as computer-readable media that store data for a short period of time, such as register memory, processor cache, and random access memory (RAM). Computer-readable media may also include non-transitory computer-readable media that store program code and / or data for a long period of time. Therefore, computer-readable media may include auxiliary or persistent long-term storage devices, such as read-only memory (ROM), optical discs or magnetic disks, and optical disc read-only memory (CD-ROM). Computer-readable media may also be any other volatile or non-volatile storage system. Computer-readable media can be considered, for example, computer-readable storage media or tangible storage devices.

[0121] While various examples and embodiments have been disclosed, other examples and embodiments will be apparent to those skilled in the art. The various examples and embodiments disclosed are for illustrative purposes and are not intended to be limiting, wherein the true scope is indicated by the appended claims.

Claims

1. A computer-implemented method, comprising: A preview of the video of the scene captured by the main camera is displayed through the viewfinder; The main camera receives pixel data corresponding to the scene from the image sensor of the auxiliary camera, wherein the main camera and the auxiliary camera share a common field of view; as well as The presence or absence of scrolling stripes in the preview of the video captured by the main camera is determined based on the pixel data from the auxiliary camera.

2. The computer-implemented method as described in claim 1, further comprising: It was determined that the scrolling stripes existed in the preview of the video; as well as Adjust the exposure time of the main camera to reduce the scrolling stripes in the preview of the video.

3. The computer-implemented method as described in claim 2, wherein, The adjustment of the exposure time includes: Determine whether to increase the exposure time of the main camera.

4. The computer-implemented method as described in claim 3, wherein, Determining whether to increase the exposure time is based on determining whether increasing the exposure time reduces the dynamic range of the video.

5. The computer-implemented method as described in claim 4, further comprising: While determining that the increase in exposure time reduces the dynamic range of the video, the exposure time of the main camera is maintained.

6. The computer-implemented method of claim 4, further comprising: The exposure time of the main camera is increased when it is determined that increasing the exposure time does not reduce the dynamic range of the video.

7. The computer-implemented method according to any one of claims 1 to 6, wherein, The auxiliary camera is maintained at an exposure time that does not exceed a time threshold.

8. The computer-implemented method as described in any one of claims 1 to 7, further comprising: Receive sensor data indicating ambient light from the ambient light sensor ALS; The first frequency of the ambient light is determined based on the sensor data; The second frequency of the ambient light is determined based on the pixel data from the image sensor of the auxiliary camera; as well as The first frequency and the second frequency are compared to determine if there is a mismatch.

9. The computer-implemented method of claim 8, further comprising: It is determined that a mismatch exists between the first frequency and the second frequency; as well as Reduce the confidence level of the ambient light measurement.

10. The computer-implemented method of claim 8, further comprising: Determine that the mismatch between the first frequency and the second frequency is within a threshold mismatch value; as well as Increase the confidence level of the ambient light measurement.

11. The computer-implemented method of claim 8, further comprising: It is determined that the mismatch between the first frequency and the second frequency exceeds a threshold mismatch value; as well as Provide notification that the ALS may have potential faults.

12. The computer-implemented method according to any one of claims 1 to 11, wherein, The main camera and the auxiliary camera are components of the mobile device.

13. The computer-implemented method as described in claim 12, wherein, The main camera and the auxiliary camera are located on the same side of the mobile device.

14. A computing device, comprising: A main camera and auxiliary cameras sharing a common field of view; One or more processors; as well as A data storage device, wherein computer-executable instructions are stored on the data storage device, and the computer-executable instructions, when executed by the one or more processors, cause the computing device to perform functions, the functions including: A preview of the video of the scene captured by the main camera is displayed through the viewfinder of the computing device; Receive pixel data corresponding to the scene from the image sensor of the auxiliary camera; and The presence or absence of scrolling stripes in the preview of the video captured by the main camera is determined based on the pixel data from the auxiliary camera.

15. The computing device of claim 14, wherein, The functions include: It was determined that the scrolling stripes existed in the preview of the video; and Adjust the exposure time of the main camera to reduce the scrolling stripes in the preview of the video.

16. The computing device of claim 15, wherein, The adjustment of the exposure time is based on determining whether increasing the exposure time reduces the dynamic range of the video.

17. The computing device according to any one of claims 14 to 16, wherein, The auxiliary camera is maintained at an exposure time that does not exceed a time threshold.

18. The computing device as claimed in claims 14 to 17, wherein, The functions include: Receive sensor data indicating ambient light from the ambient light sensor ALS; The first frequency of the ambient light is determined based on the sensor data; The second frequency of the ambient light is determined based on the pixel data from the image sensor of the auxiliary camera; Determining that the mismatch between the first frequency and the second frequency exceeds a threshold mismatch value; and Provide notification that the ALS may have potential faults.

19. The computing device according to any one of claims 14 to 18, wherein, The main camera and the auxiliary camera are located on the same side of the computing device.

20. An article comprising: One or more non-transitory computer-readable media storing computer-readable instructions that, when executed by one or more processors of a computing device, cause the computing device to perform functions including the computer-implemented method as claimed in any one of claims 1 to 13.

21. A program, when executed by one or more processors of a computing device, causes the computing device to perform functions including the computer-implemented method as claimed in any one of claims 1 to 13.

Images