Overlay video frames to enhance image brightness

Abstract

A method for enhancing the brightness of video frames includes: generating, by a processor, a brightness histogram for each video frame in a set of video frames; for each video frame in the set of video frames, determining corresponding pixels that match a predetermined darkest percentage of pixels and corresponding pixels that match a predetermined brightest percentage of pixels based on the brightness histogram of the video frame; and applying a linear correction to each video frame in the set of video frames to provide brightness enhancement, the brightness enhancement modifying at least one of the following group: modifying a first brightness corresponding to the predetermined darkest percentage of pixels to be less than a darkness threshold, modifying a second brightness corresponding to the predetermined brightest percentage of pixels to be greater than a brightness threshold, and a combination of both.

Description

[0001] Cross - reference to related applications

[0002] This application is a divisional application of the invention patent application with the application date of August 6, 2020, application number 202080077751.7, and invention title "Overlaying Video Frames to Enhance Image Brightness". It claims the priority of US Patent Application No. 16 / 712,715 titled "Overlaying Video Frames to Enhance Image Brightness" filed on December 12, 2019, the entire content of which is incorporated herein by reference. Background Art

[0003] Video calls in dark environments may result in insufficient video brightness, making it difficult to see the objects in the video. Manually increasing the exposure of the display may cause the faces of the objects to appear overexposed. Using computationally expensive hardware or software solutions to correct the brightness may reduce the frame rate of the streaming video due to the computational burden.

[0004] The background art provided herein is for the purpose of generally presenting the context of the present disclosure. The work of the inventors of the present invention is described in this background art section, as well as aspects described that may not qualify as prior art at the time of filing, neither explicitly nor implicitly admitted as prior art to the present disclosure. Summary of the Invention

[0005] Embodiments generally relate to a method of enhancing the brightness of video frames in a video during streaming. The method includes determining by a processor that a first video frame has low brightness. The method further includes applying, by the processor, a linear correction in the first video frame to provide a first brightness enhancement that modifies a first brightness of a predetermined darkest pixel percentage to be less than a darkness threshold, or modifies a second brightness of a predetermined brightest pixel percentage to be greater than a brightness threshold. The method may further include averaging, by the processor, the first brightness enhancement of the first video frame using a set of brightness enhancements corresponding to prior video frames to generate a modified video.

[0006] In some embodiments, the method further includes: by a graphics processing unit different from the processor

[0007] (The GPU) downsizes the first video frame, where the first video frame is stored in the GPU memory of the GPU and where the GPU memory is not accessible by the processor; transfers the downsized first video frame from the GPU to a computer processing memory accessible by the processor; and generates, by the processor, a luminance histogram of the downsized first video frame, where linear correction is applied to the first video frame by the GPU. In some embodiments, the number of video frames in the set of video frames being averaged is based on at least one of the frame rate of the video or the input luminance of the first video frame. In some embodiments, it is determined that the first video frame has low luminance based on at least one of the ambient light sensor value not reaching a first threshold or the exponentially moving average frame luminance not reaching a second threshold. In some embodiments, a first luminance enhancement of the first video frame is averaged with a set of luminance enhancements corresponding to prior video frames to produce a modified video. In some embodiments, the video frames are captured by a camera coupled to the processor, and the method further includes determining, by the processor, whether the modified video meets a luminance threshold, in response to determining that the modified video does not meet the luminance threshold, changing one or more camera settings by the processor, and in response to determining that the modified video does meet the luminance threshold, displaying the modified video. In some embodiments, changing one or more camera settings includes increasing the exposure time or modifying the sensitivity of the image sensor, at least one of them. In some embodiments, the method further includes applying contrast limited adaptive histogram equalization (CLAHE) to the first video frame. In some embodiments, applying CLAHE to the first video frame includes: (a) applying CLAHE to the luminance channel of the first video frame, (b) applying CLAHE to each chrominance channel of the first video frame, and (c) mixing the results from (a) and (b). In some embodiments, the method further includes determining the total luminance of the first video frame by calculating the square root of the sum of the average luminance of a set of frames, where each frame in the set of frames is squared, determining a multiplier as 0.5 divided by the total luminance, and determining a final output value by multiplying the multiplier by the square root of the sum of the squares of the input values of the first video frame. In some embodiments, determining the total luminance includes determining that a first total luminance exceeds a threshold, in response to the first total luminance exceeding the threshold, removing a second frame from the set of frames, and calculating a second total luminance.

[0008] A non - transitory computer - readable medium may include instructions stored thereon that, when executed by one or more computers, cause the one or more computers to perform operations, the operations including: determining that a first video frame has low luminance and applying linear correction to the first video frame to provide a first luminance enhancement that modifies the first luminance of a predetermined darkest pixel percentage to be less than a darkness threshold or modifies the second luminance of a predetermined brightest pixel percentage to be greater than a luminance threshold.

[0009] In some embodiments, the operations further include: downscaling the first video frame by a graphics processing unit (GPU) different from the processor, where the first video frame is stored in the GPU memory of the GPU and where the GPU memory is not accessible by the processor; transferring the downscaled first video frame by the GPU to a computer processing memory accessible by the processor; and generating, by the processor, a luminance histogram of the downscaled first video frame, where a linear correction is applied to the first video frame by the GPU. In some embodiments, the number of video frames in the set of video frames to be averaged is based on at least one of the frame rate of the video or the input luminance of the first video frame. In some embodiments, it is determined that the first video frame has low luminance based on at least one of the ambient light sensor value not reaching a first threshold or the exponentially weighted moving average frame luminance not reaching a second threshold. In some embodiments, the operations further include averaging a first luminance enhancement of the first video frame with a set of luminance enhancements corresponding to prior video frames to produce a modified video.

[0010] The system may include one or more processors; and a memory storing instructions executable by the one or more processors, the instructions including: determining that a first video frame has low luminance and applying a linear correction to the first video frame to provide a first luminance enhancement that modifies a first luminance of a predetermined darkest pixel percentage to be less than a darkness threshold or modifies a second luminance of a predetermined brightest pixel percentage to be greater than a luminance threshold.

[0011] In some embodiments, the instructions further include: downscaling the first video frame by a graphics processing unit (GPU) different from the processor, where the first video frame is stored in the GPU memory of the GPU and where the GPU memory is not accessible by the processor; transferring the downscaled first video frame by the GPU to a computer processing memory accessible by the processor; and generating, by the processor, a luminance histogram of the downscaled first video frame, where a linear correction is applied to the first video frame by the GPU. In some embodiments, the number of video frames in the set of video frames to be averaged is based on at least one of the frame rate of the video or the input luminance of the first video frame. In some embodiments, it is determined that the first video frame has low luminance based on at least one of the ambient light sensor value not reaching a first threshold or the exponentially weighted moving average frame luminance not reaching a second threshold. In some embodiments, the instructions further include averaging the first luminance enhancement of the first video frame with a set of luminance enhancements corresponding to prior video frames to produce a modified video. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] This patent or application document contains at least one color drawing. The Patent Office will provide a copy of the color drawing of this patent or patent application publication on the basis of a request and payment of the necessary fees.

[0013] This disclosure is illustrated by way of example and not limitation in the accompanying drawings, in which like reference numerals are used to refer to like elements.

[0014] Figure 1 A block diagram showing an example environment for enhancing the brightness of a video frame according to some embodiments.

[0015] Figure 2 A block diagram showing an exemplary computing device for enhancing the brightness of a video frame according to some embodiments.

[0016] Figure 3A An example user interface enabling a low-light mode according to some embodiments is shown.

[0017] Figure 3B An example of a user interface once the low-light mode is enabled according to some embodiments is shown.

[0018] Figure 4A A flowchart showing an example method of applying contrast-limited adaptive histogram equalization (CLAHE) according to some embodiments.

[0019] Figure 4B Description of using according to some embodiments Figure 4A The corresponding video frame changed by the CLAHE correction described in.

[0020] Figure 5 A flowchart showing an example method for enhancing the brightness of a video frame according to some embodiments.

[0021] Figure 6 A flowchart showing an example method of obtaining user consent to enable the low-light mode according to some embodiments. Detailed Description

[0022] To address the problem of providing high-quality video in low-light settings, techniques for enhancing the brightness of video frames are described herein. First, it is determined whether the user consents to the low-light mode. If the user does not consent to the low-light mode, the method waits until the user consents (if at all). If the user does consent to the low-light mode, the video application analyzes the video frame to determine that the first video frame has low brightness. A correction is performed on the first video frame to produce a modified video. For example, linear correction, contrast-limited adaptive histogram equalization correction, or total brightness determination is applied to provide brightness enhancement in the first video frame.

[0023] The video application determines whether the modified video meets a brightness threshold. If the modified video meets the brightness threshold, then subsequent video frames are analyzed to determine whether they have low brightness. If the modified video fails to meet the brightness threshold, then one or more camera settings are changed.

[0024] The various embodiments described below have several advantages. First, brightness enhancement avoids overexposure of objects in the video or a reduction in the frame rate by analyzing the distribution of brightness values rather than using an average brightness value. Second, the techniques described below include averaging a first brightness enhancement associated with a first video frame with a set of brightness enhancements corresponding to prior video frames to generate a modified video with reduced noise, where the noise results from amplification during the enhancement of the video frames.

[0025] Example environment

[0026] Figure 1 A block diagram of an example environment 100 in which video frames are brightened is shown. The illustrated environment 100 includes a video server 101, user devices 115a, 115n, and a network 105. Users 125a, 125n may be associated with the respective user devices 115a, 115n. In some embodiments, the environment 100 may include Figure 1 other servers or devices not shown, or may not include the video server 101. In Figure 1 and the remaining figures, the letter following a reference number (e.g., "115a") refers to a reference to an element having that particular reference number. A reference number in the text without a following letter, such as "115", refers to a general reference to an embodiment of an element with that reference number.

[0027] The video server 101 may include a processor, a memory, and network communication hardware. In some embodiments, the video server 101 is a hardware server. The video server 101 is communicatively coupled to the network 105 via a signal line 102. The signal line 102 may be a wired connection, such as Ethernet, coaxial cable, fiber optic cable, etc., or a wireless connection, such as Wi-Fi, Bluetooth, or other wireless technologies. In some embodiments, the video server 101 sends data to and receives data from one or more user devices 115a, 115n via the network 105. The video server 101 may include a video application 103a and a database 199.

[0028] The video application 103a may include operative code and routines for enhancing the brightness of video frames and / or transmitting the modified video to another user device 115n. In some embodiments, the video application 103a may be implemented using hardware including a central processing unit (CPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), any other type of processor, or a combination thereof. In some embodiments, the video application 103a may be implemented using a combination of hardware and software.

[0029] The database 199 may store videos, which include the modified videos received from the user devices 115. The videos are stored when the user associated with the client device that generates the modified videos provides permission to store the videos. The database 199 may store the videos that are indexed and associated with the identity of the users 125 of the mobile devices 115. For example, the videos may be indexed, where the index is associated with metadata that describes the user 125 as a member of a social network, and the social network includes links to the profiles of the members connected to the social network. The database 199 may also store social network data associated with the user 125, user preferences of the user 125, and the like.

[0030] In some embodiments, the video application 103 may perform facial recognition on video frames. In cases where the systems and methods discussed herein may collect or use personal information about users (such as user data, information about the user's social network, storage and analysis of videos by the video application 103, etc.), the users are provided with the opportunity to control whether personal information is collected, whether personal information is stored, whether personal information is used, whether images or videos are analyzed, and how the user's information is collected, stored, and used. That is, the systems and methods discussed herein may collect, store, and / or use user personal information only when explicit authorization is received from the relevant users. For example, the users are provided with control over whether a program or feature collects user information about that particular user or other users associated with the program or feature. One or more options are presented to each user for whom personal information is to be collected to allow control over the collection of information related to that user, thereby providing permission or authorization regarding whether information is collected and regarding which portions of the information are to be collected. For example, one or more such control options may be provided to the users via a communication network. In addition, before storing or using certain data, the data may be processed in one or more ways to remove personally identifiable information.

[0031] The user device 115 can be a computing device including a memory and a hardware processor. For example, the user device 115 can include a desktop computer, a mobile device, a tablet computer, a mobile phone, a wearable device, a head-mounted display, a mobile email device, a portable gaming console, a portable music player, a reader device, or another electronic device capable of accessing the network 105.

[0032] In the illustrated implementation, the user device 115a is coupled to the network 105 via the signal line 108, and the user device 115n is coupled to the network 105 via the signal line 110. The signal lines 108 and 110 can be wired connections such as Ethernet, coaxial cable, fiber optic cable, etc., or wireless connections such as Wi-Fi, Bluetooth, or other wireless technologies. The user devices 115a, 115n are accessed by users 125a, 125n respectively. Figure 1 The user devices 115a, 115n in [description] are used as examples. Although Figure 1 Two user devices 115a and 115n are shown, but the present disclosure is applicable to system architectures having one or more user devices 115.

[0033] In some embodiments, the user device 115 can be a wearable device worn by the user 125. For example, the user device 115a is included as part of a clip (such as a wristband), part of jewelry, or part of a pair of glasses. In another example, the user device 115a can be a smartwatch. The video application 103b can generate video frames with enhanced brightness streamed from the user device 115a.

[0034] In some embodiments, the user device 115a can include a video application 103b that generates video frames with enhanced brightness for video calls. The video application 103b can determine that a first video frame has low brightness, apply a correction for brightness enhancement to the first video frame, average the brightness enhancement of the first video frame using a set of brightness enhancements corresponding to prior video frames to generate a modified video, and send the modified video with enhanced brightness to the user device 115. The video application 103b can repeat this process for additional frames from a series of video frames (e.g., captured by the user device 115a during a video call).

[0035] During a video call over the network 105, the user device 115a can send a video stream including, for example, the set of video frames, directly or via the video server 101, to the user device 115n. The user device 115n can include a video application 103c that displays the video stream.

[0036] In the illustrated implementation, entities of environment 100 are communicatively coupled via network 105. Network 105 can be of a conventional wired or wireless type and can have a variety of different configurations, including star configuration, token ring configuration, or other configurations. Additionally, network 105 can include a local area network (LAN), a wide area network (WAN) (e.g., the Internet), and / or other interconnected data paths through which multiple devices can communicate. In some embodiments, network 105 can be a peer-to-peer network. Network 105 can also be coupled to or include a portion of a telecommunications network for sending data in a variety of different communication protocols. In some embodiments, network 105 includes a communication network defined by IEEE 902.11, a wireless local area network (WLAN), computer communication, or a cellular communication network for sending and receiving data, the data including via short message service (SMS), multimedia message service (MMS), hypertext transfer protocol (HTTP), direct data connection, email, etc. Although Figure 1 one network 105 is shown coupled to user device 115 and video server 101, in practice one or more networks 105 can be coupled to these entities.

[0037] Exemplary computing device

[0038] Figure 2 A block diagram of an exemplary computing device 200 for enhancing the brightness of a video frame is shown. Computing device 200 can be user device 115 or video server 101. Computing device 200 can include a processor 235, a memory 237, a graphics processing unit (GPU) 239, a communication unit 241, a camera 243, a display 245, and a storage device 247. Depending on the type of computing device 200, additional components can be present, or some of the previous components can be omitted. Video application 103 can be stored in memory 237. In some embodiments, computing device 200 can include other components not listed herein, such as a battery, etc. The components of computing device 200 can be communicatively coupled via a bus 218. Bus 218 can be a communication bus that conveys signals between various parts of computing device 200.

[0039] Processor 235 includes an arithmetic logic unit, a microprocessor, a general-purpose controller, or some other processor array that performs calculations and provides instructions to a display device. Processor 235 processes data and can include various computing architectures, including a complex instruction set computer (CISC) architecture, a reduced instruction set computer (RISC) architecture, or an architecture that implements a combination of instruction sets. Although Figure 2A single processor 235 is shown, but multiple processors 235 may be included. In different embodiments, processor 235 may be a single-core processor or a multi-core processor. Other processors (e.g., graphics processing units), operating systems, sensors, displays, and / or physical configurations may be part of computing device 200. Processor 235 is coupled to bus 220 to communicate with other components via signal lines 222.

[0040] Memory 237 stores instructions and / or data that can be executed by processor 235. The instructions may include code and / or routines for performing the techniques described herein. Memory 237 may be a dynamic random access memory (DRAM) device, a static RAM, or some other memory device. In some embodiments, memory 237 also includes non-volatile memory, such as a static random access memory (SRAM) device or flash memory, or similar permanent storage devices and media, including hard disk drives, compact disc read-only memory (CD-ROM) devices, DVD-ROM devices, DVD-RAM devices, DVD-RW devices, flash memory devices, or some other mass storage device for storing information on a more permanent basis. Memory 237 includes code and routines operable to execute video application program 103, which will be described in more detail below. Memory 237 is coupled to bus 220 to communicate with other components via signal lines 224.

[0041] The graphics processing unit (GPU) 239 may include hardware operable to downscale video frames. In some embodiments, the GPU 239 includes GPU memory, such as a frame buffer, that stores instructions and / or data executable by the GPU 239. The instructions may include code and / or routines for downscaling video frames. In some embodiments, the GPU 239 produces a downsampled video frame by sampling individual pixels in the video frame. In some embodiments, the original video frame and the corresponding downscaled video frame are stored in the memory of the GPU 239. The GPU 239 may transfer the downscaled video frame to the memory 237 associated with the processor 239. In some implementations, the GPU 239 may include a large number of processing cores that can execute operations in parallel, such as 256 cores, 1000 cores, etc. For example, downscaling of a video frame can be performed by dividing the video frame into multiple regions and using separate cores of the GPU to downscale each region. Such operations can provide downscaled video frames at a rate that allows processing of streamed video (e.g., video that is captured, modified, and streamed) from a client device including the GPU 239 over a network to other devices. In some implementations, the GPU 239 may be coupled to the processor 235 via a dedicated communication channel and may be isolated from other components of the device 200. In some embodiments, the GPU 239 may be coupled to the bus 220 to communicate with other components via signal lines 220.

[0042] The communication unit 241 sends data to and receives data from at least one of the user device 115 and the video server 101 according to the location where the video application 103 may be executed. In some embodiments, the communication unit 241 includes a wireless transceiver for exchanging data with the user device 115, the video server 101, or other communication channels using one or more wireless communication methods, including IEEE 802.11, IEEE 802.16, Bluetooth, or other suitable wireless communication methods. The communication unit 241 is coupled to the bus 220 to communicate with other components via signal lines 226.

[0043] In some embodiments, the communication unit 241 includes a cellular communication transceiver for sending and receiving data over a cellular communication network, including via Short Message Service (SMS), Multimedia Messaging Service (MMS), Hypertext Transfer Protocol (HTTP), direct data connection, email, or other suitable types of electronic communication. In some embodiments, the communication unit 241 includes a wired port and / or a wireless transceiver. The communication unit 241 also provides other conventional connections to the network 105 for distributing files and / or media objects using standard network protocols, including but not limited to User Datagram Protocol (UDP), TCP / IP, HTTP, HTTP Secure (HTTPS), Simple Mail Transfer Protocol (SMTP), SPDY, Quick UDP Internet Connection (QUIC), and the like.

[0044] The camera 243 may include hardware operable to capture video frames. For example, the camera 243 may receive instructions from the user interface module 206 to begin capturing video frames of a video call. The camera 243 may include one or more ambient light sensors, image sensors such as CMOS sensors, depth sensors (such as infrared sensors, time-of-flight sensors, etc.), and / or other types of sensors. The ambient light sensor may be a photodetector for detecting the amount of ambient light present in the environment. In some implementations, the camera 243 may include multiple lenses or other image capture units. The depth sensor may capture depth data that indicates the depth (distance) of one or more pixels of the captured image or video from the camera. The camera 243 may receive instructions from the video application 103 to perform at least one of increasing the exposure time or modifying the sensitivity of the image sensor. For example, the camera 243 may modify the image sensor by modifying the ISO setting. The camera 243 is coupled to the bus 220 for communicating with other components via the signal line 228.

[0045] The display 245 may include hardware operable to display graphical data received from the video application 103. For example, the display 245 may present graphics to display video frames of a video. The display 245 may be any type of display, such as a liquid crystal display (LCD), OLED, etc. In some embodiments, the display 245 may be a projection screen. In some embodiments, for example, when the device 243 is an enhanced physical device, the display 245 may be a stereoscopic display. The display 245 is coupled to the bus 220 to communicate with other components via the signal line 230.

[0046] The storage device 247 may be a non-transitory computer-readable storage medium storing data providing the functions described herein. In an embodiment where the computing device 200 is the video server 101, the storage device 247 may include Figure 1The database 199 therein. The storage device 247 can be a DRAM device, an SRAM device, a flash memory, or some other storage device. In some embodiments, the storage device 247 also includes a non-volatile memory or a similar permanent storage device and medium, including a hard disk drive, a CD-ROM device, a DVD-ROM device, a DVD-RAM device, a DVD-RW device, a flash memory device, or some other mass storage device for permanently storing information. The storage device 247 is coupled to the bus 220 for communicating with other components via signal lines 232.

[0047] The video application 103 can include a user interface module 202, a detection module 204, and a brightness module 204.

[0048] The user interface module 202 generates graphic data to display the user interface. In some embodiments, the user interface module 202 includes a set of instructions executable by the processor 235 to generate graphic data. In some embodiments, the user interface module 202 is stored in the memory 237 of the computing device 200 and can be accessed and executed by the processor 235.

[0049] In some embodiments, the user interface module 202 generates graphic data to display a user interface having options for initiating a video call and modifying settings associated with the call. The settings can include an option for a low-light mode that enhances the brightness of video frames of the video when enabled. The default setting can disable the low-light mode. In some embodiments, the user can enable the low-light mode as a setting applied to all video calls. In some embodiments, the low-light mode can be a setting that can be enabled during a specific video call.

[0050] Go to Figure 3A , an example user interface 300 is shown, which includes a low-light mode option 305 that can be enabled by selecting the "Enable" option, or the low-light mode option 305 can be cancelled by selecting the "Do not enable" option. In this example, the low-light mode option 305 is made available before activating the video call, but other options are also possible. For example, the user interface module 202 can display a settings page with different video application options, including enabling the low-light mode. In some embodiments, once the video call has started and the low-light mode is enabled, the low-light mode cannot be automatically turned off to avoid excessive on-off cycles of the low-light mode.

[0051] Go to Figure 3B , an example of the user interface 350 is shown once the low-light mode is enabled. In this example, the low-light mode option 355 notifies the user that the low-light mode is enabled. The user can change the low-light mode by going to the settings.

[0052] The detection module 204 determines whether a video frame in the video has low brightness. In some embodiments, the detection module 204 includes a set of instructions executable by the processor 235 to determine low brightness in the video. In some embodiments, the detection module 204 is stored in the memory 237 of the computing device 200 and can be accessed and executed by the processor 235.

[0053] In some embodiments, the detection module 204 may check for user consent provided by the user before determining that a video frame in the video has low brightness. In some embodiments where the low-light mode is not enabled, the detection module 204 may determine that a video frame in the video has low brightness and instruct the user interface module 202 to provide the user with a notification that the video has low brightness and recommend that the user enable the low-light mode. In some embodiments, if the user cancels the notification, the user interface module 202 may increase an exponential backoff, where after detecting low brightness, as the user continues to not enable the low-light mode, the number of days the user interface module 202 displays the notification increases. For example, if the user does not enable the low-light mode, the user interface module 202 waits one day to display the notification again, then 5 days, 25 days, 125 days, etc.

[0054] The detection module 204 may determine that a video frame in the video has low brightness based on one or more of an ambient light sensor value received from an ambient light sensor associated with the camera 243 or an exponential moving average frame brightness that does not meet a threshold. In some embodiments, if the ambient light sensor value does not meet a first threshold and the average frame brightness does not meet a second threshold, the detection module 204 may determine that the video frame has low brightness. The average frame brightness is defined as the average brightness of the pixels in the frame. For example, if the ambient sensor value is less than 1 lumen ambient light sensor value and the average frame brightness is less than 30%, the detection module 204 may determine that the video is in low brightness.

[0055] In some embodiments, the detection module 204 does not analyze every frame of the video to save power and / or battery life. Instead, the detection module 204 may analyze every predetermined number of video frames, such as every 30th frame. In the case of measuring the frame brightness for every 30th frame, the detection module 204 detects the exponential moving average of the brightness of every 30th frame. In some embodiments, once low brightness is detected, the detection module 204 may begin analyzing the frame brightness of every frame.

[0056] In some embodiments, once the detection module 204 determines that a video frame has low brightness, the detection module 204 determines whether the user has provided consent to automatically provide brightness enhancement or whether consent is needed again. For example, if consent is needed again, the detection module 204 may instruct the user interface module 202 to generate a notification to be displayed to the user to obtain user confirmation to instruct the brightness module 206 to provide brightness enhancement.

[0057] Brightness module 206 provides brightness enhancement to a video frame to produce a modified video. In some embodiments, brightness module 206 includes a set of instructions executable by processor 235 to provide brightness enhancement of the video frame. In some embodiments, brightness module 206 is stored in memory 237 of computing device 200 and can be accessed and executed by processor 235.

[0058] In some embodiments, once detection module 204 determines that a video frame has low brightness, brightness module 206 can implement one or more corrections to the video frame. For example, brightness module 206 may apply linear correction, contrast limited adaptive histogram, or total brightness formula (square root formula).

[0059] Example linear correction

[0060] In cases where the background of the video frame is dark but the person in the video frame is well illuminated, some attempts to correct the brightness may include using a single average brightness value. This may cause the person to appear overexposed. Linear correction addresses this issue by looking at the distribution of brightness values and correcting portions with different brightness levels.

[0061] In the case of performing linear correction, brightness module 206 may receive the downscaled video frame passed by GPU 239 to memory 237 from memory 237. GPU 239 performs downscaling of the video frame and stores the first video frame in GPU memory. In some embodiments, GPU memory is not accessible by processor 235.

[0062] The luminance module 206 may generate a luminance histogram of the downscaled video frame, which is used to calculate statistics regarding the downscaled video frame, such as the distribution of luminance values within the downscaled video frame. The luminance module 206 may identify over-bright or over-dark regions of the downscaled video frame. For example, the luminance module 206 may identify the darkest five percent of the downscaled video frame and transmit statistical data to the GPU 239, which linearly corrects the original video frame to correct the darkest five percent to have a luminance of less than 20%. In another example, the luminance module 206 may identify the darkest 3%, 8%, 10%, etc. The luminance module 206 may also identify the brightest five percent of the downscaled video frame. The luminance module 206 may transmit statistical data to the GPU 239, which linearly corrects the original video frame to correct the brightest five percent to have a luminance of greater than 80%. Thus, the correction is only applied to a relatively small portion of the pixels, i.e., the darkest and brightest five percent of the pixels in this example, thereby reducing the computational effort in the calculation compared to correcting the entire pixels contained in a frame. Further, the examples in this paragraph illustrate the darkest and brightest five percent of the pixels. Pixels at other percentage points within the range of the darkest and brightest 2% to 10% are equally suitable, depending on the nature of the video data and the overall lighting conditions in the video. Similarly, depending on the nature of the video data and the overall lighting conditions in the video, different darkness thresholds (e.g., within the range between 10 and 30%) and / or different brightness thresholds (e.g., within the range between 70 and 90%) may be suitable.

[0063] The GPU 239 may apply linear correction to provide luminance enhancement in the video frame, which modifies a first luminance of a preset darkest percentage (e.g., the darkest 5%) of pixels to be less than a darkness threshold (e.g., greater than 20%), and modifies a second luminance of a preset brightest percentage (e.g., the brightest 5%) of pixels to be greater than a brightness threshold (e.g., greater than 80%).

[0064] The luminance module 206 can average multiple frames together to reduce the noise caused by amplification. In some embodiments, the number of frames selected for averaging is based on at least one of the frame speed (frame rate) of the video or the average original luminance of the video frames. In some embodiments, the luminance module 206 uses a lower number of frames and frame rate to determine how to ensure that the sum of the average frame luminance is at least 30% of the luminance. In some embodiments, one frame is used in the case of 0 - 10 frames per second (FPS); for a faster frame rate, when the frame rate is 10 - 15 FPS, the maximum number of frames for averaging can be the average of two frames; when the frame rate is 15 - 20 FPS, it can be the average of three frames; and when the frame rate is greater than 20 FPS, it can be the average of four frames. In some embodiments, the luminance module 206 maintains an exponential moving average such that the average frame luminance of the average frames is greater than 30%. In some embodiments, the luminance module 206 maintains an exponential moving average such that the average frame luminance remains between 30% and 70%.

[0065] Exemplary contrast - limited adaptive histogram equalization (CLAHE) correction

[0066] In some embodiments, the luminance module 206 uses CLAHE correction to artificially increase the dynamic range of the first video frame. CLAHE correction is a full - frame histogram equalization technique that can reveal details in the dark areas of an image even when some parts of the image are well - lit. The luminance module 206 can use CLAHE correction instead of linear correction or as a supplementary correction.

[0067] Applying CLAHE correction to the luminance channel produces a brighter image, but the resulting frame may have fewer saturated colors than the original video frame. Applying CLAHE correction separately to the chrominance channels can increase saturation but can also produce color artifacts. As a result, to keep the color enhancement within a reasonable range, the luminance module 206 can apply the result of CLAHE to both the luminance channel and the chrominance channels. For example, the luminance module 206 can apply CLAHE to a video frame by the following steps: (a) applying CLAHE to the luminance channel of the first video frame, (b) applying CLAHE to each chrominance channel of the image, and (c) mixing the results from (a) and (b).

[0068] Figure 4A and 4B will be discussed together. Figure 4A Flowchart 400 showing an example method of applying CLAHE correction is shown. Figure 4B Illustrate the corresponding stage 450 of the video frame changed using CLAHE correction. In Figure 4A box 402, the video frame is divided into Y, Cb, Cr channels, where Y is the luminance channel, Cb is the blue - difference chrominance channel, and Cr is the red - difference chrominance channel. InFigure 4B In this, 455 is the original video frame. The original video frame 455 is split into Y, Cb, and Cr channels to obtain the luminance channel (Y) 460, the blue-difference chrominance channel (Cb) 465, and the red-difference chrominance channel (Cr) 470.

[0069] In Figure 4A at the box 404, the luminance module 206 applies CLAHE to the luminance channel. This corresponds to Figure 4B the CLAHE-applied luminance channel 475 in Figure 4A at the box 406, the luminance module 206 applies CLAHE to the blue-difference chrominance channel. This corresponds to Figure 4B the blue-difference chrominance channel 480 to which CLAHE is applied in Figure 4A at the box 408, the luminance module 206 applies CLAHE to the red-difference chrominance channel. This corresponds to Figure 4B the chrominance red channel 485 to which CLAHE is applied in

[0070] At the box 410, the luminance module 206 combines Figure 4B the luminance channel 475 to which CLAHE is applied in Figure 4B with the blue-difference chrominance channel 465 and the chrominance red channel 470 in Figure 4B to form the post-CbCr combined luminance channel 490. At the box 412, the luminance module 206 combines the images obtained from CLAHE to form the full-channel CLAHE, where CLAHE is applied to all channels. Specifically, in Figure 4B the luminance channel 475 to which CLAHE is applied, the blue-difference chrominance channel 480 to which CLAHE is applied, and the red-difference chrominance channel 485 to which CLAHE is applied are combined to form

[0071] the CLAHE all channels 495 in Figure 4B At the box 414, the luminance module 206 mixes the post-CbCr combined luminance channel 490 with the CLAHE all channels 495 to obtain a video frame with enhanced luminance. In some embodiments, the luminance module 206 applies a 50 / 50 mix, which corresponds to

[0072] Exemplary total brightness

[0073] In some embodiments, the luminance module 206 simulates a longer exposure time by summing multiple frames. Since the relationship between the amount of photons hitting the camera sensor and the values obtained in Red, Green, Blue (RGB) is not linear, to reach a linear space, the luminance module raises the values to the power of 2.2 (assuming the camera 243 uses the standard gamma). Then going back to the encoded space, which is subsequently decoded by the display of the receiving user equipment (such as Figure 1 the user equipment 115n in

[0074] The luminance module 206 can determine the total luminance of a frame by calculating the square root of the sum of the average luminance of a set of frames, where each frame in the set of frames is squared. In some embodiments, for each video frame, the luminance module 206 uses the previous video frame to determine the total luminance using the following formula:

[0075]

[0076] where, b i is the average luminance of frame i and n is the number of frames.

[0077] In some embodiments, the luminance module 206 performs calculations to achieve a luminance level of at least 0.3, which is the average luminance of each frame between 0 and 1. If the first total luminance exceeds a threshold luminance value (e.g., 0.7), then the luminance module 206 removes a frame from the set of frames, even if this will cause the total luminance level to be below 0.3. Then the luminance module 206 uses the second total luminance calculated from the smaller set of frames. In some embodiments, the set of frames does not include frames earlier than 0.1 seconds compared to the first frame, because if the camera frame rate is low, they may cause excessive ghosting.

[0078] After the luminance module 206 determines the multiple frames for the set of frames, the luminance module 206 calculates a multiplier equal to 0.5 / total luminance. The multiplier can eliminate any fluctuations in the number of frames used by targeting a luminance level of 0.5. Other values can be used to obtain different output luminances, where higher values result in higher output luminances.

[0079] For each pixel and each Red, Green, Blue (RGB) color channel in the video frame, the luminance module 206 calculates the final output value. The luminance module 206 can use the following equation:

[0080]

[0081] where c r is the final output, and c iis the input value for frame i. The input value can be a separate value for an RGB color channel. Since each frame has multiple inputs and outputs, there can be an input value for each color channel of each pixel in the frame.

[0082] The luminance module 206 can use the total luminance to enhance the luminance of the video frame, rather than the GPU 239 applying linear correction or the luminance module 206 applying CLAHE correction. In some embodiments, in addition to another correction method, the luminance module 206 also uses the total luminance.

[0083] In some embodiments, once the luminance module 206 performs correction on the video frame, the luminance module 206 determines whether the luminance of the modified video (e.g., the first video frame, a set of video frames, etc.) meets a threshold luminance. If the modified video fails to meet the threshold luminance, the luminance module 206 can change one or more camera settings. For example, the luminance module 206 can instruct the camera 243 to perform at least one of increasing the exposure time or modifying the sensitivity of the image sensor. The luminance module 206 can modify the image sensor by instructing the camera 243 to modify the ISO setting.

[0084] Exemplary method

[0085] Figure 5 A flowchart of an example method 500 for enhancing the luminance of a video frame is shown. Method 500 is executed by a video application 103 stored on a computing device 200 (such as a user device 115, a video server 101, or a portion of the user device 115 and a portion of the video server 101).

[0086] At block 502, it is determined whether the user agrees to the low-light mode. If the user does not agree to the low-light mode, method 500 proceeds to Figure 6 step 602 therein. If the user agrees to the low-light mode, at block 504, it is determined that the first video frame has low luminance. At block 506, linear correction is applied to provide luminance enhancement in the first video frame, where the luminance enhancement modifies the first luminance of a preset darkest percentage of pixels to be less than a darkness threshold, or modifies the second luminance of a preset brightest percentage of pixels to be greater than a luminance threshold. Other correction methods, such as CLAHE correction, can alternatively or additionally be used. At block 508, the first luminance enhancement of the first video frame is averaged with a set of luminance enhancements corresponding to prior video frames to produce a modified video.

[0087] At block 510, it is determined whether the modified video meets a brightness threshold. If the modified video meets the brightness threshold, method 500 proceeds to the next video frame. Video application 103 may also send the modified video to user device 115n and / or display the modified video in display 245 of user device 115a. For example, brightness module 206 may analyze every 30 video frames. If the modified video fails to meet the brightness threshold, method 500 proceeds to block 512 and changes one or more camera settings.

[0088] Figure 6 A flowchart of an example method 600 for obtaining user consent to enable a low-light mode is shown in accordance with some embodiments. At block 602, it is determined that there is low brightness in the video frame. At block 604, it is determined whether a notification 604 can be displayed. If the notification cannot be displayed, method 600 returns to block 602. If the notification can be shown, the notification is shown at block 606. The notification may include a message such as "Low brightness has been detected. Do you want to enable the low-light mode?"

[0089] At block 608, it is determined whether the low-light mode is enabled. If the low-light mode is not enabled, method 600 proceeds to block 610 with an increasing exponential backoff. For example, if the user does not want to enable the low-light mode, the next notification will not be shown on another day, five days, 25 days, 125 days, etc. If the low-light mode is enabled, method 600 proceeds to block 612.

[0090] At block 612, the low-light mode is enabled. At block 614, low brightness is determined in the video frame. At block 616, brightness enhancement is performed, and method 600 continues to block 608.

[0091] In the above description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the specification. However, it will be apparent to one of ordinary skill in the art that the present disclosure may be practiced without these specific details. In some instances, structures and devices are shown in block diagram form in order to avoid obscuring the description. For example, the above embodiments may be described primarily with reference to a user interface and specific hardware. However, the embodiments are applicable to any type of computing device that can receive data and commands, as well as any peripheral device that provides services.

[0092] Reference to "some embodiments" or "some examples" in the specification means that a particular feature, structure, or characteristic described in connection with the embodiments or examples may be included in at least one implementation of the specification. The phrase "in some embodiments" appearing in various places in the specification does not necessarily refer to the same embodiments.

[0093] Certain portions of the detailed description above are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, considered to be a self-consistent sequence of steps leading to a desired result. These steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic data capable of being stored, transferred, combined, compared, and otherwise manipulated. For the sake of common usage, it has sometimes proven convenient to refer to these data as bits, values, elements, symbols, characters, terms, numbers, etc.

[0094] However, it should be borne in mind that all such and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. As is apparent from the following discussion, unless specifically stated otherwise. It should be understood that throughout the specification, discussions using terms including "processing" or "computational processing" or "computing" or "determining" or "displaying", etc. refer to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities within the registers and memories of the computer system into other data similarly represented as physical quantities within the memories or registers of the computer system or other such information storage, transmission, or display devices.

[0095] Embodiments of this specification may also relate to a processor for performing one or more steps of the above methods. The processor may be a dedicated processor selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory computer-readable storage medium, including but not limited to any type of disk, including optical disks, ROM, CD-ROMs, magnetic disks, RAMs, EPROMs, EEPROMs, magnetic or optical cards, including flash memory of a USB key having non-volatile memory, or any type of medium suitable for storing electronic instructions. Each is coupled to the computer system bus.

[0096] This specification may take the form of some completely hardware embodiments, some completely software embodiments, or some embodiments incorporating both hardware and software elements. In some embodiments, the specification is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

[0097] In addition, the description may take the form of a computer program product accessible from a computer-usable or computer-readable medium that provides program code for use by or in conjunction with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer-readable medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0098] A data processing system suitable for storing or executing program code will include at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory used during actual execution of the program code, bulk storage, and cache memory that provides temporary storage of at least some program code in order to reduce the number of times the code must be retrieved from bulk storage during execution.

Claims

1. A method for enhancing the brightness of a video frame, characterized in that, The method includes: generating, by a processor, a luminance histogram for each video frame in a set of video frames; for each video frame in the set of video frames, determining, based on the luminance histogram of the video frame, corresponding pixels that match a predetermined darkest percentage of pixels and corresponding pixels that match a predetermined brightest percentage of pixels; and applying a linear correction to each video frame in the set of video frames to provide luminance enhancement, the luminance enhancement modifying at least one of the following group: modifying a first luminance corresponding to the predetermined darkest percentage of pixels to be less than a darkness threshold, modifying a second luminance corresponding to the predetermined brightest percentage of pixels to be greater than a luminance threshold, and a combination of both.

2. The method according to claim 1, characterized in that, The video is a streaming video captured by a camera during a video call, and after applying the linear correction, the streaming video is transmitted to one or more other participants in the video call.

3. The method according to claim 1, wherein The linear correction is applied on a sending device in the video call.

4. The method according to claim 1, characterized in that, It further includes: downscaling each video frame in the set of video frames by a graphics processing unit (GPU) different from the processor, where the set of video frames is stored in a GPU memory of the GPU and where the processor cannot access the GPU memory; and transmitting, by the GPU, the downscaled set of video frames to a memory accessible by the processor for computer processing; wherein the luminance histogram is generated from the downscaled video frames, and the linear correction is applied to the set of video frames by the GPU.

5. The method according to claim 1, wherein Selecting the number of frames based on the frame rate in the set of video frames to calculate an average frame luminance.

6. The method according to claim 5, characterized in that, If the frame rate is 0 - 10 frames per second, the number of frames is 1; if the frame rate is 10 - 15 frames per second, the number of frames is 2; if the frame rate is 15 - 20 frames per second, the number of frames is 3; if the frame rate is greater than 20 frames per second, the number of frames is 4.

7. The method according to claim 1, wherein It further includes: after applying the linear correction, applying a second correction to ensure that the sum of the average luminance of each video frame in the set of video frames is at least 30% of the total luminance.

8. A non-transitory computer-readable medium, characterized in that, Enhancing the luminance of video frames in a video based on an instruction, which when executed by one or more processors, causes the one or more processors to perform operations, the operations including: generating, by a processor, a luminance histogram for each video frame in a set of video frames; for each video frame in the set of video frames, determining, based on the luminance histogram of the video frame, corresponding pixels that match a predetermined darkest percentage of pixels and corresponding pixels that match a predetermined brightest percentage of pixels; and applying a linear correction to each video frame in the set of video frames to provide luminance enhancement, the luminance enhancement modifying at least one of the following group: modifying a first luminance corresponding to the predetermined darkest percentage of pixels to be less than a darkness threshold, modifying a second luminance corresponding to the predetermined brightest percentage of pixels to be greater than a luminance threshold, and a combination of both.

9. The computer-readable medium according to claim 8, wherein, The video is a streaming video captured by a camera during a video call, and the operation further includes, after applying the linear correction, transmitting the streaming video to one or more other participants in the video call.

10. The computer-readable medium according to claim 8, wherein Apply the linear correction on the sending device in the video call.

11. The computer-readable medium according to claim 8, wherein, The operation further includes: Shrinking each video frame in the set of video frames by a graphics processing unit (GPU) different from the one or more processors, where the set of video frames is stored in the GPU memory of the GPU, and where the one or more processors cannot access the GPU memory; and Transmitting the shrunk set of video frames by the GPU to a memory accessible by the one or more processors for computer processing; wherein, the luminance histogram is generated from the shrunk video frames, and the linear correction is applied to the set of video frames by the GPU.

12. The computer-readable medium according to claim 8, wherein Select the number of frames based on the frame rate in the set of video frames to calculate the average frame luminance.

13. The computer-readable medium according to claim 12, wherein, If the frame rate is 0 - 10 frames per second, the number of frames is 1; if the frame rate is 10 - 15 frames per second, the number of frames is 2; if the frame rate is 15 - 20 frames per second, the number of frames is 3; if the frame rate is greater than 20 frames per second, the number of frames is 4.

14. The computer-readable medium according to claim 8, wherein The operation further includes: After applying the linear correction, applying a second correction to ensure that the sum of the average luminance of each video frame in the set of video frames is at least 30% of the total luminance.

15. A system for enhancing the brightness of a video frame, characterized in that, Comprising: One or more processors; And A memory storing instructions executed by the one or more processors, the instructions including: Generating, by a processor, a luminance histogram for each video frame in the set of video frames; For each video frame in the set of video frames, determining, based on the luminance histogram of the video frame, a corresponding pixel that matches a predetermined darkest percentage of pixels and a corresponding pixel that matches a predetermined brightest percentage of pixels; and Applying a linear correction to each video frame in the set of video frames to provide luminance enhancement, the luminance enhancement modifying at least one of the following group: modifying a first luminance corresponding to the predetermined darkest percentage of pixels to be less than a darkness threshold, modifying a second luminance corresponding to the predetermined brightest percentage of pixels to be greater than a luminance threshold, and a combination of both.

16. The system according to claim 15, wherein, The video is a streaming video captured by a camera during a video call, and the instructions further include, after applying the linear correction, transmitting the streaming video to one or more other participants in the video call.

17. The system according to claim 15, wherein Apply the linear correction on the sending device in the video call.

18. The system according to claim 15, wherein The instructions further include: Shrinking each video frame in the set of video frames by a graphics processing unit (GPU) different from the one or more processors, where the set of video frames is stored in the GPU memory of the GPU, and where the one or more processors cannot access the GPU memory; and Transfer the set of downscaled video frames to a computer - processed memory accessible by the one or more processors via the Graphics Processing Unit (GPU). Wherein, the luminance histogram is generated from the downscaled video frames, and the linear correction is applied to the set of video frames by the Graphics Processing Unit (GPU).

19. The system according to claim 15, wherein, Select the number of frames based on the frame rate in the set of video frames to calculate the average frame luminance.

20. The system according to claim 19, wherein, If the frame rate is 0 - 10 frames per second, the number of frames is 1; if the frame rate is 10 - 15 frames per second, the number of frames is 2; if the frame rate is 15 - 20 frames per second, the number of frames is 3; if the frame rate is greater than 20 frames per second, the number of frames is 4.

Images