Video processing methods, electronic devices and storage media

Abstract

This application provides a video processing method, an electronic device, and a storage medium. In this method, for a video image frame, the electronic device determines an inverse tone mapping curve corresponding to the video image frame to obtain a grayscale gain value corresponding to the video image frame, and simultaneously determines a pixel compression coefficient corresponding to the video image frame. Then, the electronic device processes a preset 3D LUT based on the grayscale gain value and the pixel compression coefficient to obtain a target 3D LUT corresponding to the video image frame, and processes the video image frame according to the target 3D LUT. The electronic device then increases the screen backlight brightness and displays the video image frame processed by the target 3D LUT, thereby achieving a display effect that expands the dynamic range of the video image frame.

Description

Technical Field

[0001] This application relates to the field of smart terminal technology, and in particular to a video processing method, electronic device and storage medium. Background Technology

[0002] With the development of video display technology, users have increasingly higher requirements for video display effects. Compared with SDR (Standard Dynamic Range) display, HDR (Higher Dynamic Range) display can provide users with more realistic images. Therefore, compared with SDR video, HDR video has higher contrast and detail, better reproduces realistic scenes, and brings users a better visual experience.

[0003] However, due to the high production cost of HDR videos, most videos online are SDR videos. Therefore, how to improve the picture quality of SDR videos to that of HDR videos is an urgent problem to be solved. Summary of the Invention

[0004] To address the aforementioned technical problems, embodiments of this application provide a video processing method, an electronic device, and a storage medium. In this method, the electronic device determines the inverse tone mapping curve and pixel compression coefficient corresponding to each video image frame to obtain a target 3D LUT corresponding to each video image frame, and processes the corresponding video image frames. Then, the electronic device displays each video image frame processed by the corresponding target 3D LUT after increasing the screen backlight brightness, thereby achieving a display effect that expands the dynamic range of the video.

[0005] In a first aspect, embodiments of this application provide a video processing method. This method is applied to an electronic device where the screen backlight brightness is a first brightness. The method includes:

[0006] The electronic device responds to the user's operation and acquires the first video image frame;

[0007] The electronic device determines the inverse tone mapping curve corresponding to the first video image frame, and determines the gray level gain value corresponding to the first video image frame based on the inverse tone mapping curve; wherein, the gray level gain value corresponding to the diffuse region of the first video image frame is 0; the gray level value of the pixels in the diffuse region is less than or equal to the gray level segmentation threshold.

[0008] The electronic device determines the pixel compression coefficient corresponding to the first video image frame. The pixel compression coefficient is used to compress the pixel value.

[0009] The electronic device processes each color channel of the preset 3D LUT according to the grayscale gain value and pixel compression coefficient to obtain the target 3D LUT corresponding to the first video image frame;

[0010] The electronic device processes the first video image frame according to the target 3D LUT to obtain the second video image frame;

[0011] After the screen backlight brightness of the electronic device is increased to a second brightness, the electronic device displays a second video image frame; wherein the second brightness is greater than the first brightness.

[0012] When the electronic device processes the pixel brightness values ​​of the preset 3D LUT, it processes the pixel brightness values ​​of each channel (R channel, G channel, and B channel) of the preset 3D LUT separately. The resulting pixel brightness value of each channel of the target 3D LUT is I″=I*(Gain+1)*C. C is the pixel compression coefficient, and Gain is the grayscale gain value.

[0013] The second brightness can be the maximum brightness value supported or allowed by the backlight of the electronic device's display.

[0014] In this way, the electronic device performs dynamic range expansion processing on the preset 3D LUT based on the content of each video image frame to obtain a target 3D LUT corresponding to each video image frame. Then, the electronic device performs color adjustment processing on each video image frame based on the target 3D LUT corresponding to each video image frame, and displays it after increasing the backlight brightness of the display screen, thereby achieving the effect of expanding the dynamic range of the video.

[0015] According to the first aspect, the electronic device determines the inverse tone mapping curve corresponding to the first video image frame, including:

[0016] If the first video image frame does not exhibit color banding after dynamic range expansion, the electronic device uses a preset inverse tone mapping curve as the inverse tone mapping curve corresponding to the first video image frame; wherein, the preset inverse tone mapping curve is calculated based on a piecewise function method; if the first video image frame exhibits color banding after dynamic range expansion, the electronic device divides the bright grayscale range into multiple grayscale sub-ranges, and calculates a third-order inverse tone mapping curve for each grayscale sub-range based on the preset inverse tone mapping curve to obtain the inverse tone mapping curve corresponding to the first video image frame; wherein, the grayscale values ​​included in the bright grayscale range are all greater than the grayscale segmentation threshold.

[0017] In this way, when the electronic device determines the corresponding inverse tone mapping curve for each video image frame, it first determines whether the video image frame will exhibit color banding after dynamic range expansion. If the video image frame will not exhibit color banding after dynamic range expansion, the preset inverse tone mapping curve is used as the inverse tone mapping curve corresponding to the video image frame. Otherwise, the decolorized inverse tone mapping curve corresponding to the video image frame is recalculated based on the preset inverse tone mapping curve, thereby avoiding the problem of color banding in video image frames after dynamic range expansion.

[0018] According to the first aspect, or any implementation of the first aspect above, before the electronic device determines the inverse tone mapping curve corresponding to the first video image frame, the method further includes:

[0019] The electronic device statistically analyzes the grayscale histogram of the first video image frame, and performs regularization on the grayscale histogram to obtain the first regularized image;

[0020] The electronic device determines whether color banding will occur in the first video image frame after dynamic range expansion, including:

[0021] The electronic device determines whether there is a regularization value peak in the bright grayscale range of the first regularization map that is greater than the first threshold. If it exists, the electronic device determines that the first video image frame will have color banding after dynamic range expansion; otherwise, the electronic device determines that the first video image frame will not have color banding after dynamic range expansion.

[0022] Considering that if a certain grayscale pixel has a high proportion in the bright area of ​​the image, color banding is likely to occur after the dynamic range of the image is expanded, this embodiment determines whether the current video image frame will not produce color banding after the dynamic range is expanded based on whether there is a regularization value peak in the bright area that is greater than a preset threshold (th2 below).

[0023] According to the first aspect, or any implementation of the first aspect above, the electronic device divides the high-brightness grayscale range into multiple grayscale sub-ranges, including:

[0024] When there is only one regularization peak, the electronic device divides the bright grayscale range into two grayscale sub-ranges according to the grayscale value corresponding to the regularization peak.

[0025] When there are multiple regularization value peaks, the electronic device determines the regularization value valley between every two adjacent regularization value peaks, and divides the bright grayscale range into multiple grayscale sub-ranges according to the grayscale values ​​corresponding to each regularization value peak and each regularization value valley.

[0026] Since color banding is prone to occur at the gray value corresponding to the regularization value peak, in this embodiment, the bright area is divided into multiple brightness fluctuation ranges based on the gray value corresponding to the regularization value peak.

[0027] Based on the first aspect, or any implementation of the first aspect above, the formula for calculating the pre-defined inverse tone mapping curve Gain_1(i) is as follows:

[0028]

[0029] Where i represents the normalized pixel gray value, i∈[0,1], Gain_1(i) represents the gray gain value corresponding to the gray value i, th3 represents the gray segmentation threshold, α represents the exponential coefficient, v1 and v2 are two preset gray segmentation thresholds, v2>v1, a, b, c, d, e, f are preset coefficients, and a, b, d are positive numbers.

[0030] In this way, since the slope of the inverse tone mapping curve Gain_1(i) is limited, the problem of unnatural image display caused by excessively large slope of the inverse tone mapping curve after the dynamic range of video image frames is expanded can be avoided.

[0031] According to the first aspect, or any implementation of the first aspect above, the electronic device calculates a third-order inverse toning map for color banding based on a preset inverse toning map curve for each grayscale sub-range, including:

[0032] For the grayscale sub-interval (S1, S2], the third-order inverse tone mapping curve Gain_2(i) after color banding removal is as follows:

[0033] Gain_2(i)=m*i 3 +n*i 2 +p*i+q, i∈(S1,S2];

[0034] Where i represents the pixel gray value, Gain_2(i) represents the gray value gain corresponding to the gray value i; m, n, p, q are coefficients, calculated based on the values ​​of Gain_2(S1), Gain_2(S2), Gain_2'(S1) and Gain_2'(S2);

[0035]

[0036]

[0037] If the grayscale value S1 is the grayscale segmentation threshold, then Gain_2(S1) = Gain_1(S1), Gain_2′(S1) = Gain_1′(S1); otherwise, Gain_2(S1) is calculated based on the third-order inverse tone mapping curve corresponding to the adjacent previous grayscale sub-interval.

[0038] Where Gain_1(i) is the preset inverse tone mapping curve, Gain_1′(i) is the derivative of the preset inverse tone mapping curve, Δ=1 / (N-1), and N is the total number of gray levels;

[0039] H norm ′(i)=maximum(H norm (i)-Tq,0);

[0040] Where Tq is a constant, H norm (i) is the regularization value of the number of pixels corresponding to the gray value i in the gray histogram.

[0041] In this embodiment, the inverse tone mapping curve Gain_2(i) for decolorization is preset to a third-order curve. The purpose is to reduce the slope of the inverse tone mapping curve near the fault gray level by controlling the slope at the endpoints S1 and S2 of the gray range, thereby achieving the purpose of weakening the color fault.

[0042] According to the first aspect, or any implementation of the first aspect above, H norm (i) It has undergone the following processing:

[0043]

[0044] Here, th5 is a preset threshold.

[0045] In this way, pre-processing the regularization graph can enhance the brightness variation of the highlight area of ​​the image, thereby making the difference in slope limitation at different gray values ​​of the highlight area more obvious, thus improving the accuracy of the inverse tone mapping curve Gain_2(i) and ensuring the decolorization effect after the dynamic range of the video image frame is expanded.

[0046] According to the first aspect, or any implementation of the first aspect above, the electronic device determines the pixel compression coefficient corresponding to the first video image frame, including:

[0047] The electronic device calculates the average brightness of the first video image frame; the electronic device calculates the undetermined expansion factor corresponding to the first video image frame based on the average brightness; if the undetermined expansion factor is less than or equal to the backlight brightness clearance ratio of the display screen, the electronic device uses the undetermined expansion factor as the target expansion factor corresponding to the first video image frame; otherwise, the electronic device uses the backlight brightness clearance ratio of the display screen as the target expansion factor Err_target corresponding to the first video image frame; wherein, the backlight brightness clearance ratio of the display screen is used to indicate the supported or allowed increase ratio of the backlight brightness of the display screen.

[0048] The electronic device calculates the pixel compression factor C corresponding to the first video image frame according to the following formula:

[0049] C = 1 / Err_target 1 / γ γ is the gamma coefficient.

[0050] In this embodiment, the pixel compression coefficient C is specifically used to compress the pixel brightness value of the preset 3D LUT when increasing the backlight brightness of the display screen. The compression coefficient C is matched with the backlight brightness of the display screen to avoid over-expansion (also known as stretching) when the dynamic range of the video image frame is expanded based on the target 3D LUT obtained after processing, which would cause an uneven display effect. It also ensures that the dynamic range of the finally expanded video image frame can match the display capability of the display screen, thereby improving the display effect of the video.

[0051] According to the first aspect, or any implementation of the first aspect above, the electronic device calculates the undetermined expansion factor corresponding to the first video image frame based on the average brightness, including:

[0052] The undetermined expansion factor Err_tbd corresponding to the first video image frame is calculated using the following formula:

[0053]

[0054] Where APL is the average brightness of all pixels in the video image frame, th1 is the preset pixel brightness threshold, Err1 is a preset maximum expansion factor, and m, n, and k are preset coefficients.

[0055] In this embodiment, Err1 can be understood as a preset maximum dynamic range expansion factor, and the coefficients m, n, and k are determined based on a preset minimum dynamic range expansion factor (Err2), where Err2 is less than Err1. Furthermore, the undetermined dynamic range expansion factor (Err_tbd) corresponding to the current video image frame, calculated by the electronic device according to the above formula, is less than or equal to the preset maximum dynamic range expansion factor (Err1) and greater than or equal to the preset minimum dynamic range expansion factor (Err2).

[0056] According to the first aspect, or any implementation of the first aspect above, the electronic device determines the inverse tone mapping curve corresponding to the first video image frame, including:

[0057] The electronic device acquires a second regularized graph corresponding to the first video image frame and a third regularized graph corresponding to the third video image frame; wherein the third video image frame is the frame preceding the first video image frame; the electronic device calculates the similarity between the second regularized graph and the third regularized graph; if the similarity is greater than a preset similarity threshold, the electronic device uses the inverse tone mapping curve corresponding to the third video image frame as the inverse tone mapping curve corresponding to the first video image frame.

[0058] Considering that video playback has a certain degree of continuity, for example, in the case of no scene switching, the content of the current video image frame changes little compared to the content of the previous video image frame (such as the similarity between the current video image frame and its previous video image frame being greater than or equal to the similarity threshold), this embodiment can use the inverse tone mapping curve of the previous video image frame as the inverse tone mapping curve corresponding to the current video image frame, thereby reducing the amount of computation.

[0059] According to the first aspect, or any implementation of the first aspect above, the regularization values ​​in the second and third regularization graphs have been pre-processed as follows:

[0060]

[0061] Among them, H norm (i) is the regularization value of the number of pixels corresponding to the gray value i in the gray histogram, and th5 is a preset threshold.

[0062] For video frames with low regularization values ​​in highlighted areas—meaning these frames won't exhibit color banding after dynamic range expansion—the inverse tone mapping curve of the preceding video frame can be directly used as its corresponding inverse tone mapping curve. In this scenario, judging the similarity between the two frames based on the pre-processed regularization map will yield a relatively high similarity value. Therefore, it can be determined that the two video frames are similar, and the inverse tone mapping curve of the preceding frame can be directly used as its corresponding inverse tone mapping curve.

[0063] According to the first aspect, or any implementation of the first aspect above, the electronic device processes each color channel of the preset 3D LUT based on the grayscale gain value and the pixel compression coefficient to obtain the target 3D LUT corresponding to the first video image frame, including:

[0064] The electronic device processes each color channel of the preset 3D LUT according to the grayscale gain value and pixel compression coefficient to obtain the initial 3D LUT corresponding to the first video image frame; the electronic device obtains the target 3D LUT corresponding to the third video image frame; wherein, the third video image frame is the frame preceding the first video image frame; the electronic device calculates the target 3D LUT corresponding to the first video image frame by weighting the target 3D LUT corresponding to the third video image frame and the initial 3D LUT corresponding to the first video image frame.

[0065] In this way, by determining the target 3D LUT of the current video image frame based on the target 3D LUT of the previous video image frame and the initial 3D LUT of the current video image frame, the problem of flickering when electronic devices display processed video image frames can be avoided, and the transition between video image frames can be relatively smooth.

[0066] According to the first aspect, or any implementation of the first aspect above, the dynamic range of the first video image frame is the standard dynamic range; the dynamic range of the second video image frame is the high dynamic range.

[0067] Secondly, embodiments of this application provide an electronic device. The electronic device includes: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when executed by the one or more processors, the electronic device performs the video processing method of the first aspect and any one thereof.

[0068] The second aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the second aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0069] Thirdly, embodiments of this application provide a computer-readable storage medium. This computer-readable storage medium includes a computer program that, when run on an electronic device, causes the electronic device to perform the video processing method of the first aspect and any one thereof.

[0070] The third aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the third aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0071] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when run, causes a computer to perform a video processing method as described in the first aspect or any one of the first aspects.

[0072] The fourth aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the fourth aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0073] Fifthly, this application provides a chip including a processing circuit and transceiver pins. The transceiver pins and the processing circuit communicate with each other via an internal connection path. The processing circuit executes a video processing method as described in the first aspect or any one thereof, to control the receiving pin to receive signals and to control the transmitting pin to transmit signals.

[0074] The fifth aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the fifth aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here. Attached Figure Description

[0075] Figure 1 A comparative diagram illustrating the dynamic range of an image as an example;

[0076] Figure 2 A schematic diagram of the hardware structure of an electronic device as an example;

[0077] Figure 3a A schematic diagram of the software structure of an electronic device as an example;

[0078] Figure 3b The processing flow of a video image frame is shown as an example.

[0079] Figure 4 This is an illustrative diagram illustrating an application scenario of extended video dynamic range.

[0080] Figure 5 This is a schematic diagram illustrating the module interaction of the video processing method provided in an embodiment of this application, which is an example of such an application.

[0081] Figure 6 A schematic diagram illustrating the process of adjusting the content of a 3D LUT based on video image frames, provided in an embodiment of this application;

[0082] Figure 7 This is an example of a grayscale histogram processing flow;

[0083] Figure 8 This is an example of the processing effect on a video image frame;

[0084] Figure 9 This is an example illustrating whether color banding occurs after dynamic range expansion of a video image frame;

[0085] Figure 10 This is an example of the processing effect on a video image frame;

[0086] Figure 11 This is a flowchart illustrating the video processing method provided in an embodiment of this application. Detailed Implementation

[0087] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0088] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0089] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0090] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0091] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0092] To better understand the embodiments of this application, the terms or concepts that may be involved in the embodiments are explained below.

[0093] The backlight brightness of a screen, also known as screen brightness, is a physical quantity that characterizes the intensity of light emitted by the screen of an electronic device. The unit can be nits or candela per square meter (cd / m2).

[0094] Grayscale, also known as gray level, is a parameter characterizing the brightness and darkness of an image. An image is composed of multiple pixels, each of which can display various colors. These colors are composed of three sub-pixels: red, green, and blue. The value of each sub-pixel ranges from 0 to 255. The RGB values ​​of a pixel are also called its color value, color, or chromaticity. The light source behind each sub-pixel can display different brightness levels. Grayscale represents different levels of brightness from the darkest to the brightest. For an 8-bit screen, an image can include 256 brightness levels from 0 to 255, meaning the image can include 256 grayscale levels. In some embodiments, 0-255 can also be converted into corresponding values ​​within the range of 0-1 to represent grayscale.

[0095] A Look-Up Table (LUT), also known as a color lookup table or color mapping table, is used to adjust the color values ​​of an image. LUTs can be stored as LUT files with the .lut file format. Electronic devices can use LUTs to convert the original color values ​​(i.e., RGB values) of each pixel in an image into corresponding target color values, thereby changing the original color effect of the image.

[0096] A Lookup Table (LUT) is a mapping table of color values ​​used to represent the correspondence between input and output color values. LUTs are mainly divided into 1D LUTs and 3D LUTs, namely one-dimensional lookup tables and three-dimensional lookup tables. 3D LUTs can influence hue, saturation, and brightness through full-space color control, thereby changing colors. Changing any one color value will result in a corresponding change in the other colors. Compared to 1D LUTs, 3D LUTs offer richer color variations and can meet more precise color control needs.

[0097] The dynamic range (DR) of an image refers to the range between the maximum and minimum brightness of a pixel in the image. In traditional technologies, limited by the display technology of display devices, the dynamic range of images is relatively small, resulting in a significant difference between the image seen by the user and the actual image, leading to a poor user experience.

[0098] With the development of video display technology, users have increasingly higher requirements for video display effects. Compared with SDR display, HDR display can provide users with more realistic images. Therefore, compared with SDR video, HDR video has higher contrast and detail, better reproduces realistic scenes, and brings users a better visual experience. At the same time, with the improvement of display device technology, the peak brightness of screens of electronic devices such as mobile phones can exceed 1000 nits, which can fully support the display of HDR video content.

[0099] However, due to the high production cost of HDR videos, most videos online are SDR videos. Therefore, how to extend the dynamic range of SDR videos to improve their picture quality to HDR display quality is an urgent problem to be solved.

[0100] To address the aforementioned issues, fully utilize the display capabilities of the terminal screen, and enhance the user's video viewing experience, this application provides a video processing method. In this method, the electronic device employs an inverse tone mapping method to dynamically extend the dynamic range of SDR video, thereby displaying the SDR video on the screen with a visual effect similar to HDR.

[0101] For example, Figure 1 This is a comparative schematic diagram of the dynamic range of an image provided in an embodiment of this application. Figure 1 Figure (1) is the original SDR image, i.e., the image without dynamic range extension. Figure 1 Figure (2) is an image after dynamic range expansion of Figure (1). (Comparison) Figure 1 Figure (1) in the middle and Figure 1 As can be seen from Figure (2), after dynamic range expansion, the brightness of the bright areas in the image is higher, and the dynamic range of the image is increased. Therefore, the image after dynamic range expansion can better reflect the gradation and layering of light and color, and improve the display effect.

[0102] The following describes the electronic device structure and software architecture to which the video processing method provided in the embodiments of this application is applicable.

[0103] The video processing method provided in this application can be applied to electronic devices that can install applications (APPs), such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). This application does not impose any restrictions on the specific type of electronic device.

[0104] For example, such as Figure 2 The diagram shown is a structural schematic of the electronic device 100. Optionally, the electronic device 100 can be a terminal, or a terminal device; this application does not limit this. It should be understood that... Figure 2 The electronic device 100 shown is only one example of an electronic device, and the electronic device 100 may have more or fewer components than shown in the figure, may combine two or more components, or may have different component configurations. Figure 2 The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0105] Electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include pressure sensors, gyroscope sensors, accelerometers, temperature sensors, motion sensors, barometric pressure sensors, magnetic sensors, distance sensors, proximity sensors, fingerprint sensors, touch sensors, ambient light sensors, bone conduction sensors, etc.

[0106] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0107] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0108] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory.

[0109] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0110] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0111] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0112] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0113] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0114] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0115] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to perform data storage functions.

[0116] The internal memory 121 can be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121, such as enabling the electronic device 100 to implement the video processing method in the embodiments of this application. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio and video data, etc.). Furthermore, the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0117] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.

[0118] Figure 3a This is a software structure block diagram of the electronic device 100 according to an embodiment of this application.

[0119] The layered architecture of the electronic device 100 divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android Runtime and system libraries, the HAL (Hardware Abstraction Layer), and the kernel layer (also known as the driver layer).

[0120] The application layer can include a series of application packages.

[0121] like Figure 3a As shown, the application package can include applications such as video applications, live streaming applications, camera applications, and gallery applications. In addition, the application package can also include applications such as calling, calendar, maps, navigation, music, video, SMS, WLAN, and Bluetooth.

[0122] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0123] like Figure 3a As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, brightness service module, etc.

[0124] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0125] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0126] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0127] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection and disconnection).

[0128] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0129] The notification manager allows applications to display notifications in the status bar. These notifications can be used to convey informational messages and can disappear automatically after a short pause, requiring no user interaction. Examples include notifications of download completion and message alerts. Notifications can also appear as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Other examples include text messages displayed in the status bar, sound alerts, vibrations from electronic devices, and flashing indicator lights.

[0130] The brightness service module is used to manage the brightness of electronic devices. In this embodiment, the brightness service module can adjust the backlight brightness of the screen according to a preset brightness threshold. For example, when the screen brightness of the electronic device supports a maximum of 1000 nits, the backlight brightness of the screen is adjusted to 900 nits according to the preset brightness threshold of 900 nits.

[0131] The Android Runtime consists of core libraries and a virtual machine. The Android Runtime is responsible for the scheduling and management of the Android system.

[0132] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0133] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0134] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0135] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0136] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0137] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0138] A 2D graphics engine is a graphics engine for 2D drawing.

[0139] HAL (Hardware Abstraction Layer) is used to abstract low-level hardware, providing a unified, abstracted service to higher layers. Specifically, HAL can encapsulate drivers in the kernel layer and provide interfaces for the application framework layer, shielding it from the implementation details of the low-level hardware. For example... Figure 3a As shown, the HAL may include an encoding / decoding module, an image format conversion module, and a LUT mapping module. In addition, the HAL may also include a camera HAL, etc., but this embodiment does not limit this.

[0140] The encoding / decoding module can be used to encode and decode video data; the image format conversion module can be used to convert the format of images; and the LUT mapping module is used to perform color correction on image frames based on 3D LUTs.

[0141] In this embodiment, the image format conversion module can be used to convert the decoded YUV format video image frames into RGB format video image frames.

[0142] In this embodiment, the LUT mapping module can specifically be used to perform color correction processing on RGB format video image frames to expand the dynamic range of the video image frames. For example, the LUT mapping module may include a LUT processing module and an image color correction module. The LUT processing module can be used to dynamically expand the range of a preset (or default) 3D LUT to obtain 3D LUTs corresponding to each video image frame. The image color correction module can be used to apply the preset 3D LUT to the video image frames to achieve color correction processing on the corresponding video image frames, or it can be used to apply the dynamically expanded 3D LUT corresponding to each video image frame to the corresponding video image frame to achieve color correction processing on the corresponding video image frame and achieve the effect of dynamically expanding the dynamic range of the corresponding video image frame.

[0143] For example, such as Figure 3b As shown, the LUT processing module performs dynamic range expansion on a preset 3D LUT based on video image frame A to obtain a target 3D LUT corresponding to video image frame A. Then, the image color correction module can perform color correction on video image frame A based on the target 3D LUT corresponding to video image frame A to obtain video image frame A'. Relative to video image frame A, video image frame A' is the dynamically expanded video image frame corresponding to it.

[0144] The kernel layer is the layer between hardware and software. It includes at least display drivers, Wi-Fi drivers, and sensor drivers. The hardware includes at least a processor, display screen, Wi-Fi module, and sensors.

[0145] Understandable Figure 3a The layers in the illustrated software structure and the components contained in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer layers than illustrated, and each layer may include more or fewer components; this application does not impose any limitations.

[0146] It is understood that, in order to implement the video processing method in the embodiments of this application, the electronic device includes hardware and / or software modules that perform various functions. Based on the algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0147] For ease of understanding, the following embodiments of this application will be described using the following methods: Figure 2 and Figure 3a Taking the electronic device with the structure shown as an example, and in conjunction with the accompanying drawings and application scenarios, the video processing method provided in this application embodiment will be specifically described.

[0148] First, it should be noted that the video processing method provided in this application embodiment can be applied to various scenarios. In one embodiment, the method can be applied to a video playback scenario. For example, Figure 4 This is a schematic diagram illustrating an application scenario of a video processing method provided in an embodiment of this application. Taking a mobile phone as an example, as... Figure 4As shown in (1), the mobile phone runs a video application, and an SDR video is playing on its display interface. In response to user operation (e.g., swiping from the right side of the interface to the left), the mobile phone displays a control window 401, in which an "Enable HDR Effect" option 4011 is displayed. In response to the user's operation of enabling the "Enable HDR Effect" option 4011, the mobile phone increases the screen backlight brightness and uses the video processing method provided in this embodiment to process the playing SDR video to expand the dynamic range of the video, so that the SDR video played by the mobile phone presents an HDR picture effect. (See also...) Figure 4 As shown in (2).

[0149] In another embodiment, the method can also be applied to live streaming scenarios. During a user's live stream viewing, in response to the user's "Enable HDR effect" option, the phone increases the screen backlight brightness and uses the video processing method provided in this embodiment to process the live stream video, thereby expanding the dynamic range of the live stream video and enabling it to display an HDR image effect.

[0150] In another embodiment, the method can also be applied to a web browsing scenario, where the webpage includes a short video. During the playback of the short video, in response to the user's "Enable HDR effect" option, the mobile phone increases the screen backlight brightness and uses the video processing method provided in this embodiment to process the short video, thereby expanding its dynamic range and enabling it to display an HDR image effect.

[0151] Of course, the video processing method provided in this application is not limited to the above scenarios. In any other scenario where the dynamic range of a video needs to be extended, the video processing method provided in this application can be used to extend the dynamic range of the video to improve the video display effect. For example, when receiving a user instruction to extend the dynamic range of the video, the mobile phone increases the screen backlight brightness and extends the dynamic range of the saved SDR video. In summary, this application does not limit the application scenarios of this method.

[0152] The following are examples Figure 4 Taking the application scenario shown as an example, the video processing method provided in the embodiments of this application will be explained and illustrated. Figure 5 The diagram shows the interaction between each module. (Refer to...) Figure 5 The video processing method provided in this application embodiment specifically includes the following steps:

[0153] S501, the video application sends a video stream to the codec module.

[0154] Here, video stream refers to the streaming data corresponding to the video currently being played by the video application.

[0155] S502, the encoding / decoding module performs decoding operations on the video stream to obtain video image frames.

[0156] For example, the video stream is a video stream with H.264 compression format. The encoding and decoding module performs decoding operations on the video stream to obtain a YUV video stream, that is, to obtain video image frames in YUV format.

[0157] S503, the encoding / decoding module sends the decoded video image frames sequentially to the image format conversion module.

[0158] S504, the image format conversion module converts the format of the current video image frame from YUV to RGB, and sends the current video image frame in RGB format to the LUT processing module.

[0159] For each YUV format video frame, the image format conversion module sequentially converts it into an RGB format video frame. The current video frame mentioned here refers to the video frame that the image format conversion module is currently processing.

[0160] S505, the image color correction module performs color correction on the current video image frame according to the preset 3D LUT, and sends the color-corrected current video image frame to the display.

[0161] In this embodiment, for each RGB format video image frame, the image color correction module performs color correction processing according to a preset 3DLUT (or default 3D LUT) alignment, and then sequentially sends the video image frames color-corrected based on the preset 3D LUT to the display. That is, the video played on the display screen is a video that has undergone color correction processing based on the preset 3D LUT.

[0162] S506, in response to user operation, the video application sends a backlight brightness adjustment instruction to the brightness service module and a dynamic range extension instruction to the image color correction module.

[0163] Reference Figure 4 As shown, user actions, such as enabling the "Enable HDR" option in a video application, can be examples of user actions.

[0164] Among them, the backlight brightness adjustment indicator can be used to instruct the brightness service module to increase the backlight brightness of the display, such as adjusting the backlight brightness of the display to its maximum supported brightness value, such as 1000 nits.

[0165] To protect the display and extend its lifespan, the system can also preset a maximum allowed brightness value for the display. This maximum allowed brightness value is less than the maximum supported brightness value of the display, and the difference between the maximum supported brightness value and the maximum allowed brightness value is less than a threshold. For example, if the maximum supported brightness value of the display is 1000 nits, the maximum allowed brightness value is 900 nits. In this case, the backlight brightness adjustment indicator can be used to instruct the brightness service module to increase the backlight brightness of the display to the maximum allowed brightness value, such as 900 nits.

[0166] The dynamic range extension indicator can be used to instruct the image color grading module to perform color grading on video image frames based on the dynamically range extended 3D LUT, thereby expanding the dynamic range of the video image frames. In other words, the dynamic range extension indicator can instruct the image color grading module to no longer perform color grading on video image frames based on the preset 3D LUT.

[0167] S507, the brightness service module increases the backlight brightness of the display screen according to the backlight brightness adjustment instruction.

[0168] For example, the brightness service module controls the display driver to increase the backlight brightness of the display screen according to the backlight brightness adjustment instruction, thereby the display driver can control the display screen to increase its backlight brightness.

[0169] In this embodiment, the brightness service module can increase the backlight brightness of the display screen to a preset maximum brightness value, such as 900 nits, according to the backlight brightness adjustment instruction.

[0170] Understandably, the final display of a video requires adjustments and transmission based on the backlight brightness of the display screen. Increasing the backlight brightness of the display screen before extending the dynamic range of the video allows for better highlighting of the brightness differences between brighter and darker pixels when displaying video frames after dynamic range extension. This, in turn, better reflects the extended dynamic range of the video and improves the display quality.

[0171] S508, the image color adjustment module sends the current video image frame in RGB format to the LUT processing module.

[0172] After receiving the dynamic range extension instruction, the image color grading module no longer performs color grading on the current video image frame based on the preset 3D LUT. Instead, before color grading the current video image frame, it sends the current video image frame to the LUT processing module to obtain the target 3D LUT with extended dynamic range corresponding to the current video image frame. Then, it performs color grading on the current video image frame based on the target 3D LUT to achieve the effect of extending the dynamic range of the current video image frame.

[0173] S509, the LUT processing module performs dynamic range expansion processing on the preset 3D LUT according to the current video image frame to obtain the target 3D LUT corresponding to the current video image frame, and sends the target 3D LUT corresponding to the current video image frame to the image color adjustment module.

[0174] like Figure 6 As shown, the LUT processing module performs dynamic range expansion processing on the preset 3D LUT based on the current video image frame to obtain the target 3D LUT corresponding to the current video image frame. The process specifically includes:

[0175] S5091, the LUT processing module calculates the grayscale histogram (Hist) of the current video image frame.

[0176] The LUT processing module acquires the current video image frame and generates a grayscale image of that frame. A grayscale image, also known as a luminance image or grayscale map, contains multiple pixels, each corresponding to a grayscale value. The pixels in the grayscale image correspond one-to-one with the pixels in the video image frame. The number of grayscale values ​​is N, where N = 256, and the grayscale values ​​range from 0 to 255.

[0177] Then, the LUT processing module counts the gray levels of each pixel in the grayscale image of the current video image frame to obtain the grayscale histogram of the video image frame. In the grayscale histogram, the horizontal axis represents the grayscale value, and the vertical axis represents the number of pixels corresponding to the grayscale value.

[0178] S5092, the LUT processing module regularizes the grayscale histogram of the current video image frame to obtain the regularized image of the current video image frame.

[0179] The LUT processing module regularizes the grayscale histogram of the current video image frame to obtain a regularized graph of the current video image frame. In the regularized graph, the horizontal axis represents the grayscale value, and the vertical axis represents the regularization value corresponding to the grayscale value.

[0180] In one implementation, the LUT processing module can regularize the grayscale histogram according to the following formula:

[0181]

[0182] Among them, H norm(i) represents the value corresponding to any gray level (i represents the gray level value) in the regularized histogram. It is the value obtained after regularizing the number of pixels corresponding to gray level i in the gray-level histogram, specifically the regularized value corresponding to the number of pixels corresponding to the gray level value. Hereinafter, it is simply referred to as the regularized value corresponding to gray level i. H(i) represents the number of pixels corresponding to gray level i in the gray-level histogram, that is, the number of pixels corresponding to gray level i before regularization. m H represents the average number of pixels corresponding to each gray level in the gray-level histogram. It is the sum of the number of pixels corresponding to each gray level in the gray-level histogram, divided by N (256). In other words, H... m This is the ratio of the total number of pixels in a grayscale image to the total number of grayscale values.

[0183] After the LUT processing module calculates the grayscale histogram of the current video image frame, the grayscale curve of the current video image frame can be obtained. Then, the grayscale curve of the current image frame can be regularized to obtain the regularized curve of the current video image frame. In this embodiment, the regularization graph can be either a regularized histogram or a regularized curve. It is understood that the regularized curve of the current video image frame can be obtained from its regularized histogram, and the regularized histogram of the current video image frame can also be obtained from its regularized curve. The regularization value corresponding to a certain grayscale value described below can also be determined based on the regularized curve or the regularized histogram; this embodiment does not limit this.

[0184] In this embodiment, after the LUT processing module calculates the grayscale histogram of the current video image frame, it can further filter the grayscale histogram to obtain a filtered grayscale curve corresponding to the current video image frame. Then, the LUT processing module can perform regularization processing on the grayscale curve to obtain a regularized curve corresponding to the current video image frame. Optionally, the LUT processing module can use mean filtering or Gaussian filtering, etc., to filter the grayscale histogram of the current video image frame; this embodiment does not limit the filtering method.

[0185] In one example Figure 7 Example (1) shows a grayscale histogram of a video image frame, where the horizontal axis represents the grayscale value and the vertical axis represents the number of pixels corresponding to the grayscale value. The LUT processing module filters the grayscale histogram to obtain the grayscale curve of the video image frame, as shown below. Figure 7 As shown in Figure (2), the horizontal axis represents the grayscale value, and the vertical axis represents the number of pixels corresponding to the grayscale value. Furthermore, the LUT processing module processes data such as... Figure 7 The grayscale curve shown in (2) is regularized to obtain the regularized curve of the video image frame, as shown in Figure 2. Figure 7As shown in (3), the horizontal axis represents the grayscale value, and the vertical axis represents the regularization value corresponding to the grayscale value. Figure 7 As can be seen from (3), each gray value corresponds to a regularization value, and the gray value and its corresponding regularization value define a point in the regularization graph.

[0186] Understandably, grayscale values ​​of 0-255 can be normalized to grayscale values ​​(or brightness values) of 0-1, and thus... Figure 7 The horizontal coordinates of (1) to (3) can also be represented by brightness values ​​of 0-1.

[0187] S5093, the LUT processing module determines the target dynamic range expansion factor (Err_target) corresponding to the current video image frame based on the average pixel level (APL) of the current video image frame and the backlight brightness clearance ratio of the display screen.

[0188] Among them, the display backlight brightness clearance ratio (Err_cwy) is used to indicate the proportion of increase that the display backlight brightness supports or allows.

[0189] In one implementation, the display backlight brightness clearance ratio can be the ratio of the maximum brightness value supported by the display to the current brightness value of the display backlight. For example, if the maximum brightness value supported by the display is 1000 nits and the current brightness value of the display backlight is 300 nits, then the display backlight brightness clearance ratio is 1000 / 300.

[0190] In another implementation, the backlight brightness clearance ratio can be the ratio of the maximum allowable brightness value of the display to the current brightness value of the display backlight. For example, if the maximum allowable brightness value of the display is 900 nits and the current brightness value of the display backlight is 300 nits, then the backlight brightness clearance ratio is 900 / 300.

[0191] In one specific implementation, if the backlight brightness adjustment indicator is used to instruct the brightness service module to adjust the backlight brightness of the display to its maximum supported brightness value, then the display backlight brightness clearance ratio is calculated based on the maximum supported brightness value of the display and the current brightness value of the display backlight; if the backlight brightness adjustment indicator is used to instruct the brightness service module to adjust the backlight brightness of the display to its maximum allowed set brightness value, then the display backlight brightness clearance ratio is calculated based on the maximum allowed set brightness value of the display and the current brightness value of the display backlight.

[0192] In this embodiment, the LUT processing module can calculate the undetermined dynamic range expansion factor (Err_tbd) corresponding to the current video image frame based on the average brightness of the current video image frame. If the undetermined dynamic range expansion factor is less than or equal to the backlight brightness-to-clearance ratio of the display screen, the LUT processing module uses the undetermined dynamic range expansion factor (Err_tbd) corresponding to the current video image frame as the target dynamic range expansion factor (Err_target) corresponding to the current video image frame. If the undetermined dynamic range expansion factor is greater than the backlight brightness-to-clearance ratio of the display screen, the LUT processing module uses the backlight brightness-to-clearance ratio of the display screen (Err_cwy) as the target dynamic range expansion factor (Err_target) corresponding to the current video image frame.

[0193] In this embodiment, the LUT processing module can calculate the undetermined dynamic range expansion factor (Err_tbd) corresponding to the current video image frame based on the following formula:

[0194]

[0195] Where APL is the average brightness of all pixels in the video image frame, th1 is the preset pixel brightness threshold, Err1 is a preset dynamic range expansion factor (e.g., 3.0), and m, n, and k are preset coefficients.

[0196] In this embodiment, Err1 can be understood as a preset maximum dynamic range expansion factor, and the coefficients m, n, and k are determined based on a preset minimum dynamic range expansion factor (Err2), where Err2 is less than Err1. Furthermore, the undetermined dynamic range expansion factor (Err_tbd) corresponding to the current video image frame, calculated by the LUT processing module according to the above formula, is less than or equal to the preset maximum dynamic range expansion factor (Err1) and greater than or equal to the preset minimum dynamic range expansion factor (Err2).

[0197] In the application embodiment, the target dynamic range expansion factor (Err_target) is used to calculate the coefficient C that needs to be compressed in terms of image pixel brightness value after the display backlight brightness is increased.

[0198] Where C = 1 / Err_target 1 / γ γ is the gamma coefficient. For example, γ = 2.2.

[0199] In this embodiment, the compression coefficient C is specifically used to compress the pixel brightness value of the preset 3D LUT when increasing the backlight brightness of the display screen. The compression coefficient C is matched with the backlight brightness of the display screen to avoid over-expansion (also known as stretching) when the dynamic range of the video image frame is expanded based on the target 3D LUT obtained after processing, which would cause an uneven display effect. It also ensures that the dynamic range of the finally expanded video image frame can match the display capability of the display screen, thereby improving the display effect of the video.

[0200] The LUT processing module compresses the pixel brightness values ​​of a preset 3D LUT by compressing the pixel brightness values ​​of each channel (R channel, G channel, and B channel) of the preset 3D LUT separately. The compressed pixel brightness value of each channel is I′=I*C.

[0201] To make it easier to see the compression effect of pixel values Figure 8 This example demonstrates how to compress image pixel values ​​when increasing display backlight brightness. Figure 8 This is a color image, but it is displayed in grayscale here. For example... Figure 8 As shown in Figure (1), when the backlight brightness of the display screen is 300 nits, the display effect of the image on the display screen is shown in Figure 801. When the backlight brightness of the display screen is increased to 900 nits, the pixel brightness value of the image is compressed, and the display effect of the image with compressed pixel brightness value on the display screen is shown in Figure 802.

[0202] S5094, the LUT processing module determines whether color banding will occur in the current video image frame after dynamic range expansion. If not, S5095 is executed; if so, S5096 is executed.

[0203] Color banding, also known as color banding, quantization banding, or color breaks, often occurs in clean gradients within an image, such as a sunset sky, studio backgrounds, or shadows. For example, a clear blue sky may appear broken into layers of color bands. Color banding occurs because the image's color depth is insufficient to represent subtle gradations, causing what should be a smooth gradient to appear as broken, fragmented sections.

[0204] To reduce or even avoid color banding in video image frames after dynamic range expansion due to overstretching, in this embodiment, the LUT processing module first determines whether color banding will occur in the video image frame after dynamic range expansion, and uses different methods to calculate the inverse tone mapping curve corresponding to the video image frame based on the determination result, so as to obtain the grayscale gain map (or brightness gain map) corresponding to the video image frame.

[0205] In one implementation, for the current video image frame, the LUT processing module obtains its regularization map and searches for regularization value peaks in the highlighted areas of the regularization map (i.e., pixel areas with grayscale values ​​greater than the grayscale segmentation threshold th3 mentioned below). The LUT processing module determines whether there is a regularization value peak greater than the threshold th2 among the peak values. If not, the LUT processing module determines that the current video image frame will not produce color banding after dynamic range expansion. If there is at least one regularization value peak greater than the threshold th2, the LUT processing module determines that the current video image frame will produce color banding after dynamic range expansion.

[0206] Considering that if a certain grayscale pixel has a high proportion in the bright area of ​​the image, color banding is likely to occur after the dynamic range of the image is expanded, this embodiment determines whether the current video image frame will not produce color banding after the dynamic range is expanded based on whether there is a regularization value peak greater than the threshold th2 in the bright area.

[0207] S5095, the LUT processing module calculates the inverse tone mapping curve corresponding to the current video image frame, and obtains the grayscale gain map corresponding to the current video image frame based on the inverse tone mapping curve, and executes S5097.

[0208] The inverse tone mapping curve is used to determine the grayscale gain value (or brightness gain value) corresponding to the pixel grayscale. The grayscale gain value characterizes the degree of increase in pixel grayscale (or brightness), or in other words, the amount of improvement in pixel grayscale. Understandably, the larger the grayscale gain value, the greater the improvement in pixel brightness; conversely, the smaller the grayscale gain value, the less the improvement in pixel brightness.

[0209] Therefore, the LUT processing module can calculate the grayscale gain map corresponding to the current video image frame based on the inverse tone mapping curve corresponding to the current video image frame. In this embodiment, the grayscale gain map corresponding to the current video image frame is used to perform grayscale gain on each channel of the preset 3D LUT to obtain the target 3D LUT corresponding to the current video image frame. The grayscale gain map includes multiple pixels, each pixel corresponding to a grayscale gain value. The pixels in the grayscale gain map correspond one-to-one with the pixels in the video image frame.

[0210] In this embodiment, each video image frame is divided into a highlight region and a diffuse region. Brightness enhancement is performed only on the highlight region of the video image frame to expand its dynamic range. It is understood that no dynamic range expansion is performed on the diffuse region of the video image frame; that is, the grayscale gain value corresponding to the diffuse region of the video image frame can be set to 0. Thus, the LUT processing module can obtain the grayscale gain map corresponding to the current video image frame based on the grayscale gain values ​​of the highlight region and the diffuse region.

[0211] The LUT processing module obtains the grayscale segmentation threshold th3 and determines the highlight pixels and diffuse pixels in the current video image frame based on th3. Highlight pixels are those with a grayscale value greater than th3, while diffuse pixels are those with a grayscale value less than or equal to th3. Therefore, in the current video image frame, the area composed of highlight pixels is the highlight area, and the area composed of diffuse pixels is the diffuse area.

[0212] Among them, the grayscale segmentation threshold th3, also known as the brightness segmentation threshold, is the grayscale value used to distinguish between bright pixels and diffuse pixels.

[0213] Optionally, the grayscale segmentation threshold th3 can be a preset value, such as 0.5 (or 255*0.5).

[0214] Optionally, the grayscale segmentation threshold th3 can also be the average grayscale value G of the grayscale map of the video image frame. ave Among them, the average grayscale value G of the grayscale image ave It refers to the average grayscale value of each pixel in a grayscale image.

[0215] Optionally, the grayscale segmentation threshold th3 can also be a preset minimum grayscale threshold th4 and the average grayscale value G of the grayscale image. ave The larger of the two values. For example, the preset minimum grayscale threshold th4 is 0.4 (or 255*0.4), and the grayscale segmentation threshold th3 is taken as the preset minimum grayscale threshold th4 and the average grayscale value G of the grayscale image. ave The maximum value in G, i.e., th3 = maximum(th4, G ave ).

[0216] In this implementation, a preset minimum grayscale threshold th4 and the average grayscale value G of the grayscale image are taken. ave The maximum value in the grayscale image is used as the grayscale segmentation threshold th3, compared to directly using the average grayscale value G of the grayscale image. aveUsing the grayscale segmentation threshold th3 as the grayscale segmentation threshold prevents the threshold from being too low when the video image frame is extremely dark. This avoids overstretching during dynamic range expansion of the video image frame, which would stretch low-brightness areas that don't need stretching, resulting in poor image quality. Compared to directly using the preset minimum grayscale threshold th4 as the grayscale segmentation threshold th3, this method prevents the threshold from being too low when the video image frame is extremely bright. This avoids overstretching during dynamic range expansion of the video image frame, which would stretch low-brightness areas that don't need stretching, resulting in poor image quality. In summary, this implementation, by combining the brightness of the video image frame with the preset minimum grayscale threshold th4 to determine the grayscale segmentation threshold th3, can more accurately distinguish between bright and diffuse pixels, preventing overstretching during dynamic range expansion and resulting in better image quality.

[0217] If the LUT processing module determines that the current video image frame will not produce color banding after dynamic range expansion, the LUT processing module uses a piecewise function method to determine the inverse tone mapping curve corresponding to the current video image frame, that is, to determine the inverse tone mapping curve corresponding to the highlight area in the current video image frame.

[0218] In this embodiment, when the LUT processing module determines that the current video image frame will not produce color banding after dynamic range expansion, the calculation formula for the inverse tone mapping curve Gain_1(i) corresponding to the current video image frame is as follows:

[0219]

[0220] Where i represents the normalized pixel gray value, i∈[0,1], Gain_1(i) represents the gray value gain corresponding to the gray value i, th3 represents the segmentation threshold used to divide the highlight area and the diffuse area, α represents the exponential coefficient, for example α=2.2, v1 and v2 are two preset gray segmentation thresholds, v2>v1, a, b, c, d, e, f are preset coefficients, and a, b, d are positive numbers.

[0221] As can be seen from the above formula, in the first gray-level interval, i.e., when i∈[0, th3), the gray-level gain value is 0, which is equivalent to not dynamically stretching the diffuse area of ​​the image; in the second gray-level interval, i.e., when i∈(th3, v1), the gray-level gain value is positively correlated with the gray-level difference (i.e., the difference between the gray-level value and the threshold th3) ​​(or exponentially related), which can ensure the continuity of the inverse tone mapping curve and the curve slope; in the third gray-level interval, i.e., when i∈[v1, v2), the gray-level gain value is linearly related to the gray-level value, which achieves the effect of controlling the slope of the curve, that is, limiting the slope of the inverse tone mapping curve in the mid-high brightness area to b; in the fourth gray-level interval, i.e., when i∈[v2, 1], the gray-level gain value is obtained by accumulating two parts, one part is positively correlated with the gray-level value, and the other part is linearly related to the gray-level value, which can stretch the slope of the inverse tone mapping curve in the high brightness area, so that the high brightness area of ​​the image will have more layering after the dynamic range is expanded.

[0222] Thus, because the slope of the inverse tone mapping curve Gain_1(i) in this embodiment is limited, the problem of unnatural image display after dynamic range expansion of video image frames due to excessively large slopes of the inverse tone mapping curve can be avoided. For bright pixels, the larger the gray value, the larger the corresponding gray value gain, and the greater the increase in gray value gain. Therefore, applying the obtained gray value gain map to the corresponding video image frame can better expand the dynamic range of the image, highlight the brightness difference between brighter and darker pixels, and improve the display effect of the video image frame.

[0223] Understandably, if i represents the pixel grayscale value, i∈[0,255], then if the LUT processing module determines that the current video image frame will not produce color banding after dynamic range expansion, the calculation formula for the inverse tone mapping curve Gain_1(i) can be adjusted as follows:

[0224]

[0225] S5096, the LUT processing module determines each brightness fluctuation range of each current video image frame based on the regularization value peak, and generates an inverse tone mapping curve of the dequantization band for each brightness fluctuation range, so as to obtain the grayscale gain map corresponding to the current video image frame according to the inverse tone mapping curve, and executes S5097.

[0226] Since color breaks are likely to occur at the gray values corresponding to the peaks of the regularization values, in this embodiment, the highlight interval is divided into multiple brightness fluctuation intervals based on the gray values corresponding to the peaks of the regularization values. When the LUT processing module determines that color breaks will occur after the dynamic range of the current video image frame is extended, the LUT processing module obtains the number of peaks of the regularization values greater than the threshold th2, and divides the highlight interval of the current video image frame into multiple brightness fluctuation intervals according to this number. Among them, the highlight interval of the video image frame refers to the range of gray values of the highlight pixel points. If i represents the pixel gray value and i ∈ [0, 1], then the highlight interval of the video image frame is the interval (th3, 1]; if i represents the pixel gray value and i ∈ [0, 255], then the highlight interval of the video image frame is the interval (th3, 255].

[0227] The following takes the highlight interval of the video image frame as the interval (th3, 1] as an example for explanation.

[0228] If, in the regularization map of the current video image frame, the number of peaks of the regularization values greater than the threshold th2 is 1, and assuming that the gray value corresponding to this peak of the regularization value is G, the LUT processing module can divide the highlight interval of the current video image frame into two brightness fluctuation intervals based on this peak of the regularization value, namely interval 1 (th3, G] and interval 2 (G, 1].

[0229] If, in the regularization map of the current video image frame, the number of peaks of the regularization values greater than the threshold th2 is num, then the LUT processing module detects the valleys of the regularization values between every two adjacent peaks of the regularization values, that is, num - 1 valleys of the regularization values. Furthermore, the LUT processing module can divide the highlight interval of the current video image frame into 2 * num brightness fluctuation intervals based on these num peaks of the regularization values and (num - 1) valleys of the regularization values.

[0230] Optionally, if there are multiple valleys of the regularization values between two adjacent peaks of the regularization values, the valley of the regularization value with the smallest value can be used as the valley of the regularization value for dividing the brightness fluctuation intervals. [[ID=*13]]

[0231] For example, in the regularization map of the current video image frame, the number of peaks of the regularization values greater than the threshold th2 is 2. Assuming that the gray values corresponding to these two peaks of the regularization values are G1 and G3 respectively, and the gray value corresponding to the valley of the regularization values between these two peaks of the regularization values is G2, and G1 < G2 < G3. At this time, the LUT processing module can divide the highlight interval of the current video image frame into 4 brightness fluctuation intervals based on these 2 peaks of the regularization values and 1 valley of the regularization values, namely interval 1 (th3, G1], interval 2 (G1, G2], interval 3 (G2, G3], and interval 4 (G3, 1].

[0232] If the LUT processing module determines that the current video frame will produce color banding after dynamic range expansion, then for any given luminance fluctuation range, let's decolorize it as a luminance fluctuation range (S1, S2]. Within the luminance fluctuation range (S1, S2), the third-order inverse tone mapping curve Gain_2(i) for decolorizing the current video frame can be:

[0233] Gain_2(i)=m*i 3 +n*i 2 +p*i+q, i∈(S1,S2).

[0234] Where i represents the pixel grayscale value, Gain_2(i) represents the grayscale gain value corresponding to the grayscale value i, and m, n, p, and q are coefficients.

[0235] In this embodiment, the inverse tone mapping curve Gain_2(i) for de-color banding is preset as a third-order curve. The purpose is to reduce the slope of the inverse tone mapping curve near the banding gray level by controlling the slope at the endpoints S1 and S2 of the gray level range, thereby weakening the color banding. For example, assuming gray level S1 is the gray level corresponding to the regularization value trough and gray level S2 is the gray level corresponding to the regularization value peak, a color banding may exist at gray level S2. In this embodiment, the slope of the inverse tone mapping curve Gain_2(i) is smaller at gray level S2 and larger at gray level S1. In the interval (S1, S2], the closer the gray level is to S2, the smaller the slope of the inverse tone mapping curve Gain_2(i), resulting in a smaller amplitude of dynamic range stretching of the image, thus achieving the effect of weakening the color banding.

[0236] For the brightness fluctuation range (S1, S2], the values ​​of Gain_2(S1), Gain_2(S2), Gain_2'(S1), and Gain_2'(S2) are calculated respectively. This allows us to solve for the coefficients m, n, p, and q in the above formula, and thus obtain the third-order inverse tone mapping curve for the current video image frame within the brightness fluctuation range (S1, S2]. Here, Gain_2'(i) is the first derivative of the function Gain_2(i).

[0237] In this embodiment, the formula for calculating Gain_2(S2) is as follows:

[0238]

[0239] Where Gain_1′(i) is the derivative of Gain_1(i); i represents the normalized pixel gray value, i∈[0,1]; Δ is used to represent the division interval of pixel gray values, Δ=1 / (N-1), N is the total number of gray levels, when N=256, Δ=1 / 255; H norm ′(i)=maximum(H norm (i)-Tq,0), where Tq is a constant, H norm (i) is the regularization value of the number of pixels corresponding to the gray value i in the gray histogram.

[0240] If the brightness fluctuation interval (S1,S2) is the first interval in the bright interval (th3,1), that is, S1 = th3, then Gain_2(S1) = Gain_1(S1). That is, Gain_2(th3) = Gain_1(th3) = 0.

[0241] If the brightness fluctuation interval (S1, S2) is not the first interval in the highlight interval (th3, 1), that is, S1 ≠ th3, then Gain_2(S1) can also be determined based on the previous brightness fluctuation interval (S3, S1). Within the brightness fluctuation interval (S3, S1), the calculation formula for Gain_2(S1) can refer to the aforementioned calculation formula for Gain_2(S2), that is:

[0242] That is,

[0243] In the formula for calculating Gain_2(S1), the method for determining the value of Gain_2(S3) can also refer to the aforementioned method for determining the value of Gain_2(S1), and so on. It will not be repeated here.

[0244] From the formula for calculating Gain_2(S2) above, we can see that:

[0245]

[0246] Similarly, if the brightness fluctuation interval (S1, S2) is the first interval in the bright interval (th3, 1), that is, S1 = th3, then Gain_2′(S1) = Gain_1′(S1). If the brightness fluctuation interval (S1, S2) is not the first interval in the bright interval (th3, 1), that is, S1 ≠ th3, then, according to the formula for calculating Gain_2(S1) above:

[0247]

[0248] Thus, after calculating the values ​​of Gain_2(S1), Gain_2(S2), Gain_2'(S1), and Gain_2'(S2), the coefficients m, n, p, and q in the above formula can be solved, and then the third-order inverse tone mapping curve Gain_2(i) = m*i for decolorization of the current video image frame within the brightness fluctuation range (S1, S2] can be obtained. 3 +n*i 2 +p*i+q.

[0249] Based on the aforementioned inverse tone mapping curve Gain_1(i), to avoid color banding after the dynamic range of video image frames expands, this embodiment compresses the grayscale gain value of pixels when calculating the inverse tone mapping curve Gain_2(i). As shown in the formula for calculating Gain_2(S2) above, within a brightness fluctuation range, this embodiment calculates Gain_2(S2) using calculus based on the known Gain_2(S1). Here, Δ represents the grayscale interval during calculus calculation. For each grayscale interval, its slope is... This can be understood as through The original slope Gain_1′(i) was compressed. It can also be understood as the weight of the original slope Gain_1′(i).

[0250] in, The value range is (0,1], which can achieve the effect of compressing the original slope Gain_1′(i), and thus the effect of compressing the grayscale gain value.

[0251] H norm ′(i) is based on formula H norm ′(i)=maximum(H norm (i)-Tq,0) is determined. Where, if the regularization value H corresponding to pixel grayscale value i is... norm (i) If it is greater than Tq, then H norm The value of ′(i) is H norm (i) The difference between Tq and H, otherwise H norm The values ​​of ′(i) are all 0. Therefore, in the highlighted region, when the regularization value H corresponding to the grayscale value i is 0... norm (i) When it is large (greater than Tq), it indicates that the number of pixels corresponding to this grayscale value is too large, and color banding is likely to occur after pixel stretching. norm ′(i) takes a positive value, and the calculated result is A value less than 1 can achieve the effect of compressing the grayscale gain value; when the regularization value H corresponding to the grayscale value i is less than 1, it can achieve the effect of compressing the grayscale gain value. norm(i) When it is small (less than or equal to Tq), it indicates that the number of pixels corresponding to this grayscale value is relatively small, and the pixels are easy to stretch without discontinuities, then H norm ′(i) takes the value 0, and the calculated result is Since the values ​​are all 1, there is no need to perform slope compression at this time; simply maintain the initial slope Gain_1′(i).

[0252] As an optional implementation, the formula for calculating Gain_2(S2) is as follows:

[0253]

[0254] Among them, H norm The formula for calculating ′(i) can be adjusted as follows:

[0255] H norm ′(i)=maximum(H norm "(i)-Tq,0).

[0256] in, th5 is a preset threshold, H norm (i) is the regularization value of the number of pixels corresponding to the gray value i in the gray histogram.

[0257] For video frames with high regularization values ​​in highlight areas, meaning those frames exhibit color banding after dynamic range expansion, the calculation of their corresponding inverse tone mapping curve is limited by slope (i.e., through...). (Displaying the slope of the curve), thus pre-processing the regularization graph can enhance the brightness variation of the highlight area of ​​the image, thereby making the difference of the slope limit (or slope weight) at different gray values ​​in the highlight area more obvious, thereby improving the accuracy of the inverse tone mapping curve Gain_2(i) and ensuring the decolorization effect after the dynamic range of the video image frame is expanded.

[0258] If the number of regularization peaks greater than the threshold th2 in the regularization graph of the current video image frame is 1, and assuming the gray value corresponding to this regularization peak is G, then the third-order inverse tone mapping curve Gain_2(i) corresponding to the decolorization of the current video image frame is as follows:

[0259]

[0260] After calculating the values ​​of Gain_2(th3), Gain_2(G), Gain_2'(th3), and Gain_2'(G), the coefficients m1, n1, p1, and q1 in the above formula can be solved, and then the third-order inverse tone mapping curve for decolorization of the current video image frame within the brightness fluctuation range (th3, G] can be obtained:

[0261] Gain_2(i)=m1*i 3 +n1*i 2 +p1*i+q1.

[0262] Similarly, after calculating the values ​​of Gain_2(G), Gain_2(1), Gain_2'(G), and Gain_2'(1), the coefficients m2, n2, p2, and q2 in the above formulas can be solved, and then the third-order inverse tone mapping curve for decolorization of the current video image frame within the brightness fluctuation range (G,1] can be obtained:

[0263] Gain_2(i)=m2*i 3 +n2*i 2 +p2*i+q2.

[0264] If in the regularization graph of the current video image frame, the number of regularization value peaks greater than the threshold th2 is num, and assuming that the gray values ​​corresponding to each regularization value peak and valley are G1, G2, ..., G... 2*num-1 (There are 2*num-1 elements in total, sorted from smallest to largest grayscale value). The third-order inverse tone mapping curve Gain_2(i) corresponding to the current video image frame after decolorization is as follows:

[0265]

[0266] After calculating the values ​​of Gain_2(th3), Gain_2(G1), Gain_2'(th3), and Gain_2'(G1), the coefficients m1, n1, p1, and q1 in the above formula can be solved, and then the third-order inverse tone mapping curve for decolorization of the current video image frame within the brightness fluctuation range (th3, G] can be obtained:

[0267] Gain_2(i)=m1*i 3 +n1*i 2 +p1*i+q1.

[0268] Similarly, after calculating the values ​​of Gain_2(th3), Gain_2(G1), Gain_2'(th3), and Gain_2'(G1), the coefficients m1, n1, p1, and q1 in the above formula can be solved, and then the third-order inverse tone mapping curve for decolorization of the current video image frame within the brightness fluctuation range (G1, G2] can be obtained:

[0269] Gain_2(i)=m2*i 3 +n2*i 2 +p2*i+q2.

[0270] And so on, after calculating Gain_2(G) 2*num-1 ),Gain_2(1),Gain_2'(G 2*num-1 After obtaining the values ​​of Gain_2'(1), the coefficient m in the above formula can be solved. 2*num n 2*num p 2*num q 2*num Therefore, it can be concluded that the current video image frame falls within the brightness fluctuation range (G). 2*num-1 [1] Third-order inverse hue mapping curve for decolorization:

[0271] Gain_2(i)=m 2*num *i 3 +n 2*num *i 2 +p 2*num *i+q 2*num .

[0272] Understandably, since the diffuse region of the video image frame does not require dynamic range expansion, the gray gain value corresponding to the diffuse region of the video image frame is set to 0 in the calculation formula of the third-order inverse tone mapping curve Gain_2(i) above, that is, the gray gain value corresponding to the gray range [0, th3] is set to 0.

[0273] In order to prevent color banding from occurring during dynamic range stretching of video image frames, when the LUT processing module determines that color banding will occur after the current video image frame is dynamically stretched, it uses the above-mentioned de-color banding method to calculate the inverse tone mapping curve corresponding to the current video image frame in segments, so that the current video image frame will not have color banding after dynamic range stretching, thus improving the picture quality of the video image frame.

[0274] In an example ( Figure 9 In (shown as a grayscale image), such as Figure 9 The image shown in (1) is an image with pixel value compression completed after the display screen backlight is increased. For example... Figure 9By performing dynamic range extension on the image shown in (1), we can obtain the following: Figure 9 The image shown in (2). Figure 9 As shown in (2), color banding appears in image region 901. If the color banding removal method provided in this embodiment is used to calculate the inverse tone mapping curve corresponding to the current video image frame in segments, a grayscale gain map corresponding to the current video image frame can be obtained, and the grayscale gain map can be used to perform color banding on the image region 901. Figure 9 By performing dynamic range extension on the image shown in (1), we can obtain the following: Figure 9 The image shown in (3) is as follows: Figure 9 In the image shown in (3), there is no longer any such... Figure 9 The color fault shown in region 901 of (2) is shown.

[0275] After the LUT processing module calculates the inverse tone mapping curve corresponding to the current video image frame, the grayscale gain map corresponding to the current video image frame can be obtained based on the inverse tone mapping curve.

[0276] In an example ( Figure 10 (Shown in grayscale image format) Figure 10 Example (1) shows a video image frame, and the corresponding grayscale gain diagram calculated using the method provided in this embodiment is shown in (2) of 10. Figure 10 The grayscale gain map shown in (2) acts on, for example, Figure 10 The original video image frame shown in (1) can be referenced to the video image frame with extended dynamic range. Figure 10 As shown in (3).

[0277] Continue to refer to Figure 8 For example, after increasing the backlight brightness of the display to 900 nits and compressing the pixel brightness values ​​of the image, the brightness of the image pixels is stretched to expand the dynamic range of the image. The display effect of the image with the expanded dynamic range on the display screen can be as shown in Figure 803.

[0278] Thus, this application embodiment improves backlight brightness and reduces image pixel grayscale based on the backlight brightness clearance ratio of the display screen. At the same time, it uses video image frame information to divide the video image frame into bright areas and diffuse areas, and only applies brightness gain to the bright areas to improve image contrast and brightness, preserve image details, and thereby improve the dynamic range of the image.

[0279] Considering the continuity of video playback, such as the fact that the content of the current video frame changes little compared to the previous video frame in the absence of scene switching (e.g., the similarity between the current video frame and the previous video frame is greater than or equal to the similarity threshold), this embodiment can use the inverse tone mapping curve of the previous video frame as the inverse tone mapping curve corresponding to the current video frame, and use the target dynamic range expansion factor (Err_target) of the previous video frame as the target dynamic range expansion factor (Err_target) corresponding to the current video frame, thereby reducing the amount of computation.

[0280] Alternatively, in this embodiment, if the similarity between the current video image frame and its previous video image frame is greater than or equal to the similarity threshold, the inverse tone mapping curve of the previous video image frame can be used as the inverse tone mapping curve corresponding to the current video image frame, and the target dynamic range expansion factor (Err_target) corresponding to the current video image frame can be recalculated to reduce the amount of computation.

[0281] In this embodiment, the LUT processing module can determine whether to recalculate the inverse tone mapping curve corresponding to the current video image frame based on the similarity between the current video image frame and its previous video image frame. Specifically, the LUT processing module can determine the similarity between two video image frames based on the similarity of their regularized graphs. If the similarity between the regularized graph of the current video image frame and the regularized graph of its previous video image frame is greater than or equal to a similarity threshold, the LUT processing module can use the inverse tone mapping curve of the previous video image frame as the inverse tone mapping curve corresponding to the current video image frame; otherwise, the LUT processing module recalculates the inverse tone mapping curve corresponding to the current video image frame.

[0282] The similarity Coh between the regularized graph Hcurrent of the current video image frame and the regularized graph Hitm of the previous video image frame is calculated using the following formula:

[0283]

[0284] Where Hcurrent(i) represents the regularization value corresponding to grayscale value i in the regularization graph of the current video image frame, and Hitm(i) represents the regularization value corresponding to grayscale value i in the regularization graph of the previous video image frame.

[0285] In this embodiment, before calculating the similarity Coh between the regularized graph Hcurrent of the current video image frame and the regularized graph Hitm of the previous video image frame, the regularized graphs Hcurrent and Hitm can be pre-processed to obtain the regularized graph Hcurrent″ of the current video image frame and the regularized graph Hitm″ of the previous video image frame, so that the similarity calculation can be performed based on the regularized graphs Hcurrent″ and Hitm″. At this time, the formula for calculating the similarity Coh can be adjusted as follows:

[0286]

[0287] Where Hcurrent″(i)=maximum(e Hcurrent(i) , th5), Hitm″(i)=maximum(e Hitm(i) ,th5).

[0288] Hcurrent″(i) represents the regularization value corresponding to grayscale value i in the regularization graph Hcurrent″ of the current video image frame, and Hitm″(i) represents the regularization value corresponding to grayscale value i in the regularization graph Hitm″ of the previous video image frame. th5 is a preset threshold.

[0289] For video frames with low regularization values ​​in highlighted areas, meaning those frames won't exhibit color banding after dynamic range expansion, the inverse tone mapping curve of the preceding video frame can be directly used as its corresponding inverse tone mapping curve. In this scenario, the similarity between the two frames is determined based on the pre-processed regularization maps Hcurrent' and Hitm'. A high similarity Coh value indicates that the two video frames are similar, and the inverse tone mapping curve of the preceding frame can be directly used as its corresponding inverse tone mapping curve.

[0290] S5097, the LUT processing module processes the preset 3D LUT according to the grayscale gain map corresponding to the current video image frame and the target dynamic range expansion factor corresponding to the current video image frame, to obtain the target 3D LUT corresponding to the current video image frame.

[0291] After obtaining the grayscale gain map corresponding to the current video image frame and the target dynamic range expansion factor corresponding to the current video image frame, the LUT processing module can process the preset 3D LUT to obtain the target 3D LUT corresponding to the current video image frame. Specifically, the target dynamic range expansion factor corresponding to the current video image frame is used to compress the pixel brightness values ​​of the preset 3D LUT; the grayscale gain map corresponding to the current video image frame is used to gain the pixel brightness values ​​of the preset 3D LUT.

[0292] When processing the pixel brightness values ​​of the preset 3D LUT, the LUT processing module processes the pixel brightness values ​​of each channel (R channel, G channel, and B channel) of the preset 3D LUT separately. The resulting pixel brightness value of each channel of the target 3D LUT is I″ = I*(Gain+1)*C.

[0293] Where C = 1 / Err_target 1 / γ γ is the gamma coefficient. For example, γ = 2.2.

[0294] Where Gain is Gain_1 or Gain_2. If the current video frame does not exhibit color banding after dynamic range expansion, then Gain in the above formula is the inverse tone mapping curve Gain_1. If the current video frame exhibits color banding after dynamic range expansion, then Gain in the above formula is the inverse tone mapping curve Gain_2.

[0295] In one example, the preset 3D LUT is shown in Table 1. The LUT processing module processes the preset 3D LUT according to the grayscale gain map corresponding to the current video image frame and the target dynamic range expansion factor corresponding to the current video image frame, and obtains the target 3D LUT corresponding to the current video image frame as shown in Table 2.

[0296] Table 1

[0297] R G B 0 0 0 64 0 0 128 0 0 192 0 0 … … … 1024 0 0 0 64 0 64 64 0 128 64 0 192 64 0 … … … 1024 64 0 … … … 896 1024 1024 960 1024 1024 1024 1024 1024

[0298] Table 2

[0299]

[0300]

[0301] To avoid video flicker, the LUT processing module can also perform temporal filtering on the inverse tone mapping results of the preset 3D LUT.

[0302] In the video processing method provided in the embodiment, S5097 may specifically include the following steps:

[0303] S50971, the LUT processing module processes the preset 3D LUT according to the grayscale gain map corresponding to the current video image frame (e.g., video image frame n) and the dynamic range expansion factor corresponding to the current video image frame (e.g., video image frame n), to obtain the initial 3D LUT corresponding to the current video image frame (e.g., video image frame n).

[0304] In the initial 3D LUT corresponding to the current video image frame (e.g., video image frame n), the pixel brightness value I″ for each channel is I*(Gain+1)*C.

[0305] S50972, the LUT processing module obtains the target 3D LUT corresponding to the previous frame of the current video image frame (such as video image frame n-1).

[0306] The target 3D LUT corresponding to the previous frame of the current video image frame is the 3DLUT applied to the previous frame. If the previous frame did not undergo dynamic range expansion, the target 3D LUT corresponding to it is the preset 3D LUT. If the previous frame underwent dynamic range expansion, the calculation method for the target 3D LUT corresponding to it is the same as that for the current video image frame.

[0307] S50973, the LUT processing module determines the target 3D LUT corresponding to the current video image frame (e.g., video image frame n-1) based on the target 3D LUT corresponding to the previous frame (e.g., video image frame n-1) and the initial 3D LUT corresponding to the current video image frame (e.g., video image frame n).

[0308] Assuming the target 3D LUT corresponding to video frame n-1 is Vitm, and the initial 3D LUT corresponding to video frame n is Vcurrent, in Vcurrent, the pixel brightness value I″ of each channel is I*(Gain+1)*C. The target 3D LUT corresponding to video image frame n is Vcurrent', and the formula for calculating Vcurrent' is as follows:

[0309] Vcurrent' = ρ * Vitm + (1 - ρ) * Vcurrent. Where ρ is a constant.

[0310] S510, the image color correction module performs color correction processing on the current video image frame according to the target 3D LUT corresponding to the current video image frame, and sends the color-corrected current video image frame to the display.

[0311] After obtaining the target 3D LUT corresponding to the current video image frame, the image color correction module can perform color correction processing on the current video image frame according to the target 3D LUT to achieve dynamic range expansion of the current video image frame, and then send the color-corrected current video image frame to the display.

[0312] like Figure 11 As shown, assuming the currently processed video image frame is the nth frame, when performing dynamic range expansion on the nth frame: First, the grayscale histogram of the nth frame is calculated. After filtering the grayscale histogram, it is regularized to obtain a regularized map corresponding to the nth frame. Then, based on the regularized maps corresponding to the nth and (n-1)th frames, the similarity between the nth and (n-1)th frames is calculated. If the similarity is less than a threshold, the inverse tone mapping curve and the target dynamic range expansion factor (Err_target) corresponding to the nth frame are recalculated. If the similarity is greater than or equal to the threshold, the inverse tone mapping curve and the target dynamic range expansion factor (Err_target) corresponding to the (n-1)th frame are used again to generate the initial 3D LUT corresponding to the nth frame. When the similarity is less than the threshold, the average brightness of the nth frame can be calculated to determine the target dynamic range expansion factor (Err_target). Furthermore, when it is determined that there will be no color banding after the dynamic range of the nth frame image is expanded, the inverse tone mapping curve of the nth frame image is directly calculated segment by segment. When it is determined that there will be color banding after the dynamic range of the nth frame image is expanded, the inflection point of the regularization value is determined, and then the inverse tone mapping curve of the dequantization band is generated for each interval. Finally, in order to avoid video flicker, the initial 3D LUT corresponding to the nth frame image is temporally filtered based on the target 3D LUT of the (n-1)th frame image to obtain the target 3D LUT corresponding to the nth frame image. Then, the nth frame image can be color-corrected based on the target 3D LUT to obtain the nth frame image after dynamic range expansion.

[0313] For any parts of this process that are not explained in detail, please refer to the previous text; they will not be repeated here.

[0314] Furthermore, after the image color correction module performs color correction on the current video image frame based on the target 3D LUT corresponding to the current video image frame, it can continue to perform saturation enhancement processing on the color-corrected current image frame to improve the color saturation of the displayed video image frame.

[0315] like Figure 4In the scenario shown in (2), if the user performs a shutdown operation on the "Enable HDR Effect" option 4011, the mobile phone reduces the screen backlight brightness in response to the user's shutdown operation on the "Enable HDR Effect" option 4011, and no longer uses the video processing method provided in this embodiment to perform dynamic range expansion processing on the SDR video that is being played.

[0316] For example, in response to the "Enable HDR Effect" option 4011 being turned off, the video application sends a backlight brightness recovery instruction to the brightness service module and a dynamic range extension cancellation instruction to the image color grading module.

[0317] Among them, the backlight brightness recovery indicator can be used to instruct the brightness service module to restore the backlight brightness of the display to the default value, such as 300 nits.

[0318] The dynamic range extension cancellation indicator can be used to instruct the image color grading module to perform color grading on video image frames based on a preset 3D LUT, without extending the dynamic range of the video image frames.

[0319] For example, the brightness service module controls the display driver to reduce the backlight brightness of the display to the default value according to the backlight brightness recovery instruction, thereby the display driver can control the display to reduce its backlight brightness to the default value.

[0320] After receiving the dynamic range extension cancellation instruction, the image color correction module no longer sends the current RGB format video image frame to the LUT processing module to obtain the target 3D LUT corresponding to the current video image frame. Instead, it directly uses the preset 3D LUT to perform color correction on the current RGB format video image frame and sends the color-corrected current video image frame to the display.

[0321] This embodiment also provides a computer storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the video processing method described above.

[0322] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the video processing method described above.

[0323] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component, or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the video processing methods in the above-described method embodiments.

[0324] In this embodiment, the electronic devices (such as mobile phones), computer storage media, computer program products or chips are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0325] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0326] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0327] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A video processing method, characterized in that, In an electronic device, the screen backlight brightness of the electronic device is a first brightness, and the method includes: In response to a user action, acquire the first video image frame; A reverse tone mapping curve corresponding to the first video image frame is determined, and a grayscale gain value corresponding to the first video image frame is determined based on the reverse tone mapping curve; wherein, the grayscale gain value corresponding to the diffuse region of the first video image frame is 0; the grayscale value of the pixels in the diffuse region is less than or equal to the grayscale segmentation threshold. Determine the pixel compression coefficient corresponding to the first video image frame, the pixel compression coefficient being used to compress the pixel brightness value; Based on the grayscale gain value and the pixel compression coefficient, each color channel of the preset 3D LUT is processed to obtain the target 3D LUT corresponding to the first video image frame; The first video image frame is processed according to the target 3D LUT to obtain the second video image frame; After the screen backlight brightness of the electronic device is increased to a second brightness, the second video image frame is displayed; wherein the second brightness is greater than the first brightness.

2. The method according to claim 1, characterized in that, Determining the inverse tone mapping curve corresponding to the first video image frame includes: If the first video image frame does not exhibit color banding after dynamic range expansion, then a preset inverse tone mapping curve is used as the inverse tone mapping curve corresponding to the first video image frame; wherein, the preset inverse tone mapping curve is calculated based on a piecewise function method; If color banding occurs in the first video image frame after dynamic range expansion, the bright grayscale range is divided into multiple grayscale sub-ranges. For each grayscale sub-range, a third-order inverse tone mapping curve for color banding removal is calculated based on the preset inverse tone mapping curve to obtain the inverse tone mapping curve corresponding to the first video image frame. The grayscale values ​​included in the high-brightness grayscale range are all greater than the grayscale segmentation threshold.

3. The method according to claim 2, characterized in that, Before determining the inverse tone mapping curve corresponding to the first video image frame, the method further includes: The grayscale histogram of the first video image frame is statistically analyzed, and the grayscale histogram is regularized to obtain a first regularized image. Determining whether color banding will occur in the first video image frame after dynamic range expansion includes: Determine whether there is a regularization value peak with a regularization value greater than the first threshold in the bright grayscale range of the first regularization map. If it exists, determine that the first video image frame will have color banding after dynamic range expansion; otherwise, determine that the first video image frame will not have color banding after dynamic range expansion.

4. The method according to claim 3, characterized in that The bright grayscale range is divided into multiple grayscale sub-ranges, including: When the number of regularization peaks is one, the bright grayscale interval is divided into two grayscale sub-intervals according to the grayscale value corresponding to the regularization peak; When there are multiple regularization value peaks, a regularization value valley is determined between every two adjacent regularization value peaks, and the high-brightness grayscale interval is divided into multiple grayscale sub-intervals according to the grayscale values ​​corresponding to each regularization value peak and each regularization value valley.

5. The method according to claim 2, characterized in that, The formula for calculating the preset inverse tone mapping curve Gain_1(i) is as follows: Where i represents the normalized pixel gray value, i∈[0,1], Gain_1(i) represents the gray gain value corresponding to the gray value i, th3 represents the gray segmentation threshold, α represents the exponential coefficient, v1 and v2 are two preset gray segmentation thresholds, v2>v1, a, b, c, d, e, f are preset coefficients, and a, b, d are positive numbers.

6. The method according to claim 2, characterized in that, For each grayscale sub-range, a third-order inverse toning map curve for decolorization is calculated based on the preset inverse toning map curve, including: For the grayscale sub-interval (S1, S2], the third-order inverse tone mapping curve Gain_2(i) after color banding removal is as follows: Gain_2(i)=m*i 3 +n*i 2 +p*i+q,i∈(S1,S2]; Where i represents the pixel gray value, Gain_2(i) represents the gray value gain corresponding to the gray value i; m, n, p, q are coefficients, calculated based on the values ​​of Gain_2(S1), Gain_2(S2), Gain_2'(S1) and Gain_2'(S2); If the grayscale value S1 is the grayscale segmentation threshold, then Gain_2(S1) = Gain_1(S1), Gain_2′(S1) = Gain_1′(S1); otherwise, Gain_2(S1) is calculated based on the third-order inverse tone mapping curve corresponding to the adjacent previous grayscale sub-interval. Where Gain_1(i) is the preset inverse tone mapping curve, Gain_1′(i) is the derivative of the preset inverse tone mapping curve, Δ=1 / (N-1), and N is the total number of gray levels; H norm ′(i)=maximum(H norm (i)-Tq,0); Where Tq is a constant, H norm (i) is the regularization value of the number of pixels corresponding to the gray value i in the gray histogram.

7. The method according to claim 6, characterized in that, H norm (i) It has undergone the following processing: Here, th5 is a preset threshold.

8. The method according to claim 1, characterized in that Determining the pixel compression coefficient corresponding to the first video image frame includes: Calculate the average brightness of the first video image frame; Calculate the undetermined expansion factor corresponding to the first video image frame based on the average brightness value; If the undetermined expansion factor is less than or equal to the display backlight brightness clearance ratio, then the undetermined expansion factor is taken as the target expansion factor corresponding to the first video image frame; otherwise, the display backlight brightness clearance ratio is taken as the target expansion factor Err_target corresponding to the first video image frame; wherein, the display backlight brightness clearance ratio is used to indicate the supported or allowed increase ratio of the display backlight brightness. The pixel compression coefficient C corresponding to the first video image frame is calculated using the following formula: C = 1 / Err_target 1 / γ γ is the gamma coefficient.

9. The method according to claim 8, characterized in that, Calculating the undetermined expansion factor corresponding to the first video image frame based on the average brightness value includes: The undetermined expansion factor Err_tbd corresponding to the first video image frame is calculated using the following formula: Where APL is the average brightness of all pixels in the video image frame, th1 is the preset pixel brightness threshold, Err1 is a preset maximum expansion factor, and m, n, and k are preset coefficients.

10. The method according to claim 1, characterized in that, Determining the inverse tone mapping curve corresponding to the first video image frame includes: Obtain a second regularized graph corresponding to the first video image frame and a third regularized graph corresponding to the third video image frame; wherein the third video image frame is the frame preceding the first video image frame. Calculate the similarity between the second regularized graph and the third regularized graph; If the similarity is greater than a preset similarity threshold, then the inverse tone mapping curve corresponding to the third video image frame will be used as the inverse tone mapping curve corresponding to the first video image frame.

11. The method according to claim 10, characterized in that, The regularization values ​​in the second and third regularization graphs have been pre-processed as follows: Among them, H norm (i) is the regularization value of the number of pixels corresponding to the gray value i in the gray histogram, and th5 is a preset threshold.

12. The method according to claim 1, characterized in that, Based on the grayscale gain value and the pixel compression coefficient, each color channel of the preset 3D LUT is processed to obtain the target 3DLUT corresponding to the first video image frame, including: Based on the grayscale gain value and the pixel compression coefficient, each color channel of the preset 3D LUT is processed to obtain an initial 3D LUT corresponding to the first video image frame; Obtain the target 3D LUT corresponding to the third video image frame; wherein, the third video image frame is the frame preceding the first video image frame; The target 3D LUT corresponding to the third video image frame is calculated by weighting the target 3D LUT corresponding to the first video image frame based on the target 3D LUT corresponding to the third video image frame and the initial 3D LUT corresponding to the first video image frame.

13. The method according to claim 1, characterized in that, The dynamic range of the first video image frame is the standard dynamic range; the dynamic range of the second video image frame is the high dynamic range.

14. An electronic device, characterized in that, include: One or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored on the memory, and when the computer programs are executed by the one or more processors, cause the electronic device to perform the video processing method as described in any one of claims 1-13.

15. A computer-readable storage medium comprising a computer program, characterized in that, When the computer program is run on an electronic device, the electronic device causes the electronic device to perform the video processing method as described in any one of claims 1-13.

Images