Video processing method and electronic equipment

Abstract

The embodiment of the invention provides a video processing method and electronic equipment. In the method, the electronic equipment can perform inverse tone mapping processing on the picture frame in the video, and the gray-scale value distribution of the pixels in the picture frame in each channel is mapped into wider gray-scale value distribution, so that the pixels can be expressed by wider gray-scale values. In this way, the picture frame in the SDR video can represent more colors, and a larger dynamic range is presented. When the video subjected to inverse tone mapping processing is played, in order to enable a user to feel a more real and vivid picture effect, the electronic equipment can improve the brightness of a screen area (a screen area 1) used for playing the video according to a certain ratio (marked as ratio 0), so that the brightness of the screen area 1 is greater than that of the screen area 1 in time 1, and the user experience is improved. Therefore, changes of picture frames in the video can be adapted. The time 1 is the time when the video playing operation is detected.

Description

Technical Field

[0001] This application relates to the fields of electronic devices and communication technology, and in particular to video processing methods and electronic devices. Background Technology

[0002] Although High Dynamic Range (SDR) video has been introduced, it offers higher display quality compared to lower dynamic range (SDR) video, such as Standard Dynamic Range (SDR) video. However, the high display quality of HDR video requires more complex encoding formats (such as HDR10 and Dolby Vision), and SDR video typically has a smaller file size when representing the same scene. This means that SDR video requires relatively less bandwidth during transmission and less storage space. Therefore, SDR video remains one of the video types offered to users by many application providers.

[0003] Higher-quality videos typically offer advantages such as richer colors and higher contrast, enhancing the user's viewing experience. Therefore, when electronic devices receive SDR videos, how to enable SDR videos to display the effects of HDR videos is a topic worth discussing. Summary of the Invention

[0004] This application provides a video processing method and an electronic device for improving the display effect of video.

[0005] In a first aspect, this application provides a video processing method applied to an electronic device. The method includes: detecting an operation to play a first video at a first time; performing inverse tone mapping processing on the frame of the first video; the dynamic range of the processed frame is greater than the dynamic range of the frame before processing, wherein the greater dynamic range includes: the pixel with the largest grayscale value in the first frame before processing has a larger grayscale value in the first frame after processing; playing the first video through a first screen area; and increasing the brightness of the first screen area using a first ratio, thereby controlling the brightness of the first screen area to be higher than the brightness of the first screen area at the first time.

[0006] In the above embodiments, inverse tone mapping is an image or video processing technique that can be used to convert standard dynamic range (SDR) video into high dynamic range (HDR) video. Inverse tone mapping achieves this by expanding the brightness range and color depth of an image, providing a richer and more realistic visual experience when playing SDR video on devices supporting HDR displays. Inverse tone mapping processing can apply a look-up table (LUT) to adjust the color and brightness of the image. A look-up table provides a mapping method from the SDR color space to the HDR color space. Through inverse tone mapping processing, the visual effect of SDR content can be effectively enhanced, making it approach or reach the quality of HDR content. The electronic device can increase the brightness of a first screen area (screen area 1 in the following embodiments) used for playing a first video (video 1 in the following embodiments) according to a first ratio (ratio0 in the following embodiments), such that the brightness of the first screen area is greater than the brightness of the first screen area at a first time (time 1 in the following embodiments), to adapt to changes in the dynamic range of the frames in the video, while making the played video content closer to the effect of HDR video.

[0007] In conjunction with the first aspect, in some embodiments, the method further includes: when increasing the brightness of the first screen area using a first ratio, and when increasing the brightness of the second screen area using the first ratio, decreasing the grayscale value of each pixel in the second screen area using the first ratio; the second screen area does not include the image frame processed by inverse tone mapping.

[0008] In the above embodiments, when the brightness of the first screen area is increased using the first ratio, the brightness of the second screen area (screen area 2 in the following embodiments) is also increased using the same first ratio. This indicates that the parameter adjusted when the electronic device increases the brightness of the first screen area is a parameter shared across the entire screen. At this time, the first ratio can be used to darken the pixels in the second screen area, reducing the grayscale value of each pixel in the second screen area. This ensures that when playing video, the brightness of the second screen area does not visually change compared to the first time, thus preventing visual discomfort for the user.

[0009] In conjunction with the first aspect, in some embodiments, where the screen further includes a third screen area for playing a second video, the first ratio is the minimum of a first brightening ratio corresponding to the first video and a second brightening ratio corresponding to the second video; the first brightening ratio is set by the application providing the first video, and the second brightening ratio is set by the application providing the second video; where the screen does not include the third screen area, the first ratio is the first brightening ratio; where the screen further includes a third screen area for playing a second video, the method further includes: performing inverse tone mapping processing on the frames in the second video; playing the second video through the third screen area; and increasing the brightness of the third screen area using the first ratio.

[0010] In the above embodiments, the brightening ratio can be obtained through bitwise operations. The first brightening ratio is ratio1, which is used for inverse tone mapping processing of video 1 in the following embodiments. The second video is video 2, which is used for inverse tone mapping processing of video 2 in the following embodiments. When the screen includes content from two or more videos, it means that there are multiple brightening ratios. However, when adjusting the brightness of a screen area, the brightness of each pixel in the screen area is increased according to a uniform standard, so only one ratio (the first ratio) is needed. Here, the first brightening ratio and the minimum value among the first brightening ratio and other brightening ratios are used as the first ratio, so that the brightness increase of the screen area is not too large when playing videos, preventing the screen image from being overexposed.

[0011] In conjunction with the first aspect, in some embodiments, the first video frame includes a first frame, and performing inverse tone mapping processing on the frame in the first video specifically includes: performing inverse tone mapping processing on the first frame in the first video, the process of performing inverse tone mapping processing on the first frame includes: placing the first frame in a first layer; the first layer is used to carry the frame to be played in the first video; the grayscale values ​​of the pixels in the first layer are used as input grayscale values, and the input grayscale values ​​are converted into output grayscale values ​​through a first lookup table to obtain the converted first layer; the dynamic range of the converted first layer is greater than the dynamic range of the first layer before conversion.

[0012] In the above embodiments, a look-up table (LUT) is a data structure used for inverse tone mapping of images or video frames, allowing for rapid color conversion or color correction. The look-up table contains a series of input grayscale values ​​and corresponding output grayscale values. By querying this table, each pixel grayscale value (as an input grayscale value) in the image or video frame can be mapped to a new grayscale value, thereby adjusting the pixel grayscale value. When the grayscale values ​​of all color channels of a pixel are mapped, the pixel color can be adjusted.

[0013] In conjunction with the first aspect, in some embodiments, the process of determining the first lookup table includes: determining the cumulative pixel percentage of each grayscale interval of each color channel in the second frame to obtain a cumulative pixel percentage distribution; the cumulative pixel percentage of a grayscale interval under a color channel is equal to the ratio of the number of pixels whose grayscale value is less than or equal to the maximum grayscale value of the grayscale interval to the total number of pixels in the second frame; the second frame is the first frame or a frame in the first video preceding the first frame; based on the cumulative pixel percentage distribution and the first brightening ratio, a first lookup table is obtained; the first lookup table includes: the output grayscale value corresponding to each of the multiple input grayscale values ​​under each color channel. The multiple input grayscale values ​​under a color channel include a first type of input grayscale value and a second type of input grayscale value; wherein, the first type of input grayscale value under a color channel includes: the maximum grayscale value of each grayscale interval under the color channel; the first type of input grayscale value is obtained by obtaining the output grayscale value corresponding to the first type of input grayscale value based on a first mapping relationship, the first mapping relationship including the first brightening ratio and the cumulative percentage of pixels in the grayscale interval where the first type of input grayscale value is located; the second type of input grayscale value is obtained by interpolation based on the first type of input grayscale value, and the output grayscale value corresponding to the second type of grayscale value is obtained by interpolation based on the output grayscale value corresponding to the first type of grayscale value.

[0014] In the above embodiments, the first lookup table is generated based on the frames in the first video. When using such a first lookup table to perform inverse tone mapping on the frames in the first video, the adaptability is higher, which can improve the quality of inverse tone mapping. Using the pixel cumulative proportion distribution to generate the first lookup table for inverse tone mapping can ensure a more comprehensive adjustment of the frames, thereby providing a smoother color transition effect. Because the pixel cumulative proportion distribution is based on the pixel distribution, compared with the pixel distribution, the cumulative proportion of the k-th pixel in the pixel cumulative proportion distribution includes the number of pixels that obtain the cumulative proportion of the (k-1)-th pixel and the number of pixels in the k-th grayscale interval, which is a smoother image data. Therefore, when performing inverse tone mapping based on the first lookup table generated by the pixel cumulative proportion distribution, the change in dynamic range before and after processing can be more natural.

[0015] In conjunction with the first aspect, in some embodiments, after obtaining the converted first layer, the method further includes:

[0016] The converted first and second layers are merged to obtain a screen image, which is then displayed on the screen. The screen image includes a first frame displayed in the first screen area. The second layer includes a layer for carrying the content of the second screen area. If a second video exists, in addition to the layer for carrying the content of the second screen area, the second layer also includes a layer carrying a third frame. The third frame comes from the second video, and the grayscale values ​​of the pixels in the layer carrying the third frame are obtained by conversion using a second lookup table. The second lookup table is obtained based on the cumulative pixel percentage distribution of the fourth frame and the second brightening ratio. The fourth frame is the third frame or a frame preceding it.

[0017] In the above embodiments, the content in the second screen area is the non-adjusted content involved in the following embodiments. It supports simultaneous inverse tone mapping processing for frames from multiple videos. Different videos use their own frame-specific lookup tables, resulting in high adaptability.

[0018] In conjunction with the first aspect, in some embodiments, the second frame is the first frame or a frame in the first video preceding the first frame, specifically including: when the first frame is a target frame, the second frame is the first frame; when the first frame is not a target frame, the second frame is the target frame whose timestamp is closest to the first frame; the first video includes at least one target frame, the first target frame is the first frame of the first video, and the i-th target frame and the (i-1)-th target frame satisfy a preset condition, where i is an integer greater than or equal to 2.

[0019] In the above embodiments, the electronic device does not generate a lookup table for every frame. Instead, it selects the target frame to generate the lookup table. Other non-target frames can use the lookup table calculated relative to their nearest target frame for inverse tone mapping. In this way, the lookup table does not need to be updated frequently, and while achieving better inverse tone mapping results, it can save computing resources.

[0020] In conjunction with the first aspect, in some embodiments, the preset condition includes: the similarity between the i-th target frame and the (i-1)-th target frame is less than a preset threshold, or the i-th target frame and the (i-1)-th target frame are separated by a preset number of frames.

[0021] In the above embodiments, using similarity as a condition for updating the target frame can ensure that the generated lookup table is compatible with each frame in the video without frequent updates.

[0022] In conjunction with the first aspect, in some embodiments, after determining the cumulative pixel percentage of each grayscale interval of each color channel in the second frame and obtaining the cumulative pixel percentage distribution, the method further includes: the video encoder recording the cumulative pixel percentage distribution into a first buffer area via a binder thread, the first buffer area also including the first brightening ratio; the first buffer area is used to cache the first layer; the video encoder is also used to perform the operation of obtaining the cumulative pixel percentage distribution; based on the cumulative pixel percentage distribution and the first brightening ratio, a first lookup table is obtained, specifically including: obtaining the cumulative pixel percentage distribution and the first brightening ratio from the first buffer area by an image processor; the image processor includes at least one of a graphics processing unit (GPU) and a display processing unit (DPU); the image processor determines a first mapping relationship based on the cumulative pixel percentage distribution and the first brightening ratio, and obtains the first lookup table according to the first mapping relationship.

[0023] In the above embodiments, the cumulative pixel percentage is calculated in the video encoder (the media encoder involved in the following embodiments), and a lookup table is determined in the image processor based on the cumulative pixel percentage. This collaboration allows for faster lookup table generation because the video encoder acquires the second frame earlier than the image processor; that is, there is a time difference between the second frame and the image processor. Utilizing this time difference, the video encoder can determine the cumulative pixel percentage distribution and transmit it to the image processor. The amount of data for the cumulative pixel percentage distribution is less than the amount of data for the second frame. Transmitting the cumulative pixel percentage distribution from the video encoder to the image processor also speeds up the transmission speed of the binder thread.

[0024] In conjunction with the first aspect, in some embodiments, before determining the cumulative pixel percentage of each grayscale interval of each color channel in the second frame and obtaining the cumulative pixel percentage distribution, the method further includes: a video encoder determining the number of pixels in each grayscale interval of each color channel in the second frame and obtaining a pixel distribution; the number of pixels in a grayscale interval under a color channel is equal to the number of pixels in that grayscale interval; the video encoder records the pixel distribution into a first buffer through a binder thread, the first buffer also including the first brightening ratio; based on the cumulative pixel percentage distribution and the first brightening ratio, a first lookup table is obtained, specifically including: obtaining the pixel distribution and the first brightening ratio from the first buffer through an image processor, and obtaining the cumulative pixel percentage distribution based on the pixel distribution; the image processor includes at least one of a graphics processing unit (GPU) and a display processing unit (DPU); the image processor determines a first mapping relationship based on the cumulative pixel percentage distribution and the first brightening ratio, and obtains the first lookup table according to the first mapping relationship.

[0025] In the above embodiments, the pixel distribution is statistically analyzed in the video encoder (the media encoder involved in the following embodiments), and a lookup table is determined in the image processor based on the pixel distribution. This collaboration allows for faster lookup table generation because the video encoder acquires the second frame earlier than the image processor; that is, there is a time difference between the second frame and the image processor. Utilizing this time difference, the video encoder can determine the pixel distribution and transmit it to the image processor. The amount of data for the pixel distribution is less than the amount of data for the second frame. Transmitting the pixel distribution from the video encoder to the image processor also speeds up the transmission speed of the binder thread.

[0026] In conjunction with the first aspect, in some embodiments, the video encoder records the pixel distribution into a first buffer via a binder thread, specifically including: the video encoder transmitting the pixel distribution to SurfaceFlinger via a binder thread; when SurfaceFlinger receives the pixel distribution, it also receives a first application identifier corresponding to the pixel distribution issued by the video encoder; if it is determined that the attributes of the first layer include a highlighting identifier, SurfaceFlinger determines that the first layer is used to carry the frame of the first video provided by the first application based on the identifier of the first layer, and records the pixel distribution corresponding to the first application identifier into the first buffer; the highlighting identifier included in the attributes of the first layer comes from the application providing the first video; the highlighting identifier is used to indicate that the first video is a video to be subjected to inverse tone mapping processing; the playback component is located in the first screen area.

[0027] In the above embodiments, the first application identifier can be the identifier of the video application 1 involved in the following embodiments. When there are two or more videos undergoing inverse tone mapping processing, the first application identifier can be used to distinguish which layer the pixel distribution will be used to perform inverse tone mapping processing on.

[0028] In conjunction with the first aspect, in some embodiments, when the pixel distribution corresponding to the first application identifier is recorded in the first buffer, the method further includes: the SurfaceFlinger obtaining the first brightening ratio from the attributes of the first layer and recording it in the first buffer; the brightening identifier and the first brightening ratio included in the attributes of the first layer both come from the application providing the first video; the process of transferring the brightening identifier and the first brightening ratio from the application providing the first video to the attributes of the first layer includes: creating a playback component for playing the first video through the application providing the first video, and adding a brightening marker and the first brightening ratio to the attributes of the playback component; the video encoder decoding the first frame and placing it in the playback component to form a view including the first frame; the SurfaceFlinger converting the view carrying the first frame to the first layer; and the Java Native Interface transaction adding the brightening marker and the first brightening ratio from the attributes of the playback component to the attributes of the first layer.

[0029] In the above embodiments, an end-to-end marking capability is provided, so that after the application (first application) providing the video (first video) highlights and marks the video, when the frame of the first video is recorded to a layer in SurfaceFlinger, SurfaceFlinger can accurately identify that the layer carrying the frame needs to be inverted tone mapping, and determine the parameters for inverted tone mapping of the layer to complete the inverted tone mapping of the frame in the video.

[0030] In conjunction with the first aspect, in some embodiments, before increasing the brightness of the first screen area using a first ratio, the method further includes: determining that the screen is operating in pulse width modulation (PWM) mode.

[0031] In the above embodiments, the screen brightness is typically low in PWM mode. Therefore, the brightness of the first screen area can be increased independently in PWM mode using a first ratio. The brightness in DC mode is already bright enough to meet the playback requirements of the processed video. Therefore, the brightness of the first screen area is not increased using the first ratio in DC mode, simplifying the solution.

[0032] In a second aspect, embodiments of this application provide an electronic device comprising: one or more processors and a memory; the memory is coupled to the one or more processors and is used to store computer program code, the computer program code including computer instructions, wherein the one or more processors invoke the computer instructions to cause the electronic device to perform the method implemented in the first aspect.

[0033] Thirdly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the method as implemented in the first aspect.

[0034] Fourthly, embodiments of this application provide a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to perform the method as implemented in the first aspect.

[0035] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on an electronic device, cause the electronic device to perform the method as implemented in the first aspect.

[0036] It is understood that the electronic device provided in the second aspect, the computer storage medium provided in the third aspect, the chip system provided in the fourth aspect, and the computer program product provided in the fifth aspect are all used to execute the methods provided in the embodiments of this application. Therefore, other beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0037] Figure 1 An exemplary user interface is shown when an electronic device plays video;

[0038] Figure 2 This diagram illustrates the pixel distribution of a frame in a single color channel.

[0039] Figure 3 A schematic diagram illustrating the determination of the initialization of the 3D-LUT is shown;

[0040] Figure 4 This diagram illustrates the determination of inverse tone mapping for a frame.

[0041] Figure 5 This demonstrates the tagging capabilities between video-providing applications and SurfaceFlinger;

[0042] Figure 6 A schematic diagram of a system structure for implementing the video processing method in the embodiments of this application is shown;

[0043] Figure 7 An exemplary module interaction diagram is shown for implementing the video processing method in the embodiments of this application;

[0044] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0045] This application provides a video processing method in which an electronic device can perform inverse tone mapping (ITM) on frames in an SDR video. This process maps the grayscale value distribution of pixels in each color channel of the frame to a wider grayscale value distribution, allowing pixels to be represented by a broader range of grayscale values. For example, the bit depth of pixels in an SDR video is typically 8 bits, which allows for the representation of 256 grayscale values ​​(0-255). After inverse tone mapping, the grayscale value of a pixel can be greater than 255. Thus, frames in an SDR video can represent more colors, resulting in a wider color distribution and a greater dynamic range.

[0046] It's important to note that a larger dynamic range can be understood as a wider brightness range and richer color depth in the frames after inverse tone mapping. To ensure a more realistic and vivid viewing experience when playing inverse tone-mapped videos, the electronic device can further increase the brightness of the screen area (screen area 1) used for video playback by a certain ratio (denoted as ratio0), making the brightness of screen area 1 greater than its brightness at time 1, to adapt to changes in the video frames. Here, time 1 is the time at which the video playback operation is detected.

[0047] Here, the method of increasing the brightness of screen area 1 depends on the type of screen. Different types of screens have different adjustment methods when increasing brightness, and this embodiment does not limit this. However, the adjustment method here does not involve adjusting the grayscale value of the pixels.

[0048] For example, when the screen is a liquid crystal display (LCD) screen, the adjustment method to increase the brightness of screen area 1 includes: the electronic device can increase the brightness of screen area 1 by increasing the backlight brightness at screen area 1. The degree to which the backlight brightness at screen area 1 is increased is determined based on ratio0; the larger the ratio0, the greater the degree to which the backlight brightness at screen area 1 is increased.

[0049] For example, when the screen is an organic light-emitting diode (OLED) screen, the methods to increase the brightness of screen area 1 include: when the OLED screen is operating in pulse width modulation (PWM) mode, the electronic device can increase the brightness of screen area 1 by increasing the dimming frequency or the duty cycle in PWM mode. When the OLED screen is operating in direct current (DC) mode, the electronic device can increase the brightness of screen area 1 by increasing the DC voltage of the pixels in screen area 1. The degree to which the dimming frequency, duty cycle, and DC voltage are increased is determined based on ratio0; the larger the ratio0, the greater the increase.

[0050] It should also be noted that increasing the brightness of display area 1 is not mandatory. In some cases, if the brightness of screen area 1 is sufficiently high at time 1, the brightness of screen area 1 may not need to be adjusted. The definition of sufficiently high brightness can vary depending on the screen type. For example, in an LCD screen, sufficiently high brightness may mean that the backlight of screen area 1 is greater than a preset backlight value. In an OLED screen, sufficiently high brightness may mean that the screen is operating in DC mode. Therefore, the operation to increase the brightness of screen area 1 is only performed when the OLED screen is operating in PWM mode.

[0051] In some cases, when an electronic device increases the brightness of screen area 1 using parameters shared across the entire screen, increasing the brightness of screen area 1 will also increase the brightness of other screen areas besides screen area 1. For screen areas that do not include frames processed with inverse tone mapping (denoted as screen area 2), the sudden increase in brightness can cause visual discomfort to the user. Therefore, if the electronic device uses ratio0 to increase the brightness of both screen area 1 and screen area 2, it can also use ratio0 to darken the pixels in screen area 2. This darkening process reduces the grayscale value of each pixel in screen area 2, so that the brightness of screen area 2 does not visually change compared to time 1 when playing video. For example, refer to... Figure 1 As shown in (1), region 111a can be an example of the aforementioned screen region 1, used to play a video after inverse tone mapping. Other regions on the screen besides region 111a can be referred to as screen region 2.

[0052] It should be noted that when the electronic device plays a video (denoted as Video 1) that has undergone inverse tone mapping through screen area 1, it can also play Video 2 through screen area 3. Video 2 is a video other than Video 1, and Video 2 also needs to undergo inverse tone mapping. Figure 1 As shown in (2), region 111b is an example of screen region 3. At this time, in addition to region 111b, the screen also includes region 111a. At this time, screen region 2 is the other regions on the screen besides screen region 1 and screen region 3.

[0053] Generally, electronic devices can start playing video 1 first, and then start playing video 2. If the electronic device does not support simultaneous playback of multiple videos while playing video 2, video 1 can be paused during video 2 playback. Figure 1 Middle (1) and Figure 1 As shown in (2), when playing a video (Example Video 1) with inverse tone mapping in area 111a, an operation to play a video (Example Video 2) in video application 2 is detected. The electronic device can then play the video with inverse tone mapping in area 111b while pausing the video in area 111a. Here, area 111a and area 111b are video playback windows.

[0054] It should be noted that when video 1 is paused, the frame displayed in screen area 1 is the last frame displayed before the pause (after inverse tone mapping), and the brightness of screen area 1 remains unchanged, the same as when video 1 is playing.

[0055] In a screen comprising screen area 1 for playing video 1 and screen area 3 for playing video 2, ratio0 is the minimum of the brightness ratio 1 corresponding to video 1 and the brightness ratio 2 corresponding to video 2. This brightness ratio 1 can be used to implement inverse tone mapping processing for video 1. Brightness ratio 2 can be used to implement inverse tone mapping processing for video 2. The brightness ratio is a number greater than 1; the larger the brightness ratio, the greater the dynamic range of the video frames after inverse tone mapping processing, and consequently, the greater the dynamic range of the video after inverse tone mapping processing.

[0056] If the screen includes screen area 1 for playing video 1, but does not include screen area 3 for playing video 2, then ratio0 is the brightness ratio 1 corresponding to video 1. This brightness ratio 1 can be used to implement inverse tone mapping processing for video 1.

[0057] The electronic device processes the frames in video 1, including performing inverse tone mapping on frame 1. This process involves: first, placing frame 1 in layer 1, which carries the frames from video 1. Then, the electronic device uses the grayscale values ​​of the pixels in layer 1 as input grayscale values ​​and converts these input grayscale values ​​to output grayscale values ​​using lookup table 1, resulting in the converted layer 1. The dynamic range of the converted layer 1 is greater than that of the unconverted layer 1. Here, lookup table 1 is generated based on a brightening ratio of 1 and records the output grayscale values ​​corresponding to multiple input grayscale values ​​for each color channel. The generation process of lookup table 1 can be found in the following... Figures 2-4 The relevant description is provided below. An example of looking up Table 1 can be found in [reference needed]. Figure 4 The lookup table 1 shown is merely an example and should not be construed as limiting the embodiments of this application.

[0058] Determine the number of pixels in each grayscale interval of each color channel in frame 0 to obtain the pixel distribution. The number of pixels in a grayscale interval of a color channel is equal to the number of pixels in that grayscale interval. Frame 0 can be frame 1 or a frame in video 1 preceding frame 1.

[0059] The following explanation uses an example where a frame contains three color channels: red (R), green (G), and blue (B), with grayscale values ​​ranging from 0 to 255, and the grayscale values ​​of a color channel are divided into 17 grayscale ranges.

[0060] like Figure 2 As shown, the grayscale values ​​of a single color channel are divided into 17 non-overlapping grayscale intervals. The electronic device counts the number of pixels in these 17 grayscale intervals to obtain the pixel distribution of frame 0 (also known as the pixel distribution histogram). For example, the grayscale interval [0, 15] contains x1 pixels, and the grayscale interval (15, 30] contains x2 pixels.

[0061] Subsequently, based on the pixel distribution of frame 0, the electronic device determines the cumulative pixel percentage of each grayscale interval in each color channel within frame 0, thus obtaining the cumulative pixel percentage distribution of the frame. The cumulative pixel percentage of a grayscale interval under a color channel is equal to the ratio of the number of pixels whose grayscale value is less than or equal to the maximum grayscale value of that grayscale interval to the total number of pixels in frame 0. The formula for calculating the cumulative pixel percentage is as follows.

[0062]

[0063] Among them, C(b) i c(b) represents the percentage of the total number of pixels accumulated to the current grayscale range. j ) indicates the number of pixels in the current grayscale range and the previous grayscale ranges, and 17 indicates the number of grayscale ranges. TotalNo.Pixels indicates the total number of pixels in frame 0.

[0064] For example, refer to Figure 2 As shown, the cumulative percentage of pixels in the grayscale range [0, 15] is equal to x1 divided by the total number of pixels in frame 0. The cumulative percentage of pixels in the grayscale range [15, 30] is equal to (x1 + x2) divided by the total number of pixels in frame 0.

[0065] Based on the cumulative pixel proportion distribution of frame 0 and the brightening ratio of 1, lookup table 1 is obtained. Lookup table 1 includes the output grayscale values ​​corresponding to multiple input grayscale values ​​under each color channel. Multiple input grayscale values ​​under a color channel include first-type input grayscale values ​​and second-type input grayscale values. Specifically, the first-type input grayscale values ​​under a color channel include the maximum grayscale value in each grayscale interval under that color channel. For example, using the aforementioned... Figure 2 Taking the pixel cumulative distribution obtained from the pixel distribution shown in the figure as an example, the first type of input grayscale values ​​under each color channel include 17 first type of input grayscale values, namely: 15, 30, 45, 60, 75, ..., 255. A first type of input grayscale value is used to obtain the corresponding output grayscale value based on mapping relationship 1. Mapping relationship 1 includes the brightening ratio 1 and the cumulative pixel percentage of the grayscale range in which the first type of input grayscale value is located. The second type of input grayscale value is obtained by interpolation based on the first type of input grayscale value, and the corresponding output grayscale value is obtained by interpolation based on the output grayscale value corresponding to the first type of input grayscale value.

[0066] like Figure 3 As shown in (1), this is a schematic diagram of the cumulative pixel proportion distribution of frame 0. Using this cumulative pixel proportion distribution, the output grayscale values ​​corresponding to the 17×3 first-type input grayscale values ​​can be determined first, resulting in an initialized 3D look-up table (LUT). An exemplary initialized 3D-LUT can be found by referring to... Figure 3As shown in (2). Here, we will take the mapping relationship 1 as 255 × the cumulative percentage of pixels in the first type of input grayscale value × the brightening ratio 1 (e.g., 1.5) as an example. For example, the first type of input grayscale value 75 (the maximum value of the grayscale interval (60, 75)) has a corresponding output grayscale value of 46. Wherein, 46 = 255 × 0.12 × brightening ratio 1 (1.5), and 0.12 is the cumulative percentage of pixels in the grayscale interval where grayscale value 75 is located on the R channel. For example, the first type of input grayscale value 210 (the maximum value of the grayscale interval (195, 210)) has a corresponding output grayscale value of 364. Wherein, 364 = 255 × 0.95 × brightening ratio 1 (1.5), and 0.95 is the cumulative percentage of pixels in the grayscale interval where grayscale value 210 is located on the R channel.

[0067] like Figure 4 As shown, the correspondence between input and output grayscale values ​​in the initialized 3D-LUT is relatively weak. Interpolation can be performed on the initialized 3D-LUT to obtain the output grayscale values ​​corresponding to the second type of input grayscale values, generating lookup table 1. Here, lookup table 1 is still a 3D-LUT.

[0068] Continue to refer to Figure 4 Subsequently, the electronic device can use lookup table 1 to complete the inverse tone mapping of frame 1, including: converting the grayscale values ​​of each pixel in layer 1 carrying frame 1 through lookup table 1 to obtain the converted layer 1. For example, when there are pixels with RGB values ​​of (75, 60, 40) in frame 1, they can be converted to pixels with RGB values ​​of (46, 23, 17) through lookup table 1. When there are pixels with RGB values ​​of (210, 255, 240) in frame 1, they can be converted to pixels with RGB values ​​of (364, 383, 320) through lookup table 1.

[0069] Generally speaking, smaller grayscale values ​​will decrease after transformation by lookup table 1, while larger grayscale values ​​will increase after transformation by lookup table 1. This allows for a wider color distribution of pixels in frame 1, increased contrast, and a wider dynamic range, resulting in a greater brightness range and richer color depth.

[0070] It should be noted that 255 in the aforementioned mapping relationship 1 represents the maximum bit depth of the pixels in video 1. Mapping relationship 1 can also be adjusted according to actual conditions and should not constitute a limitation on the embodiments of this application. For example, mapping relationship 1 can also be the maximum grayscale value in frame 0 × cumulative pixel percentage × brightening ratio 1. The values ​​of 17, 0.12, 0.95, etc. mentioned above are also illustrative examples and should not constitute a limitation on the embodiments of this application.

[0071] In some possible cases, after the lookup table 1 is generated, it can be used to perform inverse tone mapping on all frames in video 1. In this case, frame 0 is the first frame in video 1.

[0072] In other possible cases, after the lookup table 1 is calculated, it can be used to perform inverse tone mapping on frames (including frame 1) within a preset number of frames after frame 0 in video 1. The lookup table 1 after the preset number of frames needs to be recalculated based on the target frames; that is, the lookup table will be updated when performing inverse tone mapping on the frames in video 1. The first target frame in video 1 is the first frame of video 1. The i-th target frame and the (i-1)-th target frame satisfy preset conditions, where i is an integer greater than or equal to 2. The preset conditions include: the similarity between the i-th target frame and the (i-1)-th target frame is less than a preset threshold, or the i-th target frame and the (i-1)-th target frame are separated by a preset number of frames. Here, if frame 1 is the target frame, then frame 0 is frame 1; if frame 1 is not the target frame, then frame 0 is the target frame whose timestamp is closest to frame 1.

[0073] When the lookup table can be updated, the electronic device can also smooth lookup table 1 based on lookup table a, so that the dynamic range of the image after inverse tone mapping can change smoothly and avoid jumps. Lookup table a is calculated based on the previous target frame (target frame a) of frame 0. Smoothing includes: smoothing the initial 3D-LUT of lookup table 1 based on the initial 3D-LUT obtained from lookup table a, obtaining a smoothed initial 3D-LUT, and then interpolating based on the smoothed initial 3D-LUT to obtain lookup table 1. The electronic device can generate the initial 3D-LUT of lookup table a based on target frame a; this process can refer to the relevant content on generating the initial 3D-LUT of lookup table 1 based on frame 0, and will not be repeated here.

[0074] Here, after obtaining the converted Layer 1, the electronic device merges the converted Layer 1 with Layer a to obtain the screen image. Then, the electronic device displays this screen image on the screen, which includes frame 1 of the image displayed in screen area 1.

[0075] If video 2 does not exist, layer a includes a layer to hold the content of screen area 2. For example, refer to... Figure 1 As shown in (1), it is used to generate Figure 1Layer a of the screen image shown in (1) may include: a layer carrying the content in region 112, a layer carrying the content in region 113, and a layer carrying the content in region 112, etc. Here, region 114 does not include region 111a.

[0076] If video 2 exists, in addition to the layer used to carry the content (non-adjusted content) in screen area 2, layer a also includes a layer carrying frame 3; frame 3 comes from video 2, and the grayscale values ​​of the pixels in the layer carrying frame 3 are obtained through lookup table 2; lookup table 2 is obtained based on the cumulative pixel proportion distribution of frame 4 and the brightening ratio 2. Frame 4 is frame 3 or a frame preceding it. For example, refer to... Figure 1 As shown in (2), it is used to generate Figure 1 Layer a of the screen image shown in (2) includes: a layer containing the content of area 112, a layer containing the content of area 115, and a layer containing the content of area 111b (frames of video 2). Here, area 115 does not include areas 111a and 111b. The process of obtaining lookup table 2 can be referred to the description of obtaining lookup table 1 above. Simply change frame 0 to frame 4 and brighten ratio 1 to brighten ratio 2. It will not be described again here.

[0077] As discussed above, when performing inverse tone mapping on frames in SDR video, the inverse tone mapping is performed on the layers carrying the frames. For different layers carrying different frames (from different videos), the electronic device uses different brightening ratios to provide different video applications during inverse tone mapping. To correctly perform inverse tone mapping on layers carrying different frames, the parameters for inverse tone mapping (including brightening ratios) need to be obtained. Determining the parameters for inverse tone mapping on layers carrying frames is based on the SurfaceFlinger (Surface Manager) in the electronic device's system. However, in addition to the layers carrying frames, SurfaceFlinger also includes other layers carrying non-adjustment content. Identifying the layers that need inverse tone mapping and obtaining the necessary parameters is a capability that SurfaceFlinger needs to possess. Whether a layer carrying a frame from a video undergoes inverse tone mapping is determined by the application providing the video. Therefore, an end-to-end marking capability is provided, allowing the application providing the video to highlight the video. Once a frame from the video is recorded as a layer in SurfaceFlinger, SurfaceFlinger can accurately identify whether the layer carrying that frame needs inverse tone mapping and determine the parameters for performing this process. This is used to complete the inverse tone mapping of the video frames.

[0078] It's important to note that this end-to-end marking capability allows SurfaceFlinger to accurately identify the layer to be inverted intone mapping and determine the parameters for inverted intone mapping after the video application highlights the video. The reason for identifying the layer to be inverted intone mapping and determining its parameters in SurfaceFlinger is that, while increasing the overall screen brightness during video playback, it's also necessary to reduce the grayscale values ​​of pixels in the layer carrying non-adjusted content (displayed in screen area 2). This ensures that while the overall screen brightness is increased, the brightness in screen area 2 remains visually unchanged compared to before and after video playback; only the area of ​​the screen used for video playback becomes visually brighter. The parameter (ratio0) used to reduce the grayscale values ​​of pixels in the layer carrying non-adjusted content is determined based on the parameters of the layer to be inverted intone mapping. For a further description, please refer to the following... Figure 6 The relevant descriptions of SurfaceFlinger in the document will not be described here.

[0079] Among them, the methods to increase the overall brightness of the screen include: the aforementioned method of using ratio0 to increase the brightness of screen area 1 and also using ratio0 to increase the brightness of screen area 2.

[0080] Taking the inverse tone mapping of video 1 as an example, the parameters required for inverse tone mapping of the layer carrying the frame of video 1 include metadata (e.g., metadata1) and a brightening ratio of 1 (denoted as extendedratio1, or simply ratio1). metadata1 can be the pixel distribution mentioned above, or it can be the cumulative pixel distribution mentioned above. Ratio1 is an attribute configured in playback component 1 (used to play video 1). As the object carrying the frame changes from the view corresponding to playback component 1 to a layer, SurfaceFlinger can obtain the ratio1 required for inverse tone mapping of the layer. metadata1 is determined by the media encoder and then transmitted to SurfaceFlinger.

[0081] like Figure 5 As shown, the inverse tone mapping process of frame 1 of video 1 will be used as an example for illustration.

[0082] refer to Figure 5 As shown in (11), the application providing video 1 (referred to as video application 1, located in the application layer) highlights video 1 and sends it to the media codec (also known as the video encoder). The media codec is configured with a filter 1 for decoding video 1. This filter 1 can decode video 1 to obtain the frame and can also determine metadata 1 based on the frame. Video application 1 can also create a playback component 1 (e.g., SurfaceView 1) for playing video 1 and add the highlight mark and ratio 1 to the properties of the SurfaceView 1.

[0083] refer to Figure 5 As shown in (12a), filter1 (located in the application framework layer) can transmit metadata1 and the identifier of the video application 1 corresponding to metadata1 to SurfaceFlinger through the binder thread. The identifier of the video application 1 may include the APP package name (e.g., APP package name1) and the APP activity name (APP activity name1) of the video application 1.

[0084] refer to Figure 5As shown in (12b), filter1 transmits frame 1 to the playback component 1 (e.g., SurfaceView1) created by the video application 1, thus obtaining a view carrying frame 1. Figure 1 The view Figure 1 Its shape is the same as SurfaceView1. SurfaceFlinger (located in the local frame layer) will carry the view of frame 1. Figure 1 Convert to layer (carrying the view) Figure 1 In layer 1).

[0085] refer to Figure 5 As shown in (12c), the Java Native Interface transaction (JNI transaction, located in the JNI layer) adds the highlight mark and ratio1 from the properties of SurfaceView1 to the properties of layer 1.

[0086] It should be noted that here Figure 5 Middle (12a) Figure 5 Middle (12b) and Figure 5 There is no sequential order among the operations involved in (12c).

[0087] refer to Figure 5 As shown in (3), when the attributes of layer 1 include a highlighting indicator, SurfaceFlinger determines that layer 1 is used to carry the frame of video 1 provided by video application 1 through the indicator of layer 1, records the metadata1 corresponding to the indicator of video application 1 into buffer1 (buffer, also known as buffer area) in GraphicBuffer, and SurfaceFlinger also records the ratio1 included in the attributes of layer 1 into buffer1.

[0088] During the compositing and display stage, the image processor (not shown in the figure) can use metadata1 to perform inverse tone mapping on layer 1, and merge the processed layer 1 with other layers (such as layer a mentioned above) to obtain the screen image. Then, the screen image is transmitted to the screen (also known as the display screen) for display. During display, the electronic device also increases the brightness of the area displaying the content of layer 1 (screen area 1 mentioned above) according to ratio1.

[0089] It should be noted that electronic devices can simultaneously open two or more videos that require inverse tone mapping. Each video has its own corresponding brightening ratio, and the parameters for inverse tone mapping differ for the layers representing frames from different videos. The electronic device needs to determine the parameters of each layer separately to implement the aforementioned video processing methods. A description of this process can be found below. Figure 6 The description.

[0090] Figure 6 A schematic diagram of a system structure for implementing the video processing method in the embodiments of this application is shown.

[0091] The system architecture is divided into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the system architecture is divided into seven layers, from top to bottom: application layer, application framework layer, JNI (Java Native Interface) layer, frameworknative layer, hardware abstraction layer (HAL), kernel layer (also known as the Linux layer), and hardware layer.

[0092] like Figure 6 As shown, the application layer can include a series of application packages.

[0093] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. Electronic devices can build rich application functionalities through these APIs without directly interacting with the underlying system architecture.

[0094] The JNI (Java Native Interface) layer provides a mechanism for Java to call native (i.e., C or C++) code, allowing data exchange and operation calls between Java code and native code.

[0095] The native framework layer sits between the application framework layer and the kernel layer, primarily containing native implementations of hardware access and system services. This layer provides JNI to the application framework layer, enabling applications within the application framework layer to call libraries written in C or C++ to perform lower-level operations.

[0096] The hardware abstraction layer is an interface layer located between the operating system kernel layer and other layers of electronic devices (such as the local service layer). Its purpose is to abstract the hardware and provide a virtual hardware platform for the operating system.

[0097] The kernel layer is the layer between hardware and software. It can also be called the driver layer. The kernel layer can include modules such as fingerprint drivers, touch panel (TP) drivers, and display drivers (LCD drivers).

[0098] The various levels cooperate with each other to complete the video processing method involved in the embodiments of this application. A description of this process can be found below.

[0099] Applications containing videos to be played transfer the frames to be played to SurfaceFlinger to form a layer carrying those frames. SurfaceFlinger then merges the layer carrying the frames to be played with the layer carrying the non-adjusted content to obtain the final screen display. Here, the frames to be played are the latest frames from the video to be displayed on the screen.

[0100] This example illustrates how, when playing a video that has undergone inverse tone mapping, the brightness of the entire screen is increased to enhance the brightness of the area used for video playback.

[0101] SurfaceFlinger retrieves views containing frame rates from the playback component 'a' (e.g., SurfaceView, which may include the aforementioned playback component '1') used to play videos requiring inverse tone mapping, updating the content in the corresponding layers accordingly. This composited layer refreshes the screen, allowing the user to view the video (after inverse tone mapping) as the screen refreshes. This refresh rate can be the screen refresh rate or the frame rate of each video. This embodiment does not limit this. The frame rate represents the reciprocal of the number of frames in a video within one second.

[0102] It is understandable that SurfaceFlinger can also obtain views containing non-adjustable content from other components (such as normalView, button, etc.) besides the playback component a at a certain refresh rate, and then update the content carried in the corresponding layer, which is also used for screen refresh. This refresh rate can be the screen refresh rate or the refresh rate of each non-adjustable content; this embodiment does not limit it.

[0103] like Figure 6As shown, frames from video application 1 can be transmitted to layer 1. The parameters (metadata1 and ratio1) involved in inverting tone mapping on layer 1 are obtained, and the frames carried in layer 1 are updated during video playback, allowing users to view the video provided by video application 1 (video 1 after inverting tone mapping) on ​​their screen. The process of obtaining the frames carried in layer 1 and the parameters for inverting tone mapping on layer 1 can be found in [reference needed]. Figure 6 Middle (11) Figure 6 Middle (12a) Figure 6 Middle (12b) and Figure 6 The relevant content at point (12c) relates to the process and is respectively related to Figure 5 Middle (11) Figure 5 Middle (12a) Figure 5 Middle (12b) and Figure 5 The same applies to (12c), please refer to the above. Figure 5 The descriptions of the relevant content in the text will not be repeated here.

[0104] Subsequently, SurfaceFlinger executes Figure 6 The operation at (13) records layer 1 and its attributes metadata1 and ratio1 into the corresponding buffer area (bufffer1) of layer 1 when the frame in layer 1 is updated.

[0105] It should be noted that, Figure 6 as well as Figure 5 Similarly, Figure 6 The operation at (11) is executed after the operation of playing video 1 is detected. When video 1 reaches filter1, it is decoded while the video is playing, and then executed subsequently. Figure 6 Middle (12a) Figure 6 Middle (12b) and Figure 6 The operation of (12c). Figure 6The operation at (12a) is performed at a preset frequency. This preset frequency includes performing the operation once after the video starts playing, using the first frame of video 1 to determine metadata1, which remains unchanged thereafter. Alternatively, the preset frequency includes determining metadata1 once when a target frame is determined and sending it to SurfaceFlinger. This process includes: using the initialized metadata1 determined by the first frame (first target frame) in video 1, sending the initialized metadata1 to SurfaceFlinger, updating the metadata1 based on subsequent target frames, and sending the updated metadata1 to SurfaceFlinger. For details regarding target frames, please refer to the foregoing description; it will not be repeated here. For example, there can be a 30-frame interval between the i-th target frame and the (i-1)-th target frame. In this case, filter1 can determine metadata1 every 30 frames and then send it to SurfaceFlinger. It is not limited to 30 frames; other frame intervals are also possible, such as those between 10 and 30 frames, and should not be construed as limiting the embodiments of this application. When video 1 reaches filter1, the operation of stopping video 1 is detected. Video application 1 can notify filter1 not to decode video 1 when it stops playing, and not to update the frame in SurfaceView1 or update metadata1.

[0106] When filter1 sends metadata1 to SurfaceFlinger, it also sends the identifier of the corresponding video application 1 (e.g., APP package name1 and APP activity name1). In some possible cases, such as when the electronic device is running Android, filter1 can use a binder thread to transfer metadata1 and the identifier of video application 1 to SurfaceFlinger. The binder thread is implemented using the binder mechanism, a high-level inter-process communication (IPC) mechanism in the Android operating system used for inter-component communication. The binder mechanism allows applications and services to exchange data between different processes, while providing a rich set of interfaces and system-level management. The binder mechanism supports not only basic data transfer but also complex object and interface transfer. Through the binder thread, the electronic device can effectively manage service creation, reference counting, and cross-process calls, ensuring system stability and security during output transmission.

[0107] However, if the binder thread executes too frequently, it may cause system lag, therefore here... Figure 6 The operation at (12a) can be performed only when the target frame is detected. This way, when updating metadata1 to optimize the effect of inverse tone mapping, it will not cause the system to be overloaded and cause lag.

[0108] Continue to refer to Figure 6 The operation of playing video 2 was detected, and video application 2 can execute... Figure 6 The operation at (21) marks video 2 with a highlighting mark and sends it to filter2 for decoding. In addition, a playback component 2 (e.g., SurfaceView2) is created to play video 2, and a highlighting mark and a highlighting ratio 2 (denoted as extendedratio2, or simply ratio2) are added to the properties of the SurfaceView2.

[0109] Frames from video 2 provided by video application 2 (e.g., frame 2) can be transmitted to layer 2. The parameters (metadata2 and ratio2) involved in inverting tone mapping on layer 2 are obtained, and the frames in layer 2 are updated during video playback, allowing users to view the video provided by video application 2 (video 2 after inverting tone mapping) on ​​their screen. The process of obtaining the frames in layer 2 and the parameters for inverting tone mapping on layer 2 can be found in [reference needed]. Figure 6 (21) Figure 6 (22a) Figure 6 (22b) and Figure 6 The relevant content at (22c) in the middle, this process is respectively related to Figure 5 or Figure 6 Middle (11) Figure 5 or Figure 6 Middle (12a) Figure 5 or Figure 6 Middle (12b) and Figure 5 Similar to (12c), simply modify the corresponding numbers. For example, change APP package name1 and APP activity name1 to APP package name2 and APP activity name2 respectively, and change the view carrying frame 1. Figure 1 Modified to carry frame 2 of the video. Figure 2 Please refer to the above. Figure 5 The descriptions of the relevant content in the text will not be repeated here.

[0110] Subsequently, SurfaceFlinger executes Figure 6 The operation at (23) records layer 2 and its attributes metadata2 and ratio2 into the corresponding buffer area (bufffer2) of layer 2 when the frame in layer 2 is updated.

[0111] In addition to obtaining the layer containing the frame to be inverted and the properties for inverting the layer, SurfaceFlinger also needs to generate corresponding layers for non-adjustable content to be displayed on the screen and record the layer information in the corresponding cache. Here, the layer used to hold the non-adjustable content can be... Figure 6 Layer 0 is shown in the image. The buffer for layer 0 is (buffer0). An example of non-adjustable content may include the aforementioned... Figure 1 The content displayed in area 112. When increasing the brightness of areas used for video playback, such as screen area 1 (the area playing video 1), the overall brightness of the screen is increased. The layer information stored in buffer0 includes the ratio0 used when darkening layer 0. The process of recording layer 0 and its ratio0 into buffer0 can be found in [reference needed]. Figure 6 Middle (31) and Figure 6 The relevant operations at (32) in the middle.

[0112] Figure 6 As shown in (31), a non-adjustable application creates a display component 0 to host the non-adjustable application. The type and number of display components 0 are related to the type of the non-adjustable application. SurfaceFlinger converts the view that hosts the non-adjustable content into layer 0.

[0113] Figure 6 As shown in (32), SurfaceFlinger records layer 0 and ratio0 into buffero0. In some cases, SurfaceFlinger can use the minimum of the brightening ratios such as ratio1 and ratio2 as ratio0. This can prevent the screen from being overexposed when playing videos.

[0114] like Figure 6As shown in section (4), as content is written to or updated in the buffer, SurfaceFlinger calls the Graphic Engine to process the layers in each buffer accordingly, and then merges the processed layers to obtain the final screen image to be displayed. For layers that record metadata and ratio in the buffer (such as layer 1 and layer 2), the Graphic Engine calls the Image Processor to perform an inverse tone mapping algorithm on the layer. The Image Processor may include a graphics processing unit (GPU) and / or a display processing unit (DPU). Both the DPU and GPU have inverse tone mapping algorithms, which can perform inverse tone mapping processing on the layers. For layers that do not record metadata in the buffer but record ratio0 (such as layer 0), the Graphic Engine calls the Image Processor to perform darkening processing on layer 0, reducing the grayscale value of each pixel in layer 0 based on ratio0. Here, the darkening processing can be done by dividing each grayscale value by ratio0. The image engine includes a render engine and a hardware compositor (HWC). Layer compositing can be achieved using at least one of these components. For details on inverse tone mapping and darkening, please refer to the preceding descriptions; they will not be repeated here. Subsequently, SurfaceFlinger sends the screen image to the display driver.

[0115] like Figure 6 As shown in (5), SurfaceFlinger sends ratio0 to the display driver.

[0116] like Figure 6 As shown in (6), the display driver displays the screen image on the display screen (screen). When displaying the screen image, ratio0 is used to increase the screen brightness. For details on increasing screen brightness, please refer to the description of the relevant content above, which will not be repeated here.

[0117] It should be noted here that... Figure 6 Middle (5) and Figure 6 The operations shown in (4) have no order.

[0118] It should also be noted that, Figure 6The operation of transferring ratio0 to buffer0 is optional. For example, if the screen brightness can be adjusted in sections, meaning only the brightness of the screen area used for video playback can be adjusted, the operation of transferring ratio0 to buffer0 can be omitted. Similarly, the operation of darkening layer 0 can also be omitted. In this case, in... Figure 6 At point (6), when displaying the screen image, the display driver uses ratio0 to increase the brightness of the screen area used for video playback separately.

[0119] In some possible circumstances, Figure 6 The operation of increasing brightness based on ratio0 at point (6) is optional. If the screen brightness is already bright enough, the operation of increasing brightness based on ratio0 can be omitted.

[0120] It should also be noted that, Figure 6 The process involves decoding frame 1 of a video application 1, and then using SurfaceFlinger to display the video containing frame 1. Figure 1 This explanation uses layer 1, which carries frame 1, as an example. For details on how other frames in video application 1 are transmitted to layer 1, please refer to the description of how frame 1 is transmitted to layer 1. This will not be repeated here. Figure 6 The process involves decoding frame 2 of the video application 2, and then using SurfaceFlinger to display the video containing frame 2. Figure 2 This explanation uses layer 2, which carries frame 2, as an example. For details on how other frames in video application 2 are transmitted to layer 2, please refer to the description of how frame 2 is transmitted to layer 2. These details will not be repeated here.

[0121] Figure 6 The operations of adding the highlight marker and ratio1 from the properties of SurfaceView1 to the properties of the corresponding layer (Layer 1) of SurfaceView1, and adding the highlight marker and ratio2 from the properties of SurfaceView2 to the properties of the corresponding layer (Layer 2) of SurfaceView2, are performed through JNI transactions. JNI transactions are used to implement the exchange of data and control information via JNI (Java Native Interface). JNI is a mechanism for Java to call native (i.e., code written in C or C++) code, allowing data exchange and operation calls between Java code and native code. JNI transactions are commonly used for interaction between the application framework layer and the native framework layer.

[0122] Figure 6 The filters involved (e.g., filter1 and filter2) are configured in the media codec of the electronic device. The media codec supports various standard audio and video codec formats, including but not limited to H.264 and AAC. The media codec includes a multimedia codec framework, which provides API interfaces for efficient encoding (compression) and decoding (decompression) of audio and video data. The media codec API allows applications to directly access codecs, which may be software codecs or hardware-accelerated codecs. By using the media codec, electronic devices can achieve high-performance audio and video playback, recording, and streaming functions, while ensuring good performance and low system resource consumption.

[0123] Figure 6 In SurfaceFlinger, the buffers corresponding to each layer in the GraphicBuffer can be accessed via the Gralloc library. Figure 6 The GrallocMapper (not shown in the image) allocates memory for layers. For example, GrallocMapper can allocate buffer0 for layer 0, buffer1 for layer 1, and buffer2 for layer 2. GrallocMapper provides an API for mapping and unmapping Gralloc-allocated buffers. Through GrallocMapper, applications and system services can access Gralloc-allocated memory for read and write operations, enabling efficient graphics data processing. Gralloc is a library in the Android system used for graphics memory allocation. It is part of the HAL (Hardware Abstraction Layer) and is responsible for managing the allocation, mapping, and release of graphics buffers. Gralloc efficiently manages GPU and other graphics hardware resources, ensuring efficient processing and display of graphics data. Gralloc provides an API that allows applications and system services to request buffers of specific sizes and formats, typically used in scenarios such as video playback.

[0124] The aforementioned playback component can be a SurfaceView. A SurfaceView is a view component used to display video playback or perform complex rendering operations. A SurfaceView provides an independent drawing surface, allowing drawing on a thread other than the main thread, thereby enabling non-blocking UI operations and smoother animation or video playback. Besides a SurfaceView, videos can also be played in other Views; this application embodiment does not limit this to such methods.

[0125] Figure 7 An exemplary flow is shown when implementing a video processing method in a video playback scenario.

[0126] Figure 7 The example video playback scenario shown above can be referred to the foregoing. Figure 1 As shown, video 1 is played initially, then video 2 is played, and then video 1 is paused.

[0127] When playing video 1, the relevant steps involved in using inverse tone mapping on video 1 can be found in the following description of steps S101-S106.

[0128] S101. The operation of playing video 1 is detected. The video application 1 marks video 1 with a highlight mark and transmits it to filter1. The video application 1 also creates SurfaceView1 and adds ratio1 and the highlight mark to the properties of SurfaceView1.

[0129] The highlight marker indicates that inverse tone mapping is applied to Video 1. ratio1 is the highlighting ratio used when inverse tone mapping is applied to Video 1.

[0130] For a description of this process, please refer to [the relevant documentation / reference]. Figure 6 The description at (11) will not be repeated here.

[0131] S102.filter1 decodes video 1, obtains the frame data, and transmits the frame data from video 1 to SurfaceView1 to obtain a view containing the frame data. Figure 1 Then, SurfaceFlinger will display the view containing the image frames. Figure 1 Convert it to Layer 1.

[0132] For a description of this process, please refer to the aforementioned section. Figure 6 The description at (12b) will not be repeated here.

[0133] It should be noted that the JNI transaction will also add ratio1 and highlight markers to the properties of layer 1. This step... Figure 7Not shown in the image, please refer to the foregoing. Figure 6 The description at (12c).

[0134] S103.filter1 determines metadata1 based on the target screen frame and sends metadata1 and the identifier of video application 1 to SurfaceFlinger.

[0135] The relevant content for step S103 can be found in [reference]. Figure 6 The relevant description at (12a).

[0136] For a description of the target frame, please refer to the foregoing description. When the frame transmitted to SurfaceView1 in step S102 is the target frame, the target frame in step S103 is the frame transmitted to SurfaceView1 in step S102. When the frame transmitted to SurfaceView1 in step S102 is not the target frame, the target frame in step S103 is the frame transmitted to SurfaceView1 prior to the frame transmitted to SurfaceView1. In this case, step S103 will be executed before step S102.

[0137] SurfaceFlinger iterates through the layers. If it recognizes the identifier for video application 1 in the identifier of layer 1, it transfers layer 1, metadata1, and ratio1 (an attribute from layer 1) to the corresponding buffer1 of layer 1. This step... Figure 7 Not shown in the image, please refer to the following: Figure 6 The relevant description at (13) in the middle.

[0138] S104. The non-adjustable application creates a display component 0, transfers the non-adjustable content to display component 0, and obtains a view that carries the non-adjustable content. Then, SurfaceFlinger transforms the view carrying the non-adjustable content into layer 0, and transfers layer 0 and ratio1 to the buffer0 corresponding to layer 0.

[0139] Here, since only video 1 was played, SurfaceFlinger can determine that ratio1 is the ratio0 used to darken layer 0.

[0140] For details regarding step S104, please refer to the foregoing. Figure 6 The middle (31) and Figure 6 The description at (32) will not be repeated here.

[0141] S105.SurfaceFlinger calls the image engine to darken layer 0 based on ratio1, and performs inverse tone mapping on layer 1 based on metadata1 and ratio1. Then, it composites layer 0 and layer 1 to obtain the screen image.

[0142] For a description of this process, please refer to the foregoing. Figure 6 The description at point (4) will not be repeated here.

[0143] S106. The image engine sends the screen image to the display screen for display.

[0144] When displaying an image on the screen, the screen brightness is increased by ratio 1. A description of this process can be found in [reference needed]. Figure 6 The middle (5) and Figure 6 The description at point (6) will not be repeated here.

[0145] It's important to note that the frames in Layer 1 update as the video plays, allowing Video 1 to be displayed on the screen. The related metadata1 also updates. Furthermore, when Video 1 plays, the content in Layer 0 also updates, allowing updates to non-adjusted content to be displayed on the screen.

[0146] In the scenario where video 1 is paused while video 2 is playing, the relevant information regarding the inverse tone mapping process applied to video 2 can be found in the following description of steps S107-S111.

[0147] S107. The operation of playing video 2 is detected. Video application 2 adds a highlight mark to video 2 and transmits it to filter2. Video application 2 also creates SurfaceView2 and adds ratio2 and the highlight mark to the properties of SurfaceView2.

[0148] The highlight marker indicates that inverse tone mapping is applied to Video 2. Ratio2 is the highlighting ratio used when performing inverse tone mapping on Video 2.

[0149] For a description of this process, please refer to [the relevant documentation / reference]. Figure 6 The description in (21) will not be repeated here.

[0150] S108.filter2 decodes video 2, obtaining image frames. These image frames from video 2 are then transmitted to SurfaceView2, resulting in a view containing the image frames. Figure 2 Then, SurfaceFlinger will display the view containing the image frames. Figure 2 Convert it to layer 2.

[0151] For a description of this process, please refer to the aforementioned section. Figure 6The description at (22b) will not be repeated here.

[0152] It should be noted that the JNI transaction will also add ratio2 and highlight markers to the properties of layer 2. This step... Figure 7 Not shown in the image, please refer to the foregoing. Figure 6 The description at (22c) in the middle.

[0153] S109.Filter2 determines metadata2 based on the target screen frame and sends metadata2 and the identifier of video application 2 to SurfaceFlinger.

[0154] The relevant content for step S108 can be found in [reference]. Figure 6 The relevant description at (22a).

[0155] For a description of the target frame, please refer to the preceding description. When the frame transmitted to SurfaceView2 in step S108 is the target frame, the target frame in step S109 is the frame transmitted to SurfaceView2 in step S108. When the frame transmitted to SurfaceView2 in step S108 is not the target frame, the target frame in step S109 is the frame transmitted to SurfaceView2 prior to the frame transmitted to SurfaceView2. In this case, step S109 will be executed before step S108.

[0156] SurfaceFlinger iterates through the layers. If it recognizes the identifier for video application 2 in the identifier of layer 2, it transfers layer 2, metadata2, and ratio2 (attributes from layer 2) to the corresponding buffer2 of layer 2. This step... Figure 7 Not shown in the image, please refer to the following: Figure 6 The relevant description at (23) in the middle.

[0157] S110.SurfaceFlinger calls the image engine to darken layer 0 based on ratio0, performs inverse tone mapping on layer 1 based on metadata1 and ratio1, and performs inverse tone mapping on layer 2 based on metadata2 and ratio2. Then, layers 0, 1, and 2 are composited to obtain the screen image.

[0158] The content carried in layer 0 in step S110 can be the same as or different from the content carried in layer 0 in step S105 above, and ratio0 is the minimum value between ratio1 and ratio2. The frame carried in layer 1 in step S110 is the frame when video 1 stops playing. After the playback stops, the frame in layer 1 will no longer be updated.

[0159] For a description of this process, please refer to the foregoing. Figure 6 The description at point (4) will not be repeated here.

[0160] S111. The image engine sends the screen image to the display screen for display.

[0161] When displaying an image on the screen, the screen brightness is increased by a ratio of 0. A description of this process can be found in [reference needed]. Figure 6 The middle (5) and Figure 6 The description at point (6) will not be repeated here.

[0162] It's important to note that as the video frames in Layer 2 update during playback, the related metadata 2 also updates, allowing Video 2 to play on the screen. Furthermore, as Video 2 plays, the content in Layer 0 also updates, ensuring that updates to non-adjustable content are displayed on the screen.

[0163] It should also be noted that the aforementioned Figure 7 In this process, step S104 is not related to the execution order of other steps. If there is an update to the content that is not being adjusted, step S104 can be executed.

[0164] The inverse tone mapping process in the embodiments of this application can also be referred to as the anti-tone mapping process.

[0165] It should be noted that the SDR video encoding format involved in this application embodiment uses a narrower color and brightness range to display images. This means that SDR video cannot display the details and color saturation that HDR video can display. Through the video processing method involved in this application embodiment, without changing the encoding format of the SDR video to that of HDR video, the display effect of the video during playback is presented at the level of HDR video. This method is not limited to SDR video; any video requiring improved dynamic range can adopt the video processing method involved in this application embodiment. For example, in addition to SDR video, some HDR videos can also use this video processing method to further improve their dynamic range; this application embodiment does not limit this.

[0166] The exemplary electronic device provided in the embodiments of this application will be introduced first below.

[0167] Figure 8This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0168] The following uses an electronic device as an example to illustrate the embodiments in detail. It should be understood that the electronic device may have more than Figure 8 The more or fewer components shown can be combined into two or more components, or they can have different component configurations. Figure 8 The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0169] The electronic device may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0170] The processor 110 may include one or more processing units, such as an application processor (AP), a graphics processing unit (GPU), a video codec, and a display processing unit (DPU). These different processing units may be independent devices or integrated into one or more processors.

[0171] Among them, the DPU and GPU can be used to render and synthesize the image to be displayed on the 194-inch screen.

[0172] In this embodiment of the application, the processor 110 can call computer instructions stored in the internal memory 121 to cause the electronic device to execute the method in this embodiment of the application.

[0173] This application also provides a chip system including at least one processor for implementing the functions involved in the methods performed by the electronic device in any of the above embodiments.

[0174] In one possible design, the chip system also includes a memory for storing program instructions and data, which may be located inside or outside the processor.

[0175] The chip system can consist of chips or include chips and other discrete components.

[0176] Optionally, the chip system may contain one or more processors. These processors can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.

[0177] Optionally, the chip system may contain one or more memories. These memories may be integrated with the processor or separated from it; this application does not limit the specific implementation.

[0178] For example, the memory can be a non-transient processor, such as a read-only memory (ROM), which can be integrated on the same chip as the processor or set on different chips. This application does not specifically limit the type of memory or the way the memory and processor are set.

[0179] This application also provides a computer program product comprising: a computer program (also referred to as code or instructions) that, when run, causes a computer to perform the method executed by the electronic device in any of the above embodiments.

[0180] This application also provides a computer-readable storage medium storing a computer program (also referred to as code or instructions). When the computer program is run, it causes the computer to perform the method executed by the electronic device in any of the above embodiments.

[0181] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0182] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0183] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0184] The terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0185] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0186] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A video processing method, characterized in that, Applied to electronic devices, the method includes: The operation of playing the first video was detected immediately. The frame in the first video is subjected to inverse tone mapping; the dynamic range of the frame after processing is greater than that of the frame before processing, and the greater dynamic range includes: the pixel with the largest grayscale value in the first frame before processing has a larger grayscale value in the first frame after processing. The first video is played through the first screen area; The brightness of the first screen area is increased using a first ratio, and the brightness of the first screen area is controlled to be higher than the brightness of the first screen area at the first time.

2. The method according to claim 1, characterized in that, The method further includes: When the brightness of the first screen area is increased using the first ratio, and the brightness of the second screen area is increased using the first ratio, the grayscale value of each pixel in the second screen area is decreased using the first ratio; the second screen area does not include the image frame processed by inverse tone mapping.

3. The method according to claim 1 or 2, characterized in that, If the screen further includes a third screen area for playing a second video, the first ratio is the minimum of a first brightness ratio corresponding to the first video and a second brightness ratio corresponding to the second video; the first brightness ratio is set by the application providing the first video, and the second brightness ratio is set by the application providing the second video; if the screen does not include the third screen area, the first ratio is the first brightness ratio; if the screen further includes a third screen area for playing a second video, the method further includes: Perform inverse tone mapping on the frames in the second video; The second video is played through the third screen area; Increase the brightness of the third screen area using the first ratio.

4. The method according to claim 3, characterized in that, The first video includes a first frame. Performing inverse tone mapping on the frames in the first video specifically includes: performing inverse tone mapping on the first frame in the first video. The process of performing inverse tone mapping on the first frame includes: Place the first frame into the first layer; the first layer is used to hold the frame to be played in the first video; The grayscale values ​​of the pixels in the first layer are used as input grayscale values. The input grayscale values ​​are converted into output grayscale values ​​through the first lookup table to obtain the converted first layer. The dynamic range of the converted first layer is greater than that of the first layer before conversion.

5. The method according to claim 4, characterized in that, The process of determining the first lookup table includes: The cumulative percentage of pixels in each grayscale interval of each color channel in the second frame is determined to obtain the cumulative percentage distribution of pixels; the cumulative percentage of pixels in a grayscale interval under a color channel is equal to the ratio of the number of pixels whose grayscale value is less than or equal to the maximum grayscale value of the grayscale interval to the total number of pixels in the second frame; the second frame is the first frame or a frame in the first video that precedes the first frame. Based on the cumulative pixel proportion distribution and the first brightening ratio, a first lookup table is obtained; the first lookup table includes: the output grayscale value corresponding to each of the multiple input grayscale values ​​under each color channel; the multiple input grayscale values ​​under one color channel include a first type of input grayscale value and a second type of input grayscale value; wherein, the first type of input grayscale value under one color channel includes: the maximum grayscale value of each grayscale interval under the one color channel; the first type of input grayscale value is obtained by interpolating the first type of input grayscale value based on a first mapping relationship, the first mapping relationship including the first brightening ratio and the cumulative pixel proportion of the grayscale interval in which the first type of input grayscale value is located; the second type of input grayscale value is obtained by interpolating the first type of input grayscale value, and the output grayscale value corresponding to the second type of grayscale value is obtained by interpolating the output grayscale value corresponding to the first type of grayscale value.

6. The method according to claim 5, characterized in that, After obtaining the converted first layer, the method further includes: The converted first layer and second layer are merged to obtain a screen image, which is then displayed on the screen. The screen image includes a first frame displayed in the first screen area. The second layer includes a layer for carrying the content of the second screen area. If a second video exists, in addition to the layer for carrying the content of the second screen area, the second layer also includes a layer carrying a third frame. The third frame comes from the second video, and the grayscale values ​​of the pixels in the layer carrying the third frame are obtained by conversion using a second lookup table. The second lookup table is obtained based on the cumulative pixel proportion distribution of the fourth frame and the second brightening ratio. The fourth frame is the third frame or a frame preceding it.

7. The method according to any one of claims 4-6, characterized in that, The second frame is the first frame or a frame in the first video that precedes the first frame. Specifically, if the first frame is a target frame, the second frame is the first frame; if the first frame is not a target frame, the second frame is the target frame whose timestamp is closest to the first frame. The first video includes at least one target frame, the first target frame is the first frame of the first video, and the i-th target frame and the (i-1)-th target frame satisfy a preset condition, where i is an integer greater than or equal to 2.

8. The method according to claim 7, characterized in that, The preset conditions include: the similarity between the i-th target frame and the (i-1)-th target frame is less than a preset threshold, or the i-th target frame and the (i-1)-th target frame are separated by a preset number of frames.

9. The method according to any one of claims 6-8, characterized in that, After determining the cumulative pixel percentage of each grayscale interval in each color channel of the second frame and obtaining the cumulative pixel percentage distribution, the method further includes: The video encoder records the cumulative pixel percentage distribution in a first buffer via a binder thread. The first buffer also includes the first brightening ratio. The first buffer is used to cache the first layer. The video encoder is also used to perform the operation of obtaining the cumulative pixel percentage distribution. Based on the cumulative pixel percentage distribution and the first brightening ratio, a first lookup table is obtained, specifically including: The cumulative pixel percentage distribution and the first brightening ratio are obtained from the first cache by an image processor; the image processor includes at least one of a graphics processing unit (GPU) and a display processing unit (DPU). The image processor determines a first mapping relationship based on the cumulative pixel percentage distribution and the first brightening ratio, and obtains the first lookup table according to the first mapping relationship.

10. The method according to any one of claims 6-8, characterized in that, Before determining the cumulative pixel percentage of each grayscale interval in each color channel of the second frame and obtaining the cumulative pixel percentage distribution, the method further includes: The video encoder determines the number of pixels in each grayscale interval of each color channel in the second frame to obtain the pixel distribution; the number of pixels in a grayscale interval under a color channel is equal to the number of pixels in the grayscale interval. The video encoder records the pixel distribution into a first buffer area via a binder thread, and the first buffer area also includes the first brightening ratio; Based on the cumulative pixel percentage distribution and the first brightening ratio, a first lookup table is obtained, specifically including: The image processor obtains the pixel distribution and the first brightening ratio from the first cache, and obtains the cumulative pixel percentage distribution based on the pixel distribution; the image processor includes at least one of a graphics processing unit (GPU) and a display processing unit (DPU); The image processor determines a first mapping relationship based on the cumulative pixel percentage distribution and the first brightening ratio, and obtains the first lookup table according to the first mapping relationship.

11. The method according to claim 10, characterized in that, The video encoder records the pixel distribution into the first buffer via the binder thread, specifically including: The video encoder transmits the pixel distribution to SurfaceFlinger via the binder thread. When SurfaceFlinger receives the pixel distribution, it also receives the first application identifier corresponding to the pixel distribution sent by the video encoder. If the attributes of the first layer include a highlighting indicator, the SurfaceFlinger determines that the first layer is used to carry the frame of the first video provided by the first application through the identifier of the first layer, and records the pixel distribution corresponding to the identifier of the first application in the first buffer area; the highlighting indicator included in the attributes of the first layer comes from the application providing the first video; the highlighting indicator is used to indicate that the first video is a video to be subjected to inverse tone mapping processing; the playback component is located in the first screen area.

12. The method according to claim 11, characterized in that, When recording the pixel distribution corresponding to the first application identifier into the first cache area, the method further includes: The SurfaceFlinger obtains the first brightening ratio from the properties of the first layer and records it in the first cache area; the brightening indicator and the first brightening ratio included in the properties of the first layer both come from the application that provides the first video; The process of transferring the highlighting indicator and the first highlighting ratio from the application providing the first video to the attributes of the first layer includes: A playback component for playing the first video is created by providing the application that provides the first video, and a highlighting mark and the first highlighting ratio are added to the properties of the playback component. After the video encoder decodes the first frame, it places it into the playback component to form a view including the first frame. The SurfaceFlinger transforms the view carrying the first frame into the first layer; The Java native interface transaction adds the highlight marker and the first highlight ratio from the playback component's properties to the properties of the first layer.

13. The method according to any one of claims 1-12, characterized in that, Before increasing the brightness of the first screen area using the first ratio, the method further includes: It is determined that the screen is operating in pulse width modulation (PWM) mode.

14. An electronic device, characterized in that, include: One or more processors and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1-13.

15. A chip system applied to an electronic device, characterized in that, The chip system includes one or more processors, which are configured to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1-13.

16. A computer program product containing instructions, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-11.

17. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-13.

Images