Image processing method and electronic device

Abstract

The embodiments of the present application provide an image processing method and an electronic device, and relate to the field of image processing technology. The method includes: obtaining display information of an image to be displayed at a current target backlight luminance of a display, and obtaining display information of an image to be displayed at a luminance higher than the current target backlight luminance and of the display, and obtaining the display information of the image to be displayed based on the two types of display information, so that in a scenario where the backlight of the display is adjustable, the display effect of the image can be improved.

Description

[Technical Field]

[0001] This application claims priority under Chinese Patent Application No. 202210033857.8, titled "Image Processing Method and Electronic Device," filed with the China National Intellectual Property Administration on 12 January 2022, which is incorporated herein by reference in its entirety.

[0002] Embodiments of this application relate to the field of image processing technology, and more particularly to image processing methods and electronic devices. [Background technology]

[0003] Currently, when displaying an image or video, an electronic device may perform tone mapping to the display brightness of image pixels on the display, based on the maximum display brightness of the display.

[0004] However, current tone mapping methods offer poor image or video display effectiveness in scenarios where the display backlight is adjustable. [Overview of the project]

[0005] To solve the aforementioned technical problems, this application provides an image processing method and apparatus. In the method, display information of the image to be displayed on the display can be obtained based on the maximum display brightness of the display and the current target backlight brightness, and as a result, the image display effect can be improved in a scenario in which the backlight of the display is adjustable.

[0006] According to the first aspect, an embodiment of the present invention provides an image processing method. The method comprises the steps of: acquiring first display information of a target image to be displayed based on the current target backlight brightness of the display; acquiring second display information of the target image based on the maximum display brightness and the current target backlight brightness of the display; processing the first and second display information based on preset information of the target image to acquire third display information of the target image, where the preset information includes information regarding the brightness of the target image; and displaying the target image on the display based on the third display information. Thus, considering that in a scenario in which the display has a specific backlight brightness, the actual brightness of the display may be higher than the backlight brightness, in this method, the first display information of the image may be acquired by referring to the backlight brightness, then the second display information of the image may be acquired by referring to the maximum display brightness and the backlight brightness of the display, and then the two types of display information may be processed to acquire final display information of the image. Thus, the final display information may be a combination of the backlight brightness and the maximum display brightness, as a result of improving the image matching effect on the display, thereby improving the image display effect.

[0007] For example, when second display information is acquired, the second display information for the image may be based on a brightness that is greater than the current target backlight brightness and less than or equal to the maximum display brightness. This prevents the brightness of the highlight areas from being excessively increased when the target image is displayed based on the third display information.

[0008] For example, the target image may be an independent image or an image of a single frame in a video. This is not limited to the present invention.

[0009] For example, the current target backlight brightness is the expected backlight brightness.

[0010] For example, suppose the maximum display brightness of a display is 500 nits, and the current backlight intensity level is set to 10%, resulting in a current target backlight brightness of 50 nits. However, after the current backlight brightness level is set to 10%, the current backlight brightness of the display (i.e., the actual backlight brightness) may be higher than 50 nits, for example, 100 nits. Thus, the backlight brightness corresponding to the backlight intensity level here can be referred to as the current target backlight brightness, i.e., the backlight brightness expected by the system or user.

[0011] According to the first aspect, the step of obtaining first display information of a target image to be displayed based on the current target backlight brightness of the display includes: obtaining a first tone mapping curve of the target image to be displayed; and processing the first tone mapping curve based on the current target backlight brightness of the display and obtaining a second tone mapping curve of the target image as first display information.

[0012] For example, the first tone mapping curve may be obtained from the source of the target image or generated at the image display edge. This is not limited to the present invention.

[0013] Source information can be any one or more types of information obtained by analyzing the information carried in the image. Source information can be metadata information of the image, which is carried in the bitstream for transmission.

[0014] For example, the first tone mapping curve may be a global initial tone mapping curve or a local initial tone mapping curve that is carried in the bitstream.

[0015] For example, when the first tone mapping curve is a local (e.g., local image region of the target image) initial tone mapping curve, the third display information of the local image region of the target image can be obtained according to the method in the first aspect.

[0016] For example, when the first tone mapping curve is a global (e.g., target image) initial tone mapping curve, the third display information of the target image can be obtained according to the method in the first aspect.

[0017] For example, when the first tone mapping curve is processed based on the current target backlight luminance of the display, the processing methods include, but are not limited to, scaling processing, translation processing, and the like. This is not limited in the present application.

[0018] According to any one of the first aspect or the above-mentioned implementations of the first aspect, the step of obtaining the second display information of the target image based on the maximum display luminance and the current target backlight luminance of the display includes: determining a first luminance based on the maximum display luminance and the current target backlight luminance, where the first luminance is higher than the current target backlight luminance; and processing the first tone mapping curve based on the first luminance, and obtaining the third tone mapping curve of the target image as the second display information.

[0019] For example, the first luminance may be smaller than or equal to the maximum display luminance.

[0020] For example, when the first tone mapping curve is processed based on the first luminance, the processing methods include, but are not limited to, scaling processing, translation processing, and the like. This is not limited in the present application.

[0021] According to either the first embodiment or the above-described implementation of the first embodiment, the pre-set information includes a second brightness of the target image, and the steps of processing the first display information and the second display information based on the pre-set information of the target image to obtain a third display information of the target image include: obtaining the second brightness of the target image; extracting a first subcurve from the second tone mapping curve in which the pixel brightness is less than or equal to the second brightness; obtaining a first tone mapping value in the second tone mapping curve that corresponds to the second brightness; processing the third tone mapping curve based on the first tone mapping value to generate a fourth tone mapping curve, where the tone mapping value in the fourth tone mapping curve that corresponds to the second brightness is the first tone mapping value; extracting a second subcurve from the fourth tone mapping curve in which the pixel brightness is greater than or equal to the second brightness; and connecting the first subcurve and the second subcurve based on the second brightness to obtain the target tone mapping curve of the target image as third display information.

[0022] For example, the second luminance can be called an anchor. The anchor (here, the second luminance) can be a global anchor, i.e., an anchor for the target image, or a local anchor, i.e., an anchor for a local image region in the target image. When a target tone mapping curve is generated for an image or an image region of one frame, a pair of initial tone mapping curves and anchors can be used for implementation.

[0023] For example, when a target tone mapping curve is generated for a single frame of an image, the initial tone mapping curve for the image frame and the anchor for the image frame may be used for implementation.

[0024] For example, when a target tone mapping curve is generated for a local image region in a single frame of an image, the initial tone mapping curve of the local image region in the image frame, and the anchor of the local image region, may be used for implementation.

[0025] For example, a global anchor can be used as an anchor for images of multiple frames or multiple image regions, and a global initial tone mapping curve can be used as an initial tone mapping curve for a frame sequence (i.e., images of multiple frames) or for multiple image regions.

[0026] For example, when an anchor for a target image is obtained, the anchor may be obtained from the source, that is, the anchor for the video or image may be sent to the bitstream.

[0027] For example, when an anchor for a target image is obtained, the anchor may alternatively be generated by the image or video display end (i.e., the display device to which the display belongs). This is not limited to the present invention.

[0028] For example, if the second luminance of the target image is, for instance, the global anchor of the frame of the target image, then the second luminance may be a value between the minimum and maximum luminance values ​​of the image pixels of the target image.

[0029] For example, if the second luminance of the target image is, for example, the local anchor of the local image region in the target image, then the second luminance can be a value between the minimum and maximum luminance values ​​of the image pixels in the local image region.

[0030] For example, the tone mapping curve in each implementation of the first embodiment may represent the mapping relationship between the luminance of a pixel in an image (i.e., pixel luminance, pixel brightness) and the display brightness on the display.

[0031] According to the first embodiment, or any one of the above-described implementations of the first embodiment, the step of obtaining a first tone mapping curve of a target image to be displayed includes: obtaining a first tone mapping curve for each of a plurality of local regions constituting the target image; the step of obtaining a second brightness of the target image includes: obtaining a second brightness for each of a plurality of local regions; and the step of connecting a first sub-curve and a second sub-curve based on the second brightness and obtaining the target tone mapping curve of the target image as third display information includes: for a plurality of local regions of the target image, connecting a first sub-curve and a second sub-curve for each local region based on the second brightness of each local region, and obtaining the target tone mapping curve for each of the plurality of local regions as third display information of the target image.

[0032] In this implementation, each local region in the target image has a corresponding first tone mapping curve (i.e., a local tone mapping curve) and a corresponding anchor (i.e., a local anchor). In this case, for each local region, a corresponding target tone mapping curve can be generated for each local region based on the process described in the above-described implementation of the first embodiment. Next, when the target image is displayed, tone mapping is performed on each local region based on the corresponding target tone mapping curve, thereby implementing display adaptation of the target image on a display with a specific backlight.

[0033] According to either the first embodiment or the above-described implementation of the first embodiment, the step of obtaining a second brightness of the target image includes the step of obtaining a second brightness of the target image based on the minimum and average pixel brightness of the target image.

[0034] For example, when the second luminance is an anchor for all image regions of the target image, the anchor can be obtained based on the minimum and average pixel luminances of the image pixels in the target image.

[0035] For example, when the second luminance is an anchor for the local image region of the target image, the anchor can be obtained based on the minimum pixel luminance and the average pixel luminance of the image pixels in the local image region.

[0036] According to either the first embodiment or the above-described implementation of the first embodiment, the steps of obtaining a second brightness of a target image based on the minimum and average pixel brightness of the target image include: a step of clustering image pixels of the target image based on the pixel brightness and pixel position in the target image to obtain a plurality of image regions; a step of classifying the plurality of image regions in the target image based on pixel brightness to obtain a first type image region and a second type image region, where the brightness of the first type image region is higher than the brightness of the second type image region; and a step of determining the second brightness of the target image based on the minimum and average pixel brightness of the first type image region.

[0037] For example, when a target image is divided into multiple image regions, pixels with similar pixel brightness and spatial positions can be divided into a single image region based on their pixel brightness and spatial position in the target image, thereby implementing image region division. This is not limited to the clustering method described above.

[0038] For example, the first type of image region may be called the highlight region, and the second type of image may be called the non-highlight region.

[0039] For example, to distinguish between highlighted and non-highlighted areas, these areas can be distinguished by comparing the average brightness values ​​of image regions having a preset threshold. This is not limited to the present invention.

[0040] According to either the first embodiment or the above-described implementation of the first embodiment, the step of obtaining first display information of a target image to be displayed based on the current target backlight brightness of the display includes: processing the target image to be displayed based on the current target backlight brightness of the display to obtain the first image as first display information.

[0041] For example, the image pixels of a target image may be processed based on the current target backlight brightness, and the processing method may include, but is not limited to, image processing methods such as image scaling.

[0042] According to either the first embodiment or the above-described implementation of the first embodiment, the step of obtaining second display information of a target image based on the maximum display brightness of the display and the current target backlight brightness includes: determining a third brightness based on the maximum display brightness of the display and the current target backlight brightness, wherein the third brightness is higher than the current target backlight brightness, and processing the target image based on the third brightness to obtain a second image as second display information.

[0043] For example, the third brightness level may be less than or equal to the maximum display brightness level.

[0044] For example, when a target image is processed based on a third luminance, the processing method includes, but is not limited to, image processing methods such as scaling. This is not limited to the present invention.

[0045] According to either the first embodiment or the above-described implementation of the first embodiment, the pre-configured information includes the gain coefficient of the target image, and the steps of processing the first display information and the second display information based on the pre-configured information of the target image to obtain the third display information of the target image include: obtaining the gain coefficient of the target image; and processing the first image and the second image based on the gain coefficient to generate the third image as the third display information.

[0046] For example, the gain coefficient of the target image may be at the pixel level or the downsampling gain coefficient.

[0047] For example, the gain coefficient can be used to perform brightness adjustment on the image pixels of a target image.

[0048] For example, the gain coefficient of a 5x5 pixel unit could be 1.2, which indicates that the image pixels in the 5x5 image region of the target image become 1.2 times brighter.

[0049] For example, regions with large brightness differences in the target image correspond to different gain coefficients, and the gain coefficient of the image region with higher brightness is higher than the gain coefficient of the image region with even higher brightness.

[0050] For example, the gain coefficient of the highlight region in the target image is greater than the gain coefficient of the non-highlight region in the target image.

[0051] According to either the first embodiment or the above-described implementation of the first embodiment, the steps of obtaining the gain coefficient of a target image include: a step of clustering image pixels of the target image based on pixel brightness and pixel position in the target image to obtain a plurality of image regions; a step of classifying the plurality of image regions in the target image based on pixel brightness to obtain a first type image region and a second type image region, wherein the brightness of the first type image region is higher than the brightness of the second type image region; and a step of configuring different gain coefficients for the first type image region and the second type image region, wherein the gain coefficient of the first type image region is higher than the gain coefficient of the second type image region.

[0052] For example, when a target image is divided into multiple image regions, pixels with similar pixel brightness and spatial positions can be divided into a single image region based on their pixel brightness and spatial position in the target image, thereby implementing image region division. This is not limited to the clustering method described above.

[0053] For example, the first type of image region may be called the highlight region, and the second type of image may be called the non-highlight region.

[0054] For example, to distinguish between highlighted and non-highlighted areas, these areas can be distinguished by comparing the average brightness values ​​of image regions having a preset threshold. This is not limited to the present invention.

[0055] Thus, the highlight areas in the target image may become brighter, and the non-highlight areas may become darker, thereby enabling the display adaptation of the target image to a display with a specific backlight brightness.

[0056] According to either the first embodiment or the above-described implementation of the first embodiment, the step of obtaining third display information of the target image by processing first display information and second display information based on pre-configured information of the target image includes: obtaining pre-configured information of the target image from source information of the target image to be displayed; and processing first display information and second display information based on pre-configured information to obtain third display information of the target image.

[0057] For example, pre-configured information may be transported in a bitstream and transmitted to an end corresponding to an image processing method. As a result, when a backlight is present on the display, the display adaptation of the video or image on the display may be performed based on the pre-configured information, and the display effect may be improved.

[0058] Pre-configured information is carried in the source information of the target image, and as a result, the image display end can directly use the pre-configured information to perform image display adaptation under the display's backlight, and the image display end does not need to generate the pre-configured information, thereby reducing the display delay of the image or video.

[0059] According to a second aspect, an embodiment of the present invention provides an electronic device. The electronic device comprises a memory and a processor. The memory is coupled to the processor. The memory stores program instructions. When program instructions are executed by the processor, the electronic device can implement either the first aspect or an implementation thereof.

[0060] Either the second embodiment or an implementation of the second embodiment may correspond to either the first embodiment or an implementation of the first embodiment. For technical effects corresponding to either the second embodiment or an implementation of the second embodiment, please refer to the technical effects corresponding to either the first embodiment or an implementation of the first embodiment. Further details will not be explained here.

[0061] According to a third aspect, an embodiment of the present invention provides a chip comprising one or more interface circuits and one or more processors. The interface circuits are configured to receive signals from the memory of an electronic device and to transmit signals to the processors. The signals include computer instructions stored in the memory. When the processors execute the computer instructions, the electronic device can perform a method according to either the first aspect or an implementation of the first aspect.

[0062] Any third aspect and any one implementation of the third aspect may correspond to any one of the first aspect and any one implementation of the first aspect. For technical effects corresponding to any one of the third aspect and any one implementation of the third aspect, please refer to the technical effects corresponding to any one of the first aspect and any one implementation of the first aspect. Further details will not be explained here.

[0063] According to a fourth aspect, an embodiment of the present application provides a computer-readable storage medium that stores a computer program. When the computer program is running on a computer or processor, the computer or processor can perform a method according to the first aspect or any one of the possible implementations of the first aspect.

[0064] Any fourth aspect and any one implementation of the fourth aspect may correspond to any one implementation of the first aspect. For technical effects corresponding to any one implementation of the fourth aspect and any one implementation of the fourth aspect, please refer to the technical effects corresponding to any one implementation of the first aspect and any one implementation of the first aspect. Further details will not be explained here.

[0065] According to the fifth aspect, an embodiment of the present application provides a computer program product. The computer program product includes a software program, and when the software program is executed by a computer or processor, a method according to the first aspect or one of possible implementations thereof is performed.

[0066] Either the fifth aspect or its implementation may correspond to either the first aspect or its implementation. For technical effects corresponding to either the fifth aspect or its implementation, please refer to the technical effects corresponding to either the first aspect or its implementation. Further details will not be explained here. [Brief explanation of the drawing]

[0067] To more clearly illustrate the technical solutions in the embodiments of the present application, the accompanying drawings used in describing the embodiments of the present application are briefly described below. It is clear that the accompanying drawings in the following description illustrate only a few embodiments of the present application, and those skilled in the art can still derive other drawings from these accompanying drawings without creative effort.

[0068] [Figure 1] This is an example of a system architecture diagram.

[0069] [Figure 2] This is an example of a diagram illustrating the architecture of a system application.

[0070] [Figure 3]This is an example of a diagram illustrating mapping relationships in dynamic range.

[0071] [Figure 4] This is an example diagram illustrating a scenario where the backlight brightness of a mobile phone is adjusted.

[0072] [Figure 5a] This is an example of a system architecture diagram.

[0073] [Figure 5b] This is an example of a diagram illustrating an image processing process.

[0074] [Figure 5c] This is an example of a diagram illustrating the data generation process.

[0075] [Figure 5d] This is an example of a diagram illustrating the image display adaptation process.

[0076] [Figure 5e] This is an example of a diagram illustrating the curve generation process.

[0077] [Figure 6a] This is an example of a system architecture diagram.

[0078] [Figure 6b] This is an example of a diagram illustrating an image processing process.

[0079] [Figure 6c] This is an example of a diagram illustrating the image display adaptation process.

[0080] [Figure 7] This is a diagram showing the structure of the apparatus according to the embodiment of the present application.

[0081] [Figure 8] This is a diagram showing the structure of a chip according to an embodiment of the present invention. [Modes for carrying out the invention]

[0082] Hereinafter, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings. It is clear that the embodiments described are part of, but not all of, the embodiments of the present application. All other embodiments obtained by those skilled in the art without creative effort based on the embodiments of the present application are included within the scope of protection of the present application.

[0083] In this specification, the term "and / or" describes only the relationship between the subjects in question and indicates that three relationships may exist. For example, A and / or B may mean that only A exists, both A and B exist, and only B exists.

[0084] In the specification and claims of embodiments of this application, the terms “first,” “second,” and similar are intended to distinguish different subjects and are not used to describe a particular order of subjects. For example, “first target subject,” “second target subject,” and similar are intended to distinguish different target subjects and are not used to describe a particular order of target subjects.

[0085] In embodiments of this Application, the words “example,” “for example,” or similar are used to indicate that an example, illustration, or explanation is being given. Any embodiment or design scheme described as “example” or “for example” in any of the embodiments of this Application should not be described as being preferable or having more advantages than another embodiment or design scheme. Strictly speaking, the use of the words “example,” “for example,” or similar is intended to present a related concept in a particular manner.

[0086] In the description of embodiments of this application, unless otherwise specified, “multiple” means two or more. For example, “multiple processing units” means two or more processing units, and “multiple systems” means two or more systems.

[0087] Figure 1 is an example of a diagram illustrating the structure of a system framework. The system in Figure 1 includes a transmitting end and a receiving end.

[0088] It should be understood that the system shown in Figure 1 is merely an example. The system in this application, and the transmitting and receiving ends in the system, may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. The components shown in Figure 1 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processors and / or application-specific integrated circuits.

[0089] For example, the transmitting end may be an image or video generation end, and the receiving end may be an image or video display end.

[0090] As shown in Figure 2(1), the transmitting end and the receiving end can be configured in the same electronic device.

[0091] For example, an electronic device is a mobile phone, and the transmitting end may include a camera, and the receiving end may include a display.

[0092] The transmitting end may generate a bitstream based on the video data collected by the camera, in which the bitstream carries backlight metadata. The transmitting end may transmit the bitstream to the receiving end's display for display.

[0093] For example, an application scenario of the system shown in Figure 1 could be a process in which a user takes a video using the camera of their mobile phone and then displays the captured video on the mobile phone's screen.

[0094] For example, the transmitting end may be integrated with an encoder, and the receiving end may be integrated with a decoder.

[0095] As shown in Figure 2(2), the transmitting end and the receiving end may be configured in different electronic devices.

[0096] For example, electronic device 1 is a mobile phone, and electronic device 2 is a television. The mobile phone can transmit a bitstream of video data captured by the mobile phone to the television, either by using a Wi-Fi® network or via Bluetooth®, and the video data can be displayed on the television's screen.

[0097] For example, electronic device 1 is an electronic device used by a video software vendor to capture and create video, and electronic device 1 can transmit the created video bitstream (including backlight metadata) to an application server, and electronic device 2 is a mobile phone. The mobile phone receives user input from a user to a video application installed on the mobile phone. In response to user input, the mobile phone can retrieve the video bitstream transmitted by electronic device 1 from the application server and display and play the video data on the mobile phone's display.

[0098] It should be noted that an electronic device may be a terminal, or may be referred to as a terminal device. A terminal may be a cellular phone, a pad, a wearable device, a television, a personal computer (PC), or an Internet of Things device, etc. This is not limited to the present invention.

[0099] In addition, the application scenarios of this application are not limited to the examples of scenarios described above, and this application may be applied to multiple scenarios, such as a scenario for Huawei Cloud storage (or transmission) of images (or videos), a video surveillance scenario, or a live broadcast scenario. This is not limited to this application.

[0100] The following describes the specific operating processes of the system shown in Figure 1, with reference to Figure 1.

[0101] The transmitting end may include a collection module, an optical-to-electrical conversion module, and an encoding module.

[0102] The acquisition module may collect ambient light to obtain image data of optical signals.

[0103] The optical-electrical conversion module can perform optical-electrical conversion of an optical signal to image data according to an optical-electrical transfer function in order to generate image data of an electrical signal.

[0104] The following explains the background of the optical-to-electric conversion performed by the optical-to-electric conversion module.

[0105] For example, dynamic range can represent the ratio of the maximum value to the minimum value of a variable. In digital images, dynamic range can represent the ratio of the maximum grayscale value to the minimum grayscale value within the displayable range of the image.

[0106] The dynamic range of nature is very large. As shown in Figure 3, various ambient light luminance information is displayed. For example, the luminance of a night scene in the starry sky is approximately 0.001 cd / m². 2 The sun's own brightness is 1,000,000,000 cd / m². 2 The brightness of the moonlight reaches approximately 1 cd / m². 2 Therefore, the ambient light luminance in an indoor lighting scenario is approximately 100 cd / m². 2 Therefore, the ambient light luminance in an outdoor cloudy scenario is approximately 500 cd / m². 2 Furthermore, the ambient light luminance in a sunny outdoor scenario is approximately 2000 cd / m². 2 Therefore, in this case, the natural dynamic range is 1,000,000,000 / 0.001 = 10 13can reach the magnitude of the digit. However, in the scenario of the natural reality, the brightness of the sun and the brightness of the starlight are not acquired simultaneously. For the natural scenario in the real world, the dynamic range is 10 -3 ~10 6 within the range of.

[0107] However, in most current color digital images, each of the RGB (Red, Green, Blue) channels is stored separately by using 8 bits or 10 bits, that is, the representation range of each channel is a gray scale from 0 to 255, and here 0 to 255 is the dynamic range of the image.

[0108] In this case, the dynamic range in the same scenario in the real world is within the range of 10 -3 ~10 6 which can be referred to as high dynamic range (HDR). In contrast, the dynamic range on a general picture can be referred to as low dynamic range (LDR). As shown in FIG. 3, the imaging process of an electronic device (for example, a digital camera) can be understood as a non-linear mapping from the high dynamic range of the real world (for example, 100 cd / m 2 ~2000 cd / m 2 ) to the low dynamic range of the image (from 1 cd / m 2 ~500 cd / m 2 ).

[0109] It should be noted that the mapping range from the peripheral light brightness to the display brightness of the display device is not limited to the example of the mapping range of the shadow area in FIG. 3. As the display ability of the electronic device is improved, the range of the brightness that can be displayed by the display device can be increased. For example, the maximum display brightness may be greater than 500.

[0110] For example, the display device in Figure 3 and in this specification may be an electronic device to which the receiving end in this embodiment of the present application belongs.

[0111] Based on the above background, referring to Figures 1 and 3, the dynamic range of the image data of the optical signal collected by the acquisition module is 10 -3 ~10 6 It is within a certain range, and the maximum display brightness of the display module at the receiving end is, for example, only 500, and it is found that it cannot reach the brightness information of ambient light in the real world. In order to view an image using a display device, an optical-electrical conversion module needs to convert the optical signal into an electrical signal by using an optical-electrical transfer function to obtain an 8-bit or 10-bit image (where the size of each pixel in the image is 8 bits or 10 bits).

[0112] For example, a standard dynamic range (SDR) image (an image with a dynamic range generally between 1 nit and 100 nits, where nit is a unit of light) corresponds to a high dynamic range (HDR) image (an image with a dynamic range between 0.001 and 10000 nits). 8-bit images in conventional formats such as JPEG can be considered SDR images.

[0113] It should be noted that the image format for an 8-bit or 10-bit image of an electrical signal may be any one of the following: RAW (unprocessed) image, RGB image, YUV (where "Y" represents luminance, Luma, and "U" and "V" represent chromaticity, Chroma), or Lab image. This is not limited to the present invention.

[0114] A Lab image consists of three elements. One element is luminance (L), and a and b are two color channels. a includes colors from dark green (low luminance value) to gray (medium luminance value) and then to bright pink (high luminance value); and b includes colors from bright blue (low luminance value) to gray (medium luminance value) and then to yellow (high luminance value).

[0115] In addition, in this application, the size of each pixel in an image displayed by a display device is not limited to 8 bits or 10 bits. As the display capabilities of display devices improve, image pixels may alternatively be displayed with a size greater than 10 bits.

[0116] The following describes the optical-electrical transfer function used by the optical-electrical transfer module shown in Figure 1.

[0117] For example, early display devices could be CRT displays, and the optical-electrical transfer function of a CRT display is the gamma function. The optical-electrical transfer function shown in formula (1) is defined in the ITU-R Recommendation BT.1886 standard:

number

[0118] Based on equation (1), the image obtained after optical-electrical conversion is performed on the collected optical signal, and then quantized into an 8-bit image, can be called an SDR image. The SDR image and optical-electrical transfer function in equation (1) are used for conventional display devices (illuminance is approximately 100 cd / m²). 2 It runs well under those conditions.

[0119] However, with upgrades to display devices, the illuminance range of display devices continues to increase. Typical illuminance information for HDR displays is 600 cd / m². 2 And the illuminance information for advanced HDR displays is 2000 cd / m². 2This can reach a level far greater than the illuminance information of an SDR display device. The optical-electrical transfer function shown in equation (1) of the ITU-R Recommendation BT.1886 standard cannot adequately represent the display performance of an HDR display device. Therefore, an improved electrical-optical transfer function is needed to accommodate upgrades to display devices. The concept of the optical-electrical transfer function originates from the mapping function in the tone mapping (TM) algorithm. The optical-electrical transfer function is obtained by appropriately adjusting the mapping function. Currently, there are three common optical-electrical transfer functions: PQ (perception quantization), HLG (hybrid log-gamma), and SLF optical-electrical transfer functions. These three optical-electrical transfer functions are conversion functions specified in the AVS standard.

[0120] The following sections describe the curves of the three optical-electrical transfer functions mentioned above.

[0121] The PQ optical-electrical transfer function here differs from the conventional “gamma” optical-electrical transfer function shown in equation (1), and the perceptual quantization transfer function can be proposed based on a luminance perception model of the human eye. The PQ optical-electrical transfer function represents the transformation relationship between the linear signal values ​​of the image pixels and the nonlinear signal values ​​of the PQ region, as shown in equation (2), and the PQ optical-electrical transfer function can be expressed as equation (3):

number

number

number

number

number

number

number

[0122] The HLG optical-electrical transfer function is obtained by improving the conventional gamma curve. The HLG optical-electrical transfer function uses the conventional gamma curve for segments with small horizontal coordinates, and a logarithmic curve is added for segments with large horizontal coordinates. The HLG optical-electrical transfer function describes the transformation relationship between the linear signal values ​​of image pixels and the nonlinear signal values ​​in the HLG region, and the HLG optical-electrical transfer function can be expressed as equation (4):

number

[0123] The SLF optical-electrical transfer function is the optimal curve obtained based on the luminance distribution in an HDR scene, when the optical properties of the human eye are satisfied.

[0124] The SLF optical-electrical transfer curve illustrates the conversion relationship between the linear signal value of an image pixel and the nonlinear signal value in the SLF region. This conversion relationship is expressed by equation (5):

number

number

[0125] As shown in Figure 1, the encoding module at the transmitting end can receive the electrical signal image generated by the optical-to-electrical conversion module and encode the electrical signal image to generate a bitstream.

[0126] It should be noted that the bitstream generated by the transmitting end in this application may carry backlight metadata. When the transmitting end in this embodiment of the application generates an image or video, it may generate backlight metadata for the image or video.

[0127] For example, backlight metadata can be attribute information for a video or image.

[0128] When the backlight of a display device changes, the backlight metadata indicates metadata related to the display adaptation of the image.

[0129] As shown in Figure 1, the transmitting end can send the generated bitstream to the receiving end.

[0130] For example, the image processed by the system shown in Figure 1 in this embodiment of the present application may be a single-frame image or a sequence of frames (i.e., multiple frames of images). This is not limited to the present application.

[0131] Referring to Figure 1, the receiving end may include a decoding module, an electro-optical conversion module, a display compatibility module, and a display module.

[0132] The decoding module can decode the bitstream to obtain, for example, an 8-bit electrical signal image and image attribute information (which may include backlight metadata).

[0133] The operations performed by the electro-optical conversion module are the reverse processes of those performed by the optical-electrical conversion module. That is, the electro-optical conversion module can convert a received electrical signal image into an optical signal image corresponding to the light intensity of the ambient light surrounding the image.

[0134] For example, a mapping method in dynamic range can be applied to the matching of an image (HDR signal) generated by the transmitting end and an HDR signal displayed by a display device to which the receiving end belongs.

[0135] For example, when the transmitting end generates an image, the collected ambient light is a 4000 nit optical signal, but the maximum display brightness (i.e., HDR display capability) of each pixel in the display device (e.g., a television or tablet) to which the receiving end belongs is only 500 nits. In this case, tone mapping from the image brightness to the brightness of the display device's display is necessary to display the 4000 nit optical signal on the display device's screen.

[0136] In another example, when the transmitting end generates an image, a 100-nit SDR signal is collected, but the maximum display brightness of the display device (e.g., a television) to which the receiving end belongs is 2000 nits. In this case, tone mapping from the image brightness to the brightness of the display device is also necessary in order to map the 100-nit signal to the 2000-nit display device for display.

[0137] In the prior art, the bitstream generated by the transmitting end does not include the backlight metadata in this embodiment of the application. Current tone mapping curves may include sigmoid curve-based adjustment processes proposed by Dolby and Bézier curves. The techniques of the two tone mapping curves correspond to the ST2094 standard. When the display pixels of an image are mapped to the display brightness of the display pixels on the display, the maximum and minimum brightness values ​​of the image are usually aligned with the maximum display brightness value (i.e., display capability, e.g., 500 nits) and minimum display brightness value of the display. Brightness values ​​between the maximum and minimum brightness values ​​of the image may be mapped to a range between the maximum and minimum display brightness values ​​of the display based on a global or local tone mapping curve (e.g., sigmoid curve-based adjustment processes and Bézier curves).

[0138] However, the aforementioned tone mapping solutions in the prior art do not consider scenarios where the display device's backlight is adjustable.

[0139] For example, a mobile phone display may include an array of backlight sources, which may be used to light up the display screen. When the backlight intensity level is at its maximum (e.g., 100%), the brightness of the display is the maximum display brightness of the display (e.g., 500 nits), and the backlight intensity level may be described as a percentage of the maximum display brightness. In this case, the product of the backlight intensity level and the maximum display brightness may be the current target backlight brightness of the display.

[0140] It should be noted that the maximum display brightness of a display device is not limited in this application and may be continuously improved based on product upgrades of the display device. In addition, the layout of the backlight source of a display device is not limited in this application, and the backlight sources of different types of electronic devices (e.g., mobile phones and televisions) may be arranged in different ways.

[0141] Figure 4 is an example diagram illustrating a scenario in which the backlight brightness of a mobile phone is adjusted.

[0142] Example 1

[0143] As shown in Figures 4(1) and 4(2), the mobile phone may include an ambient light sensor 101. The ambient light sensor 101 can detect ambient brightness values ​​and identify changes in ambient brightness. The mobile phone may set the current target backlight brightness of the mobile phone screen based on the ambient light brightness values ​​collected by the ambient light sensor 101, and adjust the current target backlight brightness based on changes in ambient brightness collected by the ambient light sensor 101.

[0144] For example, the system may adjust the backlight brightness of the mobile phone screen based on changes in ambient light. For instance, the backlight brightness set on the mobile phone during the day may be higher than the backlight brightness set on the mobile phone at night.

[0145] Example 2:

[0146] As shown in Figure 4(1), the mobile phone displays interface 100, and the user can perform a downward slide operation on interface 100, starting from the screen area closer to the right side, following the direction of the arrow from the top of the screen.

[0147] Interface 100 is not limited in this application and may be an application interface, an application icon interface, or similar.

[0148] In response to a user's slide operation, the mobile phone can switch its display interface from interface 100 shown in Figure 4(1) to control center interface 102 shown in Figure 4(2).

[0149] The control center interface 102 includes one or more controls, and the implementation of the controls may include, but is not limited to, icons, buttons, windows, layers, and the like.

[0150] As shown in Figure 4(2), the control center interface 102 may include the "Huawei Video" playback window, a wireless network icon, a Bluetooth icon, and window 103.

[0151] Window 103 may include a mobile data icon, a ring / mute icon, and a screen auto-rotate icon.

[0152] In addition, window 103 may further include a brightness progress bar control used to control the backlight brightness of the mobile phone screen.

[0153] It should be noted that window 103 may include more or fewer controls, and this is not limited to the present invention.

[0154] Brightness progress bar control may include icon 104, icon 105, progress bar 106, and control 107.

[0155] Icon 104 indicates minimum backlight brightness, i.e., a backlight intensity level of 0%.

[0156] Icon 105 indicates maximum backlight brightness, i.e., a backlight intensity level of 100%. In this case, the display brightness of the mobile phone screen is the maximum display brightness of the mobile phone screen (e.g., 500 nits as mentioned above).

[0157] The user's finger moves the control 107 on the progress bar 106, and the mobile phone can adjust the current backlight intensity level of the mobile phone screen in response to the user's operation, that is, adjust the backlight intensity level in the range of 0% to 100%, thereby changing the current target backlight brightness of the mobile phone.

[0158] For example, control 107 moves on progress bar 106 towards icon 105, thereby increasing the backlight intensity level of the mobile phone screen and increasing the current target backlight brightness. Control 107 also moves on progress bar 106 towards icon 104, thereby decreasing the backlight intensity level of the mobile phone screen and decreasing the current target backlight brightness.

[0159] In this embodiment, the user can set the current target backlight brightness of the mobile phone screen based on usage requirements and actual mobile phone usage.

[0160] It should be noted that the interface provided by the mobile phone for the user to manually adjust the current target backlight brightness of the screen is not limited to (2) in Figure 4, and that the backlight brightness may be adjusted by using a different interface instead.

[0161] In this scenario, where screen backlight brightness is adjustable, the final display brightness of the screen may be the result of the combined effect of the current target backlight brightness value and the screen display value. During backlight brightness adjustment, the screen brightness range changes as appropriate.

[0162] For example, an OLED screen is self-luminous. During backlight control, the brightness of the excitation light source for each pixel in the OLED screen can be directly controlled, and the backlight intensity can result in an overall change in the screen display value.

[0163] For example, in a scenario where the backlight of a mobile phone is adjustable, if the maximum display brightness (i.e., display capability) of the mobile phone screen is 500 nits and the backlight intensity level is 10%, then the current target backlight brightness is 50 nits. However, since the maximum display brightness of the mobile phone screen can reach 500 nits, when the current target backlight brightness is 50 nits, the actual brightness of the screen (current backlight brightness, i.e., actual backlight brightness) may be higher than 50 nits, for example, 60 nits or 100 nits. There may be strategies to determine the actual brightness of the screen.

[0164] Therefore, based on the above explanation that the backlight brightness of the display of an electronic device is adjustable, it can be seen that neither the global tone mapping solution nor the local tone mapping solution in the prior art takes into account that, at a given backlight intensity, the final display brightness of the screen may increase from the current target backlight brightness (e.g., 50 nits as described above) to the brightness that reaches the actual highest level of backlight for a single pixel or several pixels of the screen (i.e., the maximum display brightness as described above, e.g., 100 nits). In this case, the mechanism in the prior art solution that aligns only the maximum and minimum brightness values ​​of the image with the maximum and minimum brightness values ​​of the screen is not suitable for display adaptation when the backlight is weakened (i.e., the backlight is weakened from the strongest backlight, and the backlight intensity level begins to decrease from 100%). As a result, the contrast of the image or picture displayed by the electronic device is low, and the colors are dark.

[0165] Therefore, the present invention provides the system described above. Referring to Figure 1, the bitstream transmitted by the transmitting end in the system may carry backlight metadata, and the display adaptation module at the receiving end may perform dynamic range mapping on the optical signal image based on the backlight metadata to acquire image display information and transmit the image display information to the display module.

[0166] The CPU at the receiving end controls the circuitry of the display module based on image display information and generates light of the corresponding brightness using the pixels in the display module, thereby enabling the implementation of a match between the image (high HDR signal) generated by the transmitting end and the low HDR signal displayed by the display device to which the receiving end belongs.

[0167] In the system of this embodiment, in a scenario where the backlight is adjustable, the image can be appropriately displayed on the display device.

[0168] Embodiment 1:

[0169] In the bitstream shown in Figure 1, the backlight metadata carried in Embodiment 1 may include a local or global initial gain adjustment curve and a local or global anchor.

[0170] "Global" may refer to a single frame image or a sequence of frames.

[0171] "Local" can refer to an image frame, an image scene, or an image region within a similar image.

[0172] The backlight metadata may include at least one of the following data: local initial gain adjustment curve and local anchor; global initial gain adjustment curve and global anchor; global initial gain adjustment curve and local anchor; or local initial gain adjustment curve and global anchor.

[0173] In this specification, the anchor is a luminance value.

[0174] The local anchor of a local image can have a brightness value that is smaller than the maximum pixel brightness value of the local image (e.g., an image region) and larger than the minimum pixel brightness value of the local image.

[0175] The global anchor, i.e., the backlight control anchor, is a luminance value that is less than the maximum global (e.g., 1 frame image) pixel luminance and greater than the minimum global pixel luminance.

[0176] When an initial gain adjustment curve is generated for a local or global image, a tone mapping curve may be generated in a prior art manner and used as the initial gain adjustment curve for the image. This is not limited to the present invention.

[0177] For example, tone mapping curves can take various forms, including, but are not limited to, sigmoid, cubic spline, gamma, and segmented cubic spline functions, each of which can be used as an initial gain adjustment curve.

[0178] Figure 5a is an example of a system architecture diagram obtained after improving Figure 1. The similarities between Figure 5a and Figure 1 will not be explained again here, and only the improved parts will be described in detail.

[0179] It should be understood that the system shown in Figure 5a is merely an example. The system in this application, and the transmitting and receiving ends in the system, may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. The components shown in Figure 5a may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processors and / or application-specific integrated circuits.

[0180] As shown in Figure 5a, the transmitting end may include a backlight metadata generation module.

[0181] The backlight metadata generation module may generate backlight metadata based on the electrical signal image or image sequence generated by the optical-electrical conversion module. This backlight metadata may include local or global initial gain adjustment curves and local or global anchors (see the above explanation for specific definitions, which are not described again here).

[0182] The encoding module can encode electrical signal images and backlight metadata, generate a bitstream, and transmit the bitstream to the receiving end.

[0183] The following describes the process by which the backlight metadata generation module generates local or global initial gain adjustment curves and local or global anchors.

[0184] Referring to Figures 5a and 5b, an example of the process for generating the initial gain adjustment curve 0 and anchor for region A of Image 1 is shown. In Figure 5b, an example where Image 1 is the target of the processing is used for explanation.

[0185] In Figure 5b, the backlight metadata generation module divides pixels of the display target image 1 that have similar luminance and spatial position to be divided into the same region, performs region division on image 1, thereby obtaining multiple regions C; and then identifies the highlight region C within the multiple regions C.

[0186] When region segmentation is performed on image 1, the segmentation method is not limited to the example in Figure 5b. It should be noted that pixels in an image that have similar brightness and spatial positions can be alternatively divided into a single region using a different method, thereby implementing image region segmentation.

[0187] For example, the process in Figure 5b may include the following steps:

[0188] Optionally, the first region segmentation is performed on image 1.

[0189] For example, Image 1 can be divided into a single region (i.e., no region division is performed). In Figure 5b, Image 1 is region A.

[0190] Alternatively, for example, Image 1 can be divided into multiple rectangular regions, and multiple regions A can be obtained.

[0191] S400: The backlight metadata generation module may generate an initial gain adjustment curve of 0 for region A.

[0192] For example, Image 1 is a 64x64 image.

[0193] The initial gain adjustment curve 0 can be generated by any method used to generate tone mapping curves in the prior art, but is not limited to this application.

[0194] When the initial gain adjustment curve in the backlight metadata includes a local initial gain adjustment curve, the initial gain adjustment curve for the image region can be generated for the image region based on the method described above.

[0195] When the initial gain adjustment curve in the backlight metadata includes a global initial gain adjustment curve, and "global" means an image of one frame, the initial gain adjustment curve can be generated for an image of one frame. When "global" means a frame sequence, the initial gain adjustment curve can be generated for an image of one frame in the frame sequence, and the initial gain adjustment curve is used as the initial gain adjustment curve for the frame sequence.

[0196] S401: The backlight metadata generation module generates region B by performing an even image division on region A.

[0197] For example, region A may be divided into four equally divided regions B, each region B having a size of 32 × 32.

[0198] Please note that the image may or may not be divided equally. This is not limited to the present application.

[0199] Here, region A is divided into four regions B. This is equivalent to performing initial clustering on the pixels in region A. For example, the four regions B may be named type 0, type 1, type 2, and type 3, respectively.

[0200] S402: The backlight metadata generation module performs iterative clustering on pixels in region A based on pixel brightness and spatial position.

[0201] At this stage, iteratively updated clustering may be performed on the pixels in region B of region A, clustering pixels in region A that are close in brightness and spatial position into one type, thereby forming four new cluster types, which are the four regions C in Figure 5b. For example, the four regions C may be referred to as type 0', type 1', type 2', and type 3'.

[0202] For example, in a given region A, for each region B, the first step may calculate the cluster center location (x_centra, y_centra) and the average color value (e.g., the average of 3-channel RGB: R_avg, G_avg, and B_avg), or the average of Lab (l_avg, a_avg, and b_avg) for each region B. Here, the cluster center location may be the geometric center of a pixel in region B.

[0203] Next, in the second stage, for each pixel in a region A (coordinates (x,y), and color value RGB or lab), the geometric distance between the pixel and the cluster center location in each region B within region A, and the color difference between the color of the pixel and the color of the pixel at the cluster center location are calculated and used as the distortion value cost for the pixel in region A: cost = (√((x - x_centra)) 2 +(y‐y_centra) 2 )) / K+(√((l‐l_avg) 2 +(a‐a_avg) 2 +(b‐b_avg) 2 )) / M Here, √ represents the square root operation, K and M are two constants, and K and M are used for normalization.

[0204] For example, in Figure 5b, one region A may contain four regions B, and four distortion values ​​may be obtained for one pixel in region A.

[0205] Next, in the third stage, for each pixel in region A, the region B corresponding to the smallest distortion value is selected as the updated cluster type (e.g., type 0). In this way, iterative clustering can be implemented for all pixels in region A at once.

[0206] Thus, for all pixels in region A, the updated cluster type can be obtained through computation. This is equivalent to performing region partitioning again on region A. The four regions B' obtained through the current partitioning differ from the four regions in the regular rectangular shape shown in Figure 5b, and the shape of the four regions obtained through the current partitioning may be irregular.

[0207] Next, based on the procedures of steps 1 through 3 described above, the cluster center locations, average color values, and similar values ​​are recalculated for pixels in the regions corresponding to the four updated cluster types, and the next iterative clustering is performed until the cluster type of pixels in region A no longer changes, and the four regions C shown in Figure 5b may be obtained (as the four final cluster types).

[0208] Thus, each region A in Image 1 can be divided into four regions C.

[0209] It should be noted that the number of final regions C generated when region A is divided is not limited to 4. For example, when the image is divided equally in S401, alternative methods of equal division using any number such as 6 or 9 may be performed, and as a result, the number of regions C obtained by dividing region A may be 6, 9, or similar.

[0210] It should be noted that the method by which pixels with similar brightness and spatial position are divided into the same region in order to perform region division on the display target image 1 is not limited to the example of the solution described above. Region division on image 1 can also be implemented by using other methods in the art.

[0211] S403: The backlight metadata generation module identifies highlight areas from divided regions of an image using a pre-configured algorithm.

[0212] In possible implementations, for example, a backlight metadata generation module may perform a filtering process on image 1 (including, for example, the four regions C shown in Figure 5b) by using a bilateral filtering method or a guided filtering method, and then use pixels with large differences in pixel values ​​in the image as highlighted pixels based on the difference between the pixel values ​​of the same pixels in image 1 before and after filtering. Next, based on the four regions C obtained by dividing image 1, regions C containing highlighted pixels that are greater than or equal to a preset threshold are used as highlighted regions C, and regions C containing highlighted pixels that are less than a preset threshold are used as non-highlighted regions C. In this way, the regions contained in image 1 can be divided into two types of regions: one type is highlighted regions, and the other type is non-highlighted regions.

[0213] For example, as shown in Figure 5b, the four regions C of region A include two highlighted regions C and two non-highlighted regions C.

[0214] For example, the attributes of the corresponding photographed object in Image 1 can influence whether an image region is a highlight region. For instance, a layer of clouds or glass has high reflectivity, and an image region of a cloud or glass is readily used as a highlight region. In another example, an image region of an object with low reflectivity, such as rough cotton or linen clothing, is readily used as a non-highlight region.

[0215] It should be noted that the method for identifying which regions of an image belong to the highlight region and which belong to the non-highlight region is not limited to the bilateral filtering or guided filtering method described above, and may be an alternative method. For example, the average brightness value of pixels may be calculated separately for different regions C in region A, and regions C whose average brightness value is greater than a preset threshold may be used as highlight regions C, thereby classifying the highlight and non-highlight regions of region A. This is not limited to the present invention.

[0216] Furthermore, when Image 1 is divided into multiple regions A, it should be noted that for different regions A, the highlighted and unhighlighted regions in each region A may be implemented using the same or different implementations. This is not limited to the present invention.

[0217] S405: The backlight metadata generation module determines the brightness value to be used as the anchor for region A based on the pixel brightness of the highlight region C in region A.

[0218] For example, for a single region A, the minimum pixel brightness a1 and the average pixel brightness a2 can be obtained for all pixels in the highlight region C within region A. For a single region A, the anchor for region A is generated based on the minimum pixel brightness a1 and the average pixel brightness a2.

[0219] For example, any luminance value greater than a1 and less than a2 can be used as an anchor for region A.

[0220] For example, the average of a1 and a2 can be used as an anchor for region A.

[0221] For example, the weighted results of a1 and a2 can be used as anchors for region A.

[0222] Each region A has one anchor.

[0223] Optionally, for one region A, the minimum pixel brightness a1 and the average pixel brightness a2 can be obtained for all pixels in region C within region A (i.e., all pixels in region A). For one region A, the anchor for region A is generated based on the minimum pixel brightness a1 and the average pixel brightness a2.

[0224] It should be noted that the method for generating anchors in this application is not limited to the method in the embodiments described above, and anchors may be generated by alternative methods. This is not limited to this application.

[0225] Therefore, based on the method in the embodiments described above, a global anchor for a single frame of an image may be generated for the frame of the image, and local anchors may also be generated for regions in the image. In addition, when the global anchor in the backlight metadata is an anchor for a frame sequence, an anchor may be generated for a single frame of an image using the method described above, and then the anchor is used as an anchor for a frame sequence containing the frame images.

[0226] Figure 5b illustrates the process of generating the initial gain adjustment curve and anchor in the implementation.

[0227] In an alternative implementation, the local or global initial gain adjustment curve and the local or global anchor may be generated based on the procedure in (1) of Figure 5c.

[0228] As shown in Figure 5c (1), the process may include the following steps.

[0229] For example, (1) in Figure 5c may be performed by the backlight metadata generation module in Figure 5a. Region A in Figure 5c(1) may be an image of one frame, or a local image region in an image. This is not limited to the present invention.

[0230] S501: Generate an initial gain adjustment curve 0 for region A.

[0231] This process is similar to S400 in Figure 5b, and the details will not be explained here again.

[0232] S502: Generate a cumulative distribution histogram of pixel brightness in region A based on the brightness values ​​of the pixels in region A.

[0233] For example, (2) in Figure 5c shows a histogram of the cumulative distribution of pixel brightness in region A.

[0234] In a histogram, the horizontal axis x represents the brightness value of pixels in region A, and the vertical axis y represents the number of pixels in region A.

[0235] For example, from (2) in Figure 5c, it can be seen that region A contains 100 pixels, the number of pixels with brightness x1 in region A is 20, the number of pixels with brightness x2 is 30, and the number of pixels with brightness x3 is 50.

[0236] S503: Based on the histogram, a luminance value in region A that is higher than the luminance of pixels within region A at a predetermined ratio is used as the anchor for region A.

[0237] For example, a pre-set ratio may be 95%, 99%, 99.5%, or similar. A pre-set ratio may be greater than 50%. Specific values ​​can be flexibly configured based on requirements. This is not limited to the present invention.

[0238] For example, the pre-set ratio is 95%. Based on the histogram shown in Figure 5c (2), it can be easily determined that x3 is a luminance value in region A that is higher than the luminance of 95% of the pixels within region A, and that x3 can be used as an anchor for region A.

[0239] In the implementation of Figure 5c, when local or global anchors are generated for an image, region segmentation does not need to be performed on the image, and the required anchors can be determined solely based on the cumulative distribution histogram of the image's pixel luminances. When tone mapping is performed on the image, considering that the luminance of pixels with a luminance higher than the anchor needs to be increased and the luminance of pixels with a luminance lower than the anchor needs to be decreased, it can be determined that the luminance of pixels by a predetermined ratio needs to be decreased, and the luminance of pixels by (1 - predetermined ratio) needs to be increased, and the magnitude of the brightening and darkening can be easily determined.

[0240] It should be noted that generating the initial gain adjustment curve and generating anchors for an image, local image, or frame sequence are two independent processes. The order in which these two processes are executed is not limited in this application.

[0241] Please note that S503 is executed after S502. However, the order in which S501, S502, and S503 are executed is not limited in this application.

[0242] In the implementation described above, we use an example in Figure 5a where the transmitting end generates a local or global initial gain adjustment curve and a local or global anchor, and as a result, information about the curve and anchor can be transmitted to the receiving end as attributes of the bitstream.

[0243] Note that the receiving end in this embodiment of the present application may also include a backlight metadata generation module for performing the operations described above, generating local or global initial gain adjustment curves and local or global anchors. The method is similar and details are again not described here.

[0244] Next, we return to Figure 5a. Below, we will explain the execution process at the receiving end in Figure 5a with reference to Figure 5d.

[0245] Through the comparison between FIG. 1 and FIG. 5a, it can be seen that the display adaptation module in FIG. 1 may include the tone mapping curve generation module and the tone mapping module shown in FIG. 5a.

[0246] Referring to FIG. 5d, the operation processes of the tone mapping curve generation module and the tone mapping module in FIG. 5a will be described below.

[0247] As shown in FIG. 5d, the process may include the following steps.

[0248] S201: The receiving end acquires the display target image 1 in the bitstream.

[0249] For example, as shown in FIG. 5a, the decoding module may obtain a picture, such as Picture 1 in FIG. 5b, by decoding the bitstream.

[0250] S202: The receiving end acquires the backlight metadata in the bitstream, where the backlight metadata includes the initial gain adjustment curve 0 of region A in image 1 and the first luminance (also referred to as an anchor) of region A.

[0251] For example, as shown in FIG. 5a, the decoding module may decode the bitstream to obtain the backlight metadata, where the backlight metadata may include the local or global initial gain adjustment curve of the bitstream and the local or global anchor.

[0252] For example, Image 1 in FIG. 5b is the display target image in the bitstream. The decoding module may obtain the initial gain adjustment curve of region A and the anchor in Image 1, and may transmit the initial gain adjustment curve and the anchor of region A to the tone mapping curve generation module.

[0253] S203: The receiving end acquires the maximum display luminance of the display and the current backlight intensity level.

[0254] For example, as shown in FIG. 5a, the tone mapping curve generation module may obtain the maximum display brightness of the display and the current backlight intensity information of the display from the display module.

[0255] For example, the current backlight intensity information may be the current backlight intensity level or the current target backlight brightness.

[0256] For example, the receiving end may obtain the maximum display brightness (e.g., 500 nits) of the display from the display device to which the receiving end belongs.

[0257] For example, in the present embodiment of the present application, the test sample (test image) may further be displayed on the display of the display device. An area that occupies N% (e.g., 5% or 10%) of the area of the test sample and the area in the test sample is white, and the other area is black. The brightness of the white area having an area of N% is measured using an instrument, and the brightness obtained through the measurement is the maximum display brightness of the display.

[0258] It should be noted that the method for obtaining the maximum display brightness of the display device is not limited in the present application.

[0259] In addition, based on the description of the embodiment in FIG. 4, the current backlight intensity information obtained by the tone mapping curve generation module and belonging to the display may be the current target backlight brightness belonging to the display obtained after automatic adjustment by the system, or the current target backlight brightness belonging to the display obtained after manual adjustment by the user. Details are not described again here.

[0260] S204: The receiving end processes curve 0 based on the maximum display brightness and generates curve 1.

[0261] Note that when curve 0 in region A is processed to generate curve 1, the luminance value based on the processing is not limited to the maximum display luminance. The luminance value may be greater than the current target backlight luminance and less than or equal to the maximum display luminance of the display (e.g., a weighted result of the current target backlight luminance and the maximum display luminance).

[0262] For example, we will use an example in which curve 1 is generated by processing curve 0 based on the maximum display brightness. When the brightness value based on the processing is not the maximum display brightness, the method is similar, and the details are again not explained here.

[0263] Curve 0 can be any tone mapping curve in the prior art. In this case, curve 1 and the subsequent curve 2 are also tone mapping curves. The independent variable x of the function f(x) of any tone mapping curve represents the luminance value of an image pixel in the SDR or HDR source, and f(x) represents the luminance value of an image pixel displayed on the display.

[0264] For example, as shown in Figure 5a, the tone mapping curve generation module may process, for example, the initial gain adjustment curve of region A (which may also be called curve 0) based on the maximum display brightness maxT, to generate an updated gain adjustment curve (which may also be called curve 1).

[0265] For example, scaling may be performed on curve 0 of region A based on the maximum display brightness, thereby obtaining curve 1. It should be noted that the processing methods are not limited to the scaling methods described in Methods 1 and 2, and may further include Methods 3, 4, 5, and other methods not listed. This is not limited to the present invention.

[0266] Method 1

[0267] For example, if the maximum x value of curve 0 is maxsource and the corresponding y value is maxY, then the overall scaling factor of curve 0 is D = maxT / maxY, where maxT is the maximum display brightness of the display device to which the receiving end belongs. Curve 0 can be scaled as a whole based on the scaling factor D, and curve 1 = D * curve 0.

[0268] Curve 0 is the initial gain adjustment curve (tone mapping curve) for region A. In this case, the maximum x value of curve 0 represents the maximum brightness value of the image pixels in region A at this point, and the corresponding y value (i.e., maxY) is that of the image pixel with the maximum brightness value in region A, and represents the brightness value displayed on the display. In this case, the maximum x value of curve 0 is the maximum brightness value of the image pixels in region A.

[0269] Method 2:

[0270] Backlight metadata may include a pre-set brightness value T3.

[0271] For example, the pre-set luminance value T3 is an arbitrary target luminance target_maxdisplay originally generated by curve 0, where the target luminance is also the luminance value of an image pixel that has a specific luminance and is displayed on the display, and may be, for example, the maximum display luminance value.

[0272] The overall scaling factor of curve 0 is D = maxT / T3. Curve 0 is scaled as a whole based on the scaling factor D, where curve 1 = D * curve 0.

[0273] When the display has a specific backlight, it is considered that the actual luminance of the display may be higher than the backlight luminance. For example, when the maximum display luminance of the display is 500 nits and the backlight intensity level is 10%, the backlight luminance is 50 nits. However, the electronic device may have a strategy for adjusting the final luminance of the display so that the final luminance may be higher than 50 nits. Therefore, in the present embodiment, the tone mapping curve (i.e., curve 1) that matches the luminance value can be generated based on a luminance value that is greater than the backlight luminance and less than or equal to the maximum display luminance.

[0274] Method 3:

[0275] Curve A is obtained based on the image parameters and display luminance in region A.

[0276] For example, the image parameters include, but are not limited to, the maximum luminance value of the pixel, the minimum luminance value of the pixel, or the average luminance value of the pixel.

[0277] For example, the display luminance may include the maximum display luminance of the display or the current target backlight luminance.

[0278] The image parameters and display luminance are used as the curve parameters of the conventional tone mapping curve, thereby obtaining curve A.

[0279] The weighting value w is obtained based on the maximum display luminance and the target luminance target_maxdisplay.

[0280] For example, the difference between the maximum display luminance and the target luminance target_maxdisplay may be used as the weighting value w, but the method for obtaining the weighting value w is not limited thereto.

[0281] The weighted sum is performed on curve A and curve 0 in region A based on the weighting value w, and curve 1 is obtained.

[0282] Method 4:

[0283] The curve parameter adjustment value B1 is obtained based on the maximum display brightness and the curve adjustment parameter B.

[0284] For example, backlight metadata may include each curve parameter indicating curve 0, and curve adjustment parameter B in this case.

[0285] For example, the value of the curve adjustment parameter B is 0.2.

[0286] For example, the maximum display brightness of the display is 200, and the maximum display brightness corresponding to curve 0, which is of the display, is 500.

[0287] In this case, |(200-500)| / 100=3, and the curve parameter adjustment value B1 obtained by multiplying the curve adjustment parameter B (here 0.2) by the above result "3" is 0.6.

[0288] Curve 1 is obtained by superimposing the curve parameter adjustment value B1 on curve 0.

[0289] The curve parameter adjustment value B1 is superimposed on each curve parameter of curve 0 (e.g., by addition or subtraction), thereby obtaining each curve parameter of curve 1, and then curve 1 is obtained.

[0290] Specifically, by adding or subtracting 0.6 from each curve parameter of curve 0, a new group of curve parameters can be obtained, thereby yielding curve 1.

[0291] Method 5:

[0292] The curve parameter adjustment value B1 is obtained based on the maximum display brightness and the curve adjustment parameter B.

[0293] For example, in method 4, each curve parameter of curve 0 is adjusted based on the curve parameter adjustment value B1. Here, for example, if the target parameter in curve 0 has the greatest influence on the curve shape, the backlight metadata may include the curve adjustment parameter B of the target parameter of curve 0. Specifically, only the target parameter of curve 0 is adjusted in method 4.

[0294] Curve 0 is scaled using either method 1 or method 2 described above to obtain curve 00.

[0295] Here, during scaling, the scaling process may be performed on the parameters of curve 0 other than the target parameters mentioned above, thereby obtaining curve 00.

[0296] Curve 1 is obtained by superimposing B1 on curve 00.

[0297] For example, a new curve 1 can be obtained by adding B1 to the target parameter of curve 00, or by subtracting B1 from it.

[0298] S205: The receiving end processes curve 0 based on the maximum display brightness and the current backlight intensity level to generate curve 2.

[0299] For example, as shown in Figure 5a, the tone mapping curve generation module may process the initial gain adjustment curve (curve 0) in region A based on, for example, the current backlight intensity information (e.g., current backlight intensity level) and the maximum display brightness maxT, to generate an updated gain adjustment curve (e.g., curve 2, also referred to as curve2).

[0300] If the current backlight intensity information obtained by the tone mapping curve generation module is the current target backlight brightness information, then curve 2 does not need to be generated based on the maximum display brightness of the display module.

[0301] For example, the maximum display brightness maxT and the current backlight intensity level Level (i.e., the current target backlight brightness L) x The product of (=maxT*Level) may be used. For example, if the maximum display brightness is 500 and the current backlight intensity level is 10%, then the current target backlight brightness is 50, and the scaling process is performed on curve 0 in region A to obtain curve 1. Note that the processing method is not limited to scaling and may further include other methods. This is not limited to the present invention.

[0302] Method 1:

[0303] For example, if the maximum x value of curve 0 in region A is maxsource, and the corresponding y value is maxY, then the overall scaling factor of curve 0 is D = L x / maxY. Curve 0 is scaled as a whole based on the scaling factor D, where curve 2 = D * curve 0.

[0304] Method 2:

[0305] Backlight metadata may include a pre-set brightness value T3.

[0306] For example, the pre-set luminance value T3 is the target luminance target_maxdisplay originally generated by curve 0.

[0307] The overall scaling factor for curve 0 is D = L x / T3. Curve 0 is scaled as a whole based on the scaling factor D, where curve 2 = D * curve 0.

[0308] The display has a specific backlight; for example, when the maximum display brightness of the display is 500 nits and the backlight intensity level is 10%, the backlight brightness is 50 nits. In this embodiment, a tone mapping curve (i.e., curve 2) that matches the backlight brightness can be generated.

[0309] For processing methods other than scaling, please refer to methods 3 to 5 described in the example in S205. The principles are similar, and the details will not be explained again here.

[0310] S206: The receiving end processes curves 1 and 2 based on the anchor in region A and generates tone mapping curve 3 for region A.

[0311] Figure 5e is an example of a diagram that generates curve 3 based on curves 1 and 2.

[0312] In the three coordinate diagrams in Figure 5e, each x-axis represents the brightness value of an image pixel in region A, and the y-axis represents the brightness value of an image pixel that is displayed on the screen and belongs to region A.

[0313] For illustrative purposes, an example is used in Figure 5e where the x-value range of each tone mapping curve is 0 to 1, and an example is used where the anchor for region A is 0.7.

[0314] Referring to the curve diagram of curve 2 shown in (1) of Figure 5e, since curve 2 is a tone mapping curve generated based on the current target backlight brightness, most of the tone mapping values ​​in curve 2, i.e., the display brightness values ​​(i.e., y values) obtained by mapping the brightness values ​​of the image pixels, are appropriate, and a small number of tone mapping values ​​are inappropriate. Therefore, based on the anchor, a portion of the tone mapping curve can first be cut off from curve 2, and then curve splicing is performed based on curve 1 to generate the final tone mapping curve of region A (i.e., curve 3, also referred to as curve 3).

[0315] As shown in (1) of Figure 5e, the value of y corresponding to anchor 0.7 is y1, and curve 2 can be cut from the origin (0,0) to P1(0.7,y1) to obtain curve 2-1. In other words, for the final tone mapping curve, i.e., curve 3, curve 2 is used for the curve portion where the pixel brightness is below the anchor (0 to 0.7). For curve 3 shown in (3) of Figure 5e, the curve 2-1 portion in (1) of Figure 5e is used in the pixel brightness range of 0 to 0.7.

[0316] Next, based on the tone mapping value of curve 2 at anchor (0.7), i.e., the y value, in this case y1 in (1) of Figure 5e, curve 1 can be translated or stretched to obtain curve 1', and as a result the tone mapping values ​​(i.e., y values) of curve 1' and curve 2 at anchor are the same.

[0317] For example, (2) in Figure 5e shows the curve diagram of curve 1. When the tone mapping value at the anchor position (x=0.7) in curve 1 is y2, for example, P2(0.7,y2) in curve 1, the x value range is 0 to 1, and since the curve below the anchor was determined from curve 2, only the curve segments where the x value range is 0 to 1 need to be determined from curve 1 in Figure 5e (2), i.e., curve segments P2P3. Next, curve segments P2P3 in curve 1 can be translated in the opposite direction of the y axis toward the x axis, and the direction of translation is parallel to the y axis, as indicated by the translation arrow in Figure 5e (2). Thus, in the curve segment P2P3 obtained after translation (i.e., the curve segment MN shown in (2) of Figure 5e, also called curve 1'), when the value of x is anchor 0.7, the value of y is y1, where P2 is M(0.7, y1) after translation, and P3 is point N after translation. The amount of translation of the curve here can be (y2-y1).

[0318] In Figure 5e, it should be noted that curve 3 can be generated by translating curve 1, or by scaling or stretching curve 1. This is not limited to the present invention.

[0319] For curve 3 shown in (3) of Figure 5e, below the anchor, i.e., within the pixel brightness range of 0 to 0.7, the portion of curve 2-1 in (1) of Figure 5e is used; and above the anchor, i.e., within the pixel brightness range of 0.7 to 1, the MN segment of curve 1' obtained in (2) of Figure 5e is used, thereby obtaining the tone mapping curve, i.e., curve 3 of region A (i.e., the target tone mapping curve in Figure 5a).

[0320] Point M in (3) of Figure 5e is also point P1 in (1) of Figure 5e. Here, the letter M represents a point.

[0321] In addition, in Figure 5e (2), for example, the curve segment above the anchor in curve 1 is translated to obtain the curve segment above the anchor in curve 3. In another embodiment, curve 1 may be translated as a whole, and then the curve segment above the anchor is cut off from the curve 1 obtained after translation and connected to curve 2-1 at the anchor position, thereby obtaining curve 3. This is not limited to the present invention.

[0322] Since curve 3 is formed by connecting two curve segments from two curves at the anchor position (optional), the smoothness of curve 3 is insufficient at anchor position M, for example, as shown in (3) of Figure 5e. As a result, distortion easily occurs when the image is displayed on a screen after tone mapping has been performed on the image based on curve 3.

[0323] Therefore, in this embodiment of the present application, as shown in (3) of Figure 5e, the curve segment EF of curve 3 within the preset range (0.7-T2, 0.7+T1) before and after the anchor can be corrected, and as a result, EF is smoother at M.

[0324] Next, the corrected curve segment EF is smoothly connected at point E to curve segment 0E in curve 3 (i.e., the curve segment from the origin to point E), and the corrected curve segment EF is smoothly connected at point F to curve segment FN in curve 3 (e.g., first differential continuity at points E and F), and the corrected curve 3, i.e., the final tone mapping curve for region A, is obtained. When the corrected curve 3 is used to perform tone mapping on image pixels, high-frequency coefficients are not easily amplified, and the image is not easily distorted.

[0325] S207: The receiving end performs tone mapping on region A based on the tone mapping curve 3 and obtains image display information for region A.

[0326] Please note that the order in which S201, S202, and S203 are performed is not limited in this application.

[0327] The order in which S204 and S205 are performed is not limited in this application.

[0328] For example, as shown in Figure 5a, the tone mapping curve generation module at the receiving end may transmit the generated target tone mapping curve (e.g., curve 3 described above) to the tone mapping module.

[0329] The tone mapping module can adjust the image pixels of region A in image 1 by performing a tone mapping process on region A based on curve 3, thereby obtaining the display brightness of each pixel in region A on the display module.

[0330] The display module can display area A in image 1 based on the display brightness corresponding to each pixel in area A in image 1, and can display image 1 with adjusted pixels on the display module to adapt image 1 to the backlight brightness of the display.

[0331] Similarly, when image 1 contains multiple regions A, the display module can display each region A based on its respective image display information.

[0332] Performing tone mapping on an image can be understood as mapping in the dynamic range, for example, as shown in Figure 3.

[0333] Mapping in the dynamic range can be performed by using static mapping methods and dynamic mapping methods.

[0334] For example, a static mapping method can be described as follows: The entire tone mapping process is performed based on a single data point, referencing the same video content or the same hard disk content. In other words, tone mapping is performed on one or more frames of an image based on the same target tone mapping curve, resulting in a bitstream that carries a smaller amount of information, a simpler process for adaptively displaying the image, and shorter image display latency.

[0335] A dynamic mapping method can be described as follows: Dynamic tone mapping is performed based on different tone mapping curves, referring to the image content of a specific area, scene, or frame in an image. In this case, each frame or scene needs to carry the relevant scene information. Tone mapping processing based on different tone mapping curves can be performed based on the image of a specific area, scene, or frame, and as a result, tone mapping can be performed based on different tone mapping curves for images of different scenes, different image areas, and different frames. For example, for an image in a bright scene, tone mapping may be performed based on the scene of the shot by using a tone mapping curve that focuses on protecting the display effect of the bright areas.

[0336] The static mapping method is equivalent to performing tone mapping on multiple frames of an image based on the same target tone mapping curve. In this case, the target tone mapping curve generated for one frame of an image in a set of multiple frames can be used as the target tone mapping curve for all the frames. When the target tone mapping curve for one frame of an image is generated, the global initial gain adjustment curve and global anchor for that frame of the image can be used for implementation.

[0337] The dynamic mapping method is equivalent to performing tone mapping on images in different frames, different image regions within the same frame, and images in different scenes, based on different target tone mapping curves. In this case, local anchors and local initial tone mapping curves may be used to generate the target tone mapping curve for each local region.

[0338] As illustrated in Figure 5a, backlight metadata may include local or global initial gain adjustment curves and local or global anchors.

[0339] The initial gain adjustment curve for the local region of an image may be called the local initial gain adjustment curve, and the anchor for the local region of an image may be called the local anchor.

[0340] The initial gain adjustment curve for an image in one frame can be called the global initial gain adjustment curve, and the anchor for an image in one frame can be called the global anchor. However, the initial gain adjustment curves and anchors for images in different frames may differ.

[0341] An initial gain adjustment curve for multiple frames of images can also be referred to as a global initial gain adjustment curve. For example, an initial gain adjustment curve for multiple frames of images can be used as the global initial gain adjustment curve for multiple frames of images.

[0342] An anchor shared by images across multiple frames can be called a global anchor. For example, an anchor may be created for one of the images across multiple frames, and this anchor can then be used as the global anchor for the images across all frames.

[0343] For example, when the bitstream includes a global initial gain adjustment curve and a global anchor, the target tone mapping curve for the image in each frame can be generated according to the method described above, based on the initial gain adjustment curve and anchor for the image in each frame. Based on the target tone mapping curve for the image in each frame, tone mapping is performed on the image in each frame, and display fitting between the image and the display is implemented.

[0344] For example, when the bitstream includes a global initial gain adjustment curve and a global anchor, a target tone mapping curve applicable to images of multiple frames may be generated according to the method described above based on the global initial gain adjustment curves and global anchors of the images of multiple frames, and tone mapping may be performed on the images of multiple frames based on the same target tone mapping curve, thereby implementing display adaptation between images and displays.

[0345] For example, when a bitstream includes local initial gain adjustment curves and local anchors, for instance, different regions in image 1 may each have their own local initial gain adjustment curve and local anchor, and a target tone mapping curve (which is also a local tone mapping curve) may be generated for each region in image 1 based on the initial gain adjustment curve and anchor for each region. Next, tone mapping is performed on the different regions in image 1 based on the respective target tone mapping curves, and display adaptation between different regions of the image and the display is implemented. Tone mapping may be performed on different regions of the same image based on different tone mapping curves. In another example, for different scenes, the target tone mapping curve for each scene may further be generated based on the local initial gain adjustment curve and local anchor for each scene, and then tone mapping is performed on the images of the different scenes based on the target tone mapping curves of the corresponding scenes, and display adaptation between the images of the different scenes and the display is implemented.

[0346] In addition, since generating anchors and generating initial gain adjustment curves are two independent processes for an image (or local image), anchors may be generated for the local region, and a global initial gain adjustment curve may be generated for the global image (e.g., a frame or a sequence of frames).

[0347] For example, when the bitstream includes a global initial gain adjustment curve and local anchors, for each local image, the target tone mapping curve for each local image may be generated based on the local anchor and global initial gain adjustment curve for each local image. When the target tone mapping curve for each local image is generated, the global initial gain adjustment curve may be used as the initial gain adjustment curve for the local image. The process for generating the target tone mapping curve for each local image is the same as the process described above in Figures 5d and 5e, and the details are not described again here.

[0348] For example, when the bitstream includes a local initial gain adjustment curve and a global anchor, for each local image, the target tone mapping curve for each local image may be generated based on the local initial gain adjustment curve and global anchor for each local image. When the target tone mapping curve for each local image is generated, the global anchor may be used as the anchor for the local image, i.e., the local image reuses the same anchor. The process for generating the target tone mapping curve for each local image is the same as the process described above in Figures 5d and 5e, and the details are not described again here.

[0349] While initial gain adjustment curves and anchors in a bitstream may not have a one-to-one correspondence, during use, one initial gain adjustment curve and one anchor are used to generate a target gain adjustment curve for the image or region being displayed; in other words, initial gain adjustment curves and anchors are used in pairs.

[0350] For example, in S207 described above, when the receiving end performs tone mapping processing on region A based on tone mapping curve 3 to obtain image display information of region A, and when tone mapping curve 3 is a global tone mapping curve, method 1 is used to implement the tone mapping processing, and when tone mapping curve 3 is a global tone mapping curve, method 2 is used to implement the tone mapping processing.

[0351] Method 1:

[0352] For each pixel of the image to be displayed, the brightness value of the pixel displayed on the screen is obtained based on the global tone mapping curve.

[0353] For example, the gain can be calculated based on equation (7): gain=f[i](max(Rp,Gp,Bp)) / max(Rp,Gp,Bp) Equation (7) Here, Rp, Gp, and Bp are the different color components of the current pixel P in the image to be displayed, and max(Rp, Gp, Bp) represents the solution with the maximum component value of the three color components.

[0354] f(x) represents a function of the global tone mapping curve used for the image to be displayed, and [i] represents that the current pixel P is the i-th pixel in the image to be displayed on which tone mapping is performed.

[0355] Next, the tone mapping value of the current pixel P is calculated based on equations (8), (9), and (10). RpTM[i] = Rp * gain (Equation 8); GpTM[i]=Gp*gain Equation (9); and BpTM[i]=Bp*gain Equation (10) RpTM[i], GpTM[i], and BpTM[i] are the tone mapping values ​​of the three color components of the current pixel P: red, green, and blue (i.e., the luminance values ​​of the current pixel P in the image displayed on the display).

[0356] Method 2:

[0357] For each pixel of the local image in the image to be displayed, the brightness value of the pixel displayed on the display is obtained based on the local tone mapping curve, where the local tone mapping curve is the local target tone mapping curve generated for the local image, for example, curve 3 of region A described above.

[0358] For example, if the image to be displayed 1 is divided into four regions A (i.e., four tone mapping regions), a local target tone mapping curve for each region A can be generated for each region A using the method described above for generating the target tone mapping curve.

[0359] It should be noted that the specific number of local images into which Image 1 is divided (e.g., the number of regions A) is not limited in this application. Tone mapping may be performed on each local image based on the local tone mapping curve of each local image. In addition, the shapes of the local images in Image 1 may be the same or different. This is not limited in this application.

[0360] Regarding information about each local image (also referred to as a tone mapping region) into which Image 1 is divided, such information may include, for example, size information of the local image in Image 1, such as length and width information, for determining the local image. In addition, information describing the size of the local image may be carried in the backlight metadata described above, and as a result, the receiving end performs tone mapping separately on the local images based on the information about the local image and according to the target tone mapping curve of the local image.

[0361] For example, when tone mapping is performed on a pixel P in region A (which may be called initial region A) in image 1, the target region A closest to pixel P in space (there may be one or more target regions A) can be determined from the four regions A mentioned above, the target tone mapping curve corresponding to the target region A is used as f(x) in equation (7) in method 1 above, and the first tone mapping value of the current pixel P is calculated based on equations (7) to (10) in method 1. If there is one first region A, the first tone mapping value is used as the final tone mapping value of the current pixel P. If there are multiple target regions A, the gain[j] of the j-th target region A can be obtained based on equation (7), where j=1, 2, 3, 4, and a weighted sum can be performed on the multiple first tone mapping values ​​corresponding to the target region A, the result of the weighted sum is divided by the sum of the weighted gain[j] corresponding to the target region A, and the obtained result is used as the final tone mapping value of the current pixel P.

[0362] When at least two first tone mapping values ​​of the current pixel P are weighted, j represents the j-th region A in the four regions A of image 1, and j = 1, 2, 3, 4. In this case, the weight Weight[j] of the first tone mapping values ​​obtained based on the j-th region A can be calculated based on equation (11): Weight[j]=f(Num[j] / Numtotal) Equation (11) Here, in equation (11), j represents the j-th target region A in the four regions A of image 1, Num[j] represents the number of pixels in the j-th target region A in image 1, and Numtotal represents the total number of pixels in all target regions A around the current pixel P.

[0363] Optionally, for example, f(x) in equation (11) represents a function of the target tone mapping curve of the j-th target region A in image 1.

[0364] Optionally, for example, in equation (11), f(x) is power(x, NumStr[j]), where f(x) is a specific representation of the tone mapping curve function, NumStr[j] represents the intensity information of the j-th target region A, and the intensity information represents the decay rate of the weighting coefficient of the j-th target region, and is a preset value for each region A.

[0365] For example, the final tone mapping values ​​Rtm, Gtm, and Btm of the current pixel P can be calculated based on equations (12), (13), and (14). Rtm=Σ(RP*gain[j]*Weight[j]) / ΣWeight[j] Equation (12); Gtm = Σ(GP*gain[j]*Weight[j]) / ΣWeight[j] Equation (13); and Btm=Σ(BP*gain[j]*Weight[j]) / ΣWeight[j] Equation (14) Here, Rp, Gp, and Bp are the distinct color components of the current pixel P in region A of the image to be displayed (for example, the initial region A described above); gain[j]=f[j](max(Rp,Gp,Bp)) / max(Rp,Gp,Bp); and Rp, Gp, and Bp are distinct color components of the current pixel P, max(Rp, Gp, Bp) represents the solution of the maximum component values ​​of the three color components, f[j] is a function of the local target tone mapping curve corresponding to the j-th target region A in image 1, and j is an integer greater than or equal to 1 or less than or equal to 4.

[0366] In this embodiment of the present application, when the current pixel P is located on a boundary shared by two regions A in image 1, and the grayscale values ​​of the two pixels on the boundary are identical, and one pixel is mapped based on the target tone mapping curve of one region A and the other pixel is mapped based on the target tone mapping curve of the other region A, then the image of image 1 located on the two pixels jumps. Therefore, when there are multiple (at least two) target regions A, it indicates that the current pixel P is a pixel on a boundary shared by at least two regions A. The respective target tone mapping curves of at least two regions A around pixel P can be used to perform tone mapping separately on pixel P, and the at least two first tone mapping values ​​obtained are weighted to obtain the final tone mapping value for pixel P. In this way, in different image regions having different tone mapping curves, the display scenes of different image regions can be considered in the display of pixels shared by different image regions, avoiding the occurrence of pixels with jump exceptions in the displayed image.

[0367] Optionally, after the final tone mapping value has been obtained for pixel P, the receiving end in this embodiment of the present invention may further perform saturation adjustment based on a saturation (color) adjustment algorithm for each pixel in region A that is obtained after tone mapping has been performed based on the target tone mapping curve.

[0368] Optionally, for the local or global initial gain adjustment curves in Figure 5a, the curve data may not be directly carried in the backlight metadata, while the parameters and curve type of the initial gain adjustment curve may be carried in the backlight metadata. In this case, the receiving end may obtain the curve type and parameters from the backlight metadata and obtain the local or global target tone mapping curve based on the HDR or SDR source.

[0369] For example, an HDR source or SDR source can be understood as a video or image to be processed at the receiving end. An HDR source or SDR source may, but is not limited to, include image data such as pixel data.

[0370] For example, backlight metadata may include, but is not limited to, the data format of the HDR or SDR source, image region segmentation information, image region scanning order information, image features, curve parameters of a local or global initial gain adjustment curve, and one or more metadata information units. Metadata information units may include coordinate information, the aforementioned image features, the aforementioned initial gain adjustment curve parameters, and similar information.

[0371] It should be noted that the format of the backlight metadata is not limited in this application. The backlight metadata may include, for example, histogram information and tone mapping curve parameter information, as in the ST2094-40 standard, or, as in the ST2094-10 standard, tone mapping curve parameter information, for example.

[0372] As described above, the initial gain adjustment curve in the backlight metadata can be any conventional tone mapping curve. For example, the receiving end may obtain the initial gain adjustment curve by obtaining the curve parameters a, b, p, m, and n of the initial gain adjustment curve from the metadata information unit and HDR or SDR source in the bitstream. Next, the target tone mapping curve may be obtained based on the anchor, the initial gain adjustment curve, and the image to be displayed, where the target tone mapping curve may still include the curve parameters described above.

[0373] For example, the function of the target tone mapping curve can be given by equation (15):

number

[0374] Based on the target tone mapping curve in equation (15), a mapping relationship can be obtained between normalized HDR source data or normalized SDR source data (e.g., the image to be displayed) and normalized HDR display data (e.g., the display value of each pixel displayed on the display and belonging to the image to be displayed).

[0375] It should be noted that L and L' may be normalized optical or electrical signals, and this is not limited to the present invention.

[0376] It should be noted that the data format of the HDR or SDR source in the bitstream where the receiving end is located is not limited in this application. For example, in terms of the color space of the pixel data, pixels in the image or video to be displayed may be YUV or RGB. In another example, in terms of the bit width of the data, each pixel in the image to be displayed may be 8 bits, 10 bits, 12 bits, or similar.

[0377] With respect to images or videos in SDR or HDR format, the receiving end in this embodiment of the application can perform tone mapping on all pixels in the image or video, thereby enabling tone mapping from high dynamic range to low dynamic range in scenarios where the backlight of the display device is adjustable.

[0378] In addition, in this embodiment of the present application, the initial tone mapping curve (i.e., the initial gain adjustment curve described above) and the anchors used to generate the target tone mapping curve may be carried in a bitstream and transmitted to the receiving end. During implementation, a hardware channel used to transmit the curves and integrated into the hardware of the electronic device to which the receiving end belongs may be reused. This is more convenient for implementation from a hardware standpoint and can be applied to the hardware channels of all electronic devices.

[0379] Embodiment 2:

[0380] In the bitstream of Figure 1, the backlight metadata carried in Embodiment 2 may include pixel-level or downsampling gain adjustment coefficients.

[0381] For example, with respect to images or videos in SDR or HDR formats in a bitstream, the image of each frame in the image and video may have a pixel level or downsampling gain adjustment coefficient.

[0382] For example, a pixel level or downsampling gain adjustment coefficient can be represented as an image having the same size as the original frame image. Information about each pixel in the image is expressed as a coefficient, which is here referred to as the gain adjustment coefficient.

[0383] For example, the pixel level or downsampling gain adjustment coefficient can alternatively be expressed as follows: For a table corresponding to pixels in the original frame image, the table has a one-to-one correspondence with the pixels in the original image, and the data in the table are the coefficients of the pixels, where the coefficients are called gain adjustment coefficients.

[0384] The gain adjustment coefficient of pixels in the highlight region of each frame's image may be higher than the gain adjustment coefficient of pixels in the non-highlight region.

[0385] Figure 6a is an example of a diagram of the system architecture obtained after Figure 1 has been improved. The similarities between Figure 6a, Figure 1, and Figure 5a will not be explained again here. Only the improvements to Figure 1 and the differences between Figure 6a and Figure 5a will be explained in detail.

[0386] It should be understood that the system shown in Figure 6a is merely an example. The system in this application, and the transmitting and receiving ends in the system, may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. The components shown in Figure 6a may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processors and / or application-specific integrated circuits.

[0387] As shown in Figure 6a, the transmitting end may include a backlight metadata generation module.

[0388] The backlight metadata generation module may generate backlight metadata based on an electrical signal image or image sequence generated by the optical-electrical conversion module. This backlight metadata may include pixel levels or downsampling gain adjustment coefficients for the image.

[0389] The gain adjustment coefficient is the gain coefficient, and the gain coefficient represents the amplification factor.

[0390] The pixel-level gain adjustment coefficient for an image is used to adjust the pixel-level display pixels of an image based on the gain adjustment coefficient. For example, if the pixel-level unit is a 4x4 image area and the gain adjustment coefficient for the image area is 1.2, the value of the display pixels in the 4x4 image area may be increased by 1.2 times, for example, the brightness may be increased by 1.2 times.

[0391] The encoding module can encode electrical signal images and backlight metadata, generate a bitstream, and transmit the bitstream to the receiving end.

[0392] The following describes the process by which the backlight metadata generation module generates image pixel-level or downsampling gain adjustment coefficients.

[0393] Referring to Figures 6a and 6b, an example of the process for generating a gain adjustment coefficient for image 1 (represented as image 2 in Figure 6b) is shown. In Figure 6b, an example where the target of processing is image 1 is used for illustrative purposes.

[0394] A comparison between Figure 5b and Figure 6b reveals that the processes S401, S402, and S403 in Figure 6b are identical to the processes S401, S402, and S403 in Figure 5b when the anchor is generated. Please refer to the relevant explanation in Figure 5b. Further details will not be explained here.

[0395] Furthermore, referring to Figure 6b, after S403, the backlight metadata generation module may execute S404.

[0396] S404: The backlight metadata generation module constructs gain adjustment coefficients for image 1 based on the highlight and non-highlight regions, and generates image 2 including the gain adjustment coefficients.

[0397] For example, as shown in Figure 6b, different gain adjustment coefficients may be configured for each region A in Image 1, for pixels in the highlight region C within region A, and for pixels in the non-highlight region C within region A.

[0398] For example, in Figure 6b, coefficients 1, 2, 3, and 4 are each comprised of four regions C. Coefficients 1 and 2, corresponding to two highlighted regions C, are greater than coefficients 3 and 4, corresponding to two non-highlighted regions C.

[0399] Optionally, for different highlight regions C, the backlight metadata generation module may configure different gain adjustment coefficients for the different highlight regions C based on the average brightness of the pixels in the highlight region C. For example, the gain adjustment coefficient for region C with a higher average brightness is higher than the gain adjustment coefficient for region C with a lower average brightness.

[0400] For example, the gain adjustment coefficients for the four regions C in Figure 6b are coefficient 1 = 1.2, coefficient 2 = 1.1, coefficient 3 = 0.8, and coefficient 4 = 0.5, respectively.

[0401] For example, the gain adjustment coefficient for the highlight region C may be greater than 1, and the gain adjustment coefficient for the non-highlight region C may be less than or equal to 1.

[0402] In possible implementations, the gain adjustment coefficient can be generated for region A in the following manner.

[0403] First, an initial gain adjustment coefficient is constructed for each region C in region A, where the initial gain adjustment coefficient may be a first predetermined coefficient threshold.

[0404] For example, the first preset coefficient threshold may be 1.0, 0.8, or similar. The specific threshold may be flexibly configured based on the actual requirements; this is not limited to the present application. Optionally, the first preset coefficient threshold may be less than or equal to 1.0.

[0405] Thus, the initial gain adjustment coefficient can be constructed for each region C (which can be understood as a pixel level) in Figure 6b. For example, the initial gain coefficient for all 64 × 64 pixels in region A is 1.0.

[0406] When the gain adjustment coefficient is configured, the initial gain adjustment coefficient is configured based on region C. Specifically, the gain adjustment coefficient for all pixels in the same region C is configured as 1.0. Therefore, the gain adjustment coefficient in this embodiment of the present application may be called a pixel-level gain adjustment coefficient.

[0407] Next, for each highlight region C in region A, the initial gain adjustment coefficient is increased by the coefficient multiplier.

[0408] For example, the coefficient multiplier could be a second preset coefficient threshold, for example, 1.2. In this case, the gain adjustment coefficient for the highlight region C is updated to the first preset coefficient threshold × the second preset coefficient threshold, for example, 1.0 × 1.2 = 1.2. In this case, in Figure 6b, coefficient 1 = coefficient 2 = 1.2, and coefficient 3 = coefficient 4 = 1.0.

[0409] For example, a backlight metadata generation module may calculate the average luminance value of pixels in each highlight region C, and as a result, each highlight region C has a corresponding average luminance value. The coefficient multiplier configured for highlight regions C having a higher average luminance value (e.g., the average luminance is greater than or equal to a preset luminance threshold) is a third preset coefficient threshold, and the coefficient multiplier configured for highlight regions C having a lower average luminance value (e.g., the average luminance is less than a preset luminance threshold) is a fourth preset coefficient threshold, and the third preset coefficient threshold is greater than the fourth preset coefficient threshold.

[0410] For example, if the third preset coefficient threshold is 1.5 and the fourth preset coefficient is 1.2, then the updated coefficients for region C in Figure 6b are coefficient 1 = 1.5, coefficient 2 = 1.2, coefficient 3 = 1, and coefficient 4 = 1, respectively. For example, for region C corresponding to coefficient 1, the value of each pixel in region C in image 2 is no longer the color value in image 1, but coefficient 1, i.e., 1.5. In this way, image 2 of the same size as image 1 can be obtained, and the value of each pixel in image 2 is the pixel-level gain adjustment coefficient.

[0411] Naturally, in other implementations, different gain adjustment coefficients may also be constructed for non-highlight regions C, based on the average brightness of pixels in region C. Gain adjustment coefficients for regions C with higher average brightness are greater than those for regions C with lower average brightness.

[0412] It should be noted that the method described above for constructing the gain adjustment coefficient for region A in Image 1 is merely an implementation. In this application, the gain adjustment coefficient for region A in Image 1 may be constructed separately using a different method. The construction method is not limited to the example described above.

[0413] For example, if image 1 contains multiple regions A, different gain coefficients need to be constructed for region A based on the highlighted and unhighlighted regions within region A. The method is similar, and the details will not be explained here again.

[0414] From the above description, it can be seen that in this application, the gain adjustment coefficient can be configured for pixels in each image region, i.e., at the pixel level, for example, for image regions of size such as 4×10 or 8×6. In this case, an image of the same size as image 1, i.e., a 64×64 image 2, is acquired after S404, and the value of each pixel in image 2 is the gain adjustment coefficient configured by the backlight metadata generation module.

[0415] In the implementation described above, an example in Figure 6a where the transmitting end generates a pixel level or downsampling gain adjustment coefficient is used for illustrative purposes, and as a result, the pixel level or downsampling gain adjustment coefficient can be transmitted to the receiving end as an attribute of the bitstream.

[0416] It should be noted that the receiving end in this embodiment of the present application may also include a backlight metadata generation module for performing the operations described above, thereby generating pixel levels or downsampling gain adjustment coefficients. The method is similar and details are again not described here.

[0417] Next, we return to Figure 6a. Below, we will explain the execution process at the receiving end in Figure 6a by referring to Figure 6c.

[0418] A comparison between Figure 1 and Figure 6a reveals that the display adaptation module in Figure 1 may include the pixel processing module shown in Figure 6a.

[0419] Referring to Figure 6c, the following describes the operation process of the pixel processing module in Figure 6a.

[0420] As shown in Figure 6c, the process may include the following steps.

[0421] S301: The receiving end acquires the image 1 to be displayed in the bitstream.

[0422] This stage is similar to S201 in Figure 5d, and the details will not be explained here again.

[0423] S302: The receiving end obtains backlight metadata in the bitstream, where the backlight metadata includes the pixel level gain adjustment coefficient of image 1.

[0424] For example, as shown in Figure 6a, a decoding module may decode a bitstream to obtain backlight metadata, which may include pixel levels of the image in the bitstream or downsampling gain adjustment coefficients.

[0425] The pixel-level or downsampling gain adjustment coefficient may be the gain adjustment coefficient for an image or all images in a bitstream. This is not limited to the present invention.

[0426] For example, Image 1 in Figure 6b is the image to be displayed in the bitstream. The decoding module can obtain the pixel-level gain adjustment coefficient of Image 1 (for example, Image 2 in Figure 6b) and transmit the pixel-level gain adjustment coefficient to the pixel processing module.

[0427] S303: The receiving end obtains the maximum display brightness and current backlight intensity level of the display.

[0428] This stage is similar to S203 in Figure 5d. Please refer to the explanation above. Further details will not be explained here.

[0429] For example, referring to Figure 6a, the pixel processing module at the receiving end can acquire the maximum display brightness and current backlight intensity information (which may be the current backlight intensity level or the current target backlight brightness) of the display device to which the receiving end belongs, i.e., S303 can be performed by the pixel processing module in Figure 6a.

[0430] S304: The receiving end processes the pixels of image 1 based on the maximum display brightness and generates a first image.

[0431] When the pixels of Image 1 are processed to generate Image 1, it should be noted that the luminance value based on the processing is not limited to the maximum display luminance maxT of the display. The luminance value may be greater than the current target backlight luminance and less than or equal to the maximum display luminance of the display (e.g., a weighted result of the current target backlight luminance and the maximum display luminance).

[0432] For example, an example is used for explanation in which the pixels of Image 1 are processed based on the maximum display brightness to generate Image 1. When the brightness value based on the processing is not the maximum display brightness maxT, the method is similar, and the details are again not explained here.

[0433] For example, scaling may be performed on the display pixels (i.e., image pixels) of image 1 based on the maximum display brightness to obtain a first image (which may be represented by TMvaluemid1 below). Note that the processing method is not limited to scaling and may further include other methods. This is not limited to the present invention.

[0434] Method 1:

[0435] For example, the maximum brightness value maxsource of an image pixel in Image 1 may be used as maxY, and the overall scaling factor for the image pixels in Image 1 is S = maxT / maxY. The image pixels of Image 1 are scaled as a whole based on the scaling factor S, that is, the pixel value of each pixel in Image 1 is multiplied by the scaling factor S to obtain the first image.

[0436] The maximum display brightness of the display is maxT.

[0437] Method 2:

[0438] Backlight metadata may include a pre-set brightness value T3.

[0439] For example, the pre-set luminance value T3 could be the target luminance target_maxdisplay originally generated by curve 0 mentioned in Embodiment 1 above.

[0440] The scaling factor is S = maxT / T3. The pixels of Image 1 are scaled as a whole based on the scaling factor S; that is, the pixel value of each pixel in Image 1 is multiplied by the scaling factor S to obtain the first image.

[0441] The maximum display brightness of the display is maxT.

[0442] When a display has a specific backlight, it should be considered that the actual brightness of the display may be higher than the backlight brightness. For example, if the maximum display brightness of the display is 500 nits and the backlight intensity level is 10%, then the backlight brightness is 50 nits. However, electronic devices may have measures to adjust the final brightness of the display, and as a result, the final brightness may be higher than 50 nits. Therefore, in this embodiment, the first image obtained by processing the display target image 1 based on the brightness value may be generated based on a brightness value that is greater than the backlight brightness and less than or equal to the maximum display brightness (here, the maximum display brightness of the display is used).

[0443] S305: The receiving end processes the pixels of image 1 based on the maximum display brightness and the current backlight intensity level to generate a second image.

[0444] Similar to the execution principle of S304, for example, as shown in Figure 6a, the pixel processing module may process the display pixels of image 1 based on the current backlight intensity information (e.g., current backlight intensity level) and the maximum display brightness maxT, and generate a second image (which may be represented by TMvaluemid2 below).

[0445] If the current backlight intensity information obtained by the pixel processing module is the current target backlight brightness information, the second image does not need to be generated based on the maximum display brightness of the display module.

[0446] For example, the maximum display brightness maxT and the current backlight intensity level Level (i.e., the current target backlight brightness L) xThe product of (=maxT*Level) may be used. For example, if the maximum display brightness is 500 and the current backlight intensity level is 10%, then the current target backlight brightness is 50, and scaling may be performed on the image pixels of image 1 based on the current target backlight brightness to obtain the second image. Note that the processing method is not limited to scaling and may further include other methods. This is not limited to the present invention.

[0447] Method 1:

[0448] For example, the maximum brightness value maxsource for an image pixel in image 1 can be used as maxY, and the overall scaling factor for an image pixel in image 1 is S=L x The value is / maxY. The image pixels of Image 1 (e.g., RGB information) are scaled based on the scaling factor S; that is, the pixel value of each pixel in Image 1 is multiplied by the scaling factor S to obtain the second image.

[0449] The maximum display brightness of the display is maxT, and L x = maxT * Level, where Level is the current backlight intensity level of the display.

[0450] Method 2:

[0451] Backlight metadata may include a pre-set brightness value T3.

[0452] For example, the pre-set luminance value T3 could be the target luminance target_maxdisplay originally generated by curve 0 mentioned in Embodiment 1 above.

[0453] The scaling factor is S = L xThe value is / T3. The image pixels of Image 1 are scaled as a whole based on the scaling factor S; that is, the pixel value of each pixel in Image 1 is multiplied by the scaling factor S to obtain the second image.

[0454] The maximum display brightness of the display is maxT, and L x = maxT * Level, where Level is the current backlight intensity level of the display.

[0455] The display has a specific backlight; for example, when the maximum display brightness of the display is 500 nits and the backlight intensity level is 10%, the backlight brightness is 50 nits. In this embodiment, the display target image 1 may be processed based on the current target backlight brightness to generate a second image.

[0456] For example, when the image pixels of image 1 are scaled as a whole based on a scaling factor S to obtain the first image, this can be implemented by using any one of the following three methods, and is not limited to the three methods listed herein.

[0457] The following three methods are explained using an example of generating the first image (TMvaluemid1). When the second image (TMvaluemid2) is generated, the following three methods are similar, and the details are not explained here again.

[0458] Method 1: TMvaluemid1.R = Scaling Factor S * h (R value of the image to be displayed); TMvaluemid1.G = scaling factor S * h (G value of the image to be displayed); and TMvaluemid1.B = Scaling factor S * h (B value of the image to be displayed).

[0459] Method 2: TMvaluemid1.R = Scaling Factor S * (h (R value of the image to be displayed) + Gain Adjustment Coefficient 2); TMvaluemid1.G = Scaling factor S * (h (G value of the image to be displayed) + Gain adjustment coefficient 2); and TMvaluemid1.B = Scaling factor S * (h (B value of the image to be displayed) + Gain adjustment coefficient 2).

[0460] Method 3: TMvaluemid1.R = Scaling factor S * h (R value of the image to be displayed) + Gain adjustment coefficient 2; TMvaluemid1.G = Scaling factor S * h (G value of the image to be displayed) + Gain adjustment coefficient 2; and TMvaluemid1.B = Scaling factor S * h (B value of the image to be displayed) + Gain adjustment coefficient 2.

[0461] For example, the image to be displayed is Image 1 mentioned above, and TMvaluemid1 represents Image 1 mentioned above; TMvaluemid1.R represents the value of the R channel in the first image; TMvaluemid1.G represents the value of the G channel in the first image; TMvaluemid1.B represents the value of the B channel of the first image; and The functional representation of h(x) may be an equation for the interconversion between x, PQ, HLG, and gamma, or a functional equation for another curve. This is not limited to the present invention.

[0462] The pixel-level or downsampling gain adjustment coefficient in the backlight metadata may include gain adjustment coefficient 1 and gain adjustment coefficient 2, i.e., two groups of gain adjustment coefficients. Naturally, the gain adjustment coefficients for each group are identical to those in the original image.

[0463] For the method of generating the gain adjustment coefficient 1, please refer to the embodiment shown in Figure 6b. Further details will not be explained here.

[0464] The backlight metadata generation module can further generate a gain adjustment coefficient 2.

[0465] Referring to Figure 6a, there may be an image signal loss between the ambient light image collected by the acquisition module and the image of the electrical signal generated by the photoelectric conversion module (for example, it can be converted to a fixed-point difference). The backlight metadata generation module may use the image signal loss as the gain adjustment coefficient 2. The pixel value of each pixel in the gain adjustment coefficient 2 is the image signal lost by each pixel.

[0466] For example, a 12-bit image can be converted to an 8-bit image by using an acquisition module and an optical-electrical conversion module. In this case, 4 bits of image data are lost for each pixel in the image, and the 4 bits of data lost for each pixel can be used as the pixel value of the corresponding pixel in the gain adjustment coefficient 2.

[0467] The first and second images are acquired based on gain adjustment coefficients 2 and 1, respectively, which can improve the contrast of the images.

[0468] S306: The receiving end processes the first and second images based on the gain adjustment coefficient of image 1 to obtain image display information for image 1.

[0469] Please note that the order in which S301, S302, and S303 are performed is not limited in this application.

[0470] The order in which S304 and S305 are performed is not limited in this application.

[0471] For example, referring to Figure 6a, the pixel processing module at the receiving end can process the first and second images corresponding to Image 1 based on the gain adjustment coefficient in the pixel level gain adjustment coefficient that is for Image 1 (for example, Image 2 in Figure 6b), and obtain the image display information of Image 1 on the display, i.e., the tone mapping of each pixel in Image 1.

[0472] for example, Tmvalue=W*TMvaluemid1+(1-W)*TMvaluemid2 Equation (16) Here, Tmvalue in equation (16) represents the tone mapping value of each pixel in image 1; and TMvaluemid1 represents the first image, and TMvaluemid2 represents the second image.

[0473] For example, W may represent the gain adjustment coefficient of image 1, which can be represented, for example, as image 2 generated in Figure 6b.

[0474] Alternatively, for example, W could represent a function such as the square of the gain adjustment coefficient 1 for image 1.

[0475] Specifically, a weighted sum is performed on the first and second images by using the gain adjustment coefficient 1 of image 1 or a function of the gain adjustment coefficient 1 as a weight, thereby obtaining the tone mapping value of image 1.

[0476] Alternatively, for example, Tmvalue = W * TMvaluemid2 equation (17) Here, Tmvalue in equation (17) represents the tone mapping value of each pixel in image 1; and TMvaluemid1 represents the first image.

[0477] For example, W could be the gain adjustment coefficient for image 1 (which can be represented, for example, as image 2 generated in Figure 6b).

[0478] Alternatively, for example, W could be a function A(x) of the gain adjustment coefficient 1 of image 1, for example, A(x) = 1 + (S-1)*x, where x represents the gain adjustment coefficient 1 of image 1, and S represents the scaling factor used when the first or second image is generated. Note that the function A(x) is not limited here.

[0479] For example, Tmvalue is the final image pixel information for each pixel in Image 1 at the current target backlight brightness, and the tone mapping value for each pixel in Tmvalue is the display value (or final RGB value) for each pixel in Image 1 on the display.

[0480] In this embodiment, the display target image 1 is processed based on the current target backlight brightness to obtain a second image, and based on the respective gain adjustment coefficients for the highlight and non-highlight regions in image 1, the highlight regions in the second image are made brighter and the non-highlight regions (e.g., dark regions) are made darker, and the display brightness of each pixel in image 1 on the screen is obtained. For example, if the gain adjustment coefficient for a region in image 1 is 1.5, the display brightness of the region may be increased by 1.5 times when the region is displayed. As a result, the region may be excessively bright when displayed. In this case, optionally, clamping can be performed on Tmvalue by using the first image TMvaluemid1, thereby avoiding excessive brightness in some regions when image 1 is displayed on the display.

[0481] Optionally, when the gain adjustment coefficient in the backlight metadata is a downsampling gain adjustment coefficient, the downsampling gain coefficient and the pixels of the display target image (e.g., Image 1) may have a one-to-one correspondence, or a one-to-many correspondence such as a one-to-four or one-to-sixteen correspondence. In this case, when the display brightness of each pixel of Image 1 on the screen is obtained based on the downsampling gain adjustment coefficient, a correspondence between the downsampling gain coefficient and the pixels of Image 1 can be obtained. Next, based on the correspondence, the corresponding pixels in Image 1 are processed based on the corresponding downsampling gain coefficient. The specific processing method is similar in principle to the process of processing Image 1 based on the pixel-level gain coefficient described in Embodiment 2 above. Please refer to the above description. Further details will not be explained here.

[0482] In this embodiment of the present application, a pixel-level gain adjustment coefficient is carried in the bitstream, and as a result, when an image is displayed in a scenario where the backlight is adjustable, the display brightness of each pixel on the display can be adjusted based on the gain adjustment coefficient of each pixel in the image. This method is easier to implement in software, tone mapping can reach the pixel level, and brightness adjustment is improved, flexible, and more adjustable.

[0483] It should be noted that Embodiments 1 and 2 can also be combined. Specifically, the backlight metadata may include local or global initial gain adjustment curves, local or global anchors, and pixel-level or downsampling gain adjustment coefficients. For similarities between Embodiments 1 and 2, please refer to each other.

[0484] Optionally, for a single frame image or frame sequence in a bitstream, the receiving end in this embodiment of the application may perform tone mapping on a portion of the image, an image region, or a portion of the frame sequence based on a target tone mapping curve to generate tone mapping values, thereby obtaining image display information, and further, may obtain image display information for other portions of the image or frame sequence based on a gain adjustment coefficient corresponding to the image, the current target backlight brightness, and the maximum display brightness of the display.

[0485] Conventional dynamic metadata display compliance standards or algorithms typically do not consider the display effect of images or videos on backlight-adjustable devices. In conventional solutions, videos are primarily adapted to the screen's maximum display capability (i.e., maximum display brightness), which is unsuitable for displaying images or videos in scenarios with low backlighting and results in poor display effect.

[0486] However, in the system described above in this embodiment of the present application, when the backlight of the display is adjustable, brightness adjustment can be performed on the image or video to be displayed by referring to the maximum display brightness of the display and the current target backlight brightness, thereby automatically adjusting the display brightness of the image on the display based on the adjustment change of the backlight intensity level, thereby improving the display effect of the image or video when the backlight is weakened.

[0487] The following describes the apparatus provided in the embodiments of the present application.

[0488] Figure 7 is a diagram showing the structure of an apparatus according to an embodiment of the present invention. As shown in Figure 7, the apparatus 500 may include a processor 501 and a transceiver 505, and optionally further include a memory 502.

[0489] The transceiver 505 may be referred to as a transceiver unit, transceiver device, transceiver circuit, or similar, and is configured to implement transceiver functions. The transceiver 505 may include a receiver and a transmitter. The receiver may be referred to as a receiver device, receiver circuit, or similar, and is configured to implement receiving functions. The transmitter may be referred to as a transmitter device, transmitter circuit, or similar, and is configured to implement transmitting functions.

[0490] Memory 502 may store a computer program, software code, or instruction 504, where the computer program, software code, or instruction 504 may also be referred to as firmware. The processor 501 may implement the method provided in the embodiments of the present application by executing the computer program, software code, or instruction 503 in the processor 501, or by calling the computer program, software code, or instruction 504 stored in memory 502. The processor 501 may be a central processing unit (CPU), and memory 502 may be, for example, read-only memory (ROM) or random access memory (RAM).

[0491] The processor 501 and transceiver 505 described herein may be implemented in an integrated circuit (IC), analog IC, radio frequency integrated circuit (RFIC), mixed-signal IC, application-specific integrated circuit (ASIC), printed circuit board (PCB), electronic device, or similar.

[0492] The apparatus 500 may further include an antenna 506. The modules included in the apparatus 500 are merely illustrative examples and are not limited herein.

[0493] As described above, the structure of the device described in the embodiments described above may not be limited to Figure 7. The device may be a standalone device or part of a larger device. For example, the implementation of the device may be: (1) a standalone integrated circuit IC, chip, or chip system or subsystem; (2) a set of one or more ICs, where optionally the set of ICs may also include a storage component for storing data and instructions; (3) a module that may be embedded in another device; (4) an in-vehicle device or similar; or (5) something else.

[0494] For cases where the device is implemented as a chip or a chip system, please refer to the diagram of the chip structure shown in Figure 8. The chip shown in Figure 8 includes a processor 601 and an interface 602. There may be one or more processors 601 and multiple interfaces 602. Optionally, the chip or chip system may include memory 603.

[0495] All relevant details of the steps in the embodiments of the method described above can be referenced in the description of the functions of the corresponding functional modules. Further details are not provided here.

[0496] Based on the same technical concept, embodiments of the present invention further provide a computer-readable storage medium. The computer-readable storage medium stores a computer program, the computer program includes at least one segment of code, and at least one segment of code can be executed by the computer to control the computer to implement the embodiments of the method described above.

[0497] Based on the same technical idea, embodiments of the present application further provide a computer program. When the computer program is executed by a terminal device, the computer program is used to implement the embodiments of the method described above.

[0498] The entire program or a portion of the program may be stored in a storage medium encapsulated by the processor, or a portion of the program or the entire program may be stored in memory that is not encapsulated by the processor.

[0499] Based on the same technical concept, embodiments of the present application further provide a chip including a network interface controller and a processor. The network interface controller and processor may implement embodiments of the method described above.

[0500] The methods or algorithmic steps described in conjunction with those disclosed in embodiments of this application may be implemented by hardware or by a processor executing software instructions. Software instructions may include corresponding software modules. These software modules may be stored in random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable hard disks, compact disk read-only memory (CD-ROM), or any other form of storage medium known in the art. An exemplary storage medium may be coupled to a processor, thereby enabling the processor to read information from and write information to the storage medium. Of course, the storage medium may alternatively be a component of the processor. The processor and storage medium may reside in an ASIC.

[0501] Those skilled in the art will recognize that, in one or more of the above examples, the functions described in the embodiments of the present application may be implemented by hardware, software, firmware, or any combination thereof. When a function is implemented by software, the function may be stored in a computer-readable medium or transmitted in a computer-readable medium as one or more instructions or codes. The computer-readable medium includes computer storage media and media, where the media includes any medium that facilitates the transmission of a computer program from one place to another. The storage medium may be any available medium accessible from a general-purpose or dedicated computer.

[0502] Embodiments of the present application will be described above with reference to the accompanying drawings, but the present application is not limited to the specific implementations described above, and the specific implementations described above are merely illustrative and not limiting. Under the enlightenment of the present application, a person skilled in the art can create many forms without departing from the purpose and scope of the claims of the present application, all of which are covered by the present application. (Other possible items) [Item 1] A step in which the first display information of the target image to be displayed is obtained based on the current target backlight brightness of the display; A step of acquiring second display information of the target image based on the maximum display brightness of the display and the current target backlight brightness; A step of obtaining a third display information of the target image by processing the first display information and the second display information based on the pre-set information of the target image, wherein the pre-set information includes information regarding the brightness of the target image; and Step of displaying the target image on the display based on the third display information. An image processing method comprising: [Item 2] The step of obtaining first display information of the target image to be displayed based on the current target backlight brightness of the display is: The step of obtaining a first tone mapping curve for the target image to be displayed; and The step of processing the first tone mapping curve based on the current target backlight brightness of the display and obtaining the second tone mapping curve of the target image as first display information. The method described in item 1, including the method described in item 1. [Item 3] The step of obtaining second display information of the target image based on the maximum display brightness of the display and the current target backlight brightness is: A step of determining a first brightness based on the maximum display brightness of the display and the current target backlight brightness, wherein the first brightness is higher than the current target backlight brightness; and The step of processing the first tone mapping curve based on the first brightness and obtaining the third tone mapping curve of the target image as the second display information. The method described in item 2, including the method described in item 2. [Item 4] The aforementioned pre-configured information includes the second brightness of the target image, and the step of processing the first display information and the second display information based on the pre-configured information of the target image to obtain the third display information of the target image is: A step of acquiring the second brightness of the target image; A step of extracting a first sub-curve from the second tone mapping curve in which the pixel brightness is less than or equal to the second brightness; A step of obtaining the first tone mapping value in the second tone mapping curve that corresponds to the second brightness; A step of processing the third tone mapping curve based on the first tone mapping value to generate a fourth tone mapping curve, wherein the tone mapping value in the fourth tone mapping curve that corresponds to the second luminance is the first tone mapping value; A step of extracting a second sub-curve from the fourth tone mapping curve in which the pixel brightness is greater than or equal to the second brightness; and The step of connecting the first subcurve and the second subcurve based on the second brightness, and acquiring the target tone mapping curve of the target image as the third display information. The method described in item 3, including the method described in item 3. [Item 5] The step of obtaining the first tone mapping curve of the target image to be displayed is: Steps to acquire the first tone mapping curve for each of the multiple local regions constituting the target image. Includes, The step of obtaining the second brightness of the target image is: Steps to acquire the second brightness level for each of the aforementioned multiple local regions. It includes, and The step of connecting the first subcurve and the second subcurve based on the second brightness, and obtaining the target tone mapping curve of the target image as the third display information, is: For the plurality of local regions of the target image, the first subcurve and the second subcurve of each of the plurality of local regions are connected based on the second brightness of each of the plurality of local regions, and the target tone mapping curve of each of the plurality of local regions is acquired as the third display information of the target image. The method described in item 4, including the method described in item 4. [Item 6] The step of obtaining the second brightness of the target image is: A step of obtaining the second brightness of the target image based on the minimum value of the pixel brightness and the average value of the pixel brightness of the target image. The method described in item 4, including the method described in item 4. [Item 7] The step of obtaining the second brightness of the target image based on the minimum value and the average value of the pixel brightness of the target image is as follows: A step of clustering image pixels of the target image based on the pixel brightness and pixel position in the target image to obtain multiple image regions; A step of classifying the plurality of image regions in the target image based on the pixel brightness to obtain a first type image region and a second type image region, where the brightness of the first type image region is higher than the brightness of the second type image region; and A step of determining the second brightness of the target image based on the minimum value of the pixel brightness in the first type image region and the average value of the pixel brightness. The method described in item 6, including the method described in item 6. [Item 8] The step of obtaining first display information of the target image to be displayed based on the current target backlight brightness of the display is: The step of processing the target image to be displayed based on the current target backlight brightness of the display and acquiring the first image as the first display information. The method described in item 1, including the method described in item 1. [Item 9] The step of obtaining second display information of the target image based on the maximum display brightness of the display and the current target backlight brightness is: A step of determining a third brightness based on the maximum display brightness of the display and the current target backlight brightness, wherein the third brightness is higher than the current target backlight brightness; and The step of processing the target image based on the third brightness and acquiring the second image as second display information. The method described in item 8, including the method described in item 8. [Item 10] The aforementioned pre-configured information includes the gain coefficient of the target image, and the step of processing the first display information and the second display information based on the pre-configured information of the target image to obtain the third display information of the target image is: The step of obtaining the gain coefficient of the target image; and A step of processing the first image and the second image based on the gain coefficient to generate a third image as the third display information. The method described in item 9, including the method described in item 9. [Item 11] The step of obtaining the gain coefficient of the target image is: A step of acquiring multiple image regions by clustering the image pixels of the target image based on the pixel brightness and pixel position in the target image; A step of classifying the plurality of image regions in the target image based on the pixel brightness to obtain a first type image region and a second type image region, wherein the brightness of the first type image region is higher than the brightness of the second type image region; and A step of configuring different gain coefficients for the first type image region and the second type image region, where the gain coefficient of the first type image region is greater than the gain coefficient of the second type image region. The method described in item 10, including the method described in item 10. [Item 12] The step of processing the first display information and the second display information based on the pre-set information of the target image to obtain the third display information of the target image is as follows: The step of obtaining the pre-set information of the target image from the source information of the target image to be displayed; and A step of processing the first display information and the second display information based on the aforementioned pre-set information to obtain the third display information of the target image. The method described in item 1, including the method described in item 1. [Item 13] An electronic device comprising memory and a processor, wherein the memory is coupled to the processor, the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device is capable of performing the image processing method described in any one of items 1 to 12. [Item 14] A computer-readable storage medium comprising a computer program, wherein when the computer program operates on an electronic device, the electronic device is capable of performing the image processing method described in any one of items 1 to 12. [Item 15] A chip comprising one or more interface circuits and one or more processors, wherein the interface circuits are configured to receive signals from the memory of an electronic device and to transmit the signals to a processor, wherein the signals include computer instructions stored in the memory; and when the processor executes the computer instructions, the electronic device is capable of performing the image processing method described in any one of items 1 to 12.

Claims

1. A step in which the first display information of the target image is obtained based on the current target backlight brightness set for the display; A step of acquiring second display information of the target image based on the maximum display brightness of the display and the current target backlight brightness; A step of obtaining a third display information of the target image by processing the first display information and the second display information based on the pre-set information of the target image, wherein the pre-set information includes information regarding the brightness of the target image; and Step of displaying the target image on the display based on the third display information. An image processing method comprising:

2. The step of obtaining first display information of a target image based on the current target backlight brightness set for the display is: The step of obtaining the first tone mapping curve of the target image; and The step of processing the first tone mapping curve based on the current target backlight brightness and obtaining the second tone mapping curve of the target image as first display information. The image processing method according to claim 1, including the method described in claim 1.

3. The step of acquiring second display information of the target image based on the maximum display brightness of the display and the current target backlight brightness is as follows: A step of determining a first brightness based on the maximum display brightness of the display and the current target backlight brightness, wherein the first brightness is higher than the current target backlight brightness; and The step of processing the first tone mapping curve based on the first brightness and obtaining the third tone mapping curve of the target image as second display information. The image processing method according to claim 2, including the method described in claim 2.

4. The aforementioned pre-set information includes the second brightness of the target image, and the step of processing the first display information and the second display information based on the pre-set information of the target image to obtain the third display information of the target image is: A step of acquiring the second brightness of the target image; Steps include extracting a first sub-curve from the second tone mapping curve in which the pixel brightness is less than or equal to the second brightness; A step of obtaining a first tone mapping value in the second tone mapping curve that corresponds to the second brightness; A step of processing the third tone mapping curve based on the first tone mapping value to generate a fourth tone mapping curve, wherein the tone mapping value in the fourth tone mapping curve that corresponds to the second luminance is the first tone mapping value; A step of extracting a second sub-curve from the fourth tone mapping curve in which the pixel brightness is greater than or equal to the second brightness; and The step of connecting the first subcurve and the second subcurve based on the second brightness, and acquiring the target tone mapping curve of the target image as the third display information. The image processing method according to claim 3, including the method described in claim 3.

5. The step of obtaining the first tone mapping curve of the target image is as follows: Steps to acquire the first tone mapping curve for each of the multiple local regions constituting the target image. Includes, The step of obtaining the second brightness of the target image is: Steps to acquire the second brightness level for each of the aforementioned multiple local regions. It includes, and The step of connecting the first subcurve and the second subcurve based on the second brightness, and obtaining the target tone mapping curve of the target image as the third display information, is as follows: For the plurality of local regions of the target image, the first subcurve and the second subcurve of each of the plurality of local regions are connected based on the second brightness of each of the plurality of local regions, and the target tone mapping curve of each of the plurality of local regions is acquired as the third display information of the target image. The image processing method according to claim 4, including the method described in claim 4.

6. The step of obtaining the second brightness of the target image is: A step of obtaining the second brightness of the target image based on the minimum value of the pixel brightness of the target image and the average value of the pixel brightness. The image processing method according to claim 4, including the method described in claim 4.

7. The step of obtaining the second brightness of the target image based on the minimum value and the average value of the pixel brightness of the target image is as follows: A step of acquiring multiple image regions by clustering image pixels of the target image based on the pixel brightness and pixel position in the target image; A step of classifying the plurality of image regions in the target image based on the pixel brightness to obtain a first type image region and a second type image region, where the brightness of the first type image region is higher than the brightness of the second type image region; and A step of determining the second brightness of the target image based on the minimum value of the pixel brightness and the average value of the pixel brightness in the first type image region. The image processing method according to claim 6, including the method described in claim 6.

8. The step of obtaining first display information of a target image based on the current target backlight brightness set for the display is: In the step of acquiring the first image as the first display information, the first image is generated by processing the target image based on the current target backlight brightness. The image processing method according to claim 1, including the method described in claim 1.

9. The step of acquiring second display information of the target image based on the maximum display brightness of the display and the current target backlight brightness is as follows: A step of determining a third brightness based on the maximum display brightness and the current target backlight brightness of the display, wherein the third brightness is higher than the current target backlight brightness; and In the step of acquiring the second image as the second display information, the second image is generated by processing the target image based on the third brightness. The image processing method according to claim 8, including the method described in claim 8.

10. The aforementioned pre-configured information includes the gain coefficient of the target image, and the step of processing the first display information and the second display information based on the pre-configured information of the target image to obtain the third display information of the target image is: The step of obtaining the gain coefficient of the target image; and The third image is generated as the third display information, where the third image is generated by processing the first image and the second image based on the gain coefficient. The image processing method according to claim 9, including the method described in claim 9.

11. The step of obtaining the gain coefficient of the target image is: A step of acquiring multiple image regions by clustering the image pixels of the target image based on the pixel brightness and pixel position in the target image; A step of classifying the plurality of image regions in the target image based on the pixel brightness to obtain a first type image region and a second type image region, where the brightness of the first type image region is higher than the brightness of the second type image region; and A step of configuring different gain coefficients for the first type image region and the second type image region, where the gain coefficient of the first type image region is greater than the gain coefficient of the second type image region. The image processing method according to claim 10, including the method described in claim 10.

12. The step of processing the first display information and the second display information based on the pre-set information of the target image to obtain the third display information of the target image is as follows: The step of obtaining the pre-set information of the target image from the source information of the target image; and The step of processing the first display information and the second display information based on the pre-set information to obtain the third display information of the target image. The image processing method according to claim 1, including the method described in claim 1.

13. An electronic device comprising memory and a processor, wherein the memory is coupled to the processor, the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device is capable of performing the image processing method according to any one of claims 1 to 12.

14. A computer program that causes an electronic device to execute the image processing method described in any one of claims 1 to 12.

15. Equipped with one or more processors and memory, The aforementioned memory stores computer instructions, The chip comprises one or more processors that execute the image processing method according to any one of claims 1 to 12 by executing the computer instructions.

Images