Image processing method and electronic device

Abstract

The embodiments of the present application provide an image processing method and an electronic device, and relate to the field of image processing technology. The method includes: obtaining display information of an image to be displayed at a current target backlight luminance of a display, and obtaining display information of an image to be displayed at a luminance higher than the current target backlight luminance and of the display, and obtaining the display information of the image to be displayed based on the two types of display information, so that in a scenario where the backlight of the display is adjustable, the display effect of the image can be improved.

Description

[Technical field]

[0001] This application claims priority to Chinese Patent Application No. 202210033857.8, entitled "IMAGE PROCESSING METHOD AND ELECTRONIC DEVICE," and filed with the State Intellectual Property Office of China on January 12, 2022, the entire contents of which are incorporated herein by reference.

[0002] FIELD OF THE DISCLOSURE Embodiments of the present application relate to the field of image processing technology, and in particular to an image processing method and electronic device. [Background technology]

[0003] Currently, when displaying an image or video, an electronic device may perform tone mapping to the display luminance of the image pixels on the display based on the maximum display luminance of the display.

[0004] However, with the current tone mapping methods, the display effect of images or videos is poor in scenarios where the backlight of the display is adjustable. Summary of the Invention

[0005] In order to solve the above technical problems, the present application provides an image processing method and apparatus, in which the display information of an image to be displayed on a display can be obtained based on the maximum display brightness and the current target backlight brightness of the display, so that in a scenario where the backlight of the display is adjustable, the image display effect can be improved.

[0006] According to a first aspect, an embodiment of the present application provides an image processing method. The method includes the steps of: obtaining first display information of a target image to be displayed based on a current target backlight luminance of a display; obtaining second display information of the target image based on a maximum display luminance of the display and a current target backlight luminance; processing the first display information and the second display information based on preset information of the target image to obtain third display information of the target image, where the preset information includes information about the luminance of the target image; and displaying the target image on the display based on the third display information. Thus, considering that in a scenario in which a display has a certain backlight luminance, the actual luminance of the display may be higher than the backlight luminance, in the method, the first display information of the image may be obtained with reference to the backlight luminance, and then the second display information of the image may be obtained with reference to the maximum display luminance and the backlight luminance of the display, and the two types of display information may be processed to obtain final display information of the image. In this way, the final display information may combine the backlight luminance and the maximum display luminance, so that the adaptation effect of the image on the display is improved, thereby improving the image display effect.

[0007] For example, when the second display information is acquired, the second display information of the image may be acquired based on a luminance that is greater than the current target backlight luminance and less than or equal to the maximum display luminance, thereby avoiding a case where the luminance of the highlighted region is increased too high when the target image is displayed based on the third display information.

[0008] For example, the target image may be an independent image or may be an image of a frame in a video, which is not a limitation of the present application.

[0009] For example, the current target backlight brightness is the expected backlight brightness.

[0010] For example, the maximum display brightness of a display is 500 nit, and the current backlight intensity level is set to 10%, and a current target backlight brightness: 50 nit is obtained. However, after the current backlight brightness level is set to 10%, the current backlight brightness (i.e., actual backlight brightness) of the display may be higher than 50 nit, for example, 100 nit. Therefore, the backlight brightness corresponding to the backlight intensity level here may be referred to as the current target backlight brightness, i.e., the backlight brightness expected by the system or the user.

[0011] According to a first aspect, the step of obtaining first display information of a target image to be displayed based on a current target backlight luminance of a display includes: obtaining a first tone mapping curve of the target image to be displayed; and processing the first tone mapping curve based on a current target backlight luminance of the display to obtain a second tone mapping curve of the target image as the first display information.

[0012] For example, the first tone mapping curve may be obtained from a source of the target image, or may be generated at the image display end, which is not a limitation in this application.

[0013] The source information may be any type or types of information obtained by analyzing information carried in an image. The source information may be metadata information for the image and is carried in a bitstream for transmission.

[0014] For example, the first tone mapping curve may be a global initial tone mapping curve or a local initial tone mapping curve carried in the bitstream.

[0015] For example, when the first tone mapping curve is a local (eg, local image region of the target image) initial tone mapping curve, the third display information of the local image region of the target image may be obtained according to the method in the first aspect.

[0016] For example, when the first tone mapping curve is a global (eg, target image) initial tone mapping curve, the third display information of the target image may be obtained according to the method in the first aspect.

[0017] For example, when the first tone mapping curve is processed based on the current target backlight luminance of the display, the processing manner may include, but is not limited to, a scaling process, a translation process, and the like, which is not limited in this application.

[0018] According to the first aspect or any one of the above-mentioned implementations of the first aspect, the step of obtaining second display information of the target image based on the maximum display luminance of the display and the current target backlight luminance includes: determining a first luminance based on the maximum display luminance of the display and the current target backlight luminance, where the first luminance is higher than the current target backlight luminance; and processing the first tone mapping curve based on the first luminance and obtaining a third tone mapping curve of the target image as the second display information.

[0019] For example, the first luminance may be less than or equal to the maximum display luminance.

[0020] For example, when the first tone mapping curve is processed based on the first luminance, the processing manner includes, but is not limited to, a scaling process, a translation process, and the like, which is not limited in this application.

[0021] According to the first aspect or any one of the above-mentioned implementations of the first aspect, the preset information includes a second luminance of the target image, and the step of processing the first display information and the second display information based on the preset information of the target image to obtain the third display information of the target image includes: obtaining the second luminance of the target image; extracting a first sub-curve from the second tone mapping curve, in which the pixel luminance is less than or equal to the second luminance; obtaining a first tone mapping value in the second tone mapping curve, which corresponds to the second luminance; processing the third tone mapping curve based on the first tone mapping value to generate a fourth tone mapping curve, where the tone mapping value in the fourth tone mapping curve, which corresponds to the second luminance, is the first tone mapping value; extracting a second sub-curve from the fourth tone mapping curve, in which the pixel luminance is greater than or equal to the second luminance; and connecting the first sub-curve and the second sub-curve based on the second luminance, and obtaining a target tone mapping curve of the target image as the third display information.

[0022] For example, the second luminance may be referred to as an anchor. The anchor (here, the second luminance) may be a global anchor, i.e., an anchor of a target image, or a local anchor, i.e., an anchor of a local image region in the target image. When a target tone mapping curve is generated for a frame of image or an image region, a pair of an initial tone mapping curve and an anchor may be used for implementation.

[0023] For example, when a target tone mapping curve is generated for a frame of an image, an initial tone mapping curve of the frame of the image and an anchor of the frame of the image may be used for implementation.

[0024] For example, when a target tone mapping curve is generated for a local image region in a frame of an image, an initial tone mapping curve of the local image region in the frame of the image and an anchor of the local image region may be used for implementation.

[0025] For example, a global anchor may be used as an anchor for images of multiple frames or multiple image regions, and a global initial tone mapping curve may be used as an initial tone mapping curve for a frame sequence (i.e., images of multiple frames) or multiple image regions.

[0026] For example, when an anchor for a target image is obtained, the anchor may be obtained from the source, i.e., the video or image anchor may be sent to the bitstream.

[0027] For example, when an anchor for a target image is obtained, the anchor may alternatively be generated by the image or video display end (i.e. the display device to which the display belongs), although this is not a limitation in this application.

[0028] For example, if the second luminance of the target image is, for example, a global anchor of a frame of the target image, the second luminance may be a value between a minimum luminance value and a maximum luminance value of an image pixel of the target image.

[0029] For example, if the second luminance of the target image is, for example, a local anchor of a local image region in the target image, the second luminance may be a value between a minimum luminance value and a maximum luminance value of image pixels in the local image region.

[0030] For example, the tone mapping curve in each implementation of the first aspect may indicate a mapping relationship between the luminance of a pixel in an image (ie, pixel luminance, pixel brightness) and the display luminance on a display.

[0031] According to the first aspect, or any one of the above-mentioned implementations of the first aspect, the step of obtaining a first tone mapping curve of the target image to be displayed includes: obtaining a first tone mapping curve for each of a plurality of local regions constituting the target image, and the step of obtaining a second luminance of the target image includes: obtaining a second luminance for each of the plurality of local regions, and connecting the first sub-curve and the second sub-curve based on the second luminance, and obtaining a target tone mapping curve of the target image as third display information includes: for a plurality of local regions of the target image, connecting the first sub-curve and the second sub-curve for each of the local regions based on the second luminance of each of the local regions, and obtaining a target tone mapping curve for each of the plurality of local regions as third display information of the target image.

[0032] In this implementation, each local region in the target image has a corresponding first tone mapping curve (i.e., a local tone mapping curve) and a corresponding anchor (i.e., a local anchor). In this case, for each local region, a corresponding target tone mapping curve can be generated for each local region based on the process described in the above implementation of the first aspect. Then, when the target image is displayed, tone mapping is performed for each local region based on the corresponding target tone mapping curve, thereby implementing display adaptation of the target image on a display with a specific backlight.

[0033] According to the first aspect or any one of the above-mentioned implementations of the first aspect, the step of obtaining a second luminance of the target image includes a step of obtaining the second luminance of the target image based on a minimum pixel luminance value and an average pixel luminance value of the target image.

[0034] For example, when the second luminance is an anchor of all image regions of the target image, the anchor may be obtained based on the minimum pixel luminance value and the average pixel luminance value of image pixels in the target image.

[0035] For example, when the second luminance is an anchor for a local image region of the target image, the anchor may be obtained based on a minimum value of pixel luminance and an average value of pixel luminance for image pixels in the local image region.

[0036] According to the first aspect or any one of the above-mentioned implementations of the first aspect, the step of obtaining a second luminance of the target image based on a minimum pixel luminance value and an average pixel luminance value of the target image includes: clustering image pixels of the target image based on pixel luminance and pixel location in the target image to obtain a plurality of image regions; classifying the plurality of image regions in the target image based on pixel luminance to obtain a first type image region and a second type image region, where the luminance of the first type image region is higher than the luminance of the second type image region; and determining the second luminance of the target image based on a minimum pixel luminance value and an average pixel luminance value in the first type image region.

[0037] For example, when the target image is divided into multiple image regions, pixels with similar pixel intensities and similar spatial locations can be divided into one image region based on the pixel intensities and spatial locations of the pixels in the target image, and image region division can be implemented, which is not limited to the above-mentioned clustering scheme.

[0038] For example, the first type image regions may be referred to as highlighted regions and the second type image regions may be referred to as non-highlighted regions.

[0039] For example, to distinguish between the highlight region and the non-highlight region, the highlight region and the non-highlight region can be distinguished through a comparison between the average luminance value of the image region with a preset threshold, which is not limited in this application.

[0040] According to the first aspect or any one of the above-mentioned implementations of the first aspect, the step of obtaining first display information of the target image to be displayed based on the current target backlight luminance of the display includes: processing the target image to be displayed based on the current target backlight luminance of the display to obtain the first image as the first display information.

[0041] For example, image pixels of the target image may be processed based on the current target backlight luminance, and the processing manner includes, but is not limited to, image processing manners such as image scaling processing.

[0042] According to the first aspect or any one of the above-mentioned implementations of the first aspect, the step of obtaining second display information of the target image based on the maximum display luminance of the display and the current target backlight luminance includes: determining a third luminance based on the maximum display luminance of the display and the current target backlight luminance, where the third luminance is higher than the current target backlight luminance, and processing the target image based on the third luminance to obtain the second image as the second display information.

[0043] For example, the third luminance can be less than or equal to the maximum display luminance.

[0044] For example, when the target image is processed based on the third luminance, the processing manner includes, but is not limited to, an image processing manner such as a scaling process, which is not limited in the present application.

[0045] According to the first aspect or any one of the above-mentioned implementations of the first aspect, the preset information includes a gain coefficient of the target image, and the step of processing the first display information and the second display information based on the preset information of the target image to obtain third display information of the target image includes: obtaining a gain coefficient of the target image; and processing the first image and the second image based on the gain coefficient to generate the third image as the third display information.

[0046] For example, the gain factor for the target image may be a pixel-level or downsampling gain factor.

[0047] For example, the gain factor may be used to perform a brightness adjustment on the image pixels of the target image.

[0048] For example, the gain factor for a 5x5 pixel unit may be 1.2, indicating that the image pixels of a 5x5 image region in the target image will be 1.2 times brighter.

[0049] For example, regions in the target image with large luminance differences correspond to different gain factors, with the gain factor being higher for image regions with higher luminance than for image regions with lower luminance.

[0050] For example, the gain factor for a highlighted region in the target image is greater than the gain factor for a non-highlighted region in the target image.

[0051] According to the first aspect or any one of the above-mentioned implementations of the first aspect, obtaining a gain factor for the target image includes: clustering image pixels of the target image based on pixel brightness and pixel location in the target image to obtain a plurality of image regions; classifying the plurality of image regions in the target image based on pixel brightness to obtain a first type image region and a second type image region, where the brightness of the first type image region is higher than the brightness of the second type image region; and configuring different gain factors for the first type image region and the second type image region, where the gain factor of the first type image region is higher than the gain factor of the second type image region.

[0052] For example, when the target image is divided into multiple image regions, pixels with similar pixel intensities and similar spatial locations can be divided into one image region based on the pixel intensities and spatial locations of the pixels in the target image, and image region division can be implemented, which is not limited to the above-mentioned clustering scheme.

[0053] For example, the first type image regions may be referred to as highlighted regions and the second type image regions may be referred to as non-highlighted regions.

[0054] For example, to distinguish between the highlight region and the non-highlight region, the highlight region and the non-highlight region can be distinguished through a comparison between the average luminance value of the image region with a preset threshold, which is not limited in this application.

[0055] In this way, highlighted areas in the target image may be brightened and non-highlighted areas may be darkened, thereby implementing display adaptation of the target image to a display having a particular backlight brightness.

[0056] According to the first aspect or any one of the above-mentioned implementations of the first aspect, the step of processing the first display information and the second display information based on preset information of the target image to obtain third display information of the target image includes: a step of obtaining preset information of the target image from source information of the target image to be displayed; and a step of processing the first display information and the second display information based on the preset information to obtain third display information of the target image.

[0057] For example, the preset information can be carried in a bitstream and sent to an end corresponding to an image processing method, so that when a backlight exists on the display, the display adaptation of the video or image on the display can be performed based on the preset information, and the display effect is improved.

[0058] The preset information is carried in the source information of the target image, so that the image display end can directly use the preset information to perform display adaptation of the image under the backlight of the display, and the image display end does not need to generate the preset information, so that the display delay of the image or video can be reduced.

[0059] According to a second aspect, an embodiment of the present application provides an electronic device. The electronic device comprises a memory and a processor. The memory is coupled to the processor. The memory stores program instructions. When the program instructions are executed by the processor, the electronic device is capable of implementing the method according to the first aspect and any one of the implementations of the first aspect.

[0060] The second aspect and any one of the implementations of the second aspect may correspond to the first aspect and any one of the implementations of the first aspect, respectively. For the technical effects corresponding to the second aspect and any one of the implementations of the second aspect, please refer to the technical effects corresponding to the first aspect and any one of the implementations of the first aspect. Details will not be described again here.

[0061] According to a third aspect, an embodiment of the present application provides a chip. The chip includes one or more interface circuits and one or more processors. The interface circuits are configured to receive signals from a memory of an electronic device and transmit signals to the processor. The signals include computer instructions stored in the memory. When the processor executes the computer instructions, the electronic device is capable of performing a method according to the first aspect and any one of the implementations of the first aspect.

[0062] The third aspect and any one of the implementations of the third aspect may correspond to the first aspect and any one of the implementations of the first aspect, respectively. For the technical effects corresponding to the third aspect and any one of the implementations of the third aspect, please refer to the technical effects corresponding to the first aspect and any one of the implementations of the first aspect. Details will not be described again here.

[0063] According to a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing a computer program, the computer program being capable of causing the computer or processor, when running on the computer or processor, to perform the method according to the first aspect or any one of the possible implementations of the first aspect.

[0064] The fourth aspect and any one of the implementations of the fourth aspect may correspond to the first aspect and any one of the implementations of the first aspect, respectively. For the technical effects corresponding to the fourth aspect and any one of the implementations of the fourth aspect, please refer to the technical effects corresponding to the first aspect and any one of the implementations of the first aspect. Details will not be described again here.

[0065] According to a fifth aspect, an embodiment of the present application provides a computer program product, comprising a software program, which, when executed by a computer or processor, performs the method according to the first aspect or any one of the possible implementations of the first aspect.

[0066] The fifth aspect and any one of the implementations of the fifth aspect may correspond to the first aspect and any one of the implementations of the first aspect, respectively. For the technical effects corresponding to the fifth aspect and any one of the implementations of the fifth aspect, please refer to the technical effects corresponding to the first aspect and any one of the implementations of the first aspect. Details will not be described again here. [Brief description of the drawings]

[0067] In order to more clearly describe the technical solutions in the embodiments of the present application, the following briefly describes the accompanying drawings used in describing the embodiments of the present application. It is clear that the accompanying drawings in the following description merely show some embodiments of the present application, and those skilled in the art may still derive other drawings from these accompanying drawings without creative efforts.

[0068] [Figure 1] 1 is an example of a system architecture diagram.

[0069] [Diagram 2] 1 is an example of a system application architecture diagram.

[0070] [Diagram 3]1 is a diagram showing an example of a mapping relationship in a dynamic range.

[0071] [Figure 4] 1 is a diagrammatic example of a scenario in which the backlight brightness of a mobile phone is adjusted.

[0072] [Figure 5a] 1 is an example of a system architecture diagram.

[0073] [Figure 5b] 1 is a diagram of an example of an image processing process.

[0074] [Figure 5c] 1 is an example diagram of a data generation process.

[0075] [Figure 5d] 1 is a diagram of an example image display adaptation process.

[0076] [Figure 5e] 1 is a diagram of a curve generation process; FIG.

[0077] [Figure 6a] 1 is an example of a system architecture diagram.

[0078] [Figure 6b] 1 is a diagram of an example of an image processing process.

[0079] [Figure 6c] 1 is a diagram of an example image display adaptation process.

[0080] [Figure 7] FIG. 2 is a diagram of the structure of an apparatus according to an embodiment of the present application;

[0081] [Figure 8] FIG. 2 is a diagram of a chip structure according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0082] The following clearly and completely describes the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. It is clear that the described embodiments are only a part and not all of the embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts shall be included in the protection scope of the present application.

[0083] The term "and / or" herein describes only a relational relationship between related objects and indicates that a triple relationship may exist. For example, A and / or B may indicate that only A is present, that both A and B are present, and that only B is present.

[0084] In the description and claims of the embodiments of the present application, the terms "first," "second," and the like are intended to distinguish between different objects, and are not used to describe a particular order of the objects. For example, a first target object, a second target object, and the like are intended to distinguish between different target objects, and are not used to describe a particular order of the target objects.

[0085] In the embodiments of the present application, the words "example," "for example," or the like are used to denote giving an example, illustration, or explanation. Any embodiment or design scheme described in the embodiments of the present application as an "example" or "for example" should not be described as being preferred or having more advantages over another embodiment or design scheme. Strictly, the use of the words "example," "for example," or the like is intended to present the related concept in a particular manner.

[0086] In describing the embodiments of the present application, unless otherwise specified, "plurality" means two or more than two. For example, a plurality of processing units means two or more processing units, and a plurality of systems means two or more systems.

[0087] An example of a diagram of the structure of a system framework is shown in Figure 1. The system in Figure 1 includes a transmitting end and a receiving end.

[0088] It should be understood that the system shown in Figure 1 is merely an example. The systems herein, and the transmitting and receiving ends of the systems, may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. The components shown in Figure 1 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processors and / or application specific integrated circuits.

[0089] For example, the transmitting end may be an image or video generating end, and the receiving end may be an image or video display end.

[0090] As shown in FIG. 2(1), the transmitting end and the receiving end may be configured in the same electronic device.

[0091] For example, the electronic device may be a mobile phone, and the transmitting end may include a camera and the receiving end may include a display.

[0092] The transmitting end may generate a bitstream based on video data collected by a camera, where the bitstream carries the backlight metadata, and the transmitting end may transmit the bitstream to a display at a receiving end for display.

[0093] For example, an application scenario of the system shown in FIG. 1 may be a process in which a user takes a video using a mobile phone camera and displays the taken video on the mobile phone display.

[0094] For example, the transmitting end may be integrated into an encoder, and the receiving end may be integrated into a decoder.

[0095] As shown in FIG. 2(2), the transmitting end and the receiving end may be implemented in different electronic devices.

[0096] For example, electronic device 1 is a mobile phone and electronic device 2 is a television. The mobile phone may transmit a bitstream of video data captured by the mobile phone to the television, such as by using a Wi-Fi network or through Bluetooth, and the video data may be displayed on the television's display.

[0097] For example, electronic device 1 is an electronic device used by a video software vendor to capture and create videos, electronic device 1 may send the created video bitstream (including backlight metadata) to an application server, and electronic device 2 is a mobile phone. The mobile phone receives a user operation of a user on a video application installed on the mobile phone. In response to the user operation, the mobile phone may obtain the video bitstream sent by electronic device 1 from the application server, and display and play the video data on the display of the mobile phone.

[0098] It should be noted that the electronic device may be a terminal or may be referred to as a terminal device. The terminal may be a device such as a cellular phone, a pad, a wearable device, a television, a personal computer (PC), or an Internet of Things device. This is not a limitation in this application.

[0099] In addition, the application scenarios of the present application are not limited to the above-mentioned example scenarios, and the present application can further be applied to multiple scenarios, such as a Huawei Cloud storage (or transmission) scenario of images (or videos), a video surveillance scenario, or a live broadcast scenario, which are not limited in the present application.

[0100] The following describes a specific operation process of the system in FIG. 1 with reference to FIG.

[0101] The transmitting end may include a collecting module, an optical-to-electrical converting module and an encoding module.

[0102] The collection module may collect the ambient light to obtain image data of the light signal.

[0103] The optical-to-electrical conversion module may perform optical-to-electrical conversion on the image data of the optical signal according to an optical-to-electrical transfer function to generate the image data of the electrical signal.

[0104] The following provides background on the optical-to-electrical conversion performed by the optical-to-electrical conversion module.

[0105] For example, dynamic range may refer to the ratio of the maximum value to the minimum value of a variable. In a digital image, dynamic range may refer to the ratio of the maximum grayscale value to the minimum grayscale value in the range in which the image can be displayed.

[0106] The dynamic range of nature is very large. As shown in Figure 3, the luminance information of various ambient lights is shown. For example, the luminance of a night scene in a starry sky is about 0.001 cd / m 2 and the luminance of the sun itself is 1,000,000,000 cd / m 2 The brightness of moonlight is about 1cd / m 2 , and the ambient light luminance in an indoor lighting scenario is about 100cd / m 2 , and the ambient light luminance in an outdoor cloudy scenario is about 500cd / m 2 and the ambient light luminance in an outdoor sunny scenario is about 2000cd / m 2 In this case, the natural dynamic range is 1,000,000,000 / 0.001=10 13However, in natural real-world scenarios, the luminance of the sun and the luminance of the starlight are not captured simultaneously. For real-world natural scenarios, the dynamic range can reach an order of magnitude of 10 -3 ~10 6 is within the range.

[0107] However, in most current color digital images, each of the RGB (Red Green Blue) channels is stored separately by using 8 bits or 10 bits, that is, the representation range of each channel is a grayscale of 0 to 255, where 0 to 255 is the dynamic range of the image.

[0108] In this case, the dynamic range in the same real-world scenario is 10 -3 ~10 6 , which may be referred to as high dynamic range (HDR). In contrast, the dynamic range of a typical picture may be referred to as low dynamic range (LDR). As shown in FIG. 3, the imaging process of an electronic device (e.g., a digital camera) is limited to the real-world high dynamic range (e.g., 100 cd / m 2 ~2000cd / m 2 ) to the low dynamic range of the image (1cd / m 2 ~500cd / m 2 ) can be understood as a nonlinear mapping to

[0109] It should be noted that the range of mapping from the luminance of the ambient light to the display luminance of the display device is not limited to the example of the mapping range of the shadow area in Fig. 3. As the display capabilities of electronic devices improve, the range of luminance that can be displayed by the display device may be increased. For example, the maximum display luminance may be greater than 500.

[0110] For example, the display device in FIG. 3 and herein may be the electronic device to which the receiving end in this embodiment of the present application belongs.

[0111] Based on the above background, and referring to FIG. 1 and FIG. 3, the dynamic range of the image data of the optical signal collected by the collection module is 10 -3 ~10 6 It can be seen that the maximum display brightness of the display module at the receiving end is, for example, only 500, and cannot reach the brightness information of the ambient light in the real world. In order to view an image by using a display device, an optical-electrical conversion module needs to convert the optical signal into an electrical signal by using an optical-electrical transfer function to obtain an 8-bit or 10-bit image (the size of each pixel in the image is 8 bits or 10 bits).

[0112] For example, a standard dynamic range (SDR) image (an image whose dynamic range is generally between 1 nit and 100 nit, where a nit is a unit of light) corresponds to a high dynamic range (HDR) image (an image whose dynamic range is between 0.001 and 10000 nit). An 8-bit image in a traditional format such as JPEG may be considered an SDR image.

[0113] It should be noted that the image format of the 8-bit or 10-bit image of the electrical signal can be any one of RAW (unprocessed) image, RGB image, YUV (where "Y" stands for Luminance, Luma, and "U" and "V" stand for Chrominance, Chroma) image, and Lab image, which is not limited in this application.

[0114] A Lab image contains three components: one component is luminance (L), and a and b are two color channels: a contains colors ranging from dark green (low luminance values) to gray (medium luminance values) to light pink (high luminance values); and b contains colors ranging from light blue (low luminance values) to gray (medium luminance values) to yellow (high luminance values).

[0115] In addition, in this application, the size of each pixel in an image displayed by a display device is not limited to 8 bits or 10 bits. With the improvement of the display capability of the display device, the image pixel may alternatively be displayed with a size larger than 10 bits.

[0116] The following describes the optical-to-electrical transfer function used by the optical-to-electrical transfer module in FIG.

[0117] For example, an early display device may be a CRT display, and the optical-electrical transfer function of a CRT display is the gamma function. The optical-electrical transfer function shown in formula (1) is defined in the ITU-R Recommendation BT.1886 standard:

number

[0118] Based on Equation (1), after optical-to-electrical conversion is performed on the collected optical signal, the obtained image quantized to an 8-bit image can be called an SDR image. The SDR image and the optical-to-electrical transfer function in Equation (1) are compared with the conventional display device (illuminance is about 100 cd / m 2 It performs well on

[0119] However, with the upgrade of display devices, the illuminance range of display devices continues to increase. The illuminance information of a typical HDR display is 600cd / m 2 and the illuminance information of an advanced HDR display may be 2000cd / m 2, which is much larger than the illuminance information of an SDR display device. The optical-electrical transfer function shown in formula (1) in the ITU-R Recommendation BT.1886 standard cannot fully express the display performance of an HDR display device. Therefore, an improved electrical-optical transfer function needs to be adapted for the upgrade of the display device. The idea of ​​the optical-electrical transfer function comes from the mapping function in the tone mapping (TM) algorithm. The optical-electrical transfer function is obtained by appropriately adjusting the mapping function. At present, there are three common optical-electrical transfer functions: PQ (perception quantization), HLG (hybrid log-gamma) and SLF optical-electrical transfer functions. The three optical-electrical transfer functions are conversion functions specified in the AVS standard.

[0120] In the following, the curves of the three optical-to-electrical transfer functions mentioned above are explained.

[0121] The PQ optical-electrical transfer function here is different from the conventional "gamma" optical-electrical transfer function shown in Equation (1), and a perceptual quantization transfer function can be proposed based on the luminance perception model of the human eye. The PQ optical-electrical transfer function represents the conversion relationship between the linear signal value of the image pixel and the nonlinear signal value in the PQ domain, as shown in Equation (2), and the PQ optical-electrical transfer function can be expressed as Equation (3):

number

number

number

number

number

number

number

[0122] The HLG optical-electrical transfer function is obtained by improving the traditional gamma curve. The HLG optical-electrical transfer function uses the traditional gamma curve in the segment with a small horizontal coordinate, and a log curve is added to the segment with a large horizontal coordinate. The HLG optical-electrical transfer function indicates the conversion relationship between the linear signal value of the image pixel and the nonlinear signal value in the HLG domain, and the HLG optical-electrical transfer function can be expressed as Equation (4):

number

[0123] The SLF optical-to-electrical transfer function is an optimal curve obtained based on the luminance distribution in an HDR scene when the optical characteristics of the human eye are satisfied.

[0124] The SLF optical-electrical transfer curve shows the conversion relationship between the linear signal value of an image pixel and the nonlinear signal value in the SLF domain. The conversion relationship between the linear signal value of an image pixel and the nonlinear signal value in the SLF domain is expressed in Equation (5):

number

number

[0125] Still referring to FIG. 1, the encoding module at the transmitting end may receive the electrical signal image generated by the optical-to-electrical conversion module, and encode the electrical signal image to generate a bit stream.

[0126] It should be noted that the bitstream generated by the sending end in this application may carry backlight metadata. When the sending end in this embodiment of the application generates an image or video, it may generate backlight metadata for the image or video.

[0127] For example, the backlight metadata may be video or image attribute information.

[0128] When the backlight of the display device changes, the backlight metadata indicates metadata regarding the display adaptation of the image.

[0129] Still referring to FIG. 1, the transmitting end may transmit the generated bit stream to the receiving end.

[0130] For example, the image processed by the system shown in Fig. 1 in this embodiment of the present application may be an image of one frame, or may be an image of a frame sequence (i.e., an image of multiple frames), which is not a limitation of the present application.

[0131] Still referring to FIG. 1, the receiving end may include a decoding module, an electrical-to-optical conversion module, a display adaptation module, and a display module.

[0132] The decoding module may decode the bitstream to obtain, for example, an 8-bit electrical signal image and image attribute information (which may include backlight metadata).

[0133] The operation performed by the electrical-optical conversion module is the reverse process of the operation performed by the optical-electrical conversion module, i.e., the electrical-optical conversion module may convert the received electrical signal image into an optical signal image corresponding to the light intensity of the ambient light of the image.

[0134] For example, the dynamic range mapping method can be applied to the matching between the image (HDR signal) generated by the transmitting end and the HDR signal displayed by the display device to which the receiving end belongs.

[0135] For example, when the transmitting end generates an image, the collected ambient light is a 4000nit optical signal, but the maximum display brightness (i.e., HDR display capability) of each pixel in the display device (e.g., TV or pad) to which the receiving end belongs is only 500nit. In this case, tone mapping needs to be performed from the image brightness to the brightness of the display of the display device to display the 4000nit optical signal on the display of the display device.

[0136] In another example, when the transmitting end generates an image, a 100 nit SDR signal is collected, but the maximum display brightness of the display device (e.g., a TV) to which the receiving end belongs is 2000 nit. In this case, tone mapping from the image brightness to the display brightness of the display device also needs to be performed to map and display the 100 nit signal on the 2000 nit display device.

[0137] In the prior art, the bitstream generated by the transmitting end does not include the backlight metadata in this embodiment of the present application. Current tone mapping curves may include a sigmoid curve-based adjustment process proposed by Dolby and a Bezier curve. The two tone mapping curve techniques correspond to the ST2094 standard. When the display pixels of an image are mapped to the display luminance of the display pixels on a display, the maximum luminance value and the minimum luminance value of the image are usually aligned with the maximum display luminance value (i.e., the display capability, e.g., 500nit) and the minimum display luminance value of the display. The luminance values ​​between the maximum luminance value and the minimum luminance value of the image can be mapped to a range between the maximum display luminance value and the minimum display luminance value of the display based on a global or local tone mapping curve (e.g., a sigmoid curve-based adjustment process and a Bezier curve).

[0138] However, in the above-mentioned tone mapping solutions in the prior art, the backlight adjustable scenario of the display device is not taken into account.

[0139] For example, a mobile phone display may include an array of backlight sources, which may be used to light up the screen of the display. When the backlight intensity level is at its highest (e.g., 100%), the brightness of the display is at its maximum display brightness (e.g., 500 nits), and the backlight intensity level may be described as a percentage of the maximum display brightness. In this case, the product of the backlight intensity level and the maximum display brightness may be the current target backlight brightness of the display.

[0140] It should be noted that the maximum display brightness of the display device is not limited in this application and can be continuously improved based on the product upgrade of the display device. In addition, the layout manner of the backlight light source of the display device is also not limited in this application, and the backlight light sources of different types of electronic devices (such as mobile phones and televisions) can be arranged in different manners.

[0141] FIG. 4 is a diagram example of a scenario in which the backlight brightness of a mobile phone is adjusted.

[0142] Example 1

[0143] As shown in Figures 4(1) and 4(2), the mobile phone may include an ambient light sensor 101. The ambient light sensor 101 may sense ambient luminance values ​​and identify ambient luminance change values. The mobile phone may set a current target backlight brightness of the mobile phone screen based on the ambient light luminance values ​​collected by the ambient light sensor 101, and adjust the current target backlight brightness based on the ambient luminance change collected by the ambient light sensor 101.

[0144] For example, the system may adjust the backlight brightness of a mobile phone screen based on changes in ambient light brightness, e.g., the backlight brightness set on the mobile phone during the day is higher than the backlight brightness set on the mobile phone at night.

[0145] Example 2:

[0146] As shown in FIG. 4(1), a mobile phone displays an interface 100, and a user may perform a downward sliding action on interface 100 from the screen area near the right side, following the arrow direction from the top of the screen.

[0147] The interface 100 is not limited in this application and may be an application interface, an application icon interface, or the like.

[0148] In response to a user's sliding operation, the mobile phone can switch the display interface of the mobile phone from the interface 100 shown in FIG. 4(1) to the control center interface 102 shown in FIG. 4(2).

[0149] The control center interface 102 includes one or more controls, and implementations of the controls may include, but are not limited to, icons, buttons, windows, layers, and the like.

[0150] As shown in FIG. 4(2), the control center interface 102 may include a “Huawei Video” playback window, a wireless network icon, a Bluetooth icon, and a window 103.

[0151] Window 103 may include a mobile data icon, a ring / mute icon, and an auto-rotate screen icon.

[0152] Additionally, window 103 may further include a brightness progress bar control that is used to control the backlight brightness of the mobile phone screen.

[0153] It should be noted that window 103 may include more or fewer controls, and this is not a limitation of this application.

[0154] The brightness progress bar control may include an icon 104 , an icon 105 , a progress bar 106 , and a control 107 .

[0155] Icon 104 indicates minimum backlight brightness, ie, a backlight intensity level of 0%.

[0156] Icon 105 indicates maximum backlight brightness, ie, the backlight intensity level is 100%, in which case the display brightness of the mobile phone screen is the maximum display brightness of the mobile phone screen (eg, the above-mentioned 500 nit).

[0157] The user's finger moves the control 107 on the progress bar 106, and the mobile phone responds to the user's action by adjusting the current backlight intensity level of the mobile phone screen, i.e., adjusting the backlight intensity level in the range of 0% to 100%, thereby changing the current target backlight brightness of the mobile phone.

[0158] For example, control 107 may be moved on progress bar 106 towards icon 105, thereby increasing the backlight intensity level of the mobile phone screen and increasing the current target backlight brightness, and control 107 may be moved on progress bar 106 towards icon 104, thereby decreasing the backlight intensity level of the mobile phone screen and decreasing the current target backlight brightness.

[0159] In this embodiment, a user may set the current target backlight brightness of the mobile phone screen based on usage requirements and actual mobile phone usage situations.

[0160] It should be noted that the interface provided by the mobile phone for the user to manually adjust the current target backlight brightness of the screen is not limited to (2) in FIG. 4, and the backlight brightness may alternatively be adjusted by using another interface.

[0161] In this case, in a scenario where the screen backlight brightness is adjustable, the final display brightness of the screen may be the result of the joint action of the current target backlight brightness value and the screen display value. During the backlight brightness adjustment, the screen brightness range changes accordingly.

[0162] For example, an OLED screen is self-luminous. During backlight control, the brightness of the excitation light source of each pixel in the OLED screen can be directly controlled, and the backlight intensity can indicate the overall change of the screen display value.

[0163] For example, in a scenario where the backlight of a mobile phone is adjustable, for example, if the maximum display brightness (i.e., display capability) of the mobile phone screen is 500 nit and the backlight intensity level is 10%, the current target backlight brightness is 50 nit. However, since the maximum display brightness of the mobile phone screen can reach 500 nit, when the current target backlight brightness is 50 nit, the actual brightness of the screen (current backlight brightness, i.e., actual backlight brightness) may be higher than 50 nit, for example, 60 nit or 100 nit. There may be a strategy for configuring the actual brightness of the screen.

[0164] Therefore, based on the above description that the backlight brightness of the display of the electronic device is adjustable, it can be seen that in either the global tone mapping solution or the local tone mapping solution in the prior art, at a certain backlight intensity, the final display brightness of the screen can be increased from the current target backlight brightness (e.g., 50 nit as mentioned above) to the brightness until the highest level of the actual backlight of a single pixel or several pixels of the screen is reached (i.e., the maximum display brightness as mentioned above, e.g., 100 nit). In this case, in the prior art solution, the mechanism in which only the maximum and minimum brightness values ​​of the image are aligned to the maximum and minimum brightness values ​​of the screen is not adapted to the display adaptation when the backlight is weakened (i.e., the backlight is weakened from the strongest backlight and the backlight intensity level starts to decrease from 100%). As a result, the contrast of the video or image displayed by the electronic device is low and the colors are dark.

[0165] Therefore, the present application provides the above-mentioned system, in which, referring to Fig. 1, a bitstream sent by a sending end in the system can carry backlight metadata, and a display adaptation module at a receiving end can perform dynamic range mapping on a light signal image according to the backlight metadata to obtain image display information, and send the image display information to a display module.

[0166] The CPU of the receiving end can control the circuitry of the display module based on the image display information, and generate light of corresponding brightness by using the pixels in the display module, thereby implementing matching between the image generated by the transmitting end (high HDR signal) and the low HDR signal displayed by the display device to which the receiving end belongs.

[0167] In the system of this embodiment of the present application, in scenarios where the backlight is adjustable, images can be adaptively displayed on the display device.

[0168] Embodiment 1:

[0169] Regarding the backlight metadata carried in the bitstream in FIG. 1, in embodiment 1, the backlight metadata may include local or global initial gain adjustment curves and local or global anchors.

[0170] "Global" may refer to a single frame of an image or a sequence of frames.

[0171] "Local" may refer to an image region in a frame of an image, an image scene, or the like.

[0172] The backlight metadata may include at least one of the following data: a local initial gain adjustment curve and a local anchor; a global initial gain adjustment curve and a global anchor; a global initial gain adjustment curve and a local anchor; or a local initial gain adjustment curve and a global anchor.

[0173] The anchors in this specification are luminance values.

[0174] A local anchor of a local image may be a luminance value that is less than the maximum pixel luminance value of the local image (eg, an image region) and greater than the minimum pixel luminance value of the local image.

[0175] A global anchor, ie, a backlight control anchor, is a luminance value less than the maximum global (eg, one frame image) pixel luminance and greater than the minimum global pixel luminance.

[0176] When an initial gain adjustment curve is generated for a local or global image, a tone mapping curve can be generated in a conventional manner and used as the initial gain adjustment curve for the image, which is not a limitation in this application.

[0177] For example, tone mapping curves may have a variety of forms including, but not limited to, sigmoid, cubic spline, gamma, and piecewise cubic spline functions, each of which may be used as an initial gain adjustment curve.

[0178] Figure 5a is an example of a system architecture diagram obtained after improving Figure 1. The similarities between Figure 5a and Figure 1 will not be described again here, and only the improved parts will be described in detail.

[0179] It should be understood that the system shown in Fig. 5a is merely an example. The systems herein, and the transmitting and receiving ends thereof, may have more or less components than those shown in the figures, may combine two or more components, or may have different component configurations. The components shown in Fig. 5a may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processors and / or application specific integrated circuits.

[0180] As shown in FIG. 5a, the sending end may include a backlight metadata generating module.

[0181] The backlight metadata generating module may generate backlight metadata based on the electrical signal image or image sequence generated by the optical-electrical conversion module, where the backlight metadata may include local or global initial gain adjustment curves and local or global anchors (see above for specific definitions, and will not be described in detail again here).

[0182] The encoding module may encode the electrical signal image and the backlight metadata, generate a bitstream, and transmit the bitstream to a receiving end.

[0183] The following describes the process by which the backlight metadata generation module generates local or global initial gain adjustment curves and local or global anchors.

[0184] 5a and 5b, an example of a process for generating an initial gain adjustment curve 0 and an anchor for region A of image 1 is shown. In Fig. 5b, an example in which the processing object is image 1 is used for explanation.

[0185] In FIG. 5b, the backlight metadata generation module may perform region segmentation on the image 1 by dividing pixels of the image 1 to be displayed that have similar luminance and spatial location into identical regions, thereby obtaining multiple regions C; and then identify a highlight region C in the multiple regions C.

[0186] It should be noted that when region segmentation is performed on image 1, the segmentation method is not limited to the example of FIG. 5b, and pixels in an image that are close in luminance and close in spatial location may alternatively be segmented into a region in another manner to implement image region segmentation.

[0187] For example, the process in FIG. 5b may include the following steps.

[0188] Optionally, a first segmentation is performed on image 1 .

[0189] For example, image 1 may be divided into one region (i.e., no region segmentation is performed). In FIG. 5b, image 1 is region A.

[0190] Alternatively, for example, image 1 may be divided into multiple rectangular regions and multiple regions A may be acquired.

[0191] S400: The backlight metadata generation module may generate an initial gain adjustment curve 0 for region A.

[0192] For example, image 1 is a 64x64 image.

[0193] The initial gain adjustment curve 0 can be generated in any manner that generates a tone mapping curve in the prior art, which is not a limitation in this application.

[0194] When the initial gain adjustment curves in the backlight metadata include a local initial gain adjustment curve, an initial gain adjustment curve for an image region may be generated for the image region based on the methods described above.

[0195] When the initial gain adjustment curves in the backlight metadata include a global initial gain adjustment curve, and "global" means a frame of an image, the initial gain adjustment curve may be generated for a frame of an image. When "global" means a frame sequence, the initial gain adjustment curve may be generated for a frame of an image in the frame sequence, and the initial gain adjustment curve is used as the initial gain adjustment curve for the frame sequence.

[0196] S401: The backlight metadata generation module performs uniform image division on region A to generate region B.

[0197] For example, region A may be divided into four equally divided regions B, and the size of each region B is 32x32.

[0198] It should be noted that the image may or may not be divided evenly, this is not a limitation of this application.

[0199] Here, region A is divided into four regions B. This is equivalent to performing initial clustering on the pixels in region A. For example, the four regions B may be referred to as type 0, type 1, type 2, and type 3, respectively.

[0200] S402: The backlight metadata generation module performs iterative clustering on the pixels in region A based on the pixel's luminance and spatial location.

[0201] At this stage, an iteratively updated clustering may be performed on the pixels in four regions B in region A to cluster the pixels in region A that are close in luminance and close in spatial location into one type, thereby forming four new cluster types, which are the four regions C in Fig. 5b, respectively. For example, the four regions C may be called Type 0', Type 1', Type 2' and Type 3'.

[0202] For example, for each region B in one region A, in a first step, the cluster center location (x_centra, y_centra) and average color value (e.g., three-channel RGB average values: R_avg, G_avg, and B_avg), or Lab average values ​​(l_avg, a_avg, and b_avg) of each region B may be calculated, where the cluster center location may be the geometric center of the pixel in region B.

[0203] Then, in the second step, for each pixel (coordinates (x,y) and color value RGB or lab) in one region A, the geometric distance between this pixel and each cluster center location of region B in region A, and the color difference between the color of this pixel and the color of the pixel at the cluster center location are calculated and used as the distortion value cost of the pixel in region A: cost = (√((x‐x_centra) 2 +(y‐y_centra) 2 )) / K+(√((l‐l_avg) 2 +(a-a_avg) 2 +(b-b_avg) 2 )) / M where √ denotes the dilatation operation, K and M are two constants, and K and M are used for normalization.

[0204] For example, in FIG. 5b, one region A may contain four regions B, and four distortion values ​​may be obtained for one pixel in region A.

[0205] Then, in the third stage, for the four distortion values ​​of each pixel in region A, the region B corresponding to the smallest distortion value is selected as the updated cluster type of the pixel (e.g., type 0). In this way, the iterative clustering can be implemented for all pixels in region A at once.

[0206] In this way, for every pixel in region A, an updated cluster type can be obtained through calculation, which is equivalent to performing region segmentation again on region A. The four regions B' obtained through the current segmentation are different from the four regions in regular rectangular shapes shown in FIG. 5b, and the shapes of the four regions obtained through the current segmentation may be irregular.

[0207] Then, based on the above-mentioned steps 1 to 3, the cluster center locations, average color values, and the like are recalculated for the pixels in the regions corresponding to the four updated cluster types, and the next round of iterative clustering is performed until the cluster types of the pixels in region A no longer change, and the four regions C shown in FIG. 5b can be obtained (as the four final cluster types).

[0208] In this way, each region A in image 1 can be divided into four regions C.

[0209] It should be noted that the number of final regions C generated when region A is divided is not limited to 4. For example, when the image is divided evenly in S401, any number of equal division schemes such as 6 or 9 may alternatively be performed, so that the number of regions C obtained by dividing region A may be 6, 9, or the like.

[0210] It should be noted that in the image 1 to be displayed, the method of dividing pixels having similar luminance and similar spatial positions into the same region is not limited to the above-mentioned solution example, and the region division for the image 1 can alternatively be implemented by using another method in the art.

[0211] S403: The backlight metadata generation module identifies highlight regions from the segmented regions of the image by using a preset algorithm.

[0212] In a possible implementation, for example, the backlight metadata generation module may perform a filtering process on the image 1 (e.g., including the four regions C shown in FIG. 5b) by using a bilateral filtering method or a guided filtering method, and based on the difference between pixel values ​​of the same pixel in the image 1 before and after filtering, use the pixels with large differences in pixel values ​​in the image as highlighted pixels. Then, based on the four regions C obtained by dividing the image 1, the region C containing highlighted pixels whose number is greater than or equal to a preset threshold is used as the highlighted region C, and the region C containing highlighted pixels whose number is less than the preset threshold is used as the non-highlighted region C. In this way, the regions included in the image 1 may be divided into two types of regions, one type being the highlighted region, and the other type being the non-highlighted region.

[0213] For example, as shown in FIG. 5b, the four regions C of region A include two highlighted regions C and two non-highlighted regions C.

[0214] For example, attributes of the corresponding photographed object in image 1 may affect whether an image region is a highlight region. For example, a cloud layer or glass has high reflectivity, and image regions of a cloud layer or glass are easily used as highlight regions. In another example, image regions of objects with low reflectivity, such as coarse cotton and linen clothing, are easily used as non-highlight regions.

[0215] It should be noted that the method for identifying which regions in the divided regions of the image belong to the highlight region and which regions belong to the non-highlight region is not limited to the above-mentioned bilateral filtering or guide filtering method, and can alternatively be another method. For example, the average luminance value of pixels is calculated separately for different regions C in region A, and the region C whose average luminance value is greater than a preset threshold is used as the highlight region C, thereby classifying the highlight region and the non-highlight region of region A. This is not limited in the present application.

[0216] In addition, it should be noted that when the image 1 is divided into multiple regions A, for different regions A, the highlight regions and non-highlight regions in each region A may be implemented in the same or different implementations, which is not a limitation in this application.

[0217] S405: The backlight metadata generation module determines a luminance value to be used as an anchor for region A based on the pixel luminance of highlight region C in region A.

[0218] For example, for one region A, a minimum pixel luminance value a1 and an average pixel luminance value a2 may be obtained for pixels in all highlight regions C in region A. For one region A, an anchor for region A is generated based on the minimum pixel luminance value a1 and the average pixel luminance value a2.

[0219] For example, any luminance value greater than a1 and less than a2 can be used as an anchor for region A.

[0220] For example, the average value of a1 and a2 can alternatively be used as an anchor for region A.

[0221] For example, the weighted result of a1 and a2 may alternatively be used as an anchor for region A.

[0222] One area A has one anchor.

[0223] Optionally, for one region A, the minimum pixel luminance value a1 and the average pixel luminance value a2 may alternatively be obtained for all pixels in region C in region A (i.e., all pixels in region A). For one region A, an anchor for region A is generated based on the minimum pixel luminance value a1 and the average pixel luminance value a2.

[0224] It should be noted that the manner of generating anchors in this application is not limited to the manner in the above embodiment, and anchors may alternatively be generated in other manners, which is not limited in this application.

[0225] Therefore, based on the method in the above embodiment, a global anchor for a frame of an image can be generated for the frame of the image, and a local anchor can also be generated for a region in the image. In addition, when the global anchor in the backlight metadata is an anchor for a frame sequence, by using the above method, an anchor can be generated for a frame of an image, and then the anchor is used as an anchor for a frame sequence including the frame image.

[0226] FIG. 5b illustrates the process of generating the initial gain adjustment curves and anchors in an implementation.

[0227] In another implementation, the local or global initial gain adjustment curves and the local or global anchors may alternatively be generated based on the procedure in FIG. 5c(1).

[0228] As shown in FIG. 5c(1), the process may include the following steps.

[0229] For example, Fig. 5c(1) may be performed by the backlight metadata generation module in Fig. 5a. Region A in Fig. 5c(1) may be an image of one frame, or may be a local image region in an image, which is not limited in this application.

[0230] S501: An initial gain adjustment curve 0 is generated for region A.

[0231] This process is similar to S400 in FIG. 5b, and the details will not be described again here.

[0232] S502: A cumulative distribution histogram of the luminance of pixels in region A is generated based on the luminance values ​​of the pixels in region A.

[0233] For example, FIG. 5c(2) shows a cumulative distribution histogram of pixel luminance in region A.

[0234] In the histogram, the horizontal axis x represents the luminance values ​​of the pixels in region A, and the vertical axis y represents the number of pixels in region A.

[0235] For example, it can be seen from (2) of FIG. 5c that area A contains 100 pixels, the number of pixels in area A having brightness x1 is 20, the number of pixels having brightness x2 is 30, and the number of pixels having brightness x3 is 50.

[0236] S503: Based on the histogram, a luminance value in region A and higher than the luminance of a preset ratio of pixels within region A is used as an anchor for region A.

[0237] For example, the preset ratio may be 95%, 99%, 99.5% or the like. The preset ratio is greater than 50%. The specific value may be flexibly configured based on requirements. This is not limited in the present application.

[0238] For example, the preset ratio is 95%. Based on the histogram shown in FIG. 5c(2), it can be easily determined that the luminance value x3 is in region A and is higher than the luminance of 95% of the pixels in region A, and x3 can be used as the anchor of region A.

[0239] In the implementation of Fig. 5c, when local or global anchors are generated for an image, segmentation does not need to be performed on the image, and the required anchors can be determined based only on the cumulative distribution histogram of the brightness of the pixels of the image. Considering that the brightness of pixels whose brightness is higher than the anchor needs to be increased, and the brightness of pixels whose brightness is lower than the anchor needs to be decreased, based on a preset ratio, it can be determined that the brightness of the pixels of the preset ratio needs to be decreased, and the brightness of the pixels of (1-preset ratio) needs to be increased, and the magnitude of brightening and darkening can be easily determined.

[0240] It should be noted that generating an initial gain adjustment curve and generating anchors for an image, a local image, or a frame sequence are two independent processes, and the order of execution of the two processes is not limited in this application.

[0241] It should be noted that S503 is executed after S502. However, the order of executing S501, S502, and S503 is not limited in the present application.

[0242] In the above implementation, the example in FIG. 5a in which the transmitting end generates a local or global initial gain adjustment curve and a local or global anchor is used for illustration, so that information about the curve and anchor can be transmitted to the receiving end as attributes of the bitstream.

[0243] It should be noted that the receiving end in this embodiment of the present application may also include a backlight metadata generating module for performing the above-mentioned operations, and generate local or global initial gain adjustment curves and local or global anchors. The method is similar, and the details are not described again here.

[0244] Returning now to Fig. 5a, the execution process of the receiving end in Fig. 5a will be described below with reference to Fig. 5d.

[0245] Through comparison between FIG. 1 and FIG. 5a, it can be seen that the display adaptation module in FIG. 1 may include a tone mapping curve generation module and a tone mapping module shown in FIG. 5a.

[0246] With reference to FIG. 5d, the following describes the operation process of the tone mapping curve generation module and the tone mapping module in FIG. 5a.

[0247] As shown in FIG. 5d, the process may include the following steps.

[0248] S201: The receiving end obtains a picture 1 to be displayed in a bitstream.

[0249] For example, as shown in FIG. 5a, the decoding module may obtain a picture, such as picture 1 in FIG. 5b, by decoding the bitstream.

[0250] S202: The receiving end obtains backlight metadata in a bitstream, where the backlight metadata includes an initial gain adjustment curve 0 of area A in image 1 and a first luminance of area A (also referred to as an anchor).

[0251] For example, as shown in FIG. 5a, the decoding module may decode the bitstream to obtain backlight metadata, where the backlight metadata may include local or global initial gain adjustment curves and local or global anchors of the bitstream.

[0252] For example, image 1 in Fig. 5b is a display target image in a bitstream. The decoding module may obtain an initial gain adjustment curve of region A and an anchor of region A in image 1, and send the initial gain adjustment curve and anchor of region A to the tone mapping curve generation module.

[0253] S203: The receiving end obtains the maximum display brightness and the current backlight intensity level of the display.

[0254] For example, as shown in FIG. 5a, the tone mapping curve generation module may obtain the maximum display luminance of the display and the current backlight intensity information of the display from the display module.

[0255] For example, the current backlight intensity information may be a current backlight intensity level, or may be a current target backlight luminance.

[0256] For example, the receiving end may obtain the maximum display brightness of the display (eg, 500 nits) from the display device to which the receiving end belongs.

[0257] For example, in this embodiment of the present application, a test sample (test image) can be further displayed on the display of a display device. An area in the test sample that occupies N% (for example, 5% or 10%) of the area of ​​the test sample is white, and the other area is black. The luminance of the white area with N% of the area is measured using an instrument, and the luminance obtained through the measurement is the maximum display luminance of the display.

[0258] It should be noted that the manner of obtaining the maximum display luminance of a display device is not limited in this application.

[0259] In addition, based on the description of the embodiment in Figure 4, the current backlight intensity information obtained by the tone mapping curve generation module and belonging to the display may be the current target backlight luminance obtained after automatic adjustment by the system and belonging to the display, or may be the current target backlight luminance obtained after manual adjustment by the user and belonging to the display. Details will not be described again here.

[0260] S204: The receiving end processes the curve 0 based on the maximum display luminance to generate the curve 1.

[0261] It should be noted that when curve 0 of region A is processed to generate curve 1, the luminance value based on which the processing is performed is not limited to the maximum display luminance. The luminance value may be a luminance value greater than the current target backlight luminance and less than or equal to the maximum display luminance of the display (e.g., a weighted result of the current target backlight luminance and the maximum display luminance).

[0262] For example, an example is used for explanation in which curve 1 is generated by processing curve 0 based on the maximum display luminance. When the luminance value based on which the processing is performed is not the maximum display luminance, the method is similar, and the details are not described again here.

[0263] Curve 0 can be any tone mapping curve in the prior art. In this case, curve 1 and the following curve 2 are also tone mapping curves. The independent variable x of the function f(x) of any tone mapping curve represents the luminance value of an image pixel of an SDR or HDR source, and f(x) represents the luminance value of the image pixel displayed on a display.

[0264] For example, as shown in FIG. 5a, the tone mapping curve generation module may process, for example, an initial gain adjustment curve (e.g., curve 0, which may also be referred to as curve 0) of area A based on a maximum display luminance maxT to generate an updated gain adjustment curve (e.g., curve 1, which may also be referred to as curve 1).

[0265] For example, a scaling process may be performed on curve 0 of area A based on the maximum display luminance to obtain curve 1. It should be noted that the processing method is not limited to the scaling process described in method 1 and method 2, and may further include method 3, method 4, method 5, and other methods not listed, which is not limited in this application.

[0266] Method 1

[0267] For example, if the maximum value of x for curve 0 is maxsource and the corresponding value of y is maxY, then the overall scaling factor of curve 0 is D=maxT / maxY, where maxT is the maximum display luminance of the display of the display device to which the receiving end belongs. Curve 0 may be scaled as a whole based on a scaling factor D, and curve 1=D*curve 0.

[0268] Curve 0 is the initial gain adjustment curve (tone mapping curve) for region A. In this case, the maximum value of x of curve 0 represents the maximum luminance value of the image pixels in region A here, and the corresponding value of y (i.e., maxY) represents the luminance value that is that of the image pixel having the maximum luminance value in region A and that is displayed on the display. In this case, the maximum value of x of curve 0 is the maximum luminance value of the image pixels in region A here.

[0269] Method 2:

[0270] The backlight metadata may include a preset brightness value T3.

[0271] For example, the preset brightness value T3 may be an arbitrary target brightness target_maxdisplay originally generated by curve 0, where the target brightness is also a brightness value of an image pixel having a particular brightness and displayed on the display, e.g., a maximum display brightness value.

[0272] The overall scaling factor for curve 0 is D=maxT / T3. Curve 0 is scaled overall based on a scaling factor D, where curve 1=D*curve 0.

[0273] Consider that when a display has a certain backlight, the actual luminance of the display may be higher than the backlight luminance. For example, if the maximum display luminance of the display is 500 nit and the backlight intensity level is 10%, the backlight luminance is 50 nit. However, the electronic device may have a way to adjust the final luminance of the display so that the final luminance may be higher than 50 nit. Therefore, in this embodiment, a tone mapping curve (i.e., curve 1) that matches the luminance value may be generated based on a luminance value that is higher than the backlight luminance and lower than or equal to the maximum display luminance.

[0274] Method 3:

[0275] Curve A is obtained based on the image parameters and display luminance of region A.

[0276] For example, the image parameters may include, but are not limited to, a maximum luminance value of a pixel, a minimum luminance value of a pixel, or an average luminance value of a pixel.

[0277] For example, the display brightness may include the maximum display brightness or the current target backlight brightness of the display.

[0278] The image parameters and the display luminance are used as the curve parameters of a conventional tone mapping curve, thereby obtaining curve A.

[0279] A weighting value w is obtained based on the maximum display luminance and the target luminance target_maxdisplay.

[0280] For example, the difference between the maximum display luminance and the target luminance target_maxdisplay may be used as the weight w, but the manner of obtaining the weight w is not limited thereto.

[0281] A weighted sum is performed on curve A and curve 0 of region A based on the weights w to obtain curve 1.

[0282] Method 4:

[0283] A curve parameter adjustment value B1 is obtained based on the maximum display luminance and the curve adjustment parameter B.

[0284] For example, the backlight metadata may include curve parameters that indicate curve 0 and curve adjustment parameter B here.

[0285] For example, the value of the curve fitting parameter B is 0.2.

[0286] For example, the maximum display luminance of the display is 200 and the maximum display luminance corresponding to curve 0 and of the display is 500.

[0287] In this case, |(200-500)| / 100=3, and the curve parameter adjustment value B1 obtained by multiplying the curve adjustment parameter B (here 0.2) with the above result "3" is 0.6.

[0288] Curve 1 is obtained by convoluting curve parameter adjustment value B1 based on curve 0.

[0289] The curve parameter adjustment value B1 is superimposed (eg, an addition or subtraction operation) on each curve parameter of curve 0 to obtain each curve parameter of curve 1, and then curve 1 is further obtained.

[0290] Specifically, each curve parameter of curve 0 may be added or subtracted by 0.6 to obtain a new group of curve parameters, thereby obtaining curve 1.

[0291] Method 5:

[0292] A curve parameter adjustment value B1 is obtained based on the maximum display luminance and the curve adjustment parameter B.

[0293] For example, in Scheme 4, each curve parameter of Curve 0 is adjusted based on the curve parameter adjustment value B1. Here, for example, if the target parameter in Curve 0 has the greatest effect on the curve shape, the backlight metadata may include the curve adjustment parameter B of the target parameter of Curve 0. Specifically, only the target parameter of Curve 0 is adjusted in Scheme 4.

[0294] Curve 0 is scaled in either method 1 or method 2 above to obtain curve 00.

[0295] Here, during scaling, the scaling process may be performed on parameters of curve 0 other than the above-mentioned target parameters to obtain curve 00.

[0296] Curve 1 is obtained by convolving B1 based on curve 00.

[0297] For example, new curve 1 may be obtained by adding or subtracting B1 to the target parameters of curve 00.

[0298] S205: The receiving end processes the curve 0 to generate a curve 2 based on the maximum display brightness and the current backlight intensity level.

[0299] For example, as shown in FIG. 5a, the tone mapping curve generation module may process an initial gain adjustment curve (curve 0) for area A based on, for example, current backlight intensity information (e.g., the current backlight intensity level) and a maximum display luminance maxT to generate an updated gain adjustment curve (e.g., curve 2, also referred to as curve2).

[0300] If the current backlight intensity information obtained by the tone mapping curve generation module is the current target backlight luminance information, curve 2 does not need to be generated based on the maximum display luminance of the display module.

[0301] For example, the maximum display luminance maxT and the current backlight intensity level Level (i.e., the current target backlight luminance L x =maxT*Level) may be used. For example, if the maximum display luminance is 500 and the current backlight intensity level is 10%, the current target backlight luminance is 50, and a scaling process is performed on curve 0 of area A to obtain curve 1. It should be noted that the processing method is not limited to scaling, and may further include other methods, which is not a limitation in this application.

[0302] Method 1:

[0303] For example, if the maximum value of x for curve 0 in region A is maxsource and the corresponding value of y is maxY, then the overall scaling factor for curve 0 is D=L x / maxY. Curve 0 is scaled as a whole based on a scaling factor D, where curve 2=D*curve 0.

[0304] Method 2:

[0305] The backlight metadata may include a preset brightness value T3.

[0306] For example, the preset brightness value T3 is the target brightness target_maxdisplay originally generated by curve 0.

[0307] The overall scaling factor for curve 0 is D=L x / T3. Curve 0 is scaled as a whole based on a scaling factor D, where curve 2=D*curve 0.

[0308] When a display has a particular backlight, for example, the maximum display brightness of the display is 500 nits and the backlight intensity level is 10%, the backlight brightness is 50 nits. In this embodiment, a tone mapping curve (i.e., curve 2) that matches the backlight brightness can be generated.

[0309] For processing methods other than scaling, please refer to Method 3 to Method 5 described in the examples in S205. The principles are similar, and the details will not be described again here.

[0310] S206: The receiving end processes curve 1 and curve 2 based on the anchor of region A to generate tone mapping curve 3 of region A.

[0311] FIG. 5e is an example of a diagram for generating curve 3 based on curve 1 and curve 2.

[0312] In the three coordinate diagram in FIG. 5e, each x-axis represents the luminance value of an image pixel in region A here, and the y-axis represents the luminance value of an image pixel that is displayed on the display and is in region A.

[0313] An example in which the value range of x for each tone mapping curve in FIG. 5e is 0 to 1 is used for illustration, and an example in which the anchor of region A is 0.7 is used for illustration.

[0314] Referring to the curve diagram of curve 2 shown in (1) of Fig. 5e, since curve 2 is a tone mapping curve generated based on the current target backlight luminance, most of the tone mapping values ​​in curve 2, i.e., the display luminance values ​​(i.e., y values) obtained by mapping the luminance values ​​of image pixels, are appropriate, and a few tone mapping values ​​are inappropriate. Therefore, based on the anchor, a part of the tone mapping curve may be first cut out from curve 2, and then curve splicing is performed based on curve 1 to generate a final tone mapping curve of area A (i.e., curve 3, also referred to as curve 3).

[0315] As shown in FIG. 5e(1), the y value corresponding to anchor 0.7 is y1, and curve 2 can be cut from origin (0,0) to P1(0.7,y1) to obtain curve 2-1. In other words, for the final tone mapping curve, i.e., curve 3, curve 2 is used for the curve portion where pixel luminance is below the anchor (0 to 0.7). For curve 3 shown in FIG. 5e(3), the curve 2-1 portion in FIG. 5e(1) is used in the pixel luminance range of 0 to 0.7.

[0316] Next, based on the tone mapping value of curve 2 at the anchor (0.7), i.e., the y value, here y1 in (1) of Figure 5e, curve 1 can be translated or stretched to obtain curve 1', so that the tone mapping values ​​(i.e., the y values) of curve 1' and curve 2 at the anchor are identical.

[0317] For example, FIG. 5e(2) shows a curve diagram of curve 1. When the tone mapping value at the anchor position (x=0.7) in curve 1 is y2, for example, P2(0.7,y2) in curve 1, the value range of x is 0-1, and since the curve under the anchor has been determined from curve 2, only the curve segment whose value range of x is 0-1 needs to be determined from curve 1 in FIG. 5e(2), i.e., curve segment P2P3. Then, the curve segment P2P3 in curve 1 can be translated in the reverse direction of the y-axis toward the x-axis, and the translation direction is parallel to the y-axis, as shown by the translation arrow in FIG. 5e(2). Thus, in the curve segment P2P3 obtained after translation (i.e., the curve segment MN shown in FIG. 5e(2), also referred to as curve 1′), when the value of x is the anchor 0.7, the value of y is y1, where P2 is M(0.7,y1) after translation, and P3 is point N after translation. The amount of translation of the curve here may be (y2-y1).

[0318] It should be noted in Fig. 5e that curve 3 can be generated by translating curve 1, or by scaling or stretching curve 1. This is not a limitation of the present application.

[0319] For curve 3 shown in (3) of Figure 5e, below the anchor, i.e., in the pixel luminance range of 0 to 0.7, the curve 2-1 portion in (1) of Figure 5e is used; and above the anchor, i.e., in the pixel luminance range of 0.7 to 1, the MN segment in curve 1' obtained in (2) of Figure 5e is used, thereby obtaining the tone mapping curve, i.e., curve 3 in area A (i.e., the target tone mapping curve in Figure 5a).

[0320] Point M in (3) of Fig. 5e is also point P1 in (1) of Fig. 5e, where the letter M stands for point.

[0321] In addition, in (2) of Fig. 5e, for example, a curve segment above the anchor in curve 1 is translated to obtain a curve segment above the anchor in curve 3. In another embodiment, curve 1 can alternatively be translated as a whole, and then the curve segment above the anchor is cut from curve 1 obtained after translation and connected with curve 2-1 at the anchor position, thereby obtaining curve 3. This is not a limitation in the present application.

[0322] Optionally, the curve 3 is formed by connecting two curve segments from two curves at the anchor position, so that the smoothness of the curve 3 is insufficient at the anchor position M shown in, for example, (3) of FIG. 5e. As a result, after tone mapping is performed on the image based on the curve 3, distortion is easily generated when the image is displayed on a display.

[0323] Therefore, in this embodiment of the present application, as shown in (3) of FIG. 5e, the curve segment EF of curve 3 within a preset range (0.7-T2, 0.7+T1) before and after the anchor can be corrected so that EF is smoother at M.

[0324] Then, the curve segment EF obtained after correction is smoothly connected at point E to the curve segment 0E in the curve 3 (i.e., the curve segment from the origin to point E), and the curve segment EF obtained after correction is smoothly connected at point F to the curve segment FN in the curve 3 (e.g., first-order differential continuity at points E and F), to obtain the curve 3 obtained after correction, i.e., the final tone mapping curve of area A. When the curve 3 obtained after correction is used to perform tone mapping display for image pixels, the high-frequency coefficients are not easily enhanced, and the image is not easily distorted.

[0325] S207: The receiving end performs tone mapping processing on the area A according to the tone mapping curve 3, and obtains the image display information of the area A.

[0326] It should be noted that the order of performing S201, S202, and S203 is not limited in this application.

[0327] The order of performing S204 and S205 is not limited in this application.

[0328] For example, as shown in FIG. 5a, the tone mapping curve generation module of the receiving end may send the generated target tone mapping curve (eg, curve 3 above) to the tone mapping module.

[0329] The tone mapping module may perform a tone mapping process on area A in image 1 based on curve 3 to adjust the image pixels of area A in image 1, thereby obtaining the display brightness of each pixel brightness in area A on the display module.

[0330] The display module may display area A in image 1 based on the display brightness corresponding to each pixel in area A in image 1, and may display image 1 on the display module with the pixels adjusted to match image 1 to the backlight brightness of the display.

[0331] Similarly, when the image 1 includes multiple regions A, the display module may display each region A based on the respective image display information.

[0332] Performing tone mapping on an image may be understood as mapping in the dynamic range, for example as shown in FIG.

[0333] Mapping in the dynamic range can be performed by using static and dynamic mapping methods.

[0334] For example, the static mapping method can be described as follows: the whole tone mapping process is performed based on single data with reference to the same video content or the same hard disk content. In other words, tone mapping is performed for one or more frames of an image based on the same target tone mapping curve, so that the bitstream carries a small amount of information, the processing process of adaptively displaying the image is simpler, and the image display delay is shorter.

[0335] The dynamic mapping method can be described as follows: dynamic tone mapping is performed based on different tone mapping curves with reference to image contents of specific regions, each scene, or each frame in an image. In this case, each frame or each scene needs to carry relevant scene information. Tone mapping processing based on different tone mapping curves can be performed based on images of specific regions, each scene, or each frame, so that tone mapping can be performed based on different tone mapping curves for images of different scenes, different image regions, and different frames. For example, for images in bright scenes, tone mapping can be performed based on the shooting scene by using a tone mapping curve that focuses on protecting the display effect of bright areas.

[0336] The static mapping method is equivalent to performing tone mapping for multiple frame images based on the same target tone mapping curve. In this case, the target tone mapping curve generated for one frame image in the multiple frame images can be used as the target tone mapping curve for the multiple frame images. When the target tone mapping curve for one frame image is generated, the global initial gain adjustment curve and global anchor of the frame of the image can be used for implementation.

[0337] The dynamic mapping method is equivalent to performing tone mapping for images of different frames, different image regions in an image of the same frame, and images in different scenes based on different target tone mapping curves, where local anchors and local initial tone mapping curves can be used to generate target tone mapping curves for each local region.

[0338] As illustrated in FIG. 5a, the backlight metadata may include local or global initial gain adjustment curves and local or global anchors.

[0339] The initial gain adjustment curve for a local region of an image may be referred to as a local initial gain adjustment curve, and the anchor for the local region of an image may be referred to as a local anchor.

[0340] The initial gain adjustment curve of a frame of an image may be referred to as a global initial gain adjustment curve, and the anchor of a frame of an image may be referred to as a global anchor. However, the initial gain adjustment curves and anchors of different frame of images may be different.

[0341] The initial gain adjustment curve of the image of the multiple frames may also be referred to as a global initial gain adjustment curve. For example, the initial gain adjustment curve of the image of the multiple frames may be used as a global initial gain adjustment curve of the image of the multiple frames.

[0342] An anchor that is shared by images of multiple frames may be referred to as a global anchor. For example, an anchor may be generated for one of the images of the multiple frames, and the anchor is used as a global anchor for the images of the multiple frames.

[0343] For example, when the bitstream includes a global initial gain adjustment curve and a global anchor, a target tone mapping curve of the image of each frame can be generated according to the above-mentioned method based on the initial gain adjustment curve and the anchor of the image of each frame. Based on the target tone mapping curve of the image of each frame, tone mapping is performed on the image of each frame, and display matching between the image and the display is implemented.

[0344] For example, when a bitstream includes a global initial gain adjustment curve and a global anchor, a target tone mapping curve applicable to images of multiple frames may be generated according to the above-described method based on the global initial gain adjustment curve of the images of the multiple frames and the global anchors of the images of the multiple frames, and tone mapping may be performed on the images of the multiple frames based on the same target tone mapping curve, and display adaptation between images and displays may be implemented.

[0345] For example, when the bitstream includes a local initial gain adjustment curve and a local anchor, for example, different regions in image 1 have respective local initial gain adjustment curves and respective local anchors, and a target tone mapping curve (also a local tone mapping curve) can be generated for each region in image 1 based on the initial gain adjustment curves and anchors of each region. Then, tone mapping is performed for different regions in image 1 based on the respective target tone mapping curves, and display matching between different regions in the image and displays is implemented. Tone mapping can be performed for different regions of the same image based on different tone mapping curves. In another example, for different scenes, the target tone mapping curve of each scene can be further generated based on the local initial gain adjustment curve and the local anchor of each scene, and then tone mapping is performed for images of different scenes based on the target tone mapping curve of the corresponding scene, and display matching is implemented between images of different scenes and displays.

[0346] In addition, since generating anchors and generating an initial gain adjustment curve for an image (or local image) are two independent processes, anchors can be generated for local regions and a global initial gain adjustment curve can be generated for a global image (e.g., a frame or frame sequence).

[0347] For example, when the bitstream includes a global initial gain adjustment curve and a local anchor, for each local image, a target tone mapping curve of each local image can be generated based on the local anchor of each local image and the global initial gain adjustment curve. When the target tone mapping curve of each local image is generated, the global initial gain adjustment curve can be used as the initial gain adjustment curve of the local image. The process of generating a target tone mapping curve for each local image is similar to the above-mentioned process described in Figure 5d and Figure 5e, and the details will not be described again here.

[0348] For example, when the bitstream includes a local initial gain adjustment curve and a global anchor, for each local image, a target tone mapping curve of each local image can be generated based on the local initial gain adjustment curve of each local image and the global anchor. When the target tone mapping curve of each local image is generated, the global anchor can be used as the anchor of the local image, that is, the local image reuses the same anchor. The process of generating a target tone mapping curve for each local image is similar to the above-mentioned process described in Figure 5d and Figure 5e, and the details will not be described again here.

[0349] The initial gain adjustment curves and anchors in the bitstream may not have a one-to-one correspondence, but in use, one initial gain adjustment curve and one anchor are used to generate a target gain adjustment curve for the image or region to be displayed, i.e., the initial gain adjustment curves and anchors are used in pairs.

[0350] For example, in the above-mentioned S207, when the receiving end performs a tone mapping process for area A based on tone mapping curve 3 to obtain image display information of area A, and when the tone mapping curve 3 is a global tone mapping curve, method 1 is used to implement the tone mapping process, and when the tone mapping curve 3 is a global tone mapping curve, method 2 is used to implement the tone mapping process.

[0351] Method 1:

[0352] For each pixel of the image to be displayed, a luminance value of the pixel to be displayed on the display is obtained based on the global tone mapping curve.

[0353] For example, the gain may be calculated based on equation (7): gain=f[i](max(Rp,Gp,Bp)) / max(Rp,Gp,Bp) Equation (7) where Rp, Gp and Bp are different color components of the current pixel P in the image to be displayed, and max(Rp, Gp, Bp) represents the maximum component value solution of the three color components.

[0354] f(x) represents the function of the global tone mapping curve used for the image to be displayed, and [i] represents that the current pixel P is the i-th pixel in the image to be displayed for which tone mapping is performed.

[0355] Then, the tone mapping value of the current pixel P is calculated based on equations (8), (9) and (10). RpTM[i]=Rp*gain Equation(8); GpTM[i]=Gp*gain Equation (9); and BpTM[i]=Bp*gain Equation (10) RpTM[i], GpTM[i] and BpTM[i] are the tone mapping values ​​of the three color components: red, green and blue of the current pixel P (i.e. the luminance values ​​of the current pixel P as they are displayed on the display and in the image to be displayed).

[0356] Method 2:

[0357] For each pixel of the local image in the image to be displayed, the luminance value of the pixel to be displayed on the display is obtained based on the local tone mapping curve, where the local tone mapping curve is the local target tone mapping curve generated for the local image, for example, curve 3 in area A above.

[0358] For example, if the image 1 to be displayed is divided into four regions A (i.e., four tone mapping regions), a local target tone mapping curve for each region A may be generated for each region A in the above-described manner for generating a target tone mapping curve.

[0359] Please note that the specific number of local images into which image 1 is divided (e.g., the number of regions A) is not limited in this application. Tone mapping can be performed for each local image based on the local tone mapping curve of each local image. In addition, the shapes of the local images in image 1 can be the same or different. This is not limited in this application.

[0360] As for the information about each local image (also called tone mapping region) into which image 1 is divided, the information may be, for example, size information, e.g., length and width information, of the local image in image 1 to determine the local image. In addition, the information describing the size of the local image may be carried in the above-mentioned backlight metadata, so that the receiving end performs tone mapping on the local images separately based on the information about the local images and according to the target tone mapping curve of the local images.

[0361] For example, when tone mapping is performed on a pixel P in an area A (which may be referred to as an initial area A) in an image 1, a target area A (which may be one or more target areas A) that is closest in space to the pixel P may be determined in the above-mentioned four areas A, a target tone mapping curve corresponding to the target area A is used as f(x) in equation (7) in the above-mentioned method 1, and a first tone mapping value of a current pixel P is calculated based on equations (7) to (10) in method 1. If there is one first area A, the first tone mapping value is used as a final tone mapping value of the current pixel P. If there are multiple target areas A, a gain[j] of the jth target area A may be obtained based on equation (7), where j=1, 2, 3, 4, and for the multiple first tone mapping values ​​corresponding to the target area A, a weighted sum may be performed on the multiple first tone mapping values, and the result of the weighted sum is divided by the sum of the weighted gain[j] corresponding to the target area A, and the obtained result is used as a final tone mapping value of the current pixel P.

[0362] When at least two first tone mapping values ​​of the current pixel P are weighted, j represents the jth region A in the four regions A of the image 1, and j=1, 2, 3, 4. In this case, the weighting Weight[j] of the first tone mapping value obtained based on the jth region A can be calculated based on Equation (11): Weight[j]=f(Num[j] / Numtotal) Equation (11) Here, in equation (11), j represents the jth target region A among the four regions A in image 1, Num[j] represents the number of pixels in the jth target region A in image 1, and Numtotal represents the total number of pixels in all target regions A around the current pixel P.

[0363] Optionally, for example, f(x) in equation (11) represents a function of the target tone mapping curve of the jth target region A in image 1.

[0364] Optionally, for example, f(x) in equation (11) is power(x, NumStr[j]), where f(x) is a specific expression of the tone mapping curve function, NumStr[j] represents the intensity information of the jth target region A, and the intensity information represents the decay rate of the weighting coefficient of the jth target region and is a preset value for each region A.

[0365] For example, the final tone mapping values ​​Rtm, Gtm, and Btm of the current pixel P may be calculated based on equations (12), (13), and (14). Rtm=Σ(RP*gain[j]*Weight[j]) / ΣWeight[j] Equation (12); Gtm = Σ(GP*gain[j]*Weight[j]) / ΣWeight[j] Equation (13); and Btm=Σ(BP*gain[j]*Weight[j]) / ΣWeight[j] Equation (14) where Rp, Gp and Bp are different color components of a current pixel P in a region A (e.g., the initial region A described above) in the image 1 to be displayed; gain[j] = f[j](max(Rp,Gp,Bp)) / max(Rp,Gp,Bp); and Rp, Gp, and Bp are the different color components of the current pixel P, max(Rp, Gp, Bp) represents the maximum component value solution of the three color components, f[j] is a function of the local target tone mapping curve corresponding to the jth target region A in image 1, and j is an integer greater than or equal to 1 or less than or equal to 4.

[0366] In this embodiment of the present application, when the current pixel P is located on a boundary shared by two regions A in the image 1, if the grayscale values ​​of the two pixels on the boundary are identical, and one pixel is mapped based on the target tone mapping curve of one region A, and the other pixel is mapped based on the target tone mapping curve of the other region A, the image of the image 1 will jump. Therefore, when there are multiple (at least two) target regions A, it indicates that the current pixel P is a pixel on a boundary shared by at least two regions A. The target tone mapping curves of each of the at least two regions A around the pixel P can be used to perform tone mapping for the pixel P separately, and the at least two first tone mapping values ​​obtained are weighted to obtain the final tone mapping value of the pixel P. In this way, in different image regions with different tone mapping curves, the display scenes of the different image regions can be taken into account in the display of the pixels shared by the different image regions, and the occurrence of pixels with jump exceptions in the displayed image is avoided.

[0367] Optionally, after the final tone mapping value is obtained for pixel P, the receiving end in this embodiment of the present application may further perform saturation adjustment based on a saturation (color) adjustment algorithm for each pixel in area A obtained after tone mapping is performed based on the target tone mapping curve.

[0368] Optionally, for the local or global initial gain adjustment curve in Fig. 5a, the curve data may not be directly carried in the backlight metadata, and the parameters of the initial gain adjustment curve and the curve type may be carried in the backlight metadata, in which case the receiving end may obtain the curve type and parameters from the backlight metadata, and obtain the local or global target tone mapping curve based on the HDR source or the SDR source.

[0369] For example, an HDR source or an SDR source may be understood as a video or image to be processed at a receiving end. The HDR source or the SDR source may include, but is not limited to, image data, such as pixel data.

[0370] For example, the backlight metadata may include, but is not limited to, a data format of an HDR or SDR source, division information of an image region, scanning order information of an image region, image characteristics, curve parameters of a local or global initial gain adjustment curve, and one or more metadata information units, which may include coordinate information, the image characteristics mentioned above, curve parameters of the initial gain adjustment curve mentioned above, and the like.

[0371] It should be noted that the format of the backlight metadata is not limited in this application. The backlight metadata may include, for example, histogram information and parameter information of a tone mapping curve, as in the ST2094-40 standard, or may include, for example, parameter information of a tone mapping curve, as in the ST2094-10 standard.

[0372] As described above, the initial gain adjustment curve in the backlight metadata can be any conventional tone mapping curve. For example, the receiving end can obtain the curve parameters a, b, p, m, and n of the initial gain adjustment curve from the metadata information unit in the bitstream and the HDR source or the SDR source to obtain the initial gain adjustment curve. Then, a target tone mapping curve can be obtained based on the anchor, the initial gain adjustment curve, and the image to be displayed, where the target tone mapping curve can still include the above-mentioned curve parameters.

[0373] For example, the target tone mapping curve function may be shown in Equation (15):

number

[0374] Based on the target tone mapping curve in equation (15), a mapping relationship between normalized HDR source data or normalized SDR source data (e.g., a display image to be displayed) and normalized HDR display data (e.g., a display value of each pixel displayed on a display and belonging to the display image to be displayed) can be obtained.

[0375] It should be noted that L and L' can be normalized optical or electrical signals, which is not a limitation in this application.

[0376] It should be noted that the data format of the HDR source or SDR source in the bitstream where the receiving end is located is not limited in this application. For example, in terms of color space of pixel data, the pixels in the image or video to be displayed may be YUV or RGB. In another example, in terms of bit width of data, each pixel in the image to be displayed may be 8 bits, 10 bits, 12 bits, or the like.

[0377] For an image or video in SDR or HDR format, the receiving end in this embodiment of the present application may perform a tone mapping process for every pixel in the image or video, so that tone mapping from a high dynamic range to a low dynamic range can be implemented in scenarios where the backlight of the display device is adjustable.

[0378] In addition, in this embodiment of the present application, the initial tone mapping curve (i.e., the initial gain adjustment curve described above) and the anchors used to generate the target tone mapping curve can be carried in the bitstream and transmitted to the receiving end. During implementation, the hardware channel used to transmit the curves and integrated into the hardware of the electronic device to which the receiving end belongs can be reused. This is more convenient for implementation in terms of hardware and can be applied to the hardware channels of all electronic devices.

[0379] Embodiment 2:

[0380] Regarding the backlight metadata carried in the bitstream of FIG. 1, in embodiment 2, the backlight metadata may include pixel-level or downsampling gain adjustment factors.

[0381] For example, for images or videos in SDR or HDR format in a bitstream, the images of each frame in the images and videos may have pixel level or downsampling gain adjustment factors.

[0382] For example, the pixel level or downsampling gain adjustment factor may be represented as an image having the same size as the image of the original frame. The information for each pixel in the image is a coefficient, referred to herein as a gain adjustment factor.

[0383] For example, pixel-level or downsampling gain adjustment factors may alternatively be expressed as follows: for a table corresponding to pixels in an image of the original frame, the table is in one-to-one correspondence with the pixels in the original image, and the data in the table are coefficients for the pixels, where the coefficients are referred to as gain adjustment factors.

[0384] The gain adjustment factor for pixels in highlight regions in each frame of image may be higher than the gain adjustment factor for pixels in non-highlight regions.

[0385] Figure 6a is an example of a system architecture diagram obtained after Figure 1 is improved. The similarities between Figure 6a and Figure 1 and Figure 5a will not be described again here, and only the improvements to Figure 1 and the differences between Figure 6a and Figure 5a will be described in detail.

[0386] It should be understood that the system shown in Fig. 6a is merely an example. The systems herein, and the transmitting and receiving ends thereof, may have more or less components than those shown in the figures, may combine two or more components, or may have different component configurations. The components shown in Fig. 6a may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processors and / or application specific integrated circuits.

[0387] As shown in FIG. 6a, the sending end may include a backlight metadata generating module.

[0388] The backlight metadata generation module may generate backlight metadata based on the electrical signal image or image sequence generated by the optical-to-electrical conversion module, where the backlight metadata may include pixel-level or downsampling gain adjustment factors for the images.

[0389] The gain adjustment factor is a gain factor, and the gain factor represents an amplification factor.

[0390] The pixel level gain adjustment factor of the image is used to adjust the pixel level display pixels of the image based on the gain adjustment factor. For example, if the pixel level unit is a 4×4 image region and the gain adjustment factor of the image region is 1.2, the value of the display pixel of the 4×4 image region may be increased by 1.2 times, for example, the luminance is increased by 1.2 times.

[0391] The encoding module may encode the electrical signal image and the backlight metadata, generate a bitstream, and transmit the bitstream to a receiving end.

[0392] The following describes the process by which the backlight metadata generation module generates pixel-level or downsampling gain adjustment factors for an image.

[0393] 6a and 6b, an example of a process for generating gain adjustment coefficients for image 1 (represented as image 2 shown in FIG. 6b) is shown. In FIG. 6b, an example in which the processing object is image 1 is used for explanation.

[0394] From the comparison between Fig. 5b and Fig. 6b, it can be seen that the process of S401, S402 and S403 in Fig. 6b is the same as the process of S401, S402 and S403 in Fig. 5b when an anchor is generated. Here, please refer to the related description in Fig. 5b. The details will not be described again here.

[0395] Still referring to FIG. 6b, after S403, the backlight metadata generation module may perform S404.

[0396] S404: The backlight metadata generation module constructs gain adjustment coefficients for image 1 based on the highlight and non-highlight regions, and generates image 2 including the gain adjustment coefficients.

[0397] For example, as shown in FIG. 6b, for each region A in image 1, different gain adjustment factors may be configured for pixels in highlight regions C in region A and for pixels in non-highlight regions C in region A.

[0398] For example, in Fig. 6b, coefficients 1, 2, 3 and 4 are respectively configured for four regions C. Coefficients 1 and 2 corresponding to the two highlight regions C are larger than coefficients 3 and 4 of the two non-highlight regions C.

[0399] Optionally, for different highlight regions C, the backlight metadata generation module may configure different gain adjustment factors for the different highlight regions C based on the average luminance of the pixels in the highlight regions C. For example, the gain adjustment factor of a region C with a higher average luminance is higher than the gain adjustment factor of a region C with a lower average luminance.

[0400] For example, the gain adjustment coefficients for the four regions C in FIG. 6b are coefficient 1=1.2, coefficient 2=1.1, coefficient 3=0.8, and coefficient 4=0.5, respectively.

[0401] For example, the gain adjustment factor of the highlight region C may be greater than one, and the gain adjustment factor of the non-highlight region C may be less than or equal to one.

[0402] In a possible implementation, a gain adjustment factor may be generated for region A in the following manner.

[0403] First, an initial gain adjustment coefficient is configured for each region C in region A, where the initial gain adjustment coefficient may be a first preset coefficient threshold value.

[0404] For example, the first preset coefficient threshold value may be 1.0, 0.8, or the like. The specific threshold value may be flexibly configured based on actual requirements. This is not limited in the present application. Optionally, the first preset coefficient threshold value is less than or equal to 1.0.

[0405] In this way, an initial gain adjustment factor may be configured for each region C in Fig. 6b (which may be understood as a pixel level). For example, the initial gain factors of the 64x64 pixels in region A are all 1.0.

[0406] When the gain adjustment coefficients are configured, the initial gain adjustment coefficients are configured based on the region C. Specifically, the gain adjustment coefficients of all pixels in the same region C are configured as 1.0. Therefore, the gain adjustment coefficients in this embodiment of the present application may be referred to as pixel-level gain adjustment coefficients.

[0407] Then, for each highlight region C in region A, the initial gain adjustment factor is increased by a factor multiplier.

[0408] For example, the coefficient multiplier may be the second preset coefficient threshold, e.g., 1.2. In this case, the gain adjustment coefficient of the highlight region C is updated to the first preset coefficient threshold x the second preset coefficient threshold, e.g., 1.0 x 1.2 = 1.2. In this case, in Fig. 6b, coefficient 1 = coefficient 2 = 1.2, and coefficient 3 = coefficient 4 = 1.0.

[0409] For example, the backlight metadata generation module may calculate an average luminance value of the pixels of each highlight region C, such that each highlight region C has a corresponding average luminance value. The configured coefficient multiplier for highlight regions C having a higher average luminance value (e.g., the average luminance is greater than or equal to the preset luminance threshold) is a third preset coefficient threshold, and the configured coefficient multiplier for highlight regions C having a lower average luminance value (e.g., the average luminance is less than the preset luminance threshold) is a fourth preset coefficient threshold, and the third preset coefficient threshold is greater than the fourth preset coefficient threshold.

[0410] For example, if the third preset coefficient threshold is 1.5 and the fourth preset coefficient is 1.2, the updated coefficients of region C in Fig. 6b are coefficient 1 = 1.5, coefficient 2 = 1.2, coefficient 3 = 1, and coefficient 4 = 1, respectively. For example, for region C corresponding to coefficient 1, the value of each pixel in region C in image 2 is no longer the color value in image 1, but coefficient 1, i.e., 1.5. In this way, image 2 with the same size as image 1 can be obtained, and the value of each pixel in image 2 is the pixel-level gain adjustment coefficient.

[0411] Of course, in another implementation, for non-highlight regions C, different gain adjustment factors can also be configured based on the average luminance of the pixels in region C. The gain adjustment factor for a region C with a higher average luminance is larger than the gain adjustment factor for a region C with a lower average luminance.

[0412] It should be noted that the above-mentioned method for configuring the gain adjustment coefficient for the region A in the image 1 is merely an implementation. In the present application, the gain adjustment coefficient of the region A in the image 1 may be separately configured in another manner. The configuration method is not limited to the above-mentioned example.

[0413] For example, if image 1 contains multiple regions A, different gain coefficients need to be configured for region A based on the highlight and non-highlight regions in region A. The method is similar, and the details will not be described again here.

[0414] From the above description, it can be seen that in the present application, the gain adjustment coefficients can be configured for pixels in each image region, i.e., at the pixel level, for example, image regions of sizes such as 4×10 or 8×6, etc. In this case, an image 2 of the same size as image 1, i.e., 64×64, is obtained after S404, and the value of each pixel in image 2 is the gain adjustment coefficient configured by the backlight metadata generation module.

[0415] In the above implementation, the example in FIG. 6a in which the transmitting end generates pixel-level or downsampling gain adjustment coefficients is used for illustration, so that the pixel-level or downsampling gain adjustment coefficients can be transmitted to the receiving end as attributes of the bitstream.

[0416] It should be noted that the receiving end in this embodiment of the present application may also include a backlight metadata generating module for performing the above-mentioned operations, thereby generating pixel-level or downsampling gain adjustment coefficients. The method is similar, and the details are not described again here.

[0417] Returning now to Fig. 6a, the execution process of the receiving end in Fig. 6a will be described below with reference to Fig. 6c.

[0418] Through a comparison between FIG. 1 and FIG. 6a, it can be seen that the display adaptation module in FIG. 1 may include a pixel processing module shown in FIG. 6a.

[0419] Referring to FIG. 6c, the following describes the operation process of the pixel processing module in FIG. 6a.

[0420] As shown in FIG. 6c, the process may include the following steps.

[0421] S301: The receiving end obtains a picture 1 to be displayed in a bitstream.

[0422] This step is similar to S201 in FIG. 5d, and the details will not be described again here.

[0423] S302: The receiving end obtains backlight metadata in a bitstream, where the backlight metadata includes pixel-level gain adjustment coefficients of image 1.

[0424] For example, as shown in FIG. 6a, the decoding module may decode the bitstream to obtain backlight metadata, which may include pixel-level or downsampling gain adjustment factors for images in the bitstream.

[0425] The pixel level or downsampling gain adjustment factor can be the gain adjustment factor of the image or all images in the bitstream, which is not a limitation in this application.

[0426] For example, image 1 in Fig. 6b is a to-be-displayed image in the bitstream. The decoding module may obtain pixel-level gain adjustment coefficients for image 1 (e.g., image 2 in Fig. 6b) and send the pixel-level gain adjustment coefficients to the pixel processing module.

[0427] S303: The receiving end obtains the maximum display brightness and the current backlight intensity level of the display.

[0428] This step is similar to S203 in Fig. 5d. Please refer to the above description. The details will not be described again here.

[0429] For example, referring to FIG. 6a, a pixel processing module at the receiving end may obtain the maximum display brightness and current backlight intensity information (which may be the current backlight intensity level or the current target backlight brightness, and this is not limited in the present application) of the display of the display device to which the receiving end belongs, that is, S303 may be performed by the pixel processing module in FIG. 6a.

[0430] S304: The receiving end processes the pixels of image 1 according to the maximum display luminance to generate a first image.

[0431] It should be noted that when the pixels of Image 1 are processed to generate the first image, the luminance value based on which the processing is performed is not limited to the maximum display luminance maxT of the display. The luminance value may be a luminance value greater than the current target backlight luminance and less than or equal to the maximum display luminance of the display (e.g., a weighted result of the current target backlight luminance and the maximum display luminance).

[0432] For example, an example is used for explanation in which the pixels of image 1 are processed based on the maximum display luminance to generate a first image. When the luminance value based on which the processing is performed is not the maximum display luminance maxT, the method is similar, and the details are not described again here.

[0433] For example, a scaling process may be performed on the display pixels (i.e., image pixels) of image 1 based on the maximum display luminance to obtain a first image (which may be represented by TMvaluemid1 below). Note that the processing manner is not limited to scaling and may further include other manners, which is not a limitation in this application.

[0434] Method 1:

[0435] For example, the maximum luminance value maxsource of the image pixels of image 1 may be used as maxY, and the overall scaling factor for the image pixels of image 1 is S=maxT / maxY. The image pixels of image 1 are scaled as a whole based on the scaling factor S, i.e. the pixel value of each pixel in image 1 is multiplied by the scaling factor S to obtain the first image.

[0436] The maximum display brightness of the display is maxT.

[0437] Method 2:

[0438] The backlight metadata may include a preset brightness value T3.

[0439] For example, the preset brightness value T3 may be the target brightness target_maxdisplay originally generated by the curve 0 mentioned in the above embodiment 1.

[0440] The scaling factor is S=maxT / T3. The image pixels of image 1 are scaled as a whole based on the scaling factor S, i.e. the pixel value of each pixel in image 1 is multiplied by the scaling factor S to obtain the first image.

[0441] The maximum display brightness of the display is maxT.

[0442] Consider that when a display has a certain backlight, the actual brightness of the display may be higher than the backlight brightness. For example, if the maximum display brightness of the display is 500nit and the backlight intensity level is 10%, the backlight brightness is 50nit. However, the electronic device may have a measure to adjust the final brightness of the display, so that the final brightness may be higher than 50nit. Therefore, in this embodiment, the first image obtained by processing the image 1 to be displayed based on the brightness value may be generated based on a brightness value that is higher than the backlight brightness and lower than or equal to the maximum display brightness (here, the maximum display brightness of the display is used).

[0443] S305: The receiving end processes the pixels of image 1 according to the maximum display brightness and the current backlight intensity level to generate a second image.

[0444] Similar to the execution principle of S304, for example, as shown in FIG. 6a, the pixel processing module may process the display pixels of image 1 based on the current backlight intensity information (e.g., the current backlight intensity level) and the maximum display luminance maxT to generate a second image (which may be represented by TMvaluemid2 below).

[0445] If the current backlight intensity information obtained by the pixel processing module is the current target backlight luminance information, the second image does not need to be generated based on the maximum display luminance of the display module.

[0446] For example, the maximum display luminance maxT and the current backlight intensity level Level (i.e., the current target backlight luminance L x=maxT*Level) product may be used. For example, if the maximum display luminance is 500 and the current backlight intensity level is 10%, the current target backlight luminance is 50, and a scaling process may be performed on the image pixels of image 1 based on the current target backlight luminance to obtain the second image. It should be noted that the processing manner is not limited to scaling, and may further include other manners, which is not limited in this application.

[0447] Method 1:

[0448] For example, the maximum luminance value maxsource of the image pixels of image 1 may be used as maxY, and the overall scaling factor for the image pixels of image 1 may be S=L x / maxY. The image pixels (eg, RGB information) of image 1 are scaled based on the scaling factor S, ie, the pixel value of each pixel in image 1 is multiplied by the scaling factor S to obtain the second image.

[0449] The maximum display brightness of the display is maxT, and L x =maxT*Level, where Level is the current backlight intensity level of the display.

[0450] Method 2:

[0451] The backlight metadata may include a preset brightness value T3.

[0452] For example, the preset brightness value T3 may be the target brightness target_maxdisplay originally generated by the curve 0 mentioned in the above embodiment 1.

[0453] The scaling factor is S=L x / T3. The image pixels of image 1 are scaled as a whole based on the scaling factor S, i.e. the pixel value of each pixel in image 1 is multiplied by the scaling factor S to obtain the second image.

[0454] The maximum display brightness of the display is maxT, and L x =maxT*Level, where Level is the current backlight intensity level of the display.

[0455] For example, when the display has a certain backlight, the maximum display brightness of the display is 500 nits and the backlight intensity level is 10%, the backlight brightness is 50 nits. In this embodiment, the image to be displayed 1 can be processed based on the current target backlight brightness to generate a second image.

[0456] For example, when the image pixels of image 1 are scaled as a whole based on a scaling factor S to obtain a first image, this may be implemented by using any one of the following three methods, and is not limited to the three methods listed here.

[0457] The following three methods are explained by using the example of generating the first image (TMvaluemid1). When the second image (TMvaluemid2) is generated, the following three methods are similar, and the details will not be explained again here.

[0458] Method 1: TMvaluemid1.R = scaling factor S*h(R value of the image to be displayed); TMvaluemid1.G = scaling factor S*h (G value of the image to be displayed); and TMvaluemid1.B = scaling factor S*h (B value of the image to be displayed).

[0459] Method 2: TMvaluemid1.R = scaling factor S*(h(R value of the image to be displayed) + gain adjustment coefficient 2); TMvaluemid1.G = scaling factor S*(h(G value of the image to be displayed) + gain adjustment coefficient 2); and TMvaluemid1.B = scaling factor S * (h (B value of the image to be displayed) + gain adjustment coefficient 2).

[0460] Method 3: TMvaluemid1.R = scaling factor S*h (R value of the image to be displayed) + gain adjustment coefficient 2; TMvaluemid1.G = Scaling factor S*h (G value of the image to be displayed) + Gain adjustment coefficient 2; and TMvaluemid1.B = scaling factor S * h (B value of the image to be displayed) + gain adjustment coefficient 2.

[0461] For example, the image to be displayed is image 1 as described above, and TMvaluemid1 represents the first image as described above; TMvaluemid1.R represents the value of the R channel of the first image; TMvaluemid1.G represents the value of the G channel of the first image; TMvaluemid1.B represents the value of the B channel of the first image; and The functional expression of h(x) can be h(x)=x, a formula for interconversion between PQ, HLG, and gamma, or a function formula of another curve, which is not limited in this application.

[0462] The pixel-level or downsampling gain adjustment coefficients in the backlight metadata may include two groups of gain adjustment coefficients, namely, gain adjustment coefficient 1 and gain adjustment coefficient 2. Of course, the gain adjustment coefficients in each group are the same as in the original image.

[0463] For the manner of generating the gain adjustment factor 1, please refer to the embodiment in Fig. 6b. The details will not be described again here.

[0464] The backlight metadata generation module may further generate a gain adjustment factor 2.

[0465] Referring to Fig. 6a, there may be an image signal loss (e.g., converted to a floating point difference) between the image of the ambient light collected by the collection module and the image of the electrical signal generated by the optical-to-electrical conversion module. The backlight metadata generation module may use the image signal loss as a gain adjustment factor 2. The pixel value of each pixel at the gain adjustment factor 2 is the image signal lost by each pixel.

[0466] For example, by using the acquisition module and the optical-electrical conversion module, a 12-bit image can be converted into an 8-bit image, in which case 4-bit image data is lost by each pixel in the image, and the 4-bit data lost by each pixel can be used as the pixel value of the corresponding pixel in the gain adjustment factor 2.

[0467] The first image and the second image are obtained based on the gain adjustment factor 2 and the gain adjustment factor 1, so that the contrast of the images can be improved.

[0468] S306: The receiving end processes the first image and the second image according to the gain adjustment coefficient of image 1 to obtain image display information of image 1.

[0469] It should be noted that the order of performing S301, S302, and S303 is not limited in this application.

[0470] The order of executing S304 and S305 is not limited in this application.

[0471] For example, referring to FIG. 6a, the pixel processing module at the receiving end may process a first image and a second image corresponding to image 1 based on the gain adjustment coefficients at the pixel level and that of image 1 (e.g., image 2 in FIG. 6b) to obtain image display information of image 1 on the display, i.e., tone mapping of each pixel in image 1.

[0472] for example, Tmvalue=W*TMvaluemid1+(1-W)*TMvaluemid2 Equation (16) where Tmvalue in equation (16) represents the tone mapping value of each pixel in image 1; and TMvaluemid1 represents the first image, and TMvaluemid2 represents the second image.

[0473] For example, W may represent the gain adjustment factor for image 1, which may be represented as image 2 produced in FIG. 6b, for example.

[0474] Alternatively, for example, W may represent a function such as the square of the gain adjustment factor 1 for image 1.

[0475] Specifically, a weighted sum is performed on the first image and the second image by using the gain adjustment factor 1 of image 1 or a function of the gain adjustment factor 1 as a weight, so that a tone mapping value of image 1 can be obtained.

[0476] Alternatively, for example, Tmvalue=W*TMvaluemid2 Equation (17), where Tmvalue in equation (17) represents the tone mapping value of each pixel in image 1; and TMvaluemid1 represents the first image.

[0477] For example, W may be the gain adjustment factor for image 1 (eg, which may be represented as generated image 2 in FIG. 6b).

[0478] Alternatively, for example, W can be a function A(x) of the gain adjustment factor 1 of image 1, e.g., A(x)=1+(S-1)*x, where x represents the gain adjustment factor 1 of image 1, and S represents the scaling factor used when the first image or the second image is generated. Note that the A(x) function is not limited here.

[0479] For example, Tmvalue is the final image pixel information of each pixel in image 1 at the current target backlight luminance, and the tone mapping value of each pixel in Tmvalue is the display value (or final RGB value) of each pixel in image 1 on the display.

[0480] In this embodiment, the image 1 to be displayed may be processed based on the current target backlight luminance to obtain a second image, and based on the respective gain adjustment coefficients of the highlight and non-highlight regions in the image 1, the highlight regions in the second image may be brightened and the non-highlight regions (e.g., dark regions) may be darkened to obtain the display luminance of each pixel in the image 1 on the screen. For example, if the gain adjustment coefficient of an area in the image 1 is 1.5, when the area is displayed, the display luminance of the area may be increased by 1.5 times. As a result, the area may be overly bright when displayed. In this case, optionally, a clamping process may be performed on the Tmvalue by using the first image TMvaluemid1 to avoid over-brightness of some areas when the image 1 is displayed on the display.

[0481] Optionally, when the gain adjustment coefficient in the backlight metadata is a downsampling gain adjustment coefficient, the downsampling gain coefficient and the pixel of the image to be displayed (for example, image 1) may have a one-to-one correspondence, or a one-to-many correspondence, such as a one-to-four correspondence or a one-to-16 correspondence. In this case, when the display luminance of each pixel of image 1 on the screen is obtained based on the downsampling gain adjustment coefficient, the correspondence between the downsampling gain coefficient and the pixel of image 1 may be obtained. Then, based on the correspondence, the corresponding pixel in image 1 is processed based on the corresponding downsampling gain coefficient. The specific processing manner is similar to the principle of the process of processing image 1 based on pixel-level gain coefficient described in the above embodiment 2. Please refer to the above description. And the details will not be described again here.

[0482] In this embodiment of the present application, pixel-level gain adjustment coefficients are carried in the bitstream, so that when an image is displayed in a scenario where the backlight is adjustable, the display brightness of each pixel on the display can be adjusted based on the gain adjustment coefficient of each pixel in the image. This method is easier to implement in software, and tone mapping can reach pixel level, and brightness adjustment is more refined, flexible, and adjustable.

[0483] It should be noted that embodiment 1 and embodiment 2 can also be combined. Specifically, the backlight metadata can include local or global initial gain adjustment curves, local or global anchors, and pixel-level or downsampling gain adjustment coefficients. Please refer to each other for similarities between embodiment 1 and embodiment 2.

[0484] Optionally, for one frame image or frame sequence in the bitstream, the receiving end in this embodiment of the present application may perform tone mapping on a part of the image, an image region, or a part of the frame sequence based on the target tone mapping curve to generate a tone mapping value, thereby obtaining image display information, and further obtain image display information of another part of the image or frame sequence based on the gain adjustment coefficient corresponding to the image, the current target backlight luminance, and the maximum display luminance of the display.

[0485] In conventional dynamic metadata display conformance standards or algorithms, the display effect of images or videos on backlight adjustable devices is usually not taken into account. In conventional solutions, the video is mainly adapted to the maximum display capability (i.e., the maximum display brightness) of the screen, which is not suitable for displaying images or videos in weak backlight scenarios and has poor display effect.

[0486] However, in the above-mentioned system in this embodiment of the present application, when the backlight of the display is adjustable, brightness adaptation can be performed on the image or video to be displayed by referring to the maximum display brightness of the display and the current target backlight brightness, so that the display brightness of the image on the display can be automatically adjusted based on the adjustment change of the backlight intensity level, thereby improving the display effect of the image or video when the backlight is weak.

[0487] The following describes an apparatus provided in an embodiment of the present application.

[0488] 7 is a diagram of the structure of an apparatus according to an embodiment of the present application. As shown in FIG. 7, an apparatus 500 may include a processor 501 and a transceiver 505, and may further include a memory 502, optionally.

[0489] The transceiver 505 may be referred to as a transceiver unit, a transceiver device, a transceiver circuit, or the like, and is configured to implement a transceiver function. The transceiver 505 may include a receiver and a transmitter. The receiver may be referred to as a receiver device, a receiver circuit, or the like, and is configured to implement a receiving function. The transmitter may be referred to as a transmitter device, a transmitter circuit, or the like, and is configured to implement a transmitting function.

[0490] The memory 502 may store computer programs, software codes, or instructions 504, where the computer programs, software codes, or instructions 504 may also be referred to as firmware. The processor 501 may implement the methods provided in the embodiments of the present application by executing the computer programs or software codes or instructions 503 in the processor 501 or by calling the computer programs or software codes or instructions 504 stored in the memory 502. The processor 501 may be a central processing unit (CPU), and the memory 502 may be, for example, a read-only memory (ROM) or a random access memory (RAM).

[0491] The processor 501 and transceiver 505 described herein may be implemented in an integrated circuit (IC), an analog IC, a radio frequency integrated circuit (RFIC), a mixed signal IC, an application specific integrated circuit (ASIC), a printed circuit board (PCB), an electronic device, or the like.

[0492] The device 500 may further include an antenna 506. The modules included in the device 500 are merely examples for the purpose of illustration and are not limiting in this application.

[0493] As described above, the structure of the device described in the above embodiment may not be limited in FIG. 7. The device may be an independent device or a part of a larger device. For example, the implementation of the device may be as follows: (1) an independent integrated circuit IC, chip, or chip system or subsystem; (2) a set including one or more ICs, where optionally, the set of ICs may also include a storage component for storing data and instructions; (3) a module that may be embedded in another device; (4) an in-vehicle device or the like; or (5) others.

[0494] For the case where the implementation of the device is a chip or a chip system, please refer to the diagram of the structure of the chip shown in Figure 8. The chip shown in Figure 8 includes a processor 601 and an interface 602. There may be one or more processors 601, and there may be more than one interface 602. Optionally, the chip or chip system may include a memory 603.

[0495] All relevant contents of the steps in the above method embodiments may be referred to the functional descriptions of the corresponding functional modules, and the details will not be described again here.

[0496] Based on the same technical concept, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, the computer program includes at least one segment of code, and the at least one segment of code can be executed by a computer to control the computer to implement the above-mentioned method embodiment.

[0497] Based on the same technical idea, the embodiments of the present application further provide a computer program, which, when executed by a terminal device, is used to implement the above-mentioned method embodiments.

[0498] The entire program or a portion of the program may be stored in a storage medium encapsulated in the processor, or a portion of the program or the entire program may be stored in a memory not encapsulated in the processor.

[0499] Based on the same technical idea, an embodiment of the present application further provides a chip, including a network interface controller and a processor, which can implement the above-mentioned method embodiments.

[0500] The method or algorithm steps described in combination with the contents disclosed in the embodiments of the present application may be implemented by hardware or by a processor executing software instructions. The software instructions may include corresponding software modules. The software modules may be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable hard disk, a compact disk read-only memory (CD-ROM), or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from and write information to the storage medium. Of course, the storage medium may alternatively be a component of the processor. The processor and the storage medium may be located in an ASIC.

[0501] Those skilled in the art should recognize that the functions described in the embodiments of the present application in one or more examples above can be implemented by hardware, software, firmware, or any combination thereof. When the functions are implemented by software, the functions can be stored on a computer-readable medium or transmitted in the computer-readable medium as one or more instructions or codes. Computer-readable media includes computer storage media and media, where media includes any medium that facilitates the transmission of a computer program from one place to another. Storage media can be any available medium accessible by a general-purpose or special-purpose computer.

[0502] Although the embodiments of the present application are described above with reference to the accompanying drawings, the present application is not limited to the specific implementations described above, and the specific implementations described above are merely illustrative and not limiting. Under the enlightenment of the present application, a person skilled in the art can make many forms without departing from the purpose of the present application and the protection scope of the claims, all of which are included in the protection of the present application.

Claims

1. obtaining first display information of a target image to be displayed based on a current target backlight luminance of the display; obtaining second display information of the target image based on a maximum display luminance of the display and the current target backlight luminance; processing the first display information and the second display information based on preset information of the target image to obtain third display information of the target image, where the preset information includes information regarding the brightness of the target image; and displaying the target image on the display based on the third display information. An image processing method comprising:

2. The step of obtaining first display information of a target image to be displayed based on a current target backlight luminance of a display includes: obtaining a first tone mapping curve for the target image to be displayed; and processing the first tone mapping curve based on the current target backlight luminance of the display to obtain a second tone mapping curve of the target image as the first display information; The method of claim 1 , comprising:

3. The step of obtaining second display information of the target image based on a maximum display luminance of the display and the current target backlight luminance includes: determining a first luminance based on the maximum display luminance of the display and the current target backlight luminance, where the first luminance is greater than the current target backlight luminance; and Processing the first tone mapping curve based on the first luminance to obtain a third tone mapping curve of the target image as the second display information. The method of claim 2 , comprising:

4. The preset information includes a second luminance of the target image, and the step of processing the first display information and the second display information based on the preset information of the target image to obtain a third display information of the target image includes: obtaining the second luminance of the target image; extracting a first sub-curve from the second tone mapping curve, the first sub-curve having a pixel luminance less than or equal to the second luminance; obtaining a first tone mapping value corresponding to the second luminance and in the second tone mapping curve; processing the third tone mapping curve based on the first tone mapping values ​​to generate a fourth tone mapping curve, where a tone mapping value in the fourth tone mapping curve that corresponds to the second luminance is the first tone mapping value; extracting a second sub-curve from the fourth tone mapping curve, the second sub-curve having a pixel luminance greater than or equal to the second luminance; and Connecting the first sub-curve and the second sub-curve based on the second luminance, and obtaining a target tone mapping curve of the target image as the third display information. The method of claim 3 , comprising:

5. The step of obtaining a first tone mapping curve of the display target image comprises: obtaining a first tone mapping curve for each of a plurality of local regions constituting the target image; Including, The step of obtaining the second luminance of the target image comprises: obtaining a second luminance for each of the plurality of local regions; and The step of connecting the first sub-curve and the second sub-curve based on the second luminance, and obtaining a target tone mapping curve of the target image as the third display information includes: for the plurality of local regions of the target image, connecting a first sub-curve and a second sub-curve of each of the plurality of local regions according to a second luminance of each of the plurality of local regions, and obtaining a target tone mapping curve for each of the plurality of local regions as the third display information of the target image. The method of claim 4 , comprising:

6. The step of obtaining the second luminance of the target image comprises: obtaining the second luminance value of the target image based on a minimum value of the pixel luminance value and an average value of the pixel luminance value of the target image; The method of claim 4 , comprising:

7. The step of obtaining the second luminance value of the target image based on a minimum value of the pixel luminance value and an average value of the pixel luminance value of the target image comprises: clustering image pixels of the target image based on the pixel intensities and pixel locations in the target image to obtain a plurality of image regions; classifying the plurality of image regions in the target image based on the pixel brightness to obtain a first type image region and a second type image region, where the brightness of the first type image region is higher than the brightness of the second type image region; and determining the second luminance value of the target image based on a minimum value of the pixel luminance value and an average value of the pixel luminance value in the first type image region; The method of claim 6, comprising:

8. The step of obtaining first display information of a target image to be displayed based on a current target backlight luminance of a display includes: processing the display target image based on the current target backlight luminance of the display to obtain a first image as the first display information; The method of claim 1 , comprising:

9. The step of obtaining second display information of the target image based on a maximum display luminance of the display and the current target backlight luminance includes: determining a third luminance based on the maximum display luminance of the display and the current target backlight luminance, where the third luminance is greater than the current target backlight luminance; and Processing the target image based on the third luminance to obtain a second image as the second display information. The method of claim 8 , comprising:

10. The preset information includes a gain coefficient of the target image, and the step of processing the first display information and the second display information based on the preset information of the target image to obtain a third display information of the target image includes: obtaining the gain factor of the target image; and processing the first image and the second image based on the gain factor to generate a third image as the third display information; 10. The method of claim 9, comprising:

11. The step of obtaining the gain factor of the target image comprises: clustering image pixels of the target image based on pixel intensities and pixel locations in the target image to obtain a plurality of image regions; classifying the plurality of image regions in the target image based on the pixel brightness to obtain a first type image region and a second type image region, where the brightness of the first type image region is higher than the brightness of the second type image region; and configuring different gain factors for the first type image regions and the second type image regions, where the gain factor for the first type image regions is greater than the gain factor for the second type image regions; The method of claim 10, comprising:

12. The step of processing the first display information and the second display information to obtain third display information of the target image based on preset information of the target image includes: obtaining the preset information of the target image from source information of the display target image; and processing the first display information and the second display information based on the preset information to obtain the third display information of the target image; The method of claim 1 , comprising:

13. 13. An electronic device comprising a memory and a processor, the memory being coupled to the processor, the memory storing program instructions, and, when the program instructions are executed by the processor, the electronic device being capable of performing the image processing method according to any one of claims 1 to 12.

14. A computer readable storage medium comprising a computer program, the computer program being capable of executing an image processing method according to any one of claims 1 to 12 when the computer program is run on an electronic device.

15. A chip comprising one or more interface circuits and one or more processors, wherein the interface circuits are configured to receive signals from a memory of an electronic device and to transmit the signals to the processor, wherein the signals include computer instructions stored in the memory; and when the processor executes the computer instructions, the electronic device is capable of performing an image processing method according to any one of claims 1 to 12.

Images