Tone mapping method and apparatus
Patent Information
- Application Number
- MYPI2025004454
- Authority / Receiving Office
- MY · MY
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-02-23
- Filing Date
- 2024-02-21
- Publication Date
- 2024-08-23
AI Technical Summary
The existing tone mapping method only considers the display brightness range of the display screen, cannot achieve the best display adaptation effect, and cannot flexibly adapt to changes in different environments and devices.
This paper proposes a tone mapping method that adjusts the tone mapping parameters of the image by obtaining information such as the size of the terminal device, user viewing distance, and ambient light brightness to ensure the best display effect of the image on different devices and environments.
It improves the display adaptation effect of the image and the consistency of the user's visual perception, and improves the user's viewing experience.
Smart Images

Figure PI2025004454_DRAWING_001 
Figure PI2025004454_DRAWING_002 
Figure PI2025004454_DRAWING_003
Abstract
Description
A tone mapping method and device Technical Field
[0001] The present application relates to the field of display technology, and in particular to a tone mapping method and device. Background Art
[0002] Dynamic range is used in many fields to express the ratio of the maximum value to the minimum value of a variable. In digital images, dynamic range represents the ratio between the maximum brightness value and the minimum brightness value within the image display range. In the same scene in the real world, the dynamic range is usually around 10 -3 Candela / square meter (cd / m 2 ) to 106cd / m 2 High dynamic range (HDR) video technology expands the brightness range of displayable images, thereby recording a wider range of brightness information in the real world and showing more details in the bright and dark areas of the image. However, the brightness of videos or images often does not match the brightness range that can be displayed by standard dynamic range (SDR) display devices or many other existing HDR display devices.
[0003] Therefore, when a video or image is displayed on an existing display device, the brightness of the video or image needs to be processed according to the display capability of the display device to adapt it to the brightness range that the display device can display. This process is called tone mapping (TM).
[0004] However, existing solutions only consider the display brightness range of the display when performing tone mapping, and cannot achieve the best display adaptation effect.
[0005] Summary of the Invention
[0006] This application proposes a tone mapping method and device, which takes other information such as the terminal display device size and the user viewing distance into consideration when performing tone mapping on an image, thereby improving the display adaptation effect.
[0007] In a first aspect, an embodiment of the present application provides a tone mapping method, which includes: obtaining an image to be displayed; obtaining reference information; the reference information includes at least one of a reference display device size, a reference viewing distance, a reference viewing angle, a reference ambient light brightness, or a reference display device resolution; obtaining terminal device information; the terminal device information includes at least one of a terminal display device size, a user viewing distance, a user viewing angle, an ambient light brightness, or a terminal display device resolution; and tone mapping the image to be displayed based on the reference information and the corresponding type of the terminal device information.
[0008] In an embodiment of the present application, in the tone mapping of the image to be displayed based on the reference information and the terminal device information of the corresponding type, the reference display device size corresponds to the terminal display device size, the reference viewing distance corresponds to the user viewing distance, the reference viewing angle corresponds to the user viewing angle, the reference ambient light brightness corresponds to the ambient light brightness, and the reference display device resolution corresponds to the terminal display device resolution.
[0009] In the embodiment of the present application, when tone mapping is performed on an image, other information such as the terminal display device size and the user's viewing distance is also taken into account, thereby improving the display adaptation effect.
[0010] In one possible implementation, tone mapping is performed on the image to be displayed based on the reference information and the terminal device information of the corresponding type, including: obtaining tone mapping parameters based on the reference information and the terminal device information of the corresponding type; and tone mapping is performed on the image to be displayed based on the tone mapping parameters.
[0011] In a possible implementation, obtaining tone mapping parameters according to the reference information and the terminal device information of the corresponding type includes: adjusting the initial tone mapping parameters of the image to be displayed according to the reference information and the terminal device information of the corresponding type to obtain the tone mapping parameters.
[0012] In this application method, the initial tone mapping parameters of the image to be displayed are adjusted based on the reference information and the corresponding type of terminal device information to obtain the tone mapping parameters. The tone mapping parameters vary with the size of the terminal display device, the user's viewing distance, the user's viewing angle, the ambient light brightness, or the resolution of the terminal display device. This allows for flexible adaptation to different environments and display devices, effectively improving the display adaptation effect, and enhancing the image's tone mapping and display effects. At the same time, this allows for more consistent visual perception in different application scenarios, thereby enhancing the user's viewing experience.
[0013] In one possible implementation, adjusting the initial tone mapping parameters of the image to be displayed based on the reference information and the corresponding type of terminal device information to obtain the tone mapping parameters includes: when the relationship between the reference information and the terminal device information satisfies a first condition, adjusting the initial tone mapping parameters of the image to be displayed in a first manner to obtain the tone mapping parameters. When the relationship between the reference information and the terminal device information satisfies a second condition, adjusting the initial tone mapping parameters of the image to be displayed in a second manner to obtain the tone mapping parameters. The initial tone mapping parameters are used to represent an initial tone mapping curve, and the tone mapping parameters are used to represent an adjusted tone mapping curve.
[0014] In this embodiment, the method of adjusting the initial tone mapping parameters corresponds to the relationship between the reference information and the terminal device information. Different adjustment methods can be used for different application scenarios to obtain different adjusted tone mapping parameters, thereby meeting the user needs of different application scenarios, and adapting to changing application scenarios to improve user experience.
[0015] In a possible embodiment, the first condition includes: the terminal display screen size is smaller than the reference display device size, the user viewing distance is greater than the reference viewing distance, the user viewing angle deviates from the reference viewing angle, the ambient light brightness is greater than the reference ambient light brightness, or the terminal display device resolution is smaller than the reference display device resolution.
[0016] In this embodiment, when the size of the terminal display screen is smaller than the size of the reference display device, the image visually perceived by the user will become darker or lack contrast; similarly, when the user's viewing distance is greater than the reference viewing distance, the user's viewing angle deviates from the reference viewing angle, the ambient light brightness is greater than the reference ambient light brightness, or the resolution of the terminal display device is smaller than the reference display device resolution, the image visually perceived by the user will also become darker or blurred, affecting the user's viewing experience. Therefore, adjusting the tone mapping parameters in the first manner helps to improve the overall brightness and detail clarity of the image, and can improve the image tone mapping effect in different application scenarios, meet the user needs of different application scenarios, and improve the user's visual experience.
[0017] In one possible embodiment, the above-mentioned adjusting the initial tone mapping parameters of the image to be displayed in the first manner includes: obtaining one or more first brightness characteristic values of the image to be displayed; adjusting the initial tone mapping parameters so that the mapping value of the obtained adjusted tone mapping curve at the one or more first brightness characteristic values is greater than the mapping value of the initial tone mapping curve at the one or more first brightness characteristic values.
[0018] In this embodiment, the mapping value of the adjusted tone mapping curve at the one or more first brightness characteristic values is increased, thereby increasing the overall or partial mapping value of the adjusted tone mapping curve. After the image is tone mapped based on the tone mapping parameters, the overall or partial brightness of the image is increased to adapt to the application scenarios covered by the first condition above, improve the display adaptation effect, enhance the user's overall brightness perception of the image, and make the user's brightness perception consistent across different application scenarios, thereby improving the user experience. For example, when the terminal display screen size is smaller than the reference display device size, the mapping value of the tone mapping curve is increased to increase the image brightness, so that the user's viewing brightness perception on the small-sized display is the same as that on the large-sized display, thereby improving the user's visual comfort.
[0019] In a possible embodiment, the above-mentioned adjustment of the initial tone mapping parameters of the image to be displayed in the first manner also includes: obtaining one or more second brightness characteristic values of the image to be displayed; adjusting the initial tone mapping parameters so that the slope of the obtained adjusted tone mapping curve at the one or more second brightness characteristic values is greater than the slope of the initial tone mapping curve at the one or more second brightness characteristic values.
[0020] In this embodiment, the slope of the adjusted tone mapping curve at the one or more second brightness characteristic values increases, resulting in an increase in the slope of the adjusted tone mapping curve as a whole or in part. After tone mapping the image based on the adjusted tone mapping curve, the overall or partial contrast and brightness of the image are increased to accommodate the application scenarios covered by the first condition above, ensuring consistent user perception of contrast and image detail across different application scenarios, improving user experience, and enhancing display adaptation. For example, when the terminal display screen size is smaller than the reference display device size, the slope of the tone mapping curve is increased to improve image contrast and brightness, ensuring that the user's perception of image clarity and brightness on the small display screen is the same as on the large display screen, thereby improving user visual comfort.
[0021] In a possible embodiment, the above-mentioned adjustment of the initial tone mapping parameters of the image to be displayed in the first manner also includes: obtaining the base layer and one or more enhancement layers of the image to be displayed; the first eigenvalue of any enhancement layer among the one or more enhancement layers is mapped to a second eigenvalue by the initial tone mapping curve of any enhancement layer; adjusting the initial tone mapping parameters of any enhancement layer so that the absolute value of the value of the first eigenvalue after mapping by the adjusted tone mapping curve is greater than the absolute value of the second eigenvalue.
[0022] In this embodiment, the first eigenvalue and the second eigenvalue can be positive or negative. When comparing the value of the first eigenvalue mapped by the adjusted tone mapping curve with the size of the second eigenvalue, the absolute values of the above two values are compared.
[0023] In this embodiment, after the base layer is mixed with one or more adjusted enhancement layers, the absolute value of the mapping value corresponding to one or more enhancement layers in the image increases, thereby enhancing the features of the enhancement layer, retaining more high-frequency components, and enhancing local contrast or brightness. The enhancement layer includes an edge layer, a texture layer, etc., so the corresponding image feature enhancements include: enhanced edge sharpness, clearer texture details, etc. This adapts to the application scenarios covered by the first condition above and improves the display adaptation effect. At the same time, it enables complex images to have better tone mapping and display effects. For example, when the terminal display screen size is smaller than the reference display device size, the mapping value of the enhancement layer tone mapping curve is increased to enhance the local contrast, detail clarity or brightness of the image, so that the image clarity and detail information viewed by the user on the small-size display screen are the same as those on the large-size display screen, thereby improving the user's visual comfort.
[0024] In a possible embodiment, the above-mentioned second condition includes: the terminal display screen size is larger than the reference display device size, the user viewing distance is smaller than the reference viewing distance, the ambient light brightness is smaller than the reference ambient light brightness, or the terminal display device resolution is larger than the reference display device resolution. At least one of the above.
[0025] In this embodiment, when the size of the terminal display screen is larger than the size of the reference display device, the image visually perceived by the user will become brighter and glaring, or the contrast will be too strong, causing visual discomfort; similarly, when the user's viewing distance is less than the reference viewing distance, the ambient light brightness is less than the reference ambient light brightness, or the resolution of the terminal display device is greater than the resolution of the reference display device, the image visually perceived by the user will also become brighter, or even glaring and damaging the eyes, or the contrast will be too strong, causing visual discomfort, affecting the user's viewing experience. Therefore, adjusting the tone mapping curve in the second way to darken the overall brightness and contrast of the image can improve the image tone mapping effect and display effect in different application scenarios, meet the user needs of different application scenarios, and improve the user's visual experience.
[0026] In one possible embodiment, the above-mentioned adjustment of the initial tone mapping parameters of the image to be displayed in the second manner includes: obtaining one or more third brightness characteristic values of the image to be displayed; adjusting the initial tone mapping parameters so that the mapping value of the adjusted tone mapping curve at the one or more third brightness characteristic values is less than the mapping value of the initial tone mapping curve at the one or more third brightness characteristic values.
[0027] In this embodiment, the mapping values of the adjusted tone mapping curve at the one or more third brightness characteristic values are reduced, resulting in a reduction in the overall or partial mapping values of the adjusted tone mapping curve. After tone mapping the image based on the adjusted tone mapping curve, the overall or partial brightness of the image is reduced to accommodate the application scenarios covered by the second condition above, ensuring consistent user brightness perception across different application scenarios, thereby improving display adaptation and user experience. For example, when the terminal display screen size is larger than the reference display device size, the mapping values of the tone mapping curve are reduced to dim the image brightness, ensuring that the user's viewing brightness on the large display screen is the same as that on the small display screen, thereby improving user visual comfort.
[0028] In a possible implementation, the one or more third brightness characteristic values include one or more brightness characteristic values of a dark portion of the image to be displayed.
[0029] In this embodiment, because the human eye is more sensitive to the brightness of dark areas, the image, after tone mapping based on the adjusted tone mapping parameters, further reduces the brightness of dark areas. This avoids the increased noise in the application scenarios covered by the second condition above, ensuring consistent user perception of dark area brightness across different application scenarios, thereby improving display adaptation and user experience. For example, when the terminal display size is larger than the reference display device size, the dark area mapping value of the tone mapping curve is adjusted down to further darken the dark areas of the image, ensuring that the user's viewing brightness on a large display is consistent with that on a small display, thereby improving user visual comfort.
[0030] In a possible embodiment, the above-mentioned adjusting the initial tone mapping parameters of the image to be displayed in the second manner also includes: obtaining one or more fourth brightness characteristic values of the image to be displayed; adjusting the initial tone mapping parameters so that the slope of the obtained adjusted tone mapping curve at the one or more fourth brightness characteristic values is less than the slope of the initial tone mapping curve at the one or more fourth brightness characteristic values.
[0031] In this embodiment, the slope of the adjusted tone mapping curve at the one or more fourth brightness characteristic values is reduced, resulting in a decrease in the overall or partial slope of the adjusted tone mapping curve. After tone mapping the image based on the adjusted tone mapping curve, the overall or partial contrast and brightness of the image are reduced, preventing user discomfort caused by excessive contrast, thereby adapting to the application scenarios covered by the second condition above and improving display adaptation and user experience. For example, when the terminal display screen size is larger than the reference display device size, the slope of the tone mapping curve is reduced to reduce the image contrast and brightness, so that the image contrast perceived by the user on the large display screen is consistent with the contrast and brightness perceived on the small display screen. This prevents user discomfort caused by excessive contrast due to a larger display screen and improves user visual comfort.
[0032] In a possible embodiment, the above-mentioned adjustment of the initial tone mapping parameters of the image to be displayed in the second manner also includes: obtaining the base layer and one or more enhancement layers of the image to be displayed; the third eigenvalue of any one of the one or more enhancement layers is mapped to a fourth eigenvalue by the initial tone mapping curve of any enhancement layer; adjusting the initial tone mapping parameters of any enhancement layer so that the absolute value of the value of the third eigenvalue after mapping by the adjusted tone mapping curve is less than the absolute value of the fourth eigenvalue.
[0033] In this embodiment, the third eigenvalue and the fourth eigenvalue can be positive or negative. When comparing the value of the third eigenvalue mapped by the adjusted tone mapping curve with the size of the fourth eigenvalue, the absolute values of the above two values are compared.
[0034] In this embodiment, after the base layer is blended with one or more adjusted enhancement layers, the mapping values corresponding to the one or more enhancement layers in the image are reduced, thereby weakening the features of the enhancement layers, preserving fewer high-frequency components, and reducing local contrast and brightness, that is, pixel gradients. Enhancement layers include edge layers and texture layers, so the image feature reduction includes reduced edge sharpness and texture detail. This adapts to the application scenarios covered by the second condition above, avoids excessive contrast leading to poor display quality and impacting user viewing, and improves display adaptation. It also achieves better tone mapping and display quality for complex images. For example, when the terminal display size is larger than the reference display device size, the mapping values of the enhancement layer tone mapping curve are reduced to reduce local image contrast, detail clarity, and brightness. This ensures that the image clarity and detail perceived by users on large displays are consistent with those on small displays. This prevents user discomfort caused by excessive contrast due to larger displays and improves visual comfort.
[0035] In one possible implementation, reference tone mapping parameters of the image to be displayed are obtained; display capabilities of the terminal display device in the terminal device information are obtained; and the reference tone mapping parameters of the image to be displayed are adjusted according to the reference tone mapping parameters and the display capabilities of the terminal display device to obtain the initial tone mapping parameters.
[0036] In this embodiment, the reference tone mapping parameters of the image to be displayed are adjusted based on the reference tone mapping parameters and the display capability information of the display screen, so that the initial tone mapping parameters obtained are adapted to the terminal display device in terms of display capability, which serves as the basis for further adjusting the tone mapping parameters in combination with other information of the terminal display device.
[0037] In a possible embodiment, obtaining the reference information includes: obtaining the reference information from the metadata of the image to be displayed; or obtaining the index of the reference information from the metadata, and selecting the reference information from multiple preset reference information according to the index; or using the preset reference information as the reference information.
[0038] In a possible implementation, the acquiring of the terminal device information includes: acquiring the terminal device information through measurement; or selecting the terminal device information from a plurality of preset terminal device information according to a preset index; or using the preset terminal device information as the terminal device information.
[0039] In a second aspect, an embodiment of the present application provides an image encoding method, which includes: writing an image to be displayed into a code stream; writing reference information of the image to be displayed into the code stream; the reference information includes at least one of a reference display device size, a reference viewing distance, a reference viewing angle, a reference ambient light brightness, or a reference display device resolution; the reference information is used to perform tone mapping on the image to be displayed.
[0040] In the embodiments of the present application, tone mapping of the image to be displayed based on the reference information combined with the corresponding type of terminal device information of the decoding end can flexibly adapt to different environments and display devices, effectively improving the display adaptation effect, enhancing the tone mapping effect and display effect of the image. At the same time, it makes the user's visual perception more consistent in different application scenarios, thereby improving the user's viewing experience.
[0041] In one possible embodiment, the method further includes: writing reference tone mapping parameters of the image to be displayed into the bitstream; wherein the reference tone mapping parameters are used to obtain initial tone mapping parameters; the reference information is used by the decoding end to adjust the initial tone mapping parameters to obtain tone mapping parameters, and the tone mapping parameters are used to tone map the image to be displayed.
[0042] In this embodiment, the reference tone mapping parameters of the image to be displayed are adjusted based on the reference tone mapping parameters in combination with the display capability information of the decoding end display screen, so that the initial tone mapping parameters obtained are adapted to the terminal display device in terms of display capability, which serves as the basis for the decoding end to adjust the tone mapping parameters again in combination with other information of the terminal display device.
[0043] In a possible implementation manner, writing the reference information of the to-be-displayed image into the code stream includes: writing the reference information into the code stream; or writing an index of the reference information into the code stream.
[0044] In a possible implementation, the method further includes: writing one or more brightness characteristic values of the image to be displayed into the bitstream; wherein the one or more brightness characteristic values are used by the decoding end to adjust the initial tone mapping parameters to obtain the tone mapping parameters.
[0045] In this embodiment, the tone mapping parameters adjusted at the decoding end using one or more luminance feature values can flexibly adapt to different environments and display devices, effectively improving display adaptation, enhancing the tone mapping effect and display quality of the image. At the same time, it makes the user's visual perception more consistent in different application scenarios, thereby improving the user's viewing experience.
[0046] In a third aspect, an embodiment of the present application provides a tone mapping device, comprising a module for executing the method described in the first aspect or any possible implementation of the first aspect.
[0047] Exemplarily, the device may include:
[0048] A receiving module, used for acquiring an image to be displayed;
[0049] Also used to obtain reference information; the reference information includes at least one of a reference display device size, a reference viewing distance, a reference viewing angle, a reference ambient light brightness, or a reference display device resolution;
[0050] Also used to obtain terminal device information; the terminal device information includes at least one of the terminal display device size, user viewing distance, user viewing angle, ambient light brightness, or terminal display device resolution;
[0051] The tone mapping module is used to perform tone mapping on the image to be displayed according to the reference information and the terminal device information of the corresponding type.
[0052] In an embodiment of the present application, in the tone mapping of the image to be displayed based on the reference information and the terminal device information of the corresponding type, the reference display device size corresponds to the terminal display device size, the reference viewing distance corresponds to the user viewing distance, the reference viewing angle corresponds to the user viewing angle, the reference ambient light brightness corresponds to the ambient light brightness, and the reference display device resolution corresponds to the terminal display device resolution.
[0053] In one possible implementation, tone mapping is performed on the image to be displayed based on the reference information and the terminal device information of the corresponding type, including: obtaining tone mapping parameters based on the reference information and the terminal device information of the corresponding type; and tone mapping is performed on the image to be displayed based on the tone mapping parameters.
[0054] In a possible implementation, obtaining tone mapping parameters according to the reference information and the terminal device information of the corresponding type includes: adjusting the initial tone mapping parameters of the image to be displayed according to the reference information and the terminal device information of the corresponding type to obtain the tone mapping parameters.
[0055] In one possible implementation, adjusting initial tone mapping parameters of the image to be displayed based on the reference information and the corresponding type of terminal device information to obtain the tone mapping parameters includes: when the relationship between the reference information and the terminal device information satisfies a first condition, adjusting the initial tone mapping parameters of the image to be displayed in a first manner to obtain the adjusted tone mapping parameters. When the relationship between the reference information and the terminal device information satisfies a second condition, adjusting the initial tone mapping parameters of the image to be displayed in a second manner to obtain the tone mapping parameters. The initial tone mapping parameters are used to represent an initial tone mapping curve, and the tone mapping parameters are used to represent an adjusted tone mapping curve.
[0056] In a possible embodiment, the first condition includes: the terminal display screen size is smaller than the reference display device size, the user viewing distance is greater than the reference viewing distance, the user viewing angle deviates from the reference viewing angle, the ambient light brightness is greater than the reference ambient light brightness, or the terminal display device resolution is smaller than the reference display device resolution.
[0057] In one possible embodiment, adjusting the initial tone mapping parameters of the image to be displayed in a first manner includes: obtaining one or more first brightness characteristic values of the image to be displayed; adjusting the initial tone mapping parameters so that the mapping value of the obtained adjusted tone mapping curve at the one or more first brightness characteristic values is greater than the mapping value of the initial tone mapping curve at the one or more first brightness characteristic values.
[0058] In one possible embodiment, adjusting the initial tone mapping curve of the image to be displayed in a first manner also includes: obtaining one or more second brightness characteristic values of the image to be displayed; adjusting the initial tone mapping curve so that the slope of the obtained adjusted tone mapping curve at the one or more second brightness characteristic values is greater than the slope of the initial tone mapping curve at the one or more second brightness characteristic values.
[0059] In a possible embodiment, adjusting the initial tone mapping parameters of the image to be displayed in a first manner also includes: obtaining the base layer and one or more enhancement layers of the image to be displayed; the first eigenvalue of any one of the one or more enhancement layers is mapped to a second eigenvalue by the initial tone mapping curve of any enhancement layer; adjusting the initial tone mapping parameters of any enhancement layer so that the absolute value of the first eigenvalue after mapping by the adjusted tone mapping curve is greater than the absolute value of the second eigenvalue.
[0060] In this embodiment, the first eigenvalue and the second eigenvalue can be positive or negative. When comparing the value of the first eigenvalue mapped by the adjusted tone mapping curve with the size of the second eigenvalue, the absolute values of the above two values are compared.
[0061] In a possible embodiment, the second condition includes: the terminal display screen size is larger than the reference display device size, the user viewing distance is smaller than the reference viewing distance, the ambient light brightness is smaller than the reference ambient light brightness, or the terminal display device resolution is larger than the reference display device resolution.
[0062] In one possible embodiment, adjusting the initial tone mapping parameters of the image to be displayed in a second manner includes: obtaining one or more third brightness characteristic values of the image to be displayed; adjusting the initial tone mapping parameters so that the mapping value of the adjusted tone mapping curve at the one or more third brightness characteristic values is less than the mapping value of the initial tone mapping curve at the one or more third brightness characteristic values.
[0063] In a possible implementation, the one or more third brightness characteristic values include: one or more brightness characteristic values of a dark portion of the image to be displayed.
[0064] In a possible embodiment, adjusting the initial tone mapping parameters of the image to be displayed in a second manner also includes: obtaining one or more fourth brightness characteristic values of the image to be displayed; adjusting the initial tone mapping parameters so that the slope of the obtained adjusted tone mapping curve at the one or more fourth brightness characteristic values is less than the slope of the initial tone mapping curve at the one or more fourth brightness characteristic values.
[0065] In a possible embodiment, adjusting the initial tone mapping parameters of the image to be displayed in a second manner also includes: obtaining the base layer and one or more enhancement layers of the image to be displayed; the third eigenvalue of any one of the one or more enhancement layers is mapped to a fourth eigenvalue by the initial tone mapping curve of any enhancement layer; adjusting the initial tone mapping parameters of any enhancement layer so that the absolute value of the third eigenvalue after mapping by the adjusted tone mapping curve is less than the absolute value of the fourth eigenvalue.
[0066] In this embodiment, the third eigenvalue and the fourth eigenvalue can be positive or negative. When comparing the value of the third eigenvalue mapped by the adjusted tone mapping curve with the size of the fourth eigenvalue, the absolute values of the above two values are compared.
[0067] In one possible embodiment, the tone mapping module is also used to: obtain a reference tone mapping of the image to be displayed; obtain the display capability of the terminal display device in the terminal device information; adjust the reference tone mapping parameters of the image to be displayed according to the reference tone mapping parameters and the display capability of the terminal display device to obtain the initial tone mapping parameters.
[0068] In a possible embodiment, obtaining the reference information includes: obtaining the reference information from the metadata of the image to be displayed; or obtaining the index of the reference information from the metadata, and selecting the reference information from multiple preset reference information according to the index; or using the preset reference information as the reference information.
[0069] In a possible implementation, the acquiring of the terminal device information includes: acquiring the terminal device information through measurement; or selecting the terminal device information from a plurality of preset terminal device information according to a preset index; or using the preset terminal device information as the terminal device information.
[0070] It should be understood that the third aspect corresponds to the technical solution of the first aspect, so the beneficial effects achieved by each possible implementation method are similar and will not be repeated here.
[0071] In a fourth aspect, an embodiment of the present application provides an image encoding device, which includes a module for executing the method described in the second aspect or any possible implementation of the second aspect.
[0072] Exemplarily, the device may include:
[0073] An encoding module is configured to write an image to be displayed into a bitstream; and further configured to write reference information of the image to be displayed into the bitstream; the reference information comprising at least one of a reference display device size, a reference viewing distance, a reference viewing angle, a reference ambient light brightness, or a reference display device resolution; and the reference information is used to perform tone mapping on the image to be displayed.
[0074] The sending module is used to send the code stream.
[0075] In one possible embodiment, the encoding module is also used to: write reference tone mapping parameters of the image to be displayed into the code stream; wherein the reference tone mapping parameters are used to obtain initial tone mapping parameters; the reference information is used by the decoding end to adjust the initial tone mapping parameters to obtain tone mapping parameters, and the tone mapping parameters are used to perform tone mapping on the image to be displayed.
[0076] In a possible implementation manner, writing the reference information of the to-be-displayed image into the code stream includes: writing the reference information into the code stream; or writing an index of the reference information into the code stream.
[0077] In a possible implementation, the encoding module is further used to: write one or more brightness characteristic values of the image to be displayed into the code stream; wherein the one or more brightness characteristic values are used by the decoding end to adjust the initial tone mapping parameters to obtain the tone mapping parameters.
[0078] In a fifth aspect, an embodiment of the present application provides an electronic device comprising: a memory and a processor coupled to each other, wherein the processor calls a program code stored in the memory to execute a method as described in the first aspect or any possible implementation of the first aspect.
[0079] It should be understood that the technical effects that can be achieved in the fifth aspect can be specifically described with reference to the technical effects brought about by the above-mentioned first aspect or any possible implementation method of the first aspect, and will not be repeated here.
[0080] In a sixth aspect, an embodiment of the present application provides an electronic device comprising: a memory and a processor coupled to each other, wherein the processor calls a program code stored in the memory to execute a method as described in the second aspect or any possible implementation of the second aspect.
[0081] It should be understood that the technical effects that can be achieved in the sixth aspect can be specifically described with reference to the technical effects brought about by the above-mentioned second aspect or any possible implementation method of the second aspect, and will not be repeated here.
[0082] In the seventh aspect, an embodiment of the present application provides a tone mapping system, including the tone mapping device described in the third aspect and any possible implementation of the third aspect and the image encoding device described in the fourth aspect and any possible implementation of the fourth aspect, or including the electronic device described in the fifth aspect and the electronic device described in the sixth aspect.
[0083] It should be understood that the technical effects that can be achieved in the seventh aspect can be specifically referred to the description of the technical effects brought about by the above-mentioned first aspect or any possible implementation method of the first aspect, and the description of the technical effects brought about by the above-mentioned second aspect or any possible implementation method of the second aspect, and will not be repeated here.
[0084] In an eighth aspect, an embodiment of the present application provides a code stream structure, which includes encoded image data and parameters required for image processing, the parameters required for image processing include the reference information described in the first aspect, any possible implementation of the first aspect, the second aspect, and any possible implementation of the second aspect, and the reference information includes at least one of a reference display device size, a reference viewing distance, a reference viewing angle, a reference ambient light brightness, or a reference display device resolution.
[0085] In one possible implementation, the code stream further includes reference tone mapping parameters, and the reference tone mapping curve parameters are used to obtain initial tone mapping parameters; the reference information is used by the decoding end to adjust the initial tone mapping parameters to obtain tone mapping parameters, and the tone mapping parameters are used to perform tone mapping on the image to be displayed.
[0086] In a possible implementation manner, the code stream includes an index of the reference information.
[0087] In a possible implementation manner, the code stream includes one or more brightness characteristic values.
[0088] In the ninth aspect, an embodiment of the present application provides a computer-readable storage medium, including a program or instruction. When the program or instruction is run on a computer, the method in the first aspect or any possible implementation of the first aspect is executed, or the method in the second aspect or any possible implementation of the second aspect is executed.
[0089] It should be understood that the technical effects that can be achieved in the seventh aspect can be specifically described by referring to the technical effects brought about by the above-mentioned first aspect or any possible implementation method of the first aspect or the second aspect or any possible implementation method of the second aspect, and will not be repeated here.
[0090] In the tenth aspect, an embodiment of the present application provides a computer program product, including a program or instructions, which, when the program or instructions are run on a computer, enables the method in the first aspect or any possible implementation of the first aspect to be executed, or enables the method in the second aspect or any possible implementation of the second aspect to be executed.
[0091] It should be understood that the technical effects that can be achieved in the eighth aspect can be specifically described by referring to the technical effects brought about by the above-mentioned first aspect or any possible implementation method of the first aspect or the second aspect or any possible implementation method of the second aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0092] FIG1 is a schematic block diagram of a coding and decoding system used in an embodiment of the present application;
[0093] FIG2A is a block diagram of a content providing system for implementing a content distribution service according to an embodiment of the present application;
[0094] FIG2B is a schematic diagram of an example structure of the terminal device 2106 in FIG2A ;
[0095] FIG3A is a schematic diagram of a workflow of a streaming media system used in an embodiment of the present application;
[0096] FIG3B is a schematic diagram of a streaming media system architecture used in an embodiment of the present application;
[0097] FIG4A is a schematic diagram of a possible system architecture applicable to an embodiment of the present application;
[0098] FIG4B is a schematic diagram of the structure of an image processing system provided in an embodiment of the present application;
[0099] FIG5 is a flow chart of a tone mapping method provided in an embodiment of the present application;
[0100] FIG6 is an example diagram of an initial tone mapping parameter adjustment method provided in an embodiment of the present application;
[0101] FIG7 is an example diagram of an initial tone mapping parameter adjustment method provided in an embodiment of the present application;
[0102] FIG8 is an example diagram of a method for adjusting initial tone mapping parameters of an enhancement layer provided in an embodiment of the present application;
[0103] FIG9 is an example diagram of an initial tone mapping parameter adjustment method provided in an embodiment of the present application;
[0104] FIG10 is an example diagram of an initial tone mapping parameter adjustment method provided in an embodiment of the present application;
[0105] FIG11 is a schematic flow chart of an image encoding method provided in an embodiment of the present application;
[0106] FIG12 is an exemplary diagram of a tone mapping device provided in an embodiment of the present application;
[0107] FIG13 is an exemplary diagram of an image encoding device provided in an embodiment of the present application;
[0108] FIG14 is an exemplary diagram of an electronic device provided in an embodiment of the present application;
[0109] FIG15 is an example diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0110] The term "at least one" in the embodiments of the present application refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where any of a, b, c, ab, ac, bc, or abc can include a single a, a single b, a single c, and can also include multiple a, multiple b, and multiple c.
[0111] Furthermore, unless otherwise indicated, ordinal numbers such as "first" and "second" in the embodiments of this application are used to distinguish between multiple objects and are not used to define the order, timing, priority, or importance of multiple objects. For example, the first priority criterion and the second priority criterion are only used to distinguish different criteria and do not indicate differences in the content, priority, or importance of the two criteria.
[0112] In addition, the terms "including" and "having" in the embodiments, claims, and drawings of this application are not exclusive. For example, a process, method, system, product, or device that includes a series of steps or modules / units is not limited to the listed steps or modules and may also include steps or modules / units that are not listed.
[0113] To facilitate understanding, the technical terms involved in the embodiments of this application are first explained accordingly.
[0114] (1) Dynamic range: The ratio of the maximum brightness to the minimum brightness in a video or image signal.
[0115] (2) High dynamic range (HDR) images: images with a dynamic range between 0.001 and 10,000 nits, where nit is a unit of illumination.
[0116] (3) Standard dynamic range (SDR) images, also known as low dynamic range, are images with a dynamic range generally between 1 nit and 100 nit.
[0117] (4) Metadata, which is used to record key information of the scene or image in the frame. In the embodiment of the present application, the metadata may also include reference tone mapping parameters for characterizing the reference tone mapping curve, and reference information, including at least one of the reference display device size, reference viewing distance, reference viewing angle, reference ambient light brightness, or reference display device resolution, for tone mapping the image. Metadata may also include dynamic metadata and static metadata; dynamic metadata may be understood as data associated with each frame of the image (hereinafter also referred to as attribute information), which may change with different pictures; static metadata may be understood as data associated with the image sequence, which remains unchanged within the image sequence.
[0118] (5) Tone mapping: mapping the brightness of an image to the brightness range that the target display device can display. Specifically, it can be divided into high dynamic range to low dynamic range mapping and low dynamic range to high dynamic range mapping. For example, an HDR image includes a 4000nit light signal, but the HDR display capability of the target display device (TV, iPad) is only 500nit. In this case, the 4000nit signal needs to be mapped to a 500nit device, which is a high-to-low tone mapping process. For another example, an SDR image includes a 100nit SDR signal, but the target display device has a 2000nit display capability. In this case, the 100nit signal needs to be mapped to a 2000nit device, which is a low-to-high tone mapping process.
[0119] (6) Image contrast: This is a measure of the brightness difference between the brightest and darkest eigenvalues in the light and dark areas of an image, i.e., the grayscale contrast of an image. A larger range of difference indicates a greater contrast, while a smaller range of difference indicates a smaller contrast. A good contrast ratio of 120:1 can easily display vivid and rich colors. When the contrast ratio is as high as 300:1, it can support all levels of color.
[0120] (7) Mapping curve: a function curve for mapping the brightness of an image. The mapping curve in the embodiment of the present application may include a Sigmoid function curve, a cubic spline function curve, a Gamma function curve, etc.
[0121] (8) Display adaptation: Tone processing of video or image signals to adapt to the display characteristics of the target display device.
[0122] When a video or image is displayed on an existing display device, the brightness of the video or image needs to be processed according to the display capability of the display device so that it can adapt to the brightness range that the display device can display. This process is called tone mapping (TM). However, current display devices are diverse, and there are differences not only in the brightness range that the display can display, but also in other aspects, such as the size of the display. Therefore, the existing solution simply aligns the maximum and minimum values of the image with the maximum and minimum brightness values of the display, which cannot achieve the best display adaptation effect. Therefore, the present application proposes a tone mapping method and device, which takes into account other information such as the size of the terminal display device and the user's viewing distance when tone mapping the image, thereby improving the display adaptation effect.
[0123] The following describes the codec system used in the present application in conjunction with FIG1 . FIG1 is a schematic block diagram of a codec system used in an embodiment of the present application, such as a video codec system 10 (or simply codec system 10) that can utilize the techniques of the present application. The video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) in the video codec system 10 represent devices that can be used to perform various techniques according to the various examples described in this application.
[0124] As shown in FIG1 , a coding and decoding system 10 includes a source device 12 , which is configured to provide coded data 21 such as coded images to a destination device 14 for decoding the coded data.
[0125] The source device 12 includes an encoder 20 and, optionally, may further include an image source 16 , an image preprocessor 18 (or a preprocessing unit), and a communication interface or communication unit 22 .
[0126] The image source 16 may include or may be any type of image capture device for capturing real-world images, etc., and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images, or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage that stores any of the above images.
[0127] In order to distinguish the processing performed by the pre-processor 18 or the pre-processing unit 18 , the image or image data 17 may also be referred to as a raw image or raw image data 17 .
[0128] The preprocessor 18 is configured to receive (raw) image data 17 and preprocess the image data 17 to obtain a preprocessed image 19 or preprocessed image data 19. For example, the preprocessing performed by the preprocessor 18 may include cropping, color format conversion (e.g., from RGB to YCbCr), color grading, or denoising. It will be appreciated that the preprocessor 18 may be an optional component.
[0129] The video encoder 20 is configured to receive the pre-processed image data 19 and provide encoded image data 21 .
[0130] Communication interface 22 in source device 12 may be used to receive encoded image data 21 and transmit the encoded image data 21 (or any other processed version thereof) to another device, such as destination device 14, or any other device, via communication channel 13 for storage or direct reconstruction.
[0131] Destination device 14 includes a decoder 30 (eg, video decoder 30 ) and, in addition and optionally, may include a communication interface or communication unit 28 , a post-processor 32 (or post-processing unit 32 ), and a display device 34 .
[0132] The communication interface 28 in the destination device 14 is used to receive the encoded image data 21 (or any other processed version) directly from the source device 12 or from any other source device such as a storage device, for example, a storage device that is an encoded image data storage device, and provide the encoded image data 21 to the decoder 30.
[0133] The communication interface 22 and the communication interface 28 may be used to send or receive the encoded image data 21 or the encoded data via a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or via any type of network, such as a wired network, a wireless network, or any combination thereof, any type of private network and public network, or any combination thereof.
[0134] For example, the communication interface 22 may be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission coding or processing for transmission over a communication link or network.
[0135] The communication interface 28 corresponds to the communication interface 22 , for example, and can be used to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain encoded image data 21 .
[0136] Both the communication interface 22 and the communication interface 28 can be configured as a unidirectional communication interface as indicated by the arrow pointing from the source device 12 to the corresponding communication channel 13 of the destination device 14 in Figure 1, or a bidirectional communication interface, and can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission such as encoded image data transmission, etc.
[0137] The decoder 30 is configured to receive the encoded image data 21 and provide decoded image data 31 or a decoded image 31 .
[0138] The post-processor 32 in the destination device 14 is configured to post-process the decoded image data 31 (also referred to as reconstructed image data), such as the decoded image 31, to obtain post-processed image data 33, such as the post-processed image 33. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color grading, cropping, or resampling, or any other processing for generating the decoded image data 31 for display on a display device 34, etc.
[0139] A display device 34 in the destination device 14 is configured to receive the post-processed image data 33 and display the image to a user or viewer. The display device 34 may be or include any type of display for displaying the reconstructed image, such as an integrated or external display screen or monitor. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display screen.
[0140] Although FIG1 shows source device 12 and destination device 14 as separate devices, device embodiments may also include both source device 12 and destination device 14 or the functionality of both source device 12 and destination device 14, that is, both source device 12 or the corresponding functionality and destination device 14 or the corresponding functionality. In these embodiments, source device 12 or the corresponding functionality and destination device 14 or the corresponding functionality may be implemented using the same hardware and / or software or through separate hardware and / or software, or any combination thereof.
[0141] Based on the description, it will be apparent to those skilled in the art that the existence and (precise) division of different units or functions in the source device 12 and / or the destination device 14 shown in FIG. 1 may vary depending on actual devices and applications.
[0142] The content providing system of the content distribution service used in the present application is described below in conjunction with Figure 2A. Figure 2A is a block diagram of a content providing system for implementing the content distribution service used in an embodiment of the present application. The content providing system 2100 includes a capture device 2102, a terminal device 2106, and an (optional) display 2126. The capture device 2102 communicates with the terminal device 2106 via a communication link 2104. The communication link may include the above-mentioned communication channel 13. The communication link 2104 includes but is not limited to WiFi, Ethernet, wired, wireless (3G / 4G / 5G), USB, or any combination thereof.
[0143] Capture device 2102 generates data and can use the coding method shown in the above embodiment to encode the data. Alternatively, capture device 2102 can distribute data to a streaming media server (not shown), which encodes the data and transmits the encoded data to terminal device 2106. Capture device 2102 includes but is not limited to a camera, a smart phone or a tablet computer, a computer or a notebook computer, a video conferencing system, a PDA, a vehicle-mounted device, or any combination thereof. For example, capture device 2102 can include above-mentioned source device 12. When data includes video, the video encoder 20 in capture device 2102 can actually perform video encoding. When data includes audio (i.e., sound), the audio encoder 20 in capture device 2102 can actually perform audio encoding. In some actual scenarios, capture device 2102 distributes encoded video data and encoded audio data by multiplexing the encoded video data and the encoded audio data together. In other actual scenarios, for example, in a video conferencing system, the encoded audio data and the encoded video data are not multiplexed. Capture device 2102 distributes the encoded audio data and the encoded video data to terminal device 2106 respectively.
[0144] The terminal device 2106 in the content providing system 2100 receives and regenerates the encoded data. The terminal device 2106 can be a device with data reception and recovery capabilities, such as a smartphone or tablet computer 2108, a computer or laptop computer 2110, a network video recorder (NVR) / digital video recorder (DVR) 2112, a television 2114, a set-top box (STB) 2116, a video conferencing system 2118, a video surveillance system 2120, a personal digital assistant (PDA) 2122, an in-vehicle device 2124, or any combination thereof, or such devices capable of decoding the encoded data. For example, the terminal device 2106 can include the destination device 14 described above. When the encoded data includes video, the video decoder 30 in the terminal device prioritizes video decoding. When the encoded data includes audio, the audio decoder in the terminal device prioritizes audio decoding. The terminal device 2106 may be a video playback application, a streaming media playback application, a streaming media playback platform, a live broadcast platform, etc. running on the terminal device.
[0145] For end devices with displays, such as smartphones or tablets 2108, computers or laptops 2110, NVRs / DVRs 2112, televisions 2114, PDAs 2122, or in-vehicle devices 2124, the end device can send the decoded data to its display. For end devices without displays, such as STBs 2116, video conferencing systems 3118, or video surveillance systems 2120, the device can be connected to an external display 2126 to receive and display the decoded data.
[0146] When performing encoding or decoding, each device in this system may use the image encoding device or image decoding device shown in the above embodiments.
[0147] FIG2B is a schematic diagram of an example structure of the terminal device 2106 in FIG2A . After the terminal device 2106 receives the bitstream from the capture device 2102, the protocol processing unit 2202 analyzes the transmission protocol of the bitstream. The protocol includes, but is not limited to, Real Time Streaming Protocol (RTSP), Hyper Text Transfer Protocol (HTTP), HTTP Live Streaming Protocol (HLS), MPEG Dynamic Adaptive Streaming over HTTP (MPEG-DASH), Real-time Transport Protocol (RTP), Real Time Messaging Protocol (RTMP), or any combination thereof.
[0148] After processing the stream, protocol processing unit 2202 generates a stream file. This file is output to demultiplexing unit 2204. Demultiplexing unit 2204 can separate the multiplexed data into coded audio data and coded video data. As mentioned above, in other practical scenarios, such as video conferencing systems, coded audio data and coded video data are not multiplexed. In this case, the coded data is not transmitted to video decoder 3206 and audio decoder 2208 via demultiplexing unit 2204.
[0149] Through the demultiplexing process, a video elementary stream (ES), an audio ES, and optional subtitles are generated. The video decoder 2206, including the video decoder 30 described in the above embodiment, decodes the video ES by the decoding method shown in the above embodiment to generate video frames, and sends the data to the synchronization unit 2212. The audio decoder 2208 decodes the audio ES to generate audio frames, and sends the data to the synchronization unit 2212. Alternatively, the video frames can be stored in a buffer (not shown in the figure) before being sent to the synchronization unit 2212. Similarly, the audio frames can be stored in a buffer (not shown in the figure) before being sent to the synchronization unit 2212.
[0150] The synchronization unit 2212 synchronizes the video frames and audio frames and provides the video / audio to the video / audio display 3214. For example, the synchronization unit 2212 synchronizes the presentation of the video and audio information. The information can be encoded in the syntax using the timestamps associated with the representation of the coded audio and visual data and the timestamps associated with the transmission of the data stream.
[0151] If subtitles are included in the bitstream, the subtitle decoder 2210 decodes the subtitles, synchronizes them with the video frames and audio frames, and provides the video / audio / subtitles to the video / audio / subtitle display 2216.
[0152] The present invention is not limited to the above-mentioned system. The image encoding device or the image decoding device in the above-mentioned embodiments can be used in other systems such as automobiles.
[0153] The following describes a streaming media system applicable to an embodiment of the present application in conjunction with Figure 3A. Figure 3A is a schematic diagram of a workflow of a streaming media system applicable to an embodiment of the present application.
[0154] The streaming media system includes a content creation module that generates the required content data, such as video or audio. The streaming media system also includes a video encoding module that encodes the generated content through an encoder. The streaming media system also includes a video stream transmission module that transmits the encoded video in the form of a code stream. Optionally, the format of the video stream can be converted into a code stream format of a commonly used transmission protocol for OTT (over-the-top) devices, for example, the protocol includes but is not limited to Real Time Streaming Protocol (RTSP), Hyper Text Transfer Protocol (HTTP), HTTP Live streaming protocol (HLS), MPEG HTTP Dynamic Adaptive Streaming over HTTP (MPEG-DASH), Real-time Transport Protocol (RTP), Real Time Messaging Protocol (RTMP) or any combination thereof. Optionally, the video stream can be stored to store the original format of the video stream and / or the converted multiple code stream formats for easy use. Furthermore, the streaming system also includes a video stream encapsulation module for encapsulating the video stream to generate an encapsulated video stream. The encapsulated video stream can be referred to as a video streaming package. Exemplarily, the video streaming package can be generated based on a transcoded video stream or a stored video stream. Furthermore, the streaming system also includes a content distribution network (CDN) for distributing the video streaming package to multiple OTT devices, such as mobile phones, computers, tablets, and home projectors.
[0155] It should be noted that video encoding, video streaming transmission, video stream transcoding, video stream storage, video streaming package generation and content distribution network can all be implemented on cloud servers.
[0156] An exemplary streaming media system architecture of the present application is described below in conjunction with FIG3B . The streaming media system architecture includes: a client device, a content distribution network, and a cloud server.
[0157] The user on the client device sends a play or playback request to the cloud platform. Optionally, the content of the request can be the title of the movie or TV program to be played.
[0158] The cloud platform makes a decision and responds to the client, sending the client the address of the requested content on the CDN. Optionally, the content sent to the client can be a URL link (uniform resource locator). Specifically, the playback application service in the cloud platform checks user authorization and permissions, and then considers individual client characteristics and current network conditions to determine which specific files are needed to process the playback request. It should be noted that the content delivery network (CDN) regularly reports its operating status, learned routes, and available content (files) to the cache control service in the cloud platform.
[0159] The client then requests the CDN to play the content based on the address, and the CDN provides the content to the client, ultimately completing the client's request.
[0160] The following describes the system architecture applicable to the embodiment of the present application in conjunction with Figure 4A. Figure 4A is a schematic diagram of a possible system architecture applicable to the embodiment of the present application. The system architecture of the embodiment of the present application includes: front-end equipment, transmission links, and terminal display equipment.
[0161] Among them, the front-end equipment is used to capture or produce HDR / SDR content (for example, HDR / SDR video or images).
[0162] In a possible embodiment, the front-end device can also be used to extract corresponding metadata from the HDR content. The metadata may include global mapping information, local mapping information, and dynamic metadata and static metadata corresponding to the HDR content. The front-end device can send the HDR content and metadata to the terminal display device via a transmission link. Specifically, the HDR content and metadata can be transmitted in the form of one data packet, or respectively in two data packets, which is not specifically limited in the embodiment of the present application.
[0163] Optionally, the terminal display device can be used to receive metadata and HDR content, and extract the global mapping information, local mapping information, and the terminal display device information contained in the corresponding metadata according to the HDR content, obtain a mapping curve to perform global tone mapping and local tone mapping on the HDR content, convert it into display content adapted for the HDR display device or SDR device in the terminal display device, and display it. It should be understood that in different embodiments, the terminal display device may include a display device with a display capability of a lower dynamic range or a higher dynamic range than the HDR content generated by the front-end device, and this application is not limited thereto.
[0164] Optionally, the front-end device and the terminal display device in this application can be independent and different physical devices. For example, the front-end device can be a video capture device or a video production device, where the video capture device can be a video camera, a camera, an image rendering machine, etc. The terminal display device can be a device with video playback function, such as virtual reality (VR) glasses, a mobile phone, a tablet, a television, a projector, etc.
[0165] Optionally, the transmission link between the front-end device and the terminal display device can be a wireless connection or a wired connection, wherein the wireless connection can adopt technologies such as long term evolution (LTE), fifth generation (5G) mobile communications, and future mobile communications. Wireless connections can also include wireless fidelity (WiFi), Bluetooth, near field communication (NFC), and other technologies. Wired connections can include Ethernet connections, local area network connections, etc. There is no specific limitation on this.
[0166] The present application may also integrate the functions of the front-end device and the terminal display device into the same physical device, for example, a terminal device such as a mobile phone or tablet with a video capture function. The present application may also integrate some functions of the front-end device and some functions of the terminal display device into the same physical device. This is not specifically limited.
[0167] The following describes an end-to-end image processing system provided by an embodiment of the present application in conjunction with Figure 4B. This system can be applied to the system architecture shown in Figure 4A. Figure 4B is a schematic diagram of the structure of an image processing system provided by an embodiment of the present application. In Figure 4B, HDR video is used as an example of HDR / SDR content. The image processing system includes: an HDR preprocessing module, an HDR video encoding module, an HDR video decoding module, and a tone mapping module.
[0168] Among them, the HDR preprocessing module and the HDR video encoding module can be located in the front-end device shown in Figure 4A, and the HDR video decoding module and the tone mapping module can be located in the terminal display device shown in Figure 4A.
[0169] HDR pre-processing module: This module is used to extract dynamic metadata (e.g., maximum, minimum, average, and range of brightness) from HDR video, determine mapping curve parameters based on the dynamic metadata and the display capabilities of the target display device, write the mapping curve parameters into the dynamic metadata to obtain HDR metadata, and transmit the metadata. HDR video can be either captured or processed by a colorist; the display capabilities of the target display device are the brightness range that the target display device can display.
[0170] HDR video encoding module: used to encode HDR video and HDR metadata according to the video compression standard (for example, AVS or HEVC standard) (for example, embedding HDR metadata into the user-defined part of the bitstream) and output the corresponding bitstream (AVS or HEVC bitstream).
[0171] HDR video decoding module: used to decode the generated bitstream (AVS bitstream or HEVC bitstream) according to the standard corresponding to the bitstream format, and output the decoded HDR video and HDR metadata.
[0172] Tone mapping module: used to generate a mapping curve according to the parameters of the mapping curve in the decoded HDR metadata, and perform tone mapping on the decoded HDR video (i.e. HDR adaptation processing or SDR adaptation processing), and send the HDR adapted video after tone mapping to the HDR display terminal for display, or send the SDR adapted video to the SDR display terminal for display.
[0173] Exemplarily, the HDR pre-processing module may exist in a video acquisition device or a video production device.
[0174] Exemplarily, the HDR video encoding module may exist in a video acquisition device or a video production device.
[0175] Exemplarily, the HDR video decoding module may be present in a set-top box, a television display device, a mobile terminal display device, and a video conversion device for live broadcasting, online video applications, and the like.
[0176] For example, the tone mapping module may be present in a set-top box, a television display device, a mobile terminal display device, or a video conversion device for a live webcast or online video application. More specifically, the tone mapping module may be present in the form of a chip or software program in the set-top box, television display, or mobile terminal display, and may be present in the form of a software program in the video conversion device for a live webcast or online video application.
[0177] In a possible embodiment, when the tone mapping module and the HDR video decoding module are both present in a set-top box, the set-top box can complete the functions of receiving, decoding, and tone mapping the video stream. The set-top box sends the decoded video data to a display device through a high-definition multimedia interface (HDMI) for display, so that the user can enjoy the video content.
[0178] The following describes the process of tone mapping performed by a terminal display device with reference to specific drawings.
[0179] Please refer to FIG5 , which is a flowchart of a tone mapping method provided in an embodiment of the present application. The method can be executed by a terminal display device and specifically includes the following steps:
[0180] Step S501: Acquire an image to be displayed.
[0181] In the embodiments of the present application, the image to be displayed may be an HDR image or video, or an SDR image or video. The image to be displayed may be an optical signal (i.e., a linear signal) or an electrical signal (i.e., a nonlinear signal), which is not limited in this application. If it is an electrical signal, the electrical signal may be converted into an optical signal according to a photoelectric signal conversion function.
[0182] In a possible embodiment, the image to be displayed is transmitted in the form of a code stream. The terminal display device can receive the code stream from the front-end device and decode the code stream to obtain data of the image to be displayed.
[0183] Optionally, the data of the image to be displayed includes image data, such as pixel data. It should be understood that the embodiment of the present application does not limit the format of the image, and the image data can be in any color space, such as RGB or YUV, and the image data can be of any bit width, such as 8 bits, 10 bits, or 12 bits.
[0184] Optionally, the data of the image to be displayed includes metadata. It should be understood that the embodiment of the present application does not limit the format of the metadata. The metadata includes data such as the data format of the image or video, area division information, area traversal order information, image features or mapping curve parameters, and one or more metadata information units. The metadata information unit includes data such as coordinate information, image features and curve parameters. Exemplarily, the metadata may include reference information of the image to be displayed, such as reference display device size, reference viewing distance, reference viewing angle, reference ambient light brightness, or reference display device resolution, etc. The metadata may also include tone mapping information, such as a tone mapping curve, etc.
[0185] Optionally, a decoding device (such as a set-top box) receives a code stream from a front-end device, decodes the code stream, obtains data of the image to be displayed, and then sends the decoded data of the image to be displayed to a terminal display device.
[0186] Optionally, the process of decoding the code stream includes decoding the received code stream by a decoder to obtain data of the image to be displayed. Exemplarily, the decoder may be a HEVC decoder, a JPEG decoder, a ProRes decoder, or a VVC decoder.
[0187] In another possible embodiment, the image to be displayed is an image that can be processed by software, and the software processing includes but is not limited to tone mapping, dehaze, deblur, scaling, color gamut conversion, and neural network-based graphics processing.
[0188] In another possible embodiment, the image to be displayed may be an image that has been processed by hardware, and the hardware processing includes but is not limited to tone mapping, dehaze, deblur, scaling, color gamut conversion, and neural network-based graphics processing.
[0189] Step S502 : Acquire reference information; the reference information includes at least one of a reference display device size, a reference viewing distance, a reference viewing angle, a reference ambient light brightness, or a reference display device resolution.
[0190] In an embodiment of the present application, reference information of an image to be displayed is obtained. This reference information is used to perform tone mapping on the image to improve the user's viewing experience. The reference information includes at least one of a reference display device size, a reference viewing distance, a reference viewing angle, a reference ambient light brightness, or a reference display device resolution. Optionally, this reference information can be combined with terminal device information to adjust a reference tone mapping curve.
[0191] Optionally, the reference display device size may be the size of the display device used when producing the image to be displayed. Optionally, the reference display device size includes at least one of the length, width or diagonal length of the display device, which is not limited in this application. Optionally, the reference display device size may be obtained by converting at least one of the length, width or diagonal length of the display device, which is not limited in this application. Among them, optionally, the reason for the conversion may be an appearance reason, such as bending, folding, etc., or a visual reason, such as the change in display size caused by vision correction of a VR device. Optionally, the reference display device size may be obtained from the metadata of the image to be displayed; an index of the reference display device size may also be obtained from the metadata, and a reference device size may be selected from a plurality of preset reference display device sizes according to the index; or the preset reference display device size may be directly used as the reference display device size. The index of the reference display device size may correspond to different standards to correspond to the display device sizes specified by different standards, wherein the standards include, for example, 1920×1080HD, 1280×720HD 720P, 3840×2160Ultra HD, etc.
[0192] Optionally, the reference viewing distance may be the viewing distance of a reference display device used when producing the image to be displayed. Optionally, the reference viewing distance may be the straight-line distance between the user and the center point of the display device, the vertical distance between the user and the imaging plane of the display device, or the straight-line distance between the user and a preset position on the imaging plane of the display device, and this application does not limit this. Optionally, the reference viewing distance may be obtained from the metadata of the image to be displayed; an index of the reference viewing distance may be obtained from the metadata, and a reference viewing distance may be selected from a plurality of preset reference viewing distances based on the index; or the preset reference viewing distance may be directly used as the reference viewing distance.
[0193] Optionally, the reference viewing angle may be the viewing angle range of a reference display device used when producing the image to be displayed. Optionally, the reference viewing angle may be an angular range that deviates from a perpendicular line perpendicular to the center point of the display device. This angular range is the angular range within which a user can accurately observe the image to be displayed, and this application does not impose any restrictions thereon. Optionally, the reference viewing angle may be obtained from the metadata of the image to be displayed; an index of the reference viewing angle may be obtained from the metadata, and a reference viewing angle may be selected from a plurality of preset reference viewing angles based on the index; or a preset reference viewing angle may be directly used as the reference viewing angle.
[0194] Optionally, the reference ambient light brightness may be the ambient light brightness used when the image to be displayed was created. Alternatively, the reference ambient light brightness may be obtained from metadata of the image to be displayed; an index of the reference ambient light brightness may be obtained from the metadata, and a reference ambient light brightness may be selected from a plurality of preset reference ambient light brightnesses based on the index; or a preset reference ambient light brightness may be directly used as the reference ambient light brightness.
[0195] Optionally, the reference display device resolution may be the resolution of the reference display device used when producing the image to be displayed. Optionally, the numerical value of the reference display device resolution may be obtained from the metadata of the image to be displayed; an index of the reference display device resolution may be obtained from the metadata, and a reference display device resolution may be selected from a plurality of preset reference display device resolutions according to the index; or a preset reference display device resolution may be directly used as the reference display device resolution. The index of the reference display device resolution may correspond to different standards, to correspond to the display device resolutions specified by different standards, such as standard 1920x1080HD, 1280x720HD 720P, 3840x2160Ultra HD, etc.
[0196] Step S503: Acquire terminal device information; the terminal device information includes at least one of the terminal display device size, user viewing distance, user viewing angle, ambient light brightness, or terminal display device resolution.
[0197] In an embodiment of the present application, terminal device information is obtained and used to adjust the reference tone mapping curve in combination with reference information.
[0198] Optionally, the terminal display device size includes at least one of the length, width, or diagonal length of the terminal display device, which is not limited in this application. Optionally, the terminal display device size can be obtained by converting at least one of the length, width, or diagonal length of the display device, which is not limited in this application. Optionally, the reason for the conversion can be an appearance reason, such as bending, folding, etc., or a visual reason, such as the change in display size caused by vision correction of a VR device. Optionally, the terminal device size can be obtained by actually measuring the terminal device, and the terminal device size can be selected from a plurality of preset terminal display device sizes according to a preset index; the preset terminal display device size can also be directly used as the terminal display device size. The index can correspond to different standards to correspond to the display device sizes specified by different standards, wherein the standards include 1920×1080HD, 1280×720HD 720P, 3840×2160Ultra HD, etc.
[0199] Optionally, the user viewing distance can be the straight-line distance between the user and the center point of the terminal display device, the vertical distance between the user and the imaging plane of the terminal display device, or the straight-line distance between the user and a preset position on the imaging plane of the terminal display device. This application does not limit this. Optionally, the viewing distance of a single user can be obtained through actual measurement; or multiple user viewing distances can be obtained through actual measurement, and then a representative user viewing distance can be selected by calculation. For example, the center of multiple user viewing distances can be selected as the user viewing distance. Optionally, the user viewing distance can be selected from multiple preset user viewing distances according to a preset index; or the preset user viewing distance can be directly used as the user viewing distance. Optionally, when the number of users is single, the viewing distance of a single user can be obtained through actual measurement; when the number of users is multiple, the preset user viewing distance can be directly used as the user viewing distance.
[0200] Optionally, the user viewing angle may be an angular range of deviations from a perpendicular line perpendicular to the center point of the display device. Optionally, the viewing distance of a single user may be obtained through actual measurement; or the viewing distances of multiple users may be obtained through actual measurement, and then a representative user viewing angle may be selected by calculation. For example, the average angle of multiple user viewing angles may be selected as the user viewing angle. Optionally, the user viewing angle may be selected from multiple preset user viewing angles according to a preset index; or the preset user viewing angle may be directly used as the user viewing angle. Optionally, when the number of users is single, the viewing angle of a single user may be obtained through actual measurement; when the number of users is multiple, the preset user viewing angle may be directly used as the user viewing angle.
[0201] Optionally, the ambient light brightness may be obtained by measuring the ambient light brightness through an actual sensor; the ambient light brightness may be selected from a plurality of preset ambient light brightnesses according to a preset index; or the preset ambient light brightness may be directly used as the ambient light brightness.
[0202] Optionally, the terminal display device resolution may be an actual value of the terminal display device resolution; the terminal display device resolution may be selected from a plurality of preset terminal display device resolutions according to an index; or the preset terminal display device resolution may be directly used as the terminal display device resolution. The index of the terminal display device resolution may correspond to different standards to correspond to display device resolutions specified by different standards, such as standard 1920×1080HD, 1280×720HD 720P, 3840×2160Ultra HD, etc.
[0203] In a possible embodiment, the tone mapping method further includes adjusting the reference tone mapping parameters of the image to be displayed according to the reference tone mapping parameters and the display capability of the terminal display device to obtain initial tone mapping parameters.
[0204] In the embodiment of the present application, the reference tone mapping parameters are used to characterize the reference tone mapping curve, and the initial tone mapping parameters are used to characterize the initial tone mapping curve.
[0205] First, obtain the reference tone mapping parameters of the image to be displayed. Optionally, the reference tone mapping parameters can be obtained from the metadata of the image to be displayed. For example, the metadata can include multiple reference display device display capability information, such as peak brightness, and the initial global tone mapping curve parameters for tone mapping at the peak brightness. The metadata can also include histogram information and tone mapping curve parameter information as in standard ST2094-40. It can also include tone mapping curve parameter information as in standard ST2094-10. Optionally, the index of the reference tone mapping parameter can be obtained from the metadata, and the reference tone mapping parameter can be selected from the parameters of multiple preset reference tone mapping curves based on the index. The preset reference tone mapping parameter can also be directly used as the reference tone mapping parameter. The reference tone mapping curve parameter can be obtained from the metadata information, such as from the SEI encapsulated by HEVC or VVC, a user-defined NAL unit, or a reserved packet unit, from the APP extension information encapsulated by JFIF, or from the data segment encapsulated by MP4. This application does not limit this.
[0206] Then, the display capability of the terminal display device is obtained from the terminal device information, such as the peak brightness and the dimming brightness of the terminal display device. Optionally, the display capability of the terminal display device's screen can be obtained as the terminal display device's display capability; the terminal display device's display capability can also be selected from multiple preset terminal display device display capabilities based on a preset index; or the preset terminal display device's display capability can be directly used as the terminal display device's display capability.
[0207] Optionally, the display capabilities of a reference display device in the reference information may be obtained for adjusting the reference tone mapping parameters. The display capabilities of the reference display device may be the display capabilities of the reference display device used when producing the image to be displayed, such as the peak brightness and darkest brightness of the reference display device. Optionally, the display capabilities of the reference display device may be obtained from the metadata of the image to be displayed; an index of the display capabilities of the reference display device may also be obtained from the metadata, and a reference display capability may be selected from multiple preset reference display capabilities based on the index; or a preset reference display capability may be directly used as the reference display capability.
[0208] Finally, according to the reference tone mapping parameters and the display capability of the terminal display device, the reference tone mapping parameters of the image to be displayed are adjusted to obtain the initial tone mapping parameters.
[0209] In a possible embodiment, the initial tone mapping curve parameter information may be acquired according to the parameter information of the reference tone mapping curve and the display capability information of the terminal display device.
[0210] Optionally, after obtaining the display capability information of the terminal display device, one or more appropriate preset display device information and corresponding one or more reference tone mapping curve parameters are selected based on the display capability of the terminal display device. Initial tone mapping curve parameters are then obtained based on the one or more reference tone mapping curve parameters to adapt the display capability information of the terminal display device to the terminal display device.
[0211] Optionally, the metadata may include information about multiple preset display devices, such as peak brightness, and the initial tone mapping curve parameters for tone mapping at the peak brightness. This application does not limit the format of the metadata. The metadata may include histogram information and tone mapping curve parameter information as in standard ST2094-40, or may include tone mapping curve parameter information as in standard ST2094-10.
[0212] Optionally, multiple preset display device information, such as peak brightness, etc., and reference tone mapping curve parameters for tone mapping based on the peak brightness, may be locally preset values.
[0213] Optionally, when a reference tone mapping curve parameter is obtained, the reference tone mapping curve parameter may be adjusted to adapt to the terminal display device information.
[0214] Optionally, when multiple reference tone mapping curve parameters are obtained, the initial tone mapping curve parameters may be obtained by weighted averaging according to the relationship between multiple preset display device information and terminal display device information corresponding to the multiple reference tone mapping curves.
[0215] Optionally, the tone mapping curve may be calculated based on the terminal display device information based on partial parameter information of the reference tone mapping curve, wherein the partial parameter information of the reference tone mapping curve may be carried in metadata or may be locally preset values.
[0216] In another possible embodiment, initial tone mapping curve parameter information can be obtained based on reference tone mapping curve parameter information, image feature information, and terminal device information. The image feature information includes content feature information, such as maximum, minimum, and average brightness, as well as feature information related to a preset percentage distribution of the image. The reference tone mapping curve parameter information can be included in metadata or locally preset values, and the image feature information can be included in metadata.
[0217] In another possible embodiment, the initial tone mapping curve parameters may be calculated based on the content feature information of the image to be displayed in the metadata. Optionally, some parameter values of the initial tone mapping curve may be preset values and do not need to be calculated.
[0218] It should be noted that the tone mapping curve includes various forms, such as Sigmoid function curve, cubic spline function curve, Gamma function, etc., which are not limited in the present invention.
[0219] For example, the following function can be used to obtain the initial tone mapping curve:
[0220] Based on the above curve, a mapping relationship can be obtained from normalized HDR / SDR source data to normalized HDR display data. L can be R, G, B in the RGB color gamut, or the maximum value max(R,G,B) among R, G, B, or the brightness Y in the YUV color gamut, or the luminance component L in the Lab color gamut. L and L' can be normalized optical signals or electrical signals, which are not limited in the present invention.
[0221] Exemplarily, m=2.4, n=1, and b=the darkest brightness of the terminal display device in the above function can be preset.
[0222] The process of obtaining the parameter p in the above function based on the image feature information and the preset relationship is as follows: first, the feature information of the image is obtained from the metadata, including the maximum value maxL, the minimum value minL, the average value avgL, etc. Then, p is obtained based on the image feature information and the preset relationship.
[0223] Optionally, you can get p based on avgL and a preset relationship:
[0224] Among them, P valueH0 、P valueL0 , TPH0, TPL0 are all preset parameters, you can set P valueH0 =3.5, P valueL0 =4.0, TPH0=0.6, TPL0=0.3.
[0225] Optionally, you can get p based on maxL and a preset relationship:
[0226] in PdeltaH1 、P deltaL1 , TPH1, TPL1 are all preset parameters, you can set P deltaH1 =3.5, P deltaL1 =4.0, TPH1=0.6, TPL1=0.3.
[0227] Optionally, p can be adjusted based on maxL and a preset relationship:
[0228] According to the terminal display device information, the characteristic information of the image obtained from the metadata, and other curve parameters, a can be calculated:
[0229] The peak brightness of the terminal display device is MaxDisplay, the darkest brightness is MinDisplay, and the maximum value of the feature information of the image obtained in the metadata is maxL.
[0230] Step S504: Perform tone mapping on the image to be displayed based on the reference information and the corresponding terminal device information. In this embodiment of the present application, when tone mapping the image to be displayed based on the reference information and the corresponding type of terminal device information, the reference display device size corresponds to the terminal display device size, the reference viewing distance corresponds to the user viewing distance, the reference viewing angle corresponds to the user viewing angle, the reference ambient light brightness corresponds to the ambient light brightness, and the reference display device resolution corresponds to the terminal display device resolution.
[0231] In the embodiment of the present application, when tone mapping is performed on an image, other information such as the terminal display device size and the user's viewing distance is also taken into account, thereby improving the display adaptation effect.
[0232] In a possible embodiment, tone mapping is performed on the image to be displayed based on the reference information and the terminal device information of the corresponding type, including: obtaining tone mapping parameters based on the reference information and the terminal device information of the corresponding type; and tone mapping is performed on the image to be displayed based on the tone mapping parameters.
[0233] In a possible embodiment, obtaining tone mapping parameters according to the reference information and the terminal device information of the corresponding type includes: adjusting the initial tone mapping parameters of the image to be displayed according to the reference information and the terminal device information of the corresponding type to obtain the tone mapping parameters.
[0234] In an embodiment of the present application, the initial tone mapping parameters of the image to be displayed are adjusted according to the reference information and the terminal device information of the corresponding type to obtain the adjusted tone mapping parameters, including adjusting the initial tone mapping parameters of the image to be displayed in different ways when the relationship between the reference information and the terminal device information meets different conditions to obtain the adjusted tone mapping parameters.
[0235] In one possible embodiment, adjusting initial tone mapping parameters of the image to be displayed based on the reference information and the corresponding type of terminal device information to obtain the tone mapping parameters includes adjusting the initial tone mapping parameters of the image to be displayed in a first manner to obtain the tone mapping parameters when the relationship between the reference information and the terminal device information satisfies a first condition. When the relationship between the reference information and the corresponding type of terminal device information satisfies a second condition, adjusting the initial tone mapping parameters of the image to be displayed in a second manner to obtain the tone mapping parameters. The initial tone mapping parameters are used to represent an initial tone mapping curve, and the tone mapping parameters are used to represent an adjusted tone mapping curve.
[0236] Optionally, the first condition includes at least one of: the size of the terminal display screen is smaller than the size of the reference display device, the user viewing distance is greater than the reference viewing distance, the user viewing angle deviates from the reference viewing angle, the ambient light brightness is greater than the reference ambient light brightness, or the terminal display device resolution is smaller than the reference display device resolution. It should be understood that when the relationship between the reference information and the terminal device information meets the first condition, it is necessary to adjust the initial tone mapping parameters so that the overall brightness of the image after tone mapping based on the adjusted tone mapping parameters is improved, and the clarity of details is improved, such as local contrast enhancement, edge sharpness improvement, more high-frequency component retention, or pixel gradient improvement, so that the user perception is more consistent and the user experience is improved.
[0237] When the relationship between the reference information and the terminal device information satisfies the first condition, there are three ways to adjust the initial tone mapping parameters of the image to be displayed in a first manner: Method 1, Method 2, and Method 3. This application may select one or more of Method 1, Method 2, and Method 3 for execution, and this application does not limit this. When executing multiple methods, this application does not limit the execution order of different methods. Specific adjustment methods include:
[0238] Method 1: Adjust the initial tone mapping parameters so that the mapping value of the adjusted tone mapping curve at the one or more first brightness characteristic values is greater than the mapping value of the initial tone mapping curve at the one or more first brightness characteristic values.
[0239] In an embodiment of the present application, it is first necessary to obtain one or more first brightness characteristic values of the image to be displayed. Optionally, the one or more first brightness characteristic values can be the average brightness value of the entire frame of the image to be displayed, the average brightness value of one or more regions of interest of the image to be displayed, the weighted average brightness value of multiple regions of the image to be displayed, or the average value of one or more brightness ranges after the brightness histogram of the image to be displayed is divided into multiple brightness ranges. This application does not limit this. Among them, the one or more regions of interest of the image to be displayed can be selected from characteristic regions of the image to be displayed, such as faces, green plants, red flags, areas with high complexity, areas with obvious noise, and non-flat areas. This application does not limit the method for selecting characteristic regions. Optionally, the method for selecting the selected region can be obtained from metadata, based on preset rules, or directly based on the characteristics of the image to be displayed. Optionally, the average value can be calculated as an arithmetic mean or a geometric mean, etc., which is not limited in this application.
[0240] For example, the average value of the brightness of the entire frame of the image to be displayed can be selected as the first brightness characteristic value, and the calculation formula is as follows:
[0241] Among them L avg is the average brightness of the entire frame of the image to be displayed, L n,m L is any brightness value in the entire frame of the image to be displayed, width and height are the width and height of the image to be displayed, and n and m are the pixel indexes in the width direction and height direction of the image to be displayed respectively. n,m It can be R, G, B in the RGB color gamut, or the maximum value max(R,G,B) among R, G, B, or the brightness Y in the YUV color gamut, or the luminance component L in the Lab color gamut. L and L' can be normalized optical signals or electrical signals, which is not limited in the present invention.
[0242] Then, the initial tone mapping parameters are adjusted so that a mapping value of the obtained adjusted tone mapping curve at the one or more first brightness characteristic values is greater than a mapping value of the initial tone mapping curve at the one or more first brightness characteristic values.
[0243] In a possible embodiment, the process of adjusting the initial tone mapping parameters is: first obtain the mapping value of the initial tone mapping curve at the one or more brightness characteristic values. Exemplarily, when the initial tone mapping curve is LY=h(L), L can be R, G, B in the RGB color domain, or the maximum value max(R,G,B) of R, G, B or the brightness Y in the YUV color domain, or the brightness component L in the Lab color domain. L and L' can be normalized light signals or electrical signals, which are not limited in the present invention. LY is the mapping value of L after mapping by the tone mapping curve, and the curve is the average value L of the brightness of the entire frame of the image to be displayed. avg The mapping value at LY avg =h(L avg ). Optionally, one or more parameters may be directly adjusted to increase the mapping value of the adjusted tone mapping curve at one or more first brightness characteristic values.
[0244] Optionally, based on the reference information and the terminal display device information, a new mapping value at one or more first brightness characteristic values can be obtained according to a preset algorithm; and based on the obtained one or more new mapping values, an adjusted tone mapping parameter can be obtained. Optionally, the preset algorithm can be a preset priori function, and a gain coefficient P corresponding to the reference information and the terminal display device information can be obtained according to the preset priori function. Then, the new mapping value LYN avg =P*LY avg Optionally, the preset a priori function may be in the form of a preset lookup table of reference information, terminal display device information, and gain coefficient P.
[0245] Optionally, an equation or set of equations for tone mapping parameters may be established based on one or more new mapping values to obtain one or more parameters of the adjusted tone mapping curve, thereby obtaining all parameters of the adjusted tone mapping curve in combination with other preset parameters. Optionally, the equation or set of equations for establishing the tone mapping curve based on one or more new mapping values may be based on a tone mapping curve function. For example, when the initial tone mapping curve function is LY=h(L), the equation is LYN avg =h(L avg ); It can also be based on the inverse tone mapping curve function, for example, when the inverse initial tone mapping curve function is L = h -1 (LY), the equation is L avg =h -1 (LYN avg ). Optionally, one or more parameters may be directly adjusted so that the mapping values of the adjusted tone mapping curve at one or more first brightness characteristic values are the one or more new mapping values.
[0246] For example, the tone mapping curve adjusted using Method 1 and the initial tone mapping curve are shown in FIG6 , which illustrates an example of an initial tone mapping parameter adjustment method provided in an embodiment of the present application. The initial tone mapping curve represented by the initial tone mapping curve parameters is curve 601. The average brightness 6 of the entire image is selected as the first brightness characteristic value. After updating the parameters, the adjusted tone mapping curve is 602. The mapping value of the adjusted tone mapping curve 602 at the first brightness characteristic value 6 is greater than the mapping value of the initial tone mapping curve 601 at the first brightness characteristic value 6. Furthermore, the overall mapping value of the adjusted tone mapping curve 602 is greater than the overall mapping value of the initial tone mapping curve 601. This increases the overall brightness of the tone-mapped image based on the adjusted tone mapping parameters, enabling more consistent user perception and better display adaptation when the terminal display screen size is smaller than the reference display device size, the user viewing distance is greater than the reference viewing distance, the user viewing angle deviates from the reference viewing angle, the ambient light brightness is greater than the reference ambient light brightness, or the terminal display device resolution is smaller than the reference display device resolution.
[0247] Mode 2 adjusts the initial tone mapping parameters so that the slope of the adjusted tone mapping curve at the one or more second brightness characteristic values is greater than the slope of the initial tone mapping curve at the one or more second brightness characteristic values.
[0248] In this embodiment of the present application, it is first necessary to obtain one or more second brightness characteristic values of the image to be displayed. The method for selecting the one or more second brightness characteristic values is similar to the method for selecting the one or more first brightness characteristic values in Method 1 and is not further described here. Optionally, the one or more second brightness characteristic values may be partially or completely identical to the one or more first brightness characteristic values, or may be completely different.
[0249] Then, the parameters of the initial tone mapping curve are adjusted so that the slope of the adjusted tone mapping curve at one or more second luminance characteristic values is greater than the slope of the initial tone mapping curve at the one or more second luminance characteristic values. Alternatively, one or more parameters may be directly adjusted so that the slope of the adjusted tone mapping curve at the one or more second luminance characteristic values is increased.
[0250] For example, first, the first-order derivative d(LY) / dL=h'(L) is obtained according to the initial tone mapping curve function LY=h(L), and the curve parameter is P curve , the second brightness eigenvalue L avg The corresponding slope value is LY` avg =h`(L avg ), then, based on the reference information and the terminal display device information, the second brightness characteristic value L is obtained according to the preset algorithm avgThe new curve parameter is P1 curve The new curve parameter P1 curve So that h`(P1 curve , L avg )>h`(P curve , L avg ). Among them, the preset algorithm can avg =h`(L avg ) is a variable, and the change of the curve parameters and LY` avg and select the appropriate curve parameter as the adjustment parameter P curve .
[0251] For example, when the initial tone mapping curve is the following function curve:
[0252] Where L is the brightness, L' is the mapping value after mapping by the tone mapping curve, a, p, m, n, b are the parameters of the tone mapping curve, L can be R, G, B in the RGB color space, or the maximum value max(R,G,B) among R, G, B, or the brightness Y in the YUV color space, or the luminance component L in the Lab color space. L and L' can be normalized optical signals or electrical signals, which is not limited in the present invention.
[0253] Based on the current values of the curve parameters p and a, the slope value may be increased by reducing p and increasing a to obtain the adjusted tone mapping parameters.
[0254] Optionally, based on the reference information and the terminal display device information, a new slope at one or more first brightness characteristic values can be obtained according to a preset algorithm; an equation or set of equations for the tone mapping curve parameters is established based on the one or more new slopes, and one or more parameters of the adjusted tone mapping curve are obtained by solving the equations, thereby combining other preset parameters to obtain all parameters of the adjusted tone mapping curve. Optionally, the preset algorithm can be a preset priori function, and the gain coefficient Q corresponding to the reference information and the terminal display device information can be obtained according to the preset priori function. Then, the new slope LYN` avg =Q*LY` avg Optionally, the preset a priori function may be in the form of a preset lookup table of reference information, terminal display device information, and gain coefficient Q. For example, when the function of the initial tone mapping curve is LY=h(L), its first-order derivative is d(LY) / dL=h`(L), and the equation is LYN` avg =h`(L avg); an equation or system of equations may also be obtained based on the inverse function of the first-order derivative of the tone mapping curve function, which is not limited here. Optionally, one or more parameters may be directly adjusted so that the slope of the adjusted tone mapping curve at one or more first luminance feature values is the one or more new slopes described above.
[0255] For example, the tone mapping curve adjusted using Method 2 and the initial tone mapping curve are shown in FIG7 , which illustrates an example of an initial tone mapping parameter adjustment method provided in an embodiment of the present application. The initial tone mapping curve is curve 701, and the average brightness of the entire image, 5, is selected as the second brightness characteristic value. After the parameters are updated, the adjusted tone mapping curve is 702. The slope of the adjusted tone mapping curve 702 at the second brightness characteristic value 5 is greater than the slope of the initial tone mapping curve 701 at the second brightness characteristic value 5. Furthermore, the slopes of multiple segments of the adjusted tone mapping curve 702 are greater than the overall slope of the initial tone mapping curve 701. This increases the overall contrast and pixel gradient of the tone-mapped image based on the adjusted tone mapping parameters, thereby enhancing detail clarity. This allows for more consistent user perception and better display adaptation when the terminal display screen size is smaller than the reference display device size, when the user viewing distance is greater than the reference viewing distance, when the user viewing angle deviates from the reference viewing angle, when the ambient light brightness is greater than the reference ambient light brightness, or when the terminal display device resolution is smaller than the reference display device resolution. This improves the display quality of complex images, particularly.
[0256] Method 3: Obtain a base layer and one or more enhancement layers of the image to be displayed; a first eigenvalue of any one of the one or more enhancement layers, after being mapped by the initial tone mapping curve of any one of the enhancement layers, is a second eigenvalue; and adjust the initial tone mapping curve of any one of the enhancement layers so that the absolute value of the first eigenvalue after being mapped by the adjusted tone mapping curve is greater than the absolute value of the second eigenvalue.
[0257] In this embodiment, the first eigenvalue and the second eigenvalue can be positive or negative. When comparing the value of the first eigenvalue mapped by the adjusted tone mapping curve with the size of the second eigenvalue, the absolute values of the above two values are compared.
[0258] In an embodiment of the present application, the base layer and one or more enhancement layers of the image to be displayed are first obtained. Optionally, the enhancement layer includes at least one of an edge layer or a texture layer. Optionally, the base layer and one or more enhancement layers may be obtained by filtering with a filter, or the base layer and one or more enhancement layers may be obtained by combining image segmentation and filtering. Optionally, the filter may be a bilateral filter, a multilateral filter, etc., which is not limited in this application. Wherein, any value of any enhancement layer in the one or more enhancement layers is a first eigenvalue; the value of the first eigenvalue after being mapped by the initial tone mapping curve of any enhancement layer is a second eigenvalue.
[0259] Then, the initial tone mapping parameters of any enhancement layer are adjusted so that the absolute value of the value of the first eigenvalue mapped by the adjusted tone mapping curve is greater than the absolute value of the second eigenvalue. Optionally, the value mapped by the adjusted tone mapping curve may be scaled to N times the second eigenvalue, where N is an integer greater than 1, a fraction, or a decimal, which is not limited in this application.
[0260] Exemplarily, the image is filtered using filters in the spatial domain, temporal domain, or a combination of spatial and temporal filters to extract data from different layers.
[0261] Any luminance value L1[m] in the base layer L1 can be:
[0262] Where L is the brightness value of the original image, filter[n] is the filter coefficient at any position of the filter, and filterL is the length of the filter.
[0263] Any luminance value L2[m] (first characteristic value) in the enhancement layer L2 can be: L2[m]=L[m]-L1[m]
[0264] Where L is the brightness value of the original image, which can be R, G, B in the RGB color domain, or the maximum value max(R,G,B) among R, G, B, or the brightness Y in the YUV color domain, or the brightness component L in the Lab color domain.
[0265] Then, according to preset information or tone mapping parameter information transmitted by metadata, tone mapping curves used by multiple layers are obtained and tone mapping is performed on each layer.
[0266] For example, if the tone mapping curve obtained for the base layer is g(), then any mapping value in the base layer L1 after mapping is L1`[m], L1`[m]=g(L1[m])
[0267] Before adjustment, for the tone mapping curve f() obtained for the enhancement layer, after mapping, any mapping value L2`[m] (second eigenvalue) in the enhancement layer L2 is: L2`[m]=f(L2[m])
[0268] Expand the mapping value N times, then L2`[m] is expanded N times to L2``[m], where N is an integer or fraction greater than 1:
[0269] L2``[m]=N*L2`[m] The tone-mapped base layer mapping value L1`[m] is mixed with the tone-mapped enhancement layer L2``[m] using the preset gain coefficient gain[n] or the gain coefficient sent in the metadata to obtain the final image mapping value L`[m]: L`[m]=gain[1]*L1`[m]+gain[2]*L2``[m]=gain[1]*L1`[m]+gain[1]*N*L2`[m]
[0270] The mapping value of the enhancement layer in the final image mapping value is N times larger than that before adjustment, and the brightness of the enhancement layer in the image is increased, thereby enhancing the local contrast and detail clarity of the image, improving the user viewing experience.
[0271] Optional, where L is R, G, B in the RGB color space, L1` corresponds to RP[0], GP[0], BP[0], and L2`` corresponds to RP[1], GP[1], BP[1].
[0272] Finally, the final mapping values Rtm, Gtm, and Btm are obtained by combining the multiple layers RP[n], GP[n], and BP[n] using the preset gain coefficients or the gain coefficients sent in the metadata. Where n represents the index of the multiple layers. Rtm = ∑RP[n]*gain[n] Gtm = ∑GP[n]*gain[n] Btm = ∑BP[n]*gain[n]
[0273] For example, the tone mapping curve and the initial tone mapping curve after adjustment of the enhancement layer by method 3 are shown in FIG8 , which is an example diagram of an initial tone mapping parameter adjustment method for the enhancement layer provided in an embodiment of the present application. The tone mapping curve of the base layer is 801, and the initial tone mapping curve of the enhancement layer is curve 802. The adjustment parameters are used to double the mapping value of curve 802, and the tone mapping curve after adjustment of the enhancement layer is obtained as 803. The overall mapping value of curve 803 is doubled compared to curve 802. Therefore, based on the tone mapping parameters after the enhancement layer is adjusted, the image after tone mapping and mixing with the base layer retains more high-frequency components, the features of the enhancement layer are more obvious, the local contrast becomes larger, and the clarity of the details is improved. For example, when the enhancement layer is an edge layer, the edge sharpness is improved; when the enhancement layer is a texture layer, the texture details are clearer. Therefore, when the terminal display screen size is smaller than the reference display device size, or the user viewing distance is greater than the reference viewing distance, or the user viewing angle deviates from the reference viewing angle, or the ambient light brightness is greater than the reference ambient light brightness, or the resolution of the terminal display device is smaller than the resolution of the reference display device, the user's perception is more consistent and the display adaptation effect is better, especially improving the display effect of complex images.
[0274] In one possible embodiment, adjusting initial tone mapping parameters of the image to be displayed based on the reference information and the corresponding type of terminal device information to obtain tone mapping parameters includes adjusting the initial tone mapping parameters of the image to be displayed in a second manner to obtain tone mapping parameters when a relationship between the reference information and the terminal device information satisfies a second condition. The initial tone mapping parameters are used to represent an initial tone mapping curve, and the tone mapping parameters are used to represent an adjusted tone mapping curve.
[0275] Optionally, the second condition includes at least one of: the terminal display screen size being larger than the reference display device size, the user viewing distance being smaller than the reference viewing distance, the ambient light brightness being smaller than the reference ambient light brightness, or the terminal display device resolution being larger than the reference display device resolution. It should be understood that when the relationship between the reference information and the terminal device information satisfies the second condition, it is necessary to adjust the initial tone mapping curve parameters so that the average brightness of the image after tone mapping based on the adjusted tone mapping curve parameters is reduced, or dark areas are further darkened, in order to achieve a more consistent user perception and improve the user experience.
[0276] When the relationship between the reference information and the corresponding type of terminal device information meets the second condition, there are three ways to adjust the initial tone mapping parameters of the image to be displayed in the second manner: Method 4, Method 5, and Method 6. The application may select one or more of Method 4, Method 5, and Method 6 for execution, and this application does not limit this. When executing multiple methods, this application does not limit the execution order of different methods. Specific adjustment methods include:
[0277] Mode 4: Adjust the initial tone mapping parameters so that the mapping value of the adjusted tone mapping curve at the one or more third brightness characteristic values is greater than the mapping value of the initial tone mapping curve at the one or more third brightness characteristic values.
[0278] In the embodiment of the present application, it is first necessary to obtain one or more third brightness characteristic values of the image to be displayed. The method for selecting the one or more third brightness characteristic values is similar to the one or more first brightness characteristic values in method 1, and will not be repeated here.
[0279] Then, the parameters of the initial tone mapping curve are adjusted so that the mapping value of the adjusted tone mapping curve at the one or more third brightness characteristic values is greater than the mapping value of the initial tone mapping curve at the one or more third brightness characteristic values.
[0280] In a possible embodiment, the process of adjusting the initial tone mapping parameters is as follows: first, obtaining the mapping value of the initial tone mapping curve at the one or more brightness characteristic values. For example, when the initial tone mapping curve function is LY=h(L), the curve has an average value L of the brightness of the entire frame of the image to be displayed. avg The mapping value at LY avg =h(L avg ). Optionally, one or more parameters may be directly adjusted to reduce the mapping value of the adjusted tone mapping curve at one or more third brightness characteristic values.
[0281] Optionally, based on the reference information and the terminal display device information, a preset algorithm can be used to obtain new mapping values at one or more first brightness characteristic values; based on the obtained one or more new mapping values, the parameters of the adjusted tone mapping curve are obtained. Optionally, the preset algorithm can be a preset priori function, and the attenuation coefficient U corresponding to the reference information and the terminal display device information can be obtained based on the preset priori function. Then, the new mapping value LYN avg =U*LY avg Optionally, the preset a priori function may be in the form of a preset lookup table of reference information, terminal display device information, and attenuation coefficient U.
[0282] Optionally, an equation or set of equations for tone mapping curve parameters may be established based on one or more new mapping values to obtain one or more parameters of the adjusted tone mapping curve, thereby obtaining all parameters of the adjusted tone mapping curve in combination with other preset parameters. Optionally, the equation or set of equations for tone mapping curve parameters established based on one or more new mapping values may be based on the tone mapping curve. For example, when the initial tone mapping curve function is LY=h(L), the equation is LYN avg =h(L avg ); It can also be based on the inverse tone mapping curve function, for example, when the inverse initial tone mapping curve function is L = h -1 (LY), the equation is L avg =h -1 (LYN avg ). Optionally, one or more parameters may be directly adjusted so that the mapping values of the adjusted tone mapping curve at one or more first brightness characteristic values are the one or more new mapping values.
[0283] Optionally, the one or more third brightness characteristic values include: one or more brightness characteristic values of a dark portion of the image to be displayed, or one or more brightness characteristic values of the entire image to be displayed. The dark portion may be a brightness range below 5 nits, which is below the lowest brightness discernible by the human eye, or may be a relatively dark portion of an image obtained based on image segmentation or brightness histogram segmentation, which is not limited in this application.
[0284] For example, when the third brightness characteristic value is the average brightness of the dark portion of the image, the tone mapping curve adjusted by method 4 and the initial tone mapping curve are shown in FIG9 , which is an example diagram of an initial tone mapping parameter adjustment method provided in an embodiment of the present application. The initial tone mapping curve represented by the initial tone mapping parameters is curve 901, and the average brightness of the dark portion of the image, 4.5, is selected as the third brightness characteristic value. After the parameters are updated, the adjusted tone mapping curve is 902. The mapping value of the adjusted tone mapping curve 902 at the third brightness characteristic value of 4.5 is smaller than the mapping value of the initial tone mapping curve 901 at the third brightness characteristic value of 4.5. Furthermore, the overall mapping value of the dark portion of the adjusted tone mapping curve 902 is smaller than the overall mapping value of the initial tone mapping curve 901. As a result, the dark parts of the image after tone mapping based on the adjusted tone mapping parameters are darker. Since the human eye is more sensitive to dark parts, when the terminal display screen size is larger than the reference display device size, or the user viewing distance is smaller than the reference viewing distance, or the ambient light brightness is smaller than the reference ambient light brightness, or the resolution of the terminal display device is larger than the reference display device resolution, the human eye may see noise in the dark parts. Therefore, further darkening the dark parts can adapt to changing application scenarios, make the user's visual perception more consistent, and improve the display adaptation and display effects of the image.
[0285] For example, when the third luminance characteristic value is the average luminance value of the entire image, the tone mapping curve adjusted using Method 4 and the initial tone mapping curve are shown in FIG10 , which illustrates an example of an initial tone mapping parameter adjustment method provided in an embodiment of the present application. The initial tone mapping curve is curve 1001, and the average luminance of the entire image, 6, is selected as the third luminance characteristic value. After parameter updating, the adjusted tone mapping curve is 1002. The mapping value of the adjusted tone mapping curve 1002 at average luminance 6 is smaller than the mapping value of the initial tone mapping curve 1001 at average luminance 6. Furthermore, the overall mapping value of the adjusted tone mapping curve 1002 is smaller than the overall mapping value of the initial tone mapping curve 1001. Consequently, the overall brightness of the tone-mapped image is reduced based on the adjusted tone mapping parameters. This can improve user visual consistency and display adaptation when the terminal display screen size is larger than the reference display device size, the user viewing distance is smaller than the reference viewing distance, the ambient light brightness is smaller than the reference ambient light brightness, or the terminal display device resolution is larger than the reference display device resolution, while also protecting the user's eyes to a certain extent.
[0286] Mode 5: Adjust the initial tone mapping parameters so that the slope of the adjusted tone mapping curve at the one or more fourth brightness characteristic values is smaller than the slope of the initial tone mapping curve at the one or more fourth brightness characteristic values.
[0287] In this embodiment of the present application, it is first necessary to obtain one or more fourth brightness characteristic values of the image to be displayed. The method for selecting the one or more fourth brightness characteristic values is similar to the method for selecting the one or more first brightness characteristic values in step 1 and is not further described here. Optionally, the one or more fourth brightness characteristic values may be partially or completely identical to the one or more third brightness characteristic values, or may be completely different.
[0288] Then, the parameters of the initial tone mapping curve are adjusted so that the slope of the adjusted tone mapping curve at one or more fourth luminance eigenvalues is smaller than the slope of the initial tone mapping curve at the one or more fourth luminance eigenvalues. Alternatively, one or more parameters may be directly adjusted so that the slope of the adjusted tone mapping curve at the one or more fourth luminance eigenvalues is smaller.
[0289] For example, first, the first-order derivative d(LY) / dL=h'(L) is obtained according to the initial tone mapping curve function LY=h(L), and the curve parameter is P curve , the second brightness eigenvalue L avg The corresponding slope value is LY` avg =h`(L avg), then, based on the reference information and the terminal display device information, the second brightness characteristic value L is obtained according to the preset algorithm avg The new curve parameter is Q1 curve The new curve parameter P1 curve So that h`(P1 curve , L avg ) <h`(P curve , L avg ). Among them, the preset algorithm can avg =h`(L avg ) is a variable, and the change of the curve parameters and LY` avg and select the appropriate curve parameter as the adjustment parameter P curve .
[0290] For example, when the initial tone mapping curve is the following function curve:
[0291] Where L is the brightness, L' is the mapped value after the tone mapping curve, and a, p, m, n, b are the parameters of the tone mapping curve. L can be R, G, B in the RGB color space, or the maximum value max(R,G,B) among R, G, B, or the brightness Y in the YUV color space, or the luminance component L in the Lab color space.
[0292] Based on the current values of the curve parameters p and a, the slope value can be reduced by reducing a and increasing p to obtain the adjusted tone mapping curve parameters.
[0293] Optionally, based on the reference information and the terminal display device information, a new slope at one or more first brightness characteristic values can be obtained according to a preset algorithm; an equation or set of equations for the tone mapping curve parameters is established based on the one or more new slopes, and the solution is used to obtain one or more parameters of the adjusted tone mapping curve, thereby combining other preset parameters to obtain all parameters of the adjusted tone mapping curve. Optionally, the preset algorithm can be a preset priori function, and the attenuation coefficient V corresponding to the reference information and the terminal display device information can be obtained according to the preset priori function. Then, the new slope LYN` avg =V*LY` avg Optionally, the preset a priori function can be in the form of a preset lookup table of reference information, terminal display device information, and attenuation coefficient V. For example, when the initial tone mapping curve function is LY=h(L), its first-order derivative is d(LY) / dL=h`(L), and the equation is LYN`avg=h`(L avg); an equation or system of equations may also be obtained based on the inverse function of the first-order derivative of the tone mapping curve function, which is not limited here. Optionally, one or more parameters may be directly adjusted so that the slope of the adjusted tone mapping curve at one or more first luminance feature values is the one or more new slopes described above.
[0294] In an embodiment of the present application, the overall contrast of the image after tone mapping based on the adjusted tone mapping curve parameters becomes smaller and the pixel gradient is improved, thereby reducing the clarity of details. This can prevent the terminal display screen size from being larger than the reference display device size, or the user viewing distance from being smaller than the reference viewing distance, or the ambient light brightness from being smaller than the reference ambient light brightness, or the resolution of the terminal display device from being larger than the reference display device resolution, thereby improving the display adaptation effect and the display effect of complex images.
[0295] Method 6: Obtain a base layer and one or more enhancement layers of the image to be displayed; the value of the third eigenvalue of any one of the one or more enhancement layers after being mapped by the initial tone mapping curve of any one of the enhancement layers is a fourth eigenvalue; adjust the initial tone mapping parameters of any one of the enhancement layers so that the absolute value of the value of the third eigenvalue after being mapped by the adjusted tone mapping curve is less than the absolute value of the fourth eigenvalue.
[0296] In this embodiment, the third eigenvalue and the fourth eigenvalue can be positive or negative. When comparing the value of the third eigenvalue mapped by the adjusted tone mapping curve with the size of the fourth eigenvalue, the absolute values of the above two values are compared.
[0297] In an embodiment of the present application, the base layer and one or more enhancement layers of the image to be displayed are first obtained. Optionally, the enhancement layer includes at least one of an edge layer or a texture layer. Optionally, the base layer and one or more enhancement layers may be obtained by filtering with a filter, or the base layer and one or more enhancement layers may be obtained by combining image segmentation and filtering. Optionally, the filter may be a bilateral filter, a multilateral filter, etc., which is not limited in this application. Wherein, any value of any enhancement layer in the one or more enhancement layers is a third eigenvalue; the value of the third eigenvalue after being mapped by the initial tone mapping curve of any enhancement layer is a fourth eigenvalue.
[0298] Then, the initial tone mapping parameter of any enhancement layer is adjusted so that the absolute value of the value mapped by the adjusted tone mapping curve of the third eigenvalue is greater than the absolute value of the fourth eigenvalue.
[0299] The value mapped by the adjusted tone mapping curve may be reduced to M times the fourth eigenvalue, where M is a fraction or decimal greater than 0 and less than 1, which is not limited in this application.
[0300] Exemplarily, the image is filtered using filters in the spatial domain, temporal domain, or a combination of spatial and temporal filters to extract data from different layers.
[0301] Any luminance value L1[m] in the base layer L1 can be:
[0302] Where L is the brightness value of the original image, filter[n] is the filter coefficient at any position of the filter, and filterL is the length of the filter.
[0303] Any luminance value L2[m] (third eigenvalue) in the enhancement layer L2 can be: L2[m]=L[m]-L1[m]
[0304] Where L is the brightness value of the original image, which can be R, G, B in the RGB color domain, or the maximum value max(R,G,B) among R, G, B, or the brightness Y in the YUV color domain, or the brightness component L in the Lab color domain.
[0305] Then, according to preset information or tone mapping parameter information transmitted by metadata, tone mapping curves used by multiple layers are obtained respectively; and tone mapping is performed on each layer respectively.
[0306] For example, if the tone mapping curve obtained for the base layer is g(), then any mapping value in the base layer L1 after mapping is L1`[m], L1`[m]=g(L1[m])
[0307] Before adjustment, the tone mapping curve obtained for the enhancement layer is f(). After mapping, any mapping value L2`[m] (the fourth eigenvalue) in the enhancement layer L2 is: L2`[m]=f(L2[m])
[0308] Reduce the mapped value by M times, then L2`[m] is reduced by M times to L2``[m], where M is an integer or fraction greater than 0 and less than 1: L2``[m] = M*L2`[m]
[0309] The tone-mapped base layer mapping value L1`[m] is mixed with the tone-mapped enhancement layer L2``[m] using the preset gain coefficient gain[n] or the gain coefficient sent in the metadata to obtain the final image mapping value L`[m]: L`[m]=gain[1]*L1`[m]+gain[2]*L2``[m]=gain[1]*L1`[m]+gain[1]*M*L2`[m]
[0310] The mapping value of the enhancement layer in the final image mapping value is M times smaller than that before adjustment, and the brightness of the enhancement layer in the image is reduced, thereby weakening the local contrast and detail clarity of the image, which can prevent the user from feeling visually uncomfortable due to excessive contrast.
[0311] Optional, where L is R, G, B in the RGB color space, L1` corresponds to RP[0], GP[0], BP[0], and L2`` corresponds to RP[1], GP[1], BP[1].
[0312] Finally, the final mapping values Rtm, Gtm, and Btm are obtained by combining the multiple layers RP[n], GP[n], and BP[n] using the preset gain coefficients or the gain coefficients sent in the metadata. Where n represents the index of the multiple layers. Rtm = ∑RP[n]*gain[n] Gtm = ∑GP[n]*gain[n] Btm = ∑BP[n]*gain[n]
[0313] In an embodiment of the present application, based on the adjusted tone mapping curve parameters of the enhancement layer, the enhancement layer is tone mapped, and the image after being mixed with the base layer has a reduced mapping value of the enhancement layer, weakened features of the enhancement layer, less high-frequency components retained, and smaller local contrast, thereby reducing the clarity of details. For example, when the enhancement layer is an edge layer, the edge sharpness is reduced; when the enhancement layer is a texture layer, the texture details are reduced. Therefore, it is possible to prevent the occurrence of excessive contrast and poor display effects of images that cause user discomfort when the terminal display screen size is smaller than the reference display device size, or the user viewing distance is greater than the reference viewing distance, or the user viewing angle deviates from the reference viewing angle, or the ambient light brightness is greater than the reference ambient light brightness, or the resolution of the terminal display device is smaller than the reference display device resolution, thereby improving the display adaptation effect, especially improving the display effect of complex images and enhancing the user viewing experience.
[0314] In a possible embodiment, tone mapping is performed on the image to be displayed based on the parameters of the adjusted tone mapping curve, that is, the tone mapping parameters.
[0315] In an embodiment of the present application, tone mapping is performed on the image to be displayed based on the adjusted tone mapping parameters. This application does not limit the method and curve used in the tone mapping process. Optionally, the method used in the tone mapping process can be global tone mapping or local tone mapping, which is not limited in this application.
[0316] Exemplarily, a reference tone mapping is implemented as follows:
[0317] First, according to the adjusted tone mapping parameters, the tone mapping coefficient lookup table lut is obtained, and the lookup table lut is used to obtain the tone mapping coefficient ratio;
[0318] Then, for each pixel (R, G, B), the luminance value lum is obtained, and the tone mapping coefficient Ratio corresponding to the luminance value lum is obtained from the lookup table lut. Ratio = lut[lum].
[0319] It should be noted that this application does not limit lum, and other brightness such as R, G, B in the RGB color domain, or the brightness component L in the Lab color domain, or I in the ICtCp format, or brightness Y in the YUV color domain can be used for calculation.
[0320] Among them, when using R, G, and B in the RGB color domain to calculate lum, this application does not limit the calculation method for obtaining the brightness value lum of each pixel (R, G, B), for example, Lum = max (R, G, B), or Lum = a*R+b*G+c*B, where a, b, c are weighting coefficients
[0321] Finally, each pixel (R, G, B) is multiplied by the tone mapping coefficient Ratio for tone mapping. The pixel after tone mapping is (R TM ,G TM ,B TM ): R TM =R*Ratio,G TM =G*Ratio,B TM =B*Ratio.
[0322] Or [YUV]=M*[RGB], YTM=Y*Ratio, [RGB]=M-1*[YUV], where M is selected according to the BT.2020 standard or the BT709 standard.
[0323] Optionally, when tone mapping is performed on the image to be displayed based on the reference information and the terminal device information of the corresponding type, the reference information and the terminal device information of the corresponding type can be input into an AI neural network, and the AI neural network outputs the tone-mapped image of the image to be displayed.
[0324] Optionally, saturation adjustment can be performed on each pixel (R, G, B) in the tone-mapped image. It should be understood that metadata may include color attribute information (e.g., color saturation adjustment intensity) of at least one frame of image. Color information of at least one frame of image after mapping can be corrected based on the color attribute information and the brightness information of each pixel in the image to be displayed before and after tone mapping.
[0325] For example, taking pixel A in an image to be displayed, the ratio between the first brightness value of pixel A after tone mapping and the second brightness value of pixel A before tone mapping can be determined. A saturation adjustment factor is determined based on this ratio, the maximum display capability of the terminal display device, the maximum display capability of a reference display device in the front-end device (e.g., a video production device), color attribute information of at least one frame of the image, and the second brightness value of pixel A before tone mapping. Based on this saturation adjustment factor, the color information of pixel A is adjusted. The maximum display capability can be understood as the maximum display brightness value.
[0326] It should be understood that the color information can be a YUV signal or a color signal in other color domains, which is not specifically limited in the present embodiment. Among them, the Y in "YUV" is used to represent the brightness information of the image, and UV is used to represent the color information of the image.
[0327] In a possible implementation, the color information is a YUV signal, and color information correction can be achieved by multiplying the saturation adjustment factor by the UV component of each pixel in at least one frame of the tone-mapped image.
[0328] Exemplarily, the saturation adjustment process may be:
[0329] According to the brightness value lum of each pixel and the tone mapping coefficient Ratio, the color adjustment ratioc is obtained. Ratioc can be ratioc=(ratio) N wait;
[0330] [YUV]=M*[RGB], UTM=U*ratioc, VTM=V*ratioc, where M is selected according to the BT.2020 standard or the BT.709 standard.
[0331] It should be noted that the present invention does not limit the saturation adjustment algorithm, also known as the color adjustment algorithm, and can be performed in different color domains, such as RGB, IctCp, HSV and Lab.
[0332] Optionally, color gamut mapping can be added after tone mapping or saturation adjustment, and the current color gamut is converted to the target color gamut. The current color gamut and the target color gamut include but are not limited to the color gamuts specified by standards BT.2020, BT.709, DCI-P3, and SRGB.
[0333] The following describes the image encoding process of the front-end device with reference to specific figures.
[0334] Please refer to FIG11 , which is a flowchart of an image encoding method provided in an embodiment of the present application. The method can be executed by a front-end device and specifically includes the following steps:
[0335] Step S1101: Write the image to be displayed into the code stream.
[0336] It should be understood that the image to be displayed can be one or more frames. When the image to be displayed is a plurality of consecutive frames, the image to be displayed can be understood as a video. Optionally, the image to be displayed can be an HDR image or video, or an SDR image or video.
[0337] Step S1102: writing reference information of the image to be displayed into the code stream; the reference information includes at least one of a reference display device size, a reference viewing distance, a reference viewing angle, a reference ambient light brightness, or a reference display device resolution.
[0338] In the embodiment of the present application, the reference information is the same as the reference information in step S502, and will not be described in detail here. The reference information is used to perform tone mapping on the image to be displayed.
[0339] An embodiment of the present application provides a code stream structure, which includes at least one of a reference display device size, a reference viewing distance, a reference viewing angle, a reference ambient light brightness, or a reference display device resolution.
[0340] Optionally, the method further includes: writing reference tone mapping parameters of the image to be displayed into the bitstream; wherein the reference tone mapping parameters are used to obtain initial tone mapping parameters. The reference information is used by the decoder to adjust the initial tone mapping parameters to obtain tone mapping parameters, and the tone mapping parameters are used to tone map the image to be displayed.
[0341] Optionally, writing the reference information of the image to be displayed into the code stream includes: writing the reference information into the code stream; or writing the index of the reference information into the code stream. The index of the reference information is the same as the index of the reference information in step S502, and will not be repeated here.
[0342] Optionally, the method further includes: writing one or more luminance characteristic values of the image to be displayed into the bitstream; wherein the one or more luminance characteristic values are used by the decoder to adjust initial tone mapping parameters to obtain the adjusted tone mapping parameters. The one or more luminance characteristic values are similar to the one or more first luminance characteristic values in Method 1 and are not further described here.
[0343] This embodiment of the present application also provides a tone mapping device 1200. See Figure 12, which illustrates an example of a tone mapping device provided in this embodiment of the present application. 1200 implements the functions of the terminal display device shown in Figure 4A above. These functions, units, or means may be implemented via software or hardware, or may be implemented via hardware executing corresponding software implementations.
[0344] Receiving module 1201, used to obtain the image to be displayed;
[0345] Also used to obtain reference information; the reference information includes at least one of a reference display device size, a reference viewing distance, a reference viewing angle, a reference ambient light brightness, or a reference display device resolution;
[0346] Also used to obtain terminal device information; the terminal device information includes at least one of the terminal display device size, user viewing distance, user viewing angle, ambient light brightness, or terminal display device resolution;
[0347] The tone mapping module 1202 is configured to perform tone mapping on the image to be displayed according to the reference information and the terminal device information of the corresponding type.
[0348] In an embodiment of the present application, in the tone mapping of the image to be displayed based on the reference information and the terminal device information of the corresponding type, the reference display device size corresponds to the terminal display device size, the reference viewing distance corresponds to the user viewing distance, the reference viewing angle corresponds to the user viewing angle, the reference ambient light brightness corresponds to the ambient light brightness, and the reference display device resolution corresponds to the terminal display device resolution.
[0349] In a possible embodiment, tone mapping is performed on the image to be displayed based on the reference information and the terminal device information of the corresponding type, including: obtaining tone mapping parameters based on the reference information and the terminal device information of the corresponding type; and tone mapping is performed on the image to be displayed based on the tone mapping parameters.
[0350] In a possible embodiment, obtaining tone mapping parameters according to the reference information and the terminal device information of the corresponding type includes: adjusting the initial tone mapping parameters of the image to be displayed according to the reference information and the terminal device information of the corresponding type to obtain the tone mapping parameters.
[0351] In one possible embodiment, adjusting initial tone mapping parameters of the image to be displayed based on the reference information and the corresponding type of terminal device information to obtain the tone mapping parameters includes: when a relationship between the reference information and the terminal device information satisfies a first condition, adjusting the initial tone mapping parameters of the image to be displayed in a first manner to obtain the adjusted tone mapping parameters. The initial tone mapping parameters are used to represent an initial tone mapping curve, and the tone mapping parameters are used to represent the adjusted tone mapping curve.
[0352] In a possible embodiment, the first condition includes: the terminal display screen size is smaller than the reference display device size, the user viewing distance is greater than the reference viewing distance, the user viewing angle deviates from the reference viewing angle, the ambient light brightness is greater than the reference ambient light brightness, or the terminal display device resolution is smaller than the reference display device resolution.
[0353] In a possible embodiment, adjusting the initial tone mapping parameters of the image to be displayed in a first manner includes: obtaining one or more first brightness characteristic values of the image to be displayed; adjusting the initial tone mapping parameters so that the mapping value of the obtained adjusted tone mapping curve at the one or more first brightness characteristic values is greater than the mapping value of the initial tone mapping curve at the one or more first brightness characteristic values.
[0354] In a possible embodiment, adjusting the initial tone mapping curve of the image to be displayed in a first manner also includes: obtaining one or more second brightness characteristic values of the image to be displayed; adjusting the initial tone mapping curve so that the slope of the obtained adjusted tone mapping curve at the one or more second brightness characteristic values is greater than the slope of the initial tone mapping curve at the one or more second brightness characteristic values.
[0355] In a possible embodiment, adjusting the initial tone mapping parameters of the image to be displayed in a first manner also includes: obtaining the base layer and one or more enhancement layers of the image to be displayed; the first eigenvalue of any one of the one or more enhancement layers is mapped to a second eigenvalue by the initial tone mapping curve of any enhancement layer; adjusting the initial tone mapping parameters of any enhancement layer so that the absolute value of the first eigenvalue after being mapped by the adjusted tone mapping curve is greater than the absolute value of the second eigenvalue.
[0356] In this embodiment, the first eigenvalue and the second eigenvalue can be positive or negative. When comparing the value of the first eigenvalue mapped by the adjusted tone mapping curve with the size of the second eigenvalue, the absolute values of the above two values are compared.
[0357] In one possible embodiment, adjusting initial tone mapping parameters of the image to be displayed based on the reference information and the corresponding type of terminal device information to obtain the tone mapping parameters includes: when a relationship between the reference information and the terminal device information satisfies a second condition, adjusting the initial tone mapping parameters of the image to be displayed in a second manner to obtain the tone mapping parameters. The initial tone mapping parameters are used to represent an initial tone mapping curve, and the tone mapping parameters are used to represent an adjusted tone mapping curve.
[0358] In a possible embodiment, the second condition includes: the terminal display screen size is larger than the reference display device size, the user viewing distance is smaller than the reference viewing distance, the ambient light brightness is smaller than the reference ambient light brightness, or the terminal display device resolution is larger than the reference display device resolution.
[0359] In a possible embodiment, adjusting the initial tone mapping parameters of the image to be displayed in a second manner includes: obtaining one or more third brightness characteristic values of the image to be displayed; adjusting the initial tone mapping parameters so that the mapping value of the obtained adjusted tone mapping curve at the one or more third brightness characteristic values is less than the mapping value of the initial tone mapping curve at the one or more third brightness characteristic values.
[0360] In a possible embodiment, the one or more third brightness characteristic values include: one or more brightness characteristic values of a dark portion of the image to be displayed.
[0361] In a possible embodiment, adjusting the initial tone mapping parameters of the image to be displayed in a second manner also includes: obtaining one or more fourth brightness characteristic values of the image to be displayed; adjusting the initial tone mapping parameters so that the slope of the obtained adjusted tone mapping curve at the one or more fourth brightness characteristic values is less than the slope of the initial tone mapping curve at the one or more fourth brightness characteristic values.
[0362] In a possible embodiment, adjusting the initial tone mapping parameters of the image to be displayed in a second manner also includes: obtaining the base layer and one or more enhancement layers of the image to be displayed; the third eigenvalue of any one of the one or more enhancement layers is mapped to a fourth eigenvalue by the initial tone mapping curve of any enhancement layer; adjusting the initial tone mapping parameters of any enhancement layer so that the absolute value of the third eigenvalue after mapping by the adjusted tone mapping curve is less than the absolute value of the fourth eigenvalue.
[0363] In this embodiment, the third eigenvalue and the fourth eigenvalue can be positive or negative. When comparing the value of the third eigenvalue mapped by the adjusted tone mapping curve with the size of the fourth eigenvalue, the absolute values of the above two values are compared.
[0364] In a possible embodiment, the tone mapping module is also used to: obtain a reference tone mapping of the image to be displayed; obtain the display capability of the terminal display device in the terminal device information; adjust the reference tone mapping parameters of the image to be displayed according to the reference tone mapping parameters and the display capability of the terminal display device to obtain the initial tone mapping parameters.
[0365] In a possible embodiment, obtaining the reference information includes: obtaining the reference information from the metadata of the image to be displayed; or obtaining the index of the reference information from the metadata, and selecting the reference information from multiple preset reference information according to the index; or using the preset reference information as the reference information.
[0366] In a possible embodiment, the acquiring of the terminal device information includes: acquiring the terminal device information through measurement; or selecting the terminal device information from a plurality of preset terminal device information according to a preset index; or using the preset terminal device information as the terminal device information.
[0367] It should be understood that the third aspect corresponds to the technical solution of the first aspect, so the beneficial effects achieved by each possible implementation method are similar and will not be repeated here.
[0368] It should be understood that all relevant contents of each step involved in the above tone mapping method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.
[0369] This embodiment of the present application also provides an image coding apparatus 1300. See Figure 13, which illustrates an example of an image coding apparatus provided in this embodiment of the present application. Apparatus 1300 is capable of implementing the functions of the front-end device described in Figure 4A above. For example, the functions, units, or means may be implemented via software or hardware, or the corresponding software implementation may be executed by hardware.
[0370] For example, referring to FIG. 13 , the apparatus 1300 may include:
[0371] Coding module 1301 is used to write an image to be displayed into a code stream; it is also used to write reference information of the image to be displayed into the code stream; the reference information includes at least one of a reference display device size, a reference viewing distance, a reference viewing angle, a reference ambient light brightness, or a reference display device resolution; the reference information is used to perform tone mapping on the image to be displayed.
[0372] The sending module 1302 is configured to send the code stream.
[0373] In a possible embodiment, the encoding module is further used to: write reference tone mapping parameters of the image to be displayed into the code stream; wherein the reference tone mapping parameters are used to obtain initial tone mapping parameters; the reference information is used by the decoding end to adjust the initial tone mapping parameters to obtain tone mapping parameters, and the tone mapping parameters are used to tone map the image to be displayed.
[0374] In a possible embodiment, writing the reference information of the image to be displayed into the code stream includes: writing the reference information into the code stream; or writing an index of the reference information into the code stream.
[0375] In a possible embodiment, the encoding module is further used to: write one or more brightness characteristic values of the image to be displayed into the code stream; wherein the one or more brightness characteristic values are used by the decoding end to adjust the initial tone mapping parameters to obtain the tone mapping parameters.
[0376] It should be understood that all relevant contents of each step involved in the embodiment of the above-mentioned image coding method can be referred to the functional description of the corresponding functional module, and will not be repeated here.
[0377] The present application also provides an electronic device 1400 , as shown in FIG14 , which is an example diagram of an electronic device provided in the present application, and is used to implement the tone mapping method in the embodiment shown in FIG5 .
[0378] As shown in FIG14 , the electronic device 1400 may include a processor 1401 for executing programs or instructions stored in a memory 1402 . When the programs or instructions stored in the memory 1402 are executed, the processor is configured to execute the tone mapping method in the embodiment shown in FIG5 .
[0379] Optionally, the electronic device 1400 may further include a communication interface 1403. FIG14 uses dashed lines to indicate that the communication interface 1403 is optional for the electronic device 1400.
[0380] Among them, the number of processors 1401, memories 1402 and communication interfaces 1403 does not constitute a limitation on the embodiments of the present application. During specific implementation, they can be arbitrarily configured according to business requirements.
[0381] Optionally, the memory 1402 is located outside the electronic device 1400 .
[0382] Optionally, the electronic device 1400 includes the memory 1402, which is connected to the at least one processor 1401 and stores instructions executable by the at least one processor 1401. FIG14 uses dashed lines to indicate that the memory 142 is optional for the electronic device 1400.
[0383] The processor 1401 and the memory 1402 may be coupled via an interface circuit or may be integrated together, which is not limited here.
[0384] The specific connection medium between the processor 1401, memory 1402, and communication interface 1403 is not limited in the embodiments of the present application. In Figure 14, the processor 1401, memory 1402, and communication interface 1403 are connected via a bus 1404. The bus is represented by a bold line in Figure 14. The connection between other components is only for schematic illustration and is not intended to be limiting. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one bold line is used in Figure 14, but this does not mean that there is only one bus or one type of bus.
[0385] An embodiment of the present application further provides an electronic device 1500 , see FIG15 , which is an example diagram of an electronic device provided in an embodiment of the present application, and is used to implement the image encoding method in the embodiment shown in FIG11 .
[0386] As shown in FIG15 , the electronic device 1500 may include a processor 1501 for executing a program or instruction stored in a memory 1502 . When the program or instruction stored in the memory 1502 is executed, the processor 1501 is used to execute the image encoding method in the embodiment shown in FIG11 .
[0387] Optionally, the electronic device 1500 may further include a communication interface 1503. FIG15 uses dashed lines to indicate that the communication interface 1503 is optional for the electronic device 1500.
[0388] Among them, the number of processors 1501, memories 1502 and communication interfaces 1603 does not constitute a limitation on the embodiments of the present application. During specific implementation, they can be arbitrarily configured according to business requirements.
[0389] Optionally, the memory 1502 is located outside the electronic device 1500 .
[0390] Optionally, the electronic device 1500 includes the memory 1502, which is connected to the at least one processor 1501 and stores instructions executable by the at least one processor 1501. FIG15 uses dotted lines to indicate that the memory 1502 is optional for the electronic device 1600.
[0391] The processor 1501 and the memory 1502 may be coupled via an interface circuit or may be integrated together, which is not limited here.
[0392] The specific connection medium between the processor 1501, memory 1502, and communication interface 1503 is not limited in the embodiments of the present application. In Figure 15, the processor 1501, memory 1502, and communication interface 1503 are connected via a bus 1504. The bus is represented by a bold line in Figure 15. The connection between other components is only for schematic illustration and is not intended to be limiting. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one bold line is used in Figure 15, but this does not mean that there is only one bus or one type of bus.
[0393] It should be understood that the processors mentioned in the embodiments of the present application can be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor that is implemented by reading software code stored in a memory.
[0394] Exemplarily, the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.
[0395] It should be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).
[0396] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) can be integrated into the processor.
[0397] It should be noted that the memory described herein is intended to include, but not be limited to, these and any other suitable types of memory.
[0398] An embodiment of the present application also provides a computer-readable storage medium, including a program or instruction. When the program or instruction is run on a computer, the tone mapping method in the embodiment shown in Figure 4 or the image encoding method in the embodiment shown in Figure 12 is executed.
[0399] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0400] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each flow and / or box in the flow chart and / or block diagram, as well as the combination of the flow chart and / or box in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more flow charts and / or one or more boxes in the block diagram.
[0401] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0402] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0403] Obviously, those skilled in the art may make various modifications and variations to this application without departing from the spirit and scope of this application. Thus, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application is intended to include such modifications and variations.
Claims
1. A tone mapping method, It is characterized in that The method comprises: Get the image to be displayed; Acquire reference information; the reference information includes at least one of a reference display device size, a reference viewing distance, a reference viewing angle, a reference ambient light brightness, or a reference display device resolution; Acquire terminal device information; the terminal device information includes at least one of the terminal display device size, user viewing distance, user viewing angle, ambient light brightness, or terminal display device resolution; According to the reference information and the terminal device information of the corresponding type, tone mapping is performed on the image to be displayed.
2. The method according to claim 1, It is characterized in that The performing tone mapping on the image to be displayed according to the reference information and the terminal device information of a corresponding type includes: Acquire tone mapping parameters according to the reference information and the terminal device information of the corresponding type; Based on the tone mapping parameters, tone mapping is performed on the image to be displayed.
3. The method according to claim 2, It is characterized in that Acquiring tone mapping parameters according to the reference information and the terminal device information of a corresponding type includes: According to the reference information and the terminal device information of the corresponding type, the initial tone mapping parameters of the image to be displayed are adjusted to obtain the tone mapping parameters.
4. The method according to claim 3, It is characterized in that The adjusting the initial tone mapping parameters of the image to be displayed according to the reference information and the terminal device information of the corresponding type to obtain the tone mapping parameters includes: When the relationship between the reference information and the terminal device information of the corresponding type satisfies a first condition, adjusting the initial tone mapping parameters of the image to be displayed in a first manner to obtain the tone mapping parameters; When the relationship between the reference information and the terminal device information of the corresponding type satisfies a second condition, adjusting the initial tone mapping parameters of the image to be displayed in a second manner to obtain the tone mapping parameters; The initial tone mapping parameters are used to characterize an initial tone mapping curve, and the tone mapping parameters are used to characterize an adjusted tone mapping curve.
5. The method according to claim 4, It is characterized in that The first condition includes: At least one of the following is true: the terminal display screen size is smaller than the reference display device size, the user viewing distance is greater than the reference viewing distance, the user viewing angle deviates from the reference viewing angle, the ambient light brightness is greater than the reference ambient light brightness, or the terminal display device resolution is smaller than the reference display device resolution.
6. The method according to claim 4 or 5, It is characterized in that The adjusting the initial tone mapping parameters of the image to be displayed in a first manner comprises: Acquire one or more first brightness characteristic values of the image to be displayed; The initial tone mapping parameters are adjusted so that a mapping value of the obtained adjusted tone mapping curve at the one or more first brightness characteristic values is greater than a mapping value of the initial tone mapping curve at the one or more first brightness characteristic values.
7. The method according to any one of claims 4 to 6, It is characterized in that The step of adjusting the initial tone mapping parameters of the image to be displayed in a first manner further comprises: Acquire one or more second brightness characteristic values of the image to be displayed; The initial tone mapping parameters are adjusted so that the slope of the adjusted tone mapping curve obtained at the one or more second brightness characteristic values is greater than the slope of the initial tone mapping curve at the one or more second brightness characteristic values.
8. The method according to any one of claims 4 to 7, It is characterized in that The step of adjusting the initial tone mapping parameters of the image to be displayed in a first manner further comprises: Acquire a base layer and one or more enhancement layers of the image to be displayed; a value after a first characteristic value of any one of the one or more enhancement layers is mapped by the initial tone mapping curve of any one of the enhancement layers is a second characteristic value; The initial tone mapping parameter of any one of the enhancement layers is adjusted so that an absolute value of a value of the first eigenvalue mapped by the adjusted tone mapping curve is greater than an absolute value of the second eigenvalue.
9. The method according to claim 4, It is characterized in that The second condition includes: At least one of the following is true: the terminal display screen size is larger than the reference display device size, the user viewing distance is smaller than the reference viewing distance, the ambient light brightness is smaller than the reference ambient light brightness, or the terminal display device resolution is larger than the reference display device resolution.
10. The method according to claim 4 or 9, It is characterized in that The adjusting the initial tone mapping parameters of the image to be displayed in a second manner comprises: Acquire one or more third brightness characteristic values of the image to be displayed; The initial tone mapping parameters are adjusted so that a mapping value of the obtained adjusted tone mapping curve at the one or more third brightness characteristic values is smaller than a mapping value of the initial tone mapping curve at the one or more third brightness characteristic values.
11. The method according to claim 4, 9 or 10, It is characterized in that The one or more third brightness characteristic values include: One or more brightness characteristic values of the dark portion of the image to be displayed.
12. The method according to any one of claims 4 or 9 to 11, It is characterized in that The adjusting the initial tone mapping parameters of the image to be displayed in a second manner further includes: Acquire one or more fourth brightness characteristic values of the image to be displayed; The initial tone mapping parameters are adjusted so that a slope of the obtained adjusted tone mapping curve at the one or more fourth brightness characteristic values is smaller than a slope of the initial tone mapping curve at the one or more fourth brightness characteristic values.
13. The method according to any one of claims 4 or 9 to 12, It is characterized in that The adjusting the initial tone mapping parameters of the image to be displayed in a second manner further includes: Acquire a base layer and one or more enhancement layers of the image to be displayed; a value after a third eigenvalue of any one of the one or more enhancement layers is mapped by the initial tone mapping curve of any one of the enhancement layers is a fourth eigenvalue; The initial tone mapping parameter of any one of the enhancement layers is adjusted so that an absolute value of a value of the third eigenvalue mapped by the adjusted tone mapping curve is smaller than an absolute value of the fourth eigenvalue.
14. The method according to claim 3, It is characterized in that The method further comprises: Acquire reference tone mapping parameters of the image to be displayed; Obtaining the display capability of the terminal display device in the terminal device information; According to the reference tone mapping parameters and the display capability of the terminal display device, the reference tone mapping parameters of the image to be displayed are adjusted to obtain the initial tone mapping parameters.
15. The method according to any one of claims 1 to 14, It is characterized in that The obtaining of reference information comprises: Acquiring the reference information from metadata of the image to be displayed; or obtaining an index of the reference information from the metadata, and selecting the reference information from a plurality of preset reference information according to the index; Or the preset reference information is used as the reference information.
16. The method according to any one of claims 1 to 15, It is characterized in that The obtaining of terminal device information includes: Acquiring the terminal device information by measuring; or selecting the terminal device information from a plurality of preset terminal information according to a preset index; Or the preset terminal device information is used as the terminal device information.
17. A method for encoding an image, It is characterized in that The method comprises: Write the image to be displayed into the code stream; Writing reference information of the image to be displayed into the bitstream; the reference information includes at least one of a reference display device size, a reference viewing distance, a reference viewing angle, a reference ambient light brightness, or a reference display device resolution; The reference information is used to perform tone mapping on the image to be displayed.
18. The method of claim 17, It is characterized in that The method further comprises: The reference tone mapping parameters of the image to be displayed are written into the bitstream; wherein the reference tone mapping parameters are used to obtain initial tone mapping parameters; the reference information is used by the decoding end to adjust the initial tone mapping parameters to obtain tone mapping parameters, and the tone mapping parameters are used to tone map the image to be displayed.
19. The method according to claim 17 or 18, It is characterized in that Writing the reference information of the image to be displayed into the code stream includes: Writing the reference information into the bitstream; Or write the index of the reference information into the bitstream.
20. The method according to any one of claims 17 to 19, It is characterized in that The method further comprises: Writing one or more brightness characteristic values of the image to be displayed into the code stream; The one or more brightness characteristic values are used by the decoding end to adjust the initial tone mapping parameters to obtain the tone mapping parameter.
21. A tone mapping device, It is characterized in that The device comprises: A receiving module, used for acquiring an image to be displayed; Also used to obtain reference information; the reference information includes at least one of a reference display device size, a reference viewing distance, a reference viewing angle, a reference ambient light brightness, or a reference display device resolution; Also used to obtain terminal device information; the terminal device information includes at least one of the terminal display device size, user viewing distance, user viewing angle, ambient light brightness, or terminal display device resolution; A tone mapping module is used to perform tone mapping on the image to be displayed according to the reference information and the terminal device information of the corresponding type.
22. The device according to claim 21, It is characterized in that The performing tone mapping on the image to be displayed according to the reference information and the terminal device information of a corresponding type includes: Acquire tone mapping parameters according to the reference information and the terminal device information of the corresponding type; Based on the tone mapping parameters, tone mapping is performed on the image to be displayed.
23. The device according to claim 22, It is characterized in that Acquiring tone mapping parameters according to the reference information and the terminal device information of a corresponding type includes: According to the reference information and the terminal device information of the corresponding type, the initial tone mapping parameters of the image to be displayed are adjusted to obtain the tone mapping parameters.
24. The device according to claim 23, It is characterized in that The adjusting the initial tone mapping parameters of the image to be displayed according to the reference information and the terminal device information of the corresponding type to obtain the tone mapping parameters includes: When the relationship between the reference information and the terminal device information of the corresponding type satisfies a first condition, adjusting the initial tone mapping parameters of the image to be displayed in a first manner to obtain the tone mapping parameters; When the relationship between the reference information and the terminal device information of the corresponding type satisfies a second condition, adjusting the initial tone mapping parameters of the image to be displayed in a second manner to obtain the tone mapping parameters; The initial tone mapping parameters are used to characterize an initial tone mapping curve, and the tone mapping parameters are used to characterize an adjusted tone mapping curve.
25. The device according to claim 24, It is characterized in that The first condition includes: At least one of the following is true: the terminal display screen size is smaller than the reference display device size, the user viewing distance is greater than the reference viewing distance, the user viewing angle deviates from the reference viewing angle, the ambient light brightness is greater than the reference ambient light brightness, or the terminal display device resolution is smaller than the reference display device resolution.
26. The device according to claim 24 or 25, It is characterized in that The adjusting the initial tone mapping parameters of the image to be displayed in a first manner comprises: Acquire one or more first brightness characteristic values of the image to be displayed; The initial tone mapping parameters are adjusted so that a mapping value of the obtained adjusted tone mapping curve at the one or more first brightness characteristic values is greater than a mapping value of the initial tone mapping curve at the one or more first brightness characteristic values.
27. The device as claimed in any one of claims 24 to 26, It is characterized in that The step of adjusting the initial tone mapping parameters of the image to be displayed in a first manner further comprises: Acquire one or more second brightness characteristic values of the image to be displayed; The initial tone mapping parameters are adjusted so that the slope of the adjusted tone mapping curve obtained at the one or more second brightness characteristic values is greater than the slope of the initial tone mapping curve at the one or more second brightness characteristic values.
28. The device as claimed in any one of claims 24 to 27, It is characterized in that The step of adjusting the initial tone mapping parameters of the image to be displayed in a first manner further comprises: Acquire a base layer and one or more enhancement layers of the image to be displayed; a value after a first characteristic value of any one of the one or more enhancement layers is mapped by the initial tone mapping curve of any one of the enhancement layers is a second characteristic value; The initial tone mapping parameter of any one of the enhancement layers is adjusted so that an absolute value of a value of the first eigenvalue mapped by the adjusted tone mapping curve is greater than an absolute value of the second eigenvalue.
29. The device of claim 24, It is characterized in that The second condition includes: At least one of the following is true: the terminal display screen size is larger than the reference display device size, the user viewing distance is smaller than the reference viewing distance, the ambient light brightness is smaller than the reference ambient light brightness, or the terminal display device resolution is larger than the reference display device resolution one.
30. The device according to claim 24 or 29, It is characterized in that The adjusting the initial tone mapping parameters of the image to be displayed in a second manner comprises: Acquire one or more third brightness characteristic values of the image to be displayed; The initial tone mapping parameters are adjusted so that a mapping value of the obtained adjusted tone mapping curve at the one or more third brightness characteristic values is smaller than a mapping value of the initial tone mapping curve at the one or more third brightness characteristic values.
31. The device of claim 30, It is characterized in that The one or more third brightness characteristic values include: One or more brightness characteristic values of the dark portion of the image to be displayed.
32. The device as claimed in any one of claims 24 or 29 to 31, It is characterized in that The adjusting the initial tone mapping parameters of the image to be displayed in a second manner further includes: Acquire one or more fourth brightness characteristic values of the image to be displayed; The initial tone mapping parameters are adjusted so that a slope of the obtained adjusted tone mapping curve at the one or more fourth brightness characteristic values is smaller than a slope of the initial tone mapping curve at the one or more fourth brightness characteristic values.
33. The device as claimed in any one of claims 24 or 29 to 32, It is characterized in that The adjusting the initial tone mapping parameters of the image to be displayed in a second manner further includes: Acquire a base layer and one or more enhancement layers of the image to be displayed; a value after a third eigenvalue of any one of the one or more enhancement layers is mapped by the initial tone mapping curve of any one of the enhancement layers is a fourth eigenvalue; The initial tone mapping parameter of any one of the enhancement layers is adjusted so that an absolute value of a value of the third eigenvalue mapped by the adjusted tone mapping curve is smaller than an absolute value of the fourth eigenvalue.
34. The device of claim 23, It is characterized in that The tone mapping module is also used to: Acquire reference tone mapping parameters of the image to be displayed; Obtaining the display capability of the terminal display device in the terminal device information; According to the reference tone mapping parameters and the display capability of the terminal display device, the reference tone mapping parameters of the image to be displayed are adjusted to obtain the initial tone mapping parameters.
35. The device as claimed in any one of claims 21 to 34, It is characterized in that The obtaining of reference information comprises: Acquiring the reference information from metadata of the image to be displayed; or obtaining an index of the reference information from the metadata, and selecting the reference information from a plurality of preset reference information according to the index; Or the preset reference information is used as the reference information.
36. The device as claimed in any one of claims 21 to 35, It is characterized in that The obtaining of terminal device information includes: Acquiring the terminal device information by measuring; or selecting the terminal device information from a plurality of preset terminal information according to a preset index; Or the preset terminal device information is used as the terminal device information.
37. An image encoding device, It is characterized in that The device comprises: The encoding module is used to write the image to be displayed into the code stream; Also used for writing reference information of the image to be displayed into the code stream; the reference information includes at least one of a reference display device size, a reference viewing distance, a reference viewing angle, a reference ambient light brightness, or a reference display device resolution; The reference information is used to perform tone mapping on the image to be displayed; The sending module is used to send the code stream.
38. The device of claim 37, It is characterized in that The encoding module is also used for: The reference tone mapping parameters of the image to be displayed are written into the bitstream; wherein the reference tone mapping parameters are used to obtain the tone mapping parameters; the reference information is used by the decoding end to adjust the initial tone mapping parameters to obtain the tone mapping parameters, and the tone mapping parameters are used to tone map the image to be displayed.
39. The device according to claim 37 or 38, It is characterized in that Writing the reference information of the image to be displayed into the code stream includes: Writing the reference information into the bitstream; Or write the index of the reference information into the bitstream.
40. The device as claimed in any one of claims 37 to 39, It is characterized in that The encoding module is also used for: One or more brightness characteristic values of the image to be displayed are written into the bitstream; wherein the one or more brightness characteristic values are used by the decoding end to adjust the initial tone mapping parameters to obtain the tone mapping parameters.
41. An electronic device, It is characterized in that include: A memory and a processor coupled to each other, wherein the processor calls a program code stored in the memory to execute the method according to any one of claims 1 to 16 or claims 17 to 20.
42. A computer readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the method of any one of claims 1 to 16 or claims 17 to 20 is implemented.
43. A computer program product, It is characterized in that The method comprises a program or an instruction, which, when executed on a computer or a processor, causes the computer or the processor to execute the method according to any one of claims 1 to 16 or claims 17 to 20.
