A tone mapping method and apparatus

Abstract

This application provides a tone mapping method and apparatus. The method can be executed by a backend device. In this method, the backend device can receive metadata and an HDR image from a frontend device, generate a mapping curve according to the characteristic parameters of the mapping curve included in the metadata, and perform tone mapping on the HDR image based on the mapping curve. Among them, the characteristic parameters of the mapping curve are determined by the frontend device according to the brightness distribution of the HDR image. In this way, by determining the mapping curve parameters in combination with the brightness distribution of the HDR image, the design of the mapping curve is more flexible, and the display effect of the HDR image can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of display technologies, and in particular, to a tone mapping method and apparatus. Background Art

[0002] High dynamic range (HDR) video technology expands the brightness range of displayable images, and thus can record a large range of brightness information and display more highlight and shadow details in images.

[0003] However, the brightness of HDR videos or images usually does not match the brightness range that can be displayed by standard dynamic range (SDR) display devices or a large number of other existing HDR display devices; for example, in the same frame of an HDR image, the pixel brightness of the darkest part is 0.4 cd / m2, and the pixel brightness of the brightest part is 6000 cd / m2, while a TV can only display pixels with a brightness greater than or equal to 0.5 cd / m2 and less than or equal to 5000 cd / m2. Then, the pixels with a brightness of 0.4 cd / m2 - 0.5 cd / m2 in this frame of the image will appear as darkness on the TV, and the pixels with a brightness of 5000 cd / m2 - 6000 cd / m2 in this frame of the image will appear as white light on the TV.

[0004] Therefore, when an HDR video or image is displayed on an existing display device, it is necessary to process the brightness of the HDR video or image according to the display capabilities of the display device to obtain an SDR video or image, so that it matches the brightness range that the display device can display. The process of compressing an HDR video or image into an SDR video or image is what we call tone mapping (TM).

[0005] The industry has proposed tone mapping HDR images based on a mapping curve determined by an N - degree spline function, but the generation of the mapping curve depends on a basic curve (for example, a Bezier curve), resulting in the generation of the entire mapping curve being insufficiently flexible, and the tone mapping effect of pixels in extremely bright or extremely dark regions being poor, thereby leading to a poor display effect of HDR images. Summary of the Invention

[0006] This application provides a tone mapping method and apparatus for tone mapping HDR images and improving the display effect of HDR images.

[0007] In a first aspect, an embodiment of the present application provides a tone mapping method, in which at least one frame of image and metadata are first received; wherein the metadata includes parameters of a mapping curve, for example, the number M of mapping curves, and the coordinate values of the starting node and the ending node of each mapping curve in a first coordinate system; then, M mapping curves can be generated according to the metadata, and tone mapping can be performed on at least one frame of image based on the M mapping curves.

[0008] It should be understood that each mapping curve is an N-order spline curve, the horizontal coordinate of the first coordinate system represents the image brightness before tone mapping, and the vertical coordinate of the first coordinate system represents the image brightness after tone mapping; M is an integer greater than or equal to 1, and N is an integer greater than or equal to 3.

[0009] In this embodiment of the present application, after receiving the metadata, M N-order spline curves can be generated based on the number M of mapping curves in the metadata and the coordinate values of the starting and ending nodes of each mapping curve in the first coordinate system. Tone mapping is then performed on at least one image frame based on these M N-order spline curves. This embodiment of the present application provides a more flexible design of the mapping curves, effectively improving the tone mapping effect of HDR images and, in turn, the display quality of HDR images.

[0010] In a possible implementation, the metadata may further include a preset first-order derivative of a starting node and a preset first-order derivative of an ending node of each mapping curve.

[0011] In this embodiment, the metadata includes a preset first-order derivative of a starting node and a preset first-order derivative of an ending node of each mapping curve, so that the method of generating each mapping curve is more flexible.

[0012] In one possible embodiment, each frame image in at least one frame of image corresponds to at least one brightness range, each brightness range corresponds to at least one mapping curve, and the function expression corresponding to each mapping curve can be an N-order polynomial, which includes N+1 monomials, and each monomial corresponds to a coefficient.

[0013] In this embodiment, each frame of at least one frame of image includes one or more brightness ranges. For each brightness range, a corresponding mapping curve can be designed specifically for it, so that the tone mapping effect of at least one frame of image can be effectively improved, thereby effectively improving the display effect of at least one frame of image.

[0014] Exemplarily, the i-th mapping curve in the at least one mapping curve may be a cubic spline function curve, and its corresponding first function expression is:

[0015] y=di*(x-x²) 3 +ci*(x-x2)2 +bi*(x - x2)+ai;

[0016] Where x is the independent variable, used to represent the image brightness before tone mapping, y is the dependent variable, used to represent the image brightness after tone mapping; ai is the first coefficient of the constant term in the i-th mapping curve, bi is the second coefficient of the first-degree monomial in the i-th mapping curve, ci is the third coefficient of the second-degree monomial in the i-th mapping curve, di is the fourth coefficient of the third-degree monomial in the i-th mapping curve; x2 is the horizontal coordinate of the starting node of the i-th mapping curve; ai, bi, ci, and di are all non-zero.

[0017] Exemplarily, the above i-th mapping curve can also be a quartic spline function curve, and its corresponding first function expression is:

[0018] y = ei*(x - x2) 4 +di*(x - x2) 3 +ci*(x - x2) 2 +bi*(x - x2)+ai;

[0019] Where x is the independent variable, used to represent the image brightness before tone mapping, y is the dependent variable, used to represent the image brightness after tone mapping; ai is the coefficient of the constant term in the i-th mapping curve, bi is the coefficient of the first-degree monomial in the i-th mapping curve, ci is the coefficient of the second-degree monomial in the i-th mapping curve, di is the coefficient of the third-degree monomial in the i-th mapping curve, ei is the coefficient of the fourth-degree monomial in the i-th mapping curve; x2 is the horizontal coordinate of the starting node of the i-th mapping curve; ai, bi, ci, and di are all non-zero.

[0020] It should be understood that the above are only examples rather than limitations with cubic spline function curves and quartic spline function curves. The embodiments of the present application do not limit the specific type of the spline function curve. For example, the above i-th mapping curve can also be a quintic spline function curve, a sextic spline function curve, etc.

[0021] In one possible embodiment, a first brightness range in at least one brightness range corresponds to a first mapping curve, and the first mapping curve includes a first starting node and a first ending node. Accordingly, generating M mapping curves based on the metadata may include: substituting the coordinate value of the first starting node into the first function expression to obtain a first coefficient a1 of the first mapping curve; substituting the coordinate value of the first ending node into the first function expression to obtain a first equation; making the first-order derivative of the first mapping curve at the first starting node equal to the preset first-order derivative of the first starting node to obtain a second equation; making the first-order derivative of the first mapping curve at the first ending node equal to the preset first-order derivative of the first ending node to obtain a third equation; combining the first equation, the second equation, and the third equation to obtain a first set of equations; analyzing the first set of equations to obtain the second coefficient b1, the third coefficient c1, and the fourth coefficient d1 of the first mapping curve; and generating the first mapping curve based on the first coefficient a1, the second coefficient b1, the third coefficient c1, and the fourth coefficient d1 of the first mapping curve.

[0022] In this embodiment, an N-order spline curve is used to form a mapping curve for the same brightness range, which requires fewer mapping curve parameters and can improve the efficiency of metadata transmission.

[0023] In a possible implementation, the second brightness range in at least one brightness range corresponds to a second mapping curve and a third mapping curve. The second mapping curve corresponds to a first sub-brightness range in the second brightness range, and the third mapping curve corresponds to a second sub-brightness range in the second brightness range. The maximum value in the first sub-brightness range is equal to the minimum value in the second sub-brightness range. The second mapping curve includes a second start node and a second end node. The third mapping curve includes a third start node and a third end node. The coordinate values of the second end node and the third start node are the same. Correspondingly, according to the metadata, generating M mapping curves may include: substituting the coordinate value of the second start node into the first function expression to obtain the first coefficient a2 of the second mapping curve; substituting the coordinate value of the third start node into the first function expression to obtain the first coefficient a3 of the third mapping curve; making the first derivative of the second mapping curve at the second start node equal to the preset first derivative of the second start node to establish a fourth equation; analyzing the fourth equation to obtain the second coefficient b2 of the second mapping curve; constructing a second system of equations according to the coordinate values of the second end node, the coordinate values of the third end node, and the preset first derivative of the third end node; analyzing the second system of equations to obtain the third coefficient c2, the fourth coefficient d2 of the second mapping curve, and the second coefficient b3, the third coefficient c3, and the fourth coefficient d3 of the third mapping curve; generating the second mapping curve based on the first coefficient a2, the second coefficient b2, the third coefficient c2, and the fourth coefficient d2 of the second mapping curve; and generating the third mapping curve based on the first coefficient a3, the second coefficient b3, the third coefficient c3, and the fourth coefficient d3 of the third mapping curve.

[0024] In this implementation, the same brightness range is further divided into two sub-brightness ranges, and each sub-brightness range has a cubic spline curve. That is, the same brightness range uses two cubic spline curves to form the mapping curve, effectively improving the flexibility of the mapping curve design and thus effectively improving the effect of tone mapping.

[0025] Among them, there are multiple implementation manners for constructing the second system of equations according to the coordinate values of the second end node, the coordinate values of the third end node, and the preset first derivative of the third end node, and the embodiments of the present application do not make any restrictions.

[0026] Exemplarily, method 1, substitutes the coordinate values of the second cutoff node and the coordinate values of the third cutoff node into the first function expression to obtain the fifth equation and the sixth equation respectively, so that the first-order derivative of the third cutoff node at the third mapping curve is equal to the preset first-order derivative of the third cutoff node, and obtains the seventh equation; makes the first-order derivative of the second mapping curve at the second cutoff node equal to the first-order derivative of the third mapping curve at the third starting node, and obtains the eighth equation; makes the second-order derivative of the second mapping curve at the second cutoff node equal to the second-order derivative of the third mapping curve at the third starting node, and obtains the ninth equation; combines the fifth equation, the sixth equation, the seventh equation, the eighth equation, and the ninth equation to obtain the second group of equations.

[0027] In method 1, the function coefficients of the mapping curves are solved by ensuring that the first-order derivative of the second mapping curve at the second cutoff node is equal to the first-order derivative of the third mapping curve at the third starting node, and that the second-order derivative of the second mapping curve at the second cutoff node is equal to the second-order derivative of the third mapping curve at the third starting node. Therefore, the metadata does not need to carry the preset first-order derivatives of the second cutoff node and the third starting node, which can effectively improve the transmission efficiency of the metadata.

[0028] Exemplarily, method 2, substitutes the coordinate values of the second cutoff node and the coordinate values of the third cutoff node into the first function expression to obtain the fifth equation and the sixth equation respectively, so that the first-order derivative of the third cutoff node at the third mapping curve is equal to the preset first-order derivative of the third cutoff node, and obtains the seventh equation; makes the first-order derivative of the second mapping curve at the second cutoff node equal to the preset first-order derivative of the second cutoff node, and obtains the eighth equation; makes the first-order derivative of the third mapping curve at the third starting node equal to the preset first-order derivative of the third starting node, and obtains the ninth equation; combines the fifth equation, the sixth equation, the seventh equation, the eighth equation, and the ninth equation to obtain the second group of equations.

[0029] In method 2, the function coefficients of the mapping curve are solved by making the first-order derivative of the second mapping curve at the second cutoff node equal to the preset first-order derivative of the second cutoff node, and the first-order derivative of the third mapping curve at the third starting node equal to the preset first-order derivative of the third starting node, which can effectively reduce the amount of calculation.

[0030] In a possible implementation, performing tone mapping on at least one frame of image based on M mapping curves may include: performing tone mapping on each pixel in the at least one frame of image using a mapping curve corresponding to a brightness range of the pixel.

[0031] In this embodiment, different mapping curves are used for pixels in different brightness ranges in at least one frame of image for tone mapping, so that the display of at least one frame of image is more adapted to the display device, thereby effectively improving the display effect of at least one frame of image.

[0032] In a possible implementation, after tone mapping at least one frame of image based on M mapping curves, it may further include: correcting the color information of the tone-mapped at least one frame of image based on the brightness information of each pixel in the at least one frame of image before and after tone mapping.

[0033] In this embodiment, the backend device adjusts the color information of at least one frame of image according to the brightness information of each pixel in the at least one frame of image before and after tone mapping after tone mapping the at least one frame of image, which can further improve the display effect of the at least one frame of image.

[0034] In a second aspect, an embodiment of the present application provides a method for generating metadata. In this method, first, at least one frame of image is obtained; then, the at least one frame of image is processed to generate metadata; where the metadata includes the number M of mapping curves, and the coordinate values of the starting node and the ending node of each mapping curve in the first coordinate system.

[0035] It should be understood that the mapping curve is an Nth-order spline curve, the horizontal coordinate of the first coordinate system represents the image brightness before tone mapping, and the vertical coordinate of the first coordinate system represents the image brightness after tone mapping; M is an integer greater than or equal to 1, and N is an integer greater than or equal to 3.

[0036] In an embodiment of the present application, generating metadata including mapping curve parameters based on at least one frame of image provides a high degree of freedom for the design of the mapping curve by matching the image content, and thus can effectively improve the effect of tone mapping the image.

[0037] In a possible implementation, processing at least one frame of image to generate metadata may include: first, extracting the attribute information of the at least one frame of image, where the attribute information may include the maximum value, minimum value, average value, change range, etc. of the brightness; then, determining the metadata based on the attribute information.

[0038] In this embodiment, determining the metadata based on the maximum value, minimum value, average value, change range, etc. of the brightness of at least one frame of image and combining the mapping curve parameters with the image brightness characteristics can effectively improve the flexibility of the mapping curve design, and thus effectively improve the effect of image tone mapping.

[0039] In a possible implementation, when the metadata includes the number M of mapping curves, based on the attribute information, determining the metadata includes: determining the brightness distribution of at least one frame of image according to the attribute information; dividing the brightness of at least one frame of image into at least one brightness range based on the brightness distribution; determining the number M according to the number F of brightness ranges, where M is greater than or equal to F, and each brightness range corresponds to at least one mapping curve.

[0040] In this implementation, designing different mapping curves for different brightness ranges can further improve the flexibility of mapping curve design.

[0041] In a possible implementation, when the metadata includes the coordinate values of the cut-off node and the starting node of each mapping curve in the first coordinate system, based on the attribute information, determining the metadata may include: determining the coordinate values of the starting node and the cut-off node of each mapping curve in the first coordinate system according to the brightness distribution and the maximum brightness value that the backend device can display.

[0042] Exemplarily, according to the brightness distribution of the first image, the abscissa of the cut-off node of the mapping curve corresponding to the first image in the first coordinate system can be determined, and the ordinate of the cut-off node of the mapping curve in the first coordinate system can be determined according to the maximum brightness value that the backend device can display.

[0043] In this implementation, by combining the brightness distribution of the image and the maximum brightness value that the backend device can display to determine the parameters of each mapping curve, the flexibility of mapping curve design can be further improved.

[0044] In a possible implementation, the metadata may further include the preset first-order derivative of the starting node and the preset first-order derivative of the cut-off node.

[0045] In this implementation, when the metadata carries the preset first-order derivative of the starting node and the preset first-order derivative of the cut-off node of each mapping curve, the way of generating the mapping curve is more flexible.

[0046] In a possible implementation, when the metadata includes the preset first-order derivative of the cut-off node and the starting node of each mapping curve, based on the attribute information, determining the metadata may include: determining the preset first-order derivative of the starting node and the preset first-order derivative of the cut-off node according to the balance requirement between the brightness and the contrast of at least one frame of image.

[0047] In this implementation, determining the preset first-order derivative of the starting node and the preset first-order derivative of the cut-off node according to the balance requirement between the brightness and the contrast of at least one frame of image can further effectively improve the flexibility of mapping curve design to meet the diverse mapping requirements of users, and thus effectively improve the user experience.

[0048] In a possible implementation, after processing at least one frame of image to generate metadata, it may further include: receiving a first operation for changing at least one of the number M, the coordinate value of the starting node in the first coordinate system, the coordinate value of the ending node in the first coordinate system, the preset first-order derivative of the starting node, and the preset first-order derivative of the ending node; and modifying the metadata based on the first operation to obtain updated metadata.

[0049] In this implementation, the user can modify the parameters of the mapping curve in the metadata, which can further meet the diverse mapping needs of the user and thus effectively improve the user experience.

[0050] In a third aspect, an embodiment of the present application provides a tone mapping processing device, which includes a module for executing the method described in the first aspect or any possible implementation of the first aspect.

[0051] Exemplarily, the device may include:

[0052] a receiving module for receiving at least one frame of image and metadata; the metadata includes the number M of mapping curves, and the coordinate values of the starting node and the ending node of each mapping curve in the first coordinate system; wherein, the mapping curve is an Nth-order spline curve, the horizontal coordinate of the first coordinate system represents the image brightness before tone mapping, and the vertical coordinate of the first coordinate system represents the image brightness after tone mapping; M is an integer greater than or equal to 1, and N is an integer greater than or equal to 3;

[0053] a processing module for generating M mapping curves according to the metadata; and performing tone mapping on at least one frame of image based on the M mapping curves.

[0054] In a possible implementation, the metadata further includes the preset first-order derivative of the starting node and the preset first-order derivative of the ending node of each mapping curve.

[0055] In a possible implementation, each frame of the at least one frame of image corresponds to at least one brightness range, and each brightness range corresponds to at least one mapping curve. The function expression corresponding to the mapping curve is an Nth-order polynomial, and the Nth-order polynomial includes N + 1 monomials, and each monomial corresponds to a coefficient.

[0056] Exemplarily, the ith mapping curve in at least one mapping curve may be a cubic spline function curve, and its corresponding first function expression is:

[0057] y = di*(x - x2) 3 + ci*(x - x2) 2 + bi*(x - x2) + ai;

[0058] Where x is an independent variable used to represent the image brightness before tone mapping, and y is a dependent variable used to represent the image brightness after tone mapping; ai is the first coefficient of the constant term in the i-th mapping curve, bi is the second coefficient of the first-degree monomial in the i-th mapping curve, ci is the third coefficient of the second-degree monomial in the i-th mapping curve, di is the fourth coefficient of the third-degree monomial in the i-th mapping curve; x2 is the horizontal coordinate of the starting node of the i-th mapping curve; ai, bi, ci, and di are all non-zero.

[0059] Exemplarily, the i-th mapping curve described above may also be a quartic spline function curve, and its corresponding first function expression is:

[0060] y = ei*(x - x2) 4 + di*(x - x2) 3 + ci*(x - x2) 2 + bi*(x - x2) + ai;

[0061] Where x is an independent variable used to represent the image brightness before tone mapping, and y is a dependent variable used to represent the image brightness after tone mapping; ai is the coefficient of the constant term in the i-th mapping curve, bi is the coefficient of the first-degree monomial in the i-th mapping curve, ci is the coefficient of the second-degree monomial in the i-th mapping curve, di is the coefficient of the third-degree monomial in the i-th mapping curve, ei is the coefficient of the fourth-degree monomial in the i-th mapping curve; x2 is the horizontal coordinate of the starting node of the i-th mapping curve; ai, bi, ci, and di are all non-zero.

[0062] It should be understood that the i-th mapping curve described above may be a cubic spline function curve, a quartic spline function curve, a quintic spline function curve, etc. The above only uses the cubic spline function curve and the quartic spline function curve as examples, and the embodiments of the present application are not specifically limited.

[0063] In a possible implementation manner, the first brightness range in at least one brightness range corresponds to a first mapping curve, and the first mapping curve includes a first starting node and a first cut-off node; when the processing module is used to generate M mapping curves according to metadata, it is specifically used for:

[0064] Substitute the coordinate values of the first starting node into the first function expression to obtain the first coefficient a1 of the first mapping curve; substitute the coordinate values of the first ending node into the first function expression to obtain the first equation; make the first derivative of the first mapping curve at the first starting node equal to the preset first derivative of the first starting node to obtain the second equation; make the first derivative of the first mapping curve at the first ending node equal to the preset first derivative of the first ending node to obtain the third equation; combine the first equation, the second equation, and the third equation to obtain the first system of equations; solve the first system of equations to obtain the second coefficient b1, the third coefficient c1, and the fourth coefficient d1 of the first mapping curve; generate the first mapping curve based on the first coefficient a1, the second coefficient b1, the third coefficient c1, and the fourth coefficient d1 of the first mapping curve.

[0065] In a possible implementation, the second brightness range in at least one brightness range corresponds to the second mapping curve and the third mapping curve. The second mapping curve corresponds to the first sub-brightness range in the second brightness range, and the third mapping curve corresponds to the second sub-brightness range in the second brightness range. The maximum value in the first sub-brightness range is equal to the minimum value in the second sub-brightness range; the second mapping curve includes a second starting node and a second ending node; the third mapping curve includes a third starting node and a third ending node; the coordinate values of the second ending node and the third starting node are the same; when the processing module is used to generate M mapping curves according to the metadata, it is specifically used for:

[0066] Substitute the coordinate values of the second starting node into the first function expression to obtain the first coefficient a2 of the second mapping curve; substitute the coordinate values of the third starting node into the first function expression to obtain the first coefficient a3 of the third mapping curve; make the first derivative of the second mapping curve at the second starting node equal to the preset first derivative of the second starting node to establish the fourth equation; solve the fourth equation to obtain the second coefficient b2 of the second mapping curve; construct a second system of equations according to the coordinate values of the second ending node, the coordinate values of the third ending node, and the preset first derivative of the third ending node; solve the second system of equations to obtain the third coefficient c2, the fourth coefficient d2 of the second mapping curve, and the second coefficient b3, the third coefficient c3, and the fourth coefficient d3 of the third mapping curve; generate the second mapping curve based on the first coefficient a2, the second coefficient b2, the third coefficient c2, and the fourth coefficient d2 of the second mapping curve; and generate the third mapping curve based on the first coefficient a3, the second coefficient b3, the third coefficient c3, and the fourth coefficient d3 of the third mapping curve.

[0067] In a possible implementation, when the processing module is used to construct a second system of equations according to the coordinate values of the second ending node, the coordinate values of the third ending node, and the preset first derivative of the third ending node, it is specifically used for:

[0068] Substituting the coordinate values of the second cutoff node and the coordinate values of the third cutoff node into the first function expression, the fifth equation and the sixth equation are obtained respectively, so that the first-order derivative of the third cutoff node at the third mapping curve is equal to the preset first-order derivative of the third cutoff node, and the seventh equation is obtained; the first-order derivative of the second mapping curve at the second cutoff node is equal to the first-order derivative of the third mapping curve at the third starting node, and the eighth equation is obtained; the second-order derivative of the second mapping curve at the second cutoff node is equal to the second-order derivative of the third mapping curve at the third starting node, and the ninth equation is obtained; or, the first-order derivative of the second mapping curve at the second cutoff node is equal to the preset first-order derivative of the second cutoff node, and the eighth equation is obtained; the first-order derivative of the third mapping curve at the third starting node is equal to the preset first-order derivative of the third starting node, and the ninth equation is obtained; the fifth equation, the sixth equation, the seventh equation, the eighth equation, and the ninth equation are combined to obtain the second group of equations.

[0069] In a possible implementation, when the processing module is used to perform tone mapping on at least one frame of image based on M mapping curves, it is specifically used to: for each pixel in at least one frame of image, perform tone mapping on the pixel using the mapping curve corresponding to the brightness range of the pixel.

[0070] In one possible embodiment, after the processing module is used to perform tone mapping on at least one frame of image based on M mapping curves, it is also used to: correct the color information of the at least one frame of image after mapping based on the brightness information of each pixel in the at least one frame of image before and after the tone mapping.

[0071] It should be understood that the third aspect corresponds to the technical solution of the first aspect, so the beneficial effects achieved by each possible implementation method are similar and will not be repeated here.

[0072] In a fourth aspect, an embodiment of the present application provides a device for generating metadata, which includes a module for executing the method described in the second aspect or any possible implementation of the second aspect.

[0073] Exemplarily, the device may include:

[0074] An acquisition module, configured to acquire at least one frame of image;

[0075] A processing module is configured to process at least one frame of image and generate metadata; wherein the metadata includes the number M of mapping curves and the coordinate values of the starting node and the ending node of each mapping curve in a first coordinate system; wherein the mapping curve is an N-order spline curve, the horizontal coordinate of the first coordinate system represents the image brightness before tone mapping, and the vertical coordinate of the first coordinate system represents the image brightness after tone mapping; M is an integer greater than or equal to 1, and N is an integer greater than or equal to 3.

[0076] In a possible implementation, when the processing module is used to process at least one frame of image to generate metadata, it is specifically used for: extracting the attribute information of at least one frame of image, where the attribute information includes the maximum value, minimum value, average value, and variation range of brightness; and determining the metadata based on the attribute information.

[0077] In a possible implementation, the metadata includes the number M of mapping curves. When the processing module is used to determine the metadata based on the attribute information, it is specifically used for: determining the brightness distribution of at least one frame of image according to the attribute information; dividing the brightness of at least one frame of image into at least one brightness range based on the brightness distribution; and determining the number M according to the number F of brightness ranges, where M is greater than or equal to F, and each brightness range corresponds to at least one mapping curve.

[0078] In a possible implementation, the metadata includes the coordinate values of the cut-off node and the start node of each mapping curve in the first coordinate system. When the processing module is used to determine the metadata based on the attribute information, it is specifically used for: determining the coordinate values of the start node and the cut-off node of each mapping curve in the first coordinate system according to the brightness distribution and the maximum brightness value that the backend device can display.

[0079] In a possible implementation, the metadata further includes the preset first-order derivative of the start node and the preset first-order derivative of the cut-off node of each mapping curve.

[0080] In a possible implementation, the metadata includes the preset first-order derivative of the cut-off node and the start node of each mapping curve. When the processing module is used to determine the metadata based on the attribute information, it is specifically used for: determining the preset first-order derivative of the start node and the preset first-order derivative of the cut-off node according to the balance requirement between the brightness and contrast of at least one frame of image.

[0081] In a possible implementation, after the processing module is used to process at least one frame of image to generate metadata, it is further used for: receiving a first operation, where the first operation is used to change at least one of the number M, the coordinate value of the start node in the first coordinate system, the coordinate value of the cut-off node in the first coordinate system, the preset first-order derivative of the start node, and the preset first-order derivative of the cut-off node; and modifying the metadata based on the first operation to obtain updated metadata.

[0082] It should be understood that the technical solutions of the fourth aspect correspond to those of the second aspect, so the beneficial effects obtained by each possible implementation are similar and will not be elaborated here.

[0083] In a fifth aspect, an embodiment of the present application provides an electronic device, including: a non-volatile memory and a processor coupled to each other, and the processor calls program code stored in the memory to execute the method described in the first aspect or any possible implementation manner of the first aspect.

[0084] It should be understood that for the technical effects that can be achieved in the fifth aspect, reference can specifically be made to the description of the technical effects brought by the first aspect or any possible implementation manner of the first aspect above, and details are not elaborated here.

[0085] In a sixth aspect, an embodiment of the present application provides an electronic device, including: a non-volatile memory and a processor coupled to each other, and the processor calls program code stored in the memory to execute the method described in the second aspect or any possible implementation manner of the second aspect.

[0086] It should be understood that for the technical effects that can be achieved in the sixth aspect, reference can specifically be made to the description of the technical effects brought by the second aspect or any possible implementation manner of the second aspect above, and details are not elaborated here.

[0087] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, including a program or instructions, which when running on a computer, cause the method in the first aspect or any possible implementation manner of the first aspect to be executed, or cause the method in the second aspect or any possible implementation manner of the second aspect to be executed.

[0088] It should be understood that for the technical effects that can be achieved in the seventh aspect, reference can specifically be made to the description of the technical effects brought by the first aspect or any possible implementation manner of the first aspect or the second aspect or any possible implementation manner of the second aspect above, and details are not elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 It is a schematic diagram of a mapping curve;

[0090] Figure 2A It is a schematic diagram of a possible system architecture applicable to an embodiment of the present application;

[0091] Figure 2B It is a schematic structural diagram of an image processing system provided by an embodiment of the present application;

[0092] Figure 3A It is a schematic flowchart of a method for generating metadata provided by an embodiment of the present application;

[0093] Figure 3B It is a schematic diagram of a possible scenario provided by an embodiment of the present application;

[0094] Figure 4A Schematic diagram of a mapping curve provided by an embodiment of the present application;

[0095] Figure 4B Schematic diagram of another mapping curve provided by an embodiment of the present application;

[0096] Figure 5A Schematic diagram of a mapping curve provided by an embodiment of the present application;

[0097] Figure 5B Schematic diagram of another mapping curve provided by an embodiment of the present application;

[0098] Figure 6A Schematic diagram of a mapping curve provided by an embodiment of the present application;

[0099] Figure 6B Schematic diagram of another mapping curve provided by an embodiment of the present application;

[0100] Figure 7 Schematic flow chart of a tone mapping method provided by an embodiment of the present application;

[0101] Figure 8 Schematic diagram of the structure of a tone mapping device provided by an embodiment of the present application;

[0102] Figure 9 Schematic diagram of the structure of a device for generating metadata provided by an embodiment of the present application;

[0103] Figure 10 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application;

[0104] Figure 11 Schematic diagram of the structure of another electronic device provided by an embodiment of the present application. Detailed implementation manners

[0105] The term "at least one" in the embodiments of the present application refers to one or more, and "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one (item)" or its similar expression refers to any combination of these items, including any combination of single item (item) or plural items (items). For example, at least one (item) of a, b, or c can represent: a, b, c, a - b, a - c, b - c, or a - b - c, and any one of a, b, c, a - b, a - c, b - c, or a - b - c can include a single a, a single b, a single c, or multiple a, multiple b, multiple c.

[0106] In addition, unless otherwise stated, the ordinal numbers such as "first" and "second" mentioned in the embodiments of the present application are used to distinguish multiple objects and are not used to limit the order, time sequence, priority, or importance of multiple objects. For example, the first priority criterion and the second priority criterion are only used to distinguish different criteria, rather than indicating differences in the content, priority, or importance of these two criteria.

[0107] Furthermore, the terms "comprising" and "having" in the embodiments of the present application, the claims, and the drawings are not exclusive. For example, a process, method, system, product, or device that includes a series of steps or modules / units is not limited to the listed steps or modules and may further include steps or modules / units that are not listed.

[0108] For ease of understanding, first, the technical terms involved in the embodiments of the present application are explained accordingly.

[0109] (1) Dynamic range ((dynamic range), the ratio of the maximum brightness to the minimum brightness in a video or image signal.

[0110] (2) High dynamic range (high dynamic range, HDR), an image with a dynamic range between 0.001 and 10000 nits, where nit is a unit of illumination.

[0111] (3) Standard dynamic range (standard dynamic range, SDR), also referred to as low dynamic range hereinafter, an image with a dynamic range generally between 1 nit and 100 nits.

[0112] (4) Metadata, which is used to record the key information of the images in a scene or a frame. In the embodiments of the present application, the metadata may further include the parameters of the mapping curve, such as the coordinate values of the start node and the end node of the mapping curve in the first coordinate system. Among them, the metadata may further include dynamic metadata and static metadata; the dynamic metadata can be understood as the data associated with each frame of the image (hereinafter also referred to as attribute information), and this data can change with different pictures, such as the average value, the maximum value, the minimum value, etc. of the pixel brightness in the scene; the static metadata can be understood as the data associated with the image sequence, and this data remains unchanged within the image sequence.

[0113] (5) Tone mapping, which maps the brightness of the HDR image to the brightness range that the target display device can display. Specifically, it can be divided into the mapping from high dynamic range to low dynamic range and the mapping from low dynamic range to high dynamic range. For example, if the HDR image includes a light signal of 4000 nits, while the HDR display capability of the target display device (TV, iPad) is only 500 nits, then it is necessary to map the 4000-nit signal to the 500-nit device, that is, the tone mapping process from high to low. Another example, if the HDR image includes an SDR signal of 100 nits, while the display capability of the target display device is only 2000 nits, then it is necessary to map the 100-nit signal to the 2000-nit device, that is, the tone mapping process from low to high.

[0114] (6) Image contrast: The measurement of the different brightness levels between the brightest white and the darkest black in the light and dark areas of an image, that is, the size of the gray-scale contrast of an image. The larger the difference range, the greater the contrast; the smaller the difference range, the smaller the contrast. A good contrast ratio of 120:1 can easily display vivid and rich colors, and when the contrast ratio reaches 300:1, it can support all levels of colors.

[0115] (7) Mapping curve, which is a function curve used to map the brightness of the HDR image. The embodiments of the present application mainly relate to the Nth spline function curve.

[0116] (8) Display adaptation: Performing tone processing on the HDR video or image signal to adapt to the display characteristics of the target display device.

[0117] In order to perform tone mapping on an HDR image, a mapping curve may be generated based on a base curve, and then tone mapping may be performed on the HDR image based on the mapping curve. The base curve may be a straight line, an electro-optical curve for scene luminance fidelity (SLF), a sigmoidal function curve, or a Bezier curve. Figure 1 A mapping curve is shown, such as Figure 1 As shown, this basic curve (i.e. Figure 1 The dotted line in the figure is modified to obtain a mapping curve, where the mapping curve between point 1 and point 2 is a linear spline function curve, the mapping curves between point 2 to point 3 and point 3 to point 4 are cubic spline function curves, and the mapping curve after point 4 is the basic curve.

[0118] from Figure 1 It can be seen that the mapping curve is restricted by the base curve, which leads to poor mapping effect (for example, poor tone mapping effect for pixels in extremely bright or dark areas), which in turn leads to poor display effect of HDR images.

[0119] In order to flexibly map HDR images, an embodiment of the present application proposes a tone mapping method, which can be executed by a back-end device. In this method, the back-end device can receive metadata and HDR images from the front-end device, generate a mapping curve according to the characteristic parameters of the mapping curve contained in the metadata, and perform tone mapping on the HDR image based on the mapping curve. The characteristic parameters of the mapping curve are determined by the front-end device according to the brightness distribution of the HDR image. Therefore, the embodiment of the present application determines the mapping curve parameters by combining the brightness distribution of the HDR image without relying on the basic curve, making the design of the mapping curve more flexible, thereby enabling better brightness mapping effects for specific brightness ranges (for example, very bright or very dark), thereby improving the display effect of the HDR image.

[0120] Figure 2A This is a schematic diagram of a possible system architecture applicable to an embodiment of the present application. The system architecture of the embodiment of the present application includes: front-end equipment, transmission links, and back-end equipment.

[0121] The front-end device is used to capture or produce HDR content (e.g., video or images). An exemplary front-end device can be a video capture device or a video production device.

[0122] In a possible embodiment, the front-end device can also be used to extract corresponding metadata from the HDR content. The metadata may include parameter information of the mapping curve and dynamic metadata and static metadata corresponding to the HDR content. The front-end device can encode the HDR content and the metadata and send them to the back-end device through a transmission link. Specifically, the HDR content and the metadata can be transmitted in the form of a single data packet or separately in two data packets. The embodiments of the present application do not make specific limitations in this regard.

[0123] The back-end device can be used to receive the metadata and the HDR content, determine the mapping curve according to the mapping curve parameters included in the metadata, and perform tone mapping on the HDR content according to the mapping curve to convert it into display content adapted to the display device in the back-end device. It should be understood that Figure 2A "Standard dynamic" in "standard dynamic HDR" and "standard dynamic SDR" is relative to the HDR content generated by the front-end device.

[0124] It should also be understood that in different embodiments, the back-end device may also include a display device with a display capability having a higher dynamic range than the HDR content generated by the front-end device. The process of display adaptation can also be applied to this system architecture, and the present application does not make limitations in this regard.

[0125] The embodiments of the present application also provide an end-to-end HDR image processing system, which can be applied to Figure 2A the system architecture shown. In Figure 2B , the HDR content takes the HDR video as an example.

[0126] Please refer to Figure 2B , the image processing system includes: an HDR preprocessing module, an HDR video encoding module, an HDR video decoding module, and a tone mapping module.

[0127] Among them, the HDR preprocessing module and the HDR video encoding module can be located in Figure 2A the front-end device shown, and the HDR video decoding module and the tone mapping module can be located in Figure 2A the back-end device shown.

[0128] The HDR preprocessing module: is used to extract dynamic metadata (for example, the maximum value, minimum value, average value, and change range of brightness) from the HDR video, and determine the mapping curve parameters according to the dynamic metadata and the display capability of the target display device, write the mapping curve parameters into the dynamic metadata to obtain HDR metadata, and transmit it. Among them, the HDR video can be captured or an HDR video processed by a colorist; the display capability of the target display device is the brightness range that the target display device can display.

[0129] HDR Video Encoding Module: Used to perform video encoding on HDR video and HDR metadata according to video compression standards (e.g., AVS or HEVC standards) (e.g., embed HDR metadata into a user-defined part of the bitstream), and output the corresponding bitstream (AVS or HEVC bitstream).

[0130] HDR Video Decoding Module: Used to decode the generated bitstream (AVS bitstream or HEVC bitstream) according to the standard corresponding to the bitstream format, and output the decoded HDR video and HDR metadata.

[0131] Tone Mapping Module: Used to generate a mapping curve according to the parameters of the mapping curve in the decoded HDR metadata, and perform tone mapping on the decoded HDR video (i.e., HDR adaptation processing or SDR adaptation processing), and display the tone-mapped HDR-adapted video on an HDR display terminal, or display the SDR-adapted video on an SDR display terminal.

[0132] Exemplarily, the HDR preprocessing module can be present in a video capture device or a video production device.

[0133] Exemplarily, the HDR video encoding module can be present in a video capture device or a video production device.

[0134] Exemplarily, the HDR video decoding module can be present in a video conversion device such as a set-top box, a television display device, a mobile terminal display device, and network live broadcast and network video applications.

[0135] Exemplarily, the tone mapping module can be present in a video conversion device such as a set-top box, a television display device, a mobile terminal display device, and network live broadcast and network video applications. More specifically, the tone mapping module can exist in the form of a chip in a set-top box, a television display, and a mobile terminal display, and can exist in the form of a software program in a video conversion device such as network live broadcast and network video applications.

[0136] In a possible embodiment, when both the tone mapping module and the HDR video decoding module are present in a set-top box, the set-top box can complete the functions of receiving, decoding, and tone mapping of the video bitstream. The set-top box sends the decoded video data to a display device for display through a high definition multimedia interface (HDMI), so that users can enjoy the video content.

[0137] Next, in combination with specific drawings, the process of the front-end device generating metadata is introduced.

[0138] Please refer to Figure 3A , Figure 3AThe flowchart shows a method for generating metadata, which can be executed by a front-end device and specifically includes the following steps:

[0139] S301. Obtain at least one frame of image.

[0140] It should be understood that the at least one frame of image can be one frame or multiple frames of images. When the at least one frame of image is multiple consecutive frames of images, the at least one frame of image can be understood as a video.

[0141] S302. Process the at least one frame of image to generate metadata.

[0142] Among them, the metadata includes mapping curve parameters. For example, the number M of mapping curves, the coordinate values of the starting node and the ending node of each mapping curve in the first coordinate system; among them, each mapping curve is an N-degree spline curve. The horizontal coordinate of the first coordinate system is the image brightness before tone mapping, and the vertical coordinate of the first coordinate system is the image brightness after tone mapping. The coordinate origin of the first coordinate is (0, 0), that is, the point where the image brightness before and after tone mapping is 0. Optionally, the metadata can also include the static metadata and dynamic metadata of the at least one frame of image.

[0143] In a possible embodiment, the metadata includes the attribute information (i.e., dynamic metadata) of the at least one frame of image. The front-end device processes the at least one frame of image to generate metadata, specifically including: processing the at least one frame of image to extract the attribute information of the at least one frame of image, and using the attribute information as part of the metadata.

[0144] Optionally, the attribute information can specifically include the maximum value, minimum value, average value, and change range of the brightness in the same frame of image or multiple images corresponding to the same scene (for example, images of continuously shooting the same object in multiple consecutive frames). Among them, the change range represents the degree of deviation and the degree of concentration of the pixel brightness of the image sub-block from the average brightness, and the smaller the change range, the more concentrated the brightness. For example, the change range can specifically be variance.

[0145] Among them, there are various implementation manners for the front-end device to process the at least one frame of image to extract the attribute information of the at least one frame of image, including but not limited to the following manners:

[0146] Manner 1. When there is only one frame of image in the at least one frame of image, determine the maximum value, minimum value, average value, and change range of the pixel brightness in this frame of image, and the attribute information can be obtained.

[0147] Manner 2. When the at least one frame of image is multiple consecutive frames of images of the same scene, count the maximum value, minimum value, average value, and change range of the pixel brightness in these multiple frames of images to obtain the final attribute information.

[0148] Optionally, when the front-end device extracts the attribute information of at least one frame of image, it can specifically extract the attribute information of a local area (such as the region of interest) in the at least one frame of image. Exemplarily, as Figure 3B shown in the image scene 1, if the region of interest is the area where a cloud is located, then when the front-end device determines the attribute information of the image scene 1, it can specifically determine the pixel brightness values corresponding to the cloud in the at least one frame of image corresponding to the scene 1, and further statistically obtain the maximum value, minimum value, average value, and change range of the brightness of the cloud, and use the maximum value, minimum value, average value, and change range of the brightness of the cloud as the attribute information of the scene 1.

[0149] In a possible embodiment, after the front-end device extracts the attribute information of at least one frame of image, it can determine metadata based on the attribute information.

[0150] In a possible embodiment, the metadata includes the number of mapping curves. The front-end device determines the metadata based on the attribute information. The specific implementation manner can include: determining the brightness distribution of at least one frame of image according to the attribute information; dividing the brightness in the at least one frame of image into at least one brightness range based on the brightness distribution; and determining the number M according to the number F of the brightness ranges. Where M is greater than or equal to F, and each brightness range corresponds to at least one mapping curve. Specifically, the brightness distribution can be implemented by a brightness distribution histogram or a brightness distribution pie chart, and the embodiments of the present application do not make specific limitations.

[0151] Example 1, please refer to Figure 4A and Figure 4B , the front-end device analyzes the brightness histogram of the first image and finds that the brightness distribution of the first image is relatively average, then divides the brightness of the first image into a low brightness range (0 - 0.3 cd / m2), a medium brightness range (0.3 - 0.6 cd / m2), and a high brightness range (0.6 - 0.9 cd / m2), believes that there is interesting content in each brightness range, and generates mapping curves using an odd or even number of cubic spline functions within each brightness range.

[0152] Figure 4A and Figure 4B In the coordinate systems of

[0153] Figure 4A and Figure 4B , the horizontal coordinate of the coordinate system is used to represent the brightness of the image before tone mapping, and the vertical coordinate is used to represent the brightness of the image after tone mapping.

[0154] Such as Figure 4AAs shown, two mapping curves are respectively adopted for the low brightness range, the medium brightness range, and the high brightness range. Specifically, the starting node of the first mapping curve for the low brightness range is point 0 (0, 0), and the ending node is point 1 (0.28, 0.29); the starting node of the second mapping curve for the low brightness range is point 1 (0.28, 0.29), and the ending node is point 2 (0.3, 0.36); the starting node of the first mapping curve for the medium brightness range is point 2 (0.3, 0.36), and the ending node is point 3 (0.43, 0.55); the starting node of the second mapping curve for the medium brightness range is point 3 (0.43, 0.55), and the ending node is point 4 (0.6, 0.58); the starting node of the first mapping curve for the high brightness range is point 4 (0.6, 0.58), and the ending node is point 5 (0.75, 0.76); the starting node of the second mapping curve for the high brightness range is point 5 (0.75, 0.76), and the ending node is point 6 (0.9, 0.98).

[0155] As Figure 4B shown, one mapping curve is respectively adopted for the low brightness range, the medium brightness range, and the high brightness range. Among them, the starting node of the mapping curve for the low brightness range is point 0 (0, 0), and the ending node is point 1 (0.3, 0.38); the starting node of the mapping curve for the medium brightness range is point 1 (0.3, 0.38), and the ending node is point 2 (0.6, 0.58); the starting node of the mapping curve for the high brightness range is point 2 (0.6, 0.58), and the ending node is point 3 (0.9, 0.88).

[0156] For Example 2, please refer to Figure 5A and Figure 5B wherein, by analyzing the brightness histogram of the second image, if the front-end device determines that the brightness of the second image is relatively evenly distributed in the low brightness range and the high brightness range, then the brightness of the second image is divided into the low brightness range and the high brightness range, and an odd or even number of cubic spline functions are used to generate mapping curves in the low brightness range and the high brightness range.

[0157] Figure 5A and Figure 5B the horizontal coordinate of the coordinate system in

[0158] Figure 5A and Figure 5B is used to represent the brightness of the image before tone mapping, and the vertical coordinate is used to represent the brightness of the image after tone mapping, and the coordinate origin is (0, 0).

[0159] As Figure 5AAs shown, two mapping curves are adopted for the low brightness range and the high brightness range respectively. Among them, the starting node of the first mapping curve in the low brightness range is point 0 (0, 0), and the ending node is point 1 (0.6, 0.7). The starting node of the second mapping curve in the low brightness range is point 1 (0.6, 0.7), and the ending node is point 2 (0.46, 0.58). The starting node of the first mapping curve in the high brightness range is point 2 (0.46, 0.58), and the ending node is point 3 (0.72, 0.68). The starting node of the second mapping curve in the high brightness range is point 3 (0.72, 0.68), and the ending node is point 4 (0.9, 0.88).

[0160] As Figure 5B shown, one mapping curve is adopted for the low brightness range and the high brightness range respectively. Among them, the starting node of the mapping curve in the low brightness range is point 0 (0, 0), and the ending node is point 1 (0.46, 0.58). The starting node of the mapping curve in the high brightness range is point 1, and the ending node is point 2 (0.9, 0.88).

[0161] For Example 3, please refer to Figure 6A and Figure 6B . The front-end device analyzes the brightness histogram of the third image and determines that the brightness of the third image is evenly distributed in the medium brightness range and there is no extremely dark or extremely bright area. Then, the brightness of the third image is only divided into the medium brightness range. Among them, Figure 6A and Figure 6B in the coordinate system, the horizontal coordinate is used to represent the brightness of the image before tone mapping, and the vertical coordinate is used to represent the brightness of the image after tone mapping. The coordinate origin is (0, 0).

[0162] Figure 6A and Figure 6B differ in that the number of mapping curves in the medium brightness range is different.

[0163] As Figure 6A shown, two mapping curves are adopted for the medium brightness range. Among them, the starting node of the first mapping curve in the medium brightness range is point 0 (0, 0), and the ending node is point 1 (0.46, 0.48). The starting node of the second mapping curve in the medium brightness range is point 1 (0.46, 0.48), and the ending node is point 2 (0.9, 0.88).

[0164] ?As Figure 6B shown, one mapping curve is adopted for the medium brightness range. Among them, the starting node of the mapping curve in the medium brightness range is point (0, 0), and the ending node is point 1 (0.9, 0.88).

[0165] In Figure 4A , Figure 5A , Figure 6AIn the corresponding example, two N-degree spline curves are used for the same brightness range to form a mapping curve, effectively improving the flexibility of the mapping curve and thus enhancing the effect of tone mapping.

[0166] In Figure 4B , Figure 5B , Figure 6B In the corresponding example, one N-degree spline curve is used for the same brightness range to form a mapping curve, requiring fewer mapping curve parameters and effectively improving the efficiency of metadata transmission.

[0167] It should be noted that the brightness ranges in the above examples are taken as three ranges: low brightness range, medium brightness range, and high brightness range. In actual applications, the brightness range can also be divided in other ways. For example, the brightness range can also have medium-high brightness range, medium-low brightness range, etc. This application does not make specific restrictions.

[0168] Furthermore, it should be noted that in each of the above brightness ranges, one or two mapping curves are mainly taken as examples. In actual applications, the number of mapping curves in each brightness range can be more, and this application does not make specific restrictions.

[0169] [[ID=I8]]In a possible embodiment, the metadata includes the coordinate values of the start node and end node of each mapping curve in the first coordinate system. The front-end device determines the metadata based on the attribute information, which can specifically include: determining the coordinate values of the start node and end node of each mapping curve in the first coordinate system according to the brightness distribution and the maximum brightness value that the back-end device can display.

[0170] Example 1, when the brightness range of at least one frame of image is divided into a low brightness range, a medium brightness range, and a high brightness range, and each brightness range corresponds to one mapping curve, the start node of mapping curve 1 corresponding to the low brightness range can be set to the origin (0, 0), and the abscissa of the end node of mapping curve 1 can be set to the brightness value at which the brightness of 30% of the pixels in at least one frame of image is not greater than, and the ordinate of the end node of mapping curve 1 can be set to 50% of the maximum brightness value that the back-end device can display; the start node of mapping curve 2 corresponding to the medium brightness range is set to the end node of mapping curve 1, and the abscissa of the end node of mapping curve 2 can be set to the brightness value at which the brightness of 70% of the pixels in at least one frame of image is less than, and the ordinate of the end node of mapping curve 2 can be set to 70% of the maximum brightness value that the back-end device can display; the start node of mapping curve 3 corresponding to the high brightness range is set to the end node of mapping curve 2, the abscissa of the end node of mapping curve 3 can be set to the maximum value of the pixel brightness in at least one frame of image, and the ordinate of the end node of mapping curve 3 can be set to the maximum brightness value that the back-end device can display.

[0171] Taking Figure 4BTaking the high brightness range in [ID] as an example, if the maximum value of the pixel brightness in the first image is 0.9 cd / m2 and the maximum brightness value that the back-end device can display is 0.98 cd / m2, then the abscissa of the cut-off node of the mapping curve corresponding to the high brightness range is set to 0.9 cd / m2, and the ordinate of the cut-off node of this mapping curve is set to 0.98 cd / m2.

[0172] Example 2: When dividing the brightness range of at least one frame of image into a low brightness range and a high brightness range, and each brightness range corresponds to a mapping curve, the starting node of the mapping curve 4 corresponding to the low brightness range can be set to the origin (0, 0). The abscissa of the cut-off node of the mapping curve 4 can be set to the brightness value that the brightness of 50% of the pixels in at least one frame of image is not greater than, and the ordinate of the cut-off node of the mapping curve 4 can be set to 30% of the maximum brightness value that the back-end device can display; the starting node of the mapping curve 5 corresponding to the high brightness range is set to the cut-off node of the mapping curve 4. The abscissa of the cut-off node of the mapping curve 4 can be set to the maximum value of the pixel brightness in at least one frame of image, and the ordinate of the cut-off node of the mapping curve 4 can be set to the maximum brightness value that the back-end device can display.

[0173] Taking Figure 5B Taking the low brightness range in [ID] as an example, if the brightness of 50% of the pixels in the second image is less than 0.46 cd / m2 and 30% of the maximum brightness value that the back-end device can display is 0.48 cd / m2; then the abscissa of the cut-off node of the mapping curve corresponding to the low brightness range is set to 0.46 cd / m2, and the ordinate of the cut-off node of this mapping curve is set to 0.48 cd / m2.

[0174] Example 3: When only dividing the brightness range of at least one frame of image into a medium brightness range, and the medium brightness range corresponds to mapping curves 6 and 7, the starting node of the mapping curve 6 can be set to the origin (0, 0). The abscissa of the cut-off node of the mapping curve 6 can be set to the value that the brightness of 50% of the pixels in at least one frame of image is not greater than, and the ordinate of the cut-off node of the mapping curve 6 can be set to 53% of the maximum display brightness that the back-end device can display; the starting node of the mapping curve 7 can be set to the cut-off node of the mapping curve 6. The abscissa of the cut-off node of the mapping curve 7 can be set to the maximum value of the pixel brightness in at least one frame of image, and the ordinate of the cut-off node of the mapping curve 7 can be set to the maximum brightness value that the back-end device can display.

[0175] Taking Figure 6A Taking the medium brightness range in [ID] as an example, the medium brightness range corresponds to mapping curves 6 and 7. In Figure 6AThe curve between 0 o'clock and 1 o'clock is the mapping curve 6, and the curve between 1 o'clock and 2 o'clock is the mapping curve 7; the maximum value of the pixel brightness in the third image is 0.9 cd / m2, and the maximum brightness value that the backend device can display is 0.98 cd / m2; the brightness of 50% of the pixels in the third image is not greater than 0.46 cd / m2, and 53% of the maximum display brightness of the backend device is 0.48 cd / m2; then the abscissa of the cut-off node of the mapping curve 6 is set to 0.46 cd / m2, and the ordinate of the cut-off node of the mapping curve 6 is set to 0.48 cd / m2; the abscissa of the cut-off node of the mapping curve 7 is set to 0.9 cd / m2, and the ordinate of the cut-off node of the mapping curve 7 is set to 0.98 cd / m2.

[0176] In a possible embodiment, the metadata may further include a preset first derivative of the starting node and a preset first derivative of the cut-off node of each mapping curve.

[0177] Correspondingly, based on the attribute information, the front-end device determines the metadata, and further includes: determining a preset first derivative of the starting node and a preset first derivative of the cut-off node of each mapping curve according to the balance requirement between the brightness and the contrast of at least one frame of image. It should be understood that the contrast here is the aforementioned image contrast.

[0178] For example, if it is desired to maintain the image brightness of at least one frame of image, the first derivative can be set larger, and if it is desired to enhance the image contrast of at least one frame of image, the first derivative can be set smaller.

[0179] Based on the above description, it can be seen that in the embodiment of the present application, the parameter design of the mapping curve is determined according to the brightness distribution of the image, and can be unrestricted by the basic curve, so the form is relatively flexible, and a large adjustment space can be provided for different brightness ranges, so as to achieve a better adaptation effect.

[0180] In a possible embodiment, after the front-end device processes at least one frame of image to generate metadata, it can also receive a first operation, and based on the first operation, modify the metadata to obtain updated metadata. Among them, the first operation can be used to change at least one of the number M of mapping curves, the coordinate value of the starting node in the first coordinate system, the coordinate value of the cut-off node in the first coordinate system, the preset first derivative of the starting node, and the preset first derivative of the cut-off node.

[0181] It should be understood that the first operation can be understood as the operation of the user. For example, after the front-end device processes Image A, it divides Image A into two brightness ranges and designs the parameters of two mapping curves for it. However, in order to improve the efficiency of tone mapping, the director can adjust the number of mapping curves corresponding to Image A to 1. Then, the operation of the director to change the number of mapping curves is the first operation. In this way, the front-end device can dynamically adjust the parameter information of the mapping curve according to the user's needs, effectively meet the user's preference requirements, and thus effectively improve the user experience.

[0182] In a possible embodiment, after the front-end device generates metadata, it can encode at least one frame of image and the metadata and send them to the back-end device in the form of a bitstream. After receiving the bitstream, the back-end device decodes it, then determines the mapping curve according to the decoded metadata, and finally performs tone mapping on the images in the HDR video according to the mapping curve and converts them into display images adapted to the display device.

[0183] The following introduces the process of tone mapping in combination with specific drawings.

[0184] Please refer to Figure 7 , Figure 7 FIG. shows a schematic flowchart of a tone mapping method, which can be executed by a back-end device and specifically includes the following steps:

[0185] S701. Receive at least one frame of image and metadata.

[0186] Generally, at least one frame of image (HDR image or video) and metadata are both transmitted in the form of a bitstream. In the embodiments of the present application, the data corresponding to at least one frame of image can be an optical signal or an electrical signal, and the present application does not make any limitation in this regard. If it is an electrical signal, the electrical signal can be converted into an optical signal according to the optoelectronic signal conversion function.

[0187] It should be understood that in the embodiments of the present application, the format of the metadata is not limited. Exemplarily, the metadata is the same as the metadata sent by the front-end device above, that is, it can include the number M of mapping curves, and the coordinate values of the starting node and the ending node of each mapping curve in the first coordinate system; wherein, the mapping curve is an Nth-order spline curve, the horizontal coordinate of the first coordinate system represents the image brightness before tone mapping, and the vertical coordinate of the first coordinate system represents the image brightness after tone mapping; M is an integer greater than or equal to 1, and N is an integer greater than or equal to 3.

[0188] In a possible embodiment, the display device in the back-end device can directly receive the bitstream from the front-end device and decode the bitstream to obtain the at least one frame of image and metadata.

[0189] In another possible embodiment, the backend device includes a decoding device (such as a set-top box) and a display device (such as a TV). The decoding device receives a bitstream from the frontend device, decodes the bitstream, and after obtaining at least one frame of image and metadata, sends the decoded at least one frame of image and metadata to the display device.

[0190] Optionally, the metadata further includes the preset first-order derivative of the starting node and the preset first-order derivative of the ending node. Specifically, the metadata received by the backend device includes the preset first-order derivative of the starting node and the preset first-order derivative of the ending node of each mapping curve.

[0191] Alternatively, the metadata does not include the preset first-order derivative of the starting node and the preset first-order derivative of the ending node. Instead, the backend device stores a mapping table between the starting nodes, ending nodes, and preset first-order derivatives of all mapping curves. After receiving the metadata, the backend device can determine the preset first-order derivatives corresponding to the starting nodes and ending nodes of each curve in the metadata according to this mapping table.

[0192] S702. Generate M mapping curves according to the metadata.

[0193] It should be understood that each frame of the at least one frame of image may correspond to at least one brightness range, and each brightness range corresponds to at least one mapping curve, that is, each frame of image may correspond to one or more brightness ranges, and one brightness range may correspond to one or more mapping curves.

[0194] Exemplarily, the first image may correspond to a low brightness range, a high brightness range, a medium brightness range; or, correspond to a low brightness range, a high brightness range, or, correspond to a low brightness range. The embodiments of the present application do not make specific limitations. Each brightness range may correspond to an odd number or an even number of mapping curves, specifically, one or two mapping curves. The embodiments of the present application do not make specific limitations.

[0195] In one possible embodiment, the function expression corresponding to the mapping curve may be an Nth-order polynomial, and the Nth-order polynomial includes N + 1 monomials, and each monomial corresponds to a coefficient.

[0196] The function expression corresponding to the i-th mapping curve among the at least one mapping curve corresponding to each brightness range may specifically be:

[0197] y = di*(x - x2) N +……+ci*(x - x2) 2 +bi*(x - x2)+ai;

[0198] Where x is the independent variable, used to represent the image brightness before tone mapping, and y is the dependent variable, used to represent the image brightness after tone mapping; ai is the coefficient of the constant term in the i-th mapping curve, bi is the coefficient of the first-degree monomial in the i-th mapping curve, ci is the coefficient of the second-degree monomial in the i-th mapping curve, di is the coefficient of the N-th degree monomial in the i-th mapping curve; x2 is the horizontal coordinate of the starting node of the i-th mapping curve; ai, bi, ci, and di are all not zero.

[0199] Specifically, the first function expression corresponding to the i-th mapping curve can be a cubic spline function: y = di*(x - x2)^3 + ci*(x - x2)^2 + bi*(x - x2) + ai; where x is the independent variable, used to represent the image brightness before tone mapping, y is the dependent variable, used to represent the image brightness after tone mapping; ai is the coefficient of the constant term in the i-th mapping curve, bi is the coefficient of the first-degree monomial in the i-th mapping curve, ci is the coefficient of the second-degree monomial in the i-th mapping curve, di is the coefficient of the cubic monomial in the i-th mapping curve; x2 is the horizontal coordinate of the starting node of the i-th mapping curve; ai, bi, ci, and di are all not zero.

[0200] Optionally, the first function expression corresponding to the i-th mapping curve can also be a quartic spline function: y = ei*(x - x2) 4 + di*(x - x2) 3 + ci*(x - x2) 2 + bi*(x - x2) + ai; where x is the independent variable, used to represent the image brightness before tone mapping, y is the dependent variable, used to represent the image brightness after tone mapping; ai is the coefficient of the constant term in the i-th mapping curve, bi is the coefficient of the first-degree monomial in the i-th mapping curve, ci is the coefficient of the second-degree monomial in the i-th mapping curve, di is the coefficient of the cubic monomial in the i-th mapping curve, ei is the coefficient of the quartic monomial in the i-th mapping curve; x2 is the horizontal coordinate of the starting node of the i-th mapping curve; ai, bi, ci, and di are all not zero.

[0201] It should be understood that the above is only an example rather than a limitation with cubic spline function curves and quartic spline function curves. In actual applications, it can also be other spline function curves.

[0202] Among them, there are multiple implementation methods for the backend device to generate mapping curves corresponding to each brightness range according to the metadata, including but not limited to the following methods:

[0203] Method 1: The first brightness range in at least one brightness range corresponds to a first mapping curve, and the first mapping curve includes a first starting node and a first ending node; according to the metadata, generate M mapping curves, including: substituting the coordinate value of the first starting node into the first function expression to obtain the first coefficient a1 of the first mapping curve; substituting the coordinate value of the first ending node into the first function expression to obtain the first equation; making the first derivative of the first mapping curve at the first starting node equal to the preset first derivative of the first starting node to obtain the second equation; making the first derivative of the first mapping curve at the first ending node equal to the preset first derivative of the first ending node to obtain the third equation; combining the first equation, the second equation, and the third equation to obtain the first system of equations; analyzing the first system of equations to obtain the second coefficient b1, the third coefficient c1, and the fourth coefficient d1 of the first mapping curve; generating the first mapping curve based on the first coefficient a1, the second coefficient b1, the third coefficient c1, and the fourth coefficient d1 of the first mapping curve.

[0204] Exemplarily, please continue to refer to Figure 4B , taking the medium brightness range as an example, assume the expression of the first mapping curve from point 1 to point 2 is:

[0205] F1(x) = d1*(x - x1) 3 + c1*(x - x1) 2 + b1*(x - x1)+ a1 (1)

[0206] The starting node 1 of this first mapping curve has coordinates (x1, y1) in the first coordinate system, and the ending node 2 of the first mapping curve has coordinates (x2, y2) in the first coordinate system; where a1, b1, c1, and d1 are the function coefficients of the first mapping curve.

[0207] The specific process of solving the function coefficients is as follows:

[0208] Substitute the coordinates of point 1 into equation (1) to get y1 = a1;

[0209] Substitute the coordinates of point 1 into equation (1) to get the first equation y2 = d1*(x2 - x1) 3 + c1*(x2 - x1) 2 + b1*(x2 - x1)+ a1;

[0210] Make the first derivative of equation (1) at point 1 equal to the preset first derivative G1 of point 1 to obtain the second equation G1 = d1*3(x - x1) 2 + c1*(x - x1)+ b1;

[0211] Make the first derivative of equation (1) at point 2 equal to the preset first derivative G2 of point 2 to obtain the third equation G2 = d1*3(x2 - x1)2 +c1*(x2 - x1)+b1;

[0212] Simultaneously solve y1 = a1, the first equation, the second equation, and the third equation to obtain the first system of equations; analyze the first system of equations to obtain the second coefficient b1, the third coefficient c1, and the fourth coefficient d1 of the first mapping curve; generate the first mapping curve based on the first coefficient a1, the second coefficient b1, the third coefficient c1, and the fourth coefficient d1 of the first mapping curve.

[0213] It should be noted here that in order to ensure the continuity of the entire mapping curve, each node can only have a unique derivative value; at the same time, in order to prevent the occurrence of an oscillating cubic spline function and avoid excessive correction, the first derivative of the first mapping curve is always greater than 0.

[0214] Method 2: Further divide each brightness range in at least one brightness range into sub - brightness ranges. For example, divide the medium - brightness range into the first sub - brightness range and the second sub - brightness range, where the first sub - brightness range corresponds to the second mapping curve corresponding to the second brightness range, and the first sub - brightness range corresponds to the third mapping curve; the maximum value in the first sub - brightness range is equal to the minimum value in the second sub - brightness range; the second mapping curve includes a second starting node and a second ending node; the third mapping curve includes a third starting node and a third ending node; the coordinate values of the second ending node and the third starting node are the same.

[0215] When the backend device generates M mapping curves based on the metadata, it can simultaneously calculate the function coefficients of the second mapping curve and the third mapping curve. The specific process can be: substitute the coordinate value of the second starting node into the first function expression to obtain the first coefficient a2 of the second mapping curve; substitute the coordinate value of the third starting node into the first function expression to obtain the first coefficient a3 of the third mapping curve; the first derivative of the second mapping curve at the second starting node is equal to the preset first derivative of the second starting node, and establish the fourth equation; analyze the fourth equation to obtain the second coefficient b2 of the second mapping curve; construct the second system of equations according to the coordinate value of the second ending node, the coordinate value of the third ending node, and the preset first derivative of the third ending node; analyze the second system of equations to obtain the third coefficient c2, the fourth coefficient d2 of the second mapping curve, and the second coefficient b3, the third coefficient c3, and the fourth coefficient d3 of the third mapping curve; generate the second mapping curve based on the first coefficient a2, the second coefficient b2, the third coefficient c2, and the fourth coefficient d2 of the second mapping curve; and generate the third mapping curve based on the first coefficient a3, the second coefficient b3, the third coefficient c3, and the fourth coefficient d3 of the third mapping curve.

[0216] There are multiple implementations for constructing the second set of equations based on the coordinate value of the second cutoff node, the coordinate value of the third cutoff node, and the preset first-order derivative of the third cutoff node, including but not limited to the following:

[0217] Method 1: Substitute the coordinate values of the second cutoff node and the coordinate values of the third cutoff node into the first function expression to obtain the fifth equation and the sixth equation respectively, so that the first-order derivative of the third cutoff node at the third mapping curve is equal to the preset first-order derivative of the third cutoff node, and obtain the seventh equation; make the first-order derivative of the second mapping curve at the second cutoff node equal to the first-order derivative of the third mapping curve at the third starting node, and obtain the eighth equation; make the second-order derivative of the second mapping curve at the second cutoff node equal to the second-order derivative of the third mapping curve at the third starting node, and obtain the ninth equation; combine the fifth equation, the sixth equation, the seventh equation, the eighth equation, and the ninth equation to obtain the second group of equations.

[0218] In method 1, the function coefficients of the mapping curves are solved by ensuring that the first-order derivative of the second mapping curve at the second cutoff node is equal to the first-order derivative of the third mapping curve at the third starting node, and that the second-order derivative of the second mapping curve at the second cutoff node is equal to the second-order derivative of the third mapping curve at the third starting node. This eliminates the need to carry the second cutoff node and the third starting node and the preset first-order derivative in the metadata, effectively improving the transmission efficiency of the metadata.

[0219] Method 2: Substitute the coordinate values of the second cutoff node and the coordinate values of the third cutoff node into the first function expression to obtain the fifth equation and the sixth equation respectively, so that the first-order derivative of the third cutoff node at the third mapping curve is equal to the preset first-order derivative of the third cutoff node, and obtain the seventh equation; or, make the first-order derivative of the second mapping curve at the second cutoff node equal to the preset first-order derivative of the second cutoff node, and obtain the eighth equation, so that the first-order derivative of the third mapping curve at the third starting node is equal to the preset first-order derivative of the third starting node, and obtain the ninth equation; combine the fifth equation, the sixth equation, the seventh equation, the eighth equation, and the ninth equation to obtain the second group of equations.

[0220] In method 2, the function coefficients of the mapping curve are solved by making the first-order derivative of the second mapping curve at the second cutoff node equal to the preset first-order derivative of the second cutoff node, and the first-order derivative of the third mapping curve at the third starting node equal to the preset first-order derivative of the third starting node, which can reduce the calculation amount of the back-end device.

[0221] For example, please see Figure 4A Taking the medium brightness range as an example, the second mapping curve of the medium brightness range is a mapping curve from 2 to 3 points, and its expression is:

[0222] F2(x) = d2 * (x - x2) 3 + c2 * (x - x2) 2 + b2 * (x - x2) + a2 (2)

[0223] The third mapping curve in the medium brightness range is the mapping curve from point 3 to point 4, and its expression is:

[0224] F3(x) = d3 * (x - x3) 3 + c3 * (x - x3) 2 + b3 * (x - x3) + a3 (3)

[0225] The form of each node coordinate is (xi, yi), such as the coordinate of point 2 is (x2, y2); the coordinate of point 3 is (x3, y3); the coordinate of point 4 is (x4, y4); where, d2, c2, b2, a2 are the function coefficients of the second mapping curve, and d3, c3, b3, a3 are the function coefficients of the third mapping curve.

[0226] The process of jointly solving the function coefficients is as follows:

[0227] Substitute the coordinate values of point 2 into equation (2), and get y2 = a2;

[0228] Substitute the coordinate values of point 3 into equation (3), and get y3 = a3;

[0229] Make the first derivative of equation (2) at point 2 equal to the preset first derivative G3 at point 2, and get the fourth equation G3 = b2;

[0230] Substitute the coordinate values of point 3 into equation (2), and get the fifth equation:

[0231] y3 = d2 * (x3 - x2) 3 + c2 * (x3 - x2) 2 + b2 * (x3 - x2) + a2;

[0232] Substitute the coordinate values of point 4 into equation (3), and get the sixth equation;

[0233] y4 = d3 * (x4 - x3) 3 + c3 * (x4 - x3) 2 + b3 * (x4 - x3) + a3;

[0234] Make the first derivative of equation (3) at point 4 equal to the preset first derivative G4 at point 4, and get the seventh equation:

[0235] G4 = d3 * 3(x4 - x3) 2 + c3 * (x4 - x3) + b3;

[0236] By making the first-order derivative of equation (2) at point 3 equal to the first-order derivative of equation (3) at point 3, we obtain the eighth equation:

[0237] d2*3(x3-x2) 2 +c2*(x3-x2)+b2=b3;

[0238] The second-order derivative of equation (2) at point 3 is equal to the second-order derivative of equation (3) at point 3, and the ninth equation is obtained:

[0239] d2*6(x3-x2)+c2=0;

[0240] By combining the above equations, we can obtain the third coefficient c2 and the fourth coefficient d2 of the second mapping curve and the second coefficient b3, the third coefficient c3, and the fourth coefficient d3 of the third mapping curve; generate the second mapping curve based on the first coefficient a2, the second coefficient b2, the third coefficient c2, and the fourth coefficient d2 of the second mapping curve; and generate the third mapping curve based on the first coefficient a3, the second coefficient b3, the third coefficient c3, and the fourth coefficient d3 of the third mapping curve.

[0241] S703 : Perform tone mapping on at least one frame of image based on M mapping curves.

[0242] In a possible embodiment, tone mapping is performed on at least one frame of image based on M mapping curves, including: for each pixel in the at least one frame of image, tone mapping is performed on the pixel using a mapping curve corresponding to a brightness range of the pixel.

[0243] Exemplarily, the brightness of the first image is divided into a low brightness range, a medium brightness range, and a high brightness range, wherein the low brightness range is further divided into two sub-brightness ranges, each sub-brightness range corresponds to a mapping curve, and the brightness of pixel A in the first image belongs to the first sub-brightness range, then the pixel is mapped according to the mapping curve corresponding to the first sub-brightness range.

[0244] From the above description, it can be seen that the back-end device in the embodiment of the present application generates a mapping curve based on the characteristic parameters of the mapping curve contained in the metadata, and uses different mapping curves to perform tone mapping on pixels in different brightness ranges of the HDR image, which can make the HDR image display adaptive and effectively improve the display effect of the HDR image.

[0245] In a possible embodiment, the metadata may also include color information of at least one frame of image. After the back-end device performs tone mapping on the at least one frame of image based on M mapping curves, it further includes: correcting the color information of the at least one frame of image after mapping based on the brightness information of each pixel in the at least one frame of image before and after the tone mapping.

[0246] It should be understood that the metadata may further include attribute information of the color of at least one frame of image (for example, the intensity of color saturation adjustment). Correspondingly, the backend device may also correct the color information of the at least one frame of image after mapping according to the attribute information of the color and the brightness information of each pixel in the at least one frame of image before and after tone mapping.

[0247] Exemplarily, taking pixel A in at least one frame of image as an example, the backend device may determine the ratio between the first brightness value of pixel A after tone mapping and the second brightness value of pixel A before tone mapping, and combine this ratio with the maximum display capacity of the target display device in the backend device, the maximum display capacity of the reference display device (such as a video production device) in the frontend device, the attribute information of the color of the at least one frame of image, and the second brightness value of pixel A before tone mapping to determine a saturation adjustment factor; based on this saturation adjustment factor, adjust the color information of pixel A. Here, the maximum display capacity can be understood as the maximum display brightness value.

[0248] It should be understood that the color information may be a YUV signal or other forms of color signals, and the embodiments of the present application do not make specific limitations. Among them, Y in "YUV" is used to represent the brightness information of the image, and UV is used to represent the color information of the image.

[0249] In a possible implementation manner, the color information is a YUV signal, and the backend device multiplies the saturation adjustment factor by the UV components of each pixel in the at least one frame of image after tone mapping to achieve color information correction.

[0250] In the embodiments of the present application, the backend device adjusts the color information of the at least one frame of image according to the brightness information of each pixel in the at least one frame of image before and after tone mapping, the display capacity of the target display device in the backend device, the maximum display capacity of the reference display device in the frontend device, and the attribute information of the color of the at least one frame of image, which can achieve a good correction effect and thus obtain a good display effect.

[0251] Based on the same technical concept, the embodiments of the present application further provide a tone mapping device 800. The device 800 has the functions of the above-mentioned Figure 2A backend device. The functions or units or means can be implemented by software, or by hardware, or by hardware executing corresponding software.

[0252] Exemplarily, referring to Figure 8 , the device 800 may include:

[0253] A receiving module 801 for receiving at least one frame of image and metadata; the metadata includes the number M of mapping curves, and the coordinate values of the starting node and the ending node of each mapping curve in the first coordinate system; wherein, the mapping curve is an N-degree spline curve, the horizontal coordinate of the first coordinate system represents the image brightness before tone mapping, and the vertical coordinate of the first coordinate system represents the image brightness after tone mapping; M is an integer greater than or equal to 1, and N is an integer greater than or equal to 3;

[0254] A processing module 802 for generating M mapping curves according to the metadata;

[0255] The processing module 802 is further configured to perform tone mapping on at least one frame of image based on the M mapping curves.

[0256] It should be understood that all relevant contents of each step involved in the above embodiments of the tone mapping method can be cited in the function descriptions of the corresponding functional modules, and will not be elaborated here.

[0257] Based on the same inventive concept, an embodiment of the present application further provides a device 900 for generating metadata. The device 900 has the functions of implementing the above Figure 2A front-end device. For example, the function or unit or means can be implemented by software, or by hardware, or by hardware executing corresponding software.

[0258] Exemplarily, referring to Figure 9 , the device 900 may include:

[0259] An obtaining module 901 for obtaining at least one frame of image;

[0260] A processing module 902 for processing at least one frame of image to generate metadata; wherein, the metadata includes the number M of mapping curves, and the coordinate values of the starting node and the ending node of each mapping curve in the first coordinate system; wherein, the mapping curve is an N-degree spline curve, the horizontal coordinate of the first coordinate system represents the image brightness before tone mapping, and the vertical coordinate of the first coordinate system represents the image brightness after tone mapping; M is an integer greater than or equal to 1, and N is an integer greater than or equal to 3.

[0261] It should be understood that all relevant contents of each step involved in the above embodiments of the method for generating metadata can be cited in the function descriptions of the corresponding functional modules, and will not be elaborated here.

[0262] Based on the same inventive concept, an embodiment of the present application further provides a computer-readable storage medium, including a program or instruction, when the program or instruction runs on a computer, causing Figure 2A the methods executed by the front-end device or the back-end device in the illustrated embodiments to be executed.

[0263] Based on the same inventive concept, an embodiment of the present application further provides an electronic device 1000 for implementing Figure 3A the method for generating metadata in the embodiment shown.

[0264] As Figure 10 shown, the electronic device 1000 may include a processor 1001 for executing a program or instruction stored in a memory 1002. When the program or instruction stored in the memory 1002 is executed, the processor 1001 is used to execute Figure 3A the tone mapping method in the embodiment shown.

[0265] Optionally, the electronic device 1000 may further include a communication interface 1003. Attached Figure 10 The communication interface 1003 is shown by a dashed line and is optional for the electronic device 1000.

[0266] Wherein, the numbers of the processor 1001, the memory 1002, and the communication interface 1003 do not constitute a limitation on the embodiments of the present application. In specific implementation, they can be arbitrarily configured according to service requirements.

[0267] Optionally, the memory 1002 is located outside the electronic device 1000.

[0268] Optionally, the electronic device 1000 includes the memory 1002, the memory 1002 is connected to the at least one processor 1001, and the memory 1002 stores instructions executable by the at least one processor 1001. Attached Figure 10 The memory 1002 is shown by a dashed line and is optional for the electronic device 1000.

[0269] Wherein, the processor 1001 and the memory 1002 may be coupled through an interface circuit or integrated together, which is not limited herein.

[0270] In the embodiments of the present application, the specific connection medium between the above-mentioned processor 1001, memory 1002, and communication interface 1003 is not limited. In the embodiments of the present application Figure 10 it is shown that the processor 1001, memory 1002, and communication interface 1003 are connected through a bus 1004. The bus is shown by a thick line in Figure 10 The connection manners between other components are only for illustrative purposes and are not to be construed as limitations. The bus may be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 10 it is only shown by a thick line in

[0271] Based on the same inventive concept, an embodiment of the present application further provides an electronic device 1100 for implementing Figure 7The tone mapping method in the illustrated embodiment.

[0272] As Figure 11 shown, the electronic device 1100 may include a processor 1101 for executing programs or instructions stored in a memory 1102. When the programs or instructions stored in the memory 1102 are executed, the processor is used to execute Figure 7 the tone mapping method in the illustrated embodiment.

[0273] Optionally, the electronic device 1100 may further include a communication interface 1103. Attached Figure 11 The communication interface 1103 is shown as optional for the electronic device 1100 by a dashed line.

[0274] Wherein, the numbers of the processor 1101, the memory 1102, and the communication interface 1103 do not constitute a limitation on the embodiments of the present application. In specific implementation, they can be arbitrarily configured according to service requirements.

[0275] Optionally, the memory 1102 is located outside the electronic device 1100.

[0276] Optionally, the electronic device 1100 includes the memory 1102, the memory 1102 is connected to the at least one processor 1101, and the memory 1102 stores instructions executable by the at least one processor 1101. Attached Figure 11 The memory 1102 is shown as optional for the electronic device 1100 by a dashed line.

[0277] Wherein, the processor 1101 and the memory 1102 may be coupled through an interface circuit or integrated together, which is not limited here.

[0278] In the embodiments of the present application, the specific connection medium between the above-mentioned processor 1101, memory 1102, and communication interface 1103 is not limited. In the embodiments of the present application Figure 11 it is shown that the processor 1101, memory 1102, and communication interface 1103 are connected through a bus 1104. The bus is shown as a thick line in Figure 11 The connection manners between other components are only for illustrative purposes and are not to be construed as limitations. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 11 only one thick line is shown in

[0279] It should be understood that the processor mentioned in the embodiments of the present application can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor that realizes its functions by reading software codes stored in a memory.

[0280] Exemplarily, the processor can be a central processing unit (CPU), or can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0281] It should be understood that the memory mentioned in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced SDRAM (ESDRAM), synch link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0282] Note that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, the memory (storage module) can be integrated in the processor.

[0283] It should be noted that the memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.

[0284] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, system, or computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program code.

[0285] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0286] These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured product including instruction means, and the instruction means implement the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0287] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0288] Obviously, those skilled in the art can make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application also intends to include these changes and modifications.

Claims

1. A tone mapping method, characterized in that: The method comprises: Receive at least one frame of image and metadata, The metadata includes the number M of mapping curves, and the coordinate values of the starting node and the ending node of each mapping curve in a first coordinate system; wherein the mapping curve is an N-order spline curve, the horizontal coordinate of the first coordinate system represents the image brightness before tone mapping, and the vertical coordinate of the first coordinate system represents the image brightness after tone mapping; M is an integer greater than or equal to 1, and N is an integer greater than or equal to 3; Generate M mapping curves according to the metadata; Performing tone mapping on the at least one frame of image based on the M mapping curves; Each of the at least one frame of images corresponds to at least one brightness range, each brightness range corresponds to at least one mapping curve, and the function expression corresponding to each mapping curve is an N-order polynomial, the N-order polynomial includes N+1 monomials, and each monomial corresponds to a coefficient; The first function expression corresponding to the i-th mapping curve in the at least one mapping curve includes: y=di*(x-x²) 3 +ci*(x-x2) 2 +bi*(x-x²)+ai; or, y=ei*(x-x²) 4 +di*(x-x2) 3 +ci*(x-x2) 2 + bi*(x-x2)+ai; Among them, the x is the independent variable used to characterize the image brightness before tone mapping, and the y is the dependent variable used to characterize the image brightness after tone mapping; the ai is the first coefficient of the constant term in the i-th mapping curve, bi is the second coefficient of the linear monomial in the i-th mapping curve, ci is the third coefficient of the quadratic monomial in the i-th mapping curve, di is the fourth coefficient of the cubic monomial in the i-th mapping curve, and ei is the coefficient of the quartic monomial in the i-th mapping curve; the x2 is the horizontal coordinate of the starting node of the i-th mapping curve; the ai, bi, ci, and di are all not 0.

2. The method according to claim 1, wherein The metadata further includes a preset first-order derivative of a starting node and a preset first-order derivative of an ending node of each mapping curve.

3. The method according to claim 1, wherein A first brightness range in the at least one brightness range corresponds to a first mapping curve, and the first mapping curve includes a first starting node and a first ending node; Generating M mapping curves according to the metadata includes: Substituting the coordinate value of the first starting node into the first function expression to obtain a first coefficient a1 of the first mapping curve; Substituting the coordinate value of the first cutoff node into the first function expression to obtain a first equation; making a first-order derivative of the first mapping curve at the first starting node equal to a preset first-order derivative of the first starting node to obtain a second equation; making a first-order derivative of the first mapping curve at the first cutoff node equal to a preset first-order derivative of the first cutoff node to obtain a third-party formula; Combining the first equation, the second equation, and the third equation to obtain a first system of equations; Analyzing the first set of equations to obtain a second coefficient b1, a third coefficient c1, and a fourth coefficient d1 of the first mapping curve; The first mapping curve is generated based on the first coefficient a1, the second coefficient b1, the third coefficient c1, and the fourth coefficient d1 of the first mapping curve.

4. The method according to claim 1, wherein The second brightness range in the at least one brightness range corresponds to a second mapping curve and a third mapping curve, the second mapping curve corresponds to a first sub-brightness range in the second brightness range, the third mapping curve corresponds to a second sub-brightness range in the second brightness range, the maximum value in the first sub-brightness range is equal to the minimum value in the second sub-brightness range; the second mapping curve includes a second starting node and a second ending node; the third mapping curve includes a third starting node and a third ending node; The coordinate value of the second end node is the same as that of the third start node; Generating M mapping curves according to the metadata includes: Substituting the coordinate value of the second starting node into the first function expression to obtain a first coefficient a2 of the second mapping curve; Substituting the coordinate value of the third starting node into the first function expression to obtain a first coefficient a3 of the third mapping curve; Making the first-order derivative of the second mapping curve at the second starting node equal to the preset first-order derivative of the second starting node, establishing a fourth equation; and solving the fourth equation to obtain a second coefficient b2 of the second mapping curve; Constructing a second set of equations according to the coordinate value of the second cutoff node, the coordinate value of the third cutoff node, and a preset first-order derivative of the third cutoff node; Analyzing the second set of equations to obtain a third coefficient c2 and a fourth coefficient d2 of the second mapping curve and a second coefficient b3, a third coefficient c3, and a fourth coefficient d3 of the third mapping curve; The second mapping curve is generated based on the first coefficient a2, second coefficient b2, third coefficient c2, and fourth coefficient d2 of the second mapping curve; and the third mapping curve is generated based on the first coefficient a3, second coefficient b3, third coefficient c3, and fourth coefficient d3 of the third mapping curve.

5. The method according to claim 4, wherein The constructing of the second set of equations according to the coordinate value of the second cutoff node, the coordinate value of the third cutoff node, and the preset first-order derivative of the third cutoff node includes: Substituting the coordinate values of the second cutoff node and the coordinate values of the third cutoff node into the first function expression to obtain a fifth equation and a sixth equation, respectively, so that the first-order derivative of the third cutoff node on the third mapping curve is equal to the preset first-order derivative of the third cutoff node, to obtain a seventh equation; The eighth equation is obtained by making the first-order derivative of the second mapping curve at the second cut-off node equal to the first-order derivative of the third mapping curve at the third starting node, and the ninth equation is obtained by making the second-order derivative of the second mapping curve at the second cut-off node equal to the second-order derivative of the third mapping curve at the third starting node; or the eighth equation is obtained by making the first-order derivative of the second mapping curve at the second cut-off node equal to a preset first-order derivative at the second cut-off node, and the ninth equation is obtained by making the first-order derivative of the third mapping curve at the third starting node equal to a preset first-order derivative at the third starting node; The fifth equation, the sixth equation, the seventh equation, the eighth equation, and the ninth equation are combined to obtain the second set of equations.

6. The method according to any one of claims 2 to 5, characterized in that The performing tone mapping on the at least one frame of image based on the M mapping curves includes: For each pixel in the at least one frame of image, tone mapping is performed on the pixel using a mapping curve corresponding to the brightness range of the pixel.

7. The method according to any one of claims 2 to 5, characterized in that After tone mapping is performed on the at least one frame of image based on the M mapping curves, the method further includes: Based on the brightness information of each pixel in the at least one frame of image before and after tone mapping, the color information of the at least one frame of image after mapping is corrected.

8. The method according to claim 6, wherein After tone mapping is performed on the at least one frame of image based on the M mapping curves, the method further includes: Based on the brightness information of each pixel in the at least one frame of image before and after tone mapping, the color information of the at least one frame of image after mapping is corrected.

9. The method according to claim 7, wherein After tone mapping is performed on the at least one frame of image based on the M mapping curves, the method further includes: Based on the brightness information of each pixel in the at least one frame of image before and after tone mapping, the color information of the at least one frame of image after mapping is corrected.

10. The method according to claim 8, wherein After tone mapping is performed on the at least one frame of image based on the M mapping curves, the method further includes: Based on the brightness information of each pixel in the at least one frame of image before and after tone mapping, the color information of the at least one frame of image after mapping is corrected.

11. A method for generating metadata, characterized in that: The method comprises: Acquire at least one frame of image; Processing the at least one frame of image to generate metadata; wherein the metadata includes the number M of mapping curves, and the coordinate values of the starting node and the ending node of each mapping curve in a first coordinate system; wherein the mapping curve is an N-order spline curve, the horizontal coordinate of the first coordinate system represents the image brightness before tone mapping, and the vertical coordinate of the first coordinate system represents the image brightness after tone mapping; wherein M is an integer greater than or equal to 1, and N is an integer greater than or equal to 3; The step of processing the at least one frame of image to generate metadata includes: extracting attribute information of the at least one frame of image; determining metadata based on the attribute information; The step of determining metadata based on the attribute information includes: Determining brightness distribution of the at least one frame of image according to attribute information of the at least one frame of image; Based on the brightness distribution, dividing the brightness of the at least one frame of image into at least one brightness range; The number M is determined according to the number F of the brightness ranges, wherein M is greater than or equal to F, and each brightness range corresponds to at least one mapping curve.

12. The method according to claim 11, wherein The attribute information includes the maximum value, minimum value, average value, and variation range of brightness.

13. The method according to claim 11, wherein Determining metadata based on the attribute information includes: The coordinate values of the starting node and the ending node of each mapping curve in the first coordinate system are determined according to the brightness distribution and the maximum brightness value that can be displayed by the back-end device.

14. The method according to claim 12 or 13, wherein: The metadata further includes a preset first-order derivative of the start node and a preset first-order derivative of the end node.

15. The method according to claim 14, wherein Determining metadata based on the attribute information includes: The preset first-order derivative of the starting node and the preset first-order derivative of the ending node are determined according to a balance requirement between brightness and contrast of the at least one frame of image.

16. The method according to claim 15, wherein After processing the at least one frame of image to generate metadata, the method further includes: receiving a first operation, wherein the first operation is used to change at least one of the number M, the coordinate value of the starting node in the first coordinate system, the coordinate value of the ending node in the first coordinate system, a preset first-order derivative of the starting node, and a preset first-order derivative of the ending node; Based on the first operation, the metadata is modified to obtain updated metadata.

17. A tone mapping device, characterized in that: include: A receiving module, configured to receive at least one image frame and metadata; the metadata including the number M of mapping curves and the coordinate values of the starting and ending nodes of each mapping curve in a first coordinate system; wherein the mapping curve is an N-order spline curve, the horizontal coordinate of the first coordinate system represents the image brightness before tone mapping, and the vertical coordinate of the first coordinate system represents the image brightness after tone mapping; M is an integer greater than or equal to 1, and N is an integer greater than or equal to 3; a processing module, configured to generate M mapping curves according to the metadata; and perform tone mapping on the at least one frame of image based on the M mapping curves; Each of the at least one frame of images corresponds to at least one brightness range, each brightness range corresponds to at least one mapping curve, and the function expression corresponding to each mapping curve is an N-order polynomial, the N-order polynomial includes N+1 monomials, and each monomial corresponds to a coefficient; The first function expression corresponding to the i-th mapping curve in the at least one mapping curve includes: y=di*(x-x²) 3 +ci*(x-x2) 2 +bi*(x-x²)+ai; or, y=ei*(x-x²) 4 +di*(x-x2) 3 +ci*(x-x2) 2 + bi*(x-x2)+ai; Among them, the x is the independent variable used to characterize the image brightness before tone mapping, and the y is the dependent variable used to characterize the image brightness after tone mapping; the ai is the first coefficient of the constant term in the i-th mapping curve, bi is the second coefficient of the linear monomial in the i-th mapping curve, ci is the third coefficient of the quadratic monomial in the i-th mapping curve, di is the fourth coefficient of the cubic monomial in the i-th mapping curve, and ei is the coefficient of the quartic monomial in the i-th mapping curve; the x2 is the horizontal coordinate of the starting node of the i-th mapping curve; the ai, bi, ci, and di are all not 0.

18. A device for generating metadata, characterized in that: The device comprises: An acquisition module, configured to acquire at least one frame of image; a processing module, configured to process the at least one frame of image and generate metadata; wherein the metadata includes the number M of mapping curves and the coordinate values of the starting node and the ending node of each mapping curve in a first coordinate system; wherein the mapping curve is an N-order spline curve, the horizontal coordinate of the first coordinate system represents the image brightness before tone mapping, and the vertical coordinate of the first coordinate system represents the image brightness after tone mapping; wherein M is an integer greater than or equal to 1, and N is an integer greater than or equal to 3; The processing module processes the at least one frame of image to generate metadata, specifically for: extracting attribute information of the at least one frame of image; determining metadata based on the attribute information; The processing module, when determining metadata based on the attribute information, is specifically configured to: Determining brightness distribution of the at least one frame of image according to attribute information of the at least one frame of image; Based on the brightness distribution, dividing the brightness of the at least one frame of image into at least one brightness range; The number M is determined according to the number F of the brightness ranges, wherein M is greater than or equal to F, and each brightness range corresponds to at least one mapping curve.

19. An electronic device, characterized in that: include: A non-volatile memory and a processor coupled to each other, wherein the processor calls a program code stored in the memory to execute the method according to any one of claims 1 to 10 or claims 11 to 16.

20. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed on a processor, implements the method according to any one of claims 1 to 10 or claims 11 to 16.

Images