Image processing method and device

Abstract

The present application provides an image processing method, comprising: adaptively determining at least one interpolation point according to feature information or dynamic metadata of an image to be processed, and a modified spline curve through the at least one interpolation point, wherein the modified spline curve includes at least one of a linear spline curve or a cubic spline curve; tone mapping the image to be processed according to the modified tone mapping curve, wherein the modified tone mapping curve includes at least a portion of an initial tone mapping curve and the modified spline curve. The method provided by the present application can adaptively modify the tone mapping curve according to the needs of the scene, so that images in various scenes can obtain better display effects.

Description

Technical Field

[0001] The present application relates to the field of image processing, and more specifically, to an image processing method and device. Background Art

[0002] Dynamic range (DR) can be used to represent the ratio of the maximum grayscale value to the minimum grayscale value of an image. Dynamic range can also be used to represent the ratio of the maximum brightness value to the minimum brightness value of an image. The real-world dynamic range is between 10 -3 -10 6 The range between the two is called high dynamic range (HDR), and the display capabilities of display devices are usually limited, far lower than the brightness range of the real world. For example, the brightness of things in the real world is brighter at 2000nit, while the maximum brightness that the display device can display is only 500nit. If no processing is performed, the display device will not be able to reproduce the thing in the real world. In another case, when the brightness of things in the real world is darker, for example, the brightness value is 10nit, and the minimum brightness that the display device can display is 100nit, if no processing is performed, the display device will not be able to reproduce the things in the real world. The present application provides a method for enabling a display device to vividly and realistically reproduce scenes in the real world. Summary of the invention

[0003] The present application provides an image processing method and apparatus, the purpose of which is to improve the tone mapping effect of a display device.

[0004] In a first aspect, a method for image processing is provided, comprising: determining a first coordinate of at least one interpolation point based on feature information or dynamic metadata of an image to be processed; determining at least one second coordinate and at least one first-order derivative based on the first coordinate of the at least one interpolation point; determining coefficients of a modified spline curve based on the first coordinate, at least one second coordinate and at least one first-order derivative of the at least one interpolation point, the modified spline curve comprising at least one of a linear spline curve or a cubic spline curve; tone-mapping the image to be processed based on a modified tone mapping curve, the modified tone mapping curve comprising at least a portion of an initial tone mapping curve and the modified spline curve.

[0005] The positions of the interpolation points are adaptively determined according to the feature information of the image to be processed or the dynamic metadata of the image to be processed, such as brightness information, pixel distribution information, etc., and the modified spline curve most suitable for the image to be processed is flexibly determined according to the interpolation points. The image to be processed is tone mapped based on the modified tone mapping curve. Since the tone mapping curve is not fixed but adaptively adjusted according to the image to be processed or the dynamic metadata, images in various scenes can obtain better display effects.

[0006] The image processing method provided in the embodiment of the present application can adaptively modify the curve according to the needs of the scene to achieve a specific effect, such as increasing the brightness, increasing the contrast, blurring some information, etc. For example, the display effect of the details in the dark area can be improved by increasing the brightness of the dark area, or the contrast can be improved by reducing the brightness of the dark area; or more bits can be allocated to the highlight area to display the details of the bright area.

[0007] In a possible implementation, the at least one interpolation point includes: a first interpolation point, the modified spline curve includes: a first spline curve, the first spline curve is a linear spline curve passing through the first interpolation point, and the coefficients of the modified spline curve include a first coefficient MA0 and a second coefficient MB0; the determining of at least one second coordinate and at least one first-order derivative based on the first coordinate of the at least one interpolation point includes: determining the second coordinate VA1 of the first interpolation point on the initial tone mapping curve and the first-order derivative GD1 of the initial tone mapping curve at the first interpolation point based on the first coordinate TH1 of the first interpolation point; the determining of the coefficients of the modified spline curve based on the first coordinate, at least one second coordinate and at least one first-order derivative of the at least one interpolation point includes: determining GD1 as the second coefficient MB0 of the first spline curve; and determining VA1 as the first coefficient MA0 of the first spline curve.

[0008] The embodiment of the present application only needs to determine one interpolation point, and optionally, the first interpolation point is close to the endpoint of the bright area, corresponding to the situation where the overall scene of the image is relatively bright. At this time, the bright area is corrected to better show the bright details of the image. Specifically, tone mapping is performed on the pixel area greater than a preset threshold (TH1) based on a linear function, and the first spline curve is smoothly connected to the initial tone mapping curve at the first interpolation point to avoid the influence of the non-smooth tone mapping curve on the tone mapping effect.

[0009] In a possible implementation, the at least one interpolation point includes: a first interpolation point and a second interpolation point, the modified spline curve includes: a first spline curve, the first spline curve is a cubic spline curve passing through the first interpolation point and the second interpolation point, and the coefficients of the modified spline curve include a first coefficient MA1, a second coefficient MB1, a third coefficient MC1, and a fourth coefficient MD1; the determining of at least one second coordinate and at least one first-order derivative according to the first coordinate of the at least one interpolation point includes: respectively calculating the first interpolation point and the second interpolation point on the initial tone mapping curve according to the first coordinate TH1 of the first interpolation point and the first coordinate TH2 of the second interpolation point The second coordinates VA1 and VA2 of the initial tone mapping curve are calculated according to TH1 and TH2; the first-order derivative GD1 of the initial tone mapping curve at the first interpolation point and the first-order derivative GD2 at the second interpolation point are calculated; the coefficients of the modified spline curve are determined according to the first coordinate of the at least one interpolation point, the at least one second coordinate and the at least one first-order derivative, including: determining VA1 as the first coefficient MA1 of the first spline curve; determining GD1 as the second coefficient MB1 of the first spline curve; constructing a group of equations including two equations according to TH1, TH2, VA2 and GD2; and analyzing the group of equations to obtain the third coefficient MC1 and the fourth coefficient MD1 of the first spline curve.

[0010] The embodiment of the present application needs to determine two interpolation points. In an optional case, the two interpolation points are close to the endpoints of the bright area, corresponding to the situation that the overall scene of the image is relatively bright. At this time, the bright area is corrected to better show the bright details of the image. Specifically, tone mapping is performed on the pixel area between the first interpolation point and the second interpolation point based on the cubic spline function, and the cubic spline function is smoothly connected to the initial tone mapping curve at the first interpolation point to avoid the influence of the non-smooth tone mapping curve on the tone mapping effect.

[0011] In a possible implementation, the system of equations including two equations is constructed based on TH1, TH2, VA2 and GD2, including: making the second coordinate of the second interpolation point on the first spline curve the sum of VA2 and S1, obtaining the first equation, S1 is the coordinate offset value included in the dynamic metadata; making the first-order derivative of the first spline curve at the second interpolation point the sum of GD2 and S2, obtaining the second equation, S2 is the slope offset value included in the dynamic metadata.

[0012] In a possible implementation, the at least one interpolation point includes: a first interpolation point, the modified spline curve includes: a first spline curve, the first spline curve is a linear spline function passing through the origin and the first interpolation point, and the coefficients of the modified spline curve include: a first-order derivative MB0 of the first spline curve; before determining at least one second coordinate and at least one first-order derivative based on the first coordinate of the at least one interpolation point, the method also includes: obtaining the first-order derivative MB0 of the first spline curve based on at least one of the maximum value maxL of the brightness of the image to be processed included in the dynamic metadata and the average value avgL of the brightness of the image to be processed.

[0013] In a possible implementation, the at least one interpolation point further includes: a second interpolation point and a third interpolation point, the modified spline curve further includes: a second spline curve and a third spline curve, the second spline curve is a cubic spline curve passing through the first interpolation point and the second interpolation point, and the third spline curve is a cubic spline curve passing through the second interpolation point and the third interpolation point; the coefficients of the modified spline curve further include: a first coefficient MA1, a second coefficient MB1, a third coefficient MC1, and a fourth coefficient MD1 of the second spline curve, and a first coefficient MA2, a second coefficient MB2, a third coefficient MC2, and a fourth coefficient MD2 of the third spline curve; the coefficients of the modified spline curve further include: a first coefficient MA1, a second coefficient MB1, a third coefficient MC1, and a fourth coefficient MD1 of the second spline curve, and a first coefficient MA2, a second coefficient MB2, a third coefficient MC2, and a fourth coefficient MD2 of the third spline curve; The first coordinate of the at least one interpolation point determines at least one second coordinate and at least one first-order derivative, including: calculating the second coordinate VA1 of the first interpolation point on the first spline curve according to the first coordinate TH1 and MB0 of the first interpolation point; calculating the second coordinate VA3 of the third interpolation point on the initial tone mapping curve according to the first coordinate TH3 of the third interpolation point; calculating the second coordinate VA2 of the second interpolation point on the first straight line according to the first coordinate TH2 of the second interpolation point, the first straight line being the straight line where the first interpolation point and the third interpolation point are located; and calculating the first-order derivative GD3 of the initial tone mapping curve at the third interpolation point according to TH3.

[0014] In a possible implementation, the coefficients of the modified spline curve are determined according to the first coordinate, at least one second coordinate and at least one first-order derivative of the at least one interpolation point, including: determining VA1 as the first coefficient MA1 of the second spline curve; determining MB0 as the second coefficient MB1 of the second spline curve; determining VA2 as the first coefficient MA2 of the third spline curve; constructing a group of equations including five equations according to TH1, TH2, TH3, VA2, VA3, GD3 and the derivative relationship between the second spline curve and the third spline curve at the second interpolation point; and analyzing the group of equations including five equations to obtain MC1, MD1, MB2, MC2 and MD2.

[0015] In the embodiment of the present application, when the overall scene of the image is relatively dark, the dark area is corrected to better display the dark details of the image, so that the pixels whose brightness is lower than the display capability of the display device can have a better display effect on the display device. Specifically, the pixels below the first threshold (TH1) are tone mapped based on the linear spline function formula, which better retains the dark detail information of this part, and then the smooth transition between the linear spline function and the initial tone mapping curve is achieved based on the cubic spline function, so as to avoid the non-smooth tone mapping curve affecting the tone mapping effect. It should be understood that there are two cubic spline curves in the embodiment of the present application. In an optional case, there may be only one cubic spline curve, and the first spline curve is smoothly connected to the initial tone mapping curve through a cubic spline function. In this case, only two interpolation points need to be determined.

[0016] Specifically, the first spline curve increases the brightness of the dark area compared to the base curve, which is equivalent to allocating more bits to the dark area. The two cubic spline functions achieve a smooth transition between the first spline curve and the initial tone mapping curve, and can enhance or weaken the correction area by controlling the points and slopes of the curves.

[0017] In a possible implementation, the system of equations including five equations is constructed according to TH1, TH2, TH3, VA2, VA3, GD3 and the derivative relationship between the second spline curve and the third spline curve at the second interpolation point, including: making the second coordinate of the second interpolation point on the second spline curve be VA2, to obtain the first equation; making the second coordinate of the third interpolation point on the third spline curve be VA3, to obtain the second equation; making the first-order derivative of the third spline curve at the third interpolation point be GD3, to obtain the third equation; making the first-order derivatives of the second spline curve and the third spline curve at the second interpolation point equal, to obtain the fourth equation; making the second-order derivatives of the second spline curve and the third spline curve at the second interpolation point equal, to obtain the fifth equation.

[0018] In a possible implementation, the first coordinate of at least one interpolation point is determined according to the feature information or dynamic metadata of the image to be processed, including: determining the first coordinate TH1 of the first interpolation point according to at least one of the maximum value maxL of the brightness of the image to be processed and the average value avgL of the brightness of the image to be processed included in the dynamic metadata; TH2=TH1+B; TH3=TH2+C*TH2-D*CH2; wherein B, C and D are constants. Optionally, B, C and D are constants, and in an optional case, the constants may be included in the dynamic metadata. In an optional case, B, C and D may also be directly defined in the relevant program code without being obtained from the dynamic metadata.

[0019] In a possible implementation, the at least one interpolation point includes: a first interpolation point, a second interpolation point and a third interpolation point, and the modified spline curve includes: a first spline curve and a second spline curve, the first spline curve is a cubic spline curve passing through the first interpolation point and the second interpolation point, and the second spline curve is a cubic spline curve passing through the second interpolation point and the third interpolation point; the coefficients of the modified spline curve include: a first coefficient MA1, a second coefficient MB1, a third coefficient MC1, and a fourth coefficient MD1 of the first spline curve, and a first coefficient MA2, a second coefficient MB2, a third coefficient MC2, and a fourth coefficient MD2 of the second spline curve ; The determining of at least one second coordinate and at least one first-order derivative according to the first coordinate of the at least one interpolation point includes: respectively calculating the second coordinates VA1 and VA3 of the first interpolation point and the third interpolation point on the initial tone mapping curve according to the first coordinate TH1 of the first interpolation point and the first coordinate TH3 of the third interpolation point; calculating the second coordinate VA2 of the second interpolation point on the first straight line according to the first coordinate TH2 of the second interpolation point, the first straight line being the straight line where the first interpolation point and the third interpolation point are located; calculating the first-order derivative GD1 of the initial tone mapping curve at the first interpolation point and the first-order derivative GD3 at the third interpolation point.

[0020] The corresponding image of the embodiment of the present application contains pixels located in a specific brightness range. At this time, the specific brightness range is focused on correction to better display the brightness details of the image in this range; specifically, based on the cubic spline function, the pixels between the first interpolation point and the second interpolation point and the second interpolation point and the third interpolation point are tone mapped to improve the display effect of these pixels. Furthermore, there is a smooth transition between the first spline curve and the second spline curve, and between the first spline curve and the second spline curve and the initial tone mapping curve, so as to avoid the non-smooth tone mapping curve affecting the tone mapping effect.

[0021] In a possible implementation, the coefficients of the modified spline curve are determined according to the first coordinates, at least one second coordinate and at least one first-order derivative of the at least one interpolation point, including: determining VA1 as the first coefficient MA1 of the first spline curve and determining GD1 as the second coefficient MB1 of the first spline curve; determining the sum of VA2 and an offset value as the first coefficient MA2 of the second spline curve, and the offset value is 0, a positive number or a negative number; constructing a group of equations including five equations according to TH1, TH2, TH3, VA2, VA3, GD3 and the derivative relationship between the first spline curve and the second spline curve at the second interpolation point; and parsing the group of equations including five equations to obtain MC1, MD1, MB2, MC2 and MD2.

[0022] Optionally, the offset value is included in the dynamic metadata.

[0023] In a possible implementation, the system of equations including five equations is constructed according to TH1, TH2, TH3, VA2, VA3, GD3 and the derivative relationship between the first spline curve and the second spline curve at the second interpolation point, including: making the second coordinate of the second interpolation point on the first spline curve be VA2, to obtain the first equation; making the second coordinate of the third interpolation point on the second spline curve be VA3, to obtain the second equation; making the first-order derivative of the second spline curve at the third interpolation point be GD3, to obtain the third equation; making the first-order derivatives of the first spline curve and the second spline curve at the second interpolation point equal, to obtain the fourth equation; making the second-order derivatives of the first spline curve and the second spline curve at the second interpolation point equal, to obtain the fifth equation.

[0024] In a possible implementation, the at least one interpolation point includes: a first interpolation point, a second interpolation point, and a third interpolation point, and the modified spline curve includes: a first spline curve and a second spline curve, the first spline curve is a cubic spline curve passing through the first interpolation point and the second interpolation point, and the second spline curve is a linear spline curve passing through the second interpolation point and the third interpolation point; the coefficients of the modified spline curve include: a first coefficient MA1, a second coefficient MB1, a third coefficient MC1, and a fourth coefficient MD1 of the first spline curve, and a first coefficient MA2 and a second coefficient MB1 of the second spline curve. 2; determining at least one second coordinate and at least one first-order derivative according to the first coordinate of the at least one interpolation point, including: calculating the second coordinates VA1 and VA3 of the first interpolation point and the third interpolation point on the initial tone mapping curve according to the first coordinate TH1 of the first interpolation point and the first coordinate TH3 of the third interpolation point respectively; calculating the second coordinate VA2 of the second interpolation point on the first straight line according to the first coordinate TH2 of the second interpolation point, the first straight line being the straight line where the first interpolation point and the third interpolation point are located; calculating the first-order derivative GD1 of the initial tone mapping curve at the first interpolation point.

[0025] In a possible implementation, the coefficients of the modified spline curve are determined according to the first coordinates, at least one second coordinate and at least one first-order derivative of the at least one interpolation point, including: determining VA1 as the first coefficient MA1 of the first spline curve and determining GD1 as the second coefficient MB1 of the first spline curve; determining the sum of VA2 and an offset value as the first coefficient MA2 of the second spline curve, and the offset value is 0, a positive number or a negative number; constructing a group of equations including three equations according to TH1, TH2, TH3, VA2, VA3 and the derivative relationship between the first spline curve and the second spline curve at the second interpolation point; and analyzing the group of equations including three equations to obtain MC1, MD1 and MB2.

[0026] It should be understood that the offset value may be included in the dynamic metadata.

[0027] The embodiment of the present application corresponds to the situation where the overall scene of the image is relatively bright. At this time, the bright area is corrected to better show the bright details of the image. Specifically, tone mapping is performed on the pixel area between the second interpolation point and the third interpolation point based on a linear spline curve, and a smooth connection is achieved between the linear spline function and the initial tone mapping curve based on a cubic spline function between the first interpolation point and the second interpolation point, so as to avoid the influence of the non-smooth tone mapping curve on the tone mapping effect.

[0028] In a possible implementation, the system of equations including three equations is constructed according to TH1, TH2, TH3, VA2, VA3 and the derivative relationship between the first spline curve and the second spline curve at the second interpolation point, including: making the second coordinate of the second interpolation point on the first spline curve be VA2 to obtain the first equation; making the second coordinate of the third interpolation point on the second spline curve be VA3 to obtain the second equation; making the first-order derivatives of the first spline curve and the second spline curve at the second interpolation point equal to each other to obtain the third equation.

[0029] In one possible implementation, the at least one interpolation point constitutes at least one correction interval, wherein one correction interval corresponds to a correction spline curve, and the image to be processed is tone mapped according to the corrected tone mapping curve, including: when the pixel to be processed is located in the at least one correction interval, tone mapping is performed on the pixel to be processed based on the correction spline curve; when the pixel to be processed is outside the at least one correction interval, tone mapping is performed on the pixel to be processed based on the initial tone mapping curve.

[0030] In a possible implementation, before determining the first coordinates of at least one interpolation point based on the feature information or dynamic metadata of the image to be processed, the method further includes: determining a target scene mode applicable to the image to be processed based on a mode indicator value in the dynamic metadata, and the at least one interpolation point and the modified spline curve are related to the target scene mode.

[0031] Exemplarily, the number of interpolation points determined and the number of linear spline curves and cubic spline curves included in the modified spline curve are related to the scene mode applicable to the image to be processed. When the image to be processed is dark as a whole, the first interpolation point is located in an area close to the endpoint of the dark area, and the pixels between the origin and the first interpolation point are mapped using a linear spline curve, which can better display the details of the dark area; when the image to be processed is bright as a whole, multiple interpolation points are located in an area close to the endpoint of the bright area.

[0032] In a possible implementation, the initial tone mapping curve includes one of the following formulas:

[0033] Among them, L represents the linear signal value, L' represents the nonlinear signal value, p, a, m, b, c1, c2, c3, n, N, ρ are all dynamic range mapping coefficients, 0≤L≤1.

[0034] In the embodiments of the present application, the method provided in the first aspect can be applied to a variety of initial tone mapping curves and has strong flexibility.

[0035] In a possible implementation, the multiple interpolation points include a first interpolation point, a second interpolation point, and a third interpolation point, the spline function includes a first spline function passing through the first interpolation point and the second interpolation point, and a second spline function passing through the second interpolation point and the third interpolation point, the first spline function includes: a first coefficient MA1, a second coefficient MB1, a third coefficient MC1, and a fourth coefficient MD1; the second spline function includes: a first coefficient MA2, a second coefficient MB2, a third coefficient MC2, and a fourth coefficient MD2.

[0036] In one possible implementation, the first interpolation point is not on the initial tone mapping curve, and the determination of the spline function passing through the multiple interpolation points includes: determining the second coordinate VA1 of the first interpolation point on the linear spline function, where the linear spline function is a linear spline function between the origin and the first interpolation point; and determining VA1 as the first coefficient MA1 of the first spline function.

[0037] In a possible implementation, the first interpolation point is located on the initial tone mapping curve, and determining the spline function passing through the multiple interpolation points includes: determining the second coordinate VA1 of the first interpolation point on the initial tone mapping curve; and determining VA1 as the first coefficient MA1 of the first spline function.

[0038] In a possible implementation, the determination of the spline function passing through the multiple interpolation points also includes: determining the second coordinate VA3 of the third interpolation point on the initial tone mapping curve; determining the second coordinate VA2 of the second interpolation point on the first straight line, the first straight line being the straight line where the first interpolation point and the third interpolation point are located; and determining VA2 as the first coefficient MA2 of the second spline function.

[0039] In a possible implementation, the dynamic metadata also includes a second coordinate offset of the second interpolation point, and the determination of the spline function passing through the multiple interpolation points also includes: determining the second coordinate VA3 of the third interpolation point on the initial tone mapping curve; determining the second coordinate VA2 of the second interpolation point on the first straight line, which is the straight line where the first interpolation point and the third interpolation point are located; and determining the sum of VA2 and the second coordinate offset as the first coefficient MA2 of the second spline function.

[0040] In a possible implementation, determining the spline function passing through the plurality of interpolation points further includes: determining a first-order derivative of the linear spline function as the MB1; and determining a first-order derivative of the initial tone mapping curve at the third interpolation point as the MB2.

[0041] In a possible implementation, determining the spline function passing through the multiple interpolation points further includes: determining the first-order derivative of the initial tone mapping curve at the first interpolation point as MB1; and determining the first-order derivative of the initial tone mapping curve at the third interpolation point as MB2.

[0042] In a possible implementation, the determination of the spline function passing through the multiple interpolation points also includes: making the second coordinate value of the second interpolation point on the first spline function the VA2 to obtain the first equation; making the second coordinate value of the third interpolation point on the second spline function the VA3 to obtain the second equation; making the first-order derivative of the second spline function at the third interpolation point the first-order derivative of the initial tone mapping curve at the third interpolation point to obtain a third equation; making the first-order derivatives of the first spline function and the second spline function at the second interpolation point equal to each other to obtain a fourth equation; making the second-order derivatives of the first spline function and the second spline function at the second interpolation point equal to each other to obtain a fifth equation; analyzing the first equation, the second equation, the third equation, the fourth equation and the fifth equation to obtain the third coefficient MC1 and the fourth coefficient MD1 of the first spline function, and the third coefficient MC2 and the fourth coefficient MD2 of the second spline function.

[0043] In a possible implementation, the first interpolation point and the third interpolation point are located on the initial tone mapping curve, and the determination of the spline function passing through the multiple interpolation points includes: respectively determining the second coordinate values ​​VA1 and VA3 of the first interpolation point and the third interpolation point on the initial tone mapping curve; determining the second coordinate value VA2 corresponding to the second interpolation point on the first straight line, the first straight line being the straight line where the first interpolation point and the third interpolation point are located; determining VA1 as the first coefficient MA1 of the first spline function, and VA2 as the first coefficient MA2 of the second spline function; respectively determining the slopes GD1 and GD3 of the initial tone mapping curve at the first interpolation point and the third interpolation point, setting GD1 as the second coefficient MB1 of the first spline function, and GD3 as the second coefficient MB2 of the second spline function; making the second coordinate value of the second interpolation point on the first spline function the value of the VA1. 2, obtain the first equation; make the second coordinate value of the third interpolation point on the second spline function be the VA3, obtain the second equation; make the first-order derivative of the second spline function at the third interpolation point be GD3, obtain the third equation; make the first-order derivatives of the first spline function and the second spline function at the second interpolation point equal, obtain the fourth equation; make the second-order derivatives of the first spline function and the second spline function at the second interpolation point equal, obtain the fifth equation; analyze the first equation, the second equation, the third equation, the fourth equation and the fifth equation to obtain the third coefficient MC1 and the fourth coefficient MD1 of the first spline function, and the third coefficient MC2 and the fourth coefficient MD2 of the second spline function; determine the first spline function according to MA1, MB1, MC1 and MD1; determine the second spline function according to MA2, MB2, MC2 and MD2.

[0044] In a possible implementation, the first interpolation point is not on the initial tone mapping curve, and the determination of the spline function passing through the multiple interpolation points includes: determining the second coordinate VA1 of the first interpolation point on the linear spline function as the first coefficient MA1 of the first spline function, and the linear spline function is a straight line passing through the origin and the first interpolation point; determining the second coordinate VA3 of the third interpolation point on the initial tone mapping curve; determining the second coordinate VA2 of the second interpolation point on the first straight line as the first coefficient MA2 of the second spline function, and the first straight line is the straight line where the first interpolation point and the third interpolation point are located; determining the first-order derivative of the linear spline function as the MB1; determining the first-order derivative of the initial tone mapping curve at the third interpolation point as the MB2; making the second coordinate value of the second interpolation point on the first spline function the VA2, and obtaining the first equation; making The second coordinate value of the third interpolation point on the second spline function is VA3, and the second equation is obtained; the first-order derivative of the second spline function at the third interpolation point is GD3, and the third equation is obtained; the first-order derivatives of the first spline function and the second spline function at the second interpolation point are equal, and the fourth equation is obtained; the second-order derivatives of the first spline function and the second spline function at the second interpolation point are equal, and the fifth equation is obtained; the first equation, the second equation, the third equation, the fourth equation and the fifth equation are analyzed to obtain the third coefficient MC1 and the fourth coefficient MD1 of the first spline function, and the third coefficient MC2 and the fourth coefficient MD2 of the second spline function; the first spline function is determined according to MA1, MB1, MC1 and MD1; the second spline function is determined according to MA2, MB2, MC2 and MD2.

[0045] In a possible implementation, the dynamic metadata includes a starting position L1 of the spline function interval and an ending position L3 of the spline function interval, and multiple interpolation points are determined based on the dynamic metadata, including: TH1=L1 / 255, TH3=L3 / 255, TH2=(TH1+TH3) / 2; wherein TH1 is the first coordinate of the first interpolation point, TH2 is the first coordinate of the second interpolation point, and TH3 is the first coordinate of the third interpolation point.

[0046] In a possible implementation, the determining of multiple interpolation points according to the brightness information of the target image includes: determining a first coordinate TH1 of the first interpolation point according to the following formula:

[0047]

[0048] Among them, maxL is included in the dynamic metadata, and maxL is the maximum value of the brightness of the image to be processed; when the image to be processed is received, the first coordinate TH1 of the first interpolation point is calculated based on the maxL contained in the dynamic metadata of the image to be processed combined with the above formula. PvalueL is the first threshold of the dark area pixel point, and PvalueH is the second threshold of the brightness value of the dark area pixel point, and the first threshold is less than the second threshold. Exemplarily, the value of PvalueL can be 0.15, and the value of PvalueH can be 0.25; TPH is the third threshold of the bright area pixel point, and TPL is the fourth threshold of the bright area pixel point. Exemplarily, the value of TPH can be 0.6, and the value of TPL can be 0.1. In an optional case, maxL in the above formula can be replaced with avgL, or maxL can be replaced with a*maxL+b*avgL, where a and b are both constants, and 0≤a≤1, 0≤b≤1.

[0049] In a possible implementation manner, the method further includes determining that the first interpolation point is before the second coordinate VA1 on the linear spline function:

[0050] Determine the first-order derivative MB0 of the linear spline function,

[0051]

[0052] Among them, avgL is included in the dynamic metadata, and avgL is the average value of the brightness of the image to be processed; when the image to be processed is received, the slope MB0 of the first spline curve is calculated based on avgL contained in the dynamic metadata combined with the above formula. AvL is the fifth threshold of the average pixel point, and AvH is the sixth threshold of the average pixel point. Exemplarily, the value of AvL can be 0.15, and the value of AvH can be 0.65; SvalueL is the first slope threshold of the increase in the brightness value of the dark area pixel point, and SvalueH is the second slope threshold of the increase in the brightness value of the dark area pixel point, and the first slope threshold is less than the second slope threshold. Exemplarily, the value of SvalueL can be 1, and the value of SvalueH can be 1.2. In an optional case, avgL in the above formula can be replaced by maxL, or replaced by a*maxL+b*avgL, where a and b are both constants, and 0≤a≤1, 0≤b≤1.

[0053] In a possible implementation, the first spline function f1 is:

[0054] f1=MD1×(L-TH1) 3 +MC1×(L-TH1) 2 +MB1×(L-TH1) 1 +MA1;

[0055] The second spline function f2 is:

[0056] f2=MD2×(L-TH2) 3 +MC2×(L-TH2) 2 +MB2×(L-TH2) 1 +MA2;

[0057] Wherein, TH1 is the first coordinate of the first interpolation point, TH2 is the first coordinate of the second interpolation point, and L is the dependent variable of the function.

[0058] In a possible implementation, tone mapping is performed on pixels in the target image that are within the spline function interval according to the spline function, including: when the pixel to be processed is between the first interpolation point and the second interpolation point, tone mapping is performed on the pixel to be processed according to the first spline function; when the pixel to be processed is between the second interpolation point and the third interpolation point, tone mapping is performed on the pixel to be processed according to the second spline function.

[0059] In a second aspect, a method for image processing is provided, comprising: determining a mode indication value in dynamic metadata, the mode indication value being used to indicate a target scene mode applicable to an image to be processed; calling a process for acquiring parameters of a modified spline curve corresponding to the target scene mode according to the mode indication value to acquire a spline correction curve; and tone mapping the image to be processed according to a modified tone mapping curve, the modified tone mapping curve comprising at least a portion of an initial tone mapping curve and the modified spline curve.

[0060] The image processing method provided in the embodiment of the present application determines the scene mode applicable to the image to be processed according to the mode indicator value in the dynamic metadata, and reasonably and flexibly modifies the initial tone mapping curve, so as to adjust the image display effect for different brightness areas, enhance the brightness change details of the image, and bring a better visual experience to the user. By fine-tuning the tone mapping curve in various forms, a variety of display effects such as soft, vivid, bright, and dusk can be obtained.

[0061] In a possible implementation, the process of acquiring the modified spline curve parameters corresponding to the target scene mode according to the mode indication value includes: when the mode indication value is a first preset value, calling the first acquisition process; when the mode indication value is a second preset value, calling the second acquisition process; when the mode indication value is a third preset value, calling the third acquisition process; when the mode indication value is a fourth preset value, calling the fourth acquisition process; when the mode indication value is a fifth preset value, calling the fifth acquisition process.

[0062] In a possible implementation, the modified spline curve obtained by the first acquisition process includes: a first spline curve, a second spline curve and a third spline curve, the first spline curve is a linear spline function passing through the origin and the first interpolation point, the second spline curve is a cubic spline curve passing through the first interpolation point and the second interpolation point, and the third spline curve is a cubic spline curve passing through the second interpolation point and the third interpolation point, and calling the first acquisition process includes: determining the first coordinate TH1 of the first interpolation point according to at least one of the maximum value maxL of the brightness of the image to be processed and the average value avgL of the brightness of the image to be processed included in the dynamic metadata; acquiring the first-order derivative MB0 of the first spline curve according to at least one of the average value avgL of the brightness of the image to be processed and the maximum value maxL of the brightness of the image to be processed included in the dynamic metadata; TH2=TH1+B; TH3=TH2+C*TH2-D*CH2; wherein TH2 is the first coordinate of the second interpolation point, TH3 is the first coordinate of the third interpolation point, and B, C and D are constants. Optionally, B, C and D may be carried in the dynamic metadata; the second coordinate VA1 of the first interpolation point on the first spline is calculated according to TH1 and MB0; the second coordinate VA3 of the third interpolation point on the initial tone mapping curve is calculated according to TH3; the second coordinate VA2 of the second interpolation point on the first straight line is calculated according to TH2, and the first straight line is the straight line where the first interpolation point and the third interpolation point are located; the first-order derivative GD3 of the initial tone mapping curve at the third interpolation point is calculated according to TH3; VA1 is determined as the The first coefficient MA1 of the second spline curve; determine MB0 as the second coefficient MB1 of the second spline curve; determine the sum of VA2 and an offset value as the first coefficient MA2 of the third spline curve, and the offset value is 0, a positive number or a negative number; construct an equation group containing five equations according to TH1, TH2, TH3, VA2, VA3, GD3 and the derivative relationship between the second spline curve and the third spline curve at the second interpolation point; analyze the equation group containing five equations to obtain MC1, MD1, MB2, MC2 and MD2.

[0063] The first acquisition process corresponds to the situation where the overall scene of the image is relatively dark. At this time, the dark area is corrected to better display the dark details of the image, so that the pixels whose brightness is lower than the display capability of the display device can have a better display effect on the display device. Specifically, the pixels below the first threshold (TH1) are tone mapped based on the linear spline function formula, which better retains the dark detail information of this part, and then the linear spline function and the initial tone mapping curve are smoothly transitioned based on the cubic spline function to avoid the non-smooth tone mapping curve affecting the tone mapping effect. It should be understood that there are two cubic spline curves in the embodiment of the present application. In an optional case, there can be only one cubic spline curve. The first spline curve is smoothly connected to the initial tone mapping curve through a cubic spline function. In this case, only two interpolation points need to be determined.

[0064] The first spline curve increases the brightness of the dark area compared to the base curve, which is equivalent to allocating more bits to the dark area. The two cubic spline functions achieve a smooth transition between the first spline curve and the initial tone mapping curve, and can enhance or weaken the correction area by controlling the points and slopes of the curves.

[0065] In a possible implementation, the modified spline curve obtained by the second acquisition process includes: a first spline curve and a second spline curve, the first spline curve is a cubic spline curve passing through the first interpolation point and the second interpolation point, and the second spline curve is a cubic spline curve passing through the second interpolation point and the third interpolation point, and calling the second acquisition process includes: acquiring the first coordinate TH1 of the first interpolation point, the first coordinate TH2 of the second interpolation point, and the first coordinate TH3 of the third interpolation point according to the dynamic metadata; calculating the second coordinates VA1 and VA3 of the first interpolation point and the third interpolation point on the initial tone mapping curve according to TH1 and TH3 respectively; calculating the second coordinate VA2 of the second interpolation point on the first straight line according to TH2, and the first straight line The line is the straight line where the first interpolation point and the third interpolation point are located; the first-order derivative GD1 of the initial tone mapping curve at the first interpolation point and the first-order derivative GD3 at the third interpolation point are calculated; VA1 is determined to be the first coefficient MA1 of the first spline curve and GD1 is determined to be the second coefficient MB1 of the first spline curve; the sum of VA2 and the offset value is determined to be the first coefficient MA2 of the second spline curve, and the offset value is 0, a positive number or a negative number; according to TH1, TH2, TH3, VA2, VA3, GD3 and the derivative relationship between the first spline curve and the second spline curve at the second interpolation point, a system of equations including five equations is constructed; the system of equations including five equations is analyzed to obtain MC1, MD1, MB2, MC2 and MD2.

[0066] The image corresponding to the second acquisition process contains pixels in a specific brightness range. At this time, the specific brightness range is corrected in particular to better display the brightness details of the image in the range. Specifically, based on the cubic spline function, the pixels between the first interpolation point and the second interpolation point and the second interpolation point and the third interpolation point are tone mapped to improve the display effect of these pixels. Furthermore, there is a smooth transition between the first spline curve and the second spline curve, and between the first spline curve and the second spline curve and the initial tone mapping curve, so as to avoid the non-smooth tone mapping curve affecting the tone mapping effect.

[0067] Specifically, this correction method can correct any interval, and the correction curve of the correction interval is smoothly connected to the initial tone mapping curve. The image contrast can be improved by controlling the interpolation point and offset value, or the brightness value of some pixels can be increased or decreased to achieve the desired effect.

[0068] In a possible implementation, the modified spline curve obtained by the third acquisition process includes: a first spline curve, which is a linear spline curve passing through the first interpolation point, and calling the third acquisition process includes: acquiring the first coordinate TH1 of the first interpolation point according to the dynamic metadata; determining the second coordinate VA1 of the first interpolation point on the initial tone mapping curve and the first-order derivative GD1 of the initial tone mapping curve at the first interpolation point according to TH1; determining GD1 as the second coefficient MB0 of the first spline curve; and determining VA1 as the first coefficient MA0 of the first spline curve.

[0069] The third acquisition process corresponds to the situation where the overall scene of the image is relatively bright. At this time, the bright area is corrected to better show the bright details of the image. Specifically, tone mapping is performed on the pixel area greater than the preset threshold (TH1) based on a linear function, and the first spline curve is smoothly connected to the initial tone mapping curve at the first interpolation point to avoid the influence of the rough tone mapping curve on the tone mapping effect.

[0070] Specifically, a first-order spline curve is used to perform tone mapping on pixels greater than the first interpolation point, indicating that points after TH1 are compressed using the compression rate of the first interpolation point to avoid excessive compression of bright area pixels, thereby improving the image brightness value.

[0071] In a possible implementation, the modified spline curve obtained by the fourth acquisition process includes: a first spline curve, the first spline curve is a cubic spline curve passing through the first interpolation point and the second interpolation point, and calling the fourth acquisition process includes: acquiring the first coordinate TH1 of the first interpolation point and the first coordinate TH2 of the second interpolation point according to the dynamic metadata; calculating the second coordinates VA1 and VA2 of the first interpolation point and the second interpolation point on the initial tone mapping curve according to TH1 and TH2 respectively; calculating the initial tone mapping curve on the first interpolation point according to TH1 and TH2. A first-order derivative GD1 at an interpolation point and a first-order derivative GD2 at the second interpolation point; making the second coordinate of the second interpolation point on the first spline curve the sum of VA2 and S1, obtaining a first equation, S1 being the coordinate offset value included in the dynamic metadata; making the first-order derivative of the first spline curve at the second interpolation point the sum of GD2 and S2, obtaining a second equation, S2 being the slope offset value included in the dynamic metadata; parsing the equation group consisting of the first equation and the second equation to obtain the third coefficient MC1 and the fourth coefficient MD1 of the first spline curve.

[0072] The fourth acquisition process corresponds to the situation where the overall scene of the image is relatively bright. At this time, the bright area is corrected to better show the bright details of the image. Specifically, tone mapping is performed on the pixel area between the first interpolation point and the second interpolation point based on the cubic spline function, and the cubic spline function is smoothly connected to the initial tone mapping curve at the first interpolation point to avoid the influence of the rough tone mapping curve on the tone mapping effect.

[0073] Specifically, the brightness value of the pixel between the first interpolation point and the endpoint of the bright area pixel can be increased, or the brightness value between the first interpolation point and the endpoint of the bright area pixel can be reduced, which can satisfy the function of increasing the brightness to achieve a more dizzy effect, or reducing the brightness to achieve the function of enhancing the bright area details, etc. In addition, the increase and decrease is a nonlinear process, which can improve the contrast.

[0074] In a possible implementation, the modified spline curve obtained by the fifth acquisition process includes: a first spline curve and a second spline curve, the first spline curve is a cubic spline curve passing through the first interpolation point and the second interpolation point, and the second spline curve is a linear spline curve passing through the second interpolation point and the third interpolation point, and calling the fifth acquisition process includes: acquiring the first coordinate TH1 of the first interpolation point, the first coordinate TH2 of the second interpolation point, and the first coordinate TH3 of the third interpolation point according to the dynamic metadata; calculating the second coordinates VA1 and VA3 of the first interpolation point and the third interpolation point on the initial tone mapping curve according to TH1 and TH3 respectively; calculating the second coordinates VA1 and VA3 of the second interpolation point on the first straight line according to TH2 A second coordinate VA2 on the line, the first straight line is the straight line where the first interpolation point and the third interpolation point are located; calculate the first-order derivative GD1 of the initial tone mapping curve at the first interpolation point; determine VA1 as the first coefficient MA1 of the first spline curve; determine GD1 as the second coefficient MB1 of the first spline curve; determine the sum of VA2 and the coordinate offset value included in the dynamic metadata as the first coefficient MA2 of the second spline curve; construct an equation group containing three equations according to TH1, TH2, TH3, VA2, VA3 and the derivative relationship between the first spline curve and the second spline curve at the second interpolation point; analyze the equation group containing three equations to obtain MC1, MD1 and MB2.

[0075] In an optional case, the brightness value of the first interpolation point>the brightness value of the second interpolation point>the brightness value of the third interpolation point.

[0076] In an optional case, the brightness value of the first interpolation point<the brightness value of the second interpolation point<the brightness value of the third interpolation point.

[0077] The fifth acquisition process corresponds to the situation where the overall scene of the image is relatively bright. At this time, the bright area is corrected to better show the bright details of the image. Specifically, tone mapping is performed on the pixel area between the second interpolation point and the third interpolation point based on a linear spline curve, and a smooth connection is achieved between the linear spline function and the initial tone mapping curve based on a cubic spline function between the first interpolation point and the second interpolation point to avoid the influence of the non-smooth tone mapping curve on the tone mapping effect.

[0078] Specifically, in this scenario, a cubic spline function can be used between the first interpolation point and the second interpolation point to satisfy the increase and decrease of brightness of the initial tone mapping curve, and this increase and decrease are nonlinear. Tone mapping is performed based on a straight line between the second interpolation point and the third interpolation point, and more bits can be allocated to the pixels between the second interpolation point and the third interpolation point, thereby improving the details of the bright area.

[0079] In a possible implementation, the initial tone mapping curve includes one of the following formulas:

[0080] Among them, L represents the linear signal value, L' represents the nonlinear signal value, p, a, m, b, c1, c2, c3, n, N, ρ are all dynamic range mapping coefficients, 0≤L≤1.

[0081] In a third aspect, the present application provides an image processing device, comprising a module for executing the first aspect or any possible implementation of the first aspect.

[0082] In a fourth aspect, the present application provides an image processing device, comprising: a determination module, used to determine a mode indication value in dynamic metadata, the mode indication value being used to indicate a target scene mode applicable to an image to be processed; the determination module is also used to call a process for acquiring a modified spline curve parameter corresponding to the target scene mode according to the mode indication value to acquire a spline correction curve; a tone mapping module, used to perform tone mapping on the image to be processed according to a modified tone mapping curve, the modified tone mapping curve comprising at least a portion of an initial tone mapping curve and the modified spline curve.

[0083] In one possible implementation, the determination module is specifically used to: when the mode indication value is a first preset value, call a first acquisition process; when the mode indication value is a second preset value, call a second acquisition process; when the mode indication value is a third preset value, call a third acquisition process; when the mode indication value is a fourth preset value, call a fourth acquisition process; when the mode indication value is a fifth preset value, call a fifth acquisition process.

[0084] In a possible implementation, the modified spline curve obtained by the first acquisition process includes: a first spline curve, a second spline curve and a third spline curve, the first spline curve is a linear spline function passing through the origin and the first interpolation point, the second spline curve is a cubic spline curve passing through the first interpolation point and the second interpolation point, and the third spline curve is a cubic spline curve passing through the second interpolation point and the third interpolation point, and the determination module is specifically used to: determine the first coordinate TH1 of the first interpolation point according to at least one of the maximum value maxL of the brightness of the image to be processed and the average value avgL of the brightness of the image to be processed included in the dynamic metadata; obtain the first-order derivative MB0 of the first spline curve according to at least one of the maximum value maxL of the brightness of the image to be processed and the average value avgL of the brightness of the image to be processed included in the dynamic metadata; TH2=TH1+B; TH3=TH2+C*TH2-D*CH2; wherein TH2 is the first coordinate of the second interpolation point, TH3 is the first coordinate of the third interpolation point, B, C and D are constants, and are optional. , B, C and D are constants included in the dynamic metadata; according to TH1 and MB0, the second coordinate VA1 of the first interpolation point on the first spline is calculated; according to TH3, the second coordinate VA3 of the third interpolation point on the initial tone mapping curve is calculated; according to TH2, the second coordinate VA2 of the second interpolation point on the first straight line is calculated, and the first straight line is the straight line where the first interpolation point and the third interpolation point are located; according to TH3, the first derivative GD3 of the initial tone mapping curve at the third interpolation point is calculated; VA1 is determined as the first interpolation point of the first interpolation curve; The invention discloses a method for obtaining a first coefficient MA1 of a second spline curve; determining MB0 as the second coefficient MB1 of the second spline curve; determining the sum of VA2 and an offset value as the first coefficient MA2 of the third spline curve, wherein the offset value is 0, a positive number or a negative number; constructing an equation group including five equations according to TH1, TH2, TH3, VA2, VA3, GD3 and the derivative relationship between the second spline curve and the third spline curve at the second interpolation point; analyzing the equation group including five equations to obtain MC1, MD1, MB2, MC2 and MD2.

[0085] In a possible implementation, the modified spline curve obtained by the second acquisition process includes: a first spline curve and a second spline curve, the first spline curve is a cubic spline curve passing through the first interpolation point and the second interpolation point, and the second spline curve is a cubic spline curve passing through the second interpolation point and the third interpolation point, and the determination module is specifically used to: obtain the first coordinate TH1 of the first interpolation point, the first coordinate TH2 of the second interpolation point, and the first coordinate TH3 of the third interpolation point according to the dynamic metadata; calculate the second coordinates VA1 and VA3 of the first interpolation point and the third interpolation point on the initial tone mapping curve according to TH1 and TH3 respectively; calculate the second coordinate VA2 of the second interpolation point on the first straight line according to TH2, and the first straight line The line is the straight line where the first interpolation point and the third interpolation point are located; the first-order derivative GD1 of the initial tone mapping curve at the first interpolation point and the first-order derivative GD3 at the third interpolation point are calculated; VA1 is determined to be the first coefficient MA1 of the first spline curve and GD1 is determined to be the second coefficient MB1 of the first spline curve; the sum of VA2 and the offset value is determined to be the first coefficient MA2 of the second spline curve, and the offset value is 0, a positive number or a negative number; according to TH1, TH2, TH3, VA2, VA3, GD3 and the derivative relationship between the first spline curve and the second spline curve at the second interpolation point, a system of equations including five equations is constructed; the system of equations including five equations is analyzed to obtain MC1, MD1, MB2, MC2 and MD2.

[0086] In a possible implementation, the modified spline curve obtained by the third acquisition process includes: a first spline curve, which is a linear spline curve passing through the first interpolation point, and the determination module is specifically used to: obtain the first coordinate TH1 of the first interpolation point according to the dynamic metadata; determine the second coordinate VA1 of the first interpolation point on the initial tone mapping curve and the first-order derivative GD1 of the initial tone mapping curve at the first interpolation point according to TH1; determine GD1 as the second coefficient MB0 of the first spline curve; determine VA1 as the first coefficient MA0 of the first spline curve.

[0087] In a possible implementation manner, the modified spline curve obtained by the fourth acquisition process includes: a first spline curve, the first spline curve is a cubic spline curve passing through the first interpolation point and the second interpolation point, and the determination module is specifically used to: obtain the first coordinate TH1 of the first interpolation point and the first coordinate TH2 of the second interpolation point according to the dynamic metadata; calculate the second coordinates VA1 and VA2 of the first interpolation point and the second interpolation point on the initial tone mapping curve according to TH1 and TH2 respectively; calculate the initial tone mapping curve on the first interpolation point according to TH1 and TH2 The first-order derivative GD1 at the interpolation point and the first-order derivative GD2 at the second interpolation point; making the second coordinate of the second interpolation point on the first spline curve the sum of VA2 and S1, obtaining the first equation, S1 is the coordinate offset value included in the dynamic metadata; making the first-order derivative of the first spline curve at the second interpolation point the sum of GD2 and S2, obtaining the second equation, S2 is the slope offset value included in the dynamic metadata; parsing the equation group consisting of the first equation and the second equation to obtain the third coefficient MC1 and the fourth coefficient MD1 of the first spline curve.

[0088] In a possible implementation, the modified spline curve obtained by the fifth acquisition process includes: a first spline curve and a second spline curve, the first spline curve is a cubic spline curve passing through the first interpolation point and the second interpolation point, and the second spline curve is a linear spline curve passing through the second interpolation point and the third interpolation point, and the determination module is specifically used to: obtain the first coordinate TH1 of the first interpolation point, the first coordinate TH2 of the second interpolation point, and the first coordinate TH3 of the third interpolation point according to the dynamic metadata; calculate the second coordinates VA1 and VA3 of the first interpolation point and the third interpolation point on the initial tone mapping curve according to TH1 and TH3 respectively; calculate the second interpolation point on the first straight line according to TH2 The second coordinate VA2 on the first interpolation point, the first straight line is the straight line where the first interpolation point and the third interpolation point are located; calculate the first-order derivative GD1 of the initial tone mapping curve at the first interpolation point; determine VA1 as the first coefficient MA1 of the first spline curve; determine GD1 as the second coefficient MB1 of the first spline curve; determine the sum of VA2 and the offset value as the first coefficient MA2 of the second spline curve, and the offset value is 0, a positive number or a negative number; construct an equation group containing three equations according to TH1, TH2, TH3, VA2, VA3 and the derivative relationship between the first spline curve and the second spline curve at the second interpolation point; analyze the equation group containing three equations to obtain MC1, MD1 and MB2. In a fourth aspect, the present application provides an image processing device, including a processor and a transmission interface, the transmission interface is used to receive or send image data; the processor is used to call program instructions stored in a memory, and execute the program instructions to execute the method of the first aspect or any one of the implementations of the first aspect.

[0089] Exemplarily, the image processing device may be a chip.

[0090] In a fifth aspect, the present application provides an image processing device, comprising a processor and a transmission interface, wherein the transmission interface is used to receive or send image data; the processor is used to call program instructions stored in a memory and execute the program instructions to execute the method of the first aspect or any possible implementation method of the first aspect.

[0091] In a possible implementation manner, the image processing device may further include a memory.

[0092] In a sixth aspect, the present application provides an image processing device, comprising a processor and a transmission interface, wherein the transmission interface is used to receive or send image data; the processor is used to call program instructions stored in a memory and execute the program instructions to execute the method in the second aspect or any possible implementation manner of the second aspect.

[0093] In a seventh aspect, the present application provides an image processing device, comprising a processor and a memory, for executing the method described in the first aspect or any one of the methods of the first aspect.

[0094] In an eighth aspect, the present application provides an image processing device, comprising a processor and a memory, for executing the method described in the second aspect or any one of the methods of the second aspect.

[0095] In a ninth aspect, the present application provides a computer-readable storage medium storing program instructions, which, when executed on a computer or a processor, causes the computer or processor to execute the method described in the first aspect or any one of the implementations of the first aspect.

[0096] In the tenth aspect, the present application provides a computer-readable storage medium, which stores program instructions. When the program instructions are executed on a computer or a processor, the computer or the processor executes the method described in the second aspect or any one of the implementations of the second aspect.

[0097] In an eleventh aspect, the present application provides a computer program product, including a computer program. When the computer program is executed on a computer or a processor, it will cause the computer or the processor to execute the method described in the first aspect or any one of the implementations of the first aspect.

[0098] In a twelfth aspect, the present application provides a computer program product, including a computer program. When the computer program is executed on a computer or a processor, it will cause the computer or the processor to execute the method described in the second aspect or any one of the implementations of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] Figure 1 is a schematic block diagram of an image display system provided in an embodiment of the present application;

[0100] Figure 2 is a hardware architecture diagram of an exemplary image processing device provided in an embodiment of the present application;

[0101] Figure 3 is a flowchart of an exemplary image processing method provided in an embodiment of the present application;

[0102] Figure 4a is a schematic diagram of an exemplary modified spline curve provided in an embodiment of the present application;

[0103] Figure 4b is a schematic diagram of an exemplary modified spline curve provided in an embodiment of the present application;

[0104] Figure 4c is a schematic diagram of an exemplary modified spline curve provided in an embodiment of the present application;

[0105] Figure 4d is a schematic diagram of an exemplary modified spline curve provided in an embodiment of the present application;

[0106] Figure 4e is a schematic diagram of an exemplary modified spline curve provided in an embodiment of the present application;

[0107] Figure 4f is a schematic diagram of several exemplary initial tone mapping curves provided in an embodiment of the present application;

[0108] Figure 5 is a flowchart of an exemplary method for obtaining parameters of a modified spline curve provided in an embodiment of the present application;

[0109] Figure 6a is a flowchart of an exemplary method for obtaining parameters of a modified spline curve provided in an embodiment of the present application;

[0110] Figure 6b An exemplary flow chart of a method for obtaining parameters of a modified spline curve provided in an embodiment of the present application;

[0111] Figure 7 is a flowchart of an exemplary method for obtaining parameters of a modified spline curve provided in an embodiment of the present application;

[0112] Figure 8 is a flowchart of an exemplary method for obtaining parameters of a modified spline curve provided in an embodiment of the present application;

[0113] Fig. 9 is a flowchart of an exemplary method for obtaining parameters of a modified spline curve provided in an embodiment of the present application;

[0114] Fig.10 is a flowchart of an exemplary method for obtaining parameters of a modified spline curve provided in an embodiment of the present application;

[0115] Fig.11 It is a structural block diagram of an exemplary image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0116] The terms "first", "second", etc. in the specification embodiments and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or precedence. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, including a series of steps or units. The method, system, product or device is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0117] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0118] The image acquisition device can generate a picture or video containing a natural scene by receiving an optical signal of the natural scene. In order to facilitate the transmission of the image or video, the optical signal needs to be converted into an electrical signal, and the image information of each pixel is recorded through a brightness value or a chromaticity value with a fixed value range (for example, a grayscale value between 0 and 255). The curve for converting the optical signal into an electrical signal is a photoelectric transfer function. Common photoelectric transfer functions include PQ (perceptual quantizer) photoelectric transfer function, HLG (hybrid log-gamma) photoelectric transfer function, and scene luminance fidelity (SLF) photoelectric transfer function. The display device can reproduce the natural scene by displaying pictures or videos. In an optional case, the image acquisition device is called a front-end device, and the display device is called a back-end device. The display device can determine the brightness of an object when shooting based on the photoelectric transfer function and the brightness value or grayscale value of each pixel in the image, that is, the electronic device can convert YUV or RGB information into brightness in units of nit. However, the brightness of the object may exceed the display capability of the display device. Since the brightness information acquired by the image acquisition device does not match the brightness display capability of the display device, there are situations where a high-brightness image is displayed using a display device with a low-brightness display capability, and a low-brightness image is displayed using a display device with a high-brightness display capability. In these situations, it is necessary to perform tone mapping (tone-mapping) on ​​the image acquired by the image acquisition device so that the image conforms to the display capability of the display device. Exemplarily, the high dynamic range image can be tone mapped to a display device with a low dynamic range display capability based on a tone mapping curve. It should be understood that the tone mapping curve can be determined by the display device or by a processing device external to the display device.

[0119] However, in some cases, when the tone mapping curve is not accurately determined, the display effect of the high dynamic range image is poor. For example, during video playback, if only one tone mapping curve is determined for the video, the image content displayed between different video frames may be quite different. For example, the image displayed by some video frames is very dark, while the image displayed by other video frames is very bright. If a fixed tone mapping curve is used for the entire video, it is impossible to perfectly reproduce each frame of the video on the display device. In addition, the tone mapping curve carries a large amount of data. If a tone mapping curve is determined for each image or frame, data redundancy may be introduced.

[0120] The present application provides an image processing method and device, which flexibly corrects the initial tone mapping curve according to the scene characteristics, so that after the image tone mapping in different scenes, a better display effect or one that meets subjective needs can be obtained.

[0121] like Figure 1 The figure shows a schematic block diagram of an exemplary image display system 10 provided in an embodiment of the present application. The image display system 10 may include an image acquisition device 12, an image processing device 14 and a display device 34. For example, the image acquisition device 12 may acquire an HDR image or a Standard Dynamic Range (SDR) image. The image processing device 14 may process the HDR image or SDR image acquired by the image acquisition device 12.

[0122] The image acquisition device 12 and the image processing device 14 can be communicatively connected via a link 13, and the image processing device 14 can receive image data from the image acquisition device 12 via the link 13. The link 13 may include one or more communication media or devices. The one or more communication media may include wireless and / or wired communication media, such as a radio frequency (RF) spectrum or one or more physical transmission lines. Optionally, the one or more communication media may form part of a packet-based network, which may be, for example, a local area network, a wide area network, or a global network (e.g., the Internet). The one or more communication media may include routers, switches, base stations, or other devices that facilitate communication from the image acquisition device 12 to the image processing device 14. In an optional case, the link 13 may be a Bluetooth wireless link.

[0123] Exemplarily, the image acquisition device 12 includes an image source 16 and a communication interface 22 or a transmission interface. Alternatively, the image acquisition device 12 may also include an image processor 20. In a specific implementation, the image processor 20, the image source 16, and the communication interface 22 may be hardware components in the image acquisition device 12, or may be software programs in the image acquisition device 12, or the communication interface 22 may be a hardware module combined with a software program.

[0124] The picture source 16 may include or may be any type of picture capture device, for example, for capturing real-world pictures, and / or any type of pictures or comments (for screen content encoding, some text on the screen is also considered to be part of the picture or image to be encoded) generating device, for example, a computer graphics processor for generating computer animation pictures, or any type of device for acquiring and / or providing real-world pictures, computer animation pictures (for example, screen content, virtual reality (VR) pictures), and / or any combination thereof (for example, augmented reality (AR) pictures). Exemplarily, the picture source 16 may be a camera for capturing pictures or a memory for storing pictures, and the picture source 16 may also include any type of (internal or external) interface for storing previously captured or generated pictures and / or acquiring or receiving pictures. When the picture source 16 is a camera, the picture source 16 may be, for example, a local or integrated camera integrated in the image acquisition device; when the picture source 16 is a memory, the picture source 16 may be a local or integrated memory integrated in the image acquisition device. When the image source 16 includes an interface, the interface may be, for example, an external interface for receiving images from an external video source, the external video source being, for example, an external image capturing device such as a camera, an external memory, or an external image generating device, the external image generating device being, for example, an external computer graphics processor, a computer, or a server.

[0125] Among them, the picture can be regarded as a two-dimensional array or matrix of pixels. The pixels in the array can also be called sampling points. In an optional case, in order to represent the color, each pixel includes three color components. For example, in the RGB format or color space, the picture includes corresponding red, green and blue sampling arrays. However, in video coding, each pixel is usually represented in a brightness / chrominance format or color space, such as for a picture in the YUV format, including a brightness component indicated by Y (sometimes also indicated by L) and two chrominance components indicated by U and V. The brightness (luma) component Y represents the brightness or grayscale level intensity (for example, both are the same in a grayscale picture), while the two chroma (chroma) components U and V represent the chroma or color information components. Accordingly, the picture in the YUV format includes a brightness sampling array of brightness sampling values ​​(Y), and two chroma sampling arrays of chroma values ​​(U and V). Pictures in RGB format can be converted or transformed into YUV format, and vice versa. This process is also called color conversion or color format conversion. If the picture is black and white, the picture can only include a brightness sampling array. In the embodiment of the present application, the picture transmitted from the picture source 16 to the image processor may also be referred to as original picture data 17 .

[0126] The image processor 20 is used to perform image processing, such as brightness mapping, tone mapping, color format conversion (for example, from RGB format to YUV format), color gamut conversion, saturation adjustment, color adjustment, resolution adjustment or denoising.

[0127] The communication interface 22 may be used to receive the image data 19 after image processing, and may transmit the image data 19 after image processing to the image processing device 14 for further image processing, or to the memory for storage via the link 13. Exemplarily, the communication interface 22 may be used to encapsulate the image data 19 after image processing into a suitable format, such as a data packet, for transmission over the link 13.

[0128] The image processing device 14 includes a communication interface 28, an image processor 32 and a display device 34. They are described as follows:

[0129] The communication interface 28 can be used to receive the image data 19 after image processing from the image acquisition device 12 or any other source, such as a storage device. For example, the specific examples of the communication interface 22 and the communication interface 28 can refer to the above description of the interface, which will not be repeated here. The communication interface 28 can be used to transmit or receive the image data 19 after image processing via the link 13 between the image acquisition device 12 and the image processing device 14 or any other type of network. The communication interface 28 can be used, for example, to decapsulate the data packet transmitted by the communication interface 22 to obtain the image data 19 after image processing.

[0130] Both the communication interface 28 and the communication interface 22 can be configured as a unidirectional communication interface or a bidirectional communication interface, and can be used, for example, to send and receive messages to establish a connection, confirm and exchange any other information related to the communication link and / or, for example, image data processed by the image and / or data transmission. Exemplarily, the communication interface 28 and the communication interface 22 can be any type of interface according to any proprietary or standardized interface protocol, such as a high definition multimedia interface (HDMI), a mobile industry processor interface (MIPI), a display serial interface (DSI) standardized by MIPI, an embedded display port (eDP) standardized by the Video Electronics Standards Association (VESA), a Display Port (DP) or a V-By-One interface, which is a digital interface standard developed for image transmission, as well as various wired or wireless interfaces, optical interfaces, etc.

[0131] The image processor 32 is used to perform tone mapping processing on the image data 19 after image processing to obtain tone mapped image data 33. The processing performed by the image processor 32 may also include: super resolution, color format conversion (for example, from YUV format to RGB format), denoising, color gamut conversion, saturation adjustment, brightness mapping, upsampling, downsampling and image sharpening, and may also be used to transmit the tone mapped image data 33 to a display device 34. It should be understood that the image processor 20 and the image processor 32 may be a general-purpose central processing unit (CPU), a system on chip (SOC), a processor integrated on a SOC, a separate processor chip or controller, etc.; the image processor 20 and the image processor 32 may also be a dedicated processing device, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or a digital signal processor (DSP), a dedicated video or graphics processor, a graphics processing unit (GPU) and a neural-network processing unit (NPU), etc. The image processor 20 and the image processor 32 may also be a processor group composed of multiple processors, and the multiple processors are coupled to each other through one or more buses.

[0132] A display device 34 is used to receive the tone-mapped image data 33 to display the image to a user or viewer. The display device 34 can be or include any type of display for presenting the reconstructed image, such as an integrated or external display or monitor. For example, the display can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display. In an optional case, the display device 34 itself has an image processing function, and the tone mapping of the image can also be implemented at the display device.

[0133] It should be understood that Figure 1The image acquisition device 12, the image processing device 14, and the display device 34 are shown as separate devices. In an optional case, an image processing device may have the functions of both the image acquisition device 12 and the image processing device 14. In an optional case, an image processing device may have the functions of both the image processing device 14 and the display device 34. In an optional case, an image processing device has the functions of the image acquisition device 12, the image processing device 14, and the display device 34. For example, a smart mobile phone has a camera, an image processor, and a display screen, wherein the camera corresponds to the image acquisition device 12, the image processor corresponds to the image processing device 14, and the display screen corresponds to the display device 34. For another example, a smart TV has a camera, an image processor, and a display screen, wherein the camera corresponds to the image acquisition device 12, the image processor corresponds to the image processing device 14, and the display screen corresponds to the display device 34.

[0134] In an optional case, the image processing device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a CPU, a memory management unit (MMU) and a memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through a process, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system or a Windows operating system. The application layer includes applications such as a browser, an address book, a word processing software, and an instant messaging software. In addition, the embodiment of the present application does not specifically limit the specific structure of the execution subject of the method provided in the embodiment of the present application, as long as the image processing can be performed by running the relevant code to implement the method provided in the embodiment of the present application, for example, the execution subject of the method provided in the embodiment of the present application can be an electronic device, or a functional module in an electronic device that can call a program and execute the program, such as a processor in an electronic device.

[0135] It should be understood that the image acquisition device 12 and the image processing device 14 may include any type of handheld or stationary device, such as a notebook or laptop computer, a mobile phone, a smart phone, a tablet or tablet computer, a camcorder, a desktop computer, a set-top box, a television, a camera, a vehicle-mounted device, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or a content distribution server), a broadcast receiver device, a broadcast transmitter device, etc., and may not use or use any type of operating system.

[0136] like Figure 2, which is a hardware architecture diagram of an exemplary image processing device provided in an embodiment of the present application. The image processing device 200 may be, for example, a processor chip. Exemplarily, Figure 2 The hardware architecture diagram shown in can be Figure 1 The exemplary architecture diagram of the image processor 20 or the image processor 32 in FIG. 1 is a diagram showing an exemplary architecture of the image processor 20 or the image processor 32 in FIG. 1 , and the image processing method provided in the embodiment of the present application can be applied to the processor chip.

[0137] refer to Figure 2 The device 200 includes: at least one CPU, a microcontroller unit (MCU), a GPU, an NPU, a memory bus, a receiving interface, and a sending interface. Figure 2 Not shown, the device 200 may also include an application processor (AP), a memory, a decoder, and a dedicated video or image processor. The above-mentioned parts of the device 200 are coupled through a connector. Exemplarily, the connector includes various interfaces, transmission lines or buses, etc. These interfaces are usually electrical communication interfaces, but may also be mechanical interfaces or other forms of interfaces, which are not limited in this embodiment.

[0138] Optionally, the CPU may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor; Optionally, the CPU may be a processor group consisting of multiple processors, and the multiple processors are coupled to each other through one or more buses. The receiving interface may be an interface for data input of a processor chip. In an optional case, the types of the receiving interface and the sending interface may refer to the description of the communication interface 28 and the communication interface 22, which will not be repeated here.

[0139] The memory is coupled to the CPU, GPU, MCU, and NPU through a memory bus. The memory can be used to store computer program instructions, including a computer operating system (OS) and various user applications. The memory is also used to store image data or video data. The memory may be a non-power-off volatile memory, such as an embedded Multi Media Card (EMMC), a universal flash storage (UFS) or a read-only memory (ROM), or other types of static storage devices that can store static information and instructions, or a power-off volatile memory (volatile memory), such as a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other computer-readable storage medium that can be used to carry or store program codes in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. It should be understood that the memory may be independent of the image processing device 200.

[0140] In an optional case, the above-mentioned parts are integrated on the same chip; in another optional case, the CPU, GPU, decoder, receiving interface and sending interface are integrated on one chip, and the various parts inside the chip access the external memory through the bus. The dedicated video / graphics processor can be integrated on the same chip as the CPU, or it can exist as a separate processor chip. For example, the dedicated video / graphics processor can be a dedicated ISP. In an optional case, the NPU can also be used as an independent processor chip. The NPU is used to implement various neural network or deep learning related operations. Optionally, the image processing method and image processing framework provided in the embodiment of the present application can be implemented by a GPU or an NPU, or by a dedicated graphics processor.

[0141] The chip involved in the embodiments of the present application is a system manufactured on the same semiconductor substrate using an integrated circuit process, also called a semiconductor chip, which can be a collection of integrated circuits formed on a substrate (usually a semiconductor material such as silicon) using an integrated circuit process, and its outer layer is usually encapsulated by a semiconductor packaging material. The integrated circuit may include various functional devices, each of which includes transistors such as logic gate circuits, metal oxide semiconductor (MOS) transistors, bipolar transistors or diodes, and may also include other components such as capacitors, resistors or inductors. Each functional device can work independently or under the action of necessary driver software, and can realize various functions such as communication, calculation, or storage.

[0142] like Figure 3 FIG. 1 is a flow chart of an image processing method provided in an embodiment of the present application. It should be understood that for ease of description, Figure 3 Describe the method in steps, although in the method flow Figure 3 The order of the methods is shown in the figure, but in some cases, the steps described can be performed in an order different from that shown here.

[0143] The image processing method includes:

[0144] 301. Determine a mode indication value in dynamic metadata, where the mode indication value is used to indicate a target scene mode applicable to the image to be processed;

[0145] It should be understood that the dynamic metadata is generated by the video front-end device when generating image data or video data. The dynamic metadata includes the mode indication value. For example, the mode indication value can be obtained by the front-end device analyzing the acquired or generated image data. The video front-end device can be as follows: Figure 1The image acquisition device 12 shown. Exemplarily, when acquiring or generating image data, the image acquisition device 12 performs image frame analysis on each frame of image data to obtain feature information of each frame of image data. Further, the tone mapping mode to which the image data is adapted can be determined based on the feature information obtained by the analysis. For example, if the overall scene of the current image is relatively dark, it is confirmed that the image frame is adapted to the dark scene tone mapping mode; if the overall scene of the current image is relatively bright, it is confirmed that the image frame is adapted to the bright scene tone mapping mode; if the current image contains pixels located within a specific brightness interval, it is confirmed that the image frame is adapted to the specific scene tone mapping mode. Further, after receiving the image to be processed and the dynamic metadata of the image to be processed, the image processing device 14 or the display device 34 adaptively corrects the initial tone mapping curve according to the mode indication value in the dynamic metadata, and tone maps the image to be processed based on the corrected tone mapping curve. In an optional case, the set-top box can be referred to as a playback device. It should be understood that the execution subject of step 301 can be Figure 1 The image processing device 14 or display device 34 shown in the figure, exemplarily, the image processing device 14 may be a set-top box or an image processor in a set-top box, or may be an image processor or an image processing unit in a smart mobile terminal with playback and display capabilities.

[0146] 302. Calling a process for acquiring modified spline curve parameters corresponding to the target scene mode according to the mode indication value, and acquiring a spline curve;

[0147] Specifically,

[0148] When the mode indication value is a first preset value, calling a first acquisition process;

[0149] When the mode indication value is a second preset value, calling a second acquisition process;

[0150] When the mode indication value is a third preset value, calling a third acquisition process;

[0151] When the mode indication value is a fourth preset value, calling a fourth acquisition process;

[0152] When the mode indication value is a fifth preset value, a fifth acquisition process is called.

[0153] Exemplarily, the mode indication value contained in the dynamic metadata is set to 3Spline_num.

[0154] If 3Spline_num is equal to 0, the spline curve parameter acquisition process 0 is called to obtain the modified spline curve parameters corresponding to the first scene mode;

[0155] If 3Spline_num is equal to 1, the spline curve parameter acquisition process 1 is called to obtain the modified spline curve parameters corresponding to the second scene mode;

[0156] If 3Spline_num is equal to 2, the spline curve parameter acquisition process 2 is called to obtain the modified spline curve parameters corresponding to the third scene mode;

[0157] If 3Spline_num is equal to 3, the spline curve parameter acquisition process 3 is called to obtain the modified spline curve parameters corresponding to the fourth scene mode;

[0158] If 3Spline_num is equal to 4, the spline curve parameter acquisition process 4 is called to obtain the modified spline curve parameters corresponding to the fifth scene mode.

[0159] In an optional case, the value of 3Spline_num can also be other integers greater than or equal to 5, and corresponds to other spline curve parameter acquisition processes, which is not limited in the embodiment of the present application. Optionally, the value of 3Spline_num can also be English letters, Greek letters or other preset indication values, not necessarily Roman numerals.

[0160] Different spline curve parameters obtain different parameters of the modified spline curve, and the parameters of the modified spline curve include a first coordinate of at least one interpolation point and a plurality of coefficients of the modified spline curve;

[0161] The at least one interpolation point constitutes a correction interval, and each two adjacent interpolation points constitute a sub-correction interval. The correction spline curve passes through the at least one interpolation point. Exemplarily, the correction spline curve may include multiple spline curves. When the spline curve includes spline curve 1 and spline curve 2, spline curve 1 may be in [x M1 ,x M2 ] interval, the spline curve 2 can be in [x L1 ,x L2 ] interval L-order spline function, where the interval [x M1 ,x M2 ] and the interval [x L1 ,x L2 ] is empty, or the interval [x M1 ,x M2 ] and the interval [x L1 ,x L2] have a common endpoint value. The degree M of spline curve 1 and the degree L of spline curve 2 may be the same or different. Exemplarily, the multiple modified spline curves may include at least one of a linear spline curve or a cubic spline curve, and the number of cubic spline curves included in the modified spline curve may be one, two, three or more, and the embodiment of the present application does not limit the number of cubic spline curves. The cubic spline curve is smoothly connected to the initial tone mapping curve. Exemplarily, when the modified spline curve includes a linear spline curve and a cubic spline curve, the cubic spline curve is smoothly connected to the linear spline curve and the initial tone mapping curve.

[0162] The function form of a linear spline curve is y=ax+b, including two coefficients. At least two known conditions are required to determine a linear spline curve.

[0163] The function form of the quadratic spline curve is y = ax 2 +bx+c, and a≠0, including three coefficients, at least three known conditions are needed to determine the quadratic spline curve, such as the coordinates of three points passing through the quadratic spline curve; or, the coordinates of two points passing through the quadratic spline curve and the derivative of the quadratic spline curve at one of the points.

[0164] The function form of the cubic spline curve is y = ax 3 +bx 2 +cx+d, and a≠0, including four coefficients, at least 4 known conditions are required to determine the cubic spline curve, such as knowing in advance the coordinates of 4 points passing through the cubic spline curve; or, knowing in advance the coordinates of 2 points and the derivatives of the cubic spline curve at the two points.

[0165] The N-order spline curve includes N+1 coefficients, and at least N+1 known conditions are required to determine the N-order spline curve.

[0166] For example, Figure 4a As shown, it is a schematic diagram of an exemplary modified spline curve corresponding to the above-mentioned first scene mode provided in an embodiment of the present application. The first scene mode corresponds to the situation where the overall scene of the image is relatively dark.

[0167] The multiple interpolation points include: a first interpolation point, a second interpolation point, and a third interpolation point. The modified spline curve includes: a first spline curve, a second spline curve, and a third spline curve. The first spline curve is a linear spline curve, and the second spline curve and the third spline curve are both cubic spline curves. The first spline curve is a straight line between the origin and the first interpolation point, the second spline curve is a spline curve between the first interpolation point and the second interpolation point, and the third spline curve is a spline curve between the second interpolation point and the third interpolation point. The first spline curve is smoothly connected to the second spline curve at the first interpolation point, the second spline curve is smoothly connected to the third spline curve at the second interpolation point, and the third spline curve is smoothly connected to the initial tone mapping curve at the third interpolation point. The output parameters of the modified spline curve include: TH1, TH2, TH3, the slope MB0 of the first spline curve, the four coefficients MA1, MB1, MC1, and MD1 of the second spline curve, and the four coefficients MA2, MB2, MC2, and MD2 of the third spline curve.

[0168] The first spline: F(L) = MB0 × L(1)

[0169] Second spline curve: F(L)=MD1×(L-TH1) 3 +MC1×(L-TH1) 2 +MB1×(L-TH1) 1 +MA1 (2)

[0170] The third spline curve: F(L) = MD2 × (L-TH2) 3 +MC2×(L-TH2) 2 +MB2×(L-TH2) 1 +MA2 (3)

[0171] Among them, TH1 is the first coordinate of the first interpolation point, TH2 is the first coordinate of the second interpolation point, TH3 is the first coordinate of the third interpolation point, the first coordinate may be, for example, a horizontal coordinate, and L is the independent variable of the function.

[0172] The first scene mode corresponds to the situation where the overall scene of the image is relatively dark. At this time, the dark area is corrected to better display the dark details of the image, so that the pixels with brightness lower than the display capability of the display device can have a better display effect on the display device. Specifically, the pixels below the first threshold (TH1) are tone mapped based on the linear spline function formula (1), so as to better retain the dark detail information of this part, and then a smooth transition between the linear spline function and the initial tone mapping curve is achieved based on the cubic spline function to avoid the influence of the non-smooth tone mapping curve on the tone mapping effect. It should be understood that there are two cubic spline curves in the first scene mode. In an optional case, there may be only one cubic spline curve. The first spline curve is smoothly connected to the initial tone mapping curve through a cubic spline function. In this case, only two interpolation points need to be determined. The acquisition process in the first scene mode is the first acquisition process. For details of the first acquisition process, please refer to Figure 6a and Figure 6b The method shown or the following case 3.1. Specifically, in this scenario, the first spline curve increases the brightness of the dark area compared to the base curve, which is equivalent to allocating more bits to the dark area. The two cubic spline functions achieve a smooth transition between the first spline curve and the initial tone mapping curve, and the correction area can be enhanced or weakened by controlling the points and slopes of the curves.

[0173] like Figure 4b As shown, it is a schematic diagram of an exemplary modified spline curve corresponding to the second scene mode provided in an embodiment of the present application. The image corresponding to the second scene mode contains pixels located in a specific brightness range.

[0174] The multiple interpolation points include: a first interpolation point, a second interpolation point, and a third interpolation point. The first interpolation point and the third interpolation point are on the initial tone mapping curve. The second interpolation point may be on the initial tone mapping curve or may not be on the initial tone mapping curve. When the second interpolation point is not on the initial tone mapping curve, there is a vertical coordinate offset value between the second interpolation point and the initial tone mapping curve. The modified spline curve includes: a first spline curve and a second spline curve. Both the first spline curve and the second spline curve are cubic spline curves. The first spline curve is a spline curve between the first interpolation point and the second interpolation point, and the second spline curve is a spline curve between the second interpolation point and the third interpolation point. Exemplarily, the first spline curve is smoothly connected to the initial tone mapping curve at the first interpolation point, the second spline curve is smoothly connected to the initial tone mapping curve at the third interpolation point, and the first spline curve is smoothly connected to the second spline curve at the second interpolation point. The output parameters of the modified spline curve include: TH1, TH2, TH3, four coefficients MA1, MB1, MC1 and MD1 of the first spline curve, and four coefficients MA2, MB2, MC2 and MD2 of the second spline curve.

[0175] The first spline curve: F(L) = MD1 × (L-TH1) 3 +MC1×(L-TH1) 2 +MB1×(L-TH1) 1 +MA1 (4)

[0176] Second spline curve: F(L)=MD2×(L-TH2) 3 +MC2×(L-TH2) 2 +MB2×(L-TH2) 1 +MA2 (5)

[0177] Among them, TH1 is the first coordinate of the first interpolation point, TH2 is the first coordinate of the second interpolation point, TH3 is the first coordinate of the third interpolation point, the first coordinate may be, for example, a horizontal coordinate, and L is the independent variable of the function.

[0178] The image corresponding to the second scene mode contains pixels located in a specific brightness range. At this time, the specific brightness range is corrected in particular to better show the brightness details of the image in this range. Specifically, based on the cubic spline function, tone mapping is performed on the pixels between the first interpolation point and the second interpolation point and between the second interpolation point and the third interpolation point to improve the display effect of these pixels. Furthermore, there is a smooth transition between the first spline curve and the second spline curve, and between the first spline curve and the second spline curve and the initial tone mapping curve to avoid the influence of the non-smooth tone mapping curve on the tone mapping effect. The acquisition process in the second scene mode is the second acquisition process. For details of the second acquisition process, please refer to Figure 7 The method shown or the following situation 3.3. Specifically, the correction method can correct any interval, and the correction curve of the correction interval is smoothly connected with the initial tone mapping curve, and the image contrast can be improved by controlling the interpolation point and the offset value, or the brightness value of some pixels can be increased or decreased, so as to achieve the desired effect.

[0179] like Figure 4c As shown, it is a schematic diagram of an exemplary modified spline curve corresponding to the third scene mode provided in an embodiment of the present application. The third scene mode corresponds to the situation where the overall scene of the image is relatively bright.

[0180] The multiple interpolation points include: a first interpolation point, the modified spline curve includes a first spline curve, the first spline curve is a linear spline curve passing through the first interpolation point, and the first spline curve is smoothly connected to the initial tone mapping curve at the first interpolation point; the output parameters of the modified spline curve include: TH1, the first coefficient MA0 of the first spline curve, and the second coefficient MB0.

[0181] The first spline curve: F(L) = MB0 × (L-TH1) 1 +MA0 (6)

[0182] TH1 is the first coordinate of the first interpolation point, and the first coordinate may be a horizontal coordinate.

[0183] The third scene mode corresponds to the situation where the overall scene of the image is relatively bright. At this time, the bright area is corrected to better show the bright details of the image. Specifically, based on the linear function (first spline curve), tone mapping is performed on the pixel area greater than the preset threshold (TH1), and the first spline curve is smoothly connected to the initial tone mapping curve at the first interpolation point to avoid the non-smooth tone mapping curve affecting the tone mapping effect. Specifically, tone mapping is performed on pixels greater than the first interpolation point using a first-order spline curve, which means that the points after TH1 are compressed using the compression rate of the first interpolation point to avoid excessive compression of pixels in the bright area, thereby improving the image brightness value. The acquisition process in the third scene mode is the third acquisition process. For details of the third acquisition process, please refer to Figure 8 The method shown or the case 1 described below.

[0184] like Figure 4d As shown, it is a schematic diagram of an exemplary modified spline curve corresponding to the fourth scene mode provided in an embodiment of the present application. The fourth scene mode corresponds to the situation where the overall scene of the image is relatively bright.

[0185] The multiple interpolation points include: a first interpolation point and a second interpolation point. In an optional case, there is an offset value Spline_strength between the second interpolation point and the initial tone mapping curve in the vertical coordinate direction. The modified spline curve includes: a first spline curve, which is a cubic spline curve passing through the first interpolation point and the second interpolation point, and the first spline curve is smoothly connected to the initial tone mapping curve at the first interpolation point; the output parameters of the modified spline curve include: TH1, TH2, and 4 coefficients MA1, MB1, MC1 and MD1 of the first spline curve.

[0186] The first spline curve: F(L) = MD1 × (L-TH1) 3 +MC1×(L-TH1) 2 +MB1×(L-TH1) 1 +MA1 (7)

[0187] Among them, TH1 is the first coordinate of the first interpolation point, TH2 is the first coordinate of the second interpolation point, the first coordinate may be, for example, a horizontal coordinate, and L is the independent variable of the function.

[0188] The fourth scene mode corresponds to the situation where the overall scene of the image is relatively bright. At this time, the bright area is corrected to better show the bright details of the image. Specifically, based on the cubic spline function (formula (7)), the pixel area between the first interpolation point and the second interpolation point is tone mapped, and the cubic spline function is smoothly connected to the initial tone mapping curve at the first interpolation point to avoid the non-smooth tone mapping curve affecting the tone mapping effect. Specifically, the brightness value of the pixel between the first interpolation point and the endpoint of the bright area pixel can be increased, and the brightness value between the first interpolation point and the endpoint of the bright area pixel can also be reduced. This can satisfy the function of increasing the brightness to achieve a more dazzling effect, and can also reduce the brightness to achieve functions such as enhancing the bright area details. In addition, increasing and decreasing is a nonlinear process, which can improve the contrast. The acquisition process in the fourth scene mode is the fourth acquisition process. For the detailed process of the fourth acquisition process, refer to Fig. 9 The method shown or the case 2 described below.

[0189] like Figure 4e As shown, it is a schematic diagram of an exemplary modified spline curve corresponding to the fifth scene mode provided in an embodiment of the present application. The fifth scene mode corresponds to the situation where the overall scene of the image is relatively bright.

[0190] The multiple interpolation points include: a first interpolation point, a second interpolation point, and a third interpolation point. The modified spline curve includes: a first spline curve and a second spline curve. The first spline curve is a cubic spline curve between the first interpolation point and the second interpolation point, and the second spline curve is a linear spline curve between the second interpolation point and the third interpolation point. The first spline curve is smoothly connected to the initial tone mapping curve at the first interpolation point, the second spline curve is smoothly connected to the initial tone mapping curve at the third interpolation point, and the first spline curve is smoothly connected to the second spline curve at the second interpolation point. The output parameters of the modified spline curve include: TH1, TH2, TH3, 4 coefficients MA1, MB1, MC1 and MD1 of the first spline curve, and 2 coefficients MA2 and MB2 of the second spline curve. Among them, TH1 is the first coordinate of the first interpolation point, TH2 is the first coordinate of the second interpolation point, and TH3 is the first coordinate of the third interpolation point. The first coordinate can be, for example, a horizontal coordinate.

[0191] The first spline curve: F(L) = MD1 × (L-TH1) 3 +MC1×(L-TH1) 2 +MB1×(L-TH1) 1 +MA1 (8)

[0192] Second spline curve: F(L)=MB2×(L-TH2) 1 +MA2 (9)

[0193] Among them, L is the independent variable of the function.

[0194] It should be understood that in the fifth scene mode, the modified spline curve includes a linear spline curve and a cubic spline curve, and the positions of the two spline curves are not limited. Figure 4e In the embodiment, the brightness of the pixel points in the pixel interval corresponding to the linear spline curve is greater than the brightness of the pixel points in the pixel interval corresponding to the cubic spline curve, that is, the brightness value of the first interpolation point < the brightness value of the second interpolation point < the brightness value of the third interpolation point; in an optional case, the brightness of the pixel points in the pixel interval corresponding to the linear spline curve may also be less than the brightness of the pixel points in the pixel interval corresponding to the cubic spline curve, in which case the brightness value of the first interpolation point > the brightness value of the second interpolation point > the brightness value of the third interpolation point.

[0195] The fifth scene mode corresponds to the situation where the overall scene of the image is relatively bright. At this time, the bright area is corrected to better show the bright details of the image. Specifically, based on the linear spline curve (formula (9)), the pixel area between the second interpolation point and the third interpolation point is tone mapped, and based on the cubic spline function between the first interpolation point and the second interpolation point, the linear spline function is smoothly connected with the initial tone mapping curve to avoid the influence of the non-smooth tone mapping curve on the tone mapping effect. Specifically, in this scene, the cubic spline function can be used between the first interpolation point and the second interpolation point to meet the increase and decrease of brightness of the initial tone mapping curve, and this increase and decrease are nonlinear. Tone mapping is performed based on a straight line between the second interpolation point and the third interpolation point, and more bits can be allocated to the pixels between the second interpolation point and the third interpolation point, thereby improving the details of the bright area. The acquisition process in the fifth scene mode is the fifth acquisition process. For the detailed process of the first acquisition process, refer to Fig.10 The method shown or the following situation 3.2.

[0196] 303. Perform tone mapping on the image to be processed according to a modified tone mapping curve, where the modified tone mapping curve includes at least a portion of the initial tone mapping curve and a modified spline curve.

[0197] It should be understood that the initial tone mapping curve can be any form of curve that can be used in the tone mapping process. Specifically, Figure 4f Several possible forms of the initial tone mapping curve are shown. Exemplarily, the initial tone mapping curve may be a normalized curve, that is, the coordinates of any point in the initial tone mapping curve are (first coordinate value, second coordinate value), the first coordinate value is between 0 and 1, and the second coordinate value is between 0 and 1. The following is an explanation of the normalized initial tone mapping curve, and it should be understood that the present application is not limited to this.

[0198] Figure 4fThe initial tone mapping curve shown by the solid line is also called the SLF tone mapping curve, which satisfies:

[0199]

[0200] Wherein, L represents the signal value before mapping, L' represents the signal value after mapping, in an optional case, the signal value before mapping is a linear signal value, and the signal value after mapping is a nonlinear signal value, p, a, m, b are all dynamic range mapping coefficients, 0≤L≤1, 0≤L'≤1. a is proportional to the dynamic range display capability of the display device and inversely proportional to the dynamic range of the target image. b is proportional to the minimum brightness display capability of the display device.

[0201] Figure 4f The initial tone mapping curve shown by the single dot-dashed line is a first-order polynomial tone mapping function, which satisfies:

[0202] L′=a×L+b, (11)

[0203] Wherein, L represents the signal value before mapping, L' represents the signal value after mapping, in an optional case, the signal value before mapping is a linear signal value, and the signal value after mapping is a nonlinear signal value, a and b are both dynamic range mapping coefficients, 0≤L≤1, 0≤L'≤1. a is proportional to the dynamic range display capability of the display device and inversely proportional to the dynamic range of the target image. b is proportional to the minimum brightness display capability of the display device.

[0204] Figure 4f The initial tone mapping curve shown by the dashed line is also called the sigmoidal tone mapping curve, which satisfies:

[0205]

[0206] Wherein, L represents the signal value before mapping, L' represents the signal value after mapping, in an optional case, the signal value before mapping is a linear signal value, and the signal value after mapping is a nonlinear signal value, c1, c2, c3, n are all dynamic range mapping coefficients, 0≤L≤1, 0≤L'≤1. c1, c2, c3, n are all related to the average brightness, maximum brightness, and minimum brightness of the target image, and are related to the minimum display brightness and maximum display brightness of the display device.

[0207] Figure 4f The initial tone mapping curve shown by the double-dotted line is also called the Bezier tone mapping curve, which satisfies:

[0208]

[0209] Wherein, L represents the signal value before mapping, L' represents the signal value after mapping, in an optional case, the signal value before mapping is a linear signal value, and the signal value after mapping is a nonlinear signal value, n, N, ρ are all dynamic range mapping coefficients, 0≤L≤1, 0≤L'≤1. n, N, ρ are related to the brightness distribution of the target image, and are also related to the average brightness of the target image.

[0210] Taking the SLF formula as an example, p and m can be determined by the overall brightness of the image. When the overall image is bright, the value of p is small and the value of m is large; when the overall image is dark, the value of p is large and the value of m is small. The value range of p can be 3 to 6. The value range of m can be 2 to 4, and m is usually 2.4.

[0211] It should be understood that the modified tone mapping curve can be a piecewise polynomial function. For example, the multiple piecewise functions of the modified tone mapping curve are smoothly connected. Specifically, when the pixel to be processed is located in the correction interval, the pixel to be processed is tone mapped according to the modified spline curve; when the pixel to be processed is not in the correction interval, the pixel to be processed is tone mapped according to the initial tone mapping curve. It should be understood that when determining the interval where the pixel to be processed is located, the pixel value of the pixel to be processed can be compared with the first coordinate values ​​of multiple interpolation points. For example, when the pixel to be processed is in RGB format, the maximum value of the three RGB components can be used as the pixel value of the pixel to be processed; when the pixel to be processed is in YUV format, the Y component can be used as the pixel value of the pixel to be processed.

[0212] Exemplarily, the correction interval includes a first correction interval, a second correction interval and a third correction interval, wherein the first correction interval corresponds to a first spline function, the second correction interval corresponds to a second spline function, and the third correction interval corresponds to a third spline function. Then, when the pixel to be processed is located in the first correction interval, tone mapping is performed on the pixel to be processed based on the first spline function; when the pixel to be processed is located in the second correction interval, tone mapping is performed on the pixel to be processed based on the second spline function; when the pixel to be processed is located in the third correction interval, tone mapping is performed on the pixel to be processed based on the third spline function; when the pixel to be processed is not in these three correction intervals, tone mapping is performed on the pixel to be processed based on the initial tone mapping curve.

[0213] Exemplarily, corresponding to the first scene mode mentioned above:

[0214] When the pixel to be processed is located between the origin and the first interpolation point, tone mapping is performed on the pixel to be processed based on formula (1);

[0215] When the pixel to be processed is located between the first interpolation point and the second interpolation point, tone mapping is performed on the pixel to be processed based on formula (2);

[0216] When the pixel to be processed is located between the second interpolation point and the third interpolation point, tone mapping is performed on the pixel to be processed based on formula (3);

[0217] When the pixel value of the pixel to be processed is greater than the first coordinate TH3 of the third interpolation point, tone mapping is performed on the pixel to be processed based on the initial tone mapping curve.

[0218] Corresponding to the second scene mode mentioned above:

[0219] When the pixel to be processed is located between the first interpolation point and the second interpolation point, tone mapping is performed on the pixel to be processed based on formula (4);

[0220] When the pixel to be processed is located between the second interpolation point and the third interpolation point, tone mapping is performed on the pixel to be processed based on formula (5);

[0221] When the pixel value of the pixel to be processed is greater than the first coordinate TH3 of the third interpolation point, tone mapping is performed on the pixel to be processed based on the initial tone mapping curve.

[0222] Corresponding to the third scene mode mentioned above:

[0223] When the pixel value of the pixel to be processed is less than the first coordinate TH1 of the first interpolation point, tone mapping is performed on the pixel to be processed based on the initial tone mapping curve;

[0224] When the pixel value of the pixel to be processed is greater than the first coordinate TH1 of the first interpolation point, tone mapping is performed on the pixel to be processed based on formula (6).

[0225] Corresponding to the fourth scene mode mentioned above:

[0226] When the pixel value of the pixel to be processed is less than the first coordinate TH1 of the first interpolation point, tone mapping is performed on the pixel to be processed based on the initial tone mapping curve;

[0227] When the pixel to be processed is located between the first interpolation point and the second interpolation point, tone mapping is performed on the pixel to be processed based on formula (7).

[0228] Corresponding to the fifth scene mode mentioned above:

[0229] When the pixel value of the pixel to be processed is less than the first coordinate TH1 of the first interpolation point, tone mapping is performed on the pixel to be processed based on the initial tone mapping curve;

[0230] When the pixel to be processed is located between the first interpolation point and the second interpolation point, tone mapping is performed on the pixel to be processed based on formula (8);

[0231] When the pixel to be processed is located between the second interpolation point and the third interpolation point, tone mapping is performed on the pixel to be processed based on formula (9).

[0232] The image processing method provided in the embodiment of the present application determines the scene mode applicable to the image to be processed according to the mode indicator value in the dynamic metadata, and reasonably and flexibly modifies the initial tone mapping curve, so as to adjust the image display effect for different brightness areas, enhance the brightness change details of the image, and bring a better visual experience to the user. By fine-tuning the tone mapping curve in various forms, a variety of display effects such as soft, vivid, bright, and dusk can be obtained.

[0233] The image processing method provided in the embodiment of the present application can adaptively modify any part of the curve according to the needs of the scene, so as to improve the brightness, improve the contrast, blur some information, etc. For example, the display effect of the details of the dark area can be improved by increasing the brightness of the dark area, or the contrast can be improved by reducing the brightness of the dark area; or more bits can be allocated to the highlight area to display the details of the bright area, and any part of the curve can be modified to achieve the required specific effect.

[0234] like Figure 5 As shown, it is a flow chart of a method for obtaining parameters of a modified spline curve provided in an embodiment of the present application.

[0235] The method includes:

[0236] 501. Determine first coordinates of at least one interpolation point according to feature information or dynamic metadata of an image to be processed;

[0237] Exemplarily, the at least one interpolation point may include only the first interpolation point, or include the first difference point and the second interpolation point, or include the first interpolation point, the second interpolation point and the third interpolation point. Exemplarily, in the third scene mode shown in the present application, only one interpolation point needs to be determined; in the fourth scene mode shown in the present application, two interpolation points need to be determined; in the first scene mode, the second scene mode and the fifth scene mode of the present application, three interpolation points need to be determined.

[0238] 502. Determine at least one second coordinate and at least one first-order derivative according to the first coordinate of the at least one interpolation point;

[0239] 503. Determine coefficients of a modified spline curve according to the first coordinate, the at least one second coordinate, and the at least one first-order derivative of the at least one interpolation point, where the modified spline curve includes at least one of a linear spline curve or a cubic spline curve;

[0240] The present application embodiment provides several exemplary situations for determining a modified spline curve:

[0241] Case 1: Contains one interpolation point;

[0242] At least one interpolation point includes a first interpolation point, the modified spline curve includes a first spline curve, the first spline curve is a linear spline curve, the first spline curve is a linear spline curve passing through the first interpolation point, the coefficients of the modified spline curve include a first coefficient MA0 and a second coefficient MB0, the linear spline curve is smoothly connected to the initial tone mapping curve at the first interpolation point, specifically, the first-order derivative of the linear spline curve at the first interpolation point is equal to the first-order derivative of the initial tone mapping curve at the first interpolation point. Correspondingly,

[0243] 501 specifically includes: determining a first coordinate TH1 of a first interpolation point according to data included in the dynamic metadata;

[0244] 502 specifically includes:

[0245] Calculate the second coordinate VA1 of the first interpolation point on the initial tone mapping curve and the first-order derivative GD1 of the initial tone mapping curve at the first interpolation point according to TH1;

[0246] 503 specifically includes:

[0247] Determine GD1 as the second coefficient MB0 of the first spline curve, and VA1 as the first coefficient MA0 of the first spline curve;

[0248] The first spline is obtained from MB0 and MA0.

[0249] The specific solution for situation 1 is as follows Figure 8 As shown in the scheme.

[0250] Case 2: The case with two interpolation points;

[0251] At least one interpolation point includes a first interpolation point and a second interpolation point, and the modified spline curve includes: a first spline curve, which is a cubic spline curve, and the second interpolation point can be one of the endpoints of the initial tone mapping curve. For example, in the fourth scene mode shown in the embodiment of the present application, the second interpolation point is the endpoint of the bright area of ​​the initial tone mapping curve, and the modified spline curve is smoothly connected to the initial tone mapping curve at the first interpolation point. Correspondingly,

[0252] 501 specifically includes:

[0253] The first coordinate TH1 of the first interpolation point and the first coordinate TH2 of the second interpolation point are determined according to the data contained in the dynamic metadata. For example, TH1=A, TH2=B, wherein A and B are constants contained in the dynamic metadata; or A and B are calculated according to the data contained in the dynamic metadata.

[0254] 502 specifically includes:

[0255] Calculate second coordinates VA1 and VA2 of the first interpolation point and the second interpolation point on the initial tone mapping curve according to TH1 and TH2 respectively;

[0256] Calculate the first-order derivative GD1 of the initial tone mapping curve at the first interpolation point and the first-order derivative GD2 at the second interpolation point according to TH1 and TH2;

[0257] 503 specifically includes:

[0258] Determine VA1 as the first coefficient MA1 of the first spline curve, and determine GD1 as the second coefficient MB1 of the first spline curve;

[0259] A system of equations consisting of two equations is constructed based on TH1, TH2, VA2 and GD2;

[0260] Analyze the equation group to obtain the third coefficient MC1 and the fourth coefficient MD1 of the first spline curve;

[0261] The first spline is obtained from MA1, MB1, MC1 and MD1.

[0262] Among them, the equation system containing two equations is constructed according to TH1, TH2, VA2 and GD2, specifically including:

[0263] The second coordinate of the second interpolation point on the first spline curve is made to be the sum of VA2 and S1, and the first equation is obtained, where S1 is the coordinate offset value included in the dynamic metadata;

[0264] The first-order derivative of the first spline curve at the second interpolation point is made to be the sum of GD2 and S2, and the second equation is obtained, where S2 is the slope offset value included in the dynamic metadata.

[0265] The specific solution for situation 2 is as follows Fig. 9 As shown in the scheme.

[0266] For the case of two interpolation points, an optional case is also included, the first interpolation point is a point close to the end point of the dark area, the modified spline curve includes: a first spline curve and a second spline curve, the first spline curve is a linear spline curve, the second spline curve is a cubic spline curve, the first spline curve is a straight line passing through the origin and the first interpolation point, the second spline curve passes through the first interpolation point and the second interpolation point, the linear spline curve and the cubic spline curve are smoothly connected at the first interpolation point, and the cubic spline curve and the initial tone mapping curve are smoothly connected at the second interpolation point. In this case, the dark area can be corrected.

[0267] Case 3: Case with three interpolation points;

[0268] The at least one interpolation point includes a first interpolation point, a second interpolation point, and a third interpolation point.

[0269] Case 3.1: The modified spline curve includes: one linear spline curve and two cubic spline curves (such as the first scene mode shown in the embodiment of the present application); correspondingly,

[0270] 501 specifically includes:

[0271] Determine a first coordinate TH1 of a first interpolation point according to at least one of a maximum value maxL of brightness of the image to be processed and an average value avgL of brightness of the image to be processed included in the dynamic metadata;

[0272] TH2=TH1+B;

[0273] TH3=TH2+C*TH2-D*CH2;

[0274] Wherein, TH2 is the first coordinate of the second interpolation point, TH3 is the first coordinate of the third interpolation point, B, C and D are constants, and in an optional case, B, C and D are constants included in the dynamic metadata;

[0275] The method further includes:

[0276] Obtaining a first-order derivative MB0 of the first spline curve according to an average value avgL of brightness of the image to be processed included in the dynamic metadata;

[0277] 502 specifically includes:

[0278] Calculate the second coordinate VA1 of the first interpolation point on the first spline curve according to TH1 and MB0;

[0279] Calculate a second coordinate VA3 of the third interpolation point on the initial tone mapping curve according to TH3;

[0280] Calculate a second coordinate VA2 of the second interpolation point on the first straight line, where the first straight line is the straight line where the first interpolation point and the third interpolation point are located;

[0281] Calculate the first-order derivative GD3 of the initial tone mapping curve at the third interpolation point;

[0282] 503 specifically includes:

[0283] Determine VA1 as the first coefficient MA1 of the second spline curve;

[0284] Determine MB0 as the second coefficient MB1 of the second spline curve;

[0285] Determine the sum of VA2 and an offset value as the first coefficient MA2 of the third spline curve, wherein the offset value is 0, a positive number, or a negative number;

[0286] It should be understood that the offset value may be transmitted to the tone mapping equipment via dynamic metadata, or may be defined in the program code.

[0287] Constructing an equation group including five equations according to TH1, TH2, TH3, VA2, VA3, GD3 and the derivative relationship between the second spline curve and the third spline curve at the second interpolation point;

[0288] The system of five equations is analyzed to obtain MC1, MD1, MB2, MC2, and MD2.

[0289] Among them, according to TH1, TH2, TH3, VA2, VA3, GD3 and the derivative relationship between the second spline curve and the third spline curve at the second interpolation point, an equation group containing five equations is constructed, specifically including:

[0290] The second coordinate of the second interpolation point on the second spline curve is VA2, and the first equation is obtained;

[0291] The second coordinate of the third interpolation point on the third spline curve is VA3, and the second equation is obtained;

[0292] The first-order derivative of the third spline curve at the third interpolation point is GD3, and a third-party formula is obtained;

[0293] Making the first-order derivatives of the second spline curve and the third spline curve at the second interpolation point equal to each other, thereby obtaining a fourth equation;

[0294] The second-order derivatives of the second spline curve and the third spline curve at the second interpolation point are made equal, thereby obtaining the fifth equation.

[0295] The specific solution for situation 3.1 is as follows Figure 6a and Figure 6b As shown in the scheme.

[0296] Case 3.2: The modified spline curve includes: a first spline curve and a second spline curve, the first spline curve is a cubic spline curve passing through the first interpolation point and the second interpolation point, and the second spline curve is a linear spline curve passing through the second interpolation point and the third interpolation point; the coefficients of the modified spline curve include: a first coefficient MA1, a second coefficient MB1, a third coefficient MC1, and a fourth coefficient MD1 of the first spline curve, and a first coefficient MA2 and a second coefficient MB2 of the second spline curve (such as the fifth scene mode shown in the embodiment of the present application); correspondingly,

[0297] 501 specifically includes:

[0298] The first coordinate TH1 of the first interpolation point, the first coordinate TH2 of the second interpolation point and the first coordinate TH3 of the third interpolation point are determined according to the data contained in the dynamic metadata. For example, TH1=A, TH2=B, TH3=C, wherein A, B and C are constants contained in the dynamic metadata; or A, B and C are calculated according to the data contained in the dynamic metadata.

[0299] 502 specifically includes:

[0300] Calculate the second coordinates VA1 and VA3 of the first interpolation point and the third interpolation point on the initial tone mapping curve according to TH1 and TH3, respectively, VA1 being the first coefficient MA1 of the first spline curve;

[0301] Calculate a second coordinate VA2 of the second interpolation point on the first straight line, where the first straight line is a straight line where the first interpolation point (TH1, VA1) and the third interpolation point (TH3, VA3) are located;

[0302] Calculate the first-order derivative GD1 of the initial tone mapping curve at the first interpolation point,

[0303] 503 specifically includes:

[0304] Determine VA1 as the first coefficient MA1 of the first spline

[0305] Determine GD1 as the second coefficient MB1 of the first spline;

[0306] In an optional case, the dynamic metadata includes a coordinate offset value, and the sum of VA2 and the coordinate offset value included in the dynamic metadata is determined as the first coefficient MA2 of the second spline curve;

[0307] Constructing an equation group including three equations according to TH1, TH2, TH3, VA2, VA3 and the derivative relationship between the first spline curve and the second spline curve at the second interpolation point;

[0308] Solving the system of three equations yields MC1, MD1, and MB2.

[0309] Among them, according to TH1, TH2, TH3, VA2, VA3 and the derivative relationship between the first spline curve and the second spline curve at the second interpolation point, an equation group containing three equations is constructed, specifically including:

[0310] The second coordinate of the second interpolation point on the first spline curve is VA2, and the first equation is obtained;

[0311] The second coordinate of the third interpolation point on the second spline curve is VA3, and the second equation is obtained;

[0312] The first-order derivatives of the first spline curve and the second spline curve at the second interpolation point are made equal to obtain a third-party formula.

[0313] The specific solution for situation 3.2 is as follows Fig.10 As shown in the scheme.

[0314] Case 3.3: The modified spline curve includes: a first spline curve and a second spline curve, the first spline curve is a cubic spline curve passing through the first interpolation point and the second interpolation point, and the second spline curve is a cubic spline curve passing through the second interpolation point and the third interpolation point; the coefficients of the modified spline curve include: the first coefficient MA1, the second coefficient MB1, the third coefficient MC1, and the fourth coefficient MD1 of the first spline curve, and the first coefficient MA2, the second coefficient MB2, the third coefficient MC2, and the fourth coefficient MD2 of the second spline curve (such as the second scene mode shown in the embodiment of the present application). Correspondingly,

[0315] 501 specifically includes:

[0316] The first coordinate TH1 of the first interpolation point, the first coordinate TH2 of the second interpolation point and the first coordinate TH3 of the third interpolation point are determined according to the data contained in the dynamic metadata. For example, TH1=A, TH2=B, TH3=C, wherein A, B and C are constants contained in the dynamic metadata; or A, B and C are calculated according to the data contained in the dynamic metadata.

[0317] 502 specifically includes:

[0318] Calculate second coordinates VA1 and VA3 of the first interpolation point and the third interpolation point on the initial tone mapping curve according to TH1 and TH3 respectively;

[0319] Calculate a second coordinate VA2 of the second interpolation point on the first straight line, where the first straight line is the straight line where the first interpolation point and the third interpolation point are located;

[0320] Calculate a first-order derivative GD1 of the initial tone mapping curve at a first interpolation point and a first-order derivative GD3 at a third interpolation point;

[0321] 503 specifically includes:

[0322] Determine VA1 as the first coefficient MA1 of the first spline

[0323] Determine GD1 as the second coefficient MB1 of the first spline;

[0324] Determine that the sum of VA2 and an offset value is the first coefficient MA2 of the second spline curve, where the offset value is 0, a positive number, or a negative number; optionally, the offset value may be included in dynamic metadata.

[0325] Constructing an equation group including five equations according to TH1, TH2, TH3, VA2, VA3, GD3 and the derivative relationship between the first spline curve and the second spline curve at the second interpolation point;

[0326] The system of five equations is analyzed to obtain MC1, MD1, MB2, MC2, and MD2.

[0327] Among them, according to TH1, TH2, TH3, VA2, VA3, GD3 and the derivative relationship between the first spline curve and the second spline curve at the second interpolation point, an equation group containing five equations is constructed, specifically including:

[0328] The second coordinate of the second interpolation point on the first spline curve is VA2, and the first equation is obtained;

[0329] The second coordinate of the third interpolation point on the second spline curve is VA3, and the second equation is obtained;

[0330] The first-order derivative of the second spline curve at the third interpolation point is GD3, and a third-party formula is obtained;

[0331] Making the first-order derivatives of the first spline curve and the second spline curve at the second interpolation point equal to each other, thereby obtaining a fourth equation;

[0332] The second-order derivatives of the first spline curve and the second spline curve at the second interpolation point are made equal, thereby obtaining the fifth equation.

[0333] The specific solution for situation 3.3 is as follows Figure 7 As shown in the scheme.

[0334] 504. Perform tone mapping on the image to be processed according to the modified tone mapping curve, where the modified tone mapping curve includes at least a portion of the initial tone mapping curve and a modified spline curve.

[0335] For 504, please refer to the description of 303, which will not be repeated here.

[0336] In an optional case, before 501, the method further includes:

[0337] The target scene mode applicable to the image to be processed is determined according to the mode indicator value in the dynamic metadata, and at least one interpolation point and the modified spline curve are related to the target scene mode. Specifically, the number of at least one interpolation point and the determination of the first-order spline curve and the third-order spline curve included in the modified spline curve are related to the mode indicator value. In other words, the parameters of the modified spline curve of the above-mentioned case 1, case 2, case 3.1, case 3.2 and case 3.3 are determined according to the mode indicator value.

[0338] like Figure 6a , which is a flow chart of a method for obtaining parameters of a modified spline curve provided in an embodiment of the present application. Figure 6a The method flow chart in FIG. 3 is a diagram of obtaining the parameters of the modified spline curve in the first scene mode and case 3.1. In this case, the modified spline curve includes a first spline curve, i.e., a first spline curve, and two third spline curves: a second spline curve and a third spline curve.

[0339] 601. Obtain a first coordinate TH1 of a first interpolation point and a first-order derivative MB0 of a first spline curve;

[0340]

[0341] Wherein, maxL and avgL are included in the dynamic metadata, maxL is the maximum value of the brightness of the image to be processed, and avgL is the average value of the brightness of the image to be processed; when the image to be processed is received, the first coordinate TH1 of the first interpolation point is calculated according to maxL included in the dynamic metadata of the image to be processed in combination with formula (14), and the slope MB0 of the first spline curve is calculated according to avgL included in the dynamic metadata in combination with formula (15), and the first spline curve is, for example, the curve shown in formula (1). In an optional case, the first coordinate of the first interpolation point can be calculated according to the average value avgL of the brightness of the image to be processed, and correspondingly, maxL in formula (14) can be replaced by avgL. In another optional case, the first coordinate of the first interpolation point can be calculated according to the maximum value maxL of the brightness of the image to be processed and the average value avgL of the brightness of the image to be processed, and correspondingly, maxL in formula (14) is replaced by a*maxL+b*avgL, wherein a and b are both constants, and 0≤a≤1, 0≤b≤1.

[0342] Correspondingly, in an optional case, the first derivative MB0 of the first spline curve can be calculated according to the maximum value maxL of the brightness of the image to be processed, and correspondingly, avgL in formula (15) can be replaced by maxL. In another optional case, the first coordinate of the first interpolation point can be calculated according to the maximum value maxL of the brightness of the image to be processed and the average value avgL of the brightness of the image to be processed, and correspondingly, avgL in formula (15) can be replaced by a*maxL+b*avgL, where a and b are both constants, and 0≤a≤1, 0≤b≤1.

[0343] PvalueL is the first threshold of the dark area pixel, PvalueH is the second threshold of the brightness value of the dark area pixel, and the first threshold is less than the second threshold. Exemplarily, the value of PvalueL can be 0.15, and the value of PvalueH can be 0.25; TPH is the third threshold of the bright area pixel, TPL is the fourth threshold of the bright area pixel, and exemplary, the value of TPH can be 0.6, and the value of TPL can be 0.1; AvL is the fifth threshold of the average pixel, and AvH is the sixth threshold of the average pixel, and exemplary, the value of AvL can be 0.15, and the value of AvH can be 0.65; SvalueL is the first slope threshold of the increase of the brightness value of the dark area pixel, and SvalueH is the second slope threshold of the increase of the brightness value of the dark area pixel, and the first slope threshold is less than the second slope threshold. Exemplarily, the value of SvalueL can be 1, and the value of SvalueH can be 1.2.

[0344] 602. Obtain first coordinates TH2 and TH3 of the second interpolation point and the third interpolation point;

[0345] TH2=TH1+B (16)

[0346] TH3=TH2+C*TH2-D*TH2 (17)

[0347] Wherein, B is the first offset value of the pixel point in the dark area transition area, C is the second offset value of the pixel point in the bright area, and D is the third offset value of the pixel point in the bright area; B, C and D are constants. In an optional case, the dynamic metadata includes the above B, C and D; B, C and D can also be directly defined in the relevant program code without being obtained from the dynamic metadata. Exemplarily, B can be 0.15, and C and D can be 0.5.

[0348] 603. Determine four coefficients MA1, MB1, MC1 and MD1 of the second spline curve and four coefficients MA2, MB2, MC2 and MD2 of the third spline curve according to TH1, TH2 and TH3;

[0349] It should be understood that, in this case, the modified spline curve includes two cubic spline curves: the second spline curve and the third spline curve. In an optional case, the modified spline curve may include only one cubic spline curve.

[0350] Specifically, Figure 6b As shown, step 603 includes:

[0351] 6031. Calculate second coordinates VA1, VA2 and VA3 of the first interpolation point, the second interpolation point and the third interpolation point respectively;

[0352] Specifically, 6031 includes:

[0353] 1) Calculate the second coordinate VA1 of the first interpolation point on the first-order spline curve;

[0354] VA1 = MB0 × TH1;

[0355] 2) Calculate the second coordinate VA3 of the third interpolation point on the initial tone mapping curve;

[0356] Exemplarily, the initial tone mapping curve is taken as the SLF tone mapping curve, as shown in formula (18):

[0357]

[0358] Substituting TH3 into formula (18) yields VA3=F(TH3); it should be understood that p, m, n, a, b, k1, k2, and k3 in formula (18) are all constants. The parameters of the initial tone mapping curve can be obtained from dynamic metadata or calculated in advance based on the image to be processed.

[0359] 3) Calculate the second coordinate VA2 of the second interpolation point on the first straight line, where the first straight line is the straight line where the first interpolation point and the third interpolation point are located;

[0360]

[0361] 6032. Determine VA1 as the first coefficient MA1 of the second spline curve, and determine the sum of VA2 and the offset value as the first coefficient MA2 of the third spline curve; optionally, the offset value can be 0, a positive number, or a negative number; the offset value can be obtained from dynamic metadata, and can also be defined in the relevant program code of the spline curve parameter acquisition process.

[0362] 6033. Determine the first-order derivative MB0 of the first spline curve as the second coefficient MB1 of the second spline curve;

[0363] 6034. Calculate the first-order derivative GD3 of the initial tone mapping curve at the third interpolation point;

[0364]

[0365] 6035. Construct a system of equations according to TH1, TH2, TH3, VA2, VA3, GD3 and the derivative relationship between the second spline curve and the third spline curve at the second interpolation point;

[0366] Exemplarily, the set of equations includes: a first equation, a second equation, a third equation, a fourth equation and a fifth equation, and 6035 specifically includes:

[0367] 1) The second coordinate of the second interpolation point on the second spline curve is VA2. In an optional case, VA2 = formula (19) + offset value, and the first equation is as shown in formula (21):

[0368] MD1×(TH2-TH1) 3 +MC1×(TH2-TH1) 2 +MB1×(TH2-TH1) 1 +MA1=VA2 (21)

[0369] 2) The second coordinate of the third interpolation point on the third spline curve is VA3, and the second equation is as shown in formula (22):

[0370] MD2×(TH3-TH1) 3 +MC2×(TH3-TH1) 2 +MB2×(TH3-TH1) 1 +MA2=VA3 (22)

[0371] 3) Make the first-order derivative of the third spline curve at the third interpolation point the above GD3, and obtain the third-party formula, as shown in formula (23):

[0372] 3×MD2×(TH3-TH2) 2 +2×MC2×(TH3-TH2) 1 +MB2=GD3 (23)

[0373] 4) Make the first-order derivatives of the second spline curve and the third spline curve at the second interpolation point equal, and obtain the fourth equation, as shown in formula (24):

[0374] 3×MD1×(TH2-TH1) 2 +2×MC1×(TH2-TH1) 1 +MB1=3×MD2×(TH2-TH2) 2 +2×MC2×(TH2-TH2) 1 +MB2 (24)

[0375] 5) Make the second-order derivatives of the second spline curve and the third spline curve at the second interpolation point equal, and obtain the fifth equation, as shown in formula (25):

[0376] 6×MD1×(TH2-TH1) 1 +2×MC1=6×MD2×(TH2-TH2) 1 +2×MC2 (25)

[0377] 6036. Analyze the system of equations consisting of the first equation, the second equation, the third equation, the fourth equation and the fifth equation to obtain MC1, MD1, MB2, MC2 and MD2.

[0378] MC1=(3*VA2-2.0*GD1*h1-3*VA1-MB2*h1) / (h1*h1) (26)

[0379] MD1=(h1*GD1+h1*MB2+2*VA1-2*VA2) / (h1*h1*h1) (27)

[0380]

[0381] MC2=MC1+3*MD1*h1(29)

[0382] MD2=-(VA3-VA2-h2*GD3+MC1*h2*h2+3*MD1*h1*h2*h2) / (2*h2*h2*h2)(30)

[0383] Among them, h1=TH2-TH1, h2=TH3-TH2.

[0384] 604. Determine a second spline curve according to MA1, MB1, MC1 and MD1, and determine a third spline curve according to MA2, MB2, MC2 and MD2.

[0385] Specifically, the second spline curve is shown in formula (2), and the third spline curve is shown in formula (3).

[0386] It should be understood that for ease of description, Figure 6a and Figure 6b The method is described in the form of steps, and although the order of the method is shown in the method flow chart, in some cases, the steps described may be performed in an order different from that shown here.

[0387] like Figure 7 , which is a flow chart of a method for obtaining parameters of a modified spline curve provided in an embodiment of the present application. Figure 7 The method flow chart in is a method for obtaining the parameters of the modified spline curve in the second scene mode. In this case, the modified spline curve includes two cubic spline curves: a first spline curve and a second spline curve. In an optional case, the modified spline curve may include only one cubic spline curve, and correspondingly, only two interpolation points need to be determined.

[0388] The method includes:

[0389] 701. Obtain first coordinates TH1, TH2 and TH3 of a first interpolation point, a second interpolation point and a third interpolation point, respectively;

[0390] Exemplarily, TH1=A, TH2=B, TH3=C, where A, B, and C are all constants. Assume that AB constitutes the first adjustment interval, BC constitutes the second adjustment interval, A is the minimum brightness value of the pixel point in the first adjustment interval, B is the maximum brightness value of the pixel point in the first adjustment interval, and C is the maximum brightness value of the pixel point in the second adjustment interval. <B<C。

[0391] In an optional case, A, B and C can be obtained from dynamic metadata. For example, the dynamic metadata includes the starting position L1 and the ending position L3 of the modified spline curve interval. Then, the above A, B and C can be calculated based on the data information in the dynamic metadata:

[0392] A=L1 / 255,C=L3 / 255,B=(A+C) / 2

[0393] In an optional case, the above-mentioned A, B and C are directly carried in the dynamic metadata.

[0394] In an optional case, A, B and C can be calculated by a formula similar to formula (14).

[0395] 702. Calculate second coordinates VA1 and VA3 of the first interpolation point and the third interpolation point on the initial tone mapping curve respectively;

[0396] Exemplarily, the initial tone mapping curve is as shown in the above formula (18). Substituting TH1 into formula (18) yields VA1, and substituting TH3 into formula (18) yields VA3:

[0397]

[0398] 703. Calculate a second coordinate VA2 of the second interpolation point on the first straight line, where the first straight line is a straight line where the first interpolation point and the third interpolation point are located;

[0399] In an optional case, the calculation formula of VA2 is the same as formula (19).

[0400] In an optional case, the dynamic metadata includes a coordinate offset value Spline_strength, and it should be understood that the coordinate offset value can be a positive number, a negative number, or 0. In this case, VA2 is the sum of the second coordinate of the second interpolation point on the first straight line and the coordinate offset value carried in the dynamic metadata:

[0401]

[0402] 704. Determine VA1 as the first coefficient MA1 of the first spline curve, and VA2 as the first coefficient MA2 of the second spline curve;

[0403] 705. Calculate a first-order derivative GD1 of the initial tone mapping curve at the first interpolation point and a first-order derivative GD3 at the third interpolation point, and determine GD1 as a second coefficient MB1 of the first spline curve;

[0404]

[0405] The calculation formula of GD3 is the same as formula (20).

[0406] 706. Constructing an equation group according to TH1, TH2, TH3, VA2, VA3, GD3 and the derivative relationship between the first spline curve and the second spline curve at the second interpolation point, the equation group including 5 equations;

[0407] Exemplarily, 706 specifically includes:

[0408] 1) The second coordinate of the second interpolation point on the first spline curve is VA2, and the first equation is obtained. The first equation is shown in formula (21);

[0409] 2) The second coordinate of the third interpolation point on the second spline curve is VA3, and the second equation is obtained, as shown in formula (22);

[0410] 3) The first-order derivative of the second spline curve at the third interpolation point is GD3, and the third-party formula is obtained, as shown in formula (23);

[0411] 4) Make the first-order derivatives of the first spline curve and the second spline curve at the second interpolation point equal, and obtain the fourth equation, as shown in formula (24);

[0412] 5) Make the second-order derivatives of the first spline curve and the second spline curve at the second interpolation point equal, and obtain the fifth equation, as shown in formula (25).

[0413] 707. Analyze the equation system consisting of the first equation, the second equation, the third equation, the fourth equation and the fifth equation to obtain MC1, MD1, MB2, MC2 and MD2;

[0414] The calculation results refer to formula (26) to formula (30) and will not be repeated here.

[0415] 708. Determine a first spline curve according to MA1, MB1, MC1 and MD1, and determine a second spline curve according to MA2, MB2, MC2 and MD2.

[0416] The first spline curve: F(L) = MD1 × (L-TH1) 3 +MC1×(L-TH1) 2 +MB1×(L-TH1) 1 +MA1(35)

[0417] Second spline curve: F(L)=MD2×(L-TH2) 3 +MC2×(L-TH2) 2 +MB2×(L-TH2) 1 +MA2(36)

[0418] In this scenario, the first coordinates of the first interpolation point, the second interpolation point, and the third interpolation point are obtained according to the dynamic metadata, the first interpolation point and the third interpolation point are all on the initial tone mapping curve, and the dynamic metadata may also include a ordinate offset value of the second interpolation point, which is used to indicate the offset value of the second interpolation point in the ordinate direction from the straight line defined by the first interpolation point and the third interpolation point.

[0419] like Figure 8 , which is a flow chart of a method for obtaining parameters of a modified spline curve provided in an embodiment of the present application. Figure 8 The method flow diagram in FIG. 1 is a method for obtaining the parameters of the modified spline curve in the third scene mode. In this case, the modified spline curve includes a linear spline curve: the first spline curve, the parameters to be determined include: the first coordinate TH1 of the first interpolation point, the first coefficient MA0 and the second coefficient MB0 of the first spline curve.

[0420] The method includes:

[0421] 801. Obtain a first coordinate TH1 of a first interpolation point;

[0422] In an alternative solution, TH1 = A;

[0423] Optionally, A may be a constant carried in the dynamic metadata, or A may be calculated based on data carried in the dynamic metadata. For example, if the dynamic metadata carries the correction interval starting value L1, then A=L1 / 255.

[0424] In an optional solution, TH1 can be calculated based on feature information of the image and a segmented formula similar to formula (14).

[0425] 802. Calculate a second coordinate VA1 of the first interpolation point on the initial tone mapping curve;

[0426] The initial tone mapping curve is shown in formula (18). Substituting TH1 into formula (18) yields VA1:

[0427]

[0428] 803. Calculate the first-order derivative GD1 of the initial tone mapping curve at the first interpolation point;

[0429]

[0430] 804. Determine GD1 as the second coefficient MB0 of the first spline curve, and VA1 as the first coefficient MA0 of the first spline curve.

[0431] Correspondingly, the obtained modified spline curve is as shown in formula (6).

[0432] like Fig. 9 , which is a flow chart of a method for obtaining parameters of a modified spline curve provided in an embodiment of the present application. Fig. 9 The method flow chart in FIG. 1 is a method for obtaining the parameters of the modified spline curve in the fourth scene mode. In this case, the modified spline curve includes a cubic spline curve: a first spline curve.

[0433] The method includes:

[0434] 901. Acquire first coordinates TH1 and TH2 of a first interpolation point and a second interpolation point, respectively, according to dynamic metadata;

[0435] Exemplarily, TH1=A, TH2=B, wherein A and B are constants. Exemplarily, B may be 1.0.

[0436] In an optional case, A and B can be directly obtained from dynamic metadata; in an optional case, A and B can be calculated based on the data information carried in the dynamic metadata. A and B can also be calculated based on the characteristics of the image to be processed and a segmented formula similar to formula (14).

[0437] 902. Calculate second coordinates VA1 and VA2 of the first interpolation point and the second interpolation point on the initial tone mapping curve respectively;

[0438] Exemplarily, the initial tone mapping curve is as shown in the above formula (18). Substituting TH1 into formula (18) obtains VA1, and substituting TH2 into formula (18) obtains VA2:

[0439]

[0440] 903. Calculate a first-order derivative GD1 of the initial tone mapping curve at a first interpolation point and a first-order derivative GD2 at a second interpolation point;

[0441]

[0442] 904. Determine VA1 as the first coefficient MA1 of the first spline curve, and determine GD1 as the second coefficient MB1 of the first spline curve;

[0443] 905. Construct a set of equations according to TH1, TH2, VA2 and GD2;

[0444] Exemplarily, 905 specifically includes:

[0445] 1) The second coordinate of the second interpolation point on the first spline curve is VA2'=VA2+S1, and the first equation is obtained as formula (43);

[0446] MD1×(TH2-TH1) 3 +MC1×(TH2-TH1) 2 +MB1×(TH2-TH1) 1 +MA1=VA2+S1 (43)

[0447] Wherein, S1 is the coordinate offset value included in the dynamic metadata. Optionally, S1 can be a positive value, a negative value or 0.

[0448] 2) The first-order derivative of the first spline curve at the second interpolation point is GD2'=GD2+S2, and the second equation is obtained as formula (44);

[0449] 3×MD1×(TH2TH1) 2 +2×MC1×(TH2-TH1) 1 +MB1=GD2+S2 (44)

[0450] Wherein, S2 is the slope offset value included in the dynamic metadata. Optionally, S2 can be a positive value, a negative value or 0.

[0451] 906. Analyze the system of equations consisting of the first equation and the second equation to obtain MC1 and MD1.

[0452] Correspondingly, the obtained modified spline curve is shown in formula (45):

[0453] F(L)=MD1×(L-TH1) 3 +MC1×(L-TH1) 2 +MB1×(L-TH1) 1 +MA1 (45)

[0454] like Fig.10 , which is a flow chart of a method for obtaining parameters of a modified spline curve provided in an embodiment of the present application. Fig.10 The method flow chart in FIG. 1 is a method for obtaining the parameters of the modified spline curve in the fifth scene mode. In this case, the modified spline curve includes a cubic spline curve and a linear spline curve. The parameters of the modified spline curve include: 4 coefficients MA1, MB1, MC1 and MD1 of the first spline curve, and 2 coefficients MA2 and MB2 of the second spline curve.

[0455] The method includes:

[0456] 1001. Obtain first coordinates TH1, TH2 and TH3 of a first interpolation point, a second interpolation point and a third interpolation point respectively;

[0457] Exemplarily, TH1=A, TH2=B, TH3=C, where A, B, and C are all constants. Assume that AB constitutes the first adjustment interval, BC constitutes the second adjustment interval, A is the minimum brightness value of the pixel point in the first adjustment interval, B is the maximum brightness value of the pixel point in the first adjustment interval or the minimum brightness value of the pixel point in the second adjustment interval, C is the maximum brightness value of the pixel point in the second adjustment interval, and A <B<C。

[0458] In an optional case, A, B and C can be obtained from dynamic metadata. For example, the dynamic metadata includes the starting position L1 and the ending position L3 of the modified spline curve interval. Then, the above A, B and C can be calculated based on the data information in the dynamic metadata:

[0459] A=L1 / 255,C=L3 / 255,B=(A+C) / 2

[0460] In an optional case, the above-mentioned A, B and C are directly carried in the dynamic metadata.

[0461] In an optional case, A, B and C can be calculated by a formula similar to formula (14).

[0462] 1002. Calculate second coordinates VA1 and VA3 of the first interpolation point and the third interpolation point on the initial tone mapping curve respectively;

[0463] Please refer to the description in Section 702 for details, which will not be repeated here.

[0464] 1003. Calculate a second coordinate VA2 of the second interpolation point on the first straight line, where the first straight line is a straight line where the first interpolation point (TH1, VA1) and the third interpolation point (TH3, VA3) are located;

[0465] In an optional case, the calculation formula of VA2 is the same as formula (19).

[0466] In an optional case, the dynamic metadata includes a coordinate offset value Spline_strength. It should be understood that the coordinate offset value can be a positive number, a negative number, or 0. When Spline_strength is 0, the value of VA2 can be calculated according to formula (19). At this time, VA2 is the sum of the second coordinate of the second interpolation point on the first straight line and the coordinate offset value carried in the dynamic metadata:

[0467]

[0468] It should be understood that VA1 <VA2<VA3。

[0469] 1004. Determine VA1 as the first coefficient MA1 of the first spline curve, and VA2 as the first coefficient MA2 of the second spline curve;

[0470] 1005. Calculate a first-order derivative GD1 of the initial tone mapping curve at the first interpolation point and a first-order derivative GD3 at the third interpolation point, and determine GD1 as a second coefficient MB1 of the first spline curve;

[0471]

[0472] 1006. Construct a system of equations according to TH1, TH2, TH3, VA2, VA3 and the derivative relationship between the first spline curve and the second spline curve at the second interpolation point;

[0473] Exemplarily, 1006 specifically includes:

[0474] 1) Let the second coordinate of the second interpolation point on the first spline be VA2, and obtain the first equation, as shown in formula (49);

[0475] MD1×(TH2-TH1) 3 +MC1×(TH2-TH1) 2 +MB1×(TH2-TH1) 1 +MA1=VA2 (49)

[0476] 2) The second coordinate of the third interpolation point on the second spline curve is VA3, and the second equation is obtained, as shown in formula (50);

[0477] MB2×(TH3-TH2) 1 +MA2=VA3 (50)

[0478] 3) Make the first-order derivatives of the first spline curve and the second spline curve at the second interpolation point equal, and obtain the third-party formula, as shown in formula (51);

[0479] 3×MD1×(TH2TH1) 2 +2×MC1×(TH2-TH1) 1 +MB1=MB2 (51)

[0480] 1007. Analyze the system of equations consisting of the first equation, the second equation and the third equation to obtain MC1, MD1 and MB2.

[0481] The obtained modified spline curves are shown in formula (52) and formula (53), where formula (52) is the first spline curve and formula (53) is the second spline curve.

[0482] F(L)=MD1×(L-TH1) 3 +MC1×(L-TH1) 2 +MB1×(L-TH1) 1 +MA1 (52)

[0483] F(L)=MB2×(L-TH2) 1 +MA2 (53)

[0484] It should be understood that for ease of description, Figures 7 to 10The method flow chart shown in the method describes the method in the form of steps. Although the order of the method is shown, in some cases, the steps described can be performed in an order different from that here.

[0485] like Fig.11 FIG. 1 is a block diagram of an exemplary image processing device 1100 provided in an embodiment of the present application. The image processing device 1100 includes:

[0486] Determining module 1110: used to execute steps 301, 302, 501, 502 and 503 in the method embodiment

[0487] Tone mapping module 1120: used to execute steps 303 and 504 in the method embodiment.

[0488] Specifically, the determination module is also used to execute Figure 6a , Figure 6b ,as well as Figures 7 to 10 Some or all of the steps in the method embodiment shown.

[0489] In an optional case, the image processing device 1100 further includes: an acquisition module 1130, configured to acquire the image to be processed and the corresponding dynamic metadata.

[0490] It should be understood that Figure 1 The image processing device 14 shown and Figure 2 The image processing devices shown can all be used to implement the embodiments of the present application. Figure 3 , Figure 5 , Figure 6a , Figure 6b as well as Figures 7 to 10 Any one of the method embodiments provided. Specifically, the image processing device 14 and Figure 2 The processor in the processor calls the program instructions stored in the memory to execute the following Figure 3 , Figure 5 , Figure 6a , Figure 6b as well as Figures 7 to 10 Part or all of the steps in the method shown in any embodiment.

[0491] The embodiment of the present application also provides a computer-readable storage medium, which stores instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes one or more steps in any of the above methods. If the components of the above signal processing device are implemented in the form of software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium.

[0492] Based on such understanding, the embodiments of the present application also provide a computer program product including instructions, which, when executed on a computer or processor, enables the computer or processor to execute any of the methods provided in the embodiments of the present application. The technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device or a processor therein to execute all or part of the steps of the methods described in the embodiments of the present application.

[0493] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0494] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for image processing, characterized in that: include: Determining first coordinates of at least one interpolation point according to dynamic metadata of the image to be processed; determining at least one second coordinate and at least one first-order derivative based on the first coordinate of the at least one interpolation point; Determining coefficients of a modified spline curve according to a first coordinate, at least one second coordinate, and at least one first-order derivative of the at least one interpolation point, wherein the modified spline curve includes at least one of a linear spline curve or a cubic spline curve; Performing tone-mapping on the image to be processed according to a modified tone mapping curve, wherein the modified tone mapping curve includes at least a portion of an initial tone mapping curve and the modified spline curve; The at least one interpolation point includes a first interpolation point, the modified spline curve includes a first spline curve, the first spline curve is a linear spline function passing through an origin and the first interpolation point, and the coefficient of the modified spline curve includes a first-order derivative MB0 of the first spline curve; Before determining at least one second coordinate and at least one first-order derivative according to the first coordinate of the at least one interpolation point, the method further includes: Obtaining a first-order derivative MB0 of the first spline curve according to at least one of a maximum value maxL of the brightness of the image to be processed and an average value avgL of the brightness of the image to be processed included in the dynamic metadata; The at least one interpolation point further includes: a second interpolation point and a third interpolation point, the modified spline curve further includes: a second spline curve and a third spline curve, the second spline curve is a cubic spline curve passing through the first interpolation point and the second interpolation point, and the third spline curve is a cubic spline curve passing through the second interpolation point and the third interpolation point; the coefficients of the modified spline curve further include: a first coefficient MA1, a second coefficient MB1, a third coefficient MC1, and a fourth coefficient MD1 of the second spline curve, and a first coefficient MA2, a second coefficient MB2, a third coefficient MC2, and a fourth coefficient MD2 of the third spline curve; The determining at least one second coordinate and at least one first-order derivative according to the first coordinate of the at least one interpolation point comprises: Calculating a second coordinate VA1 of the first interpolation point on the first spline curve according to the first coordinate TH1 and MB0 of the first interpolation point; Calculating a second coordinate VA3 of the third interpolation point on the initial tone mapping curve according to the first coordinate TH3 of the third interpolation point; Calculate the second coordinate VA2 of the second interpolation point on the first straight line according to the first coordinate TH2 of the second interpolation point, where the first straight line is the straight line where the first interpolation point and the third interpolation point are located; Calculate the first-order derivative GD3 of the initial tone mapping curve at the third interpolation point according to TH3; The step of determining the coefficient of the modified spline curve according to the first coordinate, the at least one second coordinate and the at least one first-order derivative of the at least one interpolation point comprises: Determine VA1 as the first coefficient MA1 of the second spline curve; Determine MB0 as the second coefficient MB1 of the second spline curve; Determine the sum of VA2 and an offset value as the first coefficient MA2 of the third spline curve, where the offset value is 0, an integer or a negative number; MC1, MD1, MB2, MC2 and MD2 were obtained based on TH1, TH2, TH3, VA2, VA3 and GD3; MC13*A2-**3*A1-MB2*h1(h1*1 MD1(h1*GD1+h1*MB2+2*VA1-2*VA2)(h1*h1*h1 MC2MC1+3*MD1*h1 MD2-(VA3-VA2-h2*GD3+MC1*h2*h2+3*MD1*h1*h2*h2) / (2 *h2*h2*h2) Among them, h1=TH2-TH1, h2=TH3-TH2.

2. The method according to claim 1, characterized in that: The second coordinate of the second interpolation point on the second spline curve is VA2; The second coordinate of the third interpolation point on the third spline curve is VA3; The first-order derivative of the third spline curve at the third interpolation point is GD3; The first-order derivatives of the second spline curve and the third spline curve at the second interpolation point are equal; The second-order derivatives of the second spline curve and the third spline curve at the second interpolation point are equal.

3. The method according to claim 1 or 2, characterized in that: The step of determining the first coordinate of at least one interpolation point according to the dynamic metadata of the image to be processed comprises: Determine a first coordinate TH1 of the first interpolation point according to at least one of a maximum value maxL of the brightness of the image to be processed and an average value avgL of the brightness of the image to be processed included in the dynamic metadata; TH2=TH1+B; TH3=TH2+C*TH2-D*CH2; Among them, B, C and D are constants.

4. The method according to any one of claims 1 to 3, characterized in that: The at least one interpolation point constitutes at least one correction interval, wherein one correction interval corresponds to one correction spline curve, and the tone mapping of the image to be processed according to the corrected tone mapping curve comprises: When the pixel point to be processed is located in the at least one correction interval, tone mapping is performed on the pixel point to be processed based on the correction spline curve; When the pixel to be processed is outside the at least one correction interval, tone mapping is performed on the pixel to be processed based on the initial tone mapping curve.

5. An image processing device, characterized in that: include: A determination module, used to determine a first coordinate of at least one interpolation point according to dynamic metadata of the image to be processed; The determination module is further used to determine at least one second coordinate and at least one first-order derivative according to the first coordinate of the at least one interpolation point; The determination module is further used to determine coefficients of a modified spline curve according to the first coordinate, the at least one second coordinate and the at least one first-order derivative of the at least one interpolation point, wherein the modified spline curve includes at least one of a linear spline curve or a cubic spline curve; a tone mapping module, configured to perform tone mapping on the image to be processed according to a modified tone mapping curve, wherein the modified tone mapping curve includes at least a portion of an initial tone mapping curve and the modified spline curve; The at least one interpolation point includes a first interpolation point, the modified spline curve includes a first spline curve, the first spline curve is a linear spline function passing through an origin and the first interpolation point, and the coefficients of the modified spline curve include: a first-order derivative MB0 of the first spline curve; The determining module is further used for: Obtaining a first-order derivative MB0 of the first spline curve according to at least one of a maximum value maxL of the brightness of the image to be processed and an average value avgL of the brightness of the image to be processed included in the dynamic metadata; The at least one interpolation point further includes: a second interpolation point and a third interpolation point, the modified spline curve further includes: a second spline curve and a third spline curve, the second spline curve is a cubic spline curve passing through the first interpolation point and the second interpolation point, and the third spline curve is a cubic spline curve passing through the second interpolation point and the third interpolation point; the coefficients of the modified spline curve further include: a first coefficient MA1, a second coefficient MB1, a third coefficient MC1, and a fourth coefficient MD1 of the second spline curve, and a first coefficient MA2, a second coefficient MB2, a third coefficient MC2, and a fourth coefficient MD2 of the third spline curve; the determination module is specifically used to: Calculating a second coordinate VA1 of the first interpolation point on the first spline curve according to the first coordinate TH1 and MB0 of the first interpolation point; Calculating a second coordinate VA3 of the third interpolation point on the initial tone mapping curve according to the first coordinate TH3 of the third interpolation point; Calculate the second coordinate VA2 of the second interpolation point on the first straight line according to the first coordinate TH2 of the second interpolation point, where the first straight line is the straight line where the first interpolation point and the third interpolation point are located; Calculate the first-order derivative GD3 of the initial tone mapping curve at the third interpolation point according to TH3; The determining module is specifically used for: Determine VA1 as the first coefficient MA1 of the second spline curve; Determine MB0 as the second coefficient MB1 of the second spline curve; Determine the sum of VA2 and an offset value as the first coefficient MA2 of the third spline curve, where the offset value is 0, a positive number, or a negative number; According to TH1, TH2, TH3, VA2, VA3, GD3, we get MC1, MD1, MB2, MC2 and MD2, MC1 = (3*VA2-2*GD1*h1-3*VA1-MB2*h1) / (h1*h1) MD1(h1*GD1+h1*MB2+2*VA1-2*VA2)(h1*h1*h1 MC2=MC1+3*MD1*h1 MD2-(VA3-VA2-h2*GD3+MC1*h2*h2+3*MD1*h1*h2*h2) / (2 *h2*h2*h2) Among them, h1=TH2-TH1, h2=TH3-TH2.

6. The device according to claim 5, characterized in that The determining module is specifically used for: Determine a first coordinate TH1 of the first interpolation point according to at least one of a maximum value maxL of the brightness of the image to be processed and an average value avgL of the brightness of the image to be processed included in the dynamic metadata; TH2=TH1+B; TH3=TH2+C*TH2-D*CH2; Among them, B, C and D are constants.

7. The device according to claim 5 or 6, characterized in that The at least one interpolation point constitutes at least one correction interval, wherein one correction interval corresponds to one correction spline curve, and the tone mapping module is specifically used for: When the pixel point to be processed is located in the at least one correction interval, tone mapping is performed on the pixel point to be processed based on the correction spline curve; When the pixel to be processed is outside the at least one correction interval, tone mapping is performed on the pixel to be processed based on the initial tone mapping curve.

8. An image processing device, characterized in that: Including processor and transmission interface, The transmission interface is used to receive or send image data; The processor is configured to call program instructions stored in a memory and execute the program instructions to perform the method according to any one of claims 1 to 4.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and when the program instructions are executed on a computer or a processor, the computer or the processor is caused to execute the method according to any one of claims 1 to 4.

Images