Method, apparatus and computer storage medium for tone mapping

Abstract

The application discloses a tone mapping method and device and a computer storage medium. The method comprises the following steps: acquiring first image data corresponding to a video image, and determining second image data corresponding to the video image based on a first mapping relationship and the first image data; wherein the first mapping relationship is used for electro-optical conversion and tone mapping; the video image is an HDR image; performing color gamut conversion based on the second image data to obtain third image data corresponding to the video image; and determining a tone-mapped image corresponding to the video image based on a second mapping relationship and the third image data; wherein the second mapping relationship is used for photoelectric conversion; and the tone-mapped image is an SDR image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a tone mapping method and device and computer storage medium. BACKGROUND

[0002] For common global tone mapping (TMO) methods of high dynamic range (HDR) video, such as a global TMO method of HDR video, a local TMO method of HDR video, and a deep learning-based TMO method of HDR video, there is a problem of low calculation efficiency and long processing time in the actual application process, which is difficult to meet the demand of real-time use of TMO. SUMMARY

[0003] The present application provides a tone mapping method, device and computer storage medium, which can improve the processing efficiency of the HDR video tone mapping process, thereby realizing real-time processing of tone mapping.

[0004] The technical solution of the present application is as follows:

[0005] In a first aspect, the present application provides a tone mapping method, which comprises:

[0006] obtaining first image data corresponding to a video image, and determining second image data corresponding to the video image based on a first mapping relationship and the first image data; wherein the first mapping relationship is used for electro-optical conversion and tone mapping; the video image is an HDR image;

[0007] performing gamut conversion based on the second image data to obtain third image data corresponding to the video image;

[0008] determining a tone-mapped image corresponding to the video image based on a second mapping relationship and the third image data; wherein the second mapping relationship is used for photoelectric conversion; the tone-mapped image is an SDR image.

[0009] In a second aspect, the present application provides a tone mapping device, which comprises:

[0010] an acquisition unit configured to acquire first image data corresponding to a video image; the video image is an HDR image;

[0011] a determination unit configured to determine second image data corresponding to the video image based on a first mapping relationship and the first image data; wherein the first mapping relationship is used for electro-optical conversion and tone mapping;

[0012] The acquisition unit is further configured to perform color gamut conversion based on the second image data to obtain third image data corresponding to the video image.

[0013] The determination unit is further configured to determine a tone-mapped image corresponding to the video image based on the second mapping relationship and the third image data, wherein the second mapping relationship is used for photoelectric conversion, and the tone-mapped image is an SDR image.

[0014] In a third aspect, an embodiment of the present application provides a computer device, which comprises a processor and a memory storing processor-executable instructions, when the instructions are executed by the processor, the method of the first aspect is implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a program, when the program is executed by a processor, the method of the first aspect is implemented.

[0016] An embodiment of the present application provides a tone-mapping method, device and computer storage medium, first image data corresponding to a video image is acquired, and second image data corresponding to the video image is determined based on a first mapping relationship and the first image data, wherein the first mapping relationship is used for electro-optical conversion and tone mapping; the video image is an HDR image; third image data corresponding to the video image is obtained by performing color gamut conversion based on the second image data; a tone-mapped image corresponding to the video image is determined based on a second mapping relationship and the third image data, wherein the second mapping relationship is used for photoelectric conversion; and the tone-mapped image is an SDR image. That is, in the embodiment of the present application, the first mapping relationship used for electro-optical conversion and tone mapping is pre-constructed, and the electro-optical conversion and tone mapping of the video image are completed, meanwhile, the second mapping relationship used for photoelectric conversion is pre-constructed, and the photoelectric conversion of the video image is completed, so that the electro-optical conversion, tone mapping and photoelectric conversion are completed directly through the first mapping relationship and the second mapping relationship in the tone mapping process of the HDR video, the corresponding output result is obtained, the processing efficiency of the tone mapping process of the HDR video is greatly improved, and real-time processing of the tone mapping is realized. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 Flowchart of the tone mapping method implementation process provided by the embodiment of the present application Figure 1 ;

[0018] Figure 2 Flowchart of the tone mapping method implementation process provided by the embodiment of the present application Figure 2 ;

[0019] Figure 3 Flowchart of the tone mapping method implementation process provided by the embodiment of the present application Figure 3 ;

[0020] Figure 4 Implementation flowchart of the tone mapping method proposed in the embodiments of the present application Figure 4 ;

[0021] Figure 5 Implementation flowchart of the tone mapping method proposed in the embodiments of the present application Figure 5 ;

[0022] Figure 6 Implementation flowchart of the tone mapping method proposed in the embodiments of the present application Figure 6 ;

[0023] Figure 7 Implementation framework diagram of the tone mapping method proposed in the embodiments of the present application

[0024] Figure 8 Implementation diagram of the tone mapping method proposed in the embodiments of the present application

[0025] Figure 9 Composition structure diagram of the tone mapping device proposed in the embodiments of the present application

[0026] Figure 10 Composition structure diagram of the computer device proposed in the embodiments of the present application DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It can be understood that the specific embodiments described herein are only used to explain the differences of the present application, and not to limit the present application. In addition, it should be noted that only the parts related to the differences of the present application are shown in the drawings for convenience of description.

[0028] In the global tone mapping (TMO) methods of high dynamic range (HDR) video, Filmic Tone Mapping is an early method of HDR video TMO, which fits a high order curve expression by comparing the results of the Standard Dynamic Range (SDR) video rendered by professional software used by artists, and uses the expression to render HDR video. Real-time Automatic TMO algorithm clamps the black and white pixel levels according to the histogram of the video frame, and uses a leaky integrator to reduce the changes of TMO parameters between different video frames, and reduce the flicker of the video. Academy Color Encoding System (ACES) method unifies different standards of HDR video and SDR video into a new color space, solves the color conversion problem between different devices, and uses a better mapping polynomial than Filmic Tone Mapping to complete tone mapping.

[0029] The global TMO methods of HDR video use the same processing means for each pixel of the video frame to perform tone mapping. The quality of the SDR video output by most existing methods still has room for improvement, and cannot adapt to different screen brightness and video brightness. The efficiency of existing algorithms also has optimization potential, and the efficiency of some algorithms that claim to be real-time is poor in actual use.

[0030] In the local TMO methods of HDR video, the Zonal Coherence TMO method uses the histogram of the video frame in the logarithmic domain to divide the video frame into regions of different brightness. In order to maintain the consistency of the region boundaries between frames in the sequence, the method calculates a sequence histogram for region segmentation again. The Motion Path Filter TMO method uses optical flow to calculate pixel-by-pixel motion estimation, and calculates spatial filtering and temporal filtering to divide the video frame into a base layer and a detail layer for processing, and enhances the local temporal coherence of tone mapping. The Bilateral Grid TMO method divides the video frame into grid-shaped regions, and uses a bilateral filter to process the image within the region to reduce the halo effect that other TMO methods are prone to cause, and improve the quality of the generated SDR video.

[0031] Local TMO methods for HDR video use different mappings for different regions of a video frame. This class of methods usually process pixel values by designing filters, which requires a large amount of computation and is difficult to meet the real-time use requirements.

[0032] In deep learning based TMO methods for HDR video, UnCL TMO method proposes an unsupervised tone mapping framework, uses domain and instance based contrastive learning loss for training, and uses the extracted set of brightness and contrast features to represent the similarity of HDR-LDR video pairs. It uses a spatial feature enhancement module to realize the information exchange and conversion of non-local regions, and uses a feature replacement module to improve the temporal coherence of the tone mapping results.

[0033] Deep learning based TMO methods for HDR video train deep neural networks to convert HDR video to SDR video, which requires a large amount of computation time and requires the use of high-performance GPUs, and cannot meet the real-time use requirements on most devices. Most of this class of methods directly apply the TMO method for HDR image to each frame of HDR video separately, which is easy to cause large differences between different consecutive frames, forming a flicker effect and affecting the quality of the output SDR video. Only a small part of this class of methods considers the inter-frame differences of HDR video, but the algorithm efficiency still cannot meet the real-time computation requirements.

[0034] In summary, the current common TMO methods for HDR video still need to be improved to output high-quality SDR video results while meeting the real-time TMO requirements.

[0035] To solve the above problems, an embodiment of the present application provides a tone mapping method, obtaining first image data corresponding to a video image, and determining second image data corresponding to the video image based on a first mapping relationship and the first image data; wherein the first mapping relationship is used for electro-optical conversion and tone mapping; the video image is an HDR image; performing gamut conversion based on the second image data to obtain third image data corresponding to the video image; determining a tone-mapped image corresponding to the video image based on a second mapping relationship and the third image data; wherein the second mapping relationship is used for photoelectric conversion; the tone-mapped image is an SDR image. That is, in the embodiment of the present application, the first mapping relationship used for electro-optical conversion and tone mapping can be used to complete the electro-optical conversion and tone mapping processing of the video image, and the second mapping relationship used for photoelectric conversion can be used to complete the photoelectric conversion of the video image, so that the electro-optical conversion, tone mapping and photoelectric conversion can be completed directly by the first mapping relationship and the second mapping relationship in the tone mapping process of the HDR video, and the corresponding output result is obtained, which greatly improves the processing efficiency of the tone mapping process of the HDR video, thereby realizing real-time processing of tone mapping.

[0036] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application.

[0037] An embodiment of the present application provides a tone mapping method, which can be applied to a tone mapping device or an electronic device, and can also be applied to any terminal including the tone mapping device or the electronic device.

[0038] Next, the tone mapping method proposed in the embodiments of the present application will be exemplarily described taking a tone mapping device as an example.

[0039] Further, in the embodiments of the present application, Figure 1 The tone mapping method proposed in the embodiments of the present application realizes the process as shown in the figure. Figure 1 As shown in the figure, Figure 1 The tone mapping method can include the following steps:

[0040] Step 101, obtaining first image data corresponding to a video image, and determining second image data corresponding to the video image based on a first mapping relationship and the first image data; wherein the first mapping relationship is used for electro-optical conversion and tone mapping; the video image is an HDR image.

[0041] In the embodiments of the present application, the first image data corresponding to the video image can be obtained first, and then the electro-optical conversion processing and the tone mapping processing of the first image can be completed based on the first image data and the first mapping relationship used for electro-optical conversion and tone mapping, so as to obtain the processed second image data.

[0042] It can be understood that in the embodiments of the present application, the video image can be any one or more frames of images in a video to be tone mapped, wherein the video to be tone mapped can be an HDR video, and correspondingly, the video image can be an HDR image.

[0043] Further, in the embodiments of the present application, the first image data corresponding to the video image can be used to determine the image information of the video image. The first image data can include, but is not limited to, one or more of pixel information, luminance information, and light intensity information, and the present application does not make specific limitations.

[0044] For example, in some embodiments, the first image data can be pixel values corresponding to the video image, or can be light intensity values corresponding to the video image.

[0045] It can be understood that in the embodiments of the present application, RGB data is generally used in the tone mapping process, therefore, the first image data corresponding to the video image can be image data in the RGB color space, and the video to be tone mapped is generally stored based on the YUV color space, that is, the video image is originally an image in the YUV color space, so color space conversion needs to be performed on the video image.

[0046] Further, in the embodiments of the present application, when the first image data corresponding to the video image is obtained, a loop unrolling side strategy can be selected based on the NEON instruction to perform color space conversion on the initial image data corresponding to the video image, so as to obtain the first image data.

[0047] That is, in the embodiments of the present application, for the video image in the video to be tone mapped, color space conversion can be performed first to convert the initial image data of the video image from the YUV color space to the RGB color space, that is, to obtain the first image data in the RGB color space.

[0048] It can be understood that in the embodiments of the present application, any color space conversion method can be selected to convert the initial image data of the video image from the YUV color space to the RGB color space, and the present application does not make specific limitations.

[0049] For example, in some embodiments, color space conversion can be performed through formula (1):

[0050]

[0051] wherein, is the image data in the RGB color space obtained by conversion, is the image data in the YUV color space before conversion.

[0052] It can be understood that in the embodiments of the present application, in the actual conversion process, since the floating point operation takes a long time, it can be selected to be converted into fixed point calculation. Among them, the fixed point calculation can select different precision, and higher precision can obtain better effect.

[0053] Exemplarily, in some embodiments, fixed point calculation with 10-bit precision can be selected to balance between video effect and operation efficiency, and the conversion process is as formula (2)-formula (4):

[0054] ir=1192*y+1719*v-956373+512>>10 (2)

[0055] ig=1192*y-192*u-666*v+362892+512>>10 (3)

[0056] ib=1192*y+2193*u-1199171+512>>10 (4)

[0057] Among them, ir, ig, ib are integer RGB data, and the coefficients are calculated according to the original floating point coefficients and the precision. Fixed point calculation needs to increase 2 10 / 2 for rounding to reduce the discontinuity of pixel color. In order to improve the effect of the output video, the precision of the fixed point calculation can be appropriately improved without overflow.

[0058] Further, in the embodiments of the present application, when the initial image data corresponding to the video image is color space converted, the NEON instruction can be selected to be used, and the loop unrolling strategy is used to complete the calculation processing. Among them, the loop unrolling strategy can be used to realize vectorized parallel calculation processing.

[0059] In the embodiments of the present application, the NEON instruction is a set of single instruction multiple data (Single Instruction Multiple Data, SIMD) instruction set for accelerating data processing. The NEON instruction set includes 16 128-bit (bit) quad-word registers Q0-Q15 and 32 64-bit double-word registers D0-D31, which can store and operate vector data. The data types supported by the NEON instruction include unsigned integer, signed integer, floating point, etc., which are suitable for various compute-intensive tasks.

[0060] In the embodiments of the present application, when the NEON instruction is used, it is necessary to check whether the target processor supports the NEON unit. The NEON instruction can be written as part of the code, which makes the NEON programming simpler and more efficient than using external hardware accelerators. Among them, the application scenarios of the NEON instruction are very wide, for example, in image processing, the NEON instruction can be used for RGB deinterleaving and interleaved storage, table lookup operation, edge processing, etc.

[0061] Further, in the embodiments of the present application, the loop unrolling strategy can be understood as a loop unrolling (Loop unrolling) technology, which is an optimization method for sacrificing the size of the program to speed up the execution speed of the program. Among them, loop unrolling is most commonly used to reduce loop overhead and provide instruction-level parallelism for processors with multiple functional units. It is also beneficial to the scheduling of the instruction pipeline.

[0062] Exemplarily, in some embodiments, assuming that the first image data and the initial image data are both pixel values, the NEON instruction set can use a vector register with a length of 128 bits, reading the storage of the pixel values of the 10-bit HDR video requires a short type variable with a length of 16 bits, and the process of fixed-point calculation requires an int type variable with a length of 32 bits to store the intermediate value of the operation to prevent overflow, therefore, in the embodiments of the present application, based on the loop unrolling strategy, 4 video frame pixel values stored in the int16x4_t type data can be read in parallel using a 64-bit register first, and then the data is widened to a vector int32x4_t composed of 4 32-bit groups at the beginning of the calculation process and stored in a 128-bit register. In order to reduce the time consumption of data storage, in the embodiments of the present application, at the end of the calculation process, the int32x4_t vector is compressed back to int16x4_t, and two such vectors are combined into a vector of int16x8_t type, and the data of each pixel R, G, B three channels are stored in an int16x8x3_t type vector array. In order to further reduce the time consumption of this step, in the embodiments of the present application, the loop unrolling strategy is used, and the above steps are repeated twice in each step in the loop, that is, 16 pixels are processed in each step in the loop. The loop unrolling strategy can reduce the redundant execution time caused by branch operations, and further improve the efficiency of the algorithm.

[0063] Further, in the embodiments of the present application, the first mapping relationship can be a mapping relationship pre-constructed for completing the electro-optical conversion and tone mapping.

[0064] It can be understood that in the embodiments of the present application, the first mapping relationship can be in the form of a set of input-output corresponding relationship table, wherein the input can be the unprocessed data before the electro-optical conversion and tone mapping, and the output can be the processed data after the electro-optical conversion and tone mapping.

[0065] It can be understood that in the embodiments of the present application, when determining the second image data corresponding to the video image based on the first mapping relationship and the first image data, based on the first mapping relationship, the first image data can be understood as the unprocessed data before the electro-optical conversion and tone mapping, and the second data can be understood as the processed data after the electro-optical conversion and tone mapping.

[0066] That is, in the embodiments of the present application, through the first mapping relationship constructed in advance, the electro-optical conversion and tone mapping processing of the first image data can be directly completed, and the processing result, i.e., the processed second image data, is obtained, so that the processing efficiency of the electro-optical conversion and tone mapping process can be improved.

[0067] Exemplarily, in some embodiments, the first mapping relationship can be as shown in Table 1, wherein the first mapping relationship can include the input-output corresponding relationship of the electro-optical conversion and tone mapping processing, wherein the input can be the pixel value before the electro-optical conversion and tone mapping, and the output can be the pixel value after the electro-optical conversion and tone mapping.

[0068] Table 1

[0069] Before processing After processing Pixel value A1 Pixel value a1 Pixel value B1 Pixel value b1 Pixel value C1 Pixel value c1 …… …… Pixel value N1 Pixel value n1

[0070] Exemplarily, in some embodiments, the first mapping relationship can be as shown in Table 2, wherein the first mapping relationship can include the input-output corresponding relationship of the electro-optical conversion and tone mapping processing, wherein the input can be the light intensity value before the electro-optical conversion and tone mapping, and the output can be the light intensity value after the electro-optical conversion and tone mapping.

[0071] Table 2

[0072]

[0073]

[0074] Exemplarily, in some embodiments, the first mapping relationship can be as shown in Table 3, wherein the first mapping relationship can include the input-output corresponding relationship of the electro-optical conversion and tone mapping processing, wherein the input can be the pixel value before the electro-optical conversion and tone mapping, and the output can be the light intensity value after the electro-optical conversion and tone mapping.

[0075] Table 3

[0076] Before processing After processing Pixel value A1 Light intensity value a1 Pixel value B1 Light intensity value b1 Pixel value C1 Light intensity value c1 …… …… Pixel value N1 Light intensity value n1

[0077] Exemplarily, in some embodiments, the first mapping relationship can be as shown in Table 4, where the first mapping relationship can include the input-output corresponding relationship of the electro-optical conversion and the tone mapping processing, where the input can be the light intensity value before the electro-optical conversion and the tone mapping, and the output can be the pixel value after the electro-optical conversion and the tone mapping.

[0078] Table 4

[0079] Before processing After processing Light intensity value A1 Pixel value a1 Light intensity value B1 Pixel value b1 Light intensity value C1 Pixel value c1 …… …… Light intensity value N1 Pixel value n1

[0080] That is, in the embodiments of the present application, the first mapping relationship can include the corresponding relationship between the pixel values or the light intensity values before and after the electro-optical conversion and the tone mapping processing. Accordingly, in the process of completing the electro-optical conversion and the tone mapping processing of the first image data based on the first mapping relationship, if the first image data is of a different data type from the input corresponding to the first mapping relationship, the data type conversion of the first image data can be selected to match the first mapping relationship.

[0081] Exemplarily, in some embodiments, assuming that the first mapping relationship is as shown in Table 2, i.e., the first mapping relationship includes the corresponding relationship between the light intensity values before and after the electro-optical conversion and the tone mapping processing, and the first image data is a pixel value, the first image data can be converted into a corresponding light intensity value first, and then the electro-optical conversion and the tone mapping processing of the first image data is completed through the first mapping relationship to determine the corresponding processing result.

[0082] Exemplarily, in some embodiments, assuming that the first mapping relationship is as shown in Table 1, i.e., the first mapping relationship includes the corresponding relationship between the pixel values before and after the electro-optical conversion and the tone mapping processing, and the first image data is a pixel value, the electro-optical conversion and the tone mapping processing of the first image data can be directly completed through the first mapping relationship to determine the corresponding processing result.

[0083] Further, in the embodiments of the present application, the second image data corresponding to the video image can be used to determine the image information obtained after the video image undergoes the electro-optical conversion and the tone mapping. Wherein, the second image data can include but is not limited to one or more of pixel information, luminance information, light intensity information, which is not specifically limited by the present application.

[0084] Exemplarily, in some embodiments, the second image data can be the pixel value corresponding to the video image after the electro-optical conversion and the tone mapping, or can be the light intensity value corresponding to the video image after the electro-optical conversion and the tone mapping.

[0085] Correspondingly, in the embodiments of the present application, after the electro-optical conversion and tone mapping processing of the first image data based on the first mapping relationship is completed, the output result can be directly determined as the second image data, or the output result can be selected to be converted in data type to obtain the second image data, for example, the output result is a light intensity value, and the second image data obtained through the conversion in data type is a pixel value.

[0086] Further, in the embodiments of the present application, when the second image data corresponding to the video image is determined based on the first mapping relationship and the first image data, the first mapped data corresponding to the video image can be first determined based on the first mapping relationship and the first image data; then the content luminance data and the display luminance data corresponding to the video image are obtained, and the first mapped data is adjusted based on the content luminance data and the display luminance data to obtain the second image data.

[0087] It should be noted that in the embodiments of the present application, after the electro-optical conversion and tone mapping processing of the first image data based on the first mapping relationship is completed, the processing result can be further mapped to improve the effect, for example, the processing result (first mapped data) is adjusted and corrected by the obtained content luminance data and display luminance data corresponding to the video image to obtain the final second image data, so as to obtain a better display effect.

[0088] It can be understood that in the embodiments of the present application, the content luminance data can represent the video content luminance corresponding to the video image, and the display luminance data can represent the luminance of the display screen. The adaptive adjustment of the tone mapping result combined with the content luminance data and the display luminance data can make the finally obtained second image data adapt to video content of different luminance and different screen luminance.

[0089] It can be understood that in the embodiments of the present application, the maximum content luminance (MCL) can be selected from the HDR10 video metadata, and then the content luminance data is further determined based on the maximum content luminance.

[0090] It can be understood that in the embodiments of the present application, the maximum screen luminance (MML) of the display screen can be detected, and then the display luminance data is further determined based on the maximum screen luminance.

[0091] Exemplarily, in some embodiments, the first mapped data can be adjusted by a method such as formula (5), that is, the result of adaptive adjustment of tone mapping input is:

[0092] Exemplarily, in some embodiments, the first mapped data can be adjusted by a method such as formula (5), that is, the result of adaptive adjustment of tone mapping input is:

[0093] wherein, output is the data before adjustment, MCL is the content luminance data, MML is the display luminance data, to_linear_scale = 10000 / MML is used to convert the normalized linear light intensity from the luminance scale of HDR10 to the luminance scale of the screen.

[0094] Further, in the embodiments of the present application, Figure 2 The implementation process of the tone mapping method proposed in the embodiments of the present application is shown in Figure 2 As Figure 2 As shown in the figure, before determining the second image data corresponding to the video image based on the first mapping relationship and the first image data, i.e. before step 101, the tone mapping method can further include the following steps:

[0095] Step 104, constructing the first mapping relationship.

[0096] In the embodiments of the present application, the first mapping relationship for performing electro-optical conversion and tone mapping can be constructed in advance.

[0097] It can be understood that, in the embodiments of the present application, in order to improve the efficiency of the electro-optical conversion and tone mapping process, the determination and construction of the first mapping relationship can be performed in advance.

[0098] It should be noted that, in the embodiments of the present application, for different electro-optical conversion methods and / or tone mapping methods, the first mapping relationship obtained by construction can be different, i.e. the first mapping relationship corresponds to the electro-optical conversion method and the tone mapping method adopted in the construction process. Among them, the present application does not make specific limitation on the electro-optical conversion method and / or the tone mapping method.

[0099] Exemplarily, in some embodiments, in the process of constructing the first mapping relationship, the electro-optical conversion method adopted includes but is not limited to the electro-optical conversion method based on the Electro-Optical Transfer Function (EOTF), and the tone mapping method adopted includes but is not limited to the tone mapping based on the S-shaped curve.

[0100] It can be understood that, in the embodiments of the present application, the TMO curve can also be adjusted, and other TMO curves other than the S-shaped curve can be selected to adapt to other tone mapping algorithms, and the effect of the SDR video result can be adjusted according to the effect.

[0101] It can be understood that in the embodiments of the present application, the data length corresponding to the first mapping relationship can be set and limited in the process of constructing the first mapping relationship. For example, the data length corresponding to the first mapping relationship can be 1024 bits, that is, an input-output corresponding relationship table (the first mapping relationship) with a length of 1024 bits is constructed in advance.

[0102] Further, in the embodiments of the present application, Figure 3 The implementation process of the tone mapping method proposed in the embodiments of the present application is shown in Figure 3 As shown in Figure 4 The tone mapping method can further include the following steps:

[0103] Step 1041, obtaining first pre-processing image data.

[0104] In the embodiments of the present application, in the process of constructing the first mapping relationship, the image data before the electro-optical conversion and tone mapping processing, that is, the first pre-processing image data, can be obtained first.

[0105] Further, in the embodiments of the present application, the first pre-processing image data can include but is not limited to one or more of pixel information, brightness information, and light intensity information, which is not limited in the present application.

[0106] Exemplarily, in some embodiments, the first pre-processing image data can be the pixel value before the electro-optical conversion and tone mapping processing, or the light intensity value before the electro-optical conversion and tone mapping processing.

[0107] Step 1042, performing electro-optical conversion based on the first pre-processing image data to obtain first linear light intensity data corresponding to the first pre-processing image data; wherein the first linear light intensity data is HDR light intensity data.

[0108] In the embodiments of the present application, after obtaining the first pre-processing image data, electro-optical conversion can be performed based on the first pre-processing image data to obtain first linear light intensity data corresponding to the first pre-processing image data.

[0109] In the embodiments of the present application, when making pictures or videos, the acquisition device takes linear scene light as input and converts it into non-linear picture or video signals to store more low-brightness information sensitive to human eyes. Therefore, before tone mapping, the non-linear video signal needs to be converted into a linear light signal, that is, electro-optical conversion processing needs to be performed first.

[0110] Further, in the embodiments of the present application, in the process of electro-optical conversion, the corresponding non-linear light intensity data can be determined based on the first pre-processing image data, and then the non-linear light intensity data is converted into corresponding first linear light intensity data.

[0111] It can be understood that in the embodiments of the present application, the first linear light intensity data can be HDR light intensity data.

[0112] Further, in the embodiments of the present application, any electro-optical conversion method can be selected to complete the electro-optical conversion processing of the first pre-processing image data, which is not specifically limited in the present application.

[0113] Exemplarily, in some embodiments, an electro-optical conversion method based on an electro-optical conversion function can be selected, for example, using the perceptual quantizer curve (Perceptual Quantilzer) specified in the Society of Motion Picture and Television Engineers (SMPTE) Recommendation ST2084 as the electro-optical transfer function, as formula (6):

[0114]

[0115] wherein nx represents the input non-linear light intensity, such as the first pre-processing image data, represents the output linear light intensity, such as the first linear light intensity, FLT_MIN represents the minimum representable floating point number. The specific values of other parameters specified in ST2084 are shown in Table 5 as follows:

[0116] Table 5

[0117]

[0118]

[0119] Step 1043, performing tone mapping based on the first linear light intensity data to obtain first post-processing image data corresponding to the first pre-processing image data; wherein the first post-processing image data is SDR image data.

[0120] In the embodiments of the present application, after performing electro-optical conversion based on the first pre-processing image data to obtain the first linear light intensity data corresponding to the first pre-processing image data, further tone mapping can be performed based on the first linear light intensity data, so that the first post-processing image data corresponding to the first pre-processing image data can be obtained.

[0121] In the embodiments of the present application, in order to compress the data in the high dynamic range into the standard dynamic range, the image needs to be tone mapped.

[0122] It can be understood that in the embodiments of the present application, the first post-processing image data can be SDR light intensity data.

[0123] Further, in the embodiments of the present application, any tone mapping method can be selected to complete the tone mapping processing of the first linear light intensity data, which is not specifically limited in the present application.

[0124] Exemplarily, in some embodiments, a global tone mapping method based on an S-shaped curve can be selected, such as formula (7):

[0125]

[0126] wherein x is the input HDR light intensity, i.e., the first linear light intensity data, f(x) is the output SDR light intensity, i.e., the first processed image data, and a, b, c, d, e, and f are parameters of the S-shaped curve. Among them, a controls the transition intensity from the shoulder to the middle of the curve, and a larger value means a faster transition; b controls the linearity of the S-shaped curve, and a larger value means that the curve is closer to linear; c controls the linear angle of the S-shaped curve, and a smaller value means that the rising angle of the curve is steeper; d controls the transition intensity from the front to the middle of the S-shaped curve, and a larger value means a stronger and faster transition; e and f jointly control the shape of the front segment of the curve.

[0127] In step 1044, a first mapping relationship is constructed based on the first pre-processing image data and the first post-processing image data.

[0128] In the embodiments of the present application, after the tone mapping based on the first linear light intensity data is performed to obtain the first post-processing image data corresponding to the first pre-processing image data, the first mapping relationship can be further constructed based on the first pre-processing image data and the first post-processing image data.

[0129] It can be understood that in the embodiments of the present application, after the electro-optical conversion and tone mapping of the image data are respectively completed to obtain the corresponding processing results, the corresponding relationship between the pre-processing data and the post-processing data, i.e., the corresponding relationship between the first pre-processing image data and the first post-processing image data, can be established, so as to complete the construction of the first mapping relationship based on the corresponding relationship.

[0130] That is, in the embodiments of the present application, in order to improve the efficiency of the electro-optical conversion and tone mapping process, the first mapping relationship for performing the electro-optical conversion and tone mapping can be pre-constructed, and the pre-constructed first mapping relationship can be called in subsequent processing. For example, the ST2084 electro-optical conversion and the S-shaped curve are pre-calculated to generate an input-output corresponding relationship table (first mapping relationship) with a length of 1024, wherein the value of the i-th item in the input-output corresponding relationship table corresponds to the processing result of the input with a pixel value of i after the electro-optical conversion and tone mapping.

[0131] Further, in the embodiments of the present application, Figure 4 The tone mapping method implementation process of the embodiments of the present application Figure 4 For example, Before processingAs shown, after determining the second image data corresponding to the video image based on the first mapping relationship and the first image data, i.e., after step 101, the tone mapping method can further include the following steps:

[0132] Step 105, storing the second image data based on the NEON instruction.

[0133] In the embodiments of the present application, after determining the second image data corresponding to the video image based on the first mapping relationship and the first image data, the second image data can also be stored based on the NEON instruction.

[0134] Exemplarily, in some embodiments, the output result (second image data) of the electro-optical conversion and tone mapping obtained based on the first mapping relationship can be stored in a vector array of type int16x8x3_t, and the result is saved using the NEON storage instruction, which further realizes the parallelization of program running and improves the program efficiency.

[0135] Step 102, performing color gamut conversion based on the second image data to obtain third image data corresponding to the video image.

[0136] In the embodiments of the present application, after obtaining the first image data corresponding to the video image and determining the second image data corresponding to the video image based on the first mapping relationship and the first image data, the second image data can be further color gamut converted, so that the third image data corresponding to the video image can be obtained.

[0137] It should be noted that, in the embodiments of the present application, color gamut conversion is used for wide color gamut to narrow color gamut compatibility, because HDR10 video uses the BT2020 color space recommended by the International Telecommunication Union, and SDR video usually uses the BT709 color gamut.

[0138] It can be understood that, in the embodiments of the present application, any color gamut conversion method can be selected to convert the second image data of the wide color gamut to the third image data of the narrow color gamut, which is not limited in the present application.

[0139] Exemplarily, in some embodiments, color gamut conversion can be performed by formula (8):

[0140]

[0141] wherein, is the second image data before conversion, is the third image data obtained after conversion.

[0142] It can be understood that in the embodiments of the present application, in the actual conversion process, due to the long time consumption of floating point operation, it can be selected to be converted into fixed point calculation. Among them, the fixed point calculation can select different precision, and higher precision can obtain better effect.

[0143] Exemplarily, in some embodiments, fixed point calculation with 10-bit precision can be selected to balance between video effect and operation efficiency, and the conversion process is as formula (9)-formula (11):

[0144] ir′=1700*ir-602*ig-75*ib+512>>10 (9)

[0145] ig′=-128*ir+1160*ig-8*ib+512>>10 (10)

[0146] ib′=-19*ir-103*ig+1146*ib+512>>10 (11)

[0147] Among them, ir', ig', ib' are integer RGB data, and the precision of fixed point calculation can be appropriately improved to improve the display effect of the output video without overflow.

[0148] Further, in the embodiments of the present application, when the color gamut conversion is performed based on the second image data to obtain the third image data corresponding to the video image, the NEON instruction can be selected, and the loop unrolling strategy is used to complete the calculation processing. Among them, the loop unrolling strategy can be used to realize vectorized parallel calculation processing.

[0149] That is, in the embodiments of the present application, in the process of color gamut conversion processing, the SIMD instruction of the NEON instruction set can be used for parallelization acceleration.

[0150] Exemplarily, in some embodiments, in the process of color gamut conversion processing, assuming that the second image data is pixel value, the content of 24 pixels can be selected for processing, which contains 3 times of repeated loop unrolling processing, each time contains two groups of RGB data of int16x4_t type, which is widened to int32x4_t in the calculation process, and is shortened and merged to int16x8x3_t vector array type for storage after the calculation is completed.

[0151] Step 103, determining a tone-mapped image corresponding to the video image based on the second mapping relationship and the third image data; wherein the second mapping relationship is used for photoelectric conversion; the tone-mapped image is an SDR image.

[0152] In the embodiments of the present application, after the color gamut conversion is performed based on the second image data to obtain third image data corresponding to the video image, the second mapping relationship and the third image data can be used to determine a tone-mapped image corresponding to the video image.

[0153] It can be understood that, in the embodiments of the present application, after the video image is sequentially subjected to electro-optical conversion, tone mapping and photoelectric conversion, and the corresponding tone-mapped image data is obtained, the tone mapping of the video image is completed, and the finally obtained tone-mapped image can be an SDR image.

[0154] Further, in the embodiments of the present application, the second mapping relationship can be a mapping relationship pre-constructed for completing photoelectric conversion. After the electro-optical conversion and the tone mapping are completed, the linear light data (such as the third image data) needs to be converted again into the nonlinear light data (second mapped data) stored in the video file.

[0155] It can be understood that, in the embodiments of the present application, the second mapping relationship can be in the form of a set of input-output corresponding relationship tables, wherein the input can be the unprocessed data before photoelectric conversion, and the output can be the processed data after photoelectric conversion.

[0156] It can be understood that, in the embodiments of the present application, when the tone-mapped image corresponding to the video image is determined based on the second mapping relationship and the third image data, the third image data can be understood as the unprocessed data before photoelectric conversion based on the second mapping relationship, and the tone-mapped image data (second mapped data) can be understood as the processed data after photoelectric conversion.

[0157] That is, in the embodiments of the present application, through the pre-constructed second mapping relationship, the photoelectric conversion processing of the third image data can be directly completed to obtain the processing result, so that the processing efficiency of the second conversion process can be improved.

[0158] Exemplarily, in some embodiments, the second mapping relationship can be as shown in Table 6, wherein the second mapping relationship can include the input-output corresponding relationship of the photoelectric conversion processing, wherein the input can be the pixel value before photoelectric conversion, and the output can be the pixel value after photoelectric conversion.

[0159] Table 6

[0160] After processing Pixel value A2 Pixel value a2 Pixel value B2 Pixel value b2 Pixel value C2 Pixel value c2 Pixel value N2 …… …… Pixel value n2 Before processing

[0161] Exemplarily, in some embodiments, the second mapping relationship can be as shown in Table 7, wherein the second mapping relationship can include the input-output corresponding relationship of the photoelectric conversion processing, wherein the input can be the light intensity value before photoelectric conversion, and the output can be the light intensity value after photoelectric conversion.

[0162] Table 7

[0163] After processing Light intensity value A2 Light intensity value a2 Light intensity value B2 Light intensity value b2 Light intensity value C2 Light intensity value c2 Light intensity value N2 …… …… Light intensity value n2 Before processing

[0164] Exemplarily, in some embodiments, the second mapping relationship can be as shown in Table 8, where the second mapping relationship can include the input-output corresponding relationship of the photoelectric conversion processing, where the input can be the pixel value before photoelectric conversion, and the output can be the light intensity value after photoelectric conversion.

[0165] Table 8

[0166] After processing Pixel value A2 Light intensity value a2 Pixel value B2 Light intensity value b2 Pixel value C2 Light intensity value c2 Pixel value N2 …… …… Light intensity value n2 Before processing

[0167] Exemplarily, in some embodiments, the second mapping relationship can be as shown in Table 9, where the second mapping relationship can include the input-output corresponding relationship of the photoelectric conversion processing, where the input can be the light intensity value before photoelectric conversion, and the output can be the pixel value after photoelectric conversion.

[0168] Table 9

[0169] After processing Light intensity value A2 Pixel value a2 Light intensity value B2 Pixel value b2 Light intensity value C2 Pixel value c2 Light intensity value N2 …… …… Pixel value n2 Figure 5

[0170] That is, in the embodiments of the present application, the second mapping relationship can include the corresponding relationship between the pixel values or the light intensity values before and after the photoelectric conversion processing. Accordingly, in the process of completing the photoelectric conversion processing of the second image data based on the second mapping relationship, if the third image data is different from the data type of the input corresponding to the second mapping relationship, the data type conversion of the third image data can be selected to match the second mapping relationship.

[0171] Exemplarily, in some embodiments, assuming that the second mapping relationship is as shown in Table 7, that is, the second mapping relationship includes the corresponding relationship between the light intensity values before and after the photoelectric conversion processing, and the third image data is a pixel value, the third image data can be converted into a corresponding light intensity value first, and then the photoelectric conversion processing of the third image data is completed through the second mapping relationship to determine the corresponding processing result.

[0172] Exemplarily, in some embodiments, assuming that the second mapping relationship is as shown in Table 6, that is, the second mapping relationship includes the corresponding relationship between the pixel values before and after the photoelectric conversion processing, and the third image data is a pixel value, the photoelectric conversion processing of the third image data can be directly completed through the second mapping relationship through the second mapping relationship to determine the corresponding processing result.

[0173] Further, in the embodiments of the present application, after determining the tone-mapped image corresponding to the video image based on the second mapping relationship and the third image data, the third image data can be enlarged according to the length parameter corresponding to the second mapping relationship to obtain enlarged data; then the second mapping data is determined based on the second mapping relationship and the enlarged data; and finally, the tone-mapped image is determined based on the second mapping data.

[0174] It should be noted that in the embodiments of the present application, the length parameter corresponding to the second mapping relationship can represent the data length corresponding to the second mapping relationship. If the second mapping relationship maintains the same data length (length parameter) as the first mapping relationship, it may not meet the precision requirement that can make the output video have a better display effect, therefore, the data length corresponding to the second mapping relationship can be greater than the data length corresponding to the first mapping relationship. For example, the data length corresponding to the first mapping relationship can be 1024 bits, and the data length corresponding to the second mapping relationship can be 8192 bits.

[0175] It can be understood that in the embodiments of the present application, in the process of photoelectric conversion of the third image data based on the second mapping relationship, the third image data can be enlarged according to the length parameter corresponding to the second mapping relationship, so that the enlarged data can match the data length corresponding to the second mapping relationship.

[0176] For example, in some embodiments, assuming that the length parameter corresponding to the second mapping relationship is 8192 bits, the normalized light intensity (third image data) output after tone mapping can be selected to be enlarged to the range of 8192 bits for subsequent photoelectric conversion calculation.

[0177] Further, in the embodiments of the present application, after determining the second mapping data corresponding to the third image data based on the second mapping relationship, the second mapping data can be selected to be stored based on the NEON instruction.

[0178] For example, in some embodiments, the output result (second mapping data) of photoelectric conversion obtained based on the second mapping relationship can be selected to be converted into uint8x8x3_t variable data parallelization storage using the NEON instruction, so as to improve the running efficiency.

[0179] It can be understood that in the embodiments of the present application, the video image can be any one or more images in the video to be tone-mapped. After tone mapping of the video image is completed and the corresponding tone-mapped image is obtained, the tone-mapped video corresponding to the video to be tone-mapped can be further obtained, and finally the tone mapping of the HDR video is completed, realizing the conversion of the HDR video into a high-quality SDR video.

[0180] Further, in the embodiments of the present application, Figure 5 A flowchart of the implementation of the tone mapping method proposed in the embodiments of the present application is shown in Figure 5 As shown in Figure 6 Before determining the tone-mapped image corresponding to the video image based on the second mapping relationship and the third image data, i.e., before step 103, the tone mapping method can further include the following steps:

[0181] Step 107, constructing the second mapping relationship.

[0182] In the embodiments of the present application, the second mapping relationship for photoelectric conversion can be constructed in advance.

[0183] It can be understood that, in the embodiments of the present application, in order to improve the efficiency of the second photoelectric conversion process, the determination and construction of the second mapping relationship can be performed in advance.

[0184] It should be noted that, in the embodiments of the present application, for different photoelectric conversion methods, the second mapping relationship obtained by construction can be different, i.e., the second mapping relationship corresponds to the photoelectric conversion method used in the construction process. In the present application, the photoelectric conversion method is not limited.

[0185] Exemplarily, in some embodiments, in the process of constructing the second mapping relationship, the photoelectric conversion method used includes but is not limited to a photoelectric conversion method based on a photoelectric conversion function.

[0186] It can be understood that, in the embodiments of the present application, in the process of constructing the second mapping relationship, the data length corresponding to the second mapping relationship can be set and limited, for example, the data length corresponding to the second mapping relationship can be 8192 bits, i.e., an input-output corresponding relationship table (second mapping relationship) with a length of 8192 bits is constructed in advance.

[0187] Further, in the embodiments of the present application, Figure 6 A flowchart of the implementation of the tone mapping method proposed in the embodiments of the present application is shown in Figure 6 As shown in Figure 7 The tone mapping method can further include the following steps:

[0188] Step 1071, obtaining second pre-processing image data; wherein the second pre-processing image data is SDR image data obtained after photoelectric conversion, tone mapping and color gamut conversion.

[0189] In the embodiments of the present application, in the process of constructing the second mapping relationship in advance, the image data before photoelectric conversion processing, i.e., the second pre-processing image data, can be obtained first.

[0190] Further, in embodiments of the present application, the second pre-processing image data can include, but is not limited to, one or more of pixel information, brightness information, and light intensity information, which is not specifically limited in the present application.

[0191] Exemplarily, in some embodiments, the second pre-processing image data can be a pixel value before photoelectric conversion processing, or a light intensity value before photoelectric conversion processing.

[0192] It can be understood that, in embodiments of the present application, the second pre-processing image data can be SDR image data.

[0193] Step 1072, performing photoelectric conversion based on the second pre-processing image data to obtain second post-processing image data corresponding to the second pre-processing image data; wherein the second post-processing image data is SDR image data.

[0194] In embodiments of the present application, after obtaining the second pre-processing image data, photoelectric conversion can be performed based on the second pre-processing image data to obtain second post-processing image data corresponding to the second pre-processing image data.

[0195] Further, in embodiments of the present application, during photoelectric conversion, corresponding nonlinear light intensity data can be determined based on the second pre-processing image data, and then the nonlinear light intensity data is converted into corresponding light intensity data, and finally corresponding second post-processing image data is obtained.

[0196] It can be understood that, in embodiments of the present application, the second post-processing image data can be SDR image data.

[0197] Further, in embodiments of the present application, any photoelectric conversion method can be selected to complete photoelectric conversion processing of the second pre-processing image data, which is not specifically limited in the present application.

[0198] Exemplarily, in some embodiments, a photoelectric conversion method based on a photoelectric conversion function can be selected, for example, using a Gamma function specified in ST1886 for photoelectric conversion, as shown in formula (12):

[0199] nx = lx 1 / 2.4 (12)

[0200] Wherein, lx represents the input linear light intensity, and nx represents the output nonlinear light intensity.

[0201] Step 1073, constructing a second mapping relationship based on the second pre-processing image data and the second post-processing image data.

[0202] In the embodiments of the present application, after the photoelectric conversion based on the second pre-processing image data is performed to obtain the second post-processing image data corresponding to the second pre-processing image data, the second mapping relationship can be further constructed based on the second pre-processing image data and the second post-processing image data.

[0203] It can be understood that in the embodiments of the present application, after the photoelectric conversion of the image data is completed and the corresponding processing result is obtained, the corresponding relationship between the pre-processing data and the post-processing data, i.e., the corresponding relationship between the second pre-processing image data and the second post-processing image data, can be established, so that the construction of the second mapping relationship is completed based on the corresponding relationship.

[0204] That is, in the embodiments of the present application, in order to improve the efficiency of the photoelectric conversion process, the second mapping relationship used for photoelectric conversion can be pre-constructed, and the pre-constructed second mapping relationship can be called in subsequent processing. For example, the ST1886 is pre-calculated to generate an input-output corresponding relationship table (second mapping relationship) with a length of 8192.

[0205] In summary, by using the tone mapping method proposed in the present application, on the one hand, the first mapping relationship used for electro-optical conversion and tone mapping, and the second mapping relationship used for photoelectric conversion can be pre-constructed, so that in the process of tone mapping the HDR image to the SDR image, the first mapping relationship and the second mapping relationship can be directly used to complete the electro-optical conversion, the tone mapping and the photoelectric conversion, which greatly improves the processing efficiency of the HDR video tone mapping process and realizes the real-time processing of the tone mapping. On the other hand, in the process of tone mapping the HDR image to the SDR image, the NEON instruction set parallelization acceleration and the loop unrolling optimization methods are combined to further improve the efficiency of the program running. On the other hand, in the process of tone mapping the HDR image to the SDR image, the data after tone mapping (first mapping data) is adjusted by the content brightness data and the display brightness data, so that the adjusted image data can adapt to video content with different brightness and different screen brightness, and a better video vector is obtained.

[0206] Exemplarily, in some embodiments, by using the tone mapping method proposed in the present application, a frame of 1920x1080 resolution HDR video frame can be processed within 8ms on an Android mobile phone using a MediaTek 9000 processor, achieving the effect of real-time TMO of the HDR video.

[0207] It can be seen that the tone mapping method provided in the application uses the pre-calculated input-output relationship (the first mapping relationship and the second mapping relationship), the NEON instruction set parallelization acceleration, the loop unrolling optimization and the like in the tone mapping process of the HDR video, greatly improving the efficiency of the mobile end HDR video tone mapping process. In the tone mapping process, in the embodiment of the application, the difference of the luminance of different video contents and the maximum support luminance difference of different display screens are considered, so that the algorithm adapts to different luminance ranges, outputs the SDR video with appropriate luminance and color, and improves the quality of the generated SDR video.

[0208] The embodiment of the application provides a tone mapping method, obtains first image data corresponding to a video image, and determines second image data corresponding to the video image based on the first mapping relationship and the first image data; wherein the first mapping relationship is used for electro-optical conversion and tone mapping; the video image is an HDR image; color gamut conversion is performed based on the second image data to obtain third image data corresponding to the video image; a tone-mapped image corresponding to the video image is determined based on the second mapping relationship and the third image data; wherein the second mapping relationship is used for photoelectric conversion; the tone-mapped image is an SDR image. That is, in the embodiment of the application, the first mapping relationship for electro-optical conversion and tone mapping, which is constructed in advance, can be used to complete the electro-optical conversion and tone mapping processing of the video image, and the second mapping relationship for photoelectric conversion, which is constructed in advance, can be used to complete the photoelectric conversion of the video image, so that the electro-optical conversion, the tone mapping and the photoelectric conversion can be completed directly through the first mapping relationship and the second mapping relationship in the tone mapping process of the HDR video, the corresponding output result is obtained, the processing efficiency of the HDR video tone mapping process is greatly improved, and real-time processing of the tone mapping is realized.

[0209] Based on the above embodiment, another embodiment of the application provides a tone mapping method, for the tone mapping task of the HDR video, proposes a high-speed TMO method under the Android platform, can adaptively adjust the generated SDR video according to the luminance of the video content and the screen, and convert the HDR video into high-quality SDR video in real time.

[0210] That is, the tone mapping method provided in the application can output high-quality SDR video while meeting the real-time TMO requirement in the TMO processing process of the HDR video.

[0211] Further, in the embodiment of the application, Figure 7 The implementation framework schematic diagram of the tone mapping method provided in the embodiment of the application is as follows: Figure 8As shown, the overall framework for implementing the tone mapping method can be composed of five parts: a color space conversion module, an electro-optical conversion module, a tone mapping module, a color gamut conversion module, and an opto-electric conversion module.

[0212] Further, in the embodiments of the present application, Figure 8 The implementation schematic diagram of the tone mapping method proposed in the embodiments of the present application is as shown in FIG. 2. Figure 9 As shown, the implementation process of the tone mapping method mainly includes YUV and RGB conversion, ST2084 electro-optical conversion, S-shaped curve TMO, color gamut conversion (GMO), and Gamma opto-electric conversion. The color gamut conversion can be specifically Rec. 2020 to Rec. 709 color gamut conversion.

[0213] The input-output relationship can be pre-calculated in the electro-optical conversion module, the tone mapping module, and the opto-electric conversion module, and the lookup table method is used to improve the efficiency of program running at runtime. The electro-optical conversion and the TMO process can be combined for pre-calculation and result storage due to their continuity, that is, the input-output relationship is pre-calculated and combined, and the same table is used to record the input-output relationship (the first mapping relationship).

[0214] In the process of converting the YUV color space to the RGB color space and the process of color gamut conversion, the NEON instruction can be used for single instruction stream multi-data stream (SIMD) parallel processing to speed up the calculation process. In the two lookup table steps, the NEON instruction can be used for parallel storage of data. Overall, in the large loop for the video frame, according to different steps, four small loops are used for processing, one step of each small loop processes a certain number of pixels, and the completion of the small loop means that all the pixels of the frame have been processed by this part. The subdivided loop compared with the integrated loop makes the code easier to be optimized by the compiler, and improves the efficiency of the algorithm.

[0215] The processing method of each part is described in detail as follows:

[0216] 1. Color space conversion: YUV and RGB conversion

[0217] Since most video files are stored in the YUV color space, it is necessary to convert them into original data in the RGB color space for subsequent processing.

[0218] For example, for 10-bit depth YCbCr data in the tv range, the conversion process of converting it into RGB data is as shown in formula (1).

[0219] In the actual conversion process, due to the long time consumption of floating point operation, it is often converted to fixed point calculation. The efficiency of fixed point optimization is particularly obvious in the efficiency improvement of Android mobile terminal. Fixed point calculation can choose different precision, and higher precision can get better results. For example, choose to use 10-bit precision fixed point calculation to balance the video effect and operation efficiency, and the conversion process is as formula (2)-formula (4).

[0220] Where ir, ig, ib are integer RGB data, and the coefficient is calculated according to the precision and the original floating point coefficient. Fixed point calculation needs to increase 2 10 / 2 to round off to reduce the discontinuity of pixel color. In order to improve the effect of the output video, the precision of fixed point calculation can be appropriately improved without overflow.

[0221] In the color space conversion process from YUV to RGB, NEON instruction can be used for vectorized parallel calculation processing. NEON instruction set uses vector register with maximum length of 128 bits, and when reading 10-bit HDR video pixel value storage needs short type variable with length of 16 bits, and fixed point calculation process needs int type variable with length of 32 bits to store intermediate values to prevent overflow, therefore, 64-bit register can be used to read 4 length 16-bit video frame pixel values in parallel, and the data is widened to 4 32-bit vector int32x4_t at the beginning of the calculation process and stored in 128-bit register.

[0222] In order to reduce the time consumption of data storage, int32x4_t vector can be compressed back to int16x4_t at the end of the calculation process, and two vectors are combined into one int16x8_t vector, and the data of R, G, B three channels of each pixel is stored in int16x8x3_t type vector array.

[0223] In order to further reduce the time consumption of this step, the strategy of loop unrolling can be used, which processes the above steps twice in each step in the loop, that is, 16 pixels are processed in each step in the loop. The strategy of loop unrolling can reduce the redundant execution time caused by branch operation, and further improve the efficiency of the algorithm.

[0224] 2. Electro-optical conversion: ST2084 electro-optical conversion

[0225] When making pictures or videos, the acquisition device takes linear scene light as input, and converts it into a non-linear picture or video signal to store more low-brightness information sensitive to the human eye. Therefore, before tone mapping, the non-linear video signal needs to be converted into a linear light signal using an electro-optical transfer function EOTF. For example, the perceptual quantizer curve specified in the Society of Motion Picture and Television Engineers' Recommended Practice ST2084 can be used as the electro-optical transfer function, as shown in equation (6).

[0226] wherein nx represents the input non-linear light intensity, lx represents the output linear light intensity, FLT_MIN represents the minimum representable floating point number. The remaining symbols are parameters specified in ST2084, and their specific values are shown in Table 5.

[0227] 3. Tone mapping: S-shaped curve TMO tone mapping

[0228] In order to compress the data in the high dynamic range into the standard dynamic range, a global tone mapping based on the S-shaped curve can be used, as shown in equation (7).

[0229] wherein x is the input HDR light intensity, f(x) is the output SDR light intensity, a, b, c, d, e, f are parameters of the S-shaped curve. Among them, a controls the transition intensity from the shoulder to the middle of the curve, and a larger value means a faster transition; b controls the linearity of the S-shaped curve, and a larger value means that the curve is closer to linear; c controls the linear angle of the S-shaped curve, and a smaller value means that the rising angle of the curve is steeper; d controls the transition intensity from the front to the middle of the S-shaped curve, and a larger value means a stronger and faster transition; e and f jointly control the shape of the front section of the curve. Different scholars have proposed different parameters for this function, and the tone mapping effects are also different. Among them, the Hable curve specifies a = 0.15, b = 0.5, c = 0.1, d = 0.2, e = 0.02, f = 0.3, and the Uncharted2 curve specifies a = 0.22, b = 0.3, c = 0.1, d = 0.2, e = 0.01, f = 0.03, both of which are commonly used S-shaped curves for global tone mapping. In addition, there are other forms of curves such as Reinhard, ACES, and the like, and different curves or self-defined adjustments can be selected according to the specific effect in actual operation.

[0230] In order to adapt to video content of different brightness and different screen brightness, the maximum content luminance (MCL) obtained from the HDR10 video metadata can be used, and combined with the maximum monitor luminance (MML) of the screen, the result of the tone mapping is adaptively adjusted, as shown in equation (5).

[0231] Where to_linear_scale = 10000 / MML is used to convert the normalized linear light intensity from the brightness scale of HDR10 to the brightness scale of the screen.

[0232] In order to improve the efficiency of the process, the ST2084 electro-optical conversion and S-shaped curve are pre-calculated, and an input-output correspondence table with a length of 1024 is generated, and the value of the i-th item in the table corresponds to the result of the input with a pixel value of i after the two-step calculation. In the actual running process of the program, the corresponding conversion result value will be directly found according to the RGB data converted from YUV, which saves a lot of calculation time.

[0233] The table lookup operation of 8 pixels is performed in one step of the small loop, and the results are stored in a vector array of type int16x8x3_t. The NEON storage instruction is used to save the results of this step, which realizes the parallelization of the program running and improves the program efficiency.

[0234] 4. Color gamut conversion: GMO color gamut conversion

[0235] Limited by the performance and color rendering principle of the color rendering hardware, the screen cannot display all colors. The HDR10 video uses the BT2020 color space recommended by the International Telecommunication Union, and the SDR video usually uses the BT709 color gamut. In order to make the video display normally, it is necessary to use color gamut conversion (Gamut Mapping, GMO) to convert the wide color gamut to the narrow color gamut. The process of color gamut conversion is as formula (8).

[0236] Similarly, the efficiency of the calculation can be improved by converting the floating-point calculation of this step to fixed-point calculation, and the conversion process is as formula (9)-(11).

[0237] Where ir', ig', ib' are integer RGB data, and the accuracy of fixed-point calculation can be appropriately improved without overflow to improve the display effect of the output video.

[0238] 5. Photoelectric conversion: Gamma photoelectric conversion

[0239] For the processed linear light data, it needs to be converted to the nonlinear light data stored in the video file, and the Gamma function specified in ST1886 can be used for photoelectric conversion, as formula (12).

[0240] Where lx represents the input linear light intensity, and nx represents the output nonlinear light intensity.

[0241] Similarly, the input-output relationship of photoelectric conversion can be pre-calculated to save calculation time. In actual operation, an input-output table with a length of 1024 cannot meet the precision requirement of making the output video have a better display effect. Therefore, the normalized light intensity output by the tone mapping can be amplified to a range of 8192 for subsequent calculation, and a pre-calculated input-output corresponding table with a length of 8192 is used to find the corresponding nonlinear light intensity value according to the result of the GMO.

[0242] In this step, the look-up table operation processes 8 pixel values in one step of the loop, and the result is converted into uint8x8x3_t variable data parallel storage using the NEON instruction, improving the running efficiency.

[0243] In summary, the tone mapping method proposed in the present application uses pre-calculated input-output relationship, NEON instruction set parallelization acceleration, loop unrolling optimization and other methods in the tone mapping process of the HDR video, which greatly improves the efficiency of the mobile end HDR video tone mapping process. In the tone mapping process, the differences in brightness of different video contents and the maximum support brightness differences of different display screens are considered, so that the algorithm adapts to different brightness ranges, outputs SDR video with appropriate brightness and color, and improves the quality of the generated SDR video.

[0244] It should be noted that in the embodiments of the present application, the TMO curve is adjusted, which can be adapted to other tone mapping algorithms, and the effect of the SDR video result is adjusted according to the subjective effect.

[0245] The embodiment of the present application provides a tone mapping method, which can complete the electro-optical conversion and tone mapping processing of the video image through the first mapping relationship for electro-optical conversion and tone mapping which is pre-constructed, and can complete the photoelectric conversion of the video image through the second mapping relationship for photoelectric conversion which is pre-constructed, so that the electro-optical conversion, tone mapping and photoelectric conversion can be directly completed through the first mapping relationship and the second mapping relationship in the tone mapping process of the HDR video, the corresponding output result is obtained, the processing efficiency of the HDR video tone mapping process is greatly improved, and the real-time processing of the tone mapping is realized.

[0246] Based on the above embodiment, in another embodiment of the present application, Figure 9 The composition structure diagram of the tone mapping device provided in the embodiment of the present application is shown as Figure 10 As shown in the figure, the tone mapping device 90 provided in the embodiment of the present application can include:

[0247] The acquisition unit 901 is configured to acquire first image data corresponding to a video image; the video image is an HDR image;

[0248] The determining unit 902 is configured to determine second image data corresponding to the video image based on the first mapping relationship and the first image data, wherein the first mapping relationship is used for electro-optical conversion and tone mapping.

[0249] The obtaining unit 901 is further configured to perform gamut conversion based on the second image data to obtain third image data corresponding to the video image.

[0250] The determining unit 902 is further configured to determine a tone-mapped image corresponding to the video image based on the second mapping relationship and the third image data, wherein the second mapping relationship is used for photoelectric conversion, and the tone-mapped image is an SDR image.

[0251] In the embodiments of the present application, further, Figure 10 The computer device according to the embodiments of the present application can include a processor 1001, a memory 1002, a communication interface 1003, and a bus 1004 for connecting the processor 1001, the memory 1002, and the communication interface 1003. Figure 1 As shown in the figure, the computer device 100 according to the embodiments of the present application can include a processor 1001, a memory 1002, a communication interface 1003, and a bus 1004 for connecting the processor 1001, the memory 1002, and the communication interface 1003.

[0252] In the embodiments of the present application, the processor 1001 can be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It can be understood that, for different devices, the electronic device used to implement the functions of the processor can also be other devices, and the embodiments of the present application do not make specific limitations. The control device 11 can also include a memory 1002, which can be connected with the processor 1001, wherein the memory 1002 is used to store executable program codes, the program codes include computer operation instructions, and the memory 1002 can include a high-speed RAM memory and can also include a non-volatile memory, for example, at least two disk memories.

[0253] In the embodiments of the present application, the bus 1004 is used to connect the communication interface 1003, the processor 1001, and the memory 1002, and the mutual communication among these devices.

[0254] In practical applications, the memory 1002 can be a volatile memory, such as a random-access memory (RAM), or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), or a combination of the above types of memories, and provides instructions and data to the processor 1001.

[0255] Further, in the embodiments of the present application, the processor 1001 acquires first image data corresponding to a video image, and determines second image data corresponding to the video image based on the first mapping relationship and the first image data; the first mapping relationship is used for electro-optical conversion and tone mapping; the video image is an HDR image; color gamut conversion is performed based on the second image data to obtain third image data corresponding to the video image; a tone-mapped image corresponding to the video image is determined based on the second mapping relationship and the third image data; the second mapping relationship is used for photoelectric conversion; and the tone-mapped image is an SDR image.

[0256] In addition, each functional module in the embodiments can be integrated in one processing unit, or each unit can exist physically independently, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional module.

[0257] When the integrated unit is realized in the form of a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium, based on such understanding, the technical solutions of the embodiments can be embodied in the form of a software product, and the computer software product is stored in a storage medium, includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor execute all or part of the steps of the embodiments. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program codes that can be stored in the medium.

[0258] The embodiment of the present application provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the tone mapping method.

[0259] Specifically, the program instruction corresponding to the tone mapping method in the embodiment can be stored on a storage medium such as an optical disc, a hard disk, a U disk, etc. When the program instruction corresponding to the tone mapping method in the storage medium is read by an electronic device or is executed, the following steps are included.

[0260] Obtaining first image data corresponding to a video image, and determining second image data corresponding to the video image based on the first mapping relationship and the first image data; wherein the first mapping relationship is used for electro-optical conversion and tone mapping; the video image is an HDR image;

[0261] Performing gamut conversion based on the second image data to obtain third image data corresponding to the video image;

[0262] Determining a tone-mapped image corresponding to the video image based on the second mapping relationship and the third image data; wherein the second mapping relationship is used for photoelectric conversion; the tone-mapped image is an SDR image.

[0263] The embodiment of the present application further provides a computer program product.

[0264] In some embodiments, the computer program product can include a computer program or instructions.

[0265] In some embodiments, the computer program product can be applied to the computer device in the embodiment of the present application, and the computer program instructions make the computer execute the corresponding processes realized by the computer device in the various methods of the embodiment of the present application, and for the sake of brevity, details are not repeated here.

[0266] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can be in the form of a hardware embodiment, a software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer usable program code.

[0267] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks. Figure 1 The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks. Figure 1 The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks.

[0268] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks. Figure 1 The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks. Figure 1 The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks.

[0269] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks. Figure 1 The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks. ​ The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks.

[0270] The above-described embodiments are merely preferred embodiments of the present application, but not to confine the present application.

Claims

1. A tone mapping method characterized by, The method comprises: obtaining first image data corresponding to a video image, and determining second image data corresponding to the video image based on a first mapping relationship and the first image data; wherein the first mapping relationship is a set of pre-constructed input-output corresponding relationship tables for performing electro-optical conversion and tone mapping; the video image is a high dynamic range (HDR) image; performing gamut conversion based on the second image data to obtain third image data corresponding to the video image; determining a tone-mapped image corresponding to the video image based on a second mapping relationship and the third image data; wherein the second mapping relationship is a set of pre-constructed input-output corresponding relationship tables for performing photoelectric conversion; the tone-mapped image is a standard dynamic range (SDR) image.

2. The method of claim 1, wherein, The method further comprises: determining first-mapped data corresponding to the video image based on the first mapping relationship and the first image data; obtaining content brightness data and display brightness data corresponding to the video image, and adjusting the first-mapped data based on the content brightness data and the display brightness data to obtain the second image data.

3. The method of claim 1, wherein, The method further comprises: enlarging the third image data according to a length parameter corresponding to the second mapping relationship to obtain enlarged data; wherein the length parameter corresponding to the second mapping relationship represents the data length corresponding to the second mapping relationship; determining second-mapped data based on the second mapping relationship and the enlarged data; determining the tone-mapped image based on the second-mapped data.

4. The method according to claim 1 or 2, characterized in that, The method further comprises: performing color space conversion on initial image data corresponding to the video image based on NEON instructions and using a loop unrolling strategy to obtain the first image data.

5. The method according to claim 1 or 2, characterized in that, The method further comprises: performing gamut conversion on the second image data based on NEON instructions and using a loop unrolling strategy to obtain the third image data.

6. The method according to any one of claims 1-3, characterized by, The method further comprises: obtaining first pre-processing image data; performing electro-optical conversion based on the first pre-processing image data to obtain first linear light intensity data corresponding to the first pre-processing image data; wherein the first linear light intensity data is HDR light intensity data; performing tone mapping based on the first linear light intensity data to obtain first post-processing image data corresponding to the first pre-processing image data; wherein the first post-processing image data is SDR image data; constructing the first mapping relationship based on the first pre-processing image data and the first post-processing image data.

7. The method according to any one of claims 1-3, characterized by, The method further comprises: obtaining second pre-processing image data; wherein the second pre-processing image data is SDR image data obtained after electro-optical conversion, tone mapping, and gamut conversion. perform photoelectric conversion based on the second pre-processing image data to obtain second post-processing image data corresponding to the second pre-processing image data; the second post-processing image data is SDR image data; construct the second mapping relationship based on the second pre-processing image data and the second post-processing image data.

8. A tone mapping apparatus characterized by comprising: The tone mapping device comprises: an acquisition unit configured to acquire first image data corresponding to a video image; the video image is an HDR image; a determination unit configured to determine second image data corresponding to the video image based on a first mapping relationship and the first image data; the first mapping relationship is a set of pre-constructed input-output corresponding relationship tables for performing electro-optical conversion and tone mapping; the acquisition unit is further configured to perform gamut conversion based on the second image data to obtain third image data corresponding to the video image; the determination unit is further configured to determine a tone-mapped image corresponding to the video image based on a second mapping relationship and the third image data; the second mapping relationship is a set of pre-constructed input-output corresponding relationship tables for performing photoelectric conversion; the tone-mapped image is an SDR image.

9. A computer device, comprising: The computer device comprises a processor and a memory storing instructions executable by the processor, and when the instructions are executed by the processor, the method in any one of claims 1-7 is implemented.

10. A computer-readable storage medium having stored thereon a program, characterized in that, The program is executed by the processor, and the method in any one of claims 1-7 is implemented.

Images