Tone mapping method and tone mapping terminal

Abstract

According to the tone mapping method and the tone mapping terminal provided by the invention, the Raw image is converted into the grayscale image, the grayscale image is downsampled and then input into the convolutional neural network, pixels needing to be processed are reduced, the operation time, the storage space and the computing resources are reduced, and the power consumption is reduced. And the convolutional neural network outputs a reduced-size iteration coefficient matrix group, and the reduced-size iteration coefficient matrix group is subjected to interpolation up-sampling to obtain a restored-size iteration coefficient matrix group. Applying a first iteration coefficient a1 of a restoration size to an Nth iteration coefficient aN of the restoration size to each pixel in the preprocessed grayscale image, and performing polynomial fitting calculation through N iterations to obtain a pixel value of a pixel in a corresponding tone mapping output grayscale image, so that the tone mapping output grayscale image approaches to a tone mapping target image, and thus, a final convolutional neural network model is obtained. Original image details are kept, the calculation amount and the calculation complexity of the convolutional neural network are reduced, and the method can be applied to an image sensor end.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a tone mapping method and a tone mapping terminal. Background Art

[0002] The purpose of tone mapping is to make high dynamic range images adapt to low dynamic range displays. To achieve this goal, the chromaticity, brightness and dynamic range of the high dynamic range image need to be mapped to the standard range of the low dynamic range image.

[0003] Neural network algorithms can learn complex tone mapping functions to obtain good image quality. Such images bring more details in dark and bright areas, meeting the application requirements of advanced driver assistance systems on vehicles. However, the complexity of neural networks also brings problems such as high power consumption, high latency and high storage space, which makes them unusable on the actual image sensor side. Common neural network models are end-to-end designs, which cannot meet the low power consumption requirements of image sensors. Summary of the invention

[0004] The purpose of the present invention is to provide a tone mapping method, which ensures that the details of the original image are preserved while reducing the amount of convolutional neural network operations and computational complexity, and reducing power consumption. The tone mapping low-power convolutional neural network structure in the method of the present invention can be applied to the image sensor end.

[0005] The present invention provides a tone mapping method, comprising:

[0006] S1. Using an image sensor to capture Raw images and corresponding color images in multiple scenes; preprocessing the Raw images to obtain preprocessed grayscale images; the preprocessing includes converting to grayscale images and downsampling and normalizing them; the color images are used as tone mapping target images after the preprocessing;

[0007] S2, sending the pre-processed grayscale image to a convolutional neural network for processing to obtain a reduced-size iteration coefficient matrix group; the pre-processed grayscale image is iteratively calculated N times to obtain a tone-mapped output grayscale image; the reduced-size iteration coefficient matrix group includes a total of N matrices from the first reduced-size iteration coefficient matrix to the Nth reduced-size iteration coefficient matrix;

[0008] S3, linearly interpolate the N matrices in the reduced size iteration coefficient matrix group to obtain a restored size iteration coefficient matrix group; the restored size iteration coefficient matrix group includes N matrices from the first restoration size iteration coefficient matrix to the Nth restoration size iteration coefficient matrix; the restored size jth iteration coefficient matrix is ​​composed of the restored size jth iteration coefficient corresponding to each pixel in the preprocessed grayscale image; 1≤j≤N, j is an integer;

[0009] Apply the corresponding restoration size first iteration coefficient a1 to the restoration size Nth iteration coefficient a to each pixel in the preprocessed grayscale image. N , obtaining the pixel values ​​of the pixels in the corresponding tone mapping output grayscale image after N-times iterative polynomial fitting calculation, making the tone mapping output grayscale image approach the tone mapping target image, adjusting the convolutional neural network parameter values ​​so that the convolutional neural network converges, and selecting the model parameters with the smallest loss function as the final convolutional neural network model;

[0010] S4, model application, obtaining a brightness gain value through the tone mapping output grayscale image and the preprocessed grayscale image, applying the brightness gain value to the three RGB channels corresponding to the Raw image to obtain a tone mapping output color image.

[0011] Furthermore, in step S1, the Raw image preprocessing specifically includes: interpolating the Raw image to obtain an RGB three-channel image; converting the RGB three-channel image into a single-channel grayscale image through a grayscale calculation formula, and downsampling and normalizing the grayscale image; the normalization process divides the pixel value of each pixel in the downsampled image by the maximum pixel value, so that the pixel value of each pixel after normalization is between the floating point number [0, 1].

[0012] Furthermore, the number of rows of each matrix in the restoration size iteration coefficient matrix group is the same as the number of rows of pixels of the preprocessed grayscale image; the number of columns of each matrix in the restoration size iteration coefficient matrix group is the same as the number of columns of pixels of the preprocessed grayscale image.

[0013] Furthermore, in step S3, the following iterative calculation formula is used:

[0014] Tmp1=Y in +a1*Y in *(Y in -1);

[0015] Tmp2=Tmp1+a2*Tmp1*(Tmp1-1);

[0016] …

[0017] Y out =Tmp N-1 +a N *Tmp N-1 *(Tmp N-1 -1)

[0018] The pixel value of any pixel in the preprocessed grayscale image is Y in, the pixel value of the pixel in the tone mapping output grayscale image corresponding to any one pixel is Y out ; a1 is the first iteration coefficient of the corresponding restoration size, a2 is the second iteration coefficient of the corresponding restoration size, a N is the Nth iteration coefficient of the corresponding restoration size; Tmp1 is the output result after the first iteration, that is, the pixel value of the grayscale image output by the tone mapping after the first iteration; Tmp2 is the pixel value of the output image after the second iteration; Tmp N-1 It is the pixel value of the output image after the N-1th iteration.

[0019] Furthermore, the preprocessed grayscale image is iteratively calculated twice to obtain the tone-mapped output grayscale image, where N=2.

[0020] Furthermore, in step S3, a loss function is used for calculation so that each pixel of the tone mapping output grayscale image approaches a corresponding pixel in the tone mapping target image; and the convolutional neural network training uses mean square error and structural similarity index as loss functions.

[0021] Furthermore, in step S4, the pixel value in the Raw image is multiplied by the brightness gain value to obtain the pixel value of the corresponding pixel point in the tone mapping output color image.

[0022] Furthermore, in step S1, the image sensor collects indoor and outdoor scenes in different time periods and weather conditions.

[0023] Further, the number of rows of pixels of the preprocessed grayscale image is an integer multiple of the number of rows of each matrix in the reduced-size iteration coefficient matrix group;

[0024] The number of columns of pixels of the preprocessed grayscale image is an integer multiple of the number of columns of each matrix in the reduced-size iteration coefficient matrix group.

[0025] The present invention also provides a tone mapping terminal, comprising:

[0026] A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the above tone mapping method by executing the computer instructions.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The present invention provides a tone mapping method, which converts a Raw image into a grayscale image and downsamples it for preprocessing before inputting it into a convolutional neural network. The number of pixels that the convolutional neural network needs to process is reduced, which greatly reduces the processing time, storage space and computing resources, and power consumption. The convolutional neural network outputs a reduced size iteration coefficient matrix group, and the reduced size iteration coefficient matrix group is obtained by interpolation upsampling to obtain a restored size iteration coefficient matrix group. The corresponding first restoration size iteration coefficient a1 to the Nth restoration size iteration coefficient a1 are applied to each pixel in the preprocessed grayscale image. N , after N iterations and polynomial fitting calculations, the pixel values ​​of the pixels in the corresponding tone mapping output grayscale image are obtained, so that the tone mapping output grayscale image is close to the tone mapping target image, thereby obtaining the final convolutional neural network model. Compared with the iterative coefficient matrix group of the directly output restoration size, while ensuring the original image details are retained, the convolutional neural network operation amount and computational complexity are reduced. The tone mapping low-power convolutional neural network structure in the method of the present invention can be applied to the image sensor end. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 The figure is a flowchart of a tone mapping method according to an embodiment of the present invention.

[0030] Figure 2 The figure is a schematic diagram of interpolating a Raw image (Bayer format image) to obtain an RGB three-channel image in a tone mapping method according to an embodiment of the present invention.

[0031] Figure 3 The figure is a schematic diagram of the principle of a tone mapping method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present invention will become more apparent from the following description. It should be noted that the accompanying drawings are in a very simplified form and use an inaccurate scale, which is only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0033] In the drawings, the sizes of layers, regions, elements and their relative sizes may be exaggerated for clarity. Like reference numerals denote like elements throughout.

[0034] It should be understood that when an element or layer is referred to as "on ...", "adjacent to ...", "connected to" or "coupled to" other elements or layers, it can be directly on, adjacent to, connected to or coupled to other elements or layers, or there can be intervening elements or layers. On the contrary, when an element is referred to as "directly on ...", "directly adjacent to ...", "directly connected to" or "directly coupled to" other elements or layers, there is no intervening element or layer. It should be understood that although the terms first, second, third, etc. can be used to describe various elements, components, regions, layers and / or parts, these elements, components, regions, layers and / or parts should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer or part from another element, component, region, layer or part. Therefore, without departing from the teachings of the present application, the first element, component, region, layer or part discussed below can be represented as the second element, component, region, layer or part. And when the second element, component, region, layer or part is discussed, it does not indicate that the present application necessarily has the first element, component, region, layer or part.

[0035] Spatially relative terms such as "under", "beneath", "below", "under", "above", "above", etc., may be used here for convenience of description to describe the relationship between an element or feature shown in the figure and other elements or features. It should be understood that in addition to the orientation shown in the figure, the spatial relationship terms are intended to also include different orientations of the device in use and operation. For example, if the device in the accompanying drawings is turned over, then the elements or features described as "under other elements" or "under it" or "under it" will be oriented as "on" other elements or features. Therefore, the exemplary terms "under" and "under" may include both upper and lower orientations. The device can be oriented otherwise (rotated 90 degrees or other orientations) and the spatial descriptors used herein are interpreted accordingly.

[0036] The purpose of the terms used herein is only to describe specific embodiments and is not intended to be limiting of the present application. When used herein, the singular forms "a", "an" and " / the" are also intended to include plural forms, unless the context clearly indicates another way. It should also be understood that the terms "consisting of" and / or "comprising", when used in this specification, determine the presence of features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or groups. When used herein, the term "and / or" includes any and all combinations of the relevant listed items.

[0037] The embodiment of the present invention provides a tone mapping method, such as Figure 1 As shown, including:

[0038] S1. Using an image sensor to capture Raw images and corresponding color images in multiple scenes; preprocessing the Raw images to obtain preprocessed grayscale images; the preprocessing includes converting to grayscale images and downsampling and normalization; the preprocessed color images are used as tone mapping target images;

[0039] S2, sending the preprocessed grayscale image to the convolutional neural network for processing to obtain a reduced-size iteration coefficient matrix group; the preprocessed grayscale image is iteratively calculated N times to obtain a tone-mapped output grayscale image; the reduced-size iteration coefficient matrix group includes a total of N matrices from the first reduced-size iteration coefficient matrix to the Nth reduced-size iteration coefficient matrix;

[0040] S3, linearly interpolate the N matrices in the reduced size iteration coefficient matrix group to obtain a restored size iteration coefficient matrix group; the restored size iteration coefficient matrix group includes N matrices from the first restoration size iteration coefficient matrix to the Nth restoration size iteration coefficient matrix; the jth restoration size iteration coefficient matrix is ​​composed of the jth restoration size iteration coefficient corresponding to each pixel in the preprocessed grayscale image; 1≤j≤N, j is an integer;

[0041] Apply the corresponding restoration size first iteration coefficient a1 to the restoration size Nth iteration coefficient a to each pixel in the preprocessed grayscale image N , after N times of iterative polynomial fitting calculation, the pixel value of the corresponding tone mapping output grayscale image is obtained, so that the tone mapping output grayscale image is close to the tone mapping target image, and the convolutional neural network parameter values ​​are adjusted to make the convolutional neural network converge, and the model parameters with the smallest loss function are selected as the final convolutional neural network model;

[0042] S4, model application, obtain the brightness gain value through the tone mapping output grayscale image and the preprocessed grayscale image, apply the brightness gain value to the RGB three channels corresponding to the Raw image, and obtain the tone mapping output color image.

[0043] The steps of the tone mapping method according to the embodiment of the present invention are described in detail below with reference to the accompanying drawings.

[0044] Step S1, using an image sensor to obtain Raw images in multiple scenes and intuitively displayed color images corresponding to the Raw images; converting the Raw images into grayscale images and performing downsampling and normalization processing. The image dataset can be obtained by using an image sensor to collect indoor and outdoor scenes in different time periods and weather conditions. The image sensor captures images including raw images and intuitively displayed color images corresponding to the raw images. The intuitively displayed color images are used as tone mapping target images (true labels) after the same preprocessing.

[0045] To collect enough input data (Raw images), a rich image database needs to be collected according to time and weather. The image sensor collects multiple frames of static images in several scenes (including but not limited to daytime, nighttime, different color temperatures, different gains, different exposures, etc.) indoors, outdoors, and in the laboratory. Raw images are also called Bayer format images. Bayer format images include, for example, multiple 2*2 pixel units arranged in any one of RGGB, GRBG, and BGGR, or in other suitable arrangements.

[0046] like Figure 2 and Figure 3 As shown, the Raw image (Bayer format image) is interpolated respectively to obtain an RGB three-channel image; the RGB three-channel image is converted into a single-channel grayscale image through a grayscale calculation formula, and the grayscale image is downsampled and normalized to obtain a preprocessed grayscale image. Exemplarily, the size of the RGB three-channel image is, for example, 256 pixels * 256 pixels, the pixel value of each pixel is represented by 16 bits, and the pixel value range of each pixel is [0,65535]. The initial grayscale value corresponding to a pixel is obtained according to the grayscale calculation formula from the red primary color pixel value, the green primary color pixel value, and the blue primary color pixel value corresponding to a pixel. The initial grayscale value is downsampled and normalized to obtain the preprocessed grayscale value Yin. Through grayscale calculation, RGB (256*256*3) is converted into a grayscale image (256*256*1), and then downsampled to obtain a fixed-size image of 128*128*1, 128*128*1 is a single-channel image, and the image size is, for example, 128 pixels*128 pixels. Normalization processing, for example, divides the pixel value of each pixel by the maximum pixel value (for example, 65535), so that the pixel value of each pixel after normalization (that is, the grayscale value Yin after preprocessing) is between the floating point number [0,1].

[0047] In order to reduce power consumption, the present invention optimizes the input image size of the convolutional neural network (CNN). The traditional neural network model is end-to-end. If the input image is 256*256*3 (RGB model), the output is also 256*256*3. The convolutional neural network of the present invention is a local tone mapping network (LTMNet) that accepts any image size. Before inputting to the local tone mapping network, RGB (256*256*3) is first converted to a grayscale image (256*256*1), and then downsampled to obtain a preset fixed size input of 128*128*1. The pre-processed grayscale image is used as the input image of the convolutional neural network. The convolutional neural network needs to process fewer pixels, which will greatly reduce the processing time, storage space and computing resources. Any size of input is converted to a preset fixed size, so the higher the image resolution, the more the power consumption advantage of the model can be reflected. The pre-processed grayscale image (for example, 128*128*1) is used as the input of the convolutional neural network. After preprocessing, the pixel value (grayscale value) of each pixel of the grayscale image ranges from a floating point number [0, 1].

[0048] Step S2, sending the pre-processed grayscale image to a convolutional neural network (CNN) for processing and outputting a down-sizing iteration coefficient matrix group; the pre-processed grayscale image is iteratively calculated N times to obtain a tone-mapped output grayscale image (low dynamic range image). The down-sizing iteration coefficient matrix group includes a total of N matrices from the down-sizing first iteration coefficient matrix to the down-sizing Nth iteration coefficient matrix.

[0049] Step S3, linearly interpolate the N matrices in the reduced-size iteration coefficient matrix group to obtain a restored-size iteration coefficient matrix group; the restored-size iteration coefficient matrix group includes N matrices from the first restored-size iteration coefficient matrix to the Nth restored-size iteration coefficient matrix.

[0050] The reduced size is relative to the size of the preprocessed grayscale image; illustratively, the size of the preprocessed grayscale image may be an integer multiple of the reduced size. The preprocessed grayscale image is, for example, 128 pixels*128 pixels, and each matrix in the reduced size iteration coefficient matrix group is, for example, 8 rows*8 columns.

[0051] The restored size is the same as the size of the preprocessed grayscale image, and each matrix in the restoration size iteration coefficient matrix group is, for example, 128 rows * 128 columns; that is, the number of rows of each matrix in the restoration size iteration coefficient matrix group is the same as the number of rows of pixels of the preprocessed grayscale image; the number of columns of each matrix in the restoration size iteration coefficient matrix group is the same as the number of columns of pixels of the preprocessed grayscale image.

[0052] The preprocessed grayscale image needs to be calculated through N iterations to obtain the tone-mapped output grayscale image. Iteration refers to repeating the feedback process in order to achieve the target result. Each repetition of the process is called an iteration, and the result obtained in each iteration will be used as the initial value of the next iteration. The target result in the present invention is the tone-mapped target image (true label). The pixel value of any pixel in the preprocessed grayscale image is, for example, Y in , the pixel value of the pixel in the tone mapping output grayscale image corresponding to the pixel is Y out ; a1 is the first iteration coefficient corresponding to the pixel, a2 is the second iteration coefficient corresponding to the pixel, a N is the Nth iteration coefficient corresponding to the pixel; Tmp1 is the pixel value of the output image after the first iteration is completed; Tmp2 is the pixel value of the output image after the second iteration is completed; Tmp N-1 It is the pixel value of the output image after the N-1th iteration.

[0053] Tmp1=Y in +a1*Y in *(Y in -1);

[0054] Tmp2=Tmp1+a2*Tmp1*(Tmp1-1);

[0055] …

[0056] Y out =Tmp N-1 +a N *Tmp N-1 *(Tmp N-1 -1)

[0057] The present invention is explained by taking the grayscale image after preprocessing as an example that needs to be iterated twice (N=2 as an example) to obtain the tone mapping output grayscale image. A large amount of calculation is mainly in the convolutional neural network model training and convolutional neural network model application (or test verification) links.

[0058] The improved convolutional neural network model training process of the present invention adopts step-by-step iterative calculation as follows:

[0059] Tmp1=Y in +a1*Y in *(Y in -1);

[0060] Y out =Tmp2=Tmp1+a2*Tmp1*(Tmp1-1)

[0061] During model training, Y in ∈[0,1];Y inRepresents the pixel value in the grayscale image after preprocessing (including normalization).

[0062] The improved convolutional neural network model application link of the present invention adopts iterative calculation combined with shift (>>) operation as follows:

[0063] Tmp1=Y in +a1*Y in *(Y in >>16-1)

[0064] Tmp2=Tmp1+a2*Tmp1*(Tmp1>>16-1)

[0065] In the application (or verification) of the convolutional neural network model, the input is a raw image without preprocessing. For example, the pixel value of each pixel is represented by 16 bits, and the input Y in Represents the pixel value of the Raw image without preprocessing (normalization), that is, in model application (or verification), Y in ∈[0, 65535]; combined with the shift operation in each iteration, it is equivalent to converting the N-1th calculation result Tmp to N-1 Divide by the maximum pixel value (e.g. 65535).

[0066] The improved present invention adopts step-by-step iterative calculation, and Tmp1 is calculated in the first iteration, and Tmp1 is used as the initial value of the second iteration; the result obtained in each iteration will be used as the initial value of the next iteration, and the pixel values ​​in the grayscale image after preprocessing in the training of the convolutional neural network model have been normalized; in the application (or verification) of the convolutional neural network model, a shift operation is combined in each iteration. Therefore, the improved present invention adopts step-by-step iterative calculation, normalization to obtain the preprocessed grayscale image, and shift operation, which can reduce the overall calculation value, avoid multiple multiplications of high-bit data in practical applications to cause the bit width to be too large; and also reduce the complexity of the calculation.

[0067] The network model before improvement uses the following calculation formula:

[0068]

[0069] Among them, W i ∈[-∞,∞] is Y in The coefficient of the i-th order term, W i The weight coefficients are generated by neural network learning.

[0070] Take i=4 as an example, Y out =W1*Y in +W2*Y in 2+W3*Y in 3 +W4*Y in 4

[0071] In the application (or verification) of the convolutional neural network model, Y in For example, the maximum value is 65535. Then, the network model calculation formula before improvement is used, Y in The i-th power (for example, i is an integer greater than 4) is a very large number. Therefore, using the calculation formula of the network model before the improvement will result in a long bit width on the hardware, and the overall calculation value is very large, which brings difficulties to both calculation and storage.

[0072] To facilitate the comparison before and after the improvement, Tmp is not calculated directly, but the expression of Tmp is substituted into Tmp2; the following expression is obtained:

[0073] Y out =Tmp2=Tmp1+a2*Tmp1*(Tmp1-1)

[0074] =[Y in +a1*Y in *(Y in -1)]+a2*[Y in +a1*Y in *(Y in -1)]*{[Y in +a1*Y in *(Y in -1)]-1}Y out It's about Y in Polynomial, sort and combine like terms to get

[0075] Y out =W4*Y in 4 +W3*Y in 3 +W2*Y in 2 +W1*Y in

[0076] W4 is Y in The coefficient of the fourth term, W3 is Y in The cubic coefficient of W2 is Y in The coefficient of the second-order term, W1 is Y in The coefficient of the first-order term; W i Y in The coefficient of the i-th order term. i It is the value calculated by combining a1 and a2 obtained by arranging and combining similar terms.

[0077] It can be seen that the network model Y before and after improvementout About Y in The multinomial expressions are essentially the same, only the calculation methods are different. In the application (or verification) calculation of the convolutional neural network model, the calculation of the network model before improvement is directly brought into Y in The i-th order term and the high-order term are complicated to calculate, resulting in a long bit width of the hardware. The improved method of the present invention can reduce the overall value of the calculation by gradually iterating and calculating, normalizing the grayscale image after preprocessing, and performing shift operations, thereby avoiding multiple multiplications of high-bit data in practical applications that result in a large bit width. At the same time, it also reduces the complexity of the calculation.

[0078] Model training, the preprocessed grayscale image is sent to the convolutional neural network for processing to obtain a reduced size iteration coefficient matrix group; the preprocessed grayscale image needs to be calculated through N iterations to obtain the tone mapping output grayscale image. Each pixel of the input preprocessed grayscale image is calculated, and each pixel of the output constitutes the tone mapping output grayscale image. For each pixel in the preprocessed grayscale image, the corresponding first iteration coefficient a1 of the restoration size to the Nth iteration coefficient a of the restoration size is applied. N , the pixel value of the pixel in the corresponding tone mapping output grayscale image is Y after N iterations out ; The same method is used to obtain the output value of each pixel, which constitutes the tone mapping output grayscale image. To collect enough input data, it is necessary to collect a rich image database according to time and weather, and then debug the most suitable target data according to needs. The process of mapping the input data to the target data is the process of convolutional neural network processing.

[0079] The training process is obtained by taking each pixel of the tone mapping output grayscale image and each pixel of the tone mapping target image (true label) as the loss function. The tone mapping network training uses the mean square error (MSE) and the structural similarity index (SSIM) as the loss function, interpolates the iteration coefficient values ​​in the N matrices in the reduced size iteration coefficient matrix group, and then approximates the tone mapping target image after passing through the polynomial fitting function, and finally adjusts the convolutional neural network parameter value to make the network converge; it should be understood that after adjusting the convolutional neural network parameter value, the iteration coefficient value in the reduced size iteration coefficient matrix group of the corresponding convolutional neural network output also changes (adjusts) accordingly; the model parameters with the smallest loss function in the verification data set are selected as the final model. For example, 70% of the image data set in step S1 is used as the training data set, and 30% of the data set is used as the verification data set.

[0080] The image sensor directly displays the color image converted into a grayscale image and then downsampled and normalized as the tone mapping target image (true label). The relationship between the mapping function is directly learned between the preprocessed grayscale image and the tone mapping target image.

[0081] The low power consumption is mainly due to the design of the network structure, which optimizes the input image size and the number of output iteration coefficients. The raw image is downsampled to obtain a preprocessed grayscale image, which is used as the input of the convolutional neural network. In this way, the convolutional neural network needs to process fewer pixels, which will greatly reduce the processing time, storage space and computing resources. The convolutional neural network outputs the optimization of the number of iteration coefficients of the reduced-size iteration coefficient matrix group, and performs feature extraction (i.e., the training process of the convolutional neural network) on the preprocessed grayscale image (low resolution). This is mainly because the downsampled image, i.e., the preprocessed grayscale image, still retains the main features and structure of the original raw image, which is sufficient for the task of tone mapping. The convolutional neural network outputs the reduced-size iteration coefficient matrix group, which reduces the computational complexity of the convolutional neural network compared to the convolutional neural network directly outputting the restored-size iteration coefficient matrix group. The reduced-size iteration coefficient matrix group is upsampled to the restored-size iteration coefficient matrix group through interpolation. The restored size is the same as the maximum reduced size of the preprocessed grayscale image. A polynomial fitting function is calculated for the preprocessed grayscale image to ensure that the details of the original image are retained while reducing the amount of computation and complexity of the convolutional neural network.

[0082] Model application, through tone mapping output grayscale image and preprocessed grayscale image to obtain brightness gain value, the gain value is applied to the three RGB channels corresponding to the raw image collected by the image sensor, and the tone mapping output color image is obtained. Specifically, the pixel value in the Raw image is multiplied by the brightness gain value to obtain the pixel value of the corresponding pixel in the tone mapping output color image. The raw image collected by the image sensor is input into the trained local tone mapping convolutional neural network model, and the tone mapping result is output. The purpose of the tone mapping algorithm is to compress the brightness of the HDR image and then map it to the display range of the LDR display device. At the same time, in the mapping process, important information such as details and colors of the original HDR image should be kept as much as possible. Exemplarily, the output tone mapping result is a low dynamic range (LDR) image to adapt to printers, displays, projectors, etc., which have limited dynamic range (or low dynamic range).

[0083] The tone mapping network results of the present invention are evaluated. On 3000 1920*1280 test images, the LTMNet model of the present invention is compared with an existing HDRNet (high dynamic range network) model. The similarity between the tone mapping output grayscale image and the target image is equal (approximately equivalent). However, on the same hardware device, the inference time of the LTMNet model of the present invention is only about 3% of that of the HDRNet model, and the storage space occupied by the LTMNet model of the present invention is only 22% of that of the HDRNet model, which meets the actual low-power operation requirements of the image sensor neural network processor. Low-power neural network algorithms can be used on the image sensor side to improve image quality.

[0084] The present invention designs a new low-power tone mapping curve to replace the traditional neural network's calculation of each pixel value gain, and uses grayscale images instead of RGB images as the input of the neural network. The neural network will learn the reduced-size iteration coefficient matrix group required by the curve. The present invention designs a low-power convolutional neural network structure for tone mapping applied to the image sensor end, which has low power consumption, high inference speed and small storage space, can obtain the required image contrast and brightness, and avoids the complicated manual parameter adjustment process.

[0085] The present invention also provides a tone mapping terminal, comprising:

[0086] A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the above tone mapping method by executing the computer instructions.

[0087] The computer program can be divided into one or more modules, one or more modules are stored in the memory and executed by the processor to complete the various modules of the present invention. One or more modules can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.

[0088] The processor referred to in this embodiment may be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.

[0089] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card, a secure digital (SD) card, a flash memory card, at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0090] In summary, the present invention provides a tone mapping method, which converts a Raw image into a grayscale image and downsamples it for preprocessing before inputting it into a convolutional neural network. The number of pixels that the convolutional neural network needs to process is reduced, which greatly reduces the processing time, storage space and computing resources, and power consumption. The convolutional neural network outputs a reduced-size iteration coefficient matrix group, and the reduced-size iteration coefficient matrix group is obtained by interpolation upsampling to obtain a restored-size iteration coefficient matrix group. For each pixel in the preprocessed grayscale image, the corresponding first-size restoration iteration coefficient a1 to the Nth-size restoration iteration coefficient a1 are applied. N , after N iterations and polynomial fitting calculations, the pixel values ​​of the pixels in the corresponding tone mapping output grayscale image are obtained, so that the tone mapping output grayscale image is close to the tone mapping target image, thereby obtaining the final convolutional neural network model. Compared with the iterative coefficient matrix group of the directly output restoration size, while ensuring the original image details are retained, the convolutional neural network operation amount and computational complexity are reduced. The tone mapping low-power convolutional neural network structure in the method of the present invention can be applied to the image sensor end.

[0091] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the method disclosed in the embodiment, since it corresponds to the device disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0092] The above description is only a description of the preferred embodiment of the present invention, and is not any limitation on the scope of rights of the present invention. Any technical personnel in this field can make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. A tone mapping method, characterized in that: include: S1. Using an image sensor to capture and obtain Raw images and corresponding color images in multiple scenes; The Raw image is preprocessed to obtain a preprocessed grayscale image; the preprocessing includes converting to a grayscale image and downsampling and normalizing the image; the color image after the preprocessing is used as a tone mapping target image; S2, sending the pre-processed grayscale image to a convolutional neural network for processing to obtain a reduced size iteration coefficient matrix group; The preprocessed grayscale image is iteratively calculated N times to obtain a tone-mapped output grayscale image; The reduced-size iteration coefficient matrix group includes a reduced-size first iteration coefficient matrix to a reduced-size Nth iteration coefficient matrix, a total of N matrices; S3, linearly interpolating the N matrices in the reduced-size iteration coefficient matrix group to obtain a restored-size iteration coefficient matrix group; The restoration size iteration coefficient matrix group includes N matrices from the first restoration size iteration coefficient matrix to the Nth restoration size iteration coefficient matrix; The j-th iteration coefficient matrix of the restoration size is composed of the j-th iteration coefficient of the restoration size corresponding to each pixel in the preprocessed grayscale image; 1≤j≤N, j is an integer; Apply the corresponding restoration size first iteration coefficient a1 to the restoration size Nth iteration coefficient a to each pixel in the preprocessed grayscale image. N , obtaining the pixel values ​​of the pixels in the corresponding tone mapping output grayscale image after N-times iterative polynomial fitting calculation, making the tone mapping output grayscale image approach the tone mapping target image, adjusting the convolutional neural network parameter values ​​so that the convolutional neural network converges, and selecting the model parameters with the smallest loss function as the final convolutional neural network model; S4, model application, obtaining a brightness gain value through the tone mapping output grayscale image and the preprocessed grayscale image, applying the brightness gain value to the three RGB channels corresponding to the Raw image to obtain a tone mapping output color image.

2. The tone mapping method according to claim 1, characterized in that: In step S1, the Raw image preprocessing specifically includes: interpolating the Raw image to obtain an RGB three-channel image; converting the RGB three-channel image into a single-channel grayscale image through a grayscale calculation formula, and downsampling and normalizing the grayscale image; the normalization process divides the pixel value of each pixel in the downsampled image by the maximum pixel value, so that the pixel value of each pixel after normalization is between the floating point number [0,1].

3. The tone mapping method according to claim 1, wherein: The number of rows of each matrix in the restoration size iteration coefficient matrix group is the same as the number of rows of pixels of the preprocessed grayscale image; The number of columns of each matrix in the restoration size iteration coefficient matrix group is the same as the number of columns of pixels of the preprocessed grayscale image.

4. The tone mapping method according to claim 1, wherein: In step S3, the following iterative calculation formula is used: Tmp1=Y in +a1*Y in *(AND in -1); Tmp2=Tmp1+a2*Tmp1*(Tmp1-1); … Y out =Tmp N-1 +a N *Tmp N-1 *(Tmp N-1 -1) The pixel value of any pixel in the preprocessed grayscale image is Y in , the pixel value of the pixel in the tone mapping output grayscale image corresponding to any one pixel is Y out ; a1 is the first iteration coefficient of the corresponding restoration size, a2 is the second iteration coefficient of the corresponding restoration size, a N is the Nth iteration coefficient of the corresponding restoration size; Tmp1 is the output result after the first iteration, that is, the pixel value of the grayscale image output by the tone mapping after the first iteration; Tmp2 is the pixel value of the output image after the second iteration; Tmp N-1 It is the pixel value of the output image after the N-1th iteration.

5. The tone mapping method according to claim 4, characterized in that: The preprocessed grayscale image is iteratively calculated twice to obtain the tone-mapped output grayscale image, where N=2.

6. The tone mapping method according to claim 1, wherein: In step S3, a loss function is used for calculation so that each pixel of the tone mapping output grayscale image approaches the corresponding pixel in the tone mapping target image; and the convolutional neural network training uses mean square error and structural similarity index as loss functions.

7. The tone mapping method according to claim 1, wherein: In step S4, the pixel value in the Raw image is multiplied by the brightness gain value to obtain the pixel value of the corresponding pixel point in the tone mapping output color image.

8. The tone mapping method according to claim 1, wherein: In step S1, the image sensor collects indoor and outdoor scenes in different time periods and weather conditions.

9. The tone mapping method according to claim 1, wherein: The number of rows of pixels of the preprocessed grayscale image is an integer multiple of the number of rows of each matrix in the reduced-size iteration coefficient matrix group; The number of columns of pixels of the preprocessed grayscale image is an integer multiple of the number of columns of each matrix in the reduced-size iteration coefficient matrix group.

10. A tone mapping terminal, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the tone mapping method according to any one of claims 1 to 9 by executing the computer instructions.

Images