A dynamic range compression method and system based on extremely lightweight Unet

Abstract

The present invention discloses a dynamic range compression method and system based on an extremely lightweight Unet, which uses an adaptive gamma algorithm and Photoshop to batch process all data, better retains the dark details of the image, appropriately brightens the dark areas while not overexposing the bright areas, and the processed images are natural without distortion or unrealistic colors; the lightweight Unet network is adopted, and compared with other deep learning algorithms, the model of the present invention is light, greatly saves the amount of calculation, has high operating efficiency and strong flexibility, and the effect is still outstanding and stable in scenes outside the training set, retains most of the picture details, uses a brightness map for training, and then uses a weight map as the network output, the fitting method is simple and easy to understand, and can retain the contrast information of the image, avoiding excessive whitening or color cast of the image.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more specifically, to a dynamic range compression method and system based on an extremely lightweight Unet. Background Art

[0002] The human eye can observe scenes with a relatively wide brightness range in nature. For example, from the pitch-black night to the sun that is difficult to look directly at, the dynamic range can reach nearly 10 orders of magnitude. However, existing traditional display devices are limited by hardware devices, and images can only express a very small part of the brightness range. For example, a common 8-bit image can only display the brightness in the integer range from 0 to 255. Therefore, in order to better display high dynamic range (HDR) images, it is necessary to implement the mapping from HDR images to low dynamic range (LDR), complete the conversion between HDR and LDR, that is, dynamic range compression (DRC).

[0003] Common dynamic compression algorithms are divided into global mapping and local mapping. Global mapping refers to one-to-one pixel mapping, and the resulting image has a low contrast and is prone to losing some detail features. Local mapping, on the other hand, enhances the contrast and brightness of different local regions in a differentiated manner through the connection between pixels. This method can retain a certain amount of contrast and detail information, but it is relatively complex and may cause distortion in some regions.

[0004] Traditional dynamic compression algorithms include linear shift algorithms and logarithmic mapping algorithms. Among them, the linear shift algorithm belongs to the global mapping method. It directly shifts the n-bit integer bits of the HDR image to the right by (n - m) bits, and then obtains an LDR image with m (m < n) bits of integers. The resulting image has relatively low resolution and contrast, and even the pixel colors are uneven. The colors in the high numerical range are relatively single, mostly concentrated in the medium and low numerical regions. Moreover, a lot of details are lost in the obtained LDR image. The previously dark areas are now darker, and the exposed areas are more severely exposed, and the image is severely distorted. The logarithmic mapping algorithm transforms the data from the interval [0, 2 n to the interval [0, n], and then linearly transforms it to the interval [0, 2 m . This method, like the linear shift algorithm, is a global mapping algorithm. Although it is efficient, it cannot take into account the local details of the image, and the image effect is average.

[0005] Compared with traditional algorithms, images generated by deep learning-based algorithms can enhance dark area details, avoid overexposure, and preserve texture details and contrast. However, existing methods have complex networks, low efficiency, and low practicality. Most networks are not stable enough on data that is out of the training set, and even show unrealistic colors or texture distortions, resulting in distorted results.

[0006] Therefore, it is an urgent problem for technical personnel in this field to provide a dynamic range compression method that can effectively solve the problems of complex network, low efficiency, poor stability, and low practicality of deep learning algorithm generation methods. Summary of the invention

[0007] In view of this, the present invention provides a dynamic range compression method and system based on an extremely lightweight Unet, which effectively solves the problems that traditional algorithms cannot guarantee the details of dark parts of images and are prone to overexposure, and that the deep learning algorithm generation method has a relatively complex network, low efficiency, poor stability, and low practicality.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] A dynamic range compression method based on extremely lightweight Unet, comprising the following steps:

[0010] Obtain a high dynamic range image, where the high dynamic range image is an RGB color space image;

[0011] The acquired image is preprocessed, and the specific steps of the preprocessing include:

[0012] Convert the RGB color space image into a YUV color space image to obtain the initial brightness value of the image I lum and brightness map;

[0013] Combine the adaptive gamma algorithm and Photoshop to batch process the initial brightness value of the image I lum , generating a standard reference map, wherein the standard reference map includes a standard weight map W and a standard brightness map L;

[0014] After data enhancement processing is performed on the brightness map of the image, the image is input into the Unet network model; the Unet network model performs dynamic range compression on the high dynamic range image to generate a low dynamic range image.

[0015] The initial brightness value of the image I lum The calculation formula is:

[0016] ;

[0017] in, I R , I G , I B is the RGB value of the RGB color space image, where C r , C g and C b is a constant.

[0018] The initial brightness value of the image is processed by combining the adaptive gamma algorithm and Photoshop batch processing I lum , generate a standard reference diagram, the specific steps are:

[0019] According to the initial brightness value I lum Distribution range, generating initial gamma;

[0020] The initial gamma is limited to a range of [0.4, 1] to obtain a final value of gamma;

[0021] Obtain a new brightness value L according to the final value of gamma, and generate a standard brightness map according to the brightness value L;

[0022] According to the initial brightness value I lum The weight value W corresponding to each pixel is determined based on the brightness value L, and a standard weight map is generated based on the weight value W of the pixel.

[0023] The Unet network model training process specifically includes:

[0024] After data enhancement processing is performed on the brightness map data set of the image, the data set is input into the Unet network model to be trained;

[0025] The Unet network model to be trained outputs the weight graph W of the Unet network model to be trained out , and generate the brightness map L of the Unet network model to be trained out ;

[0026] According to the standard weight map W and standard brightness map L obtained in the data preprocessing step and the weight map W output by the network to be trained out and brightness map L out , calculate the current loss loss;

[0027] The Unet network model is iterated according to the current loss. After the loss converges and becomes stable, the final Unet network model is selected based on the model output result at this time.

[0028] The data enhancement processing includes: flipping, cropping and Gaussian blurring.

[0029] The process of calculating the current loss loss is:

[0030] According to the standard brightness map L obtained in the data preprocessing step and the brightness map L output by the network to be trained out , use the loss function to calculate a set of losses, recorded as lum_loss;

[0031] According to the standard weight map W obtained in the data preprocessing step and the weight map W output by the network to be trained out Use the loss function to calculate a set of losses, recorded as weight_loss;

[0032] Assign different weight values ​​to lum_loss and weight_loss, and add lum_loss and weight_loss to get the total loss.

[0033] The loss function calculation formula is:

[0034] ;

[0035] in, y i represents the standard brightness map L or the standard weight map W, f ( x i ) represents the brightness map L output by the network to be trained out Or weight graph W out .

[0036] The standard weight map W, the standard brightness map L and the weight map W output by the network to be trained obtained in the data preprocessing step out and brightness map L out , calculate the current loss loss, and also include the following steps: use the minmax_loss that limits the extreme value to limit the maximum and minimum values ​​output by the Unet network model to be trained, and eliminate the influence of the extreme value on the network results; use the edge_loss that highlights the edge to optimize the edge of the weight graph output by the Unet network model to be trained.

[0037] A dynamic range compression system based on an extremely lightweight Unet, including:

[0038] Data acquisition module, used to obtain RGB color space images;

[0039] The data preprocessing module is used to convert the RGB color space image into the YUV color space image and obtain the initial brightness value of the image. Ilum ; and combining the adaptive gamma algorithm and Photoshop to batch process the initial brightness value of the image I lum , generate the standard weight map W and the standard brightness map L;

[0040] The network training module is used to train the constructed Unet network model to obtain a trained Unet network model;

[0041] The dynamic range compression module is used to perform dynamic range compression on images based on the trained Unet network model.

[0042] The network training model includes a data enhancement submodule and a loss function calculation submodule;

[0043] The data enhancement submodule is used to perform data enhancement operations on the brightness map data set, and the data enhancement operations include flipping, cropping and Gaussian blurring;

[0044] The loss function calculation submodule is used to calculate the current loss loss based on the loss function.

[0045] It can be seen from the above technical solutions that compared with the prior art, the present invention discloses a dynamic range compression method and system based on an extremely lightweight Unet, which uses an adaptive gamma algorithm and Photoshop to batch process all data. Compared with other traditional algorithms, it better retains the dark details of the image, appropriately brightens the dark areas while not overexposing the bright areas, and the processed images are natural without distortion or unrealistic colors. By adopting a lightweight Unet network, compared with other deep learning algorithms, the model of the present invention is lightweight, greatly saves the amount of calculation, has high operating efficiency, strong flexibility, and the effect is still outstanding and stable in scenes outside the training set, retains most of the picture details, uses a brightness map for training, and then uses a weight map as the network output. The fitting method is simple and easy to understand, and can retain the contrast information of the image, avoiding excessive whitening or color cast of the image. By using multiple losses in combination, the network converges stably, and the edges of the output weight map are clear and smooth, which enhances the edge details and picture contrast of the final result image. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0047] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0048] Figure 2 It is a schematic diagram of the network model structure of the method of the present invention.

[0049] Figure 3 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] like Figure 1 As shown, the embodiment of the present invention discloses a dynamic range compression method based on an extremely lightweight Unet, comprising the following steps:

[0052] Get a high dynamic range image, which is an RGB color space image.

[0053] Preprocess the acquired images. The specific steps of preprocessing include:

[0054] Convert the RGB color space image to the YUV color space image to get the initial brightness value of the image I lum and brightness map;

[0055] The initial brightness value of the image I lum The calculation formula is:

[0056] (1);

[0057] in, I R , I G , I B is the RGB value of the RGB color space image, where C r , C g and C b is a constant, and the present invention sets C r , C g and C b They are 0.299, 0.587 and 0.114 respectively.

[0058] Combining the adaptive gamma algorithm and Photoshop to batch process the initial brightness value of the image I lum , generate a standard reference map, which includes a standard weight map W and a standard brightness map L .

[0059] Adaptive gamma is a traditional algorithm that mainly generates appropriate gamma for different brightness ranges and distributions of images to compress the brightness dynamic range. Specifically, first, the initial brightness value is substituted into formula 2 to perform logarithmic calculation with the natural number e as the base, and then the arithmetic mean calculation is performed to obtain the final result. Similarly, the brightness value is substituted into formula 3 for similar calculation to obtain the sum, and finally the sum is substituted into formula 4 to obtain an initial gamma, that is, the constant G in formula 4, τ is set to 0.5 in the present invention.

[0060] (2);

[0061] (3);

[0062] (4);

[0063] in, i avg Indicates the initial brightness value compressed with the natural number e as the base I lum The arithmetic mean of T represents the updated compressed initial brightness value. I lum The new weighting constant after averaging and τ.

[0064] In order to avoid image distortion caused by too large or too small gamma, the gamma is restricted, and the present invention limits gamma to between 0.4 and 1. In other words, if G obtained in the previous step is less than 0.4, then gamma is 0.4; if G exceeds 1, then gamma is reduced to 1, which is the step of formula 4.

[0065] (5);

[0066] in, represents a constant that limits the gamma value, and Represent the lower and upper limits respectively.

[0067] Get the new brightness value L according to the final value of gamma, generate the standard brightness map according to the brightness value L; finally, Ilum Substitute into formula 6 and perform I lum The new brightness is obtained by raising τ to the power of gamma, which can generate the brightness reference image required for subsequent training of the network.

[0068] (6);

[0069] According to the initial brightness value I lum and brightness value L Determine the weight value corresponding to each pixel W , and based on the weight value of the pixel W Generate a standard weight graph. The weight value is calculated as:

[0070] (7);

[0071] The adaptive gamma algorithm is suitable for dynamic range compression, and its advantage is that it can brighten the dark parts while avoiding overexposure, but it loses some contrast. In view of this, when making a reference standard image, the present invention uses the traditional algorithm and combines it with Photoshop's contrast enhancement function, so that the contrast is still very high while compressing the image brightness range.

[0072] After data enhancement processing, the brightness map of the image is input into the Unet network model. The data enhancement processing includes: flipping, cropping and Gaussian blur.

[0073] The Unet network model performs dynamic range compression on high dynamic range images to generate low dynamic range images.

[0074] The network trained by the present invention is a modified Unet network, and the specific network structure is as follows: Figure 2 As shown in the figure, the Unet network model training process specifically includes:

[0075] After data enhancement, the brightness map dataset of the image is input into the Unet network model to be trained;

[0076] The Unet network model to be trained outputs the weight graph W of the Unet network model to be trained out , and generate the brightness map of the Unet network model to be trained L out ;

[0077] According to the standard weight map obtained in the data preprocessing step W , Standard brightness chart L And the weight map of the network output to be trained W out and brightness map Lout , calculate the current loss loss;

[0078] The Unet network model is updated and iterated according to the current loss. After the loss converges and stabilizes, the final Unet network model is selected based on the model output results at this time.

[0079] The process of calculating the current loss loss is:

[0080] According to the standard brightness map L obtained in the data preprocessing step and the brightness map L output by the network to be trained out , use the loss function L1 to calculate a set of losses, recorded as lum_loss;

[0081] According to the standard weight map obtained in the data preprocessing step W And the weight graph W output by the network to be trained out Use the loss function L1 to calculate a set of losses, recorded as weight_loss;

[0082] Assign different weight values ​​to lum_loss and weight_loss, and add lum_loss and weight_loss to get the total loss.

[0083] The loss function calculation formula is:

[0084] ;

[0085] in, y i represents the standard brightness map L or the standard weight map W, f ( x i ) represents the brightness map L output by the network to be trained out Or weight graph W out .

[0086] In addition, the present invention also adds minmax_loss to limit extreme values ​​and edge_loss to highlight edges. Specifically, minmax_loss to limit extreme values ​​mainly limits the maximum and minimum values ​​of the network output to eliminate the influence of extreme values ​​on the network results; edge_loss to highlight edges can learn the edges of the weight graph output by the network better. The principle is to use a Gaussian kernel to traverse the edge weight graph of the current output weight graph and the corresponding reference standard graph, extract features, and then calculate the loss of these two feature graphs. After calculating the above losses, assign different weight values ​​to each loss, and add them up to get the total loss. Subsequently, the iterative training model is updated according to the total loss. After the loss converges to a stable range, the initial model with the best effect is selected in combination with the output results of the model at this time. After training and testing on multiple data sets, key scenarios, and data outside the training set, the relevant parameters of the model are re-debugged and the final model is iterated.

[0087] like Figure 3 As shown, another embodiment of the present invention discloses a dynamic range compression system based on an extremely lightweight Unet, comprising:

[0088] Data acquisition module, used to obtain RGB color space images;

[0089] Data preprocessing module, used to convert RGB color space images into YUV color space images and obtain the initial brightness value of the image I lum ; and the initial brightness value of the image by combining the adaptive gamma algorithm and Photoshop batch processing I lum , generate the standard weight graph W and standard brightness diagram L ;

[0090] The network training module is used to train the constructed Unet network model to obtain a trained Unet network model;

[0091] The dynamic range compression module is used to perform dynamic range compression on images based on the trained Unet network model.

[0092] The network training model includes a data enhancement submodule and a loss function calculation submodule;

[0093] The data enhancement submodule is used to perform data enhancement operations on the brightness map dataset. The data enhancement operations include flipping, cropping, and Gaussian blurring.

[0094] The loss function calculation submodule is used to calculate the current loss based on the loss function loss .

[0095] Compared with other traditional brightness range compression algorithms, the present invention improves dark details while retaining the contrast of dark images, suppresses the exposure of highlight areas, and can be used stably in multiple scenes. Compared with other deep learning algorithms, the present invention has a lightweight model, greatly saves computational effort, has high operating efficiency, and still has outstanding and stable effects in scenes outside the training set, retaining most of the picture details.

[0096] The brightness map is used for training, and the weight map is used as the network output. The fitting method is simple and easy to understand, and can retain the contrast information of the image to avoid excessive whitening or color cast of the image.

[0097] The use of deep learning networks enhances the adaptability of the algorithm, and the final result can be made more perfect by adjusting the weight value of the loss.

[0098] Using the lightweight Unet network model, it can be quickly and efficiently deployed on embedded devices or terminals with NPUs, meeting the computing needs of current products, giving full play to the effects of the model, and enhancing the use of products.

[0099] The selection of multiple losses makes the network convergence efficient and stable, and the edge transition of the output weight map is clear and smooth, which enhances the edge details and image contrast of the final result map.

[0100] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0101] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A dynamic range compression method based on extremely lightweight Unet, characterized in that: The following steps are involved: Obtain a high dynamic range image, where the high dynamic range image is an RGB color space image; The acquired RGB color space image is preprocessed, and the preprocessing specifically includes: Convert the RGB color space image into a YUV color space image to obtain the initial brightness value I of the RGB color space image lum and brightness map; Combine the adaptive gamma algorithm and Photoshop to batch process the initial brightness value I of the RGB color space image lum , generating a standard reference map, wherein the standard reference map includes a standard weight map and a standard brightness map; After data enhancement processing is performed on the brightness map of the RGB color space image, the map is input into the Unet network model; the Unet network model performs dynamic range compression on the high dynamic range image to generate a low dynamic range image; The Unet network model training process specifically includes: After data enhancement processing is performed on the brightness image dataset of the RGB color space image, the dataset is input into the Unet network model to be trained; The Unet network model to be trained outputs the weight graph W of the Unet network model to be trained out , and generate the brightness map L of the Unet network model to be trained out ; According to the standard weight map and standard brightness map obtained in data preprocessing and the weight map W output by the network to be trained out and brightness map L out , calculate the current loss loss; The Unet network model is updated and iterated according to the current loss, and after the loss converges and tends to be stable, the final Unet network model is selected in combination with the output result of the model at this time; The process of calculating the current loss loss is: According to the standard brightness map obtained in data preprocessing and the brightness map L output by the network to be trained out , use the loss function to calculate a set of losses, recorded as lum_loss; According to the standard weight map obtained in data preprocessing and the weight map W output by the network to be trained out Use the loss function to calculate a set of losses, recorded as weight_loss; Assign different weight values ​​to lum_loss and weight_loss, and add lum_loss and weight_loss to get the total loss; Combine the adaptive gamma algorithm and Photoshop to batch process the initial brightness value I of the RGB color space image lum , generating a standard reference image, the standard reference image including a standard weight image and a standard brightness image, generating the standard reference image specifically includes: According to the final value of gamma, a new brightness value L is obtained, and a standard brightness map is generated according to the brightness value L, specifically including: converting the initial brightness value I lum Substituting into the formula L=(I lum ) gamma In I lum The new brightness value L is obtained by performing a power operation with λ as the base and gamma as the exponent, and the brightness reference image required for subsequent training of the network can be generated; According to the initial brightness value I lum The weight value W corresponding to each pixel is determined by the new brightness value L, and a standard weight map is generated based on the weight value W of the pixel. The weight value calculation formula is:

2. A dynamic range compression method based on extremely lightweight Unet according to claim 1, characterized in that: The initial brightness value I of the RGB color space image lum The calculation formula is: I lum =C r ×I R +C g ×I G +C b ×I B ; Among them, I R ,I G ,I B is the RGB value of the RGB color space image, where C r , C g and C b is a constant.

3. A dynamic range compression method based on extremely lightweight Unet according to claim 1, characterized in that: The initial brightness value I of the RGB color space image is processed by combining the adaptive gamma algorithm and Photoshop batch processing lum , generate a standard reference diagram, the specific steps are: According to the initial brightness value I lum Distribution range, generating initial gamma; Limiting the initial gamma to obtain a final value of gamma; Obtain a new brightness value L according to the final value of gamma, and generate a standard brightness map according to the brightness value L; According to the initial brightness value I lum The weight value W corresponding to each pixel is determined by the new brightness value L, and a standard weight map is generated based on the weight value W of the pixel.

4. A dynamic range compression method based on extremely lightweight Unet according to claim 1, characterized in that: The data enhancement processing includes: flipping, cropping and Gaussian blurring.

5. The dynamic range compression method based on extremely lightweight Unet according to claim 1, characterized in that: The loss function calculation formula is: Among them, y i represents the standard brightness map or standard weight map, f(x i ) represents the brightness map L output by the network to be trained out Or weight graph W out .

6. A dynamic range compression method based on extremely lightweight Unet according to claim 1, characterized in that: The standard weight map and standard brightness map obtained in data preprocessing and the weight map W output by the network to be trained out and brightness map L out , calculate the current loss loss, and also include the following steps: use the minmax_loss that limits the extreme value to limit the maximum and minimum values ​​output by the Unet network model to be trained, and eliminate the influence of the extreme value on the network results; use the edge_loss that highlights the edge to optimize the edge of the weight graph output by the Unet network model to be trained.

7. A dynamic range compression system based on an extremely lightweight Unet, characterized in that: A method for implementing a dynamic range compression method based on an extremely lightweight Unet as claimed in any one of claims 1 to 6, comprising: A data acquisition module, used to acquire high dynamic range RGB color space images; The data preprocessing module is used to convert the RGB color space image into a YUV color space image and obtain the initial brightness value I of the RGB color space image. lum ; and combine the adaptive gamma algorithm and Photoshop batch processing initial brightness value I lum , generate a standard weight map and a standard brightness map; The network training module is used to train the constructed Unet network model to obtain a trained Unet network model; Dynamic range compression module, used to perform dynamic range compression on images based on the trained Unet network model; The network training module includes a data enhancement submodule and a loss function calculation submodule; The data enhancement submodule is used to perform data enhancement operations on the brightness map data set, and the data enhancement operations include flipping, cropping and Gaussian blurring; The loss function calculation submodule is used to calculate the current loss loss based on the loss function.

Images