A high-quality HDR image generation system

Abstract

The present invention relates to a high-quality HDR image generation system, which includes an image input module, an FPGA acceleration module, and an optimized segmentation edge module; the image input module inputs an image into the FPGA acceleration module for acceleration processing to obtain an HDR image; the HDR image is passed through the optimized segmentation edge module to remove the edge distortion effect, and the final high-quality HDR image is obtained. The present invention solves the problem that local saturation is too high while local saturation is insufficient when the whole picture is processed.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a high-quality HDR image generation system. Background Art

[0002] High dynamic range (HDR) imaging is an imaging technology with a larger dynamic range than the standard dynamic range. HDR images have the advantages of higher contrast, higher saturation, and more vivid viewing effects. At the same time, HDR optimization in algorithms can make up for the deficiencies in hardware parameters of mobile phones or cameras during shooting, and better-quality images can be obtained through software optimization. With the continuous development of deep learning, an HDRnet network was proposed in 2017. Compared with traditional HDR processing algorithms, using deep learning methods to process high dynamic range imaging pictures can obtain results quickly, and a well-trained network has stable high-quality HDR image output. With the popularization of HDR technology and the continuous progress of related technologies, people can already use mobile phones or cameras to take more beautiful HDR images. Especially the application of HDR technology on TVs enables people to see more exquisite pictures. However, in daily viewing, it is not difficult to find that the images generated through current HDR processing have problems of local overexposure and local oversaturation. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a high-quality HDR image generation system to solve the problems of easy local oversaturation and insufficient local saturation in HDR imaging based on deep learning.

[0004] To achieve the above purpose, the present invention adopts the following technical solutions:

[0005] A high-quality HDR image generation system includes an image input module, an FPGA acceleration module, and an optimized segmentation edge module; the image input module inputs an image into the FPGA acceleration module for acceleration processing to obtain an HDR image; the HDR image is passed through the optimized segmentation edge module to remove the edge distortion effect, and the final high-quality HDR image is obtained.

[0006] Further, the FPGA acceleration module includes a Segnet and an HDRnet network, and the Segnet network module and the HDRnet network module are pre-trained comprehensive neural networks.

[0007] Further, the acceleration processing is specifically as follows:

[0008] Step S1: Input the picture into the Segnet network unit for specific segmentation of the image content;

[0009] Step S2: Different scenic parts of the segmented images are respectively sent to the HDRnet processing unit for high dynamic range processing;

[0010] Step S3: Synthesize each processed content after HDR processing to obtain an HDR image.

[0011] Furthermore, removing the edge distortion effect from the HDR image through the optimized segmentation edge module specifically involves: comparing and synthesizing the processed HDR image with the original input image to optimize the unnatural transition of some edges caused by segmentation and remove the distortion effect.

[0012] Furthermore, the Segnet network includes a decoding part and an encoding part, specifically:

[0013] The encoding part uses the first 13 convolutional networks of VGG16. Each encoding layer corresponds to a decoder layer. The output of the final decoder layer is sent to a soft-max classifier to independently generate class probabilities for each pixel. In the convolutional stage, the receptive field is increased through pooling while the image size is reduced;

[0014] In the decoding stage, the features after image classification are reproduced through deconvolution, upsampling restores the image to its original size, and finally, through Softmax, the maximum value of different classifications is output to obtain the final segmentation map.

[0015] Furthermore, in the network architecture of the Segnet network, the last convolutional layer outputs all classes. A softmax layer is added at the end of the network. Softmax calculates the maximum probability of each pixel in all classes as the label of this pixel, finally completing the pixel-level classification of the image.

[0016] Furthermore, there is also an evaluation index PSNR-μ, specifically: perform tone mapping tontmapping operations on the output image and the label image respectively and then calculate their PSNR:

[0017]

[0018] where I GT is the reference image, I Pred is the predicted image, i and j are the horizontal and vertical coordinates of the pixel points, and m and n are the total number of horizontal and vertical pixels of the image.

[0019] where T(x) refers to the tonemapping operation, which performs the following processing on the image:

[0020]

[0021] where μ is a preset value.

[0022] The present invention has the following beneficial effects compared with the prior art:

[0023] 1. Before entering the HDRnet neural network, the present invention first sends the picture into the Segnet network for segmentation processing. Each segmented image content is processed separately without interference, solving the problem of local over-saturation and local under-saturation that are likely to occur when the whole picture is processed;

[0024] 2. After the HDR processing of the picture is completed, the processed picture will be compared and fused with the original picture, solving the problem of overly unnatural segmentation edges that may occur due to segmentation. At the same time, the integrity of the picture is enhanced, and the distortion effect of the picture is avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is the flowchart of the method of the present invention;

[0026] Figure 2 is the functional division of the FPGA acceleration unit in an embodiment of the present invention;

[0027] Figure 3 is the example picture input in an embodiment of the present invention;

[0028] Figure 4 is Figure 3 the result picture after segmentation processing;

[0029] Figure 5 is the final output HDR picture in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0031] Please refer to Figure 1 , the present invention provides a high-quality HDR image generation system, including an image input module, an FPGA acceleration module, and an optimized segmentation edge module; the image input module inputs the image into the FPGA acceleration module for acceleration processing to obtain an HDR image; the HDR image is passed through the optimized segmentation edge module to remove the edge distortion effect and obtain the final high-quality HDR image.

[0032] In this embodiment, the FPGA acceleration module includes the Segnet and HDRnet networks; the Segnet network module and the HDRnet network module are pre-trained comprehensive neural networks. The training set contains 1500 pictures, including different scenes such as beaches, roads, skies, food, flowers, and portraits, and the number of training times reaches 5000 times to ensure that the comprehensive network has strong processing capabilities for different scenes.

[0033] In this embodiment, an image is input into the FPGA acceleration unit, and the pre-set neural network unit is called to start computing and processing.

[0034] In this embodiment, according to the characteristics of the Segnet image segmentation network and the HDRnet high dynamic range processing network, the functions of the FPGA acceleration module are divided into an input buffer unit, a weight buffer unit, a controller, an input register, multiple arithmetic units, an output buffer unit, an output unit, an off-chip storage unit, and an ARM core. Its specific structure is shown in Figure 2. In this structure, there are multiple arithmetic units, and multiple convolution operations can be performed simultaneously within the arithmetic units, improving the output speed of the calculation results. Specifically, the FPGA acceleration module processes the image as follows:

[0035] (1) The picture sent into the FPGA first enters the Segnet calculation unit. In the Segnet unit, the picture is segmented in terms of content. For example, Figure 3 as shown in the picture, the picture contains three different scenes: trees, sky, and buildings. After the picture is sent into the Segnet network, its different scenes are segmented, forming three pictures as shown in Figure 4 . When the segmented pictures are processed separately, they will not affect each other, solving the problem of local over-saturation or local under-saturation.

[0036] (2) After the picture is segmented into the picture as shown in Figure 4 by the Segnet network module, the sky, trees, and buildings are respectively input into the HDRnet network for high dynamic range processing. Since the sky is prone to over-saturation during HDR processing while the buildings are not, we process them separately. In this way, during the processing, the image can be made as vivid as possible while avoiding over-saturation of the sky or under-saturation of the building part. At the same time, thanks to the parallel computing function of the FPGA, the computing speed will not slow down due to the separate processing of the three scenes.

[0037] (3) The separately processed sky, tree, and building parts are spliced and fused to restore the original appearance of the complete picture.

[0038] (4) After the HDR pictures that are segmented and then processed are spliced, there are certain residual segmentation grids at the transition of some edges, which affects the visual effect. Therefore, we compare and synthesize the HDR pictures with the original pictures to avoid distortion to the greatest extent. At the same time, the integrity of the picture is enhanced, improving the visual effect.

[0039] After the processing is completed, in order to evaluate whether the processing effect meets the expectation, in this embodiment, PSNR-μ is introduced as an evaluation index, which means that the tone mapping operation is first performed on the output image and the label image respectively, and then their PSNR is calculated:

[0040]

[0041] Among them, T(x) refers to the tonemapping operation, which performs the following processing on the image:

[0042]

[0043] Among them, μ = 5000.

[0044] What this example shows is the process of processing a landscape picture containing sky, buildings and trees by using the method of the present invention. In order to obtain the final result, four stages of segmentation, HDR processing, stitching processing, and fusion with the original input are mainly carried out, and an FPGA is used as a computing acceleration unit, so as to produce a better HDR effect than the prior art.

[0045] The above are only the preferred embodiments of the present invention, and all equivalent changes and modifications made according to the scope of the patent application of the present invention shall fall within the scope of the present invention.

Claims

1. A high-quality HDR image generation system, characterized in that, It includes an image input module, an FPGA acceleration module, and an optimized segmentation edge module; the image input module inputs an image into the FPGA acceleration module for acceleration processing to obtain an HDR image; the HDR image is passed through the optimized segmentation edge module to remove the edge distortion effect, and the final high-quality HDR image is obtained. The FPGA acceleration module includes a Segnet network and an HDRnet network, and the Segnet network module and the HDRnet network module are pre-trained comprehensive neural networks. The acceleration processing is specifically as follows: Step S1: Input the picture into the Segnet network unit for specific segmentation of the image content. Step S2: Different scene parts of the segmented picture are respectively sent to the HDRnet processing unit for high dynamic range processing. Step S3: Synthesize the processed contents after HDR processing to obtain an HDR image. The process of removing the edge distortion effect from the HDR image through the optimized segmentation edge module is specifically: compare and synthesize the processed HDR image with the original input image, optimize the unnatural transition of some edges caused by segmentation, and remove the distortion effect.

2. The high-quality HDR image generation system according to claim 1, characterized in that, The Segnet network includes a decoding part and an encoding part, specifically: The encoding part uses the first 13 convolutional networks of VGG16. Each encoding layer corresponds to a decoder layer. The output of the final decoder layer is sent to the softmax layer to independently generate class probabilities for each pixel. In the convolutional stage, the receptive field is increased through pooling, and at the same time, the picture becomes smaller. In the decoding stage, the features after image classification are reproduced through deconvolution, upsampling restores to the original image size, and finally, the maximum value of different classifications is output through the softmax layer to obtain the final segmentation map.

3. The high-quality HDR image generation system according to claim 1, characterized in that, In the network architecture of the Segnet network, the last convolutional layer outputs all categories. A softmax layer is added at the end of the network. The softmax layer calculates the maximum probability of each pixel in all categories as the label of the pixel, and finally completes the pixel-level classification of the image.

4. The high-quality HDR image generation system according to claim 1, characterized in that, There is also an evaluation index PSNR-μ, specifically: perform a tonemapping operation on the output picture and the label picture respectively and then calculate their PSNR. Where I GT is the reference image, and I Pred is the predicted image. i and j are the horizontal and vertical coordinates of the pixel points, and m and n are the total number of horizontal and vertical pixels of the image; Among them, T(x) refers to the tonemapping operation, and the following processing is performed on the image: Among them, μ is a preset value.

Images