A video inverse tone mapping method and related device capable of removing banding artifacts
Through the combination of neural network and mask generation network, the band-shaped artifact features are extracted and removed, and the problem of band-shaped artifacts in reverse tone mapping is solved, and high-quality HDR image conversion is achieved.
Patent Information
- Application Number
- CN202111460080.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-02
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-12-02
AI Technical Summary
The existing inverse tone mapping technology often shows ribbon artifacts when converting SDR images into HDR images, affecting the visual effect.
The neural network model is used to obtain the preliminary HDR brightness channel map, and the band-shaped artifact features are extracted through semantic segmentation, filtering and gradient analysis. The image mask is generated by combining the mask generation network, and the preliminary HDR brightness channel map and blurred map are fused to remove the band-shaped Artifact.
Effectively eliminates band-like artifacts while maintaining image details, improving the visual effect of HDR images.
Smart Images

Figure CN114331871B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a video inverse tone mapping method and related equipment capable of removing banding artifacts. Background Art
[0002] High dynamic range technology (HDR) is developing rapidly. Compared with standard dynamic range technology (SDR), HDR can bring significant improvement in visual effects. At present, the development obstacle faced by HDR display technology is that HDR content is very scarce, which makes existing HDR display devices such as HDR TVs unable to fully play their role. To address this problem, the inverse tone mapping technology (ITM) is currently used to convert a large number of existing SDR images into HDR images. However, in the current inverse tone mapping technology, when converting SDR images into HDR images, severe brightness faults, namely banding artifacts, will appear in some scenes. The existence of banding artifacts seriously reduces the visual effect of the converted image.
[0003] Therefore, the prior art still needs to be improved and enhanced. Summary of the invention
[0004] In view of the above-mentioned defects of the prior art, the present invention provides a video inverse tone mapping method and related equipment capable of removing banding artifacts, aiming to solve the problem of banding artifacts existing in the HDR image obtained after SDR conversion in the ITM technology in the prior art.
[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:
[0006] A first aspect of the present invention provides a video inverse tone mapping method capable of removing banding artifacts, the method comprising:
[0007] Acquire an SDR image, input the SDR image into a first neural network, and acquire a preliminary HDR brightness channel map output by the first neural network;
[0008] Input the SDR image into a semantic segmentation model, obtain a semantic segmentation feature map output by the semantic segmentation model, filter the preliminary HDR brightness channel map to obtain a blur map, obtain a contrast feature map according to the difference between the blur map and the preliminary HDR brightness channel map, and obtain a band artifact trajectory map according to the gradient of each pixel point of the preliminary HDR brightness channel map;
[0009] Combine the preliminary HDR brightness channel map, the semantic segmentation feature map, the contrast feature map, and the band artifact trajectory map to obtain a target feature map;
[0010] Inputting the target feature map into a mask generation network, obtaining an image mask output by the mask generation network, wherein the pixel value of each pixel in the image mask reflects the probability that the pixel is located in a banded artifact area;
[0011] fusing the preliminary HDR brightness channel map and the blur map according to the image mask to obtain a final HDR brightness channel map;
[0012] A target HDR color image is obtained according to the SDR image and the final HDR brightness channel map.
[0013] The video inverse tone mapping method capable of removing banding artifacts, wherein the step of obtaining a banding artifact trajectory map according to the gradient of each pixel point of the preliminary HDR brightness channel map comprises:
[0014] Obtaining an initial gradient of each pixel point according to the neighboring pixels of each pixel point on the preliminary HDR brightness channel map;
[0015] Obtaining the main direction intensity of the gradient of each pixel point on the preliminary HDR brightness channel map according to the initial gradient of each pixel point on the preliminary HDR brightness channel map;
[0016] The pixel value of each pixel point on the band-like artifact trajectory map is obtained according to the initial gradient and the main direction intensity corresponding to each pixel point on the initial brightness channel map.
[0017] The video inverse tone mapping method capable of removing banding artifacts, wherein the main direction intensity of the gradient of each pixel point of the preliminary HDR brightness channel image is obtained according to the initial gradient of each pixel point on the preliminary HDR brightness channel image, including:
[0018] When the initial gradient of the target pixel point on the preliminary HDR brightness channel map in the target direction is within a preset range, determining the intermediate gradient of the target pixel point in the target direction as the initial gradient;
[0019] When the gradient of the target pixel point on the preliminary HDR brightness channel map in the target direction is not within the preset range, determining that the intermediate gradient of the target pixel point in the target direction is 0;
[0020] Determine the intermediate gradients of the target pixel in each target direction, and use the largest intermediate gradient as the final gradient of the target pixel;
[0021] The main direction intensity of each pixel is determined according to the final gradient of each pixel on the preliminary HDR brightness channel map.
[0022] The video inverse tone mapping method capable of removing banding artifacts, wherein the determining the main direction intensity of each pixel point according to the final gradient of each pixel point on the preliminary HDR brightness channel map comprises:
[0023] Generate a gradient map according to the initial gradient of each pixel point on the preliminary HDR brightness channel map;
[0024] The gradient block where each pixel point is located is deformed to obtain the deformation vector of each pixel point, and the target matrix is constructed according to the deformation vector of each pixel point;
[0025] The main direction intensity corresponding to each pixel point is obtained according to the eigenvalue of the target matrix corresponding to each pixel point and the final gradient.
[0026] The video inverse tone mapping method capable of removing banding artifacts, wherein the fusing of the preliminary HDR brightness channel map and the blur map according to the image mask to obtain the final HDR brightness channel map, comprises:
[0027] Obtain the final HDR brightness channel map according to a preset formula;
[0028] The preset formula is:
[0029]
[0030] Among them, Y HDR represents the final HDR brightness channel map, M is the image mask, and Y M is the fuzzy graph, Y D This is the preliminary HDR brightness channel map.
[0031] The video inverse tone mapping method capable of removing banding artifacts, wherein the step of obtaining a target HDR color image according to the SDR image and the final HDR brightness channel map comprises:
[0032] Acquire a target brightness gain map according to the SDR image and the final HDR brightness channel map;
[0033] Convert the SDR image into a linear RGB channel, and obtain a target linear RGB image according to the target brightness gain map and the linear RGB channel of the SDR image;
[0034] The target linear RGB image is converted into a nonlinear RGB three-channel image to obtain the target HDR color image.
[0035] The video inverse tone mapping method capable of removing banding artifacts, wherein before inputting the SDR image into the first neural network, comprises:
[0036] Training the first neural network according to the multiple groups of first training data, and updating the parameters of the first neural network to first preliminary parameters according to a first objective function, wherein each group of the first training data includes a sample SDR image and an HDR color image corresponding to the sample SDR image, and the value of the first objective function is obtained by the difference between the brightness channel of the HDR color image corresponding to the sample SDR image and the output of the first neural network;
[0037] Fixing the parameters of the first neural network, training the mask generation network based on multiple sets of second training data, and updating the parameters of the mask generation network to second preliminary parameters according to a second objective function, wherein each set of the second training data includes a sample SDR image and an annotated mask corresponding to the sample SDR image, and the value of the second objective function is obtained by the difference between the annotated mask corresponding to the sample SDR image and the output of the mask generation network;
[0038] Fixing the parameters of the first neural network, training the mask generation network based on multiple sets of third training data, and updating the second preliminary parameters of the mask generation network to third preliminary parameters according to a third objective function, wherein each set of the third training data includes a sample SDR image and a real HDR color image corresponding to the sample SDR image, and the value of the third objective function is obtained by the difference between the brightness channel of the real HDR color image corresponding to the sample SDR image and the final HDR brightness channel map corresponding to the sample SDR image;
[0039] The final training of the first neural network parameters and the mask generation network is performed based on multiple groups of fourth training data to determine the final parameters of the first neural network parameters and the mask generation network.
[0040] A second aspect of the present invention provides a video inverse tone mapping device capable of removing banding artifacts, comprising:
[0041] A preliminary HDR brightness channel acquisition module, the preliminary HDR brightness channel acquisition module is used to acquire an SDR image, input the SDR image into a first neural network, and acquire a preliminary HDR brightness channel map output by the first neural network;
[0042] A feature extraction module, wherein the feature extraction module is used to input the SDR image into a semantic segmentation model, obtain a semantic segmentation feature map output by the semantic segmentation model, filter the preliminary HDR brightness channel map to obtain a blur map, obtain a contrast feature map according to the difference between the blur map and the preliminary HDR brightness channel map, and obtain a band artifact trajectory map according to the gradient of each pixel point of the preliminary HDR brightness channel map;
[0043] A feature fusion module, wherein the feature fusion module is used to combine the preliminary HDR brightness channel map, the semantic segmentation feature map, the contrast feature map and the banded artifact trajectory map to obtain a target feature map;
[0044] A mask generation module, wherein the mask generation module is used to input the target feature map into a mask generation network to obtain an image mask output by the mask generation network, wherein the pixel value of each pixel in the image mask reflects the probability that the pixel is located in a banded artifact area;
[0045] A final HDR brightness channel acquisition module, wherein the final HDR brightness channel acquisition module is used to fuse the preliminary HDR brightness channel map and the blur map according to the image mask to obtain a final HDR brightness channel map;
[0046] An HDR color image acquisition module, wherein the HDR color image acquisition model is used to acquire a target HDR color image according to the SDR image and the final HDR brightness channel map.
[0047] According to a third aspect of the present invention, a terminal is provided, comprising a processor and a computer-readable storage medium communicatively connected to the processor, wherein the computer-readable storage medium is suitable for storing a plurality of instructions, and the processor is suitable for calling the instructions in the computer-readable storage medium to execute the steps of implementing any of the classification methods described above.
[0048] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any of the above-mentioned classification methods.
[0049] Compared with the prior art, the present invention provides a video inverse tone mapping method and related equipment capable of removing banding artifacts. The video inverse tone mapping method capable of removing banding artifacts first obtains a preliminary HDR brightness channel map of the SDR image based on a neural network model, and then extracts multiple features related to banding artifacts through the SDR image and the preliminary HDR brightness channel map, including semantic segmentation of the SDR image to obtain a semantic segmentation feature map, filtering the preliminary HDR brightness channel map to obtain a blur map, obtaining a contrast feature map based on the blur map, obtaining a banding artifact trajectory map based on the gradient of each pixel point of the preliminary HDR brightness channel map, and extracting the semantic segmentation feature map from the SDR image. The feature map, the contrast feature map and the banding artifact trajectory map are combined to obtain a target feature map, and an image mask that characterizes the position, i.e., the severity, of the banding artifacts in the image is obtained based on the target feature map. Since there are no banding artifacts in the blurred map, but the image details are blurred, and there are banding artifacts in the preliminary HDR brightness channel map, but there are image details, the final HDR brightness channel map obtained by fusing the blurred map and the preliminary HDR brightness channel map based on the image mask can effectively eliminate the banding artifacts in the image while maintaining the image details, so that the target HDR color image obtained according to the SDR image and the final HDR brightness channel map can maintain the image details and have no banding artifacts. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 A flowchart of an embodiment of a video inverse tone mapping method capable of removing banding artifacts provided by the present invention;
[0051] Figure 2 An overall framework diagram of the inverse tone mapping process in an embodiment of the video inverse tone mapping method capable of removing banding artifacts provided by the present invention;
[0052] Figure 3 A schematic diagram of a feature extraction stage in an embodiment of a video inverse tone mapping method capable of removing banding artifacts provided by the present invention;
[0053] Figure 4 A schematic diagram of three feature maps extracted in an embodiment of a video inverse tone mapping method capable of removing banding artifacts provided by the present invention;
[0054] Figure 5 A schematic diagram of an image mask in an embodiment of a video inverse tone mapping method capable of removing banding artifacts provided by the present invention;
[0055] Figure 6 A schematic diagram of generating a final HDR brightness channel map in an embodiment of a video inverse tone mapping method capable of removing banding artifacts provided by the present invention;
[0056] Figure 7A schematic diagram of obtaining a target HDR color image in an embodiment of a video inverse tone mapping method capable of removing banding artifacts provided by the present invention;
[0057] Figure 8 A structural principle diagram of an embodiment of a video inverse tone mapping device capable of removing banding artifacts provided by the present invention;
[0058] Fig. 9 A schematic diagram of the principles of an embodiment of a terminal provided by the present invention. DETAILED DESCRIPTION
[0059] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0060] The video inverse tone mapping method capable of removing banding artifacts provided by the present invention can be applied to a terminal with computing capability. The terminal can execute the video inverse tone mapping method capable of removing banding artifacts provided by the present invention to convert an SDR image into a target HDR color image. The terminal can be, but is not limited to, various computers, mobile terminals, smart home appliances, wearable devices, etc.
[0061] Embodiment 1
[0062] like Figure 1 As shown, in one embodiment of the video inverse tone mapping method capable of removing banding artifacts, the method comprises the steps of:
[0063] S100, acquiring an SDR image, inputting the SDR image into a first neural network, and acquiring a preliminary HDR brightness channel map output by the first neural network;
[0064] The SDR image may be an independent SDR image or a frame image in an SDR video stream. After acquiring the SDR image, the SDR image is input into the first neural network, and the first neural network outputs the preliminary HDR brightness channel map based on the SDR image. The pixel value of each pixel on the preliminary HDR brightness channel map is a brightness value. Specifically, the parameters of the first neural network may be obtained by training based on a sample SDR image and an HDR image corresponding to the sample SDR image, so that the first neural network can output a brightness channel map as the preliminary HDR brightness channel map based on the input SDR image. The specific training process of the first neural network will be described later.
[0065] like Figure 2As shown, the structure of the first neural network can be a structure using a residual network, and the skip connection branch can be the benchmark inverse tone mapping method defined by the International Telecommunication Commission in BT.2446: the main branch consists of two convolutional layers and 16 residual blocks, each residual block includes two convolutional layers and a ReLU activation layer, in the main branch, the kernel size of all convolutional layers is 3*3, the step size is 1, the resolution of all feature maps is consistent with the input image, except that the number of output channels of the last convolutional layer is 1, and the number of output channels of other convolutional layers is 64. Of course, it can be understood that those skilled in the art can use other structures to construct the first neural network, for example, the number of convolutional layers, the size and step size of the convolutional kernel can be modified, and the present invention is not limited to the above structure.
[0066] S200, input the SDR image into a semantic segmentation model, obtain a semantic segmentation feature map output by the semantic segmentation model, filter the preliminary HDR brightness channel map to obtain a blurred map, obtain a contrast feature map based on the difference between the blurred map and the preliminary HDR brightness channel map, and obtain a band artifact trajectory map based on the gradient of each pixel point of the preliminary HDR brightness channel map.
[0067] There may be banding artifacts in the preliminary HDR brightness channel map. In order to remove the banding artifacts, Figure 2 As shown, in the video inverse tone mapping method capable of removing banding artifacts provided by this embodiment, after obtaining the preliminary HDR brightness channel map, features related to banding artifacts are extracted. Specifically, Figure 3 As shown, three feature maps are extracted: semantic segmentation feature map, contrast feature map and banding artifact trajectory map.
[0068] The semantic segmentation feature map is obtained based on the SDR image. The SDR image is input into the semantic segmentation model to obtain the semantic segmentation feature map output by the semantic segmentation model. The existing semantic segmentation model can be used in this method. Specifically, a model can be constructed based on the structure of the existing semantic segmentation model and semantic segmentation training can be performed. The trained semantic segmentation model is used to extract the semantic segmentation feature map corresponding to the SDR image. The existing trained semantic segmentation model can also be used to extract the semantic segmentation feature map in this method. The present invention is not limited to the category of the semantic segmentation model. The output of the semantic segmentation feature map can be multiple semantic channels, for example, there can be channels corresponding to 8 semantic categories such as plane, person, vehicle, building, object, nature, sky and background. The pixel value of each pixel in the semantic segmentation feature map is the corresponding semantic feature value. The semantic segmentation feature map can provide semantic information and improve the accuracy of the image mask extracted later.
[0069] The contrast feature map is obtained based on the preliminary HDR brightness channel map. Specifically, after obtaining the preliminary HDR brightness channel map, the preliminary HDR brightness channel map is first filtered, that is, the preliminary HDR brightness channel map is blurred to remove the banding artifacts in the preliminary HDR brightness channel map to obtain the blurred map. For example, the preliminary HDR brightness channel map can be mean filtered, and the filter kernel size can be 21. Of course, this is only an example, and other filtering methods and other filter kernel sizes can be used to blur the preliminary HDR brightness channel map.
[0070] After obtaining the blurred image, the blurred image and the preliminary HDR brightness channel image are subtracted to obtain the contrast feature image. There are banding artifacts and details in the preliminary HDR brightness channel image, and there are no banding artifacts in the blurred image but the details are blurred. The contrast feature image can provide a reference for the location of the banding artifacts and the degree of the generation of the banding artifacts.
[0071] The banding artifact trajectory map is obtained according to the gradient of each pixel point of the preliminary HDR brightness channel map. Specifically, the step of obtaining the banding artifact trajectory map according to the gradient of each pixel point of the preliminary HDR brightness channel map includes:
[0072] Obtaining an initial gradient of each pixel point according to the neighboring pixels of each pixel point on the preliminary HDR brightness channel map;
[0073] Obtaining the main direction intensity of the gradient of each pixel point of the preliminary HDR brightness channel image according to the initial gradient of each pixel point on the preliminary HDR brightness channel image;
[0074] The pixel value of each pixel point on the band-like artifact trajectory map is obtained according to the initial gradient and the main direction intensity corresponding to each pixel point on the initial brightness channel map.
[0075] The inventors have found that banding artifacts often appear in small gradient regions, and banding artifacts often have a strong main directionality. Therefore, in this embodiment, the gradient on the preliminary HDR brightness channel map is used to obtain a banding artifact trajectory map.
[0076] In order to extract the trajectory map of the banding artifact, on the preliminary HDR brightness channel map, firstly, the initial gradient of each pixel point is obtained according to the adjacent pixel points of each pixel point on the preliminary HDR brightness channel map, which can be specifically extracted by the following formula:
[0077] G x (x,y)=|Y D (x+1,y)-Y D (x,y)|
[0078] G y (x,y)=|Y D (x,y+1)-Y D (x,y)|
[0079] Among them, G x (x,y),G y (x, y) represent the initial gradients in the x and y directions of the pixel point with coordinates (x, y) on the preliminary HDR brightness channel map, respectively. D (x, y) represents the pixel value of the pixel point with coordinates (x, y) on the preliminary HDR brightness channel map, Y D (x+1,y),Y D (x, y+1) respectively represent the pixel values of the adjacent pixel points in the x direction and in the y direction of the pixel point with coordinates (x, y) on the preliminary HDR brightness channel map.
[0080] After obtaining the initial gradient of each pixel on the preliminary HDR brightness channel map, obtaining the main direction intensity of the gradient of each pixel on the preliminary HDR brightness channel map according to the initial gradient of each pixel on the preliminary HDR brightness channel map specifically includes:
[0081] When the initial gradient of the target pixel point on the preliminary HDR brightness channel map in the target direction is within a preset range, determining the intermediate gradient of the target pixel point in the target direction as the initial gradient;
[0082] When the gradient of the target pixel point on the preliminary HDR brightness channel map in the target direction is not within the preset range, determining that the intermediate gradient of the target pixel point in the target direction is 0;
[0083] Determine the intermediate gradients of the target pixel in each target direction, and use the largest intermediate gradient as the final gradient of the target pixel;
[0084] The main direction intensity of each pixel is determined according to the final gradient of each pixel on the preliminary HDR brightness channel map.
[0085] The inventors found that where the banding artifacts appear, the difference in pixel values on both sides of the track formed by the banding artifacts is mostly between 1 / 1023 and 4 / 1023. Therefore, in this embodiment, the preset range is 1 / 1023 to 4 / 1023. Of course, according to different accuracy requirements, the difference in pixel values on both sides of the banding artifacts in the existing image can be more accurately counted to determine different preset ranges. The solution process of the intermediate gradient of each pixel point on the preliminary HDR brightness channel map can be expressed by the formula:
[0086]
[0087]
[0088] Among them, G x ′(x,y), G y ′(x, y) represents the intermediate gradient in the x direction and in the y direction of the pixel point with coordinates (x, y) on the preliminary HDR brightness channel map, respectively. x (x,y),G y (x, y) represents the initial gradient in the x direction and y direction of the pixel point with coordinates (x, y) on the preliminary HDR brightness channel map, respectively.
[0089] The process of solving the final gradient of each pixel on the preliminary HDR brightness channel map can be expressed as follows:
[0090] G(x,y)=max(G′ x (x,y),G′ y (x,y)
[0091] G(x, y) represents the final gradient of the pixel point with coordinates (x, y) on the preliminary HDR brightness channel map.
[0092] Determining the main direction intensity of each pixel point according to the final gradient of each pixel point on the preliminary HDR brightness channel map includes:
[0093] Generate a gradient map according to the initial gradient of each pixel point on the preliminary HDR brightness channel map;
[0094] The gradient block where each pixel point is located is deformed to obtain the deformation vector of each pixel point, and the target matrix is constructed according to the deformation vector of each pixel point;
[0095] The main direction intensity corresponding to each pixel point is obtained according to the eigenvalue of the target matrix corresponding to each pixel point and the final gradient.
[0096] Specifically, the size of the gradient map is consistent with the size of the preliminary HDR brightness channel map, and the pixel value of each pixel on the gradient map is the initial gradient of the corresponding pixel on the preliminary HDR brightness channel map. For each pixel, an area of size n*n can be determined with the pixel as the center as the gradient block where the pixel is located, where n is a positive integer, for example, n can be 5, 3, etc. The gradient block is deformed and stretched into a vector of n*n*1, that is, the deformation vector of the pixel is obtained. Since the initial gradient includes gradient values in two directions, each pixel corresponds to two deformation vectors. The target matrix is constructed according to the deformation vector of each pixel as follows:
[0097]
[0098] Among them, g x (x,y),g y (x, y) are the deformation vectors of the pixel with coordinates (x, y) in the x direction and in the y direction respectively. The target matrix J(x, y) has two non-negative eigenvalues, λ1 and λ2, respectively. Assuming λ1>λ2, the main direction intensity corresponding to the pixel with coordinates (x, y) is calculated as follows:
[0099]
[0100] A J (x, y) is the main direction intensity corresponding to the pixel point with coordinates (x, y). J The larger the (x, y) is, the more likely the corresponding pixel point is on the strip contour. The strip artifact trajectory map Y is obtained by performing point-by-point convolution using the main direction intensity and the final gradient. contour , which can be expressed as: That is, the main direction intensity corresponding to the pixel point with coordinates (x, y) is multiplied by the final gradient point to obtain the pixel value of the pixel point with coordinates (x, y) on the band-like artifact trajectory map.
[0101] The SDR image, the semantic segmentation feature map, the contrast feature map and the band artifact trajectory map are as follows: Figure 4 As shown, Figure 4 From left to right in the figure are the SDR image, the second channel of the semantic segmentation feature map, the contrast feature map, and the banding artifact trajectory map.
[0102] Please refer again Figure 1 The video inverse tone mapping method capable of removing banding artifacts provided by this embodiment further includes the steps of:
[0103] S300, combining the preliminary HDR brightness channel map, the semantic segmentation feature map, the contrast feature map and the band artifact trajectory map to obtain a target feature map.
[0104] like Figure 3 As shown, after obtaining the semantic segmentation feature map, the contrast feature map and the banding artifact trajectory map, the semantic segmentation feature map, the contrast feature map, the banding artifact trajectory map and the preliminary HDR brightness channel map are combined to obtain the target feature map, that is, the number of channels of the target feature map is equal to the sum of the number of channels of the semantic segmentation feature map, the contrast feature map, the banding artifact trajectory map and the preliminary HDR brightness channel map.
[0105] S400: Input the target feature map into a mask generation network to obtain an image mask output by the mask generation network, wherein the pixel value of each pixel in the image mask reflects the probability that the pixel is located in a band artifact area.
[0106] The mask generation network is used to learn an image mask M to distinguish areas of different heights. The output layer of the mask generation network uses a sigmoid function, which means that the value of each point in the mask ranges from 0 to 1. The larger the value, the greater the probability that the point is located in the banding artifact area, and the greater the probability that the banding artifact removal process is required. The detailed structure of the mask generation network can be composed of two convolutional layers and 8 identical network blocks MasBlock, each block contains two convolutional layers and a ReLU activation layer, the kernel size of all convolutional layers is 3x3, the step size is 1, the resolution of all feature maps is consistent with the input image, except for the last convolutional layer whose output channel number is 1, and the output channel number of other convolutional layers is 64. It can be understood that the mask generation network can also adopt other network architecture designs. The image mask output by the mask generation network can be as follows Figure 5 As shown, Figure 5 The SDR image is on the left, and the corresponding image mask is on the right.
[0107] The parameters of the mask generation network are determined by pre-training so that the mask generation network has the ability to predict the probability that each pixel in the image is in the band artifact area based on the input image. The training process of the mask generation network will be described later.
[0108] S500: Fusing the preliminary HDR brightness channel map and the blur map according to the image mask to obtain a final HDR brightness channel map.
[0109] There are no banding artifacts in the blurred image, but the image details are blurred, while there are banding artifacts and image details in the preliminary HDR brightness channel image. Therefore, after using the mask generation network to generate the image mask that can distinguish between banded and non-banded areas in the image, the preliminary HDR brightness channel image and the blurred image are fused according to the image mask to obtain the final HDR brightness channel image.
[0110] Specifically, the preliminary HDR brightness channel map and the blur map are fused according to the image mask to obtain the final HDR brightness channel map, which can be achieved using a preset formula, and the preset formula is:
[0111]
[0112] Among them, Y HDR represents the final HDR brightness channel map, M is the image mask, and Y M is the fuzzy graph, Y D This is the preliminary HDR brightness channel map.
[0113] The preliminary HDR brightness channel map and the blur map are fused through the image mask that distinguishes between banded and non-banded areas in the image, which is equivalent to filtering in the banded areas and not filtering in the non-banded areas. Therefore, the banded artifacts in the image can be effectively eliminated while maintaining the image details.
[0114] like Figure 6 As shown, Figure 6 From left to right in the figure are the SDR image, the corresponding preliminary HDR brightness channel map, the blurred map, the final HDR brightness channel map, and the real HDR brightness channel map corresponding to the SDR image. It can be seen that the preliminary HDR brightness channel map does not form a good brightness transition, while the blurred map forms a transition, but the edge of the image is blurred. The final HDR brightness channel map obtained after fusion forms a brightness transition while retaining the details of the image, which is very similar to the real HDR brightness channel map.
[0115] After obtaining the final HDR brightness channel map, the method provided in this embodiment further includes the steps of:
[0116] S600: Acquire a target HDR color image according to the SDR image and the final HDR brightness channel map.
[0117] The step of acquiring a target HDR color image according to the SDR image and the final HDR brightness channel map includes:
[0118] Acquire a target brightness gain map according to the SDR image and the final HDR brightness channel map;
[0119] Convert the SDR image into a linear RGB channel, and obtain a target linear RGB image according to the target brightness gain map and the linear RGB channel of the SDR image;
[0120] The target linear RGB image is converted into a nonlinear RGB three-channel image to obtain the target HDR color image.
[0121] After obtaining the final HDR brightness channel map, the SDR image and the final HDR brightness channel map are linearized to the actual display brightness, and the gain map from the SDR image to the final HDR brightness channel map is calculated. The SDR image is then converted into a linear RGB channel, and the gain map is point-by-point multiplied with the linear RGB three channels to obtain the linear RGB value after gain, thereby generating the target linear RGB map. The target linear RGB map is then nonlinearly encoded and converted into a normal nonlinear RGB three-channel image, which is the target HDR color image. An example of the target color image is shown in FIG. Figure 7 As shown, Figure 7 From left to right in the figure are the SDR color image, the corresponding SDR brightness channel, the final HDR brightness channel map and the target HDR color image.
[0122] The training process of the first neural network and the mask generation network is described below. Specifically, before the SDR image is input into the first neural network, the process includes:
[0123] The step of acquiring a target HDR color image according to the SDR image and the final HDR brightness channel map includes:
[0124] Training the first neural network according to the multiple groups of first training data, and updating the parameters of the first neural network to first preliminary parameters according to a first objective function, wherein each group of the first training data includes a sample SDR image and an HDR color image corresponding to the sample SDR image, and the value of the first objective function is obtained by the difference between the brightness channel of the HDR color image corresponding to the sample SDR image and the output of the first neural network;
[0125] Fixing the parameters of the first neural network, training the mask generation network based on multiple sets of second training data, and updating the parameters of the mask generation network to second preliminary parameters according to a second objective function, wherein each set of the second training data includes a sample SDR image and an annotated mask corresponding to the sample SDR image, and the value of the second objective function is obtained by the difference between the annotated mask corresponding to the sample SDR image and the output of the mask generation network;
[0126] Fixing the parameters of the first neural network, training the mask generation network based on multiple sets of third training data, and updating the second preliminary parameters of the mask generation network to third preliminary parameters according to a third objective function, wherein each set of the third training data includes a sample SDR image and a real HDR color image corresponding to the sample SDR image, and the value of the third objective function is obtained by the difference between the brightness channel of the real HDR color image corresponding to the sample SDR image and the final HDR brightness channel map corresponding to the sample SDR image;
[0127] The final training of the first neural network parameters and the mask generation network is performed based on multiple groups of fourth training data to determine the final parameters of the first neural network parameters and the mask generation network.
[0128] First, the first neural network is trained separately, and multiple groups of first training data are used to train the first neural network, and the parameters of the first neural network are updated to the first preliminary parameters. Specifically, a sample SDR image in the first training data is input into the first neural network each time, and the value of the first objective function is obtained according to the output of the first neural network and the brightness channel of the HDR color image corresponding to the input sample SDR image. Specifically, the value of the first objective function can be obtained by calculating the L1 norm between the brightness channel of the HDR color image corresponding to the sample SDR image and the output of the first neural network. It can be understood that the closer the output of the first neural network is to the brightness channel of the HDR color image corresponding to the sample SDR image, the better the performance of the first neural network. Therefore, each time the parameters of the first neural network are updated to the value that minimizes the value of the first objective function. When the parameters of the first neural network converge, the training ends, and the parameters of the first neural network are the first preliminary parameters.
[0129] Then, the parameters of the first neural network are fixed, and the mask generation network is pre-trained. The pre-training of the mask generation network is completed using multiple sets of second training data. Specifically, each time a sample SDR image in the second training data is subjected to steps S100-S300, the corresponding target feature map is obtained, and then input into the mask generation network, and the value of the second objective function is obtained according to the difference between the output of the mask generation network and the annotated mask corresponding to the sample SDR image. The annotated mask corresponding to the sample SDR image can be extracted by the following formula:
[0130]
[0131] Among them, M R (x, y) represents the pixel value of the pixel with coordinates (x, y) in the annotation mask, Y M(x, y) represents the pixel value of the pixel with coordinates (x, y) in the blurred image corresponding to the sample SDR image. D (x, y) represents the pixel value of the pixel with coordinates (x, y) in the preliminary HDR brightness channel map corresponding to the sample SDR image.
[0132] The value of the second objective function can specifically be the L1 norm between the output of the mask generation network and the annotated mask corresponding to the sample SDR image. In the training of the mask generation network, in order to quickly approach the ideal parameters of the mask generation network, the mask generation network is first pre-trained according to the annotated mask. When the parameters of the mask generation network converge, the pre-training of the mask generation network ends, and the parameters of the mask generation network are the second preliminary parameters.
[0133] After the mask generation network is pre-trained, the mask generation network is formally trained. Specifically, the parameters of the first neural network are fixed, and the mask generation network is trained based on multiple sets of third training data. Each time, a set of sample SDR images in the third training data is subjected to steps S100-S500 to obtain the final HDR brightness channel map corresponding to the sample SDR image, and the value of the third objective function is obtained according to the difference between the final HDR brightness channel map corresponding to the sample SDR image and the brightness channel of the real HDR color image corresponding to the sample SDR image. Specifically, the value of the third objective function is the L1 norm between the final HDR brightness channel map corresponding to the sample SDR image and the brightness channel of the real HDR color image corresponding to the sample SDR image. When the parameters of the mask generation network converge, the formal training of the mask generation network ends, and the parameters of the mask generation network are updated from the second preliminary parameters to the third preliminary parameters.
[0134] Finally, the first neural network and the mask generation network are jointly fine-tuned, that is, the parameters of the two are not fixed, and then the parameters of the first neural network and the mask generation network are updated according to multiple groups of fourth training data, each group of the fourth training data includes a sample SDR image and a real HDR image corresponding to the sample SDR image. Specifically, it can be based on the sample SDR image. Adopt steps S100-S600 to obtain a target HDR color image, obtain the value of the fourth objective function according to the difference between the target HDR color image and the real HDR image, and then jointly update the parameters of the first neural network and the mask generation network.
[0135] In summary, this embodiment provides a video inverse tone mapping method capable of removing banding artifacts. First, a preliminary HDR brightness channel map of the SDR image is obtained based on a neural network model, and then a variety of features related to banding artifacts are extracted through the SDR image and the preliminary HDR brightness channel map, including semantic segmentation of the SDR image to obtain a semantic segmentation feature map, filtering the preliminary HDR brightness channel map to obtain a blur map, obtaining a contrast feature map based on the blur map, obtaining a banding artifact trajectory map for the gradient of each pixel point of the preliminary HDR brightness channel map, and combining the semantic segmentation feature map, the contrast feature map and the banding artifact trajectory map. The target feature map is obtained by combining the SDR image and the trace map, and an image mask is obtained based on the target feature map, which characterizes the position and severity of the banding artifacts in the image. Since there are no banding artifacts in the blurred map, but the image details are blurred, and there are banding artifacts in the preliminary HDR brightness channel map, but there are image details, the final HDR brightness channel map obtained by fusing the blurred map and the preliminary HDR brightness channel map based on the image mask can effectively eliminate the banding artifacts in the image while maintaining the image details, so that the target HDR color image obtained according to the SDR image and the final HDR brightness channel map can maintain the image details and have no banding artifacts.
[0136] It should be understood that, although the various steps in the flowcharts given in the accompanying drawings of the present specification are displayed in sequence according to the indications of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0137] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0138] Embodiment 2
[0139] Based on the above embodiments, the present invention also provides a video inverse tone mapping device capable of removing banding artifacts. Figure 8 As shown, the video inverse tone mapping device capable of removing banding artifacts includes:
[0140] A preliminary HDR brightness channel acquisition module, the preliminary HDR brightness channel acquisition module is used to acquire an SDR image, input the SDR image into a first neural network, and acquire a preliminary HDR brightness channel map output by the first neural network, as specifically described in Embodiment 1;
[0141] A feature extraction module, wherein the feature extraction module is used to input the SDR image into a semantic segmentation model, obtain a semantic segmentation feature map output by the semantic segmentation model, filter the preliminary HDR brightness channel map to obtain a blur map, obtain a contrast feature map according to the difference between the blur map and the preliminary HDR brightness channel map, and obtain a banded artifact trajectory map according to the gradient of each pixel point of the preliminary HDR brightness channel map, as described in Example 1;
[0142] A feature fusion module, wherein the feature fusion module is used to combine the preliminary HDR brightness channel map, the semantic segmentation feature map, the contrast feature map and the banded artifact trajectory map to obtain a target feature map, as specifically described in Embodiment 1;
[0143] A mask generation module, wherein the mask generation module is used to input the target feature map into a mask generation network to obtain an image mask output by the mask generation network, wherein the pixel value of each pixel in the image mask reflects the probability that the pixel is located in a banded artifact area, as specifically described in the first embodiment;
[0144] A final HDR brightness channel acquisition module, wherein the final HDR brightness channel acquisition module is used to fuse the preliminary HDR brightness channel map and the blur map according to the image mask to obtain a final HDR brightness channel map, as specifically described in Embodiment 1;
[0145] An HDR color image acquisition module, wherein the HDR color image acquisition model is used to acquire a target HDR color image according to the SDR image and the final HDR brightness channel map, as specifically described in Embodiment 1.
[0146] Embodiment 3
[0147] Based on the above embodiments, the present invention also provides a terminal, such as Fig. 9 As shown, the terminal includes a processor 10 and a memory 20. Fig. 9 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.
[0148] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a video inverse tone mapping program 30 capable of removing banding artifacts is stored on the memory 20, and the video inverse tone mapping program 30 capable of removing banding artifacts can be executed by the processor 10, thereby realizing the video inverse tone mapping method capable of removing banding artifacts in the present application.
[0149] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other chip, used to run the program code or process data stored in the memory 20, such as executing the classification method.
[0150] In one embodiment, when the processor 10 executes the video inverse tone mapping program 30 capable of removing banding artifacts in the memory 20, the following steps are implemented:
[0151] Acquire an SDR image, input the SDR image into a first neural network, and acquire a preliminary HDR brightness channel map output by the first neural network;
[0152] Input the SDR image into a semantic segmentation model, obtain a semantic segmentation feature map output by the semantic segmentation model, filter the preliminary HDR brightness channel map to obtain a blur map, obtain a contrast feature map according to the difference between the blur map and the preliminary HDR brightness channel map, and obtain a band artifact trajectory map according to the gradient of each pixel point of the preliminary HDR brightness channel map;
[0153] Combine the preliminary HDR brightness channel map, the semantic segmentation feature map, the contrast feature map, and the band artifact trajectory map to obtain a target feature map;
[0154] Inputting the target feature map into a mask generation network, obtaining an image mask output by the mask generation network, wherein the pixel value of each pixel in the image mask reflects the probability that the pixel is located in a banded artifact area;
[0155] fusing the preliminary HDR brightness channel map and the blur map according to the image mask to obtain a final HDR brightness channel map;
[0156] A target HDR color image is obtained according to the SDR image and the final HDR brightness channel map.
[0157] The step of obtaining a band artifact trajectory map according to the gradient of each pixel point of the preliminary HDR brightness channel map includes:
[0158] Obtaining an initial gradient of each pixel point according to the adjacent pixels of each pixel point on the preliminary HDR brightness channel map;
[0159] Obtaining the main direction intensity of the gradient of each pixel point on the preliminary HDR brightness channel map according to the initial gradient of each pixel point on the preliminary HDR brightness channel map;
[0160] The pixel value of each pixel point on the band-like artifact trajectory map is obtained according to the initial gradient and the main direction intensity corresponding to each pixel point on the initial brightness channel map.
[0161] Wherein, obtaining the main direction intensity of the gradient of each pixel point of the preliminary HDR brightness channel image according to the initial gradient of each pixel point on the preliminary HDR brightness channel image includes:
[0162] When the initial gradient of the target pixel point on the preliminary HDR brightness channel map in the target direction is within a preset range, determining the intermediate gradient of the target pixel point in the target direction as the initial gradient;
[0163] When the gradient of the target pixel point on the preliminary HDR brightness channel map in the target direction is not within the preset range, determining that the intermediate gradient of the target pixel point in the target direction is 0;
[0164] Determine the intermediate gradients of the target pixel in each target direction, and use the largest intermediate gradient as the final gradient of the target pixel;
[0165] The main direction intensity of each pixel is determined according to the final gradient of each pixel on the preliminary HDR brightness channel map.
[0166] Wherein, determining the main direction intensity of each pixel point according to the final gradient of each pixel point on the preliminary HDR brightness channel map includes:
[0167] Generate a gradient map according to the initial gradient of each pixel point on the preliminary HDR brightness channel map;
[0168] The gradient block where each pixel point is located is deformed to obtain the deformation vector of each pixel point, and the target matrix is constructed according to the deformation vector of each pixel point;
[0169] The main direction intensity corresponding to each pixel point is obtained according to the eigenvalue of the target matrix corresponding to each pixel point and the final gradient.
[0170] The step of fusing the preliminary HDR brightness channel map and the blur map according to the image mask to obtain a final HDR brightness channel map includes:
[0171] Obtain the final HDR brightness channel map according to a preset formula;
[0172] The preset formula is:
[0173]
[0174] Among them, YHDR represents the final HDR brightness channel map, M is the image mask, and Y M is the fuzzy graph, Y D This is the preliminary HDR brightness channel map.
[0175] Wherein, obtaining a target HDR color image according to the SDR image and the final HDR brightness channel map includes:
[0176] Acquire a target brightness gain map according to the SDR image and the final HDR brightness channel map;
[0177] Convert the SDR image into a linear RGB channel, and obtain a target linear RGB image according to the target brightness gain map and the linear RGB channel of the SDR image;
[0178] The target linear RGB image is converted into a nonlinear RGB three-channel image to obtain the target HDR color image.
[0179] Wherein, before inputting the SDR image into the first neural network, the method includes:
[0180] Training the first neural network according to the multiple groups of first training data, and updating the parameters of the first neural network to first preliminary parameters according to a first objective function, wherein each group of the first training data includes a sample SDR image and an HDR color image corresponding to the sample SDR image, and the value of the first objective function is obtained by the difference between the brightness channel of the HDR color image corresponding to the sample SDR image and the output of the first neural network;
[0181] Fixing the parameters of the first neural network, training the mask generation network based on multiple sets of second training data, and updating the parameters of the mask generation network to second preliminary parameters according to a second objective function, wherein each set of the second training data includes a sample SDR image and an annotated mask corresponding to the sample SDR image, and the value of the second objective function is obtained by the difference between the annotated mask corresponding to the sample SDR image and the output of the mask generation network;
[0182] Fixing the parameters of the first neural network, training the mask generation network based on multiple sets of third training data, and updating the second preliminary parameters of the mask generation network to third preliminary parameters according to a third objective function, wherein each set of the third training data includes a sample SDR image and a real HDR color image corresponding to the sample SDR image, and the value of the third objective function is obtained by the difference between the brightness channel of the real HDR color image corresponding to the sample SDR image and the final HDR brightness channel map corresponding to the sample SDR image;
[0183] The final training of the first neural network parameters and the mask generation network is performed based on multiple groups of fourth training data to determine the final parameters of the first neural network parameters and the mask generation network.
[0184] Embodiment 4
[0185] The present invention also provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the video inverse tone mapping method capable of removing banding artifacts as described above.
[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A video inverse tone mapping method capable of removing banding artifacts, characterized in that: The method comprises: Acquire an SDR image, input the SDR image into a first neural network, and acquire a preliminary HDR brightness channel map output by the first neural network; Input the SDR image into a semantic segmentation model, obtain a semantic segmentation feature map output by the semantic segmentation model, filter the preliminary HDR brightness channel map to obtain a blur map, obtain a contrast feature map according to the difference between the blur map and the preliminary HDR brightness channel map, and obtain a band artifact trajectory map according to the gradient of each pixel point of the preliminary HDR brightness channel map; Combine the preliminary HDR brightness channel map, the semantic segmentation feature map, the contrast feature map, and the band artifact trajectory map to obtain a target feature map; Inputting the target feature map into a mask generation network, obtaining an image mask output by the mask generation network, wherein the pixel value of each pixel in the image mask reflects the probability that the pixel is located in a banded artifact area; fusing the preliminary HDR brightness channel map and the blur map according to the image mask to obtain a final HDR brightness channel map; A target HDR color image is obtained according to the SDR image and the final HDR brightness channel map.
2. The video inverse tone mapping method capable of removing banding artifacts according to claim 1, characterized in that: The step of obtaining a banded artifact trajectory map according to the gradient of each pixel point of the preliminary HDR brightness channel map comprises: Obtaining an initial gradient of each pixel point according to the adjacent pixels of each pixel point on the preliminary HDR brightness channel map; Obtaining the main direction intensity of the gradient of each pixel point on the preliminary HDR brightness channel map according to the initial gradient of each pixel point on the preliminary HDR brightness channel map; The pixel value of each pixel point on the band-like artifact trajectory map is obtained according to the initial gradient and the main direction intensity corresponding to each pixel point on the initial brightness channel map.
3. The video inverse tone mapping method capable of removing banding artifacts according to claim 2, characterized in that: The obtaining the main direction intensity of the gradient of each pixel point of the preliminary HDR brightness channel image according to the initial gradient of each pixel point on the preliminary HDR brightness channel image comprises: When the initial gradient of the target pixel point on the preliminary HDR brightness channel map in the target direction is within a preset range, determining the intermediate gradient of the target pixel point in the target direction as the initial gradient; When the gradient of the target pixel point on the preliminary HDR brightness channel map in the target direction is not within the preset range, determining that the intermediate gradient of the target pixel point in the target direction is 0; Determine the intermediate gradients of the target pixel in each target direction, and use the largest intermediate gradient as the final gradient of the target pixel; The main direction intensity of each pixel is determined according to the final gradient of each pixel on the preliminary HDR brightness channel map.
4. The video inverse tone mapping method capable of removing banding artifacts according to claim 3, characterized in that: Determining the main direction intensity of each pixel point according to the final gradient of each pixel point on the preliminary HDR brightness channel map includes: Generate a gradient map according to the initial gradient of each pixel point on the preliminary HDR brightness channel map; The gradient block where each pixel point is located is deformed to obtain the deformation vector of each pixel point, and the target matrix is constructed according to the deformation vector of each pixel point; The main direction intensity corresponding to each pixel point is obtained according to the eigenvalue of the target matrix corresponding to each pixel point and the final gradient.
5. The video inverse tone mapping method capable of removing banding artifacts according to claim 1, characterized in that: The step of fusing the preliminary HDR brightness channel map and the blur map according to the image mask to obtain a final HDR brightness channel map comprises: Obtain the final HDR brightness channel map according to a preset formula; The preset formula is: Among them, Y HDR represents the final HDR brightness channel map, M is the image mask, and Y M is the fuzzy graph, Y D This is the preliminary HDR brightness channel map.
6. The video inverse tone mapping method capable of removing banding artifacts according to claim 1, characterized in that: The step of acquiring a target HDR color image according to the SDR image and the final HDR brightness channel map includes: Acquire a target brightness gain map according to the SDR image and the final HDR brightness channel map; Convert the SDR image into a linear RGB channel, and obtain a target linear RGB image according to the target brightness gain map and the linear RGB channel of the SDR image; The target linear RGB image is converted into a nonlinear RGB three-channel image to obtain the target HDR color image.
7. The video inverse tone mapping method capable of removing banding artifacts according to claim 1, characterized in that: Before inputting the SDR image into the first neural network, the method comprises: Training the first neural network according to multiple groups of first training data, and updating the parameters of the first neural network to first preliminary parameters according to a first objective function, wherein each group of the first training data includes a sample SDR image and an HDR color image corresponding to the sample SDR image, and the value of the first objective function is obtained by the difference between the brightness channel of the HDR color image corresponding to the sample SDR image and the output of the first neural network; Fixing the parameters of the first neural network, training the mask generation network based on multiple sets of second training data, and updating the parameters of the mask generation network to second preliminary parameters according to a second objective function, wherein each set of the second training data includes a sample SDR image and an annotated mask corresponding to the sample SDR image, and the value of the second objective function is obtained by the difference between the annotated mask corresponding to the sample SDR image and the output of the mask generation network; Fixing the parameters of the first neural network, training the mask generation network based on multiple sets of third training data, and updating the second preliminary parameters of the mask generation network to third preliminary parameters according to a third objective function, wherein each set of the third training data includes a sample SDR image and a real HDR color image corresponding to the sample SDR image, and the value of the third objective function is obtained by the difference between the brightness channel of the real HDR color image corresponding to the sample SDR image and the final HDR brightness channel map corresponding to the sample SDR image; The final training of the first neural network parameters and the mask generation network is performed based on multiple groups of fourth training data to determine the final parameters of the first neural network parameters and the mask generation network.
8. A video inverse tone mapping device capable of removing banding artifacts, characterized in that: include: A preliminary HDR brightness channel acquisition module, the preliminary HDR brightness channel acquisition module is used to acquire an SDR image, input the SDR image into a first neural network, and acquire a preliminary HDR brightness channel map output by the first neural network; A feature extraction module, wherein the feature extraction module is used to input the SDR image into a semantic segmentation model, obtain a semantic segmentation feature map output by the semantic segmentation model, filter the preliminary HDR brightness channel map to obtain a blur map, obtain a contrast feature map according to the difference between the blur map and the preliminary HDR brightness channel map, and obtain a band artifact trajectory map according to the gradient of each pixel point of the preliminary HDR brightness channel map; A feature fusion module, wherein the feature fusion module is used to combine the preliminary HDR brightness channel map, the semantic segmentation feature map, the contrast feature map and the banded artifact trajectory map to obtain a target feature map; A mask generation module, wherein the mask generation module is used to input the target feature map into a mask generation network to obtain an image mask output by the mask generation network, wherein the pixel value of each pixel in the image mask reflects the probability that the pixel is located in a banded artifact area; A final HDR brightness channel acquisition module, wherein the final HDR brightness channel acquisition module is used to fuse the preliminary HDR brightness channel map and the blur map according to the image mask to obtain a final HDR brightness channel map; An HDR color image acquisition module, wherein the HDR color image acquisition model is used to acquire a target HDR color image according to the SDR image and the final HDR brightness channel map.
9. A terminal, characterized in that: The terminal includes: a processor, a computer-readable storage medium communicatively connected to the processor, the computer-readable storage medium being suitable for storing a plurality of instructions, and the processor being suitable for calling the instructions in the computer-readable storage medium to execute the steps of the video inverse tone mapping method capable of removing banding artifacts as described in any one of claims 1-7 above.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the video inverse tone mapping method capable of removing banding artifacts as described in any one of claims 1-7.
