Coding method, training method and device of high dynamic range video

Abstract

The invention provides a high dynamic range video coding method, training method and device, which can be applied to the technical field of multimedia processing. The high dynamic range video coding method comprises the following steps: acquiring a high dynamic range video; processing a first pixel domain feature of a video frame of the high dynamic range video by using a first deep residual network, and outputting a segmentation threshold parameter of a code rate coding parameter mapping function; performing block processing on the video frame to obtain a plurality of image blocks; processing the second pixel domain features of the plurality of image blocks by using a second deep residual network, and outputting coefficients of code rate coding parameter mapping functions corresponding to the plurality of image blocks respectively; based on a code rate coding parameter mapping function, determining coding parameters of the plurality of image blocks according to the respective pre-allocated bit parameters of the plurality of image blocks; and performing coding processing on the video frame of the high dynamic range video based on the coding parameters of the plurality of image blocks.

Description

Technical Field

[0001] The present disclosure relates to the field of multimedia processing technology, and in particular to a high dynamic range video encoding method, training method and device. Background Art

[0002] With the rapid development of artificial intelligence and display technology, high dynamic range imaging technology has become one of the important technologies to enhance the visual experience. High dynamic range imaging (HDR) technology attempts to capture the real world from 10 -8 cd / m 2 to 10 9 cd / m 2 This capability allows for the creation of images and videos with more vivid colors, rich shadow and highlight details, and a superior overall visual experience. Unlike traditional 8-bit Standard Dynamic Range (SDR) video images, high dynamic range video images require a higher bit depth or floating point value to store the necessary information. Two common high dynamic range formats are Radiance RGBE and OpenEXR, with the extensions ".hdr" and ".exr". High dynamic range video can accurately record the physical brightness of the scene, but it will generate more data than standard dynamic range video, which brings new challenges to video compression and transmission.

[0003] In the process of implementing the concept of the present disclosure, the inventors found that in the related art, the rate control efficiency of intra-frame coding for high dynamic range video is low, resulting in low rate control accuracy and coding efficiency of high dynamic range video and low quality of high dynamic range video. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a high dynamic range video encoding method, training method and device.

[0005] According to a first aspect of the present disclosure, a method for encoding a high dynamic range video is provided, comprising: obtaining a high dynamic range video; using a first deep residual network to process a first pixel domain feature of a video frame of the high dynamic range video, and outputting a segmented threshold parameter of a rate coding parameter mapping function, wherein the first deep residual network is trained using the segmented threshold parameter of the rate coding parameter mapping function of a sample video frame as a label; performing block processing on the video frame to obtain a plurality of image blocks; using a second deep residual network to process a second pixel domain feature of the plurality of image blocks, and outputting coefficients of the rate coding parameter mapping function corresponding to each of the plurality of image blocks, wherein the second deep residual network is trained using the segmented threshold parameter of the rate coding parameter mapping function of a sample video frame as a label; performing block processing on the video frame to obtain a plurality of image blocks; using a second deep residual network to process a second pixel domain feature of the plurality of image blocks, and outputting coefficients of the rate coding parameter mapping function corresponding to each of the plurality of image blocks, The residual network is trained by taking the coefficients of the rate coding parameter mapping function of the sample image block as labels; based on the rate coding parameter mapping function, the coding parameters of the multiple image blocks are determined according to the pre-allocated bit parameters of each of the multiple image blocks, wherein the rate coding parameter mapping function is constructed by taking the pre-allocated bit parameters of the multiple image blocks as independent variables, the coding parameters of the multiple image blocks as dependent variables, and the coefficients of the rate coding parameter mapping function corresponding to multiple segments of pre-allocated bit parameters; the multiple segments of pre-allocated bit parameters are divided according to the segmentation threshold parameters; and the video frames of the high dynamic range video are encoded based on the coding parameters of the multiple image blocks.

[0006] A second aspect of the present invention provides a training method for a deep residual network, the method comprising: obtaining training samples of the deep residual network, wherein the deep residual network comprises a first deep residual network and a second deep residual network; using the first deep residual network to process the first pixel domain features of the sample video frame of the training sample, and outputting a prediction threshold parameter of a rate coding parameter mapping function; performing block processing on the sample video frame of the training sample to obtain a plurality of sample image blocks; using the second deep residual network to process the second pixel domain features of the plurality of sample image blocks, and outputting prediction coefficients of the rate coding parameter mapping function corresponding to each of the plurality of sample image blocks; determining a loss value based on the target loss function and the training sample and the training sample label; adjusting the prediction threshold parameter of the rate coding parameter mapping function and the prediction coefficient of the rate coding parameter mapping function based on the loss value to obtain a trained deep residual network.

[0007] A third aspect of the present invention provides a high dynamic range video encoding device, the device comprising a first acquisition module, a first extraction module, a first blocking module, a second extraction module, a first determination module and an encoding module.

[0008] The first acquisition module is used to acquire high dynamic range video.

[0009] The first extraction module is used to process the first pixel domain features of the video frame of the high dynamic range video using the first deep residual network, and output the segmentation threshold parameters of the bit rate coding parameter mapping function, wherein the first deep residual network is trained with the segmentation threshold parameters of the bit rate coding parameter mapping function of the sample video frame as labels.

[0010] The first block division module is used to perform block processing on the above video frame to obtain multiple image blocks.

[0011] The second extraction module is used to process the second pixel domain features of the above-mentioned image blocks by using a second deep residual network, and output coefficients of the rate coding parameter mapping function corresponding to each of the above-mentioned multiple image blocks, wherein the above-mentioned second deep residual network is trained with the coefficients of the rate coding parameter mapping function of the sample image block as labels.

[0012] The first determination module is used to determine the coding parameters of the above-mentioned multiple image blocks according to the respective pre-allocated bit parameters of the above-mentioned multiple image blocks, wherein the above-mentioned rate coding parameter mapping function is constructed by taking the pre-allocated bit parameters of the above-mentioned multiple image blocks as independent variables and the coding parameters of the above-mentioned multiple image blocks as dependent variables, and the coefficients of the above-mentioned rate coding parameter mapping function corresponding to each of the multiple segments of pre-allocated bit parameters; the above-mentioned multiple segments of pre-allocated bit parameters are divided according to the above-mentioned segmentation threshold parameters.

[0013] The encoding module is used to encode the video frames of the high dynamic range video based on the encoding parameters of the multiple image blocks.

[0014] According to the encoding method, training method and device of high dynamic range video provided by the present invention, by using a first deep residual network to process the first pixel domain features of the video frame of the acquired high dynamic range video, the segmentation threshold parameters of the rate coding parameter mapping function can be obtained, and the video frame is processed into blocks to obtain multiple image blocks. The second pixel domain features of the multiple image blocks are processed by a second deep residual network to obtain the coefficients of the rate coding parameter mapping function. Based on the rate coding parameter mapping function, the coding parameters of the multiple image blocks can be determined according to the pre-allocated bit parameters of each of the multiple image blocks, and the video frame of the high dynamic range video is encoded based on the coding parameters of the multiple image blocks to obtain the bit rates corresponding to the multiple image blocks, thereby improving the rate control efficiency of intra-frame coding of the high dynamic range video, thereby improving the rate control accuracy and coding efficiency of the high dynamic range video and improving the quality of the high dynamic range video. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0016] Figure 1 A diagram schematically shows an application scenario of a method for encoding high dynamic range video according to an embodiment of the present disclosure;

[0017] Figure 2 A flowchart of a method for encoding a high dynamic range video according to an embodiment of the present disclosure is schematically shown;

[0018] Figure 3 A schematic diagram of a deep residual network according to an embodiment of the present disclosure is schematically shown;

[0019] Figure 4 A flowchart of a method for training a deep residual network according to an embodiment of the present disclosure is schematically shown;

[0020] Figure 5 The structure block diagram of the high dynamic range video encoding device according to the embodiment of the present disclosure is schematically shown;

[0021] Figure 6 The structure block diagram of the training device of the deep residual network according to the embodiment of the present disclosure is schematically shown;

[0022] Figure 7 A block diagram of an electronic device suitable for implementing a method for encoding a high dynamic range video according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0023] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0024] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0025] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0026] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0027] In the process of implementing the present disclosure, it is found that in order to cope with the challenge of high dynamic range video with more data, a series of standardized and non-standardized coding tools have been developed in the related art, some of which have been integrated into the latest Versatile Video Coding (VVC) standard. Rate control technology is a key technology in video coding, which aims to find an optimal bit allocation strategy to ensure that the video coding bit rate is within the bit allocation budget while achieving the best rate distortion (RD) performance. In common video coding standards, such as HEVC / VVC, the rate control algorithm can be designed for three different coding structures, including All Intra (AI), Low Delay (LD) and Random Access (RA), to effectively cope with various practical coding application scenarios. The related technology is mainly designed for LD and RA coding structures, ignoring the important role of intra-frame coding in allocating remaining bits and controlling the quality of subsequent frames, thereby limiting the potential improvement of high dynamic range video coding performance.

[0028] In addition, the intra-frame rate control algorithm in the related art does not take into account the differences in encoding characteristics between high dynamic range video and standard dynamic range video, resulting in the existing intra-frame rate control algorithm for standard dynamic range content being difficult to directly use for intra-frame rate control of high dynamic range content.

[0029] In view of this, an embodiment of the present disclosure provides a method for encoding a high dynamic range video, including: obtaining a high dynamic range video; using a first deep residual network to process a first pixel domain feature of a video frame of the high dynamic range video, and outputting a segmented threshold parameter of a rate coding parameter mapping function, wherein the first deep residual network is trained using the segmented threshold parameter of the rate coding parameter mapping function of a sample video frame as a label; performing block processing on the video frame to obtain a plurality of image blocks; using a second deep residual network to process a second pixel domain feature of the plurality of image blocks, and outputting coefficients of the rate coding parameter mapping function corresponding to each of the plurality of image blocks, wherein In the method, the second deep residual network is trained with the coefficients of the rate coding parameter mapping function of the sample image block as labels; based on the rate coding parameter mapping function, the coding parameters of the multiple image blocks are determined according to the pre-allocated bit parameters of each of the multiple image blocks, wherein the rate coding parameter mapping function is constructed by taking the pre-allocated bit parameters of the multiple image blocks as independent variables and the coding parameters of the multiple image blocks as dependent variables, and the coefficients of the rate coding parameter mapping function corresponding to each of the multiple segments of pre-allocated bit parameters; the multiple segments of pre-allocated bit parameters are divided according to the segmentation threshold parameter; and the video frames of the high dynamic range video are encoded based on the coding parameters of the multiple image blocks.

[0030] Figure 1 The application scenario diagram of the high dynamic range video encoding method according to the embodiment of the present disclosure is schematically shown.

[0031] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0032] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0033] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0034] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0035] It should be noted that the high dynamic range video encoding method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the high dynamic range video encoding device provided in the embodiment of the present disclosure can generally be set in the server 105. The high dynamic range video encoding method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the high dynamic range video encoding device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0036] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0037] The following will be based on Figure 1 The scene described by Figure 2~Figure 4 The high dynamic range video encoding method of the disclosed embodiment is described in detail.

[0038] Figure 2 The flowchart of the high dynamic range video encoding method according to the embodiment of the present disclosure is schematically shown.

[0039] like Figure 2 As shown, the high dynamic range video encoding method 200 of this embodiment includes operations S210 to S260.

[0040] In operation S210, a high dynamic range video is acquired.

[0041] In operation S220, a first pixel domain feature of a video frame of a high dynamic range video is processed using a first deep residual network, and a segmented threshold parameter of a rate coding parameter mapping function is output.

[0042] In operation S230, the video frame is divided into blocks to obtain a plurality of image blocks.

[0043] In operation S240, the second pixel domain features of the plurality of image blocks are processed using a second deep residual network, and coefficients of rate coding parameter mapping functions corresponding to each of the plurality of image blocks are output.

[0044] In operation S250, encoding parameters of the plurality of image blocks are determined according to the respective pre-allocated bit parameters of the plurality of image blocks based on a rate encoding parameter mapping function.

[0045] In operation S260, a video frame of the high dynamic range video is encoded based on the encoding parameters of the plurality of image blocks.

[0046] According to an embodiment of the present disclosure, a high dynamic range video may represent a video formed using a high dynamic range imaging technique, which provides a richer visual experience than a traditional standard dynamic range video by extending the brightness and color range. A high dynamic range video may include multiple video frames.

[0047] According to an embodiment of the present disclosure, the deep residual network may be a deep residual network based on wavelet convolution. The deep residual network may include a first deep residual network and a second deep residual network. The first deep residual network may be obtained by training with a segmented threshold parameter of a rate coding parameter mapping function of a sample video frame as a label. The second deep residual network may be obtained by training with a coefficient of a rate coding parameter mapping function of a sample image block as a label.

[0048] According to an embodiment of the present disclosure, the first pixel domain feature of the video frame of the high dynamic range video can be obtained by performing a 3x3 convolution on the video frame of the high dynamic range video through a first convolution layer. The first pixel domain feature of the video frame of the high dynamic range video can be processed using a first deep residual network.

[0049] According to an embodiment of the present disclosure, a plurality of image blocks may be obtained by performing block processing on a video frame. The image block may be a coding tree unit. The video frame of the high dynamic range video may be cut into 128x128 image blocks in a left-to-right and top-to-bottom order as input to the second deep residual network.

[0050] According to an embodiment of the present disclosure, the rate coding parameter mapping function may be a piecewise function. For example, the rate coding parameter mapping function may be a piecewise function. Function, as follows formula (1):

[0051] (1)

[0052] in, Indicates the minimum pre-allocated bit parameter of a video frame or image block, Indicates the maximum pre-allocated bit parameter of a video frame or image block; , , , Represents the endpoints of an interval of a piecewise function.

[0053] According to an embodiment of the present disclosure, segment Number and segmentation of function intervals The fitting accuracy of the function is positively correlated. As the number of intervals increases, the difficulty of predicting the segmented threshold parameters and coefficients of the rate coding parameter mapping function becomes increasingly greater. In order to strike a balance between the fitting accuracy and the difficulty of predicting the segmented threshold parameters and coefficients of the rate coding parameter mapping function, the present disclosure conducts experiments to explore the optimal number of intervals, and the results are shown in Table 1.

[0054] Table 1 Fitting accuracy of rate coding parameter mapping function with different number of intervals

[0055]

[0056] According to an embodiment of the present disclosure, as shown in Table 1, BalloonFestival, EBU_04_Hurdles, ShowGirls, EBU_06_Starting, and S12-Sunrise represent different sequences. For different sequences, when the number of intervals is set to be greater than 2, the fitting accuracy is slightly improved, but the difficulty of predicting the segmentation threshold parameters and coefficients of the rate coding parameter mapping function will increase rapidly. Therefore, in an embodiment of the present disclosure, a rate coding parameter mapping function with two intervals can be used, as shown in the following formula (2):

[0057] (2)

[0058] in, and The coefficients of the rate coding parameter mapping function representing the first interval; and represents the coefficients of the rate coding parameter mapping function for the second interval; Represents the segmentation threshold parameter of the rate coding parameter mapping function.

[0059] According to an embodiment of the present disclosure, It can be regarded as the low bit rate segment and It can be regarded as the high bit rate segment of the bit rate encoding parameter mapping function.

[0060] According to an embodiment of the present disclosure, the second pixel domain features of the multiple image blocks can be obtained by performing 3x3 convolution on the multiple image blocks through the first convolution layer. By processing the second pixel domain features of the multiple image blocks using the second deep residual network, the coefficients of the rate coding parameter mapping function corresponding to each of the multiple image blocks can be output.

[0061] According to an embodiment of the present disclosure, the rate coding parameter mapping function can be constructed by taking the pre-allocated bit parameters of multiple image blocks as independent variables, the coding parameters of multiple image blocks as dependent variables, and the coefficients of the rate coding parameter mapping function corresponding to each of the multiple segments of pre-allocated bit parameters.

[0062] According to an embodiment of the present disclosure, the multiple segments of pre-allocated bit parameters may be divided according to segmentation threshold parameters. In one embodiment, the multiple segments of pre-allocated bit parameters may be the same.

[0063] According to an embodiment of the present disclosure, video frames of a high dynamic range video may be encoded based on encoding parameters of multiple image blocks, so that encoding may be performed based on encoding parameters of different image blocks to obtain bit rates corresponding to different image blocks.

[0064] According to an embodiment of the present disclosure, by using a first deep residual network to process the first pixel domain features of the acquired video frame of the high dynamic range video, a segmented threshold parameter of the rate coding parameter mapping function can be obtained, and the video frame is processed into blocks to obtain multiple image blocks. The second pixel domain features of the multiple image blocks are processed using a second deep residual network to obtain coefficients of the rate coding parameter mapping function. Based on the rate coding parameter mapping function, the coding parameters of the multiple image blocks can be determined according to the pre-allocated bit parameters of each of the multiple image blocks. The video frame of the high dynamic range video is encoded based on the coding parameters of the multiple image blocks, and the bit rates corresponding to the multiple image blocks can be obtained, thereby improving the rate control accuracy of intra-frame coding of the high dynamic range video, and then improving the coding efficiency of the high dynamic range video and the quality of the high dynamic range video.

[0065] Figure 3 A schematic diagram of a deep residual network according to an embodiment of the present disclosure is schematically shown.

[0066] like Figure 3 As shown, the deep residual network includes an input module, a multi-level wavelet transform module and a parameter regression module.

[0067] According to an embodiment of the present disclosure, when the deep residual network is characterized as a second deep residual network, a video frame of a high dynamic range video can be input into an input module, and the video frame can be processed in blocks to obtain multiple image blocks. Multiple image blocks are input into a multi-level wavelet transform module, and the first convolution layer C1 performs convolution processing on multiple image blocks to obtain pixel domain features. Each wavelet transform module in the multi-level wavelet transform module may include a residual submodule and a discrete wavelet submodule cascaded with each other. The multi-level wavelet transform may include three wavelet transform modules, the first wavelet transform module includes a first residual submodule R1, a second residual submodule R2 and a first discrete wavelet submodule W1, the second wavelet transform module includes a third residual submodule R3, a fourth residual word module R4 and a second discrete wavelet submodule W2, and the second wavelet transform module includes a fifth residual submodule R5, a sixth residual word module R6 and a third discrete wavelet submodule W3.

[0068] According to an embodiment of the present disclosure, the first residual submodule R1, the second residual submodule R2, the third residual submodule R3, the fourth residual word module R4, the fifth residual submodule R5 and the sixth residual word module R6 have the same structure and all include two residual blocks. The first discrete wavelet submodule W1, the second discrete wavelet submodule W2 and the third discrete wavelet submodule W3 have the same structure and are all composed of Haar filters.

[0069] According to an embodiment of the present disclosure, the parameter regression module can first perform an adaptive flattening operation on the second frequency domain features output by the multi-level wavelet transform module, and then perform mapping processing of three fully connected layers (F1, F2, F3) on the adaptive flattening operation, so as to obtain the coefficients of the bit rate coding parameter mapping function.

[0070] According to an embodiment of the present disclosure, when the deep residual network is characterized as a first deep residual network, the input module does not perform block processing on the video frames of the high dynamic range video, but directly inputs the video frames of the high dynamic range video into the first convolution layer for convolution processing to obtain a first pixel domain feature, and then inputs the first pixel domain feature into the multi-level wavelet transform module for frequency domain decomposition processing to obtain a first frequency domain feature. The parameter regression module can first perform an adaptive flattening operation on the first frequency domain feature output by the multi-level wavelet transform module in a different manner from the first frequency domain feature, and after the adaptive flattening operation, perform mapping processing of three fully connected layers (F1, F2, F3), so as to obtain the segmented threshold parameters of the bit rate coding parameter mapping function.

[0071] Table 2 Hyperparameters of deep residual network

[0072]

[0073] According to an embodiment of the present disclosure, a first deep residual network is used to process a first pixel domain feature of a video frame of a high dynamic range video, and a segmented threshold parameter of a bit rate coding parameter mapping function is output, including: performing convolution processing on the video frame using a first convolution layer, and outputting a first pixel domain feature; performing frequency domain decomposition processing on the first pixel domain feature using a multi-level wavelet transform module, and outputting a first frequency domain feature; and mapping processing on the first frequency domain feature using a parameter regression module, and outputting a segmented threshold parameter of a bit rate coding parameter mapping function.

[0074] According to an embodiment of the present disclosure, the structures of the first deep residual network and the second deep residual network may be the same, and both the first deep residual network and the second deep residual network may include a first convolutional layer, a multi-level wavelet transform module and a parameter regression module cascaded to each other.

[0075] According to an embodiment of the present disclosure, the first convolution layer may be a 3x3 convolution layer. The step size of the first convolution layer may be 1. By using the first convolution layer, the video frame may be convolved to obtain a first pixel domain feature.

[0076] According to an embodiment of the present disclosure, the multi-level wavelet transform module may include multiple wavelet transform modules. For example, it may include 3 wavelet transform modules. Each wavelet transform module may include two residual submodules and a discrete wavelet submodule.

[0077] According to an embodiment of the present disclosure, the wavelet transform module can use Haar wavelet transform to perform frequency domain decomposition processing on the first pixel domain feature to obtain the first frequency domain feature. The Haar wavelet transform decomposes the first pixel domain feature into 1 low frequency sub-band and 3 high frequency sub-bands through four convolution filters, and the four convolution filters are defined as follows:

[0078]

[0079] in, represents the approximate frequency domain component filter, represents the horizontal frequency domain component filter, represents the vertical frequency domain component filter and represents a diagonal frequency domain component filter.

[0080] According to an embodiment of the present disclosure, the approximate frequency domain component filter It can be composed of two consecutive low-pass filters. Horizontal frequency domain component filter It can be composed of a low-pass filter and a high-pass filter, and the vertical frequency domain component filter It can be composed of a high-pass filter and a low-pass filter and a diagonal frequency domain component filter It can be composed of two consecutive high-pass filters. The step size of the approximate frequency domain component filter, the horizontal frequency domain component filter, the vertical frequency domain component filter and the diagonal frequency domain component filter is 2.

[0081] According to an embodiment of the present disclosure, through Haar wavelet transform, the width and height of the pixel domain feature are downsampled to half of the original, while the channel of the pixel domain feature is expanded to four times of the original.

[0082] According to the embodiments of the present disclosure, in order to enhance the information interaction between different sub-band features, a 3×3 convolution layer with a step size of 1 can be used to achieve effective feature fusion of 1 low-frequency sub-band and 3 high-frequency sub-bands. In addition, unlike traditional downsampling operations (such as pooling, bilinear interpolation, etc.), the wavelet transform module provides a lossless downsampling method, and its working process is as follows: Formula (3):

[0083] (3)

[0084] in, represents the feature concatenation operation, represents the convolution operation, represents a 1×1 convolution operation, represents the ReLU activation function, represents the frequency domain subband features obtained by Haar wavelet transform, represents the characteristics of the wavelet transform input, Represents the features of the wavelet transform module output.

[0085] According to the embodiment of the present disclosure, the first pixel domain feature can be subjected to frequency domain decomposition processing using a multi-level wavelet transform module, thereby obtaining a first frequency domain feature. The first frequency domain feature can be subjected to mapping processing using a parameter regression module, thereby obtaining a segmented threshold parameter of a rate coding parameter mapping function.

[0086] According to an embodiment of the present disclosure, the fully connected layer may include multiple cascades. For example, three fully connected layers may be cascaded. The parameter regression module may map the first frequency domain feature after the adaptive flattening operation through the cascaded fully connected layer, thereby obtaining the segmentation threshold parameter of the rate coding parameter mapping function.

[0087] According to an embodiment of the present disclosure, a second deep residual network is used to process second pixel domain features of multiple image blocks, and coefficients of rate coding parameter mapping functions corresponding to each of the multiple image blocks are output, including: performing convolution processing on multiple image blocks using a first convolution layer, and outputting second pixel domain features; performing frequency domain decomposition processing on the second pixel domain features using a multi-level wavelet transform module, and outputting second frequency domain features; and mapping processing on the second frequency domain features using a parameter regression module, and outputting coefficients of rate coding parameter mapping functions corresponding to each of the multiple image blocks.

[0088] According to an embodiment of the present disclosure, the first convolution layer may be a 3x3 convolution layer with a stride of 1. The first convolution layer is used to perform convolution processing on multiple image blocks, so that the first convolution layer can output a second pixel domain feature.

[0089] According to an embodiment of the present disclosure, a multi-level wavelet transform module is used to perform frequency domain decomposition processing on the second pixel domain feature, so that the second frequency domain feature can be output. A parameter regression module is used to perform mapping processing on the second frequency domain feature, and the coefficients of the rate coding parameter mapping function corresponding to each of the multiple image blocks can be output.

[0090] According to an embodiment of the present disclosure, the multi-level wavelet transform module can be three wavelet transform modules cascaded to each other. The first wavelet transform module can perform frequency domain decomposition processing on the pixel domain feature to obtain the first intermediate frequency domain feature. The first intermediate frequency domain feature is input into the second wavelet transform module, and the second wavelet transform module performs frequency domain decomposition processing on the first intermediate frequency domain feature again to obtain the second intermediate frequency domain feature. The second intermediate frequency domain feature is input into the third wavelet transform module, and the third wavelet transform module performs frequency domain decomposition processing on the second intermediate frequency domain feature again to obtain the second frequency domain feature. Among them, the processing process of the multi-level wavelet transform on the first pixel domain feature is the same as the processing process on the second pixel domain feature, which will not be repeated here.

[0091] According to an embodiment of the present disclosure, the mapping process for the second frequency domain feature may be to map the second frequency domain feature using fully connected layers cascaded with each other. For example, three fully connected layers may be cascaded.

[0092] According to an embodiment of the present disclosure, by performing convolution processing on a video frame or multiple image blocks through a first convolution layer, a first pixel domain feature or a second pixel domain feature can be obtained respectively, and by processing the first pixel domain feature or the second pixel domain feature using a multi-level wavelet transform module and a parameter regression module, a segmented threshold parameter of a rate coding parameter mapping function or a coefficient of the rate coding parameter mapping function can be obtained, thereby improving the accuracy of the segmented threshold parameter of the rate coding parameter mapping function or the coefficient of the rate coding parameter mapping function.

[0093] According to an embodiment of the present disclosure, a multi-level wavelet transform module is used to perform frequency domain decomposition processing on the first pixel domain feature to output the first frequency domain feature, including: inputting the first pixel domain feature into a residual submodule to output a first intermediate pixel domain feature; and using a discrete wavelet submodule to perform frequency domain decomposition processing on the first intermediate pixel domain feature to output the first frequency domain feature.

[0094] According to an embodiment of the present disclosure, each wavelet transform module in the multi-level wavelet transform module may include a residual submodule and a discrete wavelet submodule cascaded to each other. For example, the residual submodules cascaded to each other may include two residual submodules.

[0095] According to an embodiment of the present disclosure, the residual submodule may include a first residual submodule and a second residual submodule. The residual submodule processes the first pixel domain feature to increase the depth of the deep residual network and improve the performance of the deep residual network.

[0096] According to an embodiment of the present disclosure, the first pixel domain feature can be input into the residual submodule to output the first intermediate pixel domain feature. The first intermediate pixel domain feature can be processed by the discrete wavelet submodule for frequency domain decomposition to output the first frequency domain feature. The discrete wavelet submodule can process pixel domain features and frequency domain features.

[0097] According to an embodiment of the present disclosure, by inputting the first pixel domain feature into the residual submodule, the first intermediate pixel domain feature can be output, and by inputting the first intermediate pixel domain feature into the discrete wavelet submodule, the first intermediate pixel domain feature can be subjected to frequency domain decomposition processing to output the first frequency domain feature, and the first frequency domain feature can be input into the parameter regression module, thereby improving the accuracy of the segmented threshold parameter of the rate coding parameter mapping function output by the parameter regression module.

[0098] According to an embodiment of the present disclosure, a parameter regression module is used to perform mapping processing on the first frequency domain feature, and a segmentation threshold parameter of a bit rate coding parameter mapping function is output, including: performing an adaptive flattening operation based on global maximum pooling on the first frequency domain feature to obtain a first target frequency domain feature; and using a plurality of fully connected layers cascaded to each other to perform mapping processing on the first target frequency domain feature, and outputting a segmentation threshold parameter of a bit rate coding parameter mapping function.

[0099] According to an embodiment of the present disclosure, a parameter regression module is used to map the second frequency domain features, and coefficients of rate coding parameter mapping functions corresponding to each of the multiple image blocks are output, including: performing a tiling-based adaptive flattening operation on the second frequency domain features to obtain a second target frequency domain feature; mapping the second frequency domain features using multiple fully connected layers cascaded to each other, and outputting coefficients of rate coding parameter mapping functions corresponding to each of the multiple image blocks.

[0100] According to an embodiment of the present disclosure, different adaptive flattening operations may be used for the first frequency domain feature and the second frequency domain feature. By performing an adaptive flattening operation based on global maximum pooling on the first frequency domain feature, it is possible to convert the three-dimensional first frequency domain feature into a one-dimensional first target frequency domain feature. By performing an adaptive flattening operation based on a tiling operation on the second frequency domain feature, it is possible to convert the three-dimensional second frequency domain feature into a one-dimensional second target frequency domain feature.

[0101] According to an embodiment of the present disclosure, by using a plurality of fully connected layers cascaded to each other to map the first target frequency domain feature, a segmented threshold parameter of a rate coding parameter mapping function can be output. By using a plurality of fully connected layers cascaded to each other to map the second frequency domain feature, coefficients of a rate coding parameter mapping function corresponding to each of a plurality of image blocks can be output.

[0102] According to an embodiment of the present disclosure, by using different adaptive flattening operations on the first frequency domain feature and the second frequency domain feature, a first target frequency domain feature corresponding to the first frequency domain feature and a second target frequency domain feature corresponding to the second frequency domain feature can be obtained, and the first target frequency domain feature or the second target frequency domain feature is mapped using multiple fully connected layers cascaded to each other, so that the segmentation threshold parameters of the rate coding parameter mapping function or the coefficients of the rate coding parameter mapping function can be obtained, and the segmentation threshold parameters of the rate coding parameter mapping function or the coefficients of the rate coding parameter mapping function are improved, thereby improving the efficiency of high dynamic video coding.

[0103] According to an embodiment of the present disclosure, encoding processing is performed on video frames of a high dynamic range video based on encoding parameters of multiple image blocks, including: determining frame-level encoding parameters of the high dynamic range video and bit parameter allocation weights of each of the multiple image blocks according to a bit rate encoding parameter mapping function and a total bit parameter of the video frame; determining encoding parameters of the multiple image blocks according to the bit parameter allocation weights of each of the multiple image blocks and the size parameters of each of the multiple image blocks; encoding processing is performed on video frames of the high dynamic range video based on the encoding parameters of the multiple image blocks.

[0104] According to an embodiment of the present disclosure, the total bit parameter of a video frame can represent the pre-allocated total bit of a video frame. The frame-level coding parameters of the high dynamic range video and the bit parameter allocation weights of each of the multiple image blocks can be determined according to the bit rate coding parameter mapping function and the total bit parameter of the video frame.

[0105] According to an embodiment of the present disclosure, the size of the video frame may include the length and width of the video frame. The pre-allocated bit rate of the video frame may be determined according to the total bit parameter of the video frame and the size of the video frame. The calculation is as follows:

[0106] (4)

[0107] in, Indicates the total bit parameter of the video frame. Indicates the length of the video frame. Indicates the width of the video frame.

[0108] According to an embodiment of the present disclosure, when the rate coding parameter mapping function is determined, the segmentation threshold parameter of the rate coding parameter mapping function and the coefficient of the rate coding parameter mapping function are also determined. According to the rate coding parameter mapping function, it can be determined whether the pre-allocated bits belong to the high rate segment or the low rate segment. The interval decision function of the rate coding parameter mapping function is as follows: Formula (5):

[0109] (5)

[0110] in, An interval decision function representing a segment rate coding parameter mapping function; represents the i-th image block, represents the coefficient of the rate coding parameter mapping function corresponding to the i-th image block, Represents the segmentation threshold parameter of the rate coding parameter mapping function corresponding to the i-th image block.

[0111] According to an embodiment of the present disclosure, the frame-level coding parameter λ of the high dynamic range video can be determined according to the bit rate coding parameter mapping function and the total bits of the video frame, and the Newton iteration method can be used to solve the following formula (6):

[0112] (6)

[0113] in, Represents the coefficient group of the rate coding parameter mapping function corresponding to the i-th image block.

[0114] According to an embodiment of the present disclosure, according to the Newton iteration method, calculating the frame-level encoding parameters of the high dynamic range video may include the following steps, defining the equation: ; Input: Initial ; Number of iterations and error threshold , the error threshold can be set to 0.0001; Output: Final ; For calculating the gradient: ,renew : , update equation Cycle until .

[0115] According to an embodiment of the present disclosure, when determining frame-level encoding parameters of a high dynamic range video, pre-allocated bit parameters of multiple image blocks may be determined according to the following formula (7).

[0116] (7)

[0117] in, Represents the pre-allocated bit parameter of the i-th image block.

[0118] According to an embodiment of the present disclosure, the bit parameter allocation weights of the plurality of image blocks may be determined according to the pre-allocated bit parameters of the plurality of image blocks and the total bit parameters of the video frame, as shown in the following formula (8):

[0119] (8)

[0120] in, represents the bit parameter allocation weight of the i-th image block.

[0121] According to an embodiment of the present disclosure, weights may be allocated according to respective bit parameters of the plurality of image blocks and respective size parameters of the plurality of image blocks, encoding parameters of the plurality of image blocks may be determined, and video frames of high-dynamic video may be encoded based on the respective encoding parameters of the plurality of image blocks.

[0122] According to the embodiments of the present disclosure, the frame-level coding parameters of the high dynamic range video and the bit parameter allocation weights of each of the multiple image blocks can be determined according to the bit rate coding parameter mapping function and the total bit parameter of the video frame, and then the coding parameters of each of the multiple image blocks can be determined according to the frame-level coding parameters. According to the coding parameters of each of the multiple image blocks, the video frames of the high dynamic range video can be encoded, which can improve the bit rate control accuracy of intra-frame coding and thus improve the coding efficiency of the high dynamic range video.

[0123] According to an embodiment of the present disclosure, encoding parameters of multiple image blocks are determined according to respective bit parameter allocation weights of the multiple image blocks and respective size parameters of the multiple image blocks, including: determining respective target allocation bits of the multiple image blocks according to respective bit parameter allocation weights of the multiple image blocks; determining target bit rates of the multiple image blocks according to respective target allocation bits of the multiple image blocks and respective size parameters of the multiple image blocks; determining encoding parameters of the multiple image blocks according to target bit rates of the multiple image blocks and a bit rate encoding parameter mapping function.

[0124] According to an embodiment of the present disclosure, the target allocated bits of the multiple image blocks can be determined according to the bit parameter allocation weights of the multiple image blocks, as shown in the following formula (9):

[0125] (9)

[0126] in, represents the target allocation bits of the i-th image block, represents a sliding window for smoothing bit errors, which can be set to 4 in the embodiment of the present disclosure; Indicates the number of image blocks included in a video frame. Indicates the number of image blocks that have been encoded in a video frame. Indicates the remaining bits for encoding. According to an embodiment of the present disclosure, the target bit rates of the multiple image blocks can be determined according to the target allocation bits of the multiple image blocks and the size parameters of the multiple image blocks. The encoding parameters of the multiple image blocks can be determined according to the target bit rates of the multiple image blocks and the code rate encoding parameter mapping function, as shown in the following formula (10):

[0127] (10)

[0128] According to an embodiment of the present disclosure, encoding a high dynamic range video frame based on encoding parameters of multiple image blocks can improve the accuracy of bit rate control of intra-frame encoding, thereby improving encoding efficiency.

[0129] According to an embodiment of the present disclosure, for the data-driven intra-frame rate control method of λ-domain high dynamic range video implemented by the encoding method of the high dynamic range video of the present disclosure, the high dynamic range video standard test sequences EBU_04_Hurdles, EBU_06_Starting, ShowGirl2 and S12-Sunrise, and the manually constructed test sequences TJU_Tree, TJU_Lake and TJU_Park are used to perform performance tests. The data-driven intra-frame rate control method of λ-domain high dynamic range video constructed by the present disclosure can realize accurate rate control of high dynamic range video and improve encoding performance without any prior knowledge, so each video sequence can only test the first frame. The indicators "bit allocation accuracy" and "coding rate distortion performance" used to characterize the rate control performance are shown in Tables 3 and 4.

[0130] Table 3 Comparison of bit allocation accuracy of different rate control methods in high dynamic range standard test sequence

[0131]

[0132] According to an embodiment of the present disclosure, the target bit rate may be calculated based on a fixed QP. Indicates the total number of bits allocated for a video frame. Indicates the actual allocated bits under fixed QP. Represents the error rate. Related Method 1 and Related Method 2 are both rate control methods in related technologies. As shown in Table 3, compared with Related Method 1 and Related Method 2, the bit allocation error rate of this method is 1.24%, and the bit allocation result closest to the target bit rate is obtained, so that a more accurate intra-frame rate control can be obtained. The intra-frame rate control method of λ-domain high dynamic range video implemented by the proposed high dynamic range video encoding method mainly benefits from the fact that the proposed rate coding parameter mapping function can accurately describe the intra-frame coding rate distortion characteristics of high dynamic range video. And the proposed deep residual network based on wavelet convolution can effectively analyze the content characteristics of high dynamic range video, and accurately predict the threshold parameters and coefficients of the rate coding parameter mapping function.

[0133] Table 4 Comparison of coding rate distortion performance of different rate control methods in high dynamic range standard test sequence

[0134]

[0135] According to the embodiments of the present disclosure, Bjontegaard delta rate (BD-Rate: η1) and Bjontegaard delta wPSNR (BD- wPSNR: δ1) can be used to evaluate the coding rate distortion performance. It can be seen from Table 4 that compared with the default intra-frame rate control method of VTM, the intra-frame rate control method implemented by the coding method of the high dynamic range video of the embodiment of the present disclosure achieves an average coding efficiency improvement of 1.67%. Compared with the related method 1 and the related method 2, the present method achieves an average coding efficiency improvement of 0.95% and 6.70% respectively.

[0136] Figure 4 The flowchart of the training method of the deep residual network according to the embodiment of the present disclosure is schematically shown.

[0137] like Figure 4 As shown, the training method 400 of the deep residual network includes operations S410 to S460.

[0138] In operation S410, a training sample of a deep residual network is obtained.

[0139] In operation S420, a first pixel domain feature of a sample video frame of a training sample is processed using a first deep residual network, and a prediction threshold parameter of a rate coding parameter mapping function is output.

[0140] In operation S430, a sample video frame of a training sample is divided into blocks to obtain a plurality of sample image blocks.

[0141] In operation S440, the second pixel domain features of the plurality of sample image blocks are processed using a second deep residual network, and prediction coefficients of rate coding parameter mapping functions corresponding to the plurality of sample image blocks are output.

[0142] In operation S450 , a loss value is determined based on the target loss function according to the training samples and the training sample labels.

[0143] In operation S460, based on the loss value, a prediction threshold parameter of the rate coding parameter mapping function and a prediction coefficient of the rate coding parameter mapping function are adjusted to obtain a trained deep residual network.

[0144] According to an embodiment of the present disclosure, the deep residual network may include a first deep residual network and a second deep residual network, and the structures of the first deep residual network and the second deep residual network may be the same.

[0145] According to an embodiment of the present disclosure, the training sample may be an artificially constructed high dynamic range video or a directly acquired high dynamic range video.

[0146] According to an embodiment of the present disclosure, in order to ensure the diversity of training samples during the construction of training samples, 229 groups of images are collected in one embodiment, each group of images includes 5 standard dynamic range images taken at different exposure levels under the same scene, and the 5 exposure levels are set to , and 0. In order to make the constructed training samples contain richer multi-scale details, the training samples contain samples of two different resolutions, namely 3840×2160 and 2336×2336. In addition, the images are taken in a well-lit static environment to ensure the maximum brightness level of the content of each training sample and effectively avoid the introduction of ghost images in the subsequent fusion process.

[0147] According to an embodiment of the present disclosure, a group of 5 standard dynamic range images can be fused into a high dynamic range image. In order to ensure the subsequent encoding and display of the training samples, each training sample can be converted from a floating point number to an integer using a PQ photoelectric conversion function. The training samples are then converted to YUV420 format and compressed using the VVC reference software. The compression process uses 20 to 40 intervals of 2, for a total of 11 different quantization parameters.

[0148] According to an embodiment of the present disclosure, a first deep residual network may be used to process first pixel domain features of a sample video frame of a training sample, and output a prediction threshold parameter of a rate coding parameter mapping function.

[0149] According to an embodiment of the present disclosure, multiple image blocks can be obtained by performing block processing on the training samples. The second pixel domain features of the image blocks can be processed using a second deep residual network to output prediction coefficients of the rate coding parameter mapping function corresponding to each of the multiple image blocks.

[0150] According to an embodiment of the present disclosure, the target loss function may adopt a mean absolute error. The training sample label may characterize the target threshold parameter and target coefficient of the rate coding parameter mapping function. The loss value between the predicted threshold parameter and coefficient of the rate coding parameter mapping function and the training sample label may be calculated based on the training sample and the training sample label.

[0151] According to an embodiment of the present disclosure, the training sample label can represent the target threshold parameter and target coefficient of the rate coding parameter mapping function obtained based on the verification sample. The training sample and the verification sample are artificially constructed data sets divided in a ratio of 9:1.

[0152] According to an embodiment of the present disclosure, the deep residual network can use Adam as a parameter optimizer, the number of training times is 300 epochs, the initial value of the learning rate is set to 0.0001, and the learning rate decays to 0.8 times the original learning rate every 100 epochs.

[0153] According to an embodiment of the present disclosure, the prediction threshold parameter of the rate coding parameter mapping function and the prediction coefficient of the rate coding parameter mapping function can be adjusted based on the loss value, thereby obtaining a trained deep residual network.

[0154] Based on the above-mentioned high dynamic range video encoding method, the present disclosure also provides a high dynamic range video encoding device. Figure 5 The device is described in detail.

[0155] Figure 5 The structure of a high dynamic range video encoding device according to an embodiment of the present disclosure is schematically shown.

[0156] like Figure 5 As shown, the high dynamic range video encoding device 500 of this embodiment includes a first acquisition module 510 , a first extraction module 520 , a first blocking module 530 , a second extraction module 540 , a first determination module 550 and an encoding module 560 .

[0157] The first acquisition module 510 is used to acquire a high dynamic range video. In one embodiment, the first acquisition module 510 can be used to perform the operation S210 described above, which will not be described in detail herein.

[0158] The first extraction module 520 is used to process the first pixel domain feature of the video frame of the high dynamic range video using the first deep residual network, and output the segmentation threshold parameter of the bit rate coding parameter mapping function, wherein the first deep residual network is trained with the segmentation threshold parameter of the bit rate coding parameter mapping function of the sample video frame as a label. In one embodiment, the first extraction module 520 can be used to perform the operation S220 described above, which will not be repeated here.

[0159] The first block division module 530 is used to perform block processing on the video frame to obtain multiple image blocks. In one embodiment, the first block division module 530 can be used to perform the operation S230 described above, which will not be described in detail here.

[0160] The second extraction module 540 is used to process the second pixel domain features of the plurality of image blocks using the second deep residual network, and output coefficients of the rate coding parameter mapping function corresponding to each of the plurality of image blocks, wherein the second deep residual network is trained using the coefficients of the rate coding parameter mapping function of the sample image block as labels. In one embodiment, the second extraction module 540 can be used to perform the operation S240 described above, which will not be described in detail herein.

[0161] The first determination module 550 is used to determine the coding parameters of the multiple image blocks according to the respective pre-allocated bit parameters of the multiple image blocks, wherein the rate coding parameter mapping function is constructed by taking the pre-allocated bit parameters of the multiple image blocks as independent variables, the coding parameters of the multiple image blocks as dependent variables, and the coefficients of the rate coding parameter mapping function corresponding to the multiple segments of pre-allocated bit parameters; the multiple segments of pre-allocated bit parameters are divided according to the segmentation threshold parameters. In one embodiment, the first determination module 550 can be used to perform the operation S250 described above, which will not be repeated here.

[0162] The encoding module 560 is used to encode the video frame of the high dynamic range video based on the encoding parameters of the multiple image blocks. In one embodiment, the encoding module 560 can be used to perform the operation S260 described above, which will not be described in detail here.

[0163] According to an embodiment of the present disclosure, the first deep residual network and the second deep residual network have the same structure, including a first convolutional layer, a multi-level wavelet transform module and a parameter regression module cascaded with each other, and a first extraction module 520, including:

[0164] The first extraction submodule is used to perform convolution processing on the video frame using the first convolution layer and output a first pixel domain feature.

[0165] The second extraction submodule is used to perform frequency domain decomposition processing on the first pixel domain feature by using a multi-level wavelet transform module, and output a first frequency domain feature.

[0166] The third extraction submodule is used to perform mapping processing on the first frequency domain feature by using the parameter regression module, and output a segmentation threshold parameter of the bit rate coding parameter mapping function.

[0167] According to an embodiment of the present disclosure, the second extraction module 540 includes:

[0168] The fourth extraction submodule is used to perform convolution processing on the multiple image blocks using the first convolution layer, and output second pixel domain features.

[0169] The fifth extraction submodule is used to perform frequency domain decomposition processing on the second pixel domain feature by using a multi-level wavelet transform module, and output a second frequency domain feature.

[0170] The sixth extraction submodule is used to perform mapping processing on the second frequency domain feature by using the parameter regression module, and output coefficients of the rate coding parameter mapping function corresponding to each of the plurality of image blocks.

[0171] According to an embodiment of the present disclosure, each wavelet transform module in the multi-level wavelet transform module includes a residual submodule and a discrete wavelet submodule cascaded with each other, and the second extraction submodule includes:

[0172] The first extraction unit is used to input the first pixel domain feature into the residual submodule and output the first intermediate pixel domain feature.

[0173] The second extraction unit is used to perform frequency domain decomposition processing on the first intermediate pixel domain feature by using a discrete wavelet submodule, and output a first frequency domain feature.

[0174] According to an embodiment of the present disclosure, the third extraction submodule includes:

[0175] The third extraction unit is used to perform an adaptive flattening operation based on global maximum pooling on the first frequency domain feature to obtain a first target frequency domain feature.

[0176] The fourth extraction unit is used to map the first target frequency domain feature by using a plurality of fully connected layers cascaded to each other, and output a segmented threshold parameter of a rate coding parameter mapping function.

[0177] According to an embodiment of the present disclosure, the sixth extraction submodule includes:

[0178] The fifth extraction unit is used to perform a tiling-based adaptive flattening operation on the second frequency domain feature to obtain a second target frequency domain feature.

[0179] The sixth extraction unit is used to map the second frequency domain features by using a plurality of fully connected layers cascaded to each other, and output coefficients of a rate coding parameter mapping function corresponding to each of the plurality of image blocks.

[0180] According to an embodiment of the present disclosure, the encoding module 560 includes:

[0181] The first encoding submodule is used to determine the frame-level encoding parameters of the high dynamic range video and the bit parameter allocation weights of each of the multiple image blocks according to the bit rate encoding parameter mapping function and the total bit parameter of the video frame.

[0182] The second encoding submodule is used to determine encoding parameters of the multiple image blocks according to the bit parameter allocation weights of the multiple image blocks and the size parameters of the multiple image blocks.

[0183] The third encoding submodule is used to encode the video frames of the high dynamic range video based on the encoding parameters of the multiple image blocks.

[0184] According to an embodiment of the present disclosure, the second encoding submodule includes:

[0185] The first encoding unit is used to determine target allocation bits for each of the multiple image blocks according to bit parameter allocation weights for each of the multiple image blocks.

[0186] The second coding unit is used to determine the target bit rates of the multiple image blocks according to the target allocation bits of the multiple image blocks and the size parameters of the multiple image blocks. The third coding unit is used to determine the coding parameters of the multiple image blocks according to the target bit rates of the multiple image blocks and the code rate coding parameter mapping function.

[0187] According to an embodiment of the present disclosure, any multiple modules of the first acquisition module 510, the first extraction module 520, the first block module 530, the second extraction module 540, the first determination module 550 and the encoding module 560 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the first acquisition module 510, the first extraction module 520, the first block module 530, the second extraction module 540, the first determination module 550 and the encoding module 560 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the first acquisition module 510, the first extraction module 520, the first blocking module 530, the second extraction module 540, the first determination module 550 and the encoding module 560 can be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function can be performed.

[0188] Based on the above-mentioned deep residual network training method, the present disclosure also provides a deep residual network training device. Figure 6 The device is described in detail.

[0189] Figure 6 The structural block diagram of a deep residual network training device according to an embodiment of the present disclosure is schematically shown.

[0190] like Figure 6 As shown, the training device 600 of the deep residual network of this embodiment includes a second acquisition module 610, a third extraction module 620, a second blocking module 630, a fourth extraction module 640, a second determination module 650 and a training module 660.

[0191] The second acquisition module 610 is used to acquire a training sample of a deep residual network, wherein the deep residual network includes a first deep residual network and a second deep residual network. In one embodiment, the second acquisition module 610 can be used to perform the operation S410 described above, which will not be repeated here.

[0192] The third extraction module 620 is used to process the first pixel domain features of the sample video frame of the training sample using the first deep residual network, and output the prediction threshold parameter of the bit rate coding parameter mapping function. In one embodiment, the third extraction module 620 can be used to perform the operation S420 described above, which will not be repeated here.

[0193] The second block division module 630 is used to perform block processing on the sample video frame of the training sample to obtain a plurality of sample image blocks. In one embodiment, the second block division module 630 can be used to perform the operation S430 described above, which will not be described in detail here.

[0194] The fourth extraction module 640 is used to process the second pixel domain features of the plurality of sample image blocks using the second deep residual network, and output prediction coefficients of the rate coding parameter mapping function corresponding to each of the plurality of sample image blocks. In one embodiment, the fourth extraction module 640 can be used to perform the operation S440 described above, which will not be described in detail herein.

[0195] The second determination module 650 is used to determine the loss value based on the target loss function, according to the training samples and the training sample labels. In one embodiment, the second determination module 650 can be used to perform the operation S450 described above, which will not be repeated here.

[0196] The training module 660 is used to adjust the prediction threshold parameter of the rate coding parameter mapping function and the prediction coefficient of the rate coding parameter mapping function based on the loss value to obtain a trained deep residual network. In one embodiment, the training module 660 can be used to perform the operation S460 described above, which will not be repeated here.

[0197] According to an embodiment of the present disclosure, any multiple modules of the second acquisition module 610, the third extraction module 620, the second block module 630, the fourth extraction module 640, the second determination module 650 and the training module 660 can be combined in one module, or any one of them can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the second acquisition module 610, the third extraction module 620, the second block module 630, the fourth extraction module 640, the second determination module 650 and the training module 660 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the second acquisition module 610, the third extraction module 620, the second blocking module 630, the fourth extraction module 640, the second determination module 650 and the training module 660 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is executed.

[0198] Figure 7 A block diagram of an electronic device suitable for implementing a method for encoding a high dynamic range video according to an embodiment of the present disclosure is schematically shown.

[0199] like Figure 7 As shown, the electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 to a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0200] In RAM 703, various programs and data required for the operation of electronic device 700 are stored. Processor 701, ROM 702 and RAM 703 are connected to each other via bus 704. Processor 701 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 702 and / or RAM 703. It should be noted that the program can also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0201] According to an embodiment of the present disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the I / O interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 708 including a hard disk, etc.; and a communication portion 709 including a network interface card such as a LAN card, a modem, etc. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage portion 708 as needed.

[0202] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0203] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0204] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the high dynamic range video encoding method provided by the embodiment of the present disclosure.

[0205] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 701. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0206] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 709, and / or installed from the removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0207] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0208] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0209] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0210] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0211] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A method for encoding high dynamic range video, characterized in that: include: Get high dynamic range video; Processing a first pixel domain feature of a video frame of the high dynamic range video using a first deep residual network, and outputting a segmentation threshold parameter of a rate coding parameter mapping function, wherein the first deep residual network is trained using the segmentation threshold parameter of the rate coding parameter mapping function of a sample video frame as a label; Performing block processing on the video frame to obtain multiple image blocks; Processing the second pixel domain features of the plurality of image blocks using a second deep residual network, and outputting coefficients of rate coding parameter mapping functions corresponding to the plurality of image blocks, wherein the second deep residual network is trained using coefficients of rate coding parameter mapping functions of sample image blocks as labels; Based on the rate coding parameter mapping function, the coding parameters of the multiple image blocks are determined according to the pre-allocated bit parameters of the multiple image blocks, wherein the rate coding parameter mapping function is constructed by taking the pre-allocated bit parameters of the multiple image blocks as independent variables and the coding parameters of the multiple image blocks as dependent variables, and the coefficients of the rate coding parameter mapping function corresponding to the multiple segments of pre-allocated bit parameters; the multiple segments of pre-allocated bit parameters are divided according to the segmentation threshold parameter; The video frames of the high dynamic range video are encoded based on the encoding parameters of the multiple image blocks.

2. The method according to claim 1, characterized in that The first deep residual network and the second deep residual network have the same structure, including a first convolutional layer, a multi-level wavelet transform module and a parameter regression module cascaded with each other, and the first pixel domain feature of the video frame of the high dynamic range video is processed by the first deep residual network to output the segmented threshold parameter of the bit rate coding parameter mapping function, including: Using the first convolutional layer to perform convolution processing on the video frame, and output a first pixel domain feature; Using the multi-level wavelet transform module to perform frequency domain decomposition processing on the first pixel domain feature, and output a first frequency domain feature; The parameter regression module is used to perform mapping processing on the first frequency domain feature, and a segmentation threshold parameter of a rate coding parameter mapping function is output.

3. The method according to claim 2, characterized in that The processing of the second pixel domain features of the plurality of image blocks by using the second deep residual network and outputting coefficients of the rate coding parameter mapping function corresponding to each of the plurality of image blocks comprises: Using the first convolution layer to perform convolution processing on the multiple image blocks, and output second pixel domain features; Using the multi-level wavelet transform module to perform frequency domain decomposition processing on the second pixel domain feature, and output a second frequency domain feature; The parameter regression module is used to perform mapping processing on the second frequency domain feature, and coefficients of the rate coding parameter mapping function corresponding to each of the plurality of image blocks are output.

4. The method according to claim 2, characterized in that Each wavelet transform module in the multi-level wavelet transform module includes a residual submodule and a discrete wavelet submodule cascaded with each other, and the multi-level wavelet transform module is used to perform frequency domain decomposition processing on the first pixel domain feature to output the first frequency domain feature, including: Inputting the first pixel domain feature into the residual submodule, and outputting a first intermediate pixel domain feature; The discrete wavelet submodule is used to perform frequency domain decomposition processing on the first intermediate pixel domain feature, and the first frequency domain feature is output.

5. The method according to claim 4, characterized in that The using the parameter regression module to perform mapping processing on the first frequency domain feature and outputting a segmented threshold parameter of a rate coding parameter mapping function includes: Performing an adaptive flattening operation based on global maximum pooling on the first frequency domain feature to obtain a first target frequency domain feature; The first target frequency domain feature is mapped using a plurality of fully connected layers cascaded to each other, and a segmented threshold parameter of a rate coding parameter mapping function is output.

6. The method according to claim 3, characterized in that The using the parameter regression module to perform mapping processing on the second frequency domain feature and outputting coefficients of the rate coding parameter mapping function corresponding to each of the plurality of image blocks includes: Performing a tiling-based adaptive flattening operation on the second frequency domain feature to obtain a second target frequency domain feature; The second frequency domain features are mapped using the multiple fully connected layers cascaded to each other, and coefficients of the rate coding parameter mapping function corresponding to each of the multiple image blocks are output.

7. The method according to claim 1, characterized in that The encoding process of the video frame of the high dynamic range video based on the encoding parameters of the multiple image blocks includes: Determining, according to the bit rate coding parameter mapping function and the total bit parameter of the video frame, a frame-level coding parameter of the high dynamic range video and a bit parameter allocation weight of each of the plurality of image blocks; Determining encoding parameters of the plurality of image blocks according to respective bit parameter allocation weights of the plurality of image blocks and respective size parameters of the plurality of image blocks; The video frames of the high dynamic range video are encoded based on the encoding parameters of the multiple image blocks.

8. The method according to claim 7, characterized in that The step of determining the encoding parameters of the plurality of image blocks according to the bit parameter allocation weights of the plurality of image blocks and the size parameters of the plurality of image blocks comprises: Determining target allocation bits for each of the plurality of image blocks according to bit parameter allocation weights for each of the plurality of image blocks; Determining target bit rates of the plurality of image blocks according to respective target allocated bits of the plurality of image blocks and respective size parameters of the plurality of image blocks; The encoding parameters of the plurality of image blocks are determined according to the target bit rates of the plurality of image blocks and the code rate encoding parameter mapping function.

9. A method for training a deep residual network, characterized in that: include: Acquire a training sample of the deep residual network, wherein the deep residual network includes a first deep residual network and a second deep residual network; Processing a first pixel domain feature of a sample video frame of the training sample using the first deep residual network, and outputting a prediction threshold parameter of a bit rate coding parameter mapping function; Performing block processing on the sample video frame of the training sample to obtain a plurality of sample image blocks; Processing the second pixel domain features of the plurality of sample image blocks by using the second deep residual network, and outputting prediction coefficients of rate coding parameter mapping functions corresponding to the plurality of sample image blocks respectively; Based on the target loss function, determining a loss value according to the training samples and the training sample labels; Based on the loss value, a prediction threshold parameter of the rate coding parameter mapping function and a prediction coefficient of the rate coding parameter mapping function are adjusted to obtain a trained deep residual network.

10. A high dynamic range video encoding device, characterized in that: include: A first acquisition module, used to acquire high dynamic range video; A first extraction module is used to process a first pixel domain feature of a video frame of the high dynamic range video using a first deep residual network, and output a segmentation threshold parameter of a rate coding parameter mapping function, wherein the first deep residual network is trained using the segmentation threshold parameter of the rate coding parameter mapping function of the sample video frame as a label; A first block division module, used for performing block processing on the video frame to obtain a plurality of image blocks; A second extraction module is used to process the second pixel domain features of the multiple image blocks using a second deep residual network, and output coefficients of the rate coding parameter mapping function corresponding to each of the multiple image blocks, wherein the second deep residual network is trained using the coefficients of the rate coding parameter mapping function of the sample image block as labels; A first determination module is configured to determine the coding parameters of the plurality of image blocks according to the respective pre-allocated bit parameters of the plurality of image blocks, wherein the rate coding parameter mapping function is constructed by taking the pre-allocated bit parameters of the plurality of image blocks as independent variables, the coding parameters of the plurality of image blocks as dependent variables, and the coefficients of the rate coding parameter mapping function corresponding to the plurality of segments of pre-allocated bit parameters; the plurality of segments of pre-allocated bit parameters are divided according to the segmentation threshold parameter; The encoding module is used to encode the video frames of the high dynamic range video based on the encoding parameters of the multiple image blocks.

Images