Encoding methods, training methods and devices for high dynamic range video

Abstract

This disclosure provides a high dynamic range (HDR) video encoding method, training method, and apparatus, which can be applied to the field of multimedia processing technology. The HDR video encoding method includes: acquiring HDR video; processing the first pixel domain features of the HDR video frames using a first deep residual network to output segmentation threshold parameters of the bitrate coding parameter mapping function; dividing the video frames into blocks to obtain multiple image blocks; processing the second pixel domain features of the multiple image blocks using a second deep residual network to output coefficients of the bitrate coding parameter mapping function corresponding to each of the multiple image blocks; determining the encoding parameters of the multiple image blocks based on the bitrate coding parameter mapping function and according to the pre-allocated bit parameters of each of the multiple image blocks; and encoding the HDR video frames based on the encoding parameters of the multiple image blocks.

Description

Technical Field

[0001] This disclosure relates to the field of multimedia processing technology, and in particular to a coding method, training method and apparatus for high dynamic range video. Background Technology

[0002] With the rapid development of artificial intelligence and display technology, high dynamic range (HDR) imaging technology has become one of the important technologies for enhancing visual experience. HDR technology attempts to capture the range of motion in the real world from 10... -8 cd / m 2 Up to 10 9 cd / m 2 High dynamic range (HDL) video captures the full range of brightness. This capability allows for the creation of images and videos with more vibrant colors, richer shadow and highlight details, and a superior overall visual experience. Unlike traditional 8-bit Standard Dynamic Range (SDR) video images, HDL video images require higher bit depths or floating-point values ​​to store the necessary information. Two common HDL formats are Radiance RGBE and OpenEXR, with the extensions ".hdr" and ".exr", respectively. HDL video accurately records the physical brightness of a scene but generates more data than standard dynamic range video, posing new challenges for video compression and transmission.

[0003] In realizing the concept of this disclosure, the inventors discovered that in the related technology, the bitrate control efficiency of intra-frame coding for high dynamic range video is low, resulting in low bitrate control accuracy and coding efficiency of high dynamic range video, as well as low quality of high dynamic range video. Summary of the Invention

[0004] In view of the above problems, this disclosure provides a coding method, training method and apparatus for high dynamic range video.

[0005] According to a first aspect of this disclosure, a method for encoding high dynamic range (HDR) video is provided, comprising: acquiring HDR video; processing first pixel domain features of video frames of the HDR video using a first deep residual network to output segmented threshold parameters of a bitrate coding parameter mapping function, wherein the first deep residual network is trained using segmented threshold parameters of the bitrate coding parameter mapping function of sample video frames as labels; dividing the video frames into blocks to obtain multiple image blocks; processing second pixel domain features of the multiple image blocks using a second deep residual network to output coefficients of the bitrate coding parameter mapping function corresponding to each of the multiple image blocks, wherein the second deep residual network... The residual network is trained using the coefficients of the rate-to-code parameter mapping function of the sample image blocks as labels. Based on the rate-to-code parameter mapping function, the encoding parameters of the multiple image blocks are determined according to their respective pre-allocated bit parameters. The rate-to-code parameter mapping function is constructed by taking the pre-allocated bit parameters of the multiple image blocks as independent variables and the encoding parameters of the multiple image blocks as dependent variables, and the coefficients of the rate-to-code parameter mapping function corresponding to each segment of pre-allocated bit parameters. The multiple segments of pre-allocated bit parameters are divided according to the segmentation threshold parameters. The video frames of the high dynamic range video are encoded based on the encoding parameters of the multiple image blocks.

[0006] A second aspect of the present invention provides a training method for a deep residual network. The method includes: acquiring training samples of the deep residual network, wherein the deep residual network includes a first deep residual network and a second deep residual network; processing the first pixel domain features of the sample video frames of the training samples using the first deep residual network to output a prediction threshold parameter of the bitrate coding parameter mapping function; dividing the sample video frames of the training samples into blocks to obtain multiple sample image blocks; processing the second pixel domain features of the multiple sample image blocks using the second deep residual network to output prediction coefficients of the bitrate coding parameter mapping function corresponding to each of the multiple sample image blocks; determining a loss value based on a target loss function, according to the training samples and training sample labels; and adjusting the prediction threshold parameter of the bitrate coding parameter mapping function and the prediction coefficients of the bitrate coding parameter mapping function based on the loss value to obtain a trained deep residual network.

[0007] A third aspect of the present invention provides an encoding apparatus for high dynamic range video, the apparatus comprising a first acquisition module, a first extraction module, a first block segmentation module, a second extraction module, a first determination module, and an encoding module.

[0008] The first acquisition module is used to acquire high dynamic range video.

[0009] The first extraction module is used to process the first pixel domain features of the video frames of the high dynamic range video using a first deep residual network, and output the segmented threshold parameters of the bitrate coding parameter mapping function. The first deep residual network is trained using the segmented threshold parameters of the bitrate coding parameter mapping function of the sample video frames as labels.

[0010] The first segmentation module is used to segment the above video frames into multiple image blocks.

[0011] The second extraction module is used to process the second pixel domain features of the above image blocks using a second deep residual network, and output the coefficients of the rate coding parameter mapping function corresponding to each of the above multiple image blocks. The second deep residual network is trained with the coefficients of the rate coding parameter mapping function of the sample image blocks as labels.

[0012] The first determining module is used to determine the encoding parameters of the multiple image blocks based on their respective pre-allocated bit parameters. The bit rate encoding parameter mapping function is constructed by taking the pre-allocated bit parameters of the multiple image blocks as independent variables and the encoding parameters of the multiple image blocks as dependent variables, and the coefficients of the bit rate encoding parameter mapping function corresponding to each of the multiple pre-allocated bit parameters. The multiple pre-allocated bit parameters are divided according to the segmentation threshold parameters.

[0013] The encoding module is used to encode the video frames of the high dynamic range video based on the encoding parameters of the above-mentioned multiple image blocks.

[0014] According to the high dynamic range (HDR) video coding method, training method, and apparatus provided in this disclosure, by processing the first pixel domain features of the acquired HDR video video frames using a first deep residual network, the segmentation threshold parameters of the bitrate coding parameter mapping function can be obtained. The video frames are then divided into blocks to obtain multiple image blocks. By processing the second pixel domain features of the multiple image blocks using a second deep residual network, the coefficients of the bitrate coding parameter mapping function can be obtained. Based on the bitrate coding parameter mapping function, the coding parameters of the multiple image blocks can be determined according to their respective pre-allocated bit parameters. Furthermore, by encoding the HDR video frames based on the coding parameters of the multiple image blocks, the bitrate corresponding to the multiple image blocks can be obtained. This improves the bitrate control efficiency of intra-frame coding in HDR video, thereby improving the bitrate control accuracy and coding efficiency of HDR video, and ultimately enhancing the quality of HDR video. Attached Figure Description

[0015] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0016] Figure 1 The illustration shows an application scenario of a high dynamic range video encoding method according to an embodiment of the present disclosure;

[0017] Figure 2 A flowchart illustrating a method for encoding high dynamic range video according to an embodiment of the present disclosure is shown schematically.

[0018] Figure 3 A schematic diagram of a deep residual network according to an embodiment of the present disclosure is shown.

[0019] Figure 4 A flowchart illustrating a training method for a deep residual network according to an embodiment of the present disclosure is shown schematically.

[0020] Figure 5 This schematically illustrates a structural block diagram of a high dynamic range video encoding apparatus according to an embodiment of the present disclosure;

[0021] Figure 6 A schematic block diagram of a training apparatus for a deep residual network according to an embodiment of the present disclosure is shown.

[0022] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a coding method for high dynamic range video according to an embodiment of the present disclosure. Detailed Implementation

[0023] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0026] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0027] In developing this disclosure, it was discovered that related technologies have developed a series of standardized and non-standardized coding tools to address the challenges of high dynamic range (HDR) video data volume, some of which have been integrated into the latest Versatile Video Coding (VVC) standard. Rate control is a key technology in video coding, aiming to find an optimal bit allocation strategy to ensure that the video coding bitrate is within the bit allocation budget while achieving optimal rate distortion (RD) performance. In common video coding standards such as HEVC / VVC, rate control algorithms can be designed for three different coding structures: All Intra (AI), Low Delay (LD), and Random Access (RA), to effectively address various practical coding application scenarios. Related technologies primarily focus on LD and RA coding structures, neglecting the crucial role of intra-frame coding in allocating remaining bits and controlling the quality of subsequent frames, thus limiting the potential for performance improvement in HDR video coding.

[0028] Furthermore, the intra-frame rate control algorithms in related technologies do not take into account the differences in coding characteristics between high dynamic range (HDR) video and standard dynamic range (SL) video, making it difficult to directly apply existing intra-frame rate control algorithms for SL content to HDR content intra-frame rate control.

[0029] In view of this, embodiments of the present disclosure provide a method for encoding high dynamic range (HDR) video, comprising: acquiring HDR video; processing the first pixel domain features of video frames of HDR video using a first deep residual network to output segmented threshold parameters of a bitrate coding parameter mapping function, wherein the first deep residual network is trained using the segmented threshold parameters of the bitrate coding parameter mapping function of sample video frames as labels; dividing the video frames into blocks to obtain multiple image blocks; processing the second pixel domain features of the multiple image blocks using a second deep residual network to output coefficients of the bitrate coding parameter mapping function corresponding to each of the multiple image blocks, wherein... In this process, the second deep residual network is trained using the coefficients of the rate-to-code parameter mapping function of the sample image blocks as labels. Based on the rate-to-code parameter mapping function, the coding parameters of multiple image blocks are determined according to their respective pre-allocated bit parameters. The rate-to-code parameter mapping function is constructed by taking the pre-allocated bit parameters of multiple image blocks as independent variables, the coding parameters of multiple image blocks as dependent variables, and the coefficients of the rate-to-code parameter mapping function corresponding to each segment of pre-allocated bit parameters. The segmented pre-allocated bit parameters are divided according to segmentation threshold parameters. Based on the coding parameters of multiple image blocks, the video frames of the high dynamic range video are encoded.

[0030] Figure 1 The illustration shows an application scenario of a high dynamic range video encoding method according to an embodiment of the present disclosure.

[0031] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0032] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0033] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0034] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0035] It should be noted that the high dynamic range video encoding method provided in this embodiment can generally be executed by server 105. Correspondingly, the high dynamic range video encoding apparatus provided in this embodiment can generally be located in server 105. The high dynamic range video encoding method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the high dynamic range video encoding apparatus provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0036] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0037] The following will be based on Figure 1 The described scene, through Figures 2-4 The high dynamic range video encoding method of the disclosed embodiments is described in detail.

[0038] Figure 2 A flowchart illustrating a method for encoding high dynamic range video according to an embodiment of the present disclosure is shown.

[0039] like Figure 2 As shown, the high dynamic range video encoding method 200 of this embodiment includes operations S210 to S260.

[0040] Using the S210, high dynamic range video is acquired.

[0041] In operation S220, the first pixel domain features of the video frame of the high dynamic range video are processed using the first deep residual network, and the segmented threshold parameters of the bitrate coding parameter mapping function are output.

[0042] In operation S230, the video frames are divided into blocks to obtain multiple image blocks.

[0043] In operation S240, the second deep residual network is used to process the second pixel domain features of multiple image blocks, and outputs the coefficients of the rate coding parameter mapping function corresponding to each of the multiple image blocks.

[0044] In operation S250, based on the rate coding parameter mapping function, the coding parameters of multiple image blocks are determined according to the pre-allocated bit parameters of each of the multiple image blocks.

[0045] In operation S260, video frames of high dynamic range video are encoded based on the encoding parameters of multiple image blocks.

[0046] According to embodiments of this disclosure, high dynamic range video can characterize video formed using high dynamic range imaging technology, providing a richer visual experience than conventional standard dynamic range video by expanding the luminance and color gamut. High dynamic range video may include multiple video frames.

[0047] According to embodiments of this disclosure, the deep residual network can be a wavelet convolution-based deep residual network. The deep residual network can include a first deep residual network and a second deep residual network. The first deep residual network can be trained using the segmentation threshold parameters of the bitrate coding parameter mapping function of sample video frames as labels. The second deep residual network can be trained using the coefficients of the bitrate coding parameter mapping function of sample image blocks as labels.

[0048] According to embodiments of this disclosure, the first pixel domain feature of a video frame in a high dynamic range video can be obtained by performing a 3x3 convolution on the video frame of the high dynamic range video using a first convolutional layer. The first pixel domain feature of the video frame in the high dynamic range video can be processed using a first deep residual network.

[0049] According to embodiments of this disclosure, multiple image blocks can be obtained by segmenting video frames. Each image block can be a coding tree unit. High dynamic range video frames can be segmented into 128x128 image blocks from left to right and top to bottom as input to a second deep residual network.

[0050] According to embodiments of this disclosure, the rate coding parameter mapping function can be a segmented function. For example, the rate coding parameter mapping function can be a segmented... The function is shown in the following formula (1):

[0051] (1)

[0052] in, This represents the minimum pre-allocated bit parameter for a video frame or image block. This represents the maximum pre-allocated bit parameter for a video frame or image block. , , , This represents the interval endpoints of the piecewise function.

[0053] According to embodiments of this disclosure, segmentation Number of intervals and segments of a function The fitting accuracy of the function is positively correlated. As the number of intervals increases, the difficulty of segmenting the threshold parameters and coefficients of the prediction bitrate coding parameter mapping function becomes increasingly greater. In order to achieve a balance between fitting accuracy and the difficulty of segmenting the threshold parameters and coefficients of the prediction bitrate coding parameter mapping function, experiments were conducted to explore the optimal number of intervals, and the results are shown in Table 1.

[0054] Table 1. Fitting accuracy of the rate coding parameter mapping function for different interval numbers

[0055]

[0056] According to embodiments of this disclosure, as shown in Table 1, BalloonFestival, EBU_04_Hurdles, ShowGirls, EBU_06_Starting, and S12-Sunrise represent different sequences. For different sequences, when the number of intervals is set to more than 2, the fitting accuracy is slightly improved, but the difficulty of predicting the segmented threshold parameters and coefficients of the rate coding parameter mapping function increases rapidly. Therefore, in embodiments of this disclosure, a rate coding parameter mapping function with two intervals can be used, as shown in the following formula (2):

[0057] (2)

[0058] in, and The coefficients of the rate-coding parameter mapping function for the first interval are represented. and The coefficients of the rate-coding parameter mapping function for the second interval are represented. This represents the segment threshold parameter of the rate coding parameter mapping function.

[0059] According to embodiments of this disclosure, It can be viewed as the low bitrate segment of the bitrate encoding parameter mapping function and It can be viewed as the high bitrate segment of the bitrate encoding parameter mapping function.

[0060] According to embodiments of this disclosure, the second pixel domain features of multiple image blocks can be obtained by performing a 3x3 convolution on the multiple image blocks using a first convolutional layer. By processing the second pixel domain features of the multiple image blocks using a second deep residual network, coefficients of the rate coding parameter mapping function corresponding to each of the multiple image blocks can be output.

[0061] According to embodiments of this disclosure, the rate coding parameter mapping function can be constructed by taking the pre-allocated bit parameters of multiple image blocks as independent variables, the coding parameters of multiple image blocks as dependent variables, and the coefficients of the rate coding parameter mapping function corresponding to each of the multiple pre-allocated bit parameters.

[0062] According to embodiments of this disclosure, the pre-allocated bit parameters for multiple segments can be divided based on segmentation threshold parameters. In one embodiment, the pre-allocated bit parameters for multiple segments can be the same.

[0063] According to embodiments of this disclosure, video frames of high dynamic range video can be encoded based on the encoding parameters of multiple image blocks, thereby enabling encoding based on the encoding parameters of different image blocks to obtain the bitrate corresponding to different image blocks.

[0064] According to embodiments of this disclosure, by processing the first pixel domain features of the video frames of the acquired high dynamic range video using a first deep residual network, segmentation threshold parameters of the bitrate coding parameter mapping function can be obtained. The video frames are then divided into blocks to obtain multiple image blocks. By processing the second pixel domain features of the multiple image blocks using a second deep residual network, coefficients of the bitrate coding parameter mapping function can be obtained. Based on the bitrate coding parameter mapping function, coding parameters for the multiple image blocks can be determined according to their respective pre-allocated bit parameters. Furthermore, by encoding the video frames of the high dynamic range video based on the coding parameters of the multiple image blocks, the bitrate corresponding to the multiple image blocks can be obtained. This improves the bitrate control accuracy of intra-frame coding in high dynamic range video, thereby improving the coding efficiency and quality of high dynamic range video.

[0065] Figure 3 A schematic diagram of a deep residual network according to an embodiment of the present disclosure is shown.

[0066] like Figure 3 As shown, the deep residual network includes an input module, a multi-level wavelet transform module, and a parameter regression module.

[0067] According to embodiments of this disclosure, when the deep residual network is represented as a second deep residual network, video frames of a high dynamic range video can be input into the input module, and the video frames can be segmented to obtain multiple image blocks. These multiple image blocks are then input into a multi-level wavelet transform module, where a first convolutional layer C1 performs convolution processing on the multiple image blocks to obtain pixel-domain features. Each wavelet transform module in the multi-level wavelet transform module can include cascaded residual sub-modules and discrete wavelet sub-modules. The multi-level wavelet transform module can include three wavelet transform modules: the first wavelet transform module includes a first residual sub-module R1, a second residual sub-module R2, and a first discrete wavelet sub-module W1; the second wavelet transform module includes a third residual sub-module R3, a fourth residual sub-module R4, and a second discrete wavelet sub-module W2; and the third wavelet transform module includes a fifth residual sub-module R5, a sixth residual sub-module R6, and a third discrete wavelet sub-module W3.

[0068] According to embodiments of this disclosure, the first residual submodule R1, the second residual submodule R2, the third residual submodule R3, the fourth residual word module R4, the fifth residual submodule R5, and the sixth residual word module R6 have the same structure, each including two residual blocks. The first discrete wavelet submodule W1, the second discrete wavelet submodule W2, and the third discrete wavelet submodule W3 have the same structure, each composed of a Haar filter.

[0069] According to embodiments of this disclosure, the parameter regression module can first perform an adaptive flattening operation on the second frequency domain features output by the multi-level wavelet transform module, and then perform mapping processing on the adaptive flattening operation using three fully connected layers (F1, F2, F3) to obtain the coefficients of the rate coding parameter mapping function.

[0070] According to embodiments of this disclosure, when the deep residual network is represented as a first deep residual network, the input module does not perform block processing on the video frames of the high dynamic range video. Instead, it directly inputs the video frames of the high dynamic range video into the first convolutional layer for convolution processing to obtain the first pixel domain features. Then, the first pixel domain features are input into the multi-level wavelet transform module for frequency domain decomposition processing to obtain the first frequency domain features. The parameter regression module can first perform an adaptive flattening operation on the first frequency domain features output by the multi-level wavelet transform module in a different way than the first frequency domain features. After the adaptive flattening operation, the mapping processing of three fully connected layers (F1, F2, F3) is performed to obtain the segmented threshold parameters of the bitrate coding parameter mapping function.

[0071] Table 2 Hyperparameters of Deep Residual Networks

[0072]

[0073] According to embodiments of this disclosure, a first pixel domain feature of a video frame in a high dynamic range video is processed using a first deep residual network to output a segmented threshold parameter of a bitrate coding parameter mapping function. This process includes: performing convolution processing on the video frame using a first convolutional layer to output the first pixel domain feature; performing frequency domain decomposition processing on the first pixel domain feature using a multi-level wavelet transform module to output the first frequency domain feature; and performing mapping processing on the first frequency domain feature using a parameter regression module to output the segmented threshold parameter of the bitrate coding parameter mapping function.

[0074] According to embodiments of this disclosure, the first deep residual network and the second deep residual network may have the same structure. Both the first deep residual network and the second deep residual network may include a first convolutional layer, a multi-level wavelet transform module, and a parameter regression module cascaded together.

[0075] According to embodiments of this disclosure, the first convolutional layer can be a 3x3 convolutional layer. The stride of the first convolutional layer can be 1. By utilizing the first convolutional layer to perform convolution processing on video frames, first pixel domain features can be obtained.

[0076] According to embodiments of this disclosure, a multi-level wavelet transform module may include multiple wavelet transform modules. For example, it may include three wavelet transform modules. Each wavelet transform module may include two residual submodules and a discrete wavelet submodule.

[0077] According to embodiments of this disclosure, the wavelet transform module can use Haar wavelet transform to perform frequency domain decomposition on the first pixel domain features to obtain the first frequency domain features. The Haar wavelet transform decomposes the first pixel domain features into one low-frequency sub-band and three high-frequency sub-bands using four convolutional filters. The four convolutional filters are defined as follows:

[0078]

[0079] in, This represents an approximate frequency domain component filter. This represents a horizontal frequency domain component filter. Represents the vertical frequency domain component filter and This represents a diagonal frequency domain component filter.

[0080] According to embodiments of this disclosure, an approximate frequency domain component filter It can be composed of two consecutive low-pass filters. Horizontal frequency domain component filter. It can be composed of a low-pass filter and a high-pass filter, and is a vertical frequency domain component filter. A diagonal frequency domain component filter can be constructed from a high-pass filter and a low-pass filter. It can be constructed from two consecutive high-pass filters. The step size of the approximate frequency domain component filter, the horizontal frequency domain component filter, the vertical frequency domain component filter, and the diagonal frequency domain component filter is 2.

[0081] According to embodiments of this disclosure, through Haar wavelet transform, the width and height of the pixel domain features are downsampled to half their original values, while the channels of the pixel domain features are expanded to four times their original values.

[0082] According to embodiments of this disclosure, in order to enhance the information interaction between features of different sub-bands, a 3×3 convolutional layer with a stride of 1 can be used to achieve effective feature fusion of one low-frequency sub-band and three high-frequency sub-bands. Furthermore, unlike traditional downsampling operations (such as pooling, bilinear interpolation, etc.), the wavelet transform module provides a lossless downsampling method, the working process of which is shown in the following formula (3):

[0083] (3)

[0084] in, This indicates a feature concatenation operation. This represents the convolution operation. This represents a 1×1 convolution operation. Represents the ReLU activation function. This represents the frequency domain subband characteristics obtained from the Haar wavelet transform. This represents the characteristics of the wavelet transform input. This represents the characteristics output by the wavelet transform module.

[0085] According to embodiments of this disclosure, a multi-level wavelet transform module can be used to perform frequency domain decomposition on the first pixel domain features, thereby obtaining the first frequency domain features. A parameter regression module can then be used to map the first frequency domain features to obtain the segmented threshold parameters of the rate coding parameter mapping function.

[0086] According to embodiments of this disclosure, the fully connected layers may include multiple cascaded layers. For example, three fully connected layers may be cascaded. The parameter regression module can map the first frequency domain features after adaptive flattening through the cascaded fully connected layers to obtain the segmented threshold parameters of the rate coding parameter mapping function.

[0087] According to embodiments of this disclosure, a second deep residual network is used to process the second pixel domain features of multiple image blocks, and outputs the coefficients of the rate coding parameter mapping function corresponding to each of the multiple image blocks. This includes: performing convolution processing on the multiple image blocks using a first convolutional layer to output the second pixel domain features; performing frequency domain decomposition processing on the second pixel domain features using a multi-level wavelet transform module to output the second frequency domain features; and performing mapping processing on the second frequency domain features using a parameter regression module to output the coefficients of the rate coding parameter mapping function corresponding to each of the multiple image blocks.

[0088] According to embodiments of this disclosure, the first convolutional layer can be a 3x3 convolutional layer with a stride of 1. The first convolutional layer is used to perform convolution processing on multiple image blocks, thereby outputting second pixel domain features.

[0089] According to embodiments of this disclosure, a multi-level wavelet transform module is used to perform frequency domain decomposition processing on the second pixel domain features, thereby outputting the second frequency domain features. A parameter regression module is then used to map the second frequency domain features, outputting the coefficients of the rate coding parameter mapping function corresponding to each of the multiple image blocks.

[0090] According to embodiments of this disclosure, the multi-level wavelet transform module can consist of three cascaded wavelet transform modules. The first wavelet transform module performs frequency domain decomposition on the pixel-domain features to obtain a first intermediate frequency domain feature. The first intermediate frequency domain feature is then input into the second wavelet transform module, which performs frequency domain decomposition again to obtain a second intermediate frequency domain feature. The second intermediate frequency domain feature is then input into the third wavelet transform module, which performs frequency domain decomposition again to obtain a second frequency domain feature. The processing procedures for the first pixel-domain features and the processing procedures for the second pixel-domain features using the multi-level wavelet transform are the same and will not be described again here.

[0091] According to embodiments of this disclosure, mapping the second frequency domain features can be performed using cascaded fully connected layers. For example, three fully connected layers can be cascaded.

[0092] According to embodiments of this disclosure, by performing convolution processing on video frames or multiple image blocks through a first convolutional layer, first pixel domain features or second pixel domain features can be obtained respectively. By processing the first pixel domain features or second pixel domain features using a multi-level wavelet transform module and a parameter regression module, the segmented threshold parameter of the bitrate coding parameter mapping function or the coefficient of the bitrate coding parameter mapping function can be obtained, thereby improving the accuracy of the segmented threshold parameter of the bitrate coding parameter mapping function or the coefficient of the bitrate coding parameter mapping function.

[0093] According to embodiments of this disclosure, a first pixel domain feature is decomposed in the frequency domain using a multi-level wavelet transform module to output a first frequency domain feature, comprising: inputting the first pixel domain feature into a residual submodule to output a first intermediate pixel domain feature; and decomposing the first intermediate pixel domain feature in the frequency domain using a discrete wavelet submodule to output the first frequency domain feature.

[0094] According to embodiments of this disclosure, each wavelet transform module in a multi-level wavelet transform module may include cascaded residual submodules and discrete wavelet submodules. For example, cascaded residual submodules may include two residual submodules.

[0095] According to embodiments of this disclosure, the residual submodule may include a first residual submodule and a second residual submodule. The residual submodule processes the features of the first pixel domain, which can increase the depth of the deep residual network and improve its performance.

[0096] According to embodiments of this disclosure, a first pixel-domain feature can be input into a residual submodule to output a first intermediate pixel-domain feature. A discrete wavelet submodule can then be used to perform frequency domain decomposition on the first intermediate pixel-domain feature to output a first frequency domain feature. The discrete wavelet submodule can process both pixel-domain and frequency-domain features.

[0097] According to embodiments of this disclosure, by inputting the first pixel domain feature into the residual submodule, the first intermediate pixel domain feature can be output. By inputting the first intermediate pixel domain feature into the discrete wavelet submodule, the first intermediate pixel domain feature can be subjected to frequency domain decomposition processing to output the first frequency domain feature. The first frequency domain feature can be input into the parameter regression module, thereby improving the accuracy of the segmented threshold parameter of the code rate coding parameter mapping function output by the parameter regression module.

[0098] According to embodiments of this disclosure, a parameter regression module is used to map the first frequency domain features and output a segmented threshold parameter of the rate coding parameter mapping function. This includes: performing an adaptive flattening operation based on global max pooling on the first frequency domain features to obtain a first target frequency domain feature; and using multiple fully connected layers cascaded together to map the first target frequency domain features and output a segmented threshold parameter of the rate coding parameter mapping function.

[0099] According to embodiments of this disclosure, a parameter regression module is used to map the second frequency domain features and output the coefficients of the rate coding parameter mapping function corresponding to each of the multiple image blocks. This includes: performing a tiling-based adaptive flattening operation on the second frequency domain features to obtain the second target frequency domain features; and using multiple fully connected layers cascaded together to map the second frequency domain features and output the coefficients of the rate coding parameter mapping function corresponding to each of the multiple image blocks.

[0100] According to embodiments of this disclosure, different adaptive flattening operations can be used on the first frequency domain feature and the second frequency domain feature. By performing an adaptive flattening operation based on global max pooling on the first frequency domain feature, the three-dimensional first frequency domain feature can be converted into a one-dimensional first target frequency domain feature. By performing an adaptive flattening operation based on tiling on the second frequency domain feature, the three-dimensional second frequency domain feature can be converted into a one-dimensional second target frequency domain feature.

[0101] According to embodiments of this disclosure, by mapping the first target frequency domain features using multiple cascaded fully connected layers, segmented threshold parameters of the rate coding parameter mapping function can be output. By mapping the second frequency domain features using multiple cascaded fully connected layers, coefficients of the rate coding parameter mapping function corresponding to each of the multiple image blocks can be output.

[0102] According to embodiments of this disclosure, by applying different adaptive flattening operations to the first frequency domain feature and the second frequency domain feature, a first target frequency domain feature corresponding to the first frequency domain feature and a second target frequency domain feature corresponding to the second frequency domain feature can be obtained. By mapping the first target frequency domain feature or the second target frequency domain feature using multiple fully connected layers cascaded together, the segmented threshold parameter or the coefficient of the rate coding parameter mapping function can be obtained, thereby improving the efficiency of high dynamic range video coding.

[0103] According to embodiments of this disclosure, encoding a video frame of a high dynamic range video based on the encoding parameters of multiple image blocks includes: determining frame-level encoding parameters of the high dynamic range video and assigning weights to the bit parameters of each of the multiple image blocks according to a bitrate encoding parameter mapping function and the total bit parameters of the video frame; determining encoding parameters of multiple image blocks according to the bit parameter assignment weights of each of the multiple image blocks and the size parameters of each of the multiple image blocks; and encoding the video frame of the high dynamic range video based on the encoding parameters of the multiple image blocks.

[0104] According to embodiments of this disclosure, the total bit parameter of a video frame can characterize the pre-allocated total bits of a video frame. The frame-level coding parameters of a high dynamic range video and the bit parameter allocation weights for each of the multiple image blocks can be determined based on the bitrate coding parameter mapping function and the total bit parameter of the video frame.

[0105] According to embodiments of this disclosure, the size of a video frame may include its length and width. The pre-allocated bitrate of the video frame can be determined based on the total bit parameters and the size of the video frame. The calculation is as follows: (4)

[0106] (4)

[0107] in, This represents the total bit parameters of a video frame. Indicates the length of a video frame. Indicates the width of a video frame.

[0108] According to embodiments of this disclosure, when the rate coding parameter mapping function is determined, the segmentation threshold parameter and the coefficients of the rate coding parameter mapping function are also determined. It can be determined whether the pre-allocated bits belong to a high-rate segment or a low-rate segment based on the rate coding parameter mapping function. The interval decision function of the rate coding parameter mapping function is as follows: (5)

[0109] (5)

[0110] in, The interval decision function representing the segmented code rate coding parameter mapping function; Represents the i-th image patch. This represents the coefficient of the rate coding parameter mapping function corresponding to the i-th image block. This represents the segment threshold parameter of the rate coding parameter mapping function corresponding to the i-th image block.

[0111] According to embodiments of this disclosure, the frame-level coding parameters λ of a high dynamic range video can be determined based on the bitrate coding parameter mapping function and the total bits of the video frame. The following formula (6) can be solved using Newton's iteration method:

[0112] (6)

[0113] in, This represents the set of coefficients for the rate coding parameter mapping function corresponding to the i-th image block.

[0114] According to embodiments of this disclosure, calculating the frame-level coding parameters of a high dynamic range video using Newton's iteration method may include the following steps: defining an equation: Input: Initial Number of iterations and error threshold The error threshold can be set to 0.0001; Output: Final For calculating the gradient: ,renew : Update equations Repeat until... .

[0115] According to embodiments of this disclosure, when the frame-level coding parameters of a high dynamic range video are determined, the pre-allocated bit parameters of multiple image blocks can be determined according to the following formula (7).

[0116] (7)

[0117] in, This represents the pre-allocated bit parameters for the i-th image block.

[0118] According to embodiments of this disclosure, the bit parameter allocation weights of multiple image blocks can be determined based on the pre-allocated bit parameters of multiple image blocks and the total bit parameters of the video frame, as shown in the following formula (8):

[0119] (8)

[0120] in, This represents the weighting of the bit parameters for the i-th image block.

[0121] According to embodiments of this disclosure, weights can be assigned based on the bit parameters of each of the multiple image blocks and the size parameters of each of the multiple image blocks, the encoding parameters of each of the multiple image blocks can be determined, and the video frames of the high dynamic range video can be encoded based on the encoding parameters of each of the multiple image blocks.

[0122] According to embodiments of this disclosure, frame-level coding parameters and bit parameter allocation weights for multiple image blocks can be determined based on the bitrate coding parameter mapping function and the total bit parameters of the video frame. Then, the coding parameters for each of the multiple image blocks can be determined based on the frame-level coding parameters. By encoding the video frames of the high dynamic range video based on the coding parameters of each of the multiple image blocks, the accuracy of bitrate control in intra-frame coding can be improved, thereby improving the coding efficiency of the high dynamic range video.

[0123] According to embodiments of this disclosure, the encoding parameters of multiple image blocks are determined based on the weights allocated to the bit parameters of each image block and the size parameters of each image block, including: determining the target allocated bits of each image block based on the weights allocated to the bit parameters of each image block; determining the target bit rate of each image block based on the target allocated bits of each image block and the size parameters of each image block; and determining the encoding parameters of the multiple image blocks based on the target bit rate and the bit rate encoding parameter mapping function of the multiple image blocks.

[0124] According to embodiments of this disclosure, the target allocation bits for each of the multiple image blocks can be determined based on the bit parameters of each of the multiple image blocks, as shown in the following formula (9):

[0125] (9)

[0126] in, This represents the target allocation bits for the i-th image block. The sliding window used to smooth bit errors can be set to 4 in embodiments of this disclosure; This indicates the number of image blocks included in a video frame. This indicates the number of image blocks encoded in a video frame. This represents the remaining bits for encoding. According to embodiments of this disclosure, the target bit rate of multiple image blocks can be determined based on their respective target allocation bits and size parameters. The encoding parameters of the multiple image blocks can be determined based on their target bit rates and a bit rate encoding parameter mapping function, as shown in the following formula (10):

[0127] (10)

[0128] According to embodiments of this disclosure, encoding high dynamic range video frames based on the encoding parameters of multiple image blocks can improve the accuracy of bitrate control in intra-frame coding, thereby improving coding efficiency.

[0129] According to embodiments of this disclosure, the data-driven intra-frame bitrate control method for high dynamic range (HMR) video implemented by the HMR video coding method of this disclosure is tested using the standard HMR video test sequences EBU_04_Hurdles, EBU_06_Starting, ShowGirl2, and S12-Sunrise, as well as manually constructed test sequences TJU_Tree, TJU_Lake, and TJU_Park. The data-driven intra-frame bitrate control method for HMR video constructed in this disclosure can achieve precise bitrate control and improve coding performance for HMR video without any prior knowledge; therefore, only the first frame of each video sequence needs to be tested. The metrics used to characterize bitrate control performance, "bit allocation accuracy" and "rate distortion performance," are shown in Tables 3 and 4.

[0130] Table 3 Comparison of Bit Allocation Accuracy of Different Rate Control Methods in High Dynamic Range Standard Test Sequences

[0131]

[0132] According to embodiments of this disclosure, the target bit rate can be calculated based on a fixed QP. This represents the total number of bits allocated to the video frame. This represents the actual number of bits allocated under a fixed QP. The error rate is represented by the value '\'. Both Relevant Method 1 and Relevant Method 2 are rate control methods within the relevant techniques. As shown in Table 3, compared to Relevant Method 1 and Relevant Method 2, the bit allocation error rate of this method is 1.24%, achieving a bit allocation result closest to the target bit rate, thus enabling more accurate intra-frame rate control. The proposed high dynamic range (HDR) video coding method's implementation of the λ-domain HDR video intra-frame rate control method mainly benefits from the fact that the proposed rate coding parameter mapping function can accurately describe the HDR video intra-frame rate distortion characteristics. Furthermore, the proposed wavelet convolution-based deep residual network can effectively analyze the HDR video content characteristics and accurately predict the threshold parameters and coefficients of the rate coding parameter mapping function.

[0133] Table 4 Comparison of coding rate distortion performance of different code rate control methods in high dynamic range standard test sequences.

[0134]

[0135] According to embodiments of this disclosure, Bjontegaard delta rate (BD-Rate: η1) and Bjontegaard delta wPSNR (BD-wPSNR: δ1) can be used to evaluate rate-distortion performance. As shown in Table 4, compared to the default intra-frame rate control method of VTM, the intra-frame rate control method implemented by the high dynamic range video coding method of this disclosure embodiment achieves an average coding efficiency improvement of 1.67%. Compared to related methods 1 and 2, this method achieves average coding efficiency improvements of 0.95% and 6.70%, respectively.

[0136] Figure 4 A flowchart illustrating a training method for a deep residual network according to an embodiment of the present disclosure is shown schematically.

[0137] like Figure 4 As shown, the training method 400 of the deep residual network includes operations S410 to S460.

[0138] Using the S410, training samples for the deep residual network are obtained.

[0139] In operation S420, the first pixel domain features of the sample video frames of the training samples are processed using the first deep residual network, and the prediction threshold parameter of the bitrate coding parameter mapping function is output.

[0140] In operation S430, the sample video frames of the training samples are divided into blocks to obtain multiple sample image blocks.

[0141] In operation S440, the second deep residual network is used to process the second pixel domain features of multiple sample image blocks, and the prediction coefficients of the rate coding parameter mapping function corresponding to each of the multiple sample image blocks are output.

[0142] In operation S450, the loss value is determined based on the target loss function, the training samples, and the training sample labels.

[0143] In the S460 operation, based on the loss value, the prediction threshold parameter and prediction coefficient of the rate coding parameter mapping function are adjusted to obtain the trained deep residual network.

[0144] According to embodiments of this disclosure, a deep residual network may include a first deep residual network and a second deep residual network, and the first deep residual network and the second deep residual network may have the same structure.

[0145] According to embodiments of this disclosure, the training samples may be artificially constructed high dynamic range videos or directly acquired high dynamic range videos.

[0146] According to embodiments of this disclosure, in order to ensure the diversity of training samples during the construction process, 229 sets of images were collected in one embodiment. Each set of images contains 5 standard dynamic range images of the same scene taken at different exposure levels. The 5 exposure levels are respectively set to... , And 0. To ensure the constructed training samples contain richer multi-scale details, the training samples include samples with two different resolutions: 3840×2160 and 2336×2336. Furthermore, images were captured in well-lit, static environments to ensure the maximum brightness level of each training sample and effectively avoid ghosting introduced during the subsequent fusion process.

[0147] According to embodiments of this disclosure, a set of five standard dynamic range images can be fused into a single high dynamic range image. To ensure the subsequent encoding and display of the training samples, each training sample can be converted from a floating-point number to an integer using the PQ photoelectric conversion function. The training samples are then converted to YUV420 format and compressed using VVC reference software. The compression process uses 11 different quantization parameters with intervals of 2 to 2 from 20 to 40.

[0148] According to embodiments of this disclosure, a first deep residual network can be used to process the first pixel domain features of the sample video frames of the training samples, and output the prediction threshold parameter of the bitrate coding parameter mapping function.

[0149] According to embodiments of this disclosure, multiple image blocks can be obtained by dividing the training samples into blocks. A second deep residual network can be used to process the second pixel domain features of the image blocks, outputting prediction coefficients of the rate coding parameter mapping functions corresponding to each of the multiple image blocks.

[0150] According to embodiments of this disclosure, the target loss function may be the mean absolute error. The training sample labels may characterize the target threshold parameter and target coefficients of the rate-to-code parameter mapping function. The loss value between the predicted threshold parameter and coefficients of the rate-to-code parameter mapping function and the training sample labels can be calculated based on the training samples and their labels.

[0151] According to embodiments of this disclosure, the training sample labels can characterize the target threshold parameter and target coefficients of the rate-coding parameter mapping function obtained based on the validation samples. The training and validation samples are divided into a manually constructed dataset in a 9:1 ratio.

[0152] According to embodiments of this disclosure, the deep residual network can use Adam as the parameter optimizer, with 300 epochs of training, an initial learning rate of 0.0001, and the learning rate decays to 0.8 times the original learning rate every 100 epochs.

[0153] According to embodiments of this disclosure, a trained deep residual network can be obtained by adjusting the prediction threshold parameter and prediction coefficient of the rate coding parameter mapping function based on the loss value.

[0154] Based on the above-described high dynamic range (HDR) video encoding method, this disclosure also provides an HDR video encoding apparatus. The following will be combined with... Figure 5 The device is described in detail.

[0155] Figure 5 A schematic block diagram of a high dynamic range video encoding apparatus according to an embodiment of the present disclosure is shown.

[0156] like Figure 5 As shown, the high dynamic range video encoding device 500 of this embodiment includes a first acquisition module 510, a first extraction module 520, a first block segmentation module 530, a second extraction module 540, a first determination module 550, and an encoding module 560.

[0157] The first acquisition module 510 is used to acquire high dynamic range video. In one embodiment, the first acquisition module 510 can be used to perform the operation S210 described above, which will not be repeated here.

[0158] The first extraction module 520 is used to process the first pixel domain features of the video frames of the high dynamic range video using a first deep residual network, and output the segmented threshold parameters of the bitrate coding parameter mapping function. The first deep residual network is trained using the segmented threshold parameters of the bitrate coding parameter mapping function of the sample video frames as labels. In one embodiment, the first extraction module 520 can be used to perform the operation S220 described above, which will not be repeated here.

[0159] The first segmentation module 530 is used to segment video frames into blocks to obtain multiple image blocks. In one embodiment, the first segmentation module 530 can be used to perform the operation S230 described above, which will not be repeated here.

[0160] The second extraction module 540 is used to process the second pixel domain features of multiple image blocks using a second deep residual network, and output the coefficients of the rate coding parameter mapping function corresponding to each of the multiple image blocks. The second deep residual network is trained using the coefficients of the rate coding parameter mapping function of the sample image blocks as labels. In one embodiment, the second extraction module 540 can be used to perform the operation S240 described above, which will not be repeated here.

[0161] The first determining module 550 is used to determine the encoding parameters of multiple image blocks based on their respective pre-allocated bit parameters. The rate encoding parameter mapping function is constructed by taking the pre-allocated bit parameters of the multiple image blocks as independent variables and the encoding parameters of the multiple image blocks as dependent variables, and the coefficients of the rate encoding parameter mapping functions corresponding to the pre-allocated bit parameters of each segment. The pre-allocated bit parameters are divided according to segmentation threshold parameters. In one embodiment, the first determining module 550 can be used to execute the operation S250 described above, which will not be repeated here.

[0162] The encoding module 560 is used to encode video frames of a high dynamic range video based on encoding parameters of multiple image blocks. In one embodiment, the encoding module 560 can be used to perform the operation S260 described above, which will not be repeated here.

[0163] According to embodiments of this disclosure, the first deep residual network and the second deep residual network have the same structure, including a first convolutional layer cascaded with each other, a multi-level wavelet transform module, and a parameter regression module. The first extraction module 520 includes:

[0164] The first extraction submodule is used to perform convolution processing on the video frame using the first convolutional layer and output the first pixel domain features.

[0165] The second extraction submodule is used to perform frequency domain decomposition on the first pixel domain features using a multi-level wavelet transform module, and output the first frequency domain features.

[0166] The third extraction submodule is used to map the first frequency domain features using the parameter regression module and output the segmented threshold parameters of the code rate coding parameter mapping function.

[0167] According to embodiments of this disclosure, the second extraction module 540 includes:

[0168] The fourth extraction submodule is used to perform convolution processing on multiple image blocks using the first convolutional layer to output the second pixel domain features.

[0169] The fifth extraction submodule is used to perform frequency domain decomposition on the second pixel domain features using a multi-level wavelet transform module, and output the second frequency domain features.

[0170] The sixth extraction submodule is used to map the second frequency domain features using the parameter regression module, and outputs the coefficients of the rate coding parameter mapping function corresponding to each of the multiple image blocks.

[0171] According to embodiments of this disclosure, each wavelet transform module in the multi-level wavelet transform module includes a residual sub-module and a discrete wavelet sub-module cascaded together. The second extraction sub-module includes:

[0172] The first extraction unit is used to input the first pixel domain features into the residual submodule and output the first intermediate pixel domain features.

[0173] The second extraction unit is used to perform frequency domain decomposition processing on the first intermediate pixel domain features using the discrete wavelet submodule, and output the first frequency domain features.

[0174] According to embodiments of this disclosure, the third extraction submodule includes:

[0175] The third extraction unit is used to perform an adaptive flattening operation based on global max pooling on the first frequency domain features to obtain the first target frequency domain features.

[0176] The fourth extraction unit is used to map the frequency domain features of the first target using multiple fully connected layers cascaded together, and output the segmented threshold parameters of the code rate coding parameter mapping function.

[0177] According to embodiments of this disclosure, the sixth extraction submodule includes:

[0178] The fifth extraction unit is used to perform an adaptive flattening operation based on tiling on the second frequency domain features to obtain the second target frequency domain features.

[0179] The sixth extraction unit is used to map the second frequency domain features using multiple fully connected layers cascaded together, and outputs the coefficients of the rate coding parameter mapping function corresponding to each of the multiple image blocks.

[0180] According to embodiments of this disclosure, the encoding module 560 includes:

[0181] The first encoding submodule is used to determine the frame-level encoding parameters of the high dynamic range video and assign weights to the bit parameters of multiple image blocks based on the bitrate encoding parameter mapping function and the total bit parameters of the video frames.

[0182] The second encoding submodule is used to determine the encoding parameters of multiple image blocks by allocating weights according to the bit parameters of each image block and the size parameters of each image block.

[0183] The third encoding submodule is used to encode video frames of high dynamic range video based on the encoding parameters of multiple image blocks.

[0184] According to embodiments of this disclosure, the second encoding submodule includes:

[0185] The first encoding unit is used to assign weights to multiple image blocks according to their respective bit parameters, and to determine the target allocated bits for each of the multiple image blocks.

[0186] The second encoding unit is used to determine the target bit rate of the multiple image blocks based on their respective target allocation bits and size parameters. The third encoding unit is used to determine the encoding parameters of the multiple image blocks based on their target bit rates and a bit rate encoding parameter mapping function.

[0187] According to embodiments of this disclosure, any plurality of modules among the first acquisition module 510, first extraction module 520, first block segmentation module 530, second extraction module 540, first determination module 550, and encoding module 560 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the first acquisition module 510, first extraction module 520, first block segmentation module 530, second extraction module 540, first determination module 550, and encoding module 560 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these methods. Alternatively, at least one of the first acquisition module 510, the first extraction module 520, the first block segmentation module 530, the second extraction module 540, the first determination module 550, and the encoding module 560 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0188] Based on the above-described training method for deep residual networks, this disclosure also provides a training apparatus for deep residual networks. The following will be combined with... Figure 6 The device is described in detail.

[0189] Figure 6 A schematic block diagram of a training apparatus for a deep residual network according to an embodiment of the present disclosure is shown.

[0190] like Figure 6 As shown, the deep residual network training device 600 of this embodiment includes a second acquisition module 610, a third extraction module 620, a second block segmentation module 630, a fourth extraction module 640, a second determination module 650, and a training module 660.

[0191] The second acquisition module 610 is used to acquire training samples of the deep residual network, wherein the deep residual network includes a first deep residual network and a second deep residual network. In one embodiment, the second acquisition module 610 can be used to perform the operation S410 described above, which will not be repeated here.

[0192] The third extraction module 620 is used to process the first pixel domain features of the sample video frames of the training samples using the first deep residual network, and output the prediction threshold parameter of the bitrate coding parameter mapping function. In one embodiment, the third extraction module 620 can be used to perform the operation S420 described above, which will not be repeated here.

[0193] The second block-segmentation module 630 is used to segment the sample video frames of the training samples to obtain multiple sample image blocks. In one embodiment, the second block-segmentation module 630 can be used to perform the operation S430 described above, which will not be repeated here.

[0194] The fourth extraction module 640 is used to process the second pixel domain features of multiple sample image blocks using the second deep residual network, and output the prediction coefficients of the rate coding parameter mapping function corresponding to each of the multiple sample image blocks. In one embodiment, the fourth extraction module 640 can be used to perform the operation S440 described above, which will not be repeated here.

[0195] The second determining module 650 is used to determine the loss value based on the target loss function, according to the training samples and their labels. In one embodiment, the second determining module 650 can be used to perform the operation S450 described above, which will not be repeated here.

[0196] Training module 660 is used to adjust the prediction threshold parameter and prediction coefficients of the rate coding parameter mapping function based on the loss value to obtain a trained deep residual network. In one embodiment, training module 660 can be used to perform the operation S460 described above, which will not be repeated here.

[0197] According to embodiments of this disclosure, any multiple modules among the second acquisition module 610, the third extraction module 620, the second block module 630, the fourth extraction module 640, the second determination module 650, and the training module 660 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the second acquisition module 610, the third extraction module 620, the second block module 630, the fourth extraction module 640, the second determination module 650, and the training module 660 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the second acquisition module 610, the third extraction module 620, the second block module 630, the fourth extraction module 640, the second determination module 650, and the training module 660 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0198] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a coding method for high dynamic range video according to an embodiment of the present disclosure.

[0199] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0200] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0201] According to embodiments of this disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0202] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0203] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703 described above.

[0204] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the high dynamic range video encoding method provided in the embodiments of this disclosure.

[0205] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0206] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0207] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0208] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0209] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0210] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0211] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for encoding high dynamic range video, characterized in that, include: Acquire high dynamic range video; The first pixel domain features of the video frame of the high dynamic range video are processed using a first deep residual network to output the segmented threshold parameters of the bitrate coding parameter mapping function. The first deep residual network is trained using the segmented threshold parameters of the bitrate coding parameter mapping function of the sample video frame as labels. The video frames are divided into blocks to obtain multiple image blocks; The second pixel domain features of the plurality of image blocks are processed using a second deep residual network to output the coefficients of the rate coding parameter mapping function corresponding to each of the plurality of image blocks. The second deep residual network is trained with the coefficients of the rate coding parameter mapping function of the sample image blocks as labels. Based on the rate coding parameter mapping function, the coding parameters of the plurality of image blocks are determined according to their respective pre-allocated bit parameters. The rate coding parameter mapping function is constructed by taking the pre-allocated bit parameters of the plurality of image blocks as independent variables and the coding parameters of the plurality of image blocks as dependent variables, and then mapping them with the coefficients of the rate coding parameter mapping function corresponding to each segment of pre-allocated bit parameters. The multiple segments of pre-allocated bit parameters are divided according to the segmentation threshold parameter. The video frames of the high dynamic range video are encoded based on the encoding parameters of the multiple image blocks.

2. The method according to claim 1, characterized in that, The first deep residual network and the second deep residual network have the same structure, including a first convolutional layer cascaded with each other, a multi-level wavelet transform module, and a parameter regression module. The first deep residual network is used to process the first pixel-domain features of the video frames of the high dynamic range video, outputting a segmented threshold parameter for the bitrate coding parameter mapping function, including: The video frame is convolved using the first convolutional layer to output the first pixel domain feature. The first pixel domain feature is decomposed in the frequency domain using the multi-level wavelet transform module to output the first frequency domain feature. The parameter regression module is used to map the first frequency domain features and output the segmented threshold parameters of the code rate coding parameter mapping function.

3. The method according to claim 2, characterized in that, The step of processing the second pixel domain features of the plurality of image blocks using a second deep residual network to output the coefficients of the rate coding parameter mapping function corresponding to each of the plurality of image blocks includes: The first convolutional layer is used to perform convolution processing on the plurality of image blocks to output the second pixel domain features; The second pixel domain feature is decomposed in the frequency domain using the multi-level wavelet transform module to output the second frequency domain feature. The parameter regression module is used to map the second frequency domain features, and outputs the coefficients of the rate coding parameter mapping function corresponding to each of the plurality of image blocks.

4. The method according to claim 2, characterized in that, Each wavelet transform module in the multi-level wavelet transform module includes a residual sub-module and a discrete wavelet sub-module cascaded together. The step of using the multi-level wavelet transform module to perform frequency domain decomposition processing on the first pixel domain features and outputting the first frequency domain features includes: The first pixel domain feature is input into the residual submodule, and the first intermediate pixel domain feature is output. The first intermediate pixel domain feature is decomposed in the frequency domain using the discrete wavelet submodule, and the first frequency domain feature is output.

5. The method according to claim 4, characterized in that, The step of mapping the first frequency domain features using the parameter regression module to output the segmented threshold parameters of the code rate coding parameter mapping function includes: Perform an adaptive flattening operation based on global max pooling on the first frequency domain features to obtain the first target frequency domain features; The frequency domain features of the first target are mapped using multiple fully connected layers cascaded together, and the segmented threshold parameters of the rate coding parameter mapping function are output.

6. The method according to claim 3, characterized in that, The step of mapping the second frequency domain features using the parameter regression module to output the coefficients of the rate coding parameter mapping function corresponding to each of the plurality of image blocks includes: An adaptive flattening operation based on tiling is performed on the second frequency domain features to obtain the second target frequency domain features; The second frequency domain features are mapped using the multiple fully connected layers that are cascaded together, and the coefficients of the rate coding parameter mapping function corresponding to each of the multiple image blocks are output.

7. The method according to claim 1, characterized in that, The encoding process for the video frames of the high dynamic range video based on the encoding parameters of the multiple image blocks includes: Based on the bitrate coding parameter mapping function and the total bit parameters of the video frame, determine the frame-level coding parameters of the high dynamic range video and the bit parameter allocation weights of each of the multiple image blocks; The encoding parameters of the multiple image blocks are determined by assigning weights based on the bit parameters of each of the multiple image blocks and the size parameters of each of the multiple image blocks. The video frames of the high dynamic range video are encoded based on the encoding parameters of the multiple image blocks.

8. The method according to claim 7, characterized in that, The step of assigning weights based on the bit parameters of each of the multiple image blocks and determining the encoding parameters of each of the multiple image blocks includes: Based on the bit parameters of each of the multiple image blocks, weights are assigned, and the target allocated bits for each of the multiple image blocks are determined. The target bit rate of the plurality of image blocks is determined based on the target allocated bits of each of the plurality of image blocks and the size parameters of each of the plurality of image blocks; The encoding parameters of the plurality of image blocks are determined based on the target bit rate of the plurality of image blocks and the bit rate encoding parameter mapping function.

9. A training method for a deep residual network, characterized in that, include: Obtain training samples for the deep residual network, wherein the deep residual network includes a first deep residual network and a second deep residual network; The first pixel domain features of the sample video frames of the training samples are processed using the first deep residual network to output the prediction threshold parameter of the bitrate coding parameter mapping function. The sample video frames of the training samples are divided into blocks to obtain multiple sample image blocks; The second deep residual network is used to process the second pixel domain features of the plurality of sample image blocks, and the prediction coefficients of the rate coding parameter mapping function corresponding to each of the plurality of sample image blocks are output. Based on the target loss function, the loss value is determined according to the training samples and training sample labels; Based on the loss value, the prediction threshold parameter and the prediction coefficient of the rate coding parameter mapping function are adjusted to obtain a trained deep residual network.

10. A high dynamic range video encoding apparatus, characterized in that, include: The first acquisition module is used to acquire high dynamic range video; The first extraction module is used to process the first pixel domain features of the video frame of the high dynamic range video using a first deep residual network, and output the segmented threshold parameters of the bitrate coding parameter mapping function, wherein the first deep residual network is trained with the segmented threshold parameters of the bitrate coding parameter mapping function of the sample video frame as labels. The first segmentation module is used to segment the video frame into multiple image blocks. The second extraction module is used to process the second pixel domain features of the plurality of image blocks using a second deep residual network, and output the coefficients of the rate coding parameter mapping function corresponding to each of the plurality of image blocks, wherein the second deep residual network is trained with the coefficients of the rate coding parameter mapping function of the sample image blocks as labels. The first determining module is used to determine the encoding parameters of the plurality of image blocks based on the bitrate encoding parameter mapping function and according to the pre-allocated bit parameters of each of the plurality of image blocks. The bitrate encoding parameter mapping function is constructed by taking the pre-allocated bit parameters of the plurality of image blocks as independent variables and the encoding parameters of the plurality of image blocks as dependent variables, and the coefficients of the bitrate encoding parameter mapping function corresponding to each segment of pre-allocated bit parameters. The multiple segments of pre-allocated bit parameters are divided according to the segmentation threshold parameter. The encoding module is used to encode the video frames of the high dynamic range video based on the encoding parameters of the plurality of image blocks.

Images