A dynamic metadata-based code rate control and encoding method, device, equipment and medium

Abstract

The application discloses a dynamic metadata-based code rate control and coding method, device, equipment and medium, and relates to the field of video coding, and comprises the following steps: acquiring dynamic metadata corresponding to a video stream; inputting the dynamic metadata into a visual complexity prediction model to determine a dynamic range factor and a brightness deviation factor based on the dynamic metadata, and performing weighted fusion on the dynamic range factor and the brightness deviation factor to obtain a visual coding difficulty coefficient corresponding to each frame; determining a code rate allocation weight based on the visual coding difficulty coefficient, fusing the code rate allocation weight and a corresponding initial code rate budget, and obtaining a target code rate corresponding to each frame; dividing the frame into a first target frame and a second target frame according to the visual coding difficulty coefficient; adjusting a quantization parameter of each coding tree unit in the first target frame to determine a new quantization parameter corresponding to each coding tree unit; and coding based on the target code rate corresponding to each frame and the quantization parameter. The application improves coding efficiency and picture quality.

Description

Technical Field

[0001] This application relates to the field of video coding, and in particular to a bitrate control and coding method, apparatus, device and medium based on dynamic metadata. Background Technology

[0002] With the rapid development of ultra-high-definition video technology, High Dynamic Range (HDR) video, with its wider brightness range, richer color gradations, and more delicate detail, has been widely used in film and television production, live streaming, terminal display, and other fields, becoming one of the core technologies for improving the video viewing experience. HDR Vivid, as the mainstream HDR video standard, introduces dynamic metadata to accurately describe the brightness characteristics of video frames, thereby restoring the visual intent of content creators on different display devices and presenting a viewing effect closer to real-life scenes.

[0003] However, current HDR Vivid encoding does not fully utilize this dynamic metadata, resulting in several shortcomings in existing solutions: First, current bitrate control relies primarily on pixel statistics within the encoder, estimating the complexity of the next frame based on the statistical information of already encoded frames. Since the visual characteristics of future frames cannot be predicted, the bitrate controller cannot proactively plan resources for upcoming scene transitions, making it difficult to plan and allocate bitrate resources in advance. Second, the human eye exhibits significant differences in sensitivity to different brightness areas. Current solutions allocate bitrate solely based on pixel statistics, failing to translate these visual perception differences into quantified encoding guidance signals. This leads to inaccurate bitrate allocation for dark scenes, bright scenes, and scenes with a large dynamic range, potentially causing problems such as noise in dark areas and loss of detail in bright areas. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a bitrate control and encoding method, apparatus, device, and medium based on dynamic metadata, which achieves efficient HDR encoding with both forward-looking and perceptual optimization, significantly improving the subjective image quality of video. The specific solution is as follows:

[0005] Firstly, this application provides a rate control and encoding method based on dynamic metadata, including:

[0006] Obtain dynamic metadata corresponding to the video stream; the dynamic metadata includes the maximum brightness, minimum brightness, and average brightness of each frame in the video stream.

[0007] The dynamic metadata is input into a pre-built visual complexity prediction model to determine a dynamic range factor and a brightness deviation factor based on the dynamic metadata. The dynamic range factor and the brightness deviation factor are then weighted and fused to obtain the visual coding difficulty coefficient for each frame. The dynamic range factor is used to quantify the difference between the maximum and minimum brightness within a frame, and the brightness deviation factor is used to reflect the degree to which the average brightness of the frame deviates from the preset most comfortable brightness range.

[0008] Based on the visual coding difficulty coefficient, the bitrate allocation weight corresponding to each frame is determined, and the bitrate allocation weight and the corresponding initial bitrate budget are fused to obtain the target bitrate corresponding to each frame; the initial bitrate budget is the bitrate budget allocated to each frame through a preset bitrate control model.

[0009] Based on the visual coding difficulty coefficient, the frame is divided into a first target frame that meets the preset low visual coding difficulty standard and a second target frame that meets the preset high visual coding difficulty standard.

[0010] Adaptive quantization parameter adjustment is performed on each coding tree unit in the first target frame to determine the new quantization parameter corresponding to each coding tree unit. The first target frame and the second target frame are encoded based on the target bitrate and quantization parameter corresponding to each frame. Each frame contains at least one coding tree unit.

[0011] Optionally, the process of determining the preset most comfortable brightness range includes:

[0012] The lower limit and upper limit of the preset most comfortable brightness range are set based on the characteristics of human vision.

[0013] Alternatively, determine the statistical value of the average brightness of a preset number of consecutive frames in the video stream, multiply the statistical value by a first preset coefficient to obtain the lower limit of the preset most comfortable brightness range, and multiply the statistical value by a second preset coefficient to obtain the upper limit of the preset most comfortable brightness range; the statistical value is the median or average value, the first preset coefficient is less than 1, and the second preset coefficient is greater than 1.

[0014] Optionally, the step of determining the dynamic range factor and the brightness deviation factor based on the dynamic metadata, and then weighting and fusing the dynamic range factor and the brightness deviation factor to obtain the visual coding difficulty coefficient corresponding to each frame, includes:

[0015] The dynamic range factor is obtained based on the difference between the maximum and minimum brightness in the dynamic metadata, and the brightness deviation factor is determined based on the preset most comfortable brightness range and the average brightness in the dynamic metadata.

[0016] The dynamic range factor is mapped using a first preset monotonically increasing function to obtain a first mapped value. The brightness deviation factor is mapped using a second preset monotonically increasing function to obtain a second mapped value. The first mapped value and the second mapped value are then weighted and fused to obtain a target weighted sum.

[0017] The target weighted sum is multiplied by the function value output by the preset masking function to obtain the visual coding difficulty coefficient corresponding to each frame. The preset masking function is a function that is related to the average brightness of the frame and conforms to the visual characteristics of the human eye. It outputs a high function value in the preset low brightness area to maintain or amplify the coding difficulty, and outputs a low function value in the preset high brightness area to suppress the coding difficulty.

[0018] Optionally, the step of determining the bitrate allocation weights corresponding to each frame based on the visual coding difficulty coefficient, and fusing the bitrate allocation weights with the corresponding initial bitrate budget to obtain the target bitrate for each frame, includes:

[0019] For any image group, the bitrate allocation weight corresponding to each frame in the image group is determined based on the visual coding difficulty coefficient; wherein, the image group is a group of consecutive video frames, the bitrate allocation weight is the ratio of the visual coding difficulty coefficient of the current frame to the sum of the visual coding difficulty coefficients of all frames in the image group, and the sum of the bitrate allocation weights of all frames in the image group is 1.

[0020] Multiply the bitrate allocation weight corresponding to each frame in any image group by the total bitrate budget corresponding to any image group to obtain the bitrate allocation component corresponding to each frame in any image group.

[0021] Using a preset fusion factor, the bitrate allocation component corresponding to each frame in any image group is linearly weighted and fused with the corresponding initial bitrate budget to obtain the target bitrate corresponding to each frame in any image group.

[0022] Optionally, dividing the frame into a first target frame that meets a preset low visual coding difficulty standard and a second target frame that meets a preset high visual coding difficulty standard according to the visual coding difficulty coefficient includes:

[0023] Determine whether the visual encoding difficulty coefficient of the current frame is lower than the preset difficulty coefficient threshold. If the visual encoding difficulty coefficient of the current frame is lower than the preset difficulty coefficient threshold, then the current frame is determined as the first target frame that meets the preset low visual encoding difficulty standard; otherwise, the current frame is determined as the second target frame that meets the preset high visual encoding difficulty standard.

[0024] Optionally, the step of adaptively adjusting the quantization parameters of each coding tree unit within the first target frame to determine the new quantization parameters corresponding to each coding tree unit includes:

[0025] Determine the average brightness of each coding tree unit in the first target frame, and identify coding tree units with an average brightness higher than a preset brightness threshold as specular candidate coding units;

[0026] Feature detection is performed on the candidate high-brightness coding unit. If the detection result meets the preset conditions, the candidate high-brightness coding unit is determined to be a key high-brightness unit.

[0027] A negative quantization parameter offset is set for the key highlight unit, and a positive quantization parameter offset is set for the other units excluding the key highlight unit; wherein, setting the negative quantization parameter offset is subtracting a preset value from the original quantization parameter, and setting the positive quantization parameter offset is adding a preset value to the original quantization parameter.

[0028] Optionally, the step of performing feature detection on the candidate specular coding unit, and determining the candidate specular coding unit as a key specular unit if the detection result meets a preset condition, includes:

[0029] The brightness gradient intensity and color saturation corresponding to the highlight candidate coding unit are detected; the brightness gradient intensity is used to characterize the drastic degree of brightness change within the highlight candidate coding unit, and the color saturation is used to characterize the vividness of the color within the highlight candidate coding unit.

[0030] If the brightness gradient intensity is greater than a preset brightness gradient intensity threshold and the color saturation is greater than a preset color saturation threshold, then the highlight candidate coding unit is determined to be a key highlight unit.

[0031] Secondly, this application provides a rate control and encoding device based on dynamic metadata, comprising:

[0032] The data acquisition module is used to acquire dynamic metadata corresponding to the video stream; the dynamic metadata includes the maximum brightness, minimum brightness and average brightness corresponding to each frame in the video stream;

[0033] The coefficient determination module is used to input the dynamic metadata into a pre-built visual complexity prediction model, so as to use the visual complexity prediction model to determine the dynamic range factor and the brightness deviation factor based on the dynamic metadata, and to perform weighted fusion of the dynamic range factor and the brightness deviation factor to obtain the visual coding difficulty coefficient corresponding to each frame; the dynamic range factor is used to quantify the difference between the maximum brightness and the minimum brightness within the frame, and the brightness deviation factor is used to reflect the degree to which the average brightness of the frame deviates from the preset most comfortable brightness range;

[0034] The bitrate determination module is used to determine the bitrate allocation weight corresponding to each frame based on the visual coding difficulty coefficient, and to fuse the bitrate allocation weight with the corresponding initial bitrate budget to obtain the target bitrate corresponding to each frame; the initial bitrate budget is the bitrate budget allocated to each frame through a preset bitrate control model;

[0035] The frame division module is used to divide the frame into a first target frame that meets a preset low visual coding difficulty standard and a second target frame that meets a preset high visual coding difficulty standard according to the visual coding difficulty coefficient.

[0036] The encoding module is used to adaptively adjust the quantization parameters of each coding tree unit in the first target frame to determine the new quantization parameters corresponding to each coding tree unit, and to encode the first target frame and the second target frame based on the target bitrate and quantization parameters corresponding to each frame; wherein, each frame contains at least one coding tree unit.

[0037] Thirdly, this application provides an electronic device, comprising:

[0038] Memory, used to store computer programs;

[0039] A processor is used to execute the computer program to implement the aforementioned rate control and encoding method based on dynamic metadata.

[0040] Fourthly, this application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned rate control and encoding method based on dynamic metadata.

[0041] In this application, dynamic metadata corresponding to a video stream is obtained; the dynamic metadata includes the maximum brightness, minimum brightness, and average brightness of each frame in the video stream; the dynamic metadata is input into a pre-built visual complexity prediction model to determine a dynamic range factor and a brightness deviation factor based on the dynamic metadata, and the dynamic range factor and the brightness deviation factor are weighted and fused to obtain the visual coding difficulty coefficient for each frame; the dynamic range factor is used to quantify the difference between the maximum and minimum brightness within a frame, and the brightness deviation factor is used to reflect the degree to which the average brightness of the frame deviates from a preset most comfortable brightness range; the visual coding difficulty coefficient is used to determine the visual coding difficulty coefficient for each frame. The corresponding bitrate allocation weights are fused with the corresponding initial bitrate budget to obtain the target bitrate for each frame; the initial bitrate budget is the bitrate budget allocated to each frame through a preset bitrate control model; the frames are divided into a first target frame that meets a preset low visual coding difficulty standard and a second target frame that meets a preset high visual coding difficulty standard according to the visual coding difficulty coefficient; the quantization parameters of each coding tree unit in the first target frame are adaptively adjusted to determine the new quantization parameters corresponding to each coding tree unit, and the first target frame and the second target frame are encoded based on the target bitrate and quantization parameters corresponding to each frame; wherein, each frame contains at least one coding tree unit. As can be seen from the above, this application obtains dynamic metadata containing the maximum brightness, minimum brightness and average brightness of each frame, and inputs it into the visual complexity prediction model, so that the encoding system can know the brightness dynamic range and overall brightness level of subsequent video frames in advance before actually encoding the current frame. This prediction based on dynamic metadata breaks the limitation of traditional schemes that can only rely on pixel statistics of encoded frames for lag estimation. It can make forward-looking judgments on upcoming scene changes and brightness variations, thus enabling more reasonable bitrate resource planning and allocation for subsequent frames. On the other hand, the dynamic range factor directly quantifies the contrast of brightness within a frame, and the brightness deviation factor scientifically assesses the deviation between the average brightness of the image and the brightness range most comfortable for the human eye. The visual coding difficulty coefficient obtained by weighted fusion of the two transforms the difference in human eye sensitivity to high dynamic range scenes, excessively dark or excessively bright scenes into a quantifiable coding guidance signal. Bitrate allocation weight calculation and target bitrate determination are based on the visual coding difficulty coefficient, which further guides the adjustment of quantization parameters at the coding tree unit level. This ensures that more bitrate resources are allocated to visually more sensitive and more difficult-to-encode areas, thereby effectively suppressing noise in dark areas and loss of detail in bright areas, achieving accurate bitrate allocation that conforms to the characteristics of human eye perception. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0043] Figure 1 This is a flowchart of a rate control and encoding method based on dynamic metadata disclosed in this application;

[0044] Figure 2 This is a schematic diagram of a system structure disclosed in this application;

[0045] Figure 3 This is a schematic diagram of a rate control and encoding device based on dynamic metadata disclosed in this application;

[0046] Figure 4 This is a schematic diagram of the structure of an electronic device disclosed in this application. Detailed Implementation

[0047] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] Currently, HDR Vivid encoding does not fully utilize dynamic metadata, resulting in several shortcomings in existing solutions: First, it cannot proactively plan resources for upcoming scene transitions, making it difficult to plan and allocate bitrate resources in advance. Second, the bitrate allocation for dark scenes, bright scenes, and scenes with a large dynamic range is not precise enough, potentially causing problems such as noise in dark areas and loss of detail in bright areas. To address these issues, this application provides a bitrate control and encoding method based on dynamic metadata, achieving efficient HDR encoding that combines proactive planning with perceptual optimization. This effectively avoids problems such as noise in dark areas and loss of detail in bright areas, significantly improving the subjective image quality of the video.

[0049] See Figure 1 As shown in the figure, this application discloses a bitrate control and encoding method based on dynamic metadata, including:

[0050] Step S11: Obtain the dynamic metadata corresponding to the video stream; the dynamic metadata includes the maximum brightness, minimum brightness and average brightness of each frame in the video stream.

[0051] In this embodiment, before the formal encoding begins, the HDR Vivid dynamic metadata information accompanying the input video stream can be read, mainly including the maximum brightness Lmax(i), minimum brightness Lmin(i), and average brightness Lavg(i) of the content displayed in the current frame; where i represents the i-th frame.

[0052] Step S12: Input the dynamic metadata into the pre-built visual complexity prediction model, so as to use the visual complexity prediction model to determine the dynamic range factor and the brightness deviation factor based on the dynamic metadata, and perform weighted fusion on the dynamic range factor and the brightness deviation factor to obtain the visual coding difficulty coefficient corresponding to each frame; the dynamic range factor is used to quantify the difference between the maximum brightness and the minimum brightness within the frame, and the brightness deviation factor is used to reflect the degree to which the average brightness of the frame deviates from the preset most comfortable brightness range.

[0053] This embodiment constructs a visual complexity prediction model to calculate the visual coding difficulty coefficient of each frame. This model comprehensively considers factors such as the average brightness and brightness range of the frame, assigning higher visual difficulty coefficients to dark-field frames and frames with a large dynamic range containing both extremely bright and extremely dark areas.

[0054] like Figure 2 As shown, dynamic metadata is input into a visual complexity prediction model to determine the visual coding difficulty coefficient for each frame. Specifically, this may include: first, obtaining a dynamic range factor based on the difference between the maximum and minimum brightness in the dynamic metadata; and then determining a brightness deviation factor based on a preset most comfortable brightness range and the average brightness in the dynamic metadata. For example:

[0055] The formula for calculating the dynamic range factor is:

[0056] ;

[0057] The formula for calculating the brightness deviation factor is:

[0058] ;

[0059] This is the upper limit of the most comfortable brightness range. This is the lower limit of the most comfortable brightness range.

[0060] In one specific implementation, the process of determining the preset most comfortable brightness range may include setting a lower limit and an upper limit for the preset most comfortable brightness range based on the characteristics of human vision. For example, brightness below 1 nit is considered extremely dark, requiring dark adaptation of the human eye and making it extremely sensitive to noise; it is a visually challenging area. 100 nits is the reference white level for traditional SDR (Standard Dynamic Range) and also an upper limit for comfortable viewing in bright environments over extended periods. Above this value, content begins to enter the highlight area, and the human eye's ability to distinguish details decreases. Therefore, the lower limit... The value range can be from 0.5 nits to 5 nits, corresponding to the nonlinearity after photoelectric conversion, with an upper limit of... The value ranges from 50 nits to 200 nits, corresponding to the nonlinear value after photoelectric conversion.

[0061] In another specific implementation, it may include: determining a statistical value of the average brightness of a preset number of consecutive frames in the video stream; multiplying the statistical value by a first preset coefficient to obtain the lower limit of a preset most comfortable brightness range; and multiplying the statistical value by a second preset coefficient to obtain the upper limit of the preset most comfortable brightness range; wherein the statistical value is the median or average, the first preset coefficient is less than 1, and the second preset coefficient is greater than 1. For example:

[0062] ;

[0063] ;

[0064] in, M is a statistical value.

[0065] Then, the dynamic range factor is mapped using a first preset monotonically increasing function to obtain a first mapped value. The brightness deviation factor is mapped using a second preset monotonically increasing function to obtain a second mapped value. The first and second mapped values ​​are then weighted and fused to obtain a target weighted sum. Next, the target weighted sum is multiplied by the function value output by a preset masking function to obtain the visual coding difficulty coefficient for each frame. The preset masking function is a function related to the average brightness of the frame and conforms to the characteristics of human vision. It outputs a high function value (close to 1) in preset low-brightness regions, indicating high visual sensitivity to maintain or amplify coding difficulty, and outputs a low function value (close to 0) in preset high-brightness regions, indicating low visual sensitivity to suppress coding difficulty.

[0066] For example, the formula for calculating the visual coding difficulty coefficient is:

[0067] ;

[0068] f() and g() are monotonically increasing functions used to non-linearly map the original values ​​of DR(i) and B(i) to the interval [0,1]. These are the weighting coefficients; M(Lavg(i)) is the brightness-related masking function; This indicates rounding down to the nearest integer.

[0069] The formula for calculating the visual coding difficulty coefficient ensures that high V values ​​are obtained in dark scenes and high dynamic range scenes.

[0070] Step S13: Determine the bitrate allocation weight for each frame based on the visual coding difficulty coefficient, and fuse the bitrate allocation weight with the corresponding initial bitrate budget to obtain the target bitrate for each frame; the initial bitrate budget is the bitrate budget allocated to each frame through a preset bitrate control model.

[0071] In this embodiment, as Figure 2 As shown, the initial bitrate budget allocated to each frame through a preset bitrate control model can be combined with the visual coding difficulty coefficient predicted based on dynamic metadata. This increases the bitrate budget weight for high-difficulty frames and decreases the bitrate budget weight for low-difficulty frames, thereby generating a forward-looking, perceptually optimized GOP (Group of Pictures) bitrate allocation blueprint and achieving frame-level bitrate adjustment.

[0072] Specifically, this can include: for any image group, determining the bitrate allocation weights for each frame within that image group based on the visual coding difficulty coefficient; wherein, an image group is a set of consecutive video frames, and the bitrate allocation weight is the ratio of the visual coding difficulty coefficient of the current frame to the sum of the visual coding difficulty coefficients of all frames within that image group, and the sum of the bitrate allocation weights for all frames within that image group is 1. Then, the bitrate allocation weights for each frame within that image group are multiplied by the total bitrate budget for that image group to obtain the bitrate allocation components for each frame within that image group. Finally, using a preset fusion factor, the bitrate allocation components for each frame within that image group are linearly weighted and fused with their respective initial bitrate budgets to obtain the target bitrate for each frame within that image group.

[0073] For example, pre-analyze a GOP to obtain V(i) for each frame, and calculate the bitrate allocation weight for each frame:

[0074] ;

[0075] Where V(i) is the visual coding difficulty coefficient, and Σ(V(j)) represents the sum of the visual coding difficulty coefficients of all frames in the current GOP.

[0076] If the preset rate control model allocates a rate budget of Rc(i) to each frame, then the final frame-level target rate Rfinal(i) can be calculated using the following formula:

[0077] ;

[0078] Where Tgop is the current GOP's total bitrate budget. As a fusion factor, .

[0079] Step S14: Divide the frame into a first target frame that meets the preset low visual coding difficulty standard and a second target frame that meets the preset high visual coding difficulty standard according to the visual coding difficulty coefficient.

[0080] To meet the image quality balance requirements of high-difficulty frames, achieve refined and efficient utilization of bitrate resources, and optimize overall subjective image quality, this embodiment can divide frames into different frame types based on visual coding difficulty coefficients. For example, frames can be divided into a first target frame that meets a preset low visual coding difficulty standard and a second target frame that meets a preset high visual coding difficulty standard. Then, as follows... Figure 2 As shown, the first target frame is optimized at the block level before encoding, while the second target frame is encoded directly.

[0081] Specifically, it can include: determining whether the visual encoding difficulty coefficient of the current frame is lower than a preset difficulty coefficient threshold; if the visual encoding difficulty coefficient of the current frame is lower than the preset difficulty coefficient threshold, then the current frame is determined as the first target frame that meets the preset low visual encoding difficulty standard; otherwise, the current frame is determined as the second target frame that meets the preset high visual encoding difficulty standard.

[0082] Step S15: Adaptively adjust the quantization parameters of each coding tree unit in the first target frame to determine the new quantization parameters corresponding to each coding tree unit, and encode the first target frame and the second target frame based on the target bitrate and quantization parameters corresponding to each frame; wherein, each frame contains at least one coding tree unit.

[0083] For the first target frame identified as meeting the preset low visual coding difficulty standard, the visual tolerance is high, and a relatively strict frame-level bitrate will be assigned to it. Therefore, block-level optimization needs to be initiated. This involves guiding the encoder to perform perceptually optimized bitrate micro-allocation within the frame, prioritizing contours and colors while discarding secondary texture details, thereby maximizing the overall subjective quality. Specifically, this can include: first, determining the average brightness of each coding tree unit within the first target frame, and identifying coding tree units with an average brightness higher than a preset brightness threshold as specular candidate coding units. Then, performing feature detection on the specular candidate coding units. If the detection results meet preset conditions, the specular candidate coding units are determined as key specular units. Finally, setting a negative quantization parameter offset for key specular units and a positive quantization parameter offset for other units excluding key specular units; where setting a negative quantization parameter offset involves subtracting a preset value from the original quantization parameter, and setting a positive quantization parameter offset involves adding a preset value to the original quantization parameter.

[0084] The process involves feature detection of candidate highlight coding units. If the detection results meet preset conditions, the candidate highlight coding unit is identified as a key highlight unit. This can include: firstly, detecting the luminance gradient intensity and color saturation corresponding to the candidate highlight coding unit; the luminance gradient intensity is used to characterize the drastic degree of luminance change within the candidate highlight coding unit, and the color saturation is used to characterize the vividness of the color within the candidate highlight coding unit. If the luminance gradient intensity is greater than a preset luminance gradient intensity threshold, and the color saturation is greater than a preset color saturation threshold, then the candidate highlight coding unit is identified as a key highlight unit.

[0085] Finally, the first target frame and the second target frame are encoded based on the target bit rate and quantization parameters corresponding to each frame.

[0086] As can be seen from the above, in this embodiment, on the one hand, the HDR Vivid dynamic metadata is transformed from display-end guidance information into pre-sensory signals at the encoding end, and a visual complexity prediction model based on metadata is constructed. This enables the encoder to predict the brightness structure and scene change trends of the video content in advance, thereby achieving forward-looking and intelligent allocation of bitrate resources. On the other hand, by quantifying the visual encoding difficulty coefficient, the sensitivity differences of the human eye to dark scenes and high dynamic range scenes are accurately reflected. Based on this difficulty coefficient, the bitrate is adjusted at the frame level for perceptual optimization. At the same time, combined with block-level adaptive quantization parameter fine-tuning, collaborative optimization is achieved at both the macro-resource planning and micro-visual quality control levels. Ultimately, the subjective visual quality of HDR video is significantly improved at the same bitrate, and dark noise and highlight detail loss are effectively suppressed.

[0087] See Figure 3 As shown in the embodiments, this application also discloses a rate control and encoding device based on dynamic metadata, including:

[0088] Data acquisition module 11 is used to acquire dynamic metadata corresponding to the video stream; the dynamic metadata includes the maximum brightness, minimum brightness and average brightness corresponding to each frame in the video stream;

[0089] The coefficient determination module 12 is used to input the dynamic metadata into a pre-built visual complexity prediction model, so as to use the visual complexity prediction model to determine the dynamic range factor and the brightness deviation factor based on the dynamic metadata, and to perform weighted fusion of the dynamic range factor and the brightness deviation factor to obtain the visual coding difficulty coefficient corresponding to each frame; the dynamic range factor is used to quantify the difference between the maximum brightness and the minimum brightness within the frame, and the brightness deviation factor is used to reflect the degree to which the average brightness of the frame deviates from the preset most comfortable brightness range;

[0090] The bitrate determination module 13 is used to determine the bitrate allocation weight corresponding to each frame based on the visual coding difficulty coefficient, and to fuse the bitrate allocation weight and the corresponding initial bitrate budget to obtain the target bitrate corresponding to each frame; the initial bitrate budget is the bitrate budget allocated to each frame through a preset bitrate control model.

[0091] Frame segmentation module 14 is used to segment a frame into a first target frame that meets a preset low visual coding difficulty standard and a second target frame that meets a preset high visual coding difficulty standard according to the visual coding difficulty coefficient.

[0092] The encoding module 15 is used to adaptively adjust the quantization parameters of each coding tree unit in the first target frame to determine the new quantization parameters corresponding to each coding tree unit, and to encode the first target frame and the second target frame based on the target bitrate and quantization parameters corresponding to each frame; wherein, each frame contains at least one coding tree unit.

[0093] In some specific embodiments, the coefficient determination module 12 includes:

[0094] The first interval setting unit is used to set the lower limit value and the upper limit value of the preset most comfortable brightness interval based on the characteristics of human eye vision.

[0095] The second interval setting unit is used to determine the statistical value of the average brightness of a preset number of consecutive frames in the video stream, multiply the statistical value by a first preset coefficient to obtain the lower limit of the preset most comfortable brightness interval, and multiply the statistical value by a second preset coefficient to obtain the upper limit of the preset most comfortable brightness interval; the statistical value is the median or average value, the first preset coefficient is less than 1, and the second preset coefficient is greater than 1.

[0096] In some specific embodiments, the coefficient determination module 12 includes:

[0097] The first data determination unit is used to obtain a dynamic range factor based on the difference between the maximum and minimum brightness in the dynamic metadata, and to determine a brightness deviation factor based on a preset most comfortable brightness range and the average brightness in the dynamic metadata.

[0098] The data weighting unit is used to map the dynamic range factor using a first preset monotonically increasing function to obtain a first mapping value, and to map the brightness deviation factor using a second preset monotonically increasing function to obtain a second mapping value. The first mapping value and the second mapping value are then weighted and fused to obtain a target weighted sum.

[0099] The coefficient determination unit is used to multiply the target weighted sum with the function value output by the preset masking function to obtain the visual coding difficulty coefficient corresponding to each frame. The preset masking function is a function that is related to the average brightness of the frame and conforms to the visual characteristics of the human eye. It outputs a high function value in the preset low brightness area to maintain or amplify the coding difficulty, and outputs a low function value in the preset high brightness area to suppress the coding difficulty.

[0100] In some specific embodiments, the bit rate determination module 13 includes:

[0101] The weight determination unit is used to determine the bitrate allocation weight corresponding to each frame in any image group based on the visual coding difficulty coefficient; wherein, the image group is a group of consecutive video frames, the bitrate allocation weight is the ratio of the visual coding difficulty coefficient of the current frame to the sum of the visual coding difficulty coefficients of all frames in any image group, and the sum of the bitrate allocation weights of all frames in any image group is 1.

[0102] The second data determination unit is used to multiply the bitrate allocation weight corresponding to each frame in any image group by the total bitrate budget corresponding to any image group to obtain the bitrate allocation component corresponding to each frame in any image group.

[0103] The bitrate determination unit is used to linearly weight and fuse the bitrate allocation components corresponding to each frame in any image group with the corresponding initial bitrate budget using a preset fusion factor, so as to obtain the target bitrate corresponding to each frame in any image group.

[0104] In some specific embodiments, the frame division module 14 includes:

[0105] The frame division unit is used to determine whether the visual coding difficulty coefficient of the current frame is lower than the preset difficulty coefficient threshold. If the visual coding difficulty coefficient of the current frame is lower than the preset difficulty coefficient threshold, the current frame is determined to be the first target frame that meets the preset low visual coding difficulty standard; otherwise, the current frame is determined to be the second target frame that meets the preset high visual coding difficulty standard.

[0106] In some specific embodiments, the encoding module 15 includes:

[0107] The filtering unit is used to determine the average brightness of each coding tree unit in the first target frame and to identify coding tree units with an average brightness higher than a preset brightness threshold as specular candidate coding units.

[0108] The determination submodule is used to perform feature detection on the candidate high-light coding unit. If the detection result meets the preset conditions, the candidate high-light coding unit is determined to be a key high-light unit.

[0109] The quantization parameter setting unit is used to set a negative quantization parameter offset for the key highlight unit and a positive quantization parameter offset for other units excluding the key highlight unit; wherein, setting a negative quantization parameter offset is to subtract a preset value from the original quantization parameter, and setting a positive quantization parameter offset is to add a preset value to the original quantization parameter.

[0110] In some specific embodiments, the determination submodule includes:

[0111] The detection unit is used to detect the brightness gradient intensity and color saturation corresponding to the highlight candidate coding unit; the brightness gradient intensity is used to characterize the drastic degree of brightness change within the highlight candidate coding unit, and the color saturation is used to characterize the vividness of the color within the highlight candidate coding unit.

[0112] The determination unit is used to determine the highlight candidate coding unit as a key highlight unit if the brightness gradient intensity is greater than a preset brightness gradient intensity threshold and the color saturation is greater than a preset color saturation threshold.

[0113] Furthermore, embodiments of this application also disclose an electronic device, Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0114] Figure 4 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the dynamic metadata-based rate control and encoding method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0115] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0116] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0117] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs capable of performing the dynamic metadata-based rate control and encoding methods executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0118] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed rate control and encoding method based on dynamic metadata. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0119] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0120] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0121] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0122] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0123] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A rate control and encoding method based on dynamic metadata, characterized in that, include: Obtain the dynamic metadata corresponding to the video stream; The dynamic metadata includes the maximum brightness, minimum brightness, and average brightness corresponding to each frame in the video stream; The dynamic metadata is input into a pre-built visual complexity prediction model to determine the dynamic range factor and brightness deviation factor based on the dynamic metadata. The dynamic range factor and brightness deviation factor are then weighted and fused to obtain the visual coding difficulty coefficient for each frame. The dynamic range factor is used to quantify the difference between the maximum and minimum brightness within a frame, and the brightness deviation factor is used to reflect the degree to which the average brightness of the frame deviates from the preset most comfortable brightness range. Based on the visual coding difficulty coefficient, the bitrate allocation weight corresponding to each frame is determined, and the bitrate allocation weight and the corresponding initial bitrate budget are fused to obtain the target bitrate corresponding to each frame; the initial bitrate budget is the bitrate budget allocated to each frame through a preset bitrate control model. Based on the visual coding difficulty coefficient, the frame is divided into a first target frame that meets the preset low visual coding difficulty standard and a second target frame that meets the preset high visual coding difficulty standard. Adaptive quantization parameter adjustment is performed on each coding tree unit in the first target frame to determine the new quantization parameters corresponding to each coding tree unit. The first target frame and the second target frame are encoded based on the target bitrate and quantization parameters corresponding to each frame. Each frame contains at least one coding tree unit. The dynamic range factor and the brightness deviation factor are weighted and fused to obtain the visual coding difficulty coefficient for each frame, including: The dynamic range factor is mapped using a first preset monotonically increasing function to obtain a first mapped value. The brightness deviation factor is mapped using a second preset monotonically increasing function to obtain a second mapped value. The first mapped value and the second mapped value are then weighted and fused to obtain a target weighted sum. The target weighted sum is multiplied by the function value output by the preset masking function to obtain the visual coding difficulty coefficient corresponding to each frame; the preset masking function is a function that is related to the average brightness of the frame and conforms to the visual characteristics of the human eye. It outputs a high function value in the preset low brightness area to maintain or amplify the coding difficulty, and outputs a low function value in the preset high brightness area to suppress the coding difficulty. The formula for calculating the visual coding difficulty coefficient is as follows: ； Where V(i) is the visual coding difficulty coefficient, DR(i) is the dynamic range factor, B(i) is the brightness deviation factor, and f() and g() are monotonically increasing functions used to non-linearly map the original values ​​of DR(i) and B(i) to the interval [0,1]. These are the weighting coefficients; M(Lavg(i)) is the brightness-related masking function; Indicates rounding down; The adaptive quantization parameter adjustment for each coding tree unit within the first target frame includes: Determine the average brightness of each coding tree unit in the first target frame, and identify coding tree units with an average brightness higher than a preset brightness threshold as specular candidate coding units; Feature detection is performed on the candidate high-brightness coding unit. If the detection result meets the preset conditions, the candidate high-brightness coding unit is determined to be a key high-brightness unit. A negative quantization parameter offset is set for the key highlight unit, and a positive quantization parameter offset is set for the other units excluding the key highlight unit; wherein, setting the negative quantization parameter offset is subtracting a preset value from the original quantization parameter, and setting the positive quantization parameter offset is adding a preset value to the original quantization parameter.

2. The rate control and encoding method based on dynamic metadata according to claim 1, characterized in that, The process of determining the preset most comfortable brightness range includes: The lower limit and upper limit of the preset most comfortable brightness range are set based on the characteristics of human vision. Alternatively, determine the statistical value of the average brightness of a preset number of consecutive frames in the video stream, multiply the statistical value by a first preset coefficient to obtain the lower limit of the preset most comfortable brightness range, and multiply the statistical value by a second preset coefficient to obtain the upper limit of the preset most comfortable brightness range; the statistical value is the median or average value, the first preset coefficient is less than 1, and the second preset coefficient is greater than 1.

3. The rate control and encoding method based on dynamic metadata according to claim 1, characterized in that, The determination of the dynamic range factor and brightness deviation factor based on the dynamic metadata includes: The dynamic range factor is obtained based on the difference between the maximum and minimum brightness in the dynamic metadata, and the brightness deviation factor is determined based on the preset most comfortable brightness range and the average brightness in the dynamic metadata.

4. The rate control and encoding method based on dynamic metadata according to claim 1, characterized in that, The step of determining the bitrate allocation weight for each frame based on the visual coding difficulty coefficient, and fusing the bitrate allocation weight with the corresponding initial bitrate budget to obtain the target bitrate for each frame includes: For any image group, the bitrate allocation weight corresponding to each frame in the image group is determined based on the visual coding difficulty coefficient; wherein, the image group is a group of consecutive video frames, the bitrate allocation weight is the ratio of the visual coding difficulty coefficient of the current frame to the sum of the visual coding difficulty coefficients of all frames in the image group, and the sum of the bitrate allocation weights of all frames in the image group is 1. Multiply the bitrate allocation weight corresponding to each frame in any image group by the total bitrate budget corresponding to any image group to obtain the bitrate allocation component corresponding to each frame in any image group. Using a preset fusion factor, the bitrate allocation component corresponding to each frame in any image group is linearly weighted and fused with the corresponding initial bitrate budget to obtain the target bitrate corresponding to each frame in any image group.

5. The rate control and encoding method based on dynamic metadata according to claim 1, characterized in that, The step of dividing a frame into a first target frame that meets a preset low visual coding difficulty standard and a second target frame that meets a preset high visual coding difficulty standard based on the visual coding difficulty coefficient includes: Determine whether the visual encoding difficulty coefficient of the current frame is lower than the preset difficulty coefficient threshold. If the visual encoding difficulty coefficient of the current frame is lower than the preset difficulty coefficient threshold, then the current frame is determined as the first target frame that meets the preset low visual encoding difficulty standard; otherwise, the current frame is determined as the second target frame that meets the preset high visual encoding difficulty standard.

6. The rate control and encoding method based on dynamic metadata according to claim 1, characterized in that, The step of performing feature detection on the candidate specular coding unit, and determining the candidate specular coding unit as a key specular unit if the detection result meets a preset condition, includes: The brightness gradient intensity and color saturation corresponding to the highlight candidate coding unit are detected; the brightness gradient intensity is used to characterize the drastic degree of brightness change within the highlight candidate coding unit, and the color saturation is used to characterize the vividness of the color within the highlight candidate coding unit. If the brightness gradient intensity is greater than a preset brightness gradient intensity threshold and the color saturation is greater than a preset color saturation threshold, then the highlight candidate coding unit is determined to be a key highlight unit.

7. A rate control and encoding device based on dynamic metadata, characterized in that, include: The data acquisition module is used to acquire dynamic metadata corresponding to the video stream; The dynamic metadata includes the maximum brightness, minimum brightness, and average brightness corresponding to each frame in the video stream; The coefficient determination module is used to input the dynamic metadata into a pre-built visual complexity prediction model, so as to use the visual complexity prediction model to determine the dynamic range factor and the brightness deviation factor based on the dynamic metadata, and to perform weighted fusion of the dynamic range factor and the brightness deviation factor to obtain the visual coding difficulty coefficient corresponding to each frame. The dynamic range factor is used to quantify the difference between the maximum and minimum brightness within a frame, and the brightness deviation factor is used to reflect the degree to which the average brightness of the frame deviates from the preset most comfortable brightness range. The bitrate determination module is used to determine the bitrate allocation weight corresponding to each frame based on the visual coding difficulty coefficient, and to fuse the bitrate allocation weight with the corresponding initial bitrate budget to obtain the target bitrate corresponding to each frame; the initial bitrate budget is the bitrate budget allocated to each frame through a preset bitrate control model; The frame division module is used to divide the frame into a first target frame that meets a preset low visual coding difficulty standard and a second target frame that meets a preset high visual coding difficulty standard according to the visual coding difficulty coefficient. The encoding module is used to adaptively adjust the quantization parameters of each coding tree unit in the first target frame to determine the new quantization parameters corresponding to each coding tree unit, and to encode the first target frame and the second target frame based on the target bitrate and quantization parameters corresponding to each frame; wherein, each frame contains at least one coding tree unit; The coefficient determination module includes: The data weighting unit is used to map the dynamic range factor using a first preset monotonically increasing function to obtain a first mapping value, and to map the brightness deviation factor using a second preset monotonically increasing function to obtain a second mapping value. The first mapping value and the second mapping value are then weighted and fused to obtain a target weighted sum. The coefficient determination unit is used to multiply the target weighted sum with the function value output by the preset masking function to obtain the visual coding difficulty coefficient corresponding to each frame; the preset masking function is a function that is related to the average brightness of the frame and conforms to the visual characteristics of the human eye. It outputs a high function value in the preset low brightness area to maintain or amplify the coding difficulty, and outputs a low function value in the preset high brightness area to suppress the coding difficulty. The formula for calculating the visual coding difficulty coefficient is as follows: ； Where V(i) is the visual coding difficulty coefficient, DR(i) is the dynamic range factor, B(i) is the brightness deviation factor, and f() and g() are monotonically increasing functions used to non-linearly map the original values ​​of DR(i) and B(i) to the interval [0,1]. These are the weighting coefficients; M(Lavg(i)) is the brightness-related masking function; Indicates rounding down; The encoding module includes: The filtering unit is used to determine the average brightness of each coding tree unit in the first target frame and to identify coding tree units with an average brightness higher than a preset brightness threshold as specular candidate coding units. The determination submodule is used to perform feature detection on the candidate high-light coding unit. If the detection result meets the preset conditions, the candidate high-light coding unit is determined to be a key high-light unit. The quantization parameter setting unit is used to set a negative quantization parameter offset for the key highlight unit and a positive quantization parameter offset for other units excluding the key highlight unit; wherein, setting a negative quantization parameter offset is to subtract a preset value from the original quantization parameter, and setting a positive quantization parameter offset is to add a preset value to the original quantization parameter.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the bitrate control and encoding method based on dynamic metadata as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, Used to store computer programs, which, when executed by a processor, implement the bitrate control and encoding method based on dynamic metadata as described in any one of claims 1 to 6.

Images