A video coding rate control method, device and storage medium

Abstract

The present application provides a video coding rate control method, device and storage medium, the method comprising: determining the video quality characteristics, time complexity characteristics and space complexity characteristics of the target video; inputting the video quality characteristics, time complexity characteristics and space complexity characteristics into a pre-trained user perception training model, and outputting the user distortion perception; based on the functional relationship between the quantization parameter and the user distortion perception and the rate-distortion function, determining the first objective function between the user distortion perception and the encoding rate; based on the second objective function and the first objective function, controlling the rate of each video frame of the target video. The frame-level rate control is realized by combining the buffer status and the user's subjective perception quality, which can effectively reduce the overflow and underflow of the encoder buffer during the encoding process, and can improve the subjective perception quality index of the video in consideration of the frame loss of the encoder buffer.

Description

Technical Field

[0001] The present application relates to the technical field of video coding, and in particular to a method, device and storage medium for controlling a bit rate of video coding. Background Art

[0002] In recent years, video surveillance systems have been widely used in industries such as industry, transportation, shopping malls, banks, public security, and smart residential areas. They are one of the important means of modern management and monitoring, greatly improving management efficiency and automation level. Video coding, as an effective video compression solution, has received widespread attention from researchers, and bit rate control is one of the very important technologies in audio and video coding. Its main task is to effectively control the audio and video encoder to select the quantization step size so that the size of its output bit stream meets the actual bandwidth limitation of the transmission channel, and to obtain the best decoded image or audio sampling as much as possible. Bit rate control has always been a hot issue in the field of audio and video coding technology research. Any audio and video standard will be limited in application without bit rate control.

[0003] At present, most observers of encoded videos are humans. The perception of the image and video quality by the human eye perception system cannot be perfectly reflected by objective indicators. At present, there is still room for improvement in the optimization of objective distortion. In addition, in scenes with more drastic video changes, the human eye is not very sensitive to distortion. At the same time, the encoded bit stream will fluctuate greatly, which will cause overflow and underflow of the encoding buffer. Therefore, how to control the bit rate has become a technical issue that cannot be underestimated. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a video encoding bit rate control method, device and storage medium, which realizes frame-level bit rate control by combining buffer status and user subjective perceived quality through a second objective function corresponding to the state stability target of the encoder buffer status and a first objective function between user distortion perception and encoding bit rate. It can effectively reduce the overflow and underflow of the encoder buffer during the encoding process, and can improve the subjective perceived quality index of the video while considering the frame loss of the encoder buffer.

[0005] The embodiment of the present application provides a video coding rate control method, the rate control method comprising:

[0006] Acquire a target video to be processed, and determine the video quality characteristics, time complexity characteristics, and space complexity characteristics of the target video;

[0007] Inputting the video quality feature, the temporal complexity feature and the spatial complexity feature into a pre-trained user perception training model, and outputting the user distortion perception;

[0008] Determining a first objective function between the user distortion perception and the encoding bit rate based on a functional relationship between a quantization parameter and the user distortion perception and a rate-distortion function;

[0009] Based on a second objective function corresponding to a state stability target of an encoder buffer state and a first objective function between the user distortion perception and the encoding bit rate, the bit rate of each video frame of the target video is controlled.

[0010] In a possible implementation manner, the video quality feature is determined by the following steps:

[0011] Performing distortion prediction processing on the target video to determine a distorted video;

[0012] Dividing the distorted video into a plurality of sub-video segments;

[0013] For each of the sub-video segments, determining a video degradation feature vector based on a target video quality value of the sub-video segment;

[0014] The video quality feature is determined based on the video degradation feature vector of each of the sub-video segments.

[0015] In a possible implementation manner, performing distortion prediction processing on the target video to determine a distorted video includes:

[0016] Performing discrete cosine transform processing on the target video to determine multiple frequency domains of the target video;

[0017] Quantization processing, inverse quantization processing and inverse transformation processing are respectively performed on each of the frequency domains according to a plurality of different quantization step sizes, so as to determine the distorted video corresponding to the different quantization step sizes.

[0018] In a possible implementation, the functional relationship includes a normal distribution probability density cumulative relationship; and determining a first objective function between the user distortion perception and the encoding bit rate based on the functional relationship between the quantization parameter and the user distortion perception and the rate-distortion function includes:

[0019] Determining a first function based on a cumulative relationship between the user's distortion perception and a probability density of a normal distribution presented by the quantization parameter;

[0020] Performing an approximate transformation on the first function to determine a second function;

[0021] Determine a third function based on the first function and the second function;

[0022] Based on the third function and the rate-distortion function, a first objective function between the distortion perception and the encoding bit rate is determined.

[0023] In a possible implementation manner, the second objective function corresponding to the state stability target of the encoder buffer state is determined by the following steps:

[0024] Obtaining a preset buffer occupancy ratio, where the buffer occupancy ratio represents an occupancy rate of a buffer state after encoding each video frame;

[0025] Based on the buffer interval in which the currently encoded video frame is located, the preset buffer occupancy ratio, the target bit rate and the frame rate, a second objective function corresponding to the state stability target of the encoder buffer state is determined.

[0026] In a possible implementation manner, after controlling the bit rate of each video frame of the target video, the bit rate control method further includes:

[0027] For the current video frame, detecting whether the intra-frame prediction CTU ratio of the current video frame is greater than a preset value; wherein the intra-frame prediction CTU ratio of the current video frame is the ratio of the CTU data of the intra-frame prediction mode between the preset number of video frames encoded before the current video frame in the target video to the total number of CTUs between the video frames;

[0028] If yes, weight adjustment is performed on the first objective function of the preset number of video frames before the current video frame, and the video frame after the current video frame is encoded according to the inter-frame relationship between the first objective function after weight adjustment, the second objective function and the preset number of video frames before the current video frame;

[0029] If not, the video frames following the current video frame are encoded according to the first objective function and the second objective function.

[0030] The embodiment of the present application further provides a video coding bit rate control device, the bit rate control device comprising:

[0031] A determination module, used to obtain a target video to be processed, and determine the video quality characteristics, time complexity characteristics, and space complexity characteristics of the target video;

[0032] A distortion perception determination module, used for inputting the video quality feature, the temporal complexity feature and the spatial complexity feature into a pre-trained user perception training model, and outputting the user distortion perception;

[0033] A first objective function determination module, configured to determine a first objective function between the user distortion perception and the encoding bit rate based on a functional relationship between a quantization parameter and the user distortion perception and a rate-distortion function;

[0034] The bit rate adjustment module is used to control the bit rate of each video frame of the target video based on a second objective function corresponding to the state stability target of the encoder buffer state and a first objective function between the user distortion perception and the encoding bit rate.

[0035] In a possible implementation manner, the determination module determines the video quality feature through the following steps:

[0036] Performing distortion prediction processing on the target video to determine a distorted video;

[0037] Dividing the distorted video into a plurality of sub-video segments;

[0038] For each of the sub-video segments, determining a video degradation feature vector based on the video quality value of the sub-video segment;

[0039] The video quality feature is determined based on the video degradation feature vector of each of the sub-video segments.

[0040] An embodiment of the present application also provides an electronic device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the video encoding bit rate control method as described above are performed.

[0041] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the bit rate control method for video encoding as described above are executed.

[0042] The embodiment of the present application provides a video encoding rate control method, device and storage medium, the rate control method comprising: obtaining a target video to be processed, and determining the video quality characteristics, time complexity characteristics and space complexity characteristics of the target video; inputting the video quality characteristics, the time complexity characteristics and the space complexity characteristics into a pre-trained user perception training model, and outputting the user distortion perception; determining the first objective function between the user distortion perception and the encoding rate based on the functional relationship between the quantization parameter and the user distortion perception and the rate distortion function; controlling the rate of each video frame of the target video based on the second objective function corresponding to the state stability target of the encoder buffer state and the first objective function between the user distortion perception and the encoding rate. Through the second objective function corresponding to the state stability target of the encoder buffer state and the first objective function between the user distortion perception and the encoding rate, frame-level rate control is achieved by combining the buffer state and the user's subjective perception quality, which can effectively reduce the overflow and underflow of the encoder buffer during the encoding process, and can improve the subjective perception quality index of the video when considering the frame loss of the encoder buffer.

[0043] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 A flowchart of a video encoding rate control method provided in an embodiment of the present application;

[0046] Figure 2 One of the structural schematic diagrams of a video encoding bit rate control device provided in an embodiment of the present application;

[0047] Figure 3 A second structural diagram of a video coding bit rate control device provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art, under the guidance of the content of the present application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0050] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0051] In order to enable those skilled in the art to use the contents of this application, the following implementation is provided in combination with a specific application scenario of "controlling the bit rate of video encoding". For those skilled in the art, the general principles defined herein can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application.

[0052] The following methods, devices, electronic devices or computer-readable storage media of the embodiments of the present application can be applied to any scenario where the bit rate of video encoding needs to be controlled. The embodiments of the present application are not limited to specific application scenarios. Any scheme using a method, device and storage medium for controlling the bit rate of video encoding provided by the embodiments of the present application is within the scope of protection of the present application.

[0053] First, the application scenarios to which the present application is applicable are introduced. The present application can be applied to the field of video coding technology.

[0054] Research has found that at this stage, most observers of encoded videos are humans, and the perception of the image and video quality by the human eye perception system cannot be perfectly reflected by objective indicators. At present, there is still room for improvement in the optimization of objective distortion. In addition, in scenes with more drastic video changes, the human eye is not very sensitive to distortion. At the same time, the encoded bit stream will fluctuate greatly, which will cause overflow and underflow of the encoding buffer. Therefore, how to control the bit rate has become a technical issue that cannot be underestimated.

[0055] Based on this, an embodiment of the present application provides a method for bit rate control of video encoding. Through a second objective function corresponding to the state stability target of the encoder buffer state and a first objective function between the user distortion perception and the encoding bit rate, frame-level bit rate control is achieved by combining the buffer state and the user's subjective perceived quality. This can effectively reduce the overflow and underflow of the encoder buffer during the encoding process, and can improve the subjective perceived quality index of the video while considering frame loss in the encoder buffer.

[0056] See also Figure 1 , Figure 1 This is a flow chart of a video encoding rate control method provided in an embodiment of the present application. Figure 1 As shown in, the bit rate control method provided by the embodiment of the present application includes:

[0057] S101: Acquire a target video to be processed, and determine video quality features, temporal complexity features, and spatial complexity features of the target video.

[0058] In this step, the target video to be processed is obtained, and the video quality characteristics, time complexity characteristics, and space complexity characteristics of the target video are determined.

[0059] Here, the target video to be processed is a video that has not been encoded.

[0060] Among them, the video quality feature is a VMAF (Video Multi-method Assessment Fusion, video quality assessment) feature.

[0061] In a possible implementation manner, the video quality feature is determined by the following steps:

[0062] A: Perform distortion prediction processing on the target video to determine a distorted video.

[0063] Here, distortion prediction processing is performed on the target video to determine the distorted video.

[0064] In a possible implementation manner, performing distortion prediction processing on the target video to determine a distorted video includes:

[0065] a: Perform discrete cosine transform processing on the target video to determine multiple frequency domains of the target video.

[0066] Here, discrete cosine transform (DCT) processing is performed on the target video, and multiple frequency domains of the target video are determined after the discrete cosine transform processing.

[0067] b: performing quantization processing, inverse quantization processing and inverse transformation processing on each of the frequency domains according to a plurality of different quantization step sizes, and determining the distorted video corresponding to the different quantization step sizes.

[0068] Here, each bias is quantized using a different quantization step size, and then dequantized and transformed to obtain the distorted video corresponding to the different quantization step sizes.

[0069] Here, the quantization step size can be 0.625, 1.4062, 3.1875, 7.125, 6, 36, 89, 180, etc.

[0070] Here, according to the corresponding relationship between the quantization step size and the quantization parameter, it can be determined that the quantization parameter QP corresponds to 0, 7, 14, 21, 28, 35, 42 and 49.

[0071] In a specific embodiment, in order to effectively reduce the time complexity of the user distortion perception generation process, a fast VMAF quality prediction method is designed. Through DCT transformation and simulation of the quantization process in the HEVC encoder, the VMAF score of the video frame under different encoding conditions can be quickly and effectively predicted without actual encoding. The specific method is: DCT transformation is performed on each complete video frame, and then different quantization steps (0.625, 1.4062, 3.1875, 7.125, 16, 36, 89, 180) are used for the frequency domain coefficients, and then inverse quantization and inverse transformation are performed to obtain the distorted video frame. The score of the distorted video frame obtained by this method after VMAF scoring can be well fitted by the function y=ax^b with the VMAF value of the distorted video frame fully encoded by the HEVC encoder (corresponding to QP of 0, 7, 14, 21, 28, 35, 42, 49). Therefore, the VMAF score of the encoded video frame can be well predicted without actual encoding. The fast VMAF quality prediction method processes the target video to predict the distortion degree of the target video under different encoding conditions without actual encoding and decoding. The VMAF quality prediction value of the video under different encoding conditions will be used to extract video quality features, which will be sent to the trained model together with the video temporal and spatial features to predict the user distortion perception of the video.

[0072] B: Divide the distorted video into multiple sub-video segments.

[0073] Here, the distorted video is divided into several sub-video segments with fixed length (0.5S) and fixed ratio (320x180).

[0074] C: For each of the sub-video segments, a video degradation feature vector is determined based on the video quality value of the sub-video segment.

[0075] Here, for each sub-video segment, a video degradation feature vector is determined according to the video quality value of the sub-video segment.

[0076] The video quality value of the sub-video clip may be determined by a VMAF calculation tool or other tools.

[0077] Among them, a video degradation feature vector with a dimension of twenty can be generated by the difference between the video quality value of the sub-video clip and the VMAF full score, or by comparing the video quality value of the sub-video clip with a preset VMAF value. If it is less than the preset VMAF value, a video degradation feature vector with a dimension of twenty is generated by the difference between the video quality value of the sub-video clip and the VMAF full score. If it is greater than, there is no need to determine the video degradation feature vector. Here, the method for determining the video degradation feature vector is not limited.

[0078] Here, the value at the subscript index of the feature vector position i (i is [0,20) represents the ratio of the difference between the video quality value in the video segment formed by cutting the sub-video segment and the VMAF full score greater than or equal to 2*i.

[0079] D: Determine the video quality feature based on the video degradation feature vector of each of the sub-video clips.

[0080] Here, the video quality feature is determined according to the video degradation feature vector of each sub-video segment.

[0081] In a possible implementation, the time complexity characteristic is determined by the following steps:

[0082] Based on the historical frames of the video to be predicted, singular value decomposition is performed to predict the predicted grayscale value of the current frame of the video to be predicted; based on the predicted grayscale value of the current frame and the true grayscale value of the current frame, a residual value is determined; based on multiple residual values, a norm of the residual vector is calculated, and then the average is taken at the frame level to determine the time complexity feature.

[0083] In a specific embodiment, SVD decomposition (singular value decomposition) is performed on the historical frames, and the predicted grayscale value of the current frame is predicted using linear mapping. The residual value is determined based on the difference between the predicted grayscale value and the true grayscale value, and the temporal complexity characteristics are determined based on multiple residual values.

[0084] In a possible implementation manner, the spatial complexity feature is determined by the following steps:

[0085] The video to be predicted is divided into multiple sub-video segments, and each of the sub-video segments is divided into video blocks of a preset size; for each sub-video segment, the grayscale variance value of each video block of the sub-video segment is calculated, and the grayscale variance values ​​of multiple video blocks of the sub-video segment are averaged to determine the spatial complexity of the sub-video segment; based on the spatial complexity of the sub-video segment, the spatial complexity feature is determined.

[0086] In a specific embodiment, the spatial complexity feature is calculated by dividing the video frame in the sub-video segment into blocks of fixed size 32x32 (32*32 blocks are only for calculating the severity of grayscale changes in each local area, and finally the average of all 32*32 small blocks can represent the severity of grayscale changes in this frame, that is, the spatial complexity), calculating the grayscale value variance of each block, and finally averaging the grayscale value variance values ​​of different blocks to get the spatial complexity. The interval is divided into ten parts according to the preset max and min. This vector reflects the proportion of spatial complexity within each interval, and then the spatial complexity feature vector of the target video is determined.

[0087] S102: Input the video quality feature, the temporal complexity feature, and the spatial complexity feature into a pre-trained user perception training model, and output the user distortion perception.

[0088] In this step, the video quality features, time complexity features and space complexity features are input into a pre-trained user perception training model, and the user distortion perception is output.

[0089] Here, the user distortion perception refers to the user's perception of the distortion of the target video, which can also be used as the user satisfaction rate SUR (Satisfied User Ratio). The value is between 0 and 1, which directly reflects the proportion of users who can perceive the distorted image video. If the value is 1, it means that all users believe that there is no visible difference between the distorted image video and the original image video.

[0090] Here, the subjective perceived quality indicator is, for example, user distortion perception, that is, user satisfaction rate.

[0091] Here, the user perception training model is trained in the following way:

[0092] First, the original version of the dataset video contained in the VideoSet dataset, the distorted version under different encoding conditions, and the user distortion perception corresponding to the distorted version are used. Before extracting features, the video is divided into several video clips of fixed length (usually 0.5s) and fixed ratio (usually 320x180). The VMAF tool is used to calculate the VMAF value of each video clip, and a VMAF quality degradation feature vector with a dimension of 20 is generated based on the difference between the VMAF value and the VMAF full score. The calculation of the spatial complexity feature is to divide the video frame in the video clip into fixed 32x32 blocks (32*32 blocks are only for calculating the intensity of each local grayscale change, and finally all 32*32 small blocks are averaged to represent the intensity of grayscale change of this frame, that is, spatial complexity), calculate the color variance value of each block (here it needs to be changed to the grayscale value variance value of each block, and the calculation is to calculate the variance), and finally the variance values ​​of different blocks are averaged to obtain the spatial complexity feature. The spatial complexity of different video clips can be converted into a frequency distribution vector with a dimension of ten, which can be used as the spatial complexity feature vector of the original video. The temporal complexity feature is obtained by performing SVD decomposition (singular value decomposition) on the historical frames, and using linear mapping to predict the current frame. The residual value (the predicted value minus the true value is the residual, and the predicted value is used as the temporal complexity (after changing the shape of the residual into a one-dimensional vector, the 1 norm is calculated as the temporal complexity). The operation of converting the temporal complexity of the video clip into the temporal feature vector of the original video is the same as that of the spatial feature. It should be noted that the calculation of temporal features and spatial features are both performed in the grayscale domain. The three features are simultaneously input into the machine learning model for user distortion perception training to determine the user perception training model.

[0093] Here, in the actual encoding process, in order to effectively save computing resources, the perceptual quality modeling of the video frame is performed once every certain time period (about half a second) (the process of perceptual modeling includes: extracting video quality features of the target video (using the fast VMAF quality prediction method), extracting spatiotemporal features; sending the extracted features to the trained machine learning model, fitting the NSUR-R function; the above perceptual modeling process is processed in units of sliding windows, and each sliding window concentrates on processing 0.5s video frames. In order to save computing resources, some simplified operations are performed when perceptual modeling of the encoded video, including: 1. Each frame It is scaled to 320x180 resolution for spatiotemporal feature extraction; 2. When extracting video quality features, each sliding window will be directly subjected to fast VMAF distortion prediction and VMAF feature extraction (instead of being divided into small blocks). It can be considered that the video of each sliding window can only be divided into one video segment, rather than being divided into many small segments as in the training process. 3. In the actual encoding process, we found that for the three data points with the worst encoding quality, the NSUR value is often 1, and for the two data points with the best encoding quality, the NSUR value is often 0, so we directly set its value and skip calculating these data points to save computing resources.

[0094] S103: Determine a first objective function between the user distortion perception and the encoding bit rate based on the functional relationship between the quantization parameter and the user distortion perception and the rate-distortion function.

[0095] In this step, a first objective function between the user distortion perception and the encoding bit rate is determined according to the functional relationship between the quantization parameter corresponding to the quantization step size and the user distortion perception and the rate-distortion function.

[0096] Here, the relationship between the user's distortion perception and the quantization parameter QP can be established through a probability density accumulation function of a normal distribution.

[0097] In a possible implementation, the functional relationship includes a normal distribution probability density cumulative relationship; and determining a first objective function between the user distortion perception and the encoding bit rate based on the functional relationship between the quantization parameter and the user distortion perception and the rate-distortion function includes:

[0098] (1): Determine a first function based on the cumulative relationship between the user's distortion perception and the probability density of the normal distribution presented by the quantization parameter.

[0099] Here, the first function is:

[0100]

[0101] Among them, S is the user's distortion perception, QP is the quantization parameter, μ is the mean of the normal distribution, σ represents the variance of the normal distribution, and e is the natural logarithm.

[0102] (2) Perform an approximate transformation on the first function to determine a second function.

[0103] Here, the first function is approximately transformed to determine the second function.

[0104] Among them, the second function is:

[0105]

[0106] Among them, S is the user's distortion perception, QP is the quantization parameter, μ is the mean of the normal distribution, σ represents the variance of the normal distribution, e is the natural logarithm, and k is a constant.

[0107] (3): Based on the first function and the second function, a third function is determined.

[0108] Here, the third function is:

[0109]

[0110] Here, NSUR is 1-user distortion sensitivity, c is a parameter, d is a parameter, and R is the bit rate.

[0111] (4): Based on the third function and the rate-distortion function, a first objective function between the distortion perception and the encoding bit rate is determined.

[0112] Here, the first objective function is:

[0113]

[0114] Among them, R is the bit rate, C v (R) is the first objective function, c is a parameter, e is a parameter, i is the number of video frames, N is the total number of video frames, and f is the weight.

[0115] Here, the study shows that the relationship between the user's distortion perception and the quantization parameter QP can be established through the probability density accumulation function of the normal distribution. However, the probability density accumulation function involves integral operations, which is not conducive to subsequent derivation. Therefore, a clear functional relationship between the quantization parameter QP and the user's distortion perception can be obtained. On these basis, the first objective function between distortion perception and encoding bit rate is derived through the ready-made relationship model of QP, λ (Lambda is the rate-distortion weight in video coding, and the rate-distortion function is expressed as J=D+λR in the coding field, and λ can be understood as the importance of the bit rate in the optimization process) and bit rate R.

[0116] Here, generally at least seven anchor points are needed to generate a user satisfaction rate curve. In order to avoid the huge computational cost of repeating encoding seven times, a fast distortion VMAF prediction method is designed to obtain anchor points more accurately without repeating encoding to generate a user satisfaction rate curve, in preparation for subsequent function optimization.

[0117] S104: Based on a second objective function corresponding to a state stability target of an encoder buffer state and a first objective function between the user distortion perception and the encoding bit rate, the bit rate of each video frame of the target video is controlled.

[0118] In this step, the bit rate of each video frame of the target video is controlled according to the second objective function corresponding to the state stability target of the encoder buffer state and the first objective function between the user distortion perception and the encoding bit rate.

[0119] Here, code rate generally refers to bit rate, which means the data transmitted in bits per unit time. The code rate can be used to calculate the speed of digital image information transmission based on the number of bits occupied by each frame of image storage and the transmission bit rate. In channel coding, a source data block of K symbol size is mapped to a code word of N symbol size through encoding, and K / N is the code rate. The higher the code rate, the better the quality of audio and video, but the larger the encoded file. If the bit rate is lower, the situation is just the opposite.

[0120] Here, the expressions of the first objective function and the second objective function are:

[0121]

[0122] Among them, R is the bit rate, ω is the weight, C b (R) is the second objective function, height is the height of the video frame, width is the width of the video frame, R i is the bit rate of the i-th video frame, N is the total number of video frames, C V (R) is the first objective function, R budget is the total bit rate of N video frames.

[0123] In a possible implementation manner, the second objective function corresponding to the state stability target of the encoder buffer state is determined by the following steps:

[0124] I: obtaining a preset buffer occupancy ratio, where the preset buffer occupancy ratio is used to indicate that the buffer state after encoding each video frame satisfies a preset occupancy rate.

[0125] II: Based on the buffer interval of the currently encoded video frame, the preset buffer occupancy ratio, the target bit rate and the frame rate, determine the second objective function corresponding to the state stability target of the encoder buffer state.

[0126] Here, the second objective function is:

[0127]

[0128] Among them, B0 represents the buffer area where the currently encoded video frame is located, R j represents the bit rate of the jth frame, kB represents the preset buffer occupancy ratio, B r Indicates the target bit rate, F r Indicates the frame rate, C b (R) is the second objective function.

[0129] In a possible implementation manner, after the bit rate of each video frame of the target video is controlled based on the second objective function corresponding to the state stability target of the encoder buffer state and the first objective function between the user distortion perception and the encoding bit rate, the bit rate control method further includes:

[0130] i: for the current video frame, detecting whether the intra-frame prediction CTU ratio of the current video frame is greater than a preset value; wherein the intra-frame prediction CTU ratio of the current video frame is the ratio of the number of CTUs in the intra-frame prediction mode between a preset number of video frames encoded before the current video frame in the target video to the total number of CTUs between the video frames;

[0131] Here, for the current video frame, it is detected whether the ratio of intra-frame prediction CTUs between a preset number of video frames encoded before the current video frame in the target video is greater than a preset value.

[0132] Here, the preset value may be 0.1.

[0133] ii: If yes, weight adjustment is performed on the first objective function of the preset number of video frames before the current video frame, and the video frame after the current video frame is encoded according to the inter-frame relationship between the first objective function after weight adjustment, the second objective function and the preset number of video frames before the current video frame;

[0134] Here, if the ratio of intra-frame predicted CTUs between a preset number of video frames before the current video frame is greater than a preset value, it is considered that the inter-frame relationship needs to be considered, and the weight adjustment is performed through the second objective function with a fixed bit allocation ratio for the preset number of video frames before the current video frame in the HM reference software, and the preset number of video frames after the current video frame are encoded according to the second objective function after adjusting the weights, the first objective function, and the inter-frame relationship between the preset number of video frames before the current video frame.

[0135] iii: If not, encoding the video frames subsequent to the current video frame according to the first objective function and the second objective function.

[0136] Here, if the intra-frame prediction CTU ratio between a preset number of video frames before the current video frame is less than a preset value, it is considered that the inter-frame relationship does not need to be considered, and the preset number of video frames after the current video frame are encoded according to the first objective function and the second objective function.

[0137] In a specific embodiment, there are frequent inter-frame references in video coding. If the degree of change between the previous and next frames is not large, the inter-frame reference is very effective. If the degree of change between the previous and next frames is large, the inter-frame reference will be inaccurate. Here, the proportion of intra-frame references is selected by counting the CTU in the coded frame. If the proportion is greater than 0.1, the inter-frame reference is considered invalid, otherwise the inter-frame reference is considered valid. When the inter-frame reference is valid, we need to focus on optimizing those video frames that are frequently referenced, that is, by adjusting the weights in the first objective function.

[0138] Most observers of encoded videos are humans. The perception of the image and video quality by the human eye perception system cannot be perfectly reflected by objective indicators such as MSE and PSNR. At present, there is room for improvement in the optimization of objective distortion. In addition, in scenes with more drastic video changes, the human eye is not very sensitive to distortion. At the same time, the encoded bitstream will fluctuate greatly, causing overflow and underflow of the encoding buffer. Therefore, combining human eye perception quality and buffer status control for bitrate control is a meaningful work improvement.

[0139] The embodiment of the present application provides a rate control method for video encoding, the rate control method comprising: obtaining a target video to be processed, and determining the video quality characteristics, time complexity characteristics and space complexity characteristics of the target video; inputting the video quality characteristics, the time complexity characteristics and the space complexity characteristics into a pre-trained user perception training model, and outputting the user distortion perception; determining the first objective function between the user distortion perception and the encoding bit rate based on the functional relationship between the quantization parameter and the user distortion perception and the rate distortion function; and controlling the bit rate of each video frame of the target video based on the second objective function corresponding to the state stability target of the encoder buffer state and the first objective function between the user distortion perception and the encoding bit rate. Through the second objective function corresponding to the state stability target of the encoder buffer state and the first objective function between the user distortion perception and the encoding bit rate, the frame-level rate allocation is realized by combining the buffer state and the user subjective perception quality index, which can effectively reduce the overflow and underflow of the encoder buffer during the encoding process, and can improve the subjective perception quality index of the video when considering the frame loss of the encoder buffer.

[0140] See also Figure 2 , Figure 3 , Figure 2 One of the structural schematic diagrams of a video encoding bit rate control device provided in an embodiment of the present application; Figure 3 The second structural diagram of a video coding rate control device provided in an embodiment of the present application is as follows. The video coding rate control device 200 includes:

[0141] The determination module 210 is used to obtain a target video to be processed and determine the video quality characteristics, time complexity characteristics and space complexity characteristics of the target video;

[0142] The distortion perception determination module 220 is used to input the video quality feature, the time complexity feature and the space complexity feature into a pre-trained user perception training model, and output the user distortion perception;

[0143] A first objective function determination module 230, configured to determine a first objective function between the user distortion perception and the encoding bit rate based on a functional relationship between a quantization parameter and the user distortion perception and a rate-distortion function;

[0144] The bit rate adjustment module 240 is used to control the bit rate of each video frame of the target video based on the second objective function corresponding to the state stability target of the encoder buffer state and the first objective function between the user distortion perception and the encoding bit rate.

[0145] Furthermore, the determination module 210 determines the video quality feature through the following steps:

[0146] Performing distortion prediction processing on the target video to determine a distorted video;

[0147] Dividing the distorted video into a plurality of sub-video segments;

[0148] For each of the sub-video segments, determining a video degradation feature vector based on the video quality value of the sub-video segment;

[0149] The video quality feature is determined based on the video degradation feature vector of each of the sub-video segments.

[0150] Furthermore, when the determination module 210 is used to perform distortion prediction processing on the target video to determine the distorted video, the determination module 210 is specifically used to:

[0151] Performing discrete cosine transform processing on the target video to determine multiple frequency domains of the target video;

[0152] Quantization processing, inverse quantization processing and inverse transformation processing are respectively performed on each of the frequency domains according to a plurality of different quantization step sizes, so as to determine the distorted video corresponding to the different quantization step sizes.

[0153] Further, when the first objective function determination module 230 is used for the functional relationship including the normal distribution probability density cumulative relationship; the functional relationship between the quantization parameter and the user distortion perception and the rate-distortion function are used to determine the first objective function between the user distortion perception and the encoding bit rate, the first objective function determination module 230 is specifically used to:

[0154] Determining a first function based on a cumulative relationship between the user's distortion perception and a probability density of a normal distribution presented by the quantization parameter;

[0155] Performing an approximate transformation on the first function to determine a second function;

[0156] Determine a third function based on the first function and the second function;

[0157] Based on the third function and the rate-distortion function, a first objective function between the distortion perception and the encoding bit rate is determined.

[0158] Further, the bit rate adjustment module 240 determines a second objective function corresponding to the state stability target of the encoder buffer state through the following steps:

[0159] Obtaining a preset buffer occupancy ratio, where the buffer occupancy ratio represents an occupancy rate of a buffer state after encoding each video frame;

[0160] Based on the buffer interval in which the currently encoded video frame is located, the preset buffer occupancy ratio, the target bit rate and the frame rate, a second objective function corresponding to the state stability target of the encoder buffer state is determined.

[0161] Further, such as Figure 3 As shown, the video coding bit rate control device 200 further includes a coding module 250, and the coding module 250 is used to:

[0162] For the current video frame, detecting whether the intra-frame prediction CTU ratio of the current video frame is greater than a preset value; wherein the intra-frame prediction CTU ratio of the current video frame is the ratio of the number of CTUs in the intra-frame prediction mode between a preset number of video frames encoded before the current video frame in the target video to the number of CTUs between the video frames;

[0163] If yes, weight adjustment is performed on the first objective function of the preset number of video frames before the current video frame, and the video frame after the current video frame is encoded according to the inter-frame relationship between the first objective function after weight adjustment, the second objective function and the preset number of video frames before the current video frame;

[0164] If not, the video frames following the current video frame are encoded according to the first objective function and the second objective function.

[0165] An embodiment of the present application provides a video encoding bit rate control device, the bit rate control device comprising: a determination module, used to obtain a target video to be processed, and determine the video quality characteristics, time complexity characteristics and space complexity characteristics of the target video; a distortion perception determination module, used to input the video quality characteristics, the time complexity characteristics and the space complexity characteristics into a pre-trained user perception training model, and output the user distortion perception; a first objective function determination module, used to determine a first objective function between the user distortion perception and the encoding bit rate based on a functional relationship between a quantization parameter and the user distortion perception and a rate-distortion function; a bit rate adjustment module, used to control the bit rate of each video frame of the target video based on a second objective function corresponding to a state stability target of an encoder buffer state and a first objective function between the user distortion perception and the encoding bit rate. Through the second objective function corresponding to the state stability target of the encoder buffer state and the first objective function between the user distortion perception and the encoding bit rate, frame-level bit rate control is achieved by combining the buffer state and the user's subjective perception quality index. This can effectively reduce the overflow and underflow of the encoder buffer during the encoding process, and can improve the subjective perception quality index of the video while considering the frame loss of the encoder buffer.

[0166] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown in , the electronic device 400 includes a processor 410 , a memory 420 and a bus 430 .

[0167] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, the above-mentioned Figure 1 The steps of the video coding rate control method in the method embodiment shown in the embodiment and the specific implementation thereof can be referred to the method embodiment, which will not be described in detail here.

[0168] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The steps of the video coding rate control method in the method embodiment shown in the embodiment and the specific implementation thereof can be referred to the method embodiment, which will not be described in detail here.

[0169] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0170] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0171] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0172] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0173] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0174] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A video coding rate control method, characterized in that: The bit rate control method comprises: Acquire a target video to be processed, and determine the video quality characteristics, time complexity characteristics, and space complexity characteristics of the target video; Inputting the video quality feature, the temporal complexity feature and the spatial complexity feature into a pre-trained user perception training model, and outputting the user distortion perception; Determining a first objective function between the user distortion perception and the encoding bit rate based on a functional relationship between a quantization parameter and the user distortion perception and a rate-distortion function; Controlling the bit rate of each video frame of the target video based on a second objective function corresponding to a state stability target of an encoder buffer state and a first objective function between the user distortion perception and the encoding bit rate; The functional relationship includes a normal distribution probability density cumulative relationship; the functional relationship between the quantization parameter and the user distortion perception and the rate-distortion function are used to determine a first objective function between the user distortion perception and the encoding bit rate, including: Determining a first function based on a cumulative relationship between the user's distortion perception and a probability density of a normal distribution presented by the quantization parameter; Among them, the first function is: Where S is the user's distortion perception, QP is the quantization parameter, μ is the mean of the normal distribution, σ represents the variance of the normal distribution, and e is the natural logarithm; Performing an approximate transformation on the first function to determine a second function; Among them, the second function is: Where S is the user's distortion perception, QP is the quantization parameter, μ is the mean of the normal distribution, σ represents the variance of the normal distribution, e is the natural logarithm, and k is a constant; Determine a third function based on the first function and the second function; Among them, the third function is: Among them, NSUR is 1-user distortion perception, c is a parameter, d is a parameter, and R is the bit rate; Determining a first objective function between the distortion perception and the encoding bit rate based on the third function and the rate-distortion function; Among them, the first objective function is: Among them, R is the bit rate, C v (R) is the first objective function, c is a parameter, e is a parameter, i is the number of video frames, N is the total number of video frames, and f is the weight; The second objective function corresponding to the state stability target of the encoder buffer state is determined by the following steps: Obtaining a preset buffer occupancy ratio, where the buffer occupancy ratio represents an occupancy rate of a buffer state after encoding each video frame; Determine a second objective function corresponding to a state stability target of the encoder buffer state based on the buffer interval of the currently encoded video frame, the preset buffer occupancy ratio, the target bit rate, and the frame rate; Among them, the second objective function is: After controlling the bit rate of each video frame of the target video, the bit rate control method further includes: For the current video frame, detecting whether the intra-frame prediction CTU ratio of the current video frame is greater than a preset value; wherein the intra-frame prediction CTU ratio of the current video frame is the ratio of the number of CTUs in the intra-frame prediction mode between a preset number of video frames encoded before the current video frame in the target video to the total number of CTUs between the video frames; If yes, weight adjustment is performed on the first objective function of the preset number of video frames before the current video frame, and the video frame after the current video frame is encoded according to the inter-frame relationship between the first objective function after weight adjustment, the second objective function and the preset number of video frames before the current video frame; If not, the video frames following the current video frame are encoded according to the first objective function and the second objective function.

2. The bit rate control method according to claim 1, characterized in that: The video quality characteristics are determined by the following steps: Performing distortion prediction processing on the target video to determine a distorted video; Dividing the distorted video into a plurality of sub-video segments; For each of the sub-video segments, determining a video degradation feature vector based on the video quality value of the sub-video segment; The video quality feature is determined based on the video degradation feature vector of each of the sub-video segments.

3. The bit rate control method according to claim 2, characterized in that: The performing distortion prediction processing on the target video to determine the distorted video includes: Performing discrete cosine transform processing on the target video to determine multiple frequency domains of the target video; Quantization processing, inverse quantization processing and inverse transformation processing are respectively performed on each of the frequency domains according to a plurality of different quantization step sizes, so as to determine the distorted video corresponding to the different quantization step sizes.

4. A video coding bit rate control device, characterized in that: The bit rate control device comprises: A determination module, used to obtain a target video to be processed, and determine the video quality characteristics, time complexity characteristics, and space complexity characteristics of the target video; A distortion perception determination module, used for inputting the video quality feature, the temporal complexity feature and the spatial complexity feature into a pre-trained user perception training model, and outputting the user distortion perception; A first objective function determination module, configured to determine a first objective function between the user distortion perception and the encoding bit rate based on a functional relationship between a quantization parameter and the user distortion perception and a rate-distortion function; A bit rate adjustment module, configured to control the bit rate of each video frame of the target video based on a second objective function corresponding to a state stability target of an encoder buffer state and a first objective function between the user distortion perception and the encoding bit rate; When the first objective function determination module is used for the functional relationship including the normal distribution probability density cumulative relationship; and the first objective function between the user distortion perception and the encoding bit rate is determined based on the functional relationship between the quantization parameter and the user distortion perception and the rate-distortion function, the first objective function determination module is specifically used for: Determining a first function based on a cumulative relationship between the user's distortion perception and a probability density of a normal distribution presented by the quantization parameter; Among them, the first function is: Where S is the user's distortion perception, QP is the quantization parameter, μ is the mean of the normal distribution, σ represents the variance of the normal distribution, and e is the natural logarithm; Performing an approximate transformation on the first function to determine a second function; Among them, the second function is: Where S is the user's distortion perception, QP is the quantization parameter, μ is the mean of the normal distribution, σ represents the variance of the normal distribution, e is the natural logarithm, and k is a constant; Determine a third function based on the first function and the second function; Among them, the third function is: Among them, NSUR is 1-user distortion perception, c is a parameter, d is a parameter, and R is the bit rate; Determining a first objective function between the distortion perception and the encoding bit rate based on the third function and the rate-distortion function; Among them, the first objective function is: Among them, R is the bit rate, C v (R) is the first objective function, c is a parameter, e is a parameter, i is the number of video frames, N is the total number of video frames, and f is the weight; The bit rate adjustment module determines a second objective function corresponding to the state stability target of the encoder buffer state through the following steps: Obtaining a preset buffer occupancy ratio, where the buffer occupancy ratio represents an occupancy rate of a buffer state after encoding each video frame; Determine a second objective function corresponding to a state stability target of the encoder buffer state based on the buffer interval of the currently encoded video frame, the preset buffer occupancy ratio, the target bit rate, and the frame rate; Among them, the second objective function is: The video coding bit rate control device also includes a coding module, which is used to: For the current video frame, detecting whether the intra-frame prediction CTU ratio of the current video frame is greater than a preset value; wherein the intra-frame prediction CTU ratio of the current video frame is the ratio of the number of CTUs in the intra-frame prediction mode between a preset number of video frames encoded before the current video frame in the target video to the total number of CTUs between the video frames; If yes, weight adjustment is performed on the first objective function of the preset number of video frames before the current video frame, and the video frame after the current video frame is encoded according to the inter-frame relationship between the first objective function after weight adjustment, the second objective function and the preset number of video frames before the current video frame; If not, the video frames following the current video frame are encoded according to the first objective function and the second objective function.

5. The bit rate control device according to claim 4, characterized in that: The determining module determines the video quality feature by the following steps: Performing distortion prediction processing on the target video to determine a distorted video; Dividing the distorted video into a plurality of sub-video segments; For each of the sub-video segments, determining a video degradation feature vector based on the video quality value of the sub-video segment; The video quality feature is determined based on the video degradation feature vector of each of the sub-video segments.

6. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the video encoding bit rate control method as described in any one of claims 1 to 3.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video encoding bit rate control method according to any one of claims 1 to 3 are executed.

Images