Video coding method and device, electronic equipment, storage medium, program product and method for generating bit stream

Abstract

The invention provides a video coding method and device, electronic equipment, a storage medium, a program product and a bit stream generation method, and the video coding method comprises the steps: obtaining a frame set which comprises a current frame and a plurality of subsequent frames; time-domain correlation analysis is carried out on the frame set to determine a correlation category of the frame set from a plurality of preset correlation categories, the time-domain correlation is time correlation between adjacent frames, and the plurality of preset correlation categories are used for representing different correlation degrees of the time-domain correlation; each preset correlation category in the plurality of preset correlation categories has a corresponding code rate prediction parameter; according to a code rate prediction parameter corresponding to the correlation category of the frame set, performing code rate prediction on the frame set to obtain a predicted code rate of the frame set; determining a quantization parameter according to the predicted code rate of the frame set; and encoding the current frame by using the quantization parameter. The method is helpful for improving the accuracy of code rate prediction.

Description

Technical Field

[0001] The present disclosure relates to video coding and compression. More specifically, the present disclosure relates to a video coding method, apparatus, electronic device, storage medium, program product and method for generating a bit stream. Background Art

[0002] In real-time video transmission scenarios such as live video broadcasting and remote conferencing, whether the bitrate fluctuation of the video stream after encoding is stable will greatly affect the user's playback experience. Frames with higher bitrates often take more time to decode, and the drastic changes in bitrates will cause a huge difference in the decoding time of each frame, which will cause playback jams. In order to reduce the fluctuation of bitrate, a common method in the industry is to establish a virtual video buffer on the encoding end to simulate a series of operations from receiving the bitstream to decoding on the decoding end, thereby smoothing the output bitrate of the encoding end. This module is called Video Buffer Verifier (VBV).

[0003] VBV's bitrate control can be summarized as a leaky bucket model. Since the simulation is of the decoding process, the input bitrate of the VBV cache is constant, that is, the maximum bitrate set by the encoding end, and the output is the actual bitrate for encoding each frame. In order to ensure a stable bitrate fluctuation, the number of bits in the VBV cache needs to be within a reasonable range, that is, between the overflow warning line and the underflow warning line, to avoid cache overflow and underflow.

[0004] In the existing technical solution, the coding complexity of the current frame and the subsequent N frames is obtained in advance to estimate the VBV buffer fullness after the N+1 frames are fully coded, and then the quantization parameters (QP) used to encode the current frame are determined based on the buffer fullness. However, the existing solution uses the same set of bit rate prediction parameters for the entire video. Such a one-size-fits-all solution may affect the accuracy of bit rate prediction, so there is still room for optimization. Summary of the invention

[0005] Embodiments of the present disclosure provide a video encoding method, apparatus, electronic device, storage medium, program product, and method for generating a bitstream, for solving at least one of the above problems.

[0006] According to one aspect of the present disclosure, a video encoding method is provided, the video encoding method comprising: acquiring a frame set, wherein the frame set comprises a current frame and a plurality of subsequent frames thereof; performing a time domain correlation analysis on the frame set to determine a correlation category of the frame set from a plurality of preset correlation categories, wherein the time domain correlation is a temporal correlation between adjacent frames, the plurality of preset correlation categories are used to represent different correlation degrees of the time domain correlation, and each of the plurality of preset correlation categories has a corresponding rate prediction parameter; performing a rate prediction on the frame set according to the rate prediction parameter corresponding to the correlation category of the frame set to obtain a predicted rate of the frame set; determining a quantization parameter according to the predicted rate of the frame set; and encoding the current frame using the quantization parameter.

[0007] Optionally, performing time domain correlation analysis on the frame set includes: performing time domain correlation analysis on the frame set based on feature information of each frame in the frame set, wherein the feature information includes at least one of the following: intra-frame prediction loss, inter-frame prediction loss, length of motion vector, and texture complexity.

[0008] Optionally, in the case that the characteristic information includes the intra-frame prediction loss and the inter-frame prediction loss, the time domain correlation analysis is performed on the frame set based on the characteristic information of each frame in the frame set, including: for each frame in the frame set, by comparing the intra-frame prediction loss with the inter-frame prediction loss, determining the correlation measure of each frame; determining the statistical value of the correlation measure of each frame in the frame set; by comparing the statistical value of the correlation measure with a statistical value threshold, and comparing the intra-frame prediction loss of the current frame with the intra-frame prediction loss threshold, performing time domain correlation analysis on the frame set.

[0009] Optionally, the video encoding method further includes: updating the rate prediction parameters corresponding to the correlation category of the frame set according to the quantization parameter and the actual bit rate for encoding the current frame, wherein the updated rate prediction parameters are applied to other frames in the current video that are encoded after the current frame.

[0010] Optionally, each of the multiple preset correlation categories has a corresponding multiple groups of rate prediction parameters, wherein the multiple groups of rate prediction parameters correspond one-to-one to multiple frame types; wherein, performing rate prediction on the frame set according to the rate prediction parameters corresponding to the correlation category of the frame set includes: for each frame in the frame set, performing rate prediction on each frame separately according to the correlation category of the frame set and the rate prediction parameters corresponding to the frame type of each frame, so as to perform rate prediction on the frame set; wherein, updating the rate prediction parameters corresponding to the correlation category of the frame set according to the quantization parameter and the actual bit rate for encoding the current frame includes: updating the rate prediction parameters corresponding to the correlation category of the frame set and the frame type of the current frame according to the quantization parameter and the actual bit rate for encoding the current frame.

[0011] Optionally, the rate prediction parameters corresponding to each preset correlation category in the multiple preset correlation categories are obtained through the following steps: for each preset correlation category in the multiple preset correlation categories, multiple sample frame sets of the preset correlation category are obtained; the multiple sample frame sets are respectively encoded using preset quantization parameters to obtain multiple groups of target encoding data, wherein the target encoding data includes encoding data related to the rate prediction parameters; and the multiple groups of target encoding data are fitted to obtain the rate prediction parameters corresponding to the preset correlation category.

[0012] Optionally, the multiple sample frame sets include frame sets in sample videos of a preset video category.

[0013] According to another aspect of the present disclosure, a video encoding device is provided, the video encoding device comprising: an acquisition unit, configured to acquire a frame set, wherein the frame set comprises a current frame and a plurality of subsequent frames thereof; an analysis unit, configured to perform a time domain correlation analysis on the frame set to determine a correlation category of the frame set from a plurality of preset correlation categories, wherein the time domain correlation is a time correlation between adjacent frames, the plurality of preset correlation categories are used to represent different correlation degrees of the time domain correlation, and each preset correlation category in the plurality of preset correlation categories has a corresponding rate prediction parameter; a prediction unit, configured to perform a rate prediction on the frame set according to the rate prediction parameter corresponding to the correlation category of the frame set to obtain a predicted rate of the frame set; a determination unit, configured to determine a quantization parameter according to the predicted rate of the frame set; and an encoding unit, configured to encode the current frame using the quantization parameter.

[0014] Optionally, the analysis unit is further configured to perform time domain correlation analysis on the frame set based on feature information of each frame in the frame set, wherein the feature information includes at least one of the following: intra-frame prediction loss, inter-frame prediction loss, length of motion vector, and texture complexity.

[0015] Optionally, in the case where the feature information includes the intra-frame prediction loss and the inter-frame prediction loss, the analysis unit is further configured to: for each frame in the frame set, determine the correlation measure of each frame by comparing the intra-frame prediction loss with the inter-frame prediction loss; determine the statistical value of the correlation measure of each frame in the frame set; perform time domain correlation analysis on the frame set by comparing the statistical value of the correlation measure with a statistical value threshold and comparing the intra-frame prediction loss of the current frame with the intra-frame prediction loss threshold.

[0016] Optionally, the video encoding device also includes an updating unit, which is configured to update the rate prediction parameters corresponding to the correlation category of the frame set based on the quantization parameter and the actual bit rate for encoding the current frame, wherein the updated rate prediction parameters are applied to other frames in the current video that are encoded after the current frame.

[0017] Optionally, each of the multiple preset correlation categories has corresponding multiple groups of rate prediction parameters, wherein the multiple groups of rate prediction parameters correspond one-to-one to multiple frame types; the prediction unit is also configured to perform rate prediction on each frame in the frame set according to the correlation category of the frame set and the rate prediction parameters corresponding to the frame type of each frame, so as to perform rate prediction on the frame set; the update unit is also configured to update the rate prediction parameters corresponding to the correlation category of the frame set and the frame type of the current frame according to the quantization parameter and the actual bit rate for encoding the current frame.

[0018] Optionally, the rate prediction parameters corresponding to each preset correlation category in the multiple preset correlation categories are obtained through the following steps: for each preset correlation category in the multiple preset correlation categories, multiple sample frame sets of the preset correlation category are obtained; the multiple sample frame sets are respectively encoded using preset quantization parameters to obtain multiple groups of target encoding data, wherein the target encoding data includes encoding data related to the rate prediction parameters; and the multiple groups of target encoding data are fitted to obtain the rate prediction parameters corresponding to the preset correlation category.

[0019] Optionally, the multiple sample frame sets include frame sets in sample videos of a preset video category.

[0020] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the video encoding method as described above.

[0021] According to another aspect of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to perform the video encoding method as described above.

[0022] According to another aspect of the present disclosure, a computer program product is provided, comprising computer instructions, which implement the video encoding method as described above when the computer instructions are executed by at least one processor.

[0023] According to another aspect of the present disclosure, a method for generating a bitstream is provided, comprising: generating a bitstream according to the video encoding method as described above.

[0024] According to the video encoding method, apparatus, electronic device, storage medium, program product and method for generating a bitstream according to the exemplary embodiments of the present disclosure, by analyzing the time domain correlation of frame sets to classify the frame sets accordingly, and configuring different rate prediction parameters for different correlation categories, the frame sets can be reasonably classified, and the rate prediction can be performed differentially in subsequent steps, which helps to improve the accuracy of rate prediction, improve encoding efficiency and encoding quality.

[0025] It is to be understood that both the foregoing general description and the following detailed description are examples only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate examples according to the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0027] Figure 1 is a block diagram illustrating an exemplary system for encoding and decoding video blocks according to some embodiments of the present disclosure.

[0028] Figure 2 is a block diagram illustrating an exemplary video encoder according to some embodiments of the present disclosure.

[0029] Figure 3 is a block diagram illustrating an exemplary video decoder according to some embodiments of the present disclosure.

[0030] Figure 4 is a schematic diagram showing a leaky bucket model for rate control according to some embodiments of the present disclosure.

[0031] Figure 5 is a schematic diagram showing an algorithm flow of a video cache checker according to some embodiments of the present disclosure.

[0032] Figure 6 is a flowchart illustrating a video encoding method according to some embodiments of the present disclosure.

[0033] Figure 7 1 is a schematic diagram showing an algorithm flow of a video cache checker according to other embodiments of the present disclosure.

[0034] Figure 8 is a block diagram illustrating a video encoding apparatus according to some embodiments of the present disclosure.

[0035] Fig. 9 is a diagram illustrating a computing environment coupled with a user interface according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0036] Reference will now be made in detail to specific embodiments, examples of which are shown in the accompanying drawings. In the following detailed description, a number of non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, various alternatives may be used without departing from the scope of the claims, and the subject matter may be practiced without these specific details. For example, the subject matter presented herein may be implemented on many types of electronic devices having digital video capabilities.

[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the drawings are used to distinguish objects, but not to describe any specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those shown in the drawings or described in the present disclosure.

[0038] Figure 1 is a block diagram illustrating an exemplary system 10 for encoding and decoding video blocks in parallel according to some embodiments of the present disclosure. Figure 1 As shown in , system 10 includes a source device 12 that generates and encodes video data to be later decoded by a target device 14. Source device 12 and target device 14 may include any of a wide variety of electronic devices, including cloud servers, server computers, desktop or laptop computers, tablet computers, smart phones, set-top boxes, digital televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some implementations, source device 12 and target device 14 are equipped with wireless communication capabilities.

[0039] In some implementations, target device 14 may receive the encoded video data to be decoded via link 16. Link 16 may include any type of communication medium or device capable of moving the encoded video data from source device 12 to target device 14.

[0040] In some other embodiments, the encoded video data may be sent from the output interface 22 to the storage device 32. The encoded video data in the storage device 32 may then be accessed by the target device 14 via the input interface 28.

[0041] like Figure 1 , source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Video source 18 may include sources such as, or a combination of such sources: a video capture device (e.g., a camera), a video archive containing previously captured video, a video feed interface for receiving video from a video content provider, and / or a computer graphics system for generating computer graphics data as source video.

[0042] The captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video data may be sent directly to target device 14 via output interface 22 of source device 12. The encoded video data may also (or alternatively) be stored on storage device 32 for later access by target device 14 or other devices for decoding and / or playback.

[0043] Target device 14 includes input interface 28, video decoder 30, and display device 34. Input interface 28 may include a receiver and / or a modem and receives encoded video data via link 16. The encoded video data transmitted via link 16 or provided on storage device 32 may include various syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements may be included within the encoded video data sent over a communication medium, stored on a storage medium, or stored on a file server.

[0044] The video encoder 20 and the video decoder 30 can operate according to a proprietary standard or an industry standard (e.g., VVC, HEVC, Part 10 of MPEG-4, AVC) or an extension of such a standard. It should be understood that the present disclosure is not limited to a specific video encoding / decoding standard and can be applied to other video encoding / decoding standards. It is generally believed that the video encoder 20 of the source device 12 can be configured to encode the video data according to any of these current standards or future standards. Similarly, it is also generally believed that the video decoder 30 of the target device 14 can be configured to decode the video data according to any of these current standards or future standards.

[0045] The video encoder 20 and the video decoder 30 may be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic devices, software, hardware, firmware, or any combination thereof. When partially implemented in software, the electronic device may store instructions for the software in a suitable non-transitory computer-readable medium, and use one or more processors to execute the instructions in the hardware to perform the video encoding / decoding operations disclosed in the present disclosure. Each of the video encoder 20 and the video decoder 30 may be included in one or more encoders or decoders, and either of the encoders or decoders may be integrated as part of a combined encoder / decoder (CODEC) in the corresponding device.

[0046] Figure 2 is a block diagram illustrating an exemplary video encoder 20 according to some embodiments described in the present disclosure. The video encoder 20 can perform intra-frame prediction coding and inter-frame prediction coding on video blocks within a video frame. Intra-frame prediction coding relies on spatial prediction to reduce or remove spatial redundancy in video data within a given video frame or picture. Inter-frame prediction coding relies on temporal prediction to reduce or remove temporal redundancy in video data within adjacent video frames or pictures of a video sequence. It should be noted that in the field of video coding and decoding, the term "frame" can be used as a synonym for the term "image" or "picture".

[0047] like Figure 2As shown in , the video encoder 20 includes a video data memory 40, a prediction processing unit 41, a decoded picture buffer (DPB) 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The prediction processing unit 41 further includes a motion estimation unit 42, a motion compensation unit 44, a segmentation unit 45, an intra-frame prediction processing unit 46, and an intra-frame block copy (BC) unit 48. In some embodiments, the video encoder 20 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and an adder 62 for video block reconstruction. A loop filter 63 such as a deblocking filter can be located between the adder 62 and the DPB 64 to filter the block boundary to remove the block effect from the reconstructed video. In addition to the deblocking filter, another loop filter (e.g., a sample adaptive offset (SAO) filter, a cross-component sample adaptive offset (CCSAO) filter, and / or an adaptive loop filter (ALF)) can also be used to filter the output of the adder 62. In some examples, the loop filter may be omitted, and the decoded video blocks may be provided directly by adder 62 to DPB 64. Video encoder 20 may take the form of a fixed or programmable hardware unit, or may be distributed among one or more of the illustrated fixed or programmable hardware units.

[0048] Video data memory 40 may store video data to be encoded by components of video encoder 20. Figure 1 Video source 18 is shown obtaining video data in video data memory 40. DPB 64 is a buffer that stores reference video data (eg, reference frames or pictures) for use by video encoder 20 in encoding the video data (eg, in intra or inter prediction coding modes).

[0049] like Figure 2 As shown in , after receiving the video data, the segmentation unit 45 within the prediction processing unit 41 segments the video data into video blocks. This segmentation may also include segmenting the video frame into strips, tiles (e.g., a set of video blocks), or other larger coding units (CUs) according to a predefined splitting structure associated with the video data, such as a quadtree (QT) structure. It should be noted that the terms "block" or "video block" used herein may be a portion of a frame or picture, in particular a rectangular (square or non-square) portion. With reference to, for example, HEVC and VVC, a block or video block may be or correspond to a coding tree unit (CTU), a CU, a prediction unit (PU), or a transform unit (TU) and / or may be or correspond to a corresponding block (e.g., a coding tree block (CTB), a coding block (CB), a prediction block (PB), or a transform block (TB)) and / or a sub-block.

[0050] The prediction processing unit 41 may select one of a plurality of feasible prediction coding modes for the current video block based on the error results (e.g., coding rate and distortion level), such as one of one or more inter-frame prediction coding modes in a plurality of intra-frame prediction coding modes. The prediction processing unit 41 may provide the resulting intra-frame prediction coding block or inter-frame prediction coding block to the adder 50 to generate a residual block, and to the adder 62 to reconstruct the coding block for subsequent use as part of a reference frame. The prediction processing unit 41 also provides syntax elements (e.g., motion vectors, intra-frame mode indicators, partition information, and other such syntax information) to the entropy coding unit 56.

[0051] To select an appropriate intra-prediction coding mode for the current video block, intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-prediction coding of the current video block in relation to one or more neighboring blocks in the same frame as the current block to be encoded to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 may perform inter-prediction coding of the current video block in relation to one or more prediction blocks in one or more reference frames to provide temporal prediction. Video encoder 20 may perform multiple encoding passes, for example, to select an appropriate coding mode for each block of video data.

[0052] In some embodiments, motion estimation unit 42 determines an inter-prediction mode for a current video frame by generating a motion vector according to a predetermined pattern within a sequence of video frames, the motion vector indicating the displacement of a video block within the current video frame relative to a prediction block within a reference video frame. Motion estimation performed by motion estimation unit 42 is the process of generating a motion vector that estimates the motion of a video block. For example, a motion vector may indicate the displacement of a video block within a current video frame or picture relative to a prediction block within a reference frame associated with a current block being encoded within the current frame. The predetermined pattern may designate a video frame in a sequence as a P frame or a B frame. Intra BC unit 48 may determine a vector (e.g., a block vector) for intra BC coding in a manner similar to the motion vector determined by motion estimation unit 42 for inter prediction, or may utilize motion estimation unit 42 to determine a block vector.

[0053] Regardless of whether the prediction block is from the same frame according to intra-frame prediction or from a different frame according to inter-frame prediction, the video encoder 20 can form a residual video block by subtracting the pixel values ​​of the prediction block from the pixel values ​​of the current video block being encoded. The pixel difference values ​​forming the residual video block may include both luma component differences and chroma component differences.

[0054] Intra-prediction processing unit 46 may encode the current block using various intra-prediction modes, for example, during separate encoding passes, and intra-prediction processing unit 46 (or, in some examples, mode selection unit) may select an appropriate intra-prediction mode to use from the tested intra-prediction modes. Intra-prediction processing unit 46 may provide information indicating the intra-prediction mode selected for the block to entropy encoding unit 56. Entropy encoding unit 56 may encode the information indicating the selected intra-prediction mode into the bitstream.

[0055] After prediction processing unit 41 determines a prediction block for the current video block via inter-prediction or intra-prediction, adder 50 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block may be included in one or more TUs and provided to transform processing unit 52. Transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform (e.g., a discrete cosine transform (DCT) or a conceptually similar transform).

[0056] The transform processing unit 52 may send the resulting transform coefficients to a quantization unit 54. The quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may also reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, the quantization unit 54 may then perform a scan on the matrix including the quantized transform coefficients. Alternatively, the entropy coding unit 56 may perform the scan.

[0057] After quantization, entropy coding unit 56 entropy encodes the quantized transform coefficients into a video bitstream using, for example, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding method or technique. The encoded bitstream may then be sent to a video bitstream such as a Figure 1 The video decoder 30 shown, or archived as Figure 1 The video frame may be stored in storage device 32 as shown for later transmission to or retrieval by video decoder 30. Entropy encoding unit 56 may also entropy encode motion vectors and other syntax elements for the current video frame being encoded.

[0058] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual video block in the pixel domain for use in generating reference blocks for predicting other video blocks. As noted above, motion compensation unit 44 may generate a motion compensated prediction block from one or more reference blocks of a frame stored in DPB 64. Motion compensation unit 44 may also apply one or more interpolation filters to the prediction block to calculate sub-integer pixel values ​​for use in motion estimation.

[0059] Adder 62 adds the reconstructed residual block to the motion compensated prediction block generated by motion compensation unit 44 to generate a reference block for storage in DPB 64. The reference block may then be used as a prediction block by intra BC unit 48, motion estimation unit 42, and motion compensation unit 44 to inter-predict another video block in a subsequent video frame.

[0060] Figure 3 is a block diagram showing an exemplary video decoder 30 according to some embodiments of the present disclosure. The video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, and a DPB 92. The prediction processing unit 81 further includes a motion compensation unit 82, an intra-frame prediction unit 84, and an intra-frame BC unit 85. The video decoder 30 may perform the above-mentioned Figure 2 The encoding process is substantially the inverse of the decoding process described with respect to video encoder 20. For example, motion compensation unit 82 may generate prediction data based on motion vectors received from entropy decoding unit 80, and intra-prediction unit 84 may generate prediction data based on intra-prediction mode indicators received from entropy decoding unit 80.

[0061] In some examples, embodiments of the present disclosure may be dispersed in one or more of the units of the video decoder 30. For example, the intra BC unit 85 may perform embodiments of the present disclosure alone or in combination with other units of the video decoder 30, such as the motion compensation unit 82, the intra prediction unit 84, and the entropy decoding unit 80. In some examples, the video decoder 30 may not include the intra BC unit 85, and the functions of the intra BC unit 85 may be performed by other components of the prediction processing unit 81, such as the motion compensation unit 82.

[0062] Video data memory 79 may store video data, such as an encoded video bitstream, to be decoded by other components of video decoder 30. Video data stored in video data memory 79 may be obtained, for example, from storage device 32, a local video source (e.g., a camera), via a wired or wireless network communication of video data, or by accessing a physical data storage medium (e.g., a flash drive or hard disk).

[0063] During the decoding process, the video decoder 30 receives an encoded video bitstream representing video blocks of an encoded video frame and associated syntax elements. The entropy decoding unit 80 of the video decoder 30 entropy decodes the bitstream to generate quantization coefficients, motion vectors or intra-frame prediction mode indicators, and other syntax elements. The entropy decoding unit 80 then forwards the motion vectors or intra-frame prediction mode indicators, and other syntax elements to the prediction processing unit 81.

[0064] When a video frame is encoded as an intra-prediction coded (I) frame or for intra-coded prediction blocks in other types of frames, the intra-prediction unit 84 of the prediction processing unit 81 can generate prediction data for a video block of the current video frame based on the intra-prediction mode transmitted by the signal and reference data from a previously decoded block of the current frame.

[0065] When the video frame is encoded as an inter-frame prediction coded (i.e., B or P) frame, motion compensation unit 82 of prediction processing unit 81 generates one or more prediction blocks for the video block of the current video frame based on the motion vectors and other syntax elements received from entropy decoding unit 80.

[0066] In some examples, when a video block is encoded according to the intra BC mode described herein, intra BC unit 85 of prediction processing unit 81 generates a prediction block for the current video block based on the block vector and other syntax elements received from entropy decoding unit 80. The prediction block may be within a reconstructed region of the same picture as the current video block as defined by video encoder 20.

[0067] Motion compensation unit 82 and / or intra BC unit 85 determine prediction information for a video block of the current video frame by parsing motion vectors and other syntax elements, and then use the prediction information to generate a prediction block for the current video block being decoded.

[0068] Inverse quantization unit 86 inverse quantizes the quantized transform coefficients provided in the bitstream and entropy decoded by entropy decoding unit 80, using the same quantization parameters calculated by video encoder 20 for each video block in the video frame to determine the degree of quantization. Inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients to reconstruct the residual block in the pixel domain.

[0069] After the motion compensation unit 82 or the intra BC unit 85 generates a prediction block for the current video block based on the vector and other syntax elements, the adder 90 reconstructs the decoded video block for the current video block by adding the residual block from the inverse transform processing unit 88 to the corresponding prediction block generated by the motion compensation unit 82 and the intra BC unit 85. A loop filter 91 (e.g., a deblocking filter, an SAO filter, a CCSAO filter, and / or an ALF) may be located between the adder 90 and the DPB 92 to further process the decoded video block. In some examples, the loop filter 91 may be omitted.

[0070] In real-time video transmission scenarios such as live video broadcasting and remote conferencing, whether the bitrate fluctuation of the video stream after encoding is stable will greatly affect the user's playback experience. Frames with higher bitrates often take more time to decode. The drastic changes in bitrates will cause a huge difference in the decoding time of each frame, which will cause playback jams. In order to reduce the fluctuation of bitrate, a common method in the industry is to establish a virtual video buffer on the encoding side to simulate a series of operations from receiving the bitstream to decoding on the decoding side, thereby smoothing the output bitrate of the encoding side. This module is called a video buffer verifier (VBV).

[0071] VBV's bit rate control can be summarized as follows: Figure 4 The leaky bucket model shown has a constant input bit rate, which is the maximum bit rate set by the encoding end, and the output is the actual bit rate for encoding each frame. In order to ensure a smooth fluctuation of the bit rate, the number of bits in the VBV cache needs to be within a reasonable range, that is, between the overflow warning line and the underflow warning line, so as to avoid cache overflow and underflow. In the existing technical solution, the number of bits in the VBV cache after all the N+1 frames are encoded is estimated by obtaining the complexity of the subsequent N+1 frames in advance. If the number of bits in the cache exceeds the overflow warning line, the quantization parameter (QP) of the current frame needs to be lowered. On the contrary, if the number of bits in the cache is lower than the underflow warning line, the QP of the current frame needs to be increased.

[0072] The technical solutions and processes currently used by the open source software x264 / x265 are as follows Figure 5 As shown, first, the coding complexity and frame type (including I frame, P frame, B frame) of the current frame and the subsequent N frames need to be analyzed by the pre-analysis module. In real-time scenarios, a fixed frame type structure is often used, and the frame type of the subsequent frames can be easily obtained in the pre-analysis stage. The coding complexity is obtained by downsampling the source image by 1 / 2 and dividing it into 8×8 blocks. Pre-analysis is performed in units of blocks. The pre-analysis process includes calculating the intra-frame prediction loss (Cost) of each block. intra ), and inter-frame prediction loss (Cost inter), both are measured using the SATD (Sum of Absolute Transformed Difference) indicator. The coding complexity of the i-th frame is frameCost i It is the accumulation of the complexity of all m×n 8×8 blocks in the frame, which can be expressed by the following formula.

[0073]

[0074] After obtaining the coding complexity of all frames from the i-th frame to the i+N-th frame, the bit rate control module will first calculate the initial QP of the current frame according to the target bit rate or CRF (Constant Rate Factor) value set by the user, and convert it into Qscale. The conversion formula is as follows.

[0075]

[0076] After getting the initial Qscale of the i-th frame, you need to calculate the number of bits in the current VBV buffer BufferFill i-1 Next, the code rates of the i-th to i+N-th frames are predicted by the coding complexity and Qscale of the current frame (i.e., the i-th frame) and subsequent frames (i.e., the i+1-th to i+N-th frames), and the sum is calculated. The code rate prediction formula for the i-th frame is as follows.

[0077]

[0078] in, Indicates the number of predicted coding bits for the i-th frame (the bit rate is the number of bits transmitted per second, so the bit rate can be predicted). The Qscale of subsequent frames can be obtained based on the Qscale of the current frame and the fixed ratio of Qscale between different frame types. The Qscale ratio in x264 / x265 is as follows. The IPratio and PBratio here can be customized by the user, and the default values ​​are 1.40 and 1.30.

[0079]

[0080] Qsclae B =Qsclae P *PBratio

[0081] After obtaining the predicted bit rate of all frames in the entire sliding window (whose length is N+1), it is necessary to estimate the number of bits in the VBV cache after encoding these N+1 frames. The specific calculation formula is as follows.

[0082]

[0083] Among them, VBVMaxRate indicates the maximum bit rate of the input VBV, and FPS indicates the frame rate (Frames Per Second). The goal of bit rate control is to make the estimated number of bits in the buffer between the overflow warning line and the underflow warning line of VBV, otherwise the Qscale of the current frame needs to be adjusted. The adjustment method is as follows.

[0084]

[0085] in, Indicates the adjusted Qscale of the i-th frame. BufferSize is the capacity of the VBV buffer, that is, the maximum number of bits that can be accommodated. underflow And Th overflow The ratio thresholds of underflow warning line and overflow warning line are fixed in x264 / x265, set to 0.5 and 0.8 respectively. If the requirements are met, exit the loop and use the Qscale in this loop i Encode the i-th frame. Otherwise, Restart the next round of bit estimation and Qscale adjustment.

[0086] After encoding the i-th frame, it is necessary to use the Qscale (expressed as ) and the number of bits obtained (expressed as ) is used to update the bit rate prediction parameters (i.e., the parameters coeff and offset in the bit rate prediction formula). The specific updating method is as follows.

[0087]

[0088]

[0089] The subscript old of the bit rate prediction parameter indicates before updating, and the subscript new indicates after updating.

[0090] The above existing solutions share a set of bit rate prediction parameters and update mechanisms for different frames. Such a one-size-fits-all solution may affect the accuracy of bit rate prediction, so there is still room for optimization.

[0091] Figure 6 FIG. 2 is a flowchart showing an exemplary video encoding method according to some embodiments of the present disclosure. The video encoding method can be implemented in the video encoder 20 described above.

[0092] Reference Figure 6 , in step S601, a frame set is obtained.

[0093] The frame set includes the current frame and multiple subsequent frames, which are used to estimate the cache occupancy of a future event before encoding the current frame. The i-th to i+N-th frames in the existing scheme described above can be referred to, and can be acquired through a sliding window. As an example, the length N of the sliding window can be a fixed value or an adjustable value, which is not limited in the present disclosure. Of course, other reasonable methods other than sliding windows can also be used to acquire the frame set, which is also not limited in the present disclosure.

[0094] In step S602, a time domain correlation analysis is performed on the frame set to determine a correlation category of the frame set from a plurality of preset correlation categories.

[0095] Time domain correlation is the time correlation between adjacent frames. Multiple preset correlation categories are used to represent different correlation degrees of time domain correlation. Each preset correlation category in the multiple preset correlation categories has a corresponding rate prediction parameter. After analysis, it is found that different time domain complexities (that is, the degree of change in the time dimension) will have different effects on the final rate during encoding, and thus affect the accuracy of rate prediction. Based on this, by analyzing the time domain correlation of a frame set to classify the frame set accordingly, and configuring different rate prediction parameters for different correlation categories, the frame set can be reasonably classified, and the rate prediction can be performed in a differentiated manner in subsequent steps, which helps to improve the accuracy of rate prediction, improve coding efficiency and coding quality. As an example, multiple preset correlation categories include, but are not limited to, static categories, continuous categories, and complex categories, and the time domain correlation degrees represented by the three gradually decrease.

[0096] Regarding the execution of step S602, optionally, the operation of performing temporal correlation analysis on the frame set in step S602 includes: performing temporal correlation analysis on the frame set according to feature information of each frame in the frame set, wherein the feature information includes at least one of the following: intra-frame prediction loss, inter-frame prediction loss, length of motion vector, and texture complexity. As mentioned above, temporal correlation is the temporal correlation between adjacent frames. By acquiring feature information related thereto, such as but not limited to intra-frame prediction loss, inter-frame prediction loss, length of motion vector, and texture complexity, and analyzing them, temporal correlation analysis can be achieved. As an example, the smaller the intra-frame prediction loss, the more suitable it is to use the intra-frame prediction coding mode when encoding the frame set, which means that the lower the temporal correlation degree is, the higher the temporal complexity is. The inter-frame prediction loss is just the opposite of the intra-frame prediction loss. The larger its value is, the more suitable it is to use the intra-frame prediction coding mode when encoding the frame set, which means that the lower the temporal correlation degree is, the higher the temporal complexity is. The longer the motion vector is, or the greater the texture complexity is, the more intense the motion of each frame in the frame set is, and the more difficult it is to find the corresponding prediction block in the reference frame. This means that the temporal correlation is lower and the temporal complexity is higher, which is similar to the inter-frame prediction loss.

[0097] As an example, in the case where the feature information includes intra-frame prediction loss and inter-frame prediction loss, the above-mentioned operation of performing time domain correlation analysis on the frame set according to the feature information of each frame in the frame set includes: for each frame in the frame set, by comparing the intra-frame prediction loss with the inter-frame prediction loss, determining the correlation measure of each frame; determining the statistical value of the correlation measure of each frame in the frame set; by comparing the statistical value of the correlation measure with the statistical value threshold, and comparing the intra-frame prediction loss of the current frame with the intra-frame prediction loss threshold, performing time domain correlation analysis on the frame set. Considering that from the perspective of coding, whether it is intra-frame prediction loss or inter-frame prediction loss, the smaller the value, the better the coding effect, based on this, by comparing the intra-frame prediction loss and the inter-frame prediction loss, two feature information with opposite correlation relationships with the time domain correlation, a relative value can be obtained as a correlation measure to characterize whether the inter-frame prediction coding mode or the intra-frame prediction coding mode is more suitable at present, thereby reducing the influence of the absolute values ​​of the two losses on the time domain correlation analysis. On this basis, by introducing a statistical value threshold, a reference benchmark can be provided for the statistical value of the correlation measure (for example, including but not limited to the sum, mean, mode, median, etc.), so as to realize time domain correlation analysis. In addition, if the intra-frame prediction loss of the current frame itself is very large, it means that the texture inside the current frame is very complex. Then, even if the current frame is found to be more suitable for the inter-frame prediction coding mode by comparing the two losses, it may be difficult to find a suitable reference frame and reference block. In other words, the time domain correlation at this time may actually be low. Based on this, by additionally judging the intra-frame prediction loss of the current frame, specifically combining the intra-frame prediction loss threshold for comparative judgment, the analysis of the spatial complexity of the current frame can be increased, thereby optimizing the judgment of the time domain correlation and improving the analysis effect. For example, still taking the above multiple preset correlation categories including the static category, continuous category and complex category with gradually decreasing temporal correlation as an example, the ratio of inter-frame prediction loss to intra-frame prediction loss can be used as a correlation measure. At this time, the smaller the value of the correlation measure, the higher the temporal correlation. Two statistical value thresholds can be configured, respectively recorded as the first statistical value threshold and the second statistical value threshold, the former is greater than the latter, and an intra-frame prediction loss threshold is configured. When the statistical value of the correlation measure is less than the second statistical value threshold, it is determined to be a static category. When the statistical value of the correlation measure is greater than the first statistical value threshold or the intra-frame prediction loss of the current frame is greater than the intra-frame prediction loss threshold (that is, the spatial complexity of the current frame is too high, and it is not easy to find the reference frame and reference block), it is determined to be a complex category. In other cases, it is determined to be a continuous category. Of course, other reasonable methods can also be used to analyze the intra-frame prediction loss and inter-frame prediction loss of each frame in the frame set, and the present disclosure does not limit this.

[0098] As an example, when the feature information includes the length of the motion vector, two statistical value thresholds can be configured, which are respectively recorded as the first statistical value threshold and the second statistical value threshold. The former is greater than the latter. When the length of the motion vector is less than the second statistical value threshold, it is determined to be a stationary category, when it is greater than the first statistical value threshold, it is determined to be a complex category, and when it is between the first statistical value threshold and the second statistical value threshold, it is determined to be a continuous category. When the feature information includes texture complexity, it is similar, and the specific value of the indicating statistical value threshold needs to be adjusted accordingly. Of course, other reasonable methods can also be used to analyze the length of the motion vector or texture complexity, or even other feature information, of each frame in the frame set, and the present disclosure does not limit this.

[0099] Regarding the determination of the rate prediction parameters, optionally, the rate prediction parameters corresponding to each preset correlation category in the multiple preset correlation categories are obtained by the following steps: for each preset correlation category in the multiple preset correlation categories, multiple sample frame sets of the preset correlation category are obtained; the multiple sample frame sets are respectively encoded using preset quantization parameters to obtain multiple groups of target coded data, wherein the target coded data includes coded data related to the rate prediction parameters; the multiple groups of target coded data are fitted to obtain the rate prediction parameters corresponding to the preset correlation category. By obtaining a sample frame set of a specific preset correlation category and encoding it using preset quantization parameters, the target coded data for actually encoding these sample frame sets can be obtained, and the rate prediction parameters are specifically used to predict these target coded data (for example, including but not limited to the Qscale, coding complexity, and actual bit rate introduced above), so the corresponding rate prediction parameters can be reversely fitted to achieve the determination of the rate prediction parameters.

[0100] As an example, a sample video may be obtained first, and based on each frame in the sample video, a corresponding plurality of sample frame sets may be obtained, and then the operation of step S602 may be performed on each of the sample frame sets to determine the correlation category of each sample frame set, and then the sample frame sets may be grouped according to the corresponding correlation category, and then the bit rate prediction parameters may be calculated for each group of sample frame sets. It should be understood that when the number of sample frame sets in a certain correlation category is insufficient, new sample videos may continue to be obtained.

[0101] As an example, the number of preset quantization parameters can be multiple, so that different quantization parameters are used for encoding, so that each sample frame set can obtain multiple sets of target encoding data. Accordingly, when performing fitting processing on multiple sets of target encoding data, it can be specifically firstly fitting multiple sets of target encoding data of the same sample frame set to obtain the fitting results of the sample frame set (specifically the fitted rate prediction parameters), and then statistical processing is performed on the fitting results of the multiple sample frame sets, for example, the average value of each rate prediction parameter is calculated to obtain the rate prediction parameter of the preset correlation category.

[0102] Further optionally, the multiple sample frame sets include frame sets in sample videos of preset video categories. By using sample videos of preset video categories and extracting sample frame sets therefrom, it is possible to use more representative sample videos in a targeted manner based on the encoding requirements of specific scenes corresponding to the preset video categories, thereby obtaining better encoding effects in specific scenes. As an example, the video cache verifier (VBV) mentioned above is usually used in real-time video transmission scenarios such as live video broadcasting and remote conferencing. Based on this, for live video broadcasting scenarios, live screen recording videos can be selected as sample videos, or for remote conferencing scenarios, remote conference recording videos can be selected as sample videos. The present disclosure does not limit this.

[0103] In step S603, the bit rate of the frame set is predicted according to the bit rate prediction parameters corresponding to the correlation category of the frame set to obtain the predicted bit rate of the frame set.

[0104] In step S604, a quantization parameter is determined according to the predicted bit rate of the frame set.

[0105] In step S605, the current frame is encoded using the quantization parameter.

[0106] When the specific bit rate prediction parameters are determined, the bit rate prediction, quantization parameter determination and encoding performed in step S603 to step S605 may refer to the existing solutions and will not be described in detail here.

[0107] Next, an exemplary video encoding method according to the present disclosure is further introduced.

[0108] In some embodiments, optionally, the exemplary video encoding method according to the present disclosure further includes: updating the rate prediction parameters corresponding to the correlation category of the frame set according to the quantization parameter and the actual bit rate for encoding the current frame, wherein the updated rate prediction parameters are applied to other frames encoded after the current frame in the current video. By updating the corresponding rate prediction parameters according to the actual encoded quantization parameter and the actual bit rate for the corresponding correlation category, the rate prediction parameters can be continuously optimized, thereby continuously improving the accuracy of rate prediction, and improving encoding efficiency and encoding quality. It should be understood that the rate prediction parameters corresponding to each preset correlation category obtained by fitting the sample frame set described above can be used as the initial rate prediction parameters of each preset correlation category. When encoding a frame of a preset correlation category in the currently encoded video, the initial rate prediction parameters of the preset correlation category are used for rate prediction, and the rate prediction parameters can be updated as needed. When encoding other frames of the subsequent preset correlation category, the updated latest rate prediction parameters of the preset correlation category are used for rate prediction, and can continue to be updated. It should also be understood that, in addition to being updated accordingly according to different relevance categories, the bit rate prediction parameters are only applied to the current video, which means that when encoding another video, the initial bit rate prediction parameters need to be reused because the video content has changed, but the bit rate prediction parameters can also continue to be updated.

[0109] Further optionally, each preset correlation category in the plurality of preset correlation categories has a corresponding plurality of groups of rate prediction parameters, wherein the plurality of groups of rate prediction parameters correspond to the plurality of frame types one by one; accordingly, step S603 comprises: for each frame in the frame set, according to the correlation category of the frame set and the rate prediction parameters corresponding to the frame type of each frame, rate prediction is performed on each frame respectively, so as to perform rate prediction on the frame set; the above-mentioned step of updating the rate prediction parameters corresponding to the correlation category of the frame set according to the quantization parameter and the actual bit rate of encoding the current frame comprises: updating the rate prediction parameters corresponding to the correlation category of the frame set and the frame type of the current frame according to the quantization parameter and the actual bit rate of encoding the current frame. By using different rate prediction parameters for different correlation categories respectively, different rate prediction parameters are further used for different frame types under the same correlation category to perform rate prediction, and differentiated rate prediction parameter updates are performed, that is, the rate prediction parameters of the frame type of the current frame are updated only according to the actual encoding situation of the current frame, so that more detailed and accurate rate prediction can be achieved. It should be understood that, still taking the above multiple preset correlation categories including the three categories of static category, continuous category and complex category as an example, since the number of frame types is also three, namely, I frame, P frame, B frame, a total of 9 groups of rate prediction parameters can be formed. As an example, different frame types under the same preset correlation category can use the same initial rate prediction parameters, and then update the rate prediction parameters according to different frame types. Of course, different initial rate prediction parameters can also be used for different frame types, and the present disclosure does not limit this.

[0110] Next, combine Figure 7 A video encoding method according to a specific embodiment of the present disclosure is introduced.

[0111] like Figure 7 As shown, in this specific embodiment, when the encoder starts encoding the current frame, it will calculate the inter-frame prediction loss Cost of the current frame (i-th frame) and the subsequent frames (n subsequent frames). intra And intra-frame prediction loss Cost inter The time domain correlation ratio δ of the current frame and subsequent frames is calculated (as the statistical value of the correlation measure).

[0112]

[0113] According to the calculated cumulative time domain correlation ratio δ, the specific classification method is as follows.

[0114]

[0115] Among them, the value of f(δ) can be 0, 1, and 2, which respectively represent that the frame set of the current sliding window is classified into three time domain correlation categories: static, continuous, and complex. mΔn is the number of 8x8 blocks that the current frame can be divided into, τ1 and τ2 correspond to the first statistical value threshold and the second statistical value threshold introduced in the previous article, and (m×n)×T corresponds to the intra-frame prediction loss threshold introduced in the previous article. Through experiments, τ1, τ2 and T are set to 0.6, 0.3 and 2 respectively. 11 .

[0116] After obtaining the category to which the frame set belongs, the predicted coding bit number bits of each frame in the frame set is calculated according to the following formula, where the subscript δ is used to indicate the correlation category to which the frame set belongs, and the superscript type is used to indicate the frame type of the frame for which the predicted coding bit number needs to be calculated.

[0117]

[0118] After obtaining the predicted number of coding bits for the subsequent frames, the following algorithm is similar to the existing scheme described above, estimating the number of bits in the VBV buffer after encoding the N+1 frame in the frame set.

[0119]

[0120] After encoding the current frame, according to the actual number of bits encoded in the current frame and To update the rate prediction parameters, and only update the rate prediction parameters of the correlation category shown in the frame set and the frame type of the current frame and That is, the parameter updates of each correlation category and frame type are independent of each other, which reduces the mutual influence of the parameter update processes of different correlation categories and frame types and improves the prediction accuracy.

[0121]

[0122]

[0123] For different relevance categories, and After fitting the initial value, the parameters can be updated for different frame types. Here, the first 200 frames of each sequence in the HEVC standard sequence are selected as sample frames, and the corresponding sample frame sets are determined respectively. Then, the time domain correlation analysis is performed to determine the correlation category, and then the four QP points of 22, 27, 32, and 37 are selected to encode these 200 frames respectively. The initial value fitted by the bit rate prediction formula is as follows.

[0124]

[0125] This specific embodiment innovatively proposes a cache check rate control method based on time domain correlation. It uses the information obtained from the pre-analysis of the current frame and its subsequent frames to classify the frame set corresponding to the frame to be encoded according to the time domain correlation, and fits different initial rate prediction parameters to different correlation categories, making the rate prediction process more accurate. This specific embodiment also innovatively designs a rate prediction parameter update mechanism based on correlation category and frame type. After the current frame is encoded, the rate prediction parameters are updated according to the correlation category and frame type to which it belongs, thereby reducing the error of rate prediction and improving encoding performance. After testing, this specific embodiment can be used in the BD-rate ( -Delta rate, Bjorntgaard-Delta rate) obtained a PSNR (Peak Signal-to-Noise Rate) decrease of about 0.357%, a SSIM (Structural Similarity Index) decrease of about 0.260%, and almost no change in encoding time.

[0126] Figure 8 is a block diagram illustrating a video encoding apparatus according to an exemplary embodiment of the present disclosure.

[0127] Reference Figure 8 The video encoding device 800 includes an acquisition unit 801, an analysis unit 802, a prediction unit 803, a determination unit 804, and an encoding unit 805.

[0128] The acquisition unit 801 may acquire a frame set, wherein the frame set includes a current frame and a plurality of subsequent frames.

[0129] The analysis unit 802 can perform time domain correlation analysis on the frame set to determine the correlation category of the frame set from multiple preset correlation categories, wherein the time domain correlation is the time correlation between adjacent frames, and the multiple preset correlation categories are used to represent different correlation degrees of the time domain correlation, and each preset correlation category in the multiple preset correlation categories has a corresponding bit rate prediction parameter.

[0130] The prediction unit 803 may perform bit rate prediction on the frame set according to the bit rate prediction parameter corresponding to the correlation category of the frame set to obtain the predicted bit rate of the frame set.

[0131] The determination unit 804 may determine a quantization parameter according to the predicted bit rate of the frame set.

[0132] The encoding unit 805 may encode the current frame using the quantization parameter.

[0133] Optionally, the analysis unit 802 may also: perform time domain correlation analysis on the frame set according to feature information of each frame in the frame set, wherein the feature information includes at least one of the following: intra-frame prediction loss, inter-frame prediction loss, length of motion vector, and texture complexity.

[0134] Optionally, when the feature information includes intra-frame prediction loss and inter-frame prediction loss, the analysis unit 802 may also: determine the correlation measure of each frame in the frame set by comparing the intra-frame prediction loss with the inter-frame prediction loss; determine the statistical value of the correlation measure of each frame in the frame set; perform time domain correlation analysis on the frame set by comparing the statistical value of the correlation measure with a statistical value threshold and comparing the intra-frame prediction loss of the current frame with the intra-frame prediction loss threshold.

[0135] Optionally, the video encoding device 800 also includes an updating unit (not shown in the figure), which can update the rate prediction parameters corresponding to the correlation category of the frame set according to the quantization parameter and the actual bit rate for encoding the current frame, wherein the updated rate prediction parameters are applied to other frames in the current video that are encoded after the current frame.

[0136] Optionally, each of the multiple preset correlation categories has corresponding multiple groups of rate prediction parameters, wherein the multiple groups of rate prediction parameters correspond one-to-one to the multiple frame types; the prediction unit 803 may also perform rate prediction on each frame in the frame set according to the correlation category of the frame set and the rate prediction parameters corresponding to the frame type of each frame, so as to perform rate prediction on the frame set; the update unit may also update the rate prediction parameters corresponding to the correlation category of the frame set and the frame type of the current frame according to the quantization parameter and the actual bit rate for encoding the current frame.

[0137] Optionally, the rate prediction parameters corresponding to each preset correlation category in multiple preset correlation categories are obtained through the following steps: for each preset correlation category in the multiple preset correlation categories, multiple sample frame sets of the preset correlation category are obtained; the multiple sample frame sets are respectively encoded using preset quantization parameters to obtain multiple groups of target encoding data, wherein the target encoding data includes encoding data related to the rate prediction parameters; and the multiple groups of target encoding data are fitted to obtain the rate prediction parameters corresponding to the preset correlation categories.

[0138] Optionally, the multiple sample frame sets include frame sets in sample videos of a preset video category.

[0139] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0140] Fig. 9 A computing environment 910 is shown coupled to a user interface 950. The computing environment 910 may be part of a data processing server. The computing environment 910 includes a processor 920, a memory 930, and an input / output (I / O) interface 940.

[0141] The processor 920 generally controls the overall operation of the computing environment 910, such as operations associated with display, data acquisition, data communication, and image processing. The processor 920 may include one or more processors for executing instructions to perform all or some steps in the above method. In addition, the processor 920 may include one or more modules that facilitate the interaction between the processor 920 and other components. The processor may be a central processing unit (CPU), a microprocessor, a single-chip microcomputer, a graphics processing unit (GPU), etc.

[0142] The memory 930 is configured to store various types of data to support the operation of the computing environment 910. The memory 930 may include predetermined software 932. Examples of such data include instructions for any application or method operating on the computing environment 910, video data sets, image data, etc. The memory 930 may be implemented by using any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0143] The I / O interface 940 provides an interface between the processor 920 and peripheral interface modules (e.g., keyboard, click wheel, buttons, etc.). The buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. The I / O interface 940 may be coupled to an encoder and a decoder.

[0144] In an embodiment, a non-transitory computer-readable storage medium including, for example, a plurality of programs in a memory 930 and / or storing a bit stream generated by the above encoding method or a bit stream to be decoded by the above decoding method is also provided, wherein the plurality of programs can be executed by a processor 920 in a computing environment 910 to perform the above method. In one example, the plurality of programs can be executed by a processor 920 in a computing environment 910 to (for example, from Figure 2The video encoder 20 in the computing environment 910 receives a bit stream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or one or more associated syntax elements, etc.), and can also be executed by the processor 920 in the computing environment 910 to perform the above-mentioned decoding method according to the received bit stream or data stream. In another example, the multiple programs can be executed by the processor 920 in the computing environment 910 to perform the above-mentioned encoding method to encode the video information (e.g., video blocks representing video frames, and / or one or more associated syntax elements, etc.) into a bit stream or data stream, and can also be executed by the processor 920 in the computing environment 910 to (e.g., to Figure 3 Alternatively, a non-transitory computer-readable storage medium may store the bit stream or data stream generated by the encoder (e.g., Figure 2 The video encoder 20 in FIG. 1 generates a video signal for a decoder (eg, Figure 3 A bitstream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or associated one or more syntax elements, etc.) used by the video decoder 30 in the video decoder 30 when decoding video data. The non-transitory computer-readable storage medium can be, for example, a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0145] In an embodiment, a bit stream generated by the above encoding method or a bit stream to be decoded by the above decoding method is provided. In an embodiment, a bit stream including encoded video information generated by the above encoding method or encoded video information to be decoded by the above decoding method is provided.

[0146] In an embodiment, a computing device is also provided, which includes: one or more processors (e.g., processor 920); and a non-transitory computer-readable storage medium or memory 930 in which multiple programs that can be executed by the one or more processors are stored, wherein the one or more processors are configured to perform the above-mentioned method when executing the multiple programs.

[0147] In an embodiment, a computer program product having instructions for storing or transmitting a bitstream is also provided, wherein the bitstream includes encoded video information generated by the above encoding method or encoded video information to be decoded by the above decoding method. In an embodiment, a computer program product including, for example, a plurality of programs in a memory 930 is also provided, wherein the plurality of programs can be executed by a processor 920 in a computing environment 910 for performing the above method. For example, the computer program product may include a non-transitory computer-readable storage medium.

[0148] In an embodiment, the computing environment 910 may be implemented by one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs, GPUs, controllers, microcontrollers, microprocessors, or other electronic components for performing the above methods.

[0149] In an embodiment, a method for storing a bit stream is also provided, comprising: storing the bit stream on a digital storage medium, wherein the bit stream comprises encoded video information generated by the above encoding method or encoded video information to be decoded by the above decoding method.

[0150] In an embodiment, a method for transmitting a bit stream generated by the above encoder is also provided. In an embodiment, a method for receiving a bit stream to be decoded by the above decoder is also provided.

[0151] The description of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or limited to the present disclosure. Many modifications, variations, and alternative embodiments will be apparent to one of ordinary skill in the art having the benefit of the teachings presented in the foregoing description and the associated drawings.

[0152] Unless otherwise specifically stated, the order of steps of the method according to the present disclosure is intended to be illustrative only, and the steps of the method according to the present disclosure are not limited to the order specifically described above, but can be changed according to actual conditions. In addition, at least one of the steps of the method according to the present disclosure can be adjusted, combined or deleted according to actual needs.

[0153] The examples are chosen and described in order to explain the principles of the present disclosure and to enable other persons skilled in the art to understand the various embodiments of the present disclosure and to best utilize the basic principles and various embodiments with various modifications suitable for the specific use contemplated. Therefore, it will be understood that the scope of the present disclosure is not limited to the specific examples of the disclosed embodiments, and modifications and other embodiments are intended to be included within the scope of the present disclosure.

Claims

1. A video encoding method, characterized in that: The video encoding method comprises: Acquire a frame set, wherein the frame set includes a current frame and a plurality of subsequent frames; Performing a time domain correlation analysis on the frame set to determine a correlation category of the frame set from a plurality of preset correlation categories, wherein the time domain correlation is a time correlation between adjacent frames, the plurality of preset correlation categories are used to represent different correlation degrees of the time domain correlation, and each of the plurality of preset correlation categories has a corresponding bit rate prediction parameter; Performing bit rate prediction on the frame set according to a bit rate prediction parameter corresponding to the correlation category of the frame set to obtain a predicted bit rate of the frame set; Determining a quantization parameter according to the predicted bit rate of the frame set; The current frame is encoded using the quantization parameter.

2. The video encoding method according to claim 1, characterized in that: The performing time domain correlation analysis on the frame set includes: According to feature information of each frame in the frame set, a time domain correlation analysis is performed on the frame set, wherein the feature information includes at least one of the following: intra-frame prediction loss, inter-frame prediction loss, length of motion vector, and texture complexity.

3. The video encoding method according to claim 2, characterized in that: In a case where the feature information includes the intra-frame prediction loss and the inter-frame prediction loss, performing a time domain correlation analysis on the frame set according to the feature information of each frame in the frame set includes: For each frame in the set of frames, determining a correlation metric for each frame by comparing the intra-frame prediction loss with the inter-frame prediction loss; Determining a statistical value of a correlation metric for each frame in the set of frames; The frame set is subjected to a temporal correlation analysis by comparing the statistical value of the correlation measure with a statistical value threshold and comparing the intra-frame prediction loss of the current frame with an intra-frame prediction loss threshold.

4. The video encoding method according to any one of claims 1 to 3, characterized in that: The video encoding method further comprises: The rate prediction parameters corresponding to the correlation category of the frame set are updated according to the quantization parameter and the actual bit rate for encoding the current frame, wherein the updated bit rate prediction parameters are applied to other frames in the current video that are encoded after the current frame.

5. The video encoding method according to claim 4, characterized in that: Each of the plurality of preset correlation categories has a corresponding plurality of groups of rate prediction parameters, wherein the plurality of groups of rate prediction parameters correspond one-to-one to a plurality of frame types; The performing bit rate prediction on the frame set according to the bit rate prediction parameter corresponding to the correlation category of the frame set includes: For each frame in the frame set, according to the correlation category of the frame set and the rate prediction parameter corresponding to the frame type of each frame, perform rate prediction on each frame respectively, so as to perform rate prediction on the frame set; The updating of the bit rate prediction parameter corresponding to the correlation category of the frame set according to the quantization parameter and the actual bit rate for encoding the current frame includes: The bit rate prediction parameter corresponding to the correlation category of the frame set and the frame type of the current frame is updated according to the quantization parameter and the actual bit rate of encoding the current frame.

6. The video encoding method according to any one of claims 1 to 3, characterized in that: The bit rate prediction parameter corresponding to each preset correlation category in the plurality of preset correlation categories is obtained by the following steps: For each preset correlation category among the plurality of preset correlation categories, acquiring a plurality of sample frame sets of the preset correlation category; Using preset quantization parameters to encode the plurality of sample frame sets respectively, to obtain a plurality of groups of target coded data, wherein the target coded data includes coded data related to the bit rate prediction parameter; Fitting processing is performed on the multiple groups of target coding data to obtain code rate prediction parameters corresponding to the preset correlation categories.

7. The video encoding method according to claim 6, characterized in that: The plurality of sample frame sets include frame sets in sample videos of a preset video category.

8. A video encoding device, characterized in that: The video encoding device comprises: An acquisition unit is configured to acquire a frame set, wherein the frame set includes a current frame and a plurality of subsequent frames; an analyzing unit, configured to perform a time domain correlation analysis on the frame set to determine a correlation category of the frame set from a plurality of preset correlation categories, wherein the time domain correlation is a time correlation between adjacent frames, the plurality of preset correlation categories are used to represent different correlation degrees of the time domain correlation, and each of the plurality of preset correlation categories has a corresponding bit rate prediction parameter; A prediction unit, configured to perform bit rate prediction on the frame set according to a bit rate prediction parameter corresponding to a correlation category of the frame set to obtain a predicted bit rate of the frame set; A determination unit, configured to determine a quantization parameter according to a predicted bit rate of the frame set; The encoding unit is configured to encode the current frame using the quantization parameter.

9. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, When the computer executable instructions are executed by at least one processor, the at least one processor is prompted to execute the video encoding method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to perform the video encoding method according to any one of claims 1 to 7.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by at least one processor, the video encoding method according to any one of claims 1 to 7 is implemented.

12. A method for generating a bit stream, characterized in that: include: A bit stream is generated according to the video encoding method according to any one of claims 1 to 7.

Images