Video coding method and device, bit stream generation method, electronic equipment, storage medium and program product

Abstract

The invention relates to a video coding method and device, a bit stream generation method, electronic equipment, a storage medium and a program product, and the video coding method comprises the steps: segmenting a to-be-coded video, and obtaining a plurality of video segments; for each video segment, determining a segment coding rate of each video segment according to the coding complexity of each video segment and the coding complexity of the video to be coded; and coding the video frame in each video segment based on the segment coding rate of each video segment. The problem that subjective image quality experience is affected due to the fact that code rate distribution of an encoder is not ideal is solved, the encoding complexity of each video segment can be analyzed, and the encoding code rate can be distributed to each video segment, so that the encoding code rate of each video segment can be reasonably distributed, the video encoding quality is improved, and the subjective image quality experience of the video is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of video technology, and in particular to a video encoding method and apparatus, a bit stream generation method, an electronic device, a storage medium, and a program product. Background Art

[0002] Various electronic devices (e.g., digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smart phones, video teleconferencing devices, video streaming devices, etc.) support digital video. Electronic devices send and receive or otherwise transmit digital video data through a communication network, and / or store digital video data on a storage device. Due to the limited bandwidth capacity of the communication network and the limited storage resources of the storage device, before the video data is transmitted or stored, the video data can be compressed using a video codec according to one or more video codec standards. For example, video codec standards include general video codec (VVC), joint exploration test model (JEM), high efficiency video codec (HEVC / H.265), advanced video codec (AVC / H.264), moving picture expert group (MPEG) codec, etc. Video codecs generally use prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that utilize the inherent redundancy in video data. Video codecs are intended to compress video data into a form that uses a lower bit rate while avoiding or minimizing the degradation of video quality.

[0003] In the field of video encoding and decoding, the bit rate allocation strategies of some existing mainstream hardware encoders are not ideal, which affects the subjective picture quality experience and brings higher transmission bandwidth costs.

[0004] In addition, the hardware encoder can only be configured and used through the official peripheral interface. The internal implementation mechanism is in a black box state, with limited adjustable parameters, low algorithm flexibility and accuracy, and unable to distinguish different scenes in the video for bit rate allocation. Summary of the invention

[0005] The present disclosure provides a video encoding method and apparatus, a bit stream generation method, an electronic device, a storage medium and a program product, so as to at least solve the problem in the related art that the encoder does not allocate the bit rate well and affects the subjective picture quality experience. The technical solution of the present disclosure is as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, a video encoding method is provided, the video encoding method comprising: dividing a video to be encoded to obtain a plurality of video segments; for each video segment, determining a segment encoding bit rate of each video segment according to the encoding complexity of each video segment and the encoding complexity of the video to be encoded; and encoding video frames in each video segment based on the segment encoding bit rate of each video segment.

[0007] Optionally, the segmenting of the video to be encoded to obtain a plurality of video segments includes: determining an intra-frame complexity of a first video frame in the video to be encoded and an inter-frame complexity of the first video frame relative to a second video frame, wherein the second video frame is before the first video frame; determining whether the first video frame is a transition frame based on the inter-frame complexity and the intra-frame complexity of the first video frame; in response to the first video frame being a transition frame, segmenting the video to be encoded based on the transition frame to obtain the plurality of video segments.

[0008] Optionally, determining whether the first video frame is a transition frame based on the inter-frame complexity and intra-frame complexity of the first video frame includes: in response to a complexity difference between the inter-frame complexity and the intra-frame complexity of the first video frame exceeding a difference threshold, determining that the first video frame is a transition frame.

[0009] Optionally, for each video segment, the segmented encoding bit rate of each video segment is determined according to the encoding complexity of each video segment and the encoding complexity of the video to be encoded, including: according to the encoding complexity of the video segment and the encoding complexity of the video to be encoded, adjusting the original encoding bit rate of the video to be encoded to determine the segmented encoding bit rate of each video segment.

[0010] Optionally, the adjusting the original encoding bit rate of the video to be encoded to determine the segmented encoding bit rate of each video segment includes: adjusting the original encoding bit rate of the video to be encoded to determine the segmented encoding bit rate of each video segment according to the encoding complexity of the video segment, the encoding complexity of the video to be encoded, and an offset coefficient, wherein the offset coefficient represents the degree of deviation of the average encoding bit rate of the video to be encoded compared to the original encoding bit rate after the encoding bit rate of each video segment is adjusted.

[0011] Optionally, the step of adjusting the original encoding bit rate of the video to be encoded according to the encoding complexity of the video segment, the encoding complexity of the video to be encoded, and the offset coefficient to determine the segmented encoding bit rate of each video segment includes: determining a first value according to the encoding complexity of all video frames in the current video segment, the encoding complexity of all video frames in the video to be encoded, and the offset coefficient; and determining the segmented encoding bit rate of each video segment based on the first value and the original encoding bit rate.

[0012] Optionally, the coding complexity of the video segment is determined in the following manner: determining the coding complexity of each video frame in the video segment; determining the normalized complexity of each video frame based on the minimum complexity and the maximum complexity of the coding complexity of each video frame in the video segment, a first coefficient and a second coefficient, wherein the relationship between the first coefficient and the second coefficient is used to control the difference between the segmented coding bit rates of each video segment; determining the coding complexity of the video segment based on the normalized complexity of each video frame in the video segment.

[0013] Optionally, the video encoding method further comprises: in response to an actual encoding codeword of the current video segment being smaller than a predicted encoding codeword of the current video segment, increasing a segment encoding bitrate of a video segment subsequent to the current video segment.

[0014] Optionally, increasing the segmented encoding bit rate of the video segment after the current video segment includes: determining a remaining codeword based on a difference between an actual encoding codeword of the current video segment and a predicted encoding codeword of the current video segment; determining a bit rate offset for the subsequent video segment based on the remaining codeword, the number of subsequent video frames to be encoded in the video to be encoded, and the number of frames of the subsequent video segment; and obtaining an increased segmented encoding bit rate for the subsequent video segment based on the bit rate offset and the segmented encoding bit rate of the subsequent video segment.

[0015] According to a second aspect of an embodiment of the present disclosure, a video encoding device is provided, comprising: a segmentation unit, configured to segment a video to be encoded to obtain a plurality of video segments; a determination unit, configured to determine, for each video segment, a segment encoding bit rate of each video segment according to the encoding complexity of each video segment and the encoding complexity of the video to be encoded; and an encoding unit, configured to encode video frames in each video segment based on the segment encoding bit rate of each video segment.

[0016] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions, wherein the processor-executable instructions, when executed by the processor, enable the processor to execute the video encoding method according to the embodiment of the present disclosure.

[0017] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the video encoding method according to the embodiment of the present disclosure.

[0018] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the video encoding method according to the embodiment of the present disclosure.

[0019] According to a sixth aspect of an embodiment of the present disclosure, a method for generating a bitstream is provided, comprising: generating a bitstream according to the video encoding method described in an embodiment of the present disclosure.

[0020] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, having instructions for storing a bit stream, wherein the bit stream includes: video data generated by the video encoding method according to the embodiment of the present disclosure.

[0021] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0022] By dividing the video to be encoded, multiple video segments are obtained, and the segment encoding bit rate of each video segment is determined according to the encoding complexity of each video segment and the encoding complexity of the video to be encoded, so as to encode the video frames in the corresponding video segment according to the segment encoding bit rate. In this way, by analyzing the encoding complexity of each video segment, the encoding bit rate can be allocated to each video segment, so that the encoding bit rate of each video segment can be reasonably allocated, the video encoding quality can be improved, and the subjective picture quality experience of the video can be improved.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0025] Figure 1 is a block diagram showing an exemplary system for encoding and decoding video blocks according to an exemplary embodiment of the present disclosure.

[0026] Figure 2 is a block diagram showing an exemplary video encoder according to an exemplary embodiment of the present disclosure.

[0027] Figure 3 is a block diagram showing an exemplary video decoder according to an exemplary embodiment of the present disclosure.

[0028] Figure 4 It is a schematic diagram of an example implementation scenario of a video encoding method according to an exemplary embodiment of the present disclosure.

[0029] Figure 5 The flowchart of a video encoding method is shown according to an exemplary embodiment of the present disclosure.

[0030] Figure 6 The present invention is a flowchart showing the steps of obtaining multiple video segments in a video encoding method according to an exemplary embodiment of the present disclosure.

[0031] Figure 7 It is a flowchart of a video encoding method according to an exemplary embodiment of the present disclosure.

[0032] Figure 8 The present invention is a flowchart showing the steps of coding complexity of video segments in a video coding method according to an exemplary embodiment of the present disclosure.

[0033] Fig. 9 The flowchart is a step of increasing the segment encoding bit rate of a video segment after a current video segment in a video encoding method according to an exemplary embodiment of the present disclosure.

[0034] Fig.10 It is a schematic diagram comparing image quality before and after bit rate reallocation in an application example of a video encoding method according to an exemplary embodiment of the present disclosure.

[0035] Fig.11 is a block diagram showing a video encoding device according to an exemplary embodiment of the present disclosure.

[0036] Fig.12 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure.

[0037] Fig.13 is a diagram of a computing environment coupled with a user interface according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0038] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0039] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0040] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0041] Figure 1 is a block diagram illustrating an exemplary system 10 for encoding and decoding video blocks in parallel according to some embodiments of the present disclosure. Figure 1 As shown in , system 10 includes a source device 12 that generates and encodes video data to be later decoded by a target device 14. Source device 12 and target device 14 may include any of a wide variety of electronic devices, including cloud servers, server computers, desktop or laptop computers, tablet computers, smart phones, set-top boxes, digital televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some implementations, source device 12 and target device 14 are equipped with wireless communication capabilities.

[0042] In some implementations, target device 14 may receive the encoded video data to be decoded via link 16. Link 16 may include any type of communication medium or device capable of moving the encoded video data from source device 12 to target device 14.

[0043] In some other embodiments, the encoded video data may be sent from the output interface 22 to the storage device 32. The encoded video data in the storage device 32 may then be accessed by the target device 14 via the input interface 28.

[0044] like Figure 1, source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Video source 18 may include sources such as, or a combination of such sources: a video capture device (e.g., a camera), a video archive containing previously captured video, a video feed interface for receiving video from a video content provider, and / or a computer graphics system for generating computer graphics data as source video.

[0045] The captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video data may be sent directly to target device 14 via output interface 22 of source device 12. The encoded video data may also (or alternatively) be stored on storage device 32 for later access by target device 14 or other devices for decoding and / or playback.

[0046] Target device 14 includes input interface 28, video decoder 30, and display device 34. Input interface 28 may include a receiver and / or a modem and receives encoded video data via link 16. The encoded video data transmitted via link 16 or provided on storage device 32 may include various syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements may be included within the encoded video data sent over a communication medium, stored on a storage medium, or stored on a file server.

[0047] The video encoder 20 and the video decoder 30 can operate according to a proprietary standard or an industry standard (e.g., VVC, HEVC, Part 10 of MPEG-4, AVC) or an extension of such a standard. It should be understood that the present application is not limited to a specific video encoding / decoding standard and can be applied to other video encoding / decoding standards. It is generally believed that the video encoder 20 of the source device 12 can be configured to encode the video data according to any of these current standards or future standards. Similarly, it is also generally believed that the video decoder 30 of the target device 14 can be configured to decode the video data according to any of these current standards or future standards.

[0048] The video encoder 20 and the video decoder 30 may be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic devices, software, hardware, firmware, or any combination thereof. When partially implemented in software, the electronic device may store instructions for the software in a suitable non-transitory computer-readable medium, and use one or more processors to execute the instructions in the hardware to perform the video encoding / decoding operations disclosed in the present disclosure. Each of the video encoder 20 and the video decoder 30 may be included in one or more encoders or decoders, and either of the encoders or decoders may be integrated as part of a combined encoder / decoder (CODEC) in the corresponding device.

[0049] Figure 2 is a block diagram illustrating an exemplary video encoder 20 according to some embodiments described in the present application. The video encoder 20 can perform intra-frame prediction coding and inter-frame prediction coding on video blocks within a video frame. Intra-frame prediction coding relies on spatial prediction to reduce or remove spatial redundancy in video data within a given video frame or picture. Inter-frame prediction coding relies on temporal prediction to reduce or remove temporal redundancy in video data within adjacent video frames or pictures of a video sequence. It should be noted that in the field of video coding and decoding, the term "frame" can be used as a synonym for the term "image" or "picture".

[0050] like Figure 2As shown in , the video encoder 20 includes a video data memory 40, a prediction processing unit 41, a decoded picture buffer (DPB) 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The prediction processing unit 41 further includes a motion estimation unit 42, a motion compensation unit 44, a segmentation unit 45, an intra-frame prediction processing unit 46, and an intra-frame block copy (BC) unit 48. In some embodiments, the video encoder 20 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and an adder 62 for video block reconstruction. A loop filter 63 such as a deblocking filter can be located between the adder 62 and the DPB 64 to filter the block boundary to remove the block effect from the reconstructed video. In addition to the deblocking filter, another loop filter (e.g., a sample adaptive offset (SAO) filter, a cross-component sample adaptive offset (CCSAO) filter, and / or an adaptive loop filter (ALF)) can also be used to filter the output of the adder 62. In some examples, the loop filter may be omitted, and the decoded video blocks may be provided directly by adder 62 to DPB 64. Video encoder 20 may take the form of a fixed or programmable hardware unit, or may be distributed among one or more of the illustrated fixed or programmable hardware units.

[0051] Video data memory 40 may store video data to be encoded by components of video encoder 20. Figure 1 Video source 18 is shown obtaining video data in video data memory 40. DPB 64 is a buffer that stores reference video data (eg, reference frames or pictures) for use by video encoder 20 in encoding the video data (eg, in intra or inter prediction coding modes).

[0052] like Figure 2 As shown in , after receiving the video data, the segmentation unit 45 within the prediction processing unit 41 segments the video data into video blocks. This segmentation may also include segmenting the video frame into strips, tiles (e.g., a set of video blocks), or other larger coding units (CUs) according to a predefined splitting structure associated with the video data, such as a quadtree (QT) structure. It should be noted that the terms "block" or "video block" used herein may be a portion of a frame or picture, in particular a rectangular (square or non-square) portion. With reference to, for example, HEVC and VVC, a block or video block may be or correspond to a coding tree unit (CTU), a CU, a prediction unit (PU), or a transform unit (TU) and / or may be or correspond to a corresponding block (e.g., a coding tree block (CTB), a coding block (CB), a prediction block (PB), or a transform block (TB)) and / or a sub-block.

[0053] The prediction processing unit 41 may select one of a plurality of feasible prediction coding modes for the current video block based on the error results (e.g., coding rate and distortion level), such as one of one or more inter-frame prediction coding modes in a plurality of intra-frame prediction coding modes. The prediction processing unit 41 may provide the resulting intra-frame prediction coding block or inter-frame prediction coding block to the adder 50 to generate a residual block, and to the adder 62 to reconstruct the coding block for subsequent use as part of a reference frame. The prediction processing unit 41 also provides syntax elements (e.g., motion vectors, intra-frame mode indicators, partition information, and other such syntax information) to the entropy coding unit 56.

[0054] To select an appropriate intra-prediction coding mode for the current video block, intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-prediction coding of the current video block in relation to one or more neighboring blocks in the same frame as the current block to be encoded to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 may perform inter-prediction coding of the current video block in relation to one or more prediction blocks in one or more reference frames to provide temporal prediction. Video encoder 20 may perform multiple encoding passes, for example, to select an appropriate coding mode for each block of video data.

[0055] In some embodiments, motion estimation unit 42 determines an inter-prediction mode for a current video frame by generating a motion vector according to a predetermined pattern within a sequence of video frames, the motion vector indicating the displacement of a video block within the current video frame relative to a prediction block within a reference video frame. Motion estimation performed by motion estimation unit 42 is the process of generating a motion vector that estimates the motion of a video block. For example, a motion vector may indicate the displacement of a video block within a current video frame or picture relative to a prediction block within a reference frame associated with a current block being encoded within the current frame. The predetermined pattern may designate a video frame in a sequence as a P frame or a B frame. Intra BC unit 48 may determine a vector (e.g., a block vector) for intra BC coding in a manner similar to the motion vector determined by motion estimation unit 42 for inter prediction, or may utilize motion estimation unit 42 to determine a block vector.

[0056] Regardless of whether the prediction block is from the same frame according to intra-frame prediction or from a different frame according to inter-frame prediction, the video encoder 20 can form a residual video block by subtracting the pixel values ​​of the prediction block from the pixel values ​​of the current video block being encoded. The pixel difference values ​​forming the residual video block may include both luma component differences and chroma component differences.

[0057] Intra-prediction processing unit 46 may encode the current block using various intra-prediction modes, for example, during separate encoding passes, and intra-prediction processing unit 46 (or, in some examples, mode selection unit) may select an appropriate intra-prediction mode to use from the tested intra-prediction modes. Intra-prediction processing unit 46 may provide information indicating the intra-prediction mode selected for the block to entropy encoding unit 56. Entropy encoding unit 56 may encode the information indicating the selected intra-prediction mode into the bitstream.

[0058] After prediction processing unit 41 determines a prediction block for the current video block via inter-prediction or intra-prediction, adder 50 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block may be included in one or more TUs and provided to transform processing unit 52. Transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform (e.g., a discrete cosine transform (DCT) or a conceptually similar transform).

[0059] The transform processing unit 52 may send the resulting transform coefficients to a quantization unit 54. The quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may also reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, the quantization unit 54 may then perform a scan on the matrix including the quantized transform coefficients. Alternatively, the entropy coding unit 56 may perform the scan.

[0060] After quantization, entropy coding unit 56 entropy encodes the quantized transform coefficients into a video bitstream using, for example, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding method or technique. The encoded bitstream may then be sent to a video bitstream such as a Figure 1 The video decoder 30 shown, or archived as Figure 1 The video frame may be stored in storage device 32 as shown for later transmission to or retrieval by video decoder 30. Entropy encoding unit 56 may also entropy encode motion vectors and other syntax elements for the current video frame being encoded.

[0061] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual video block in the pixel domain for use in generating reference blocks for predicting other video blocks. As noted above, motion compensation unit 44 may generate a motion compensated prediction block from one or more reference blocks of a frame stored in DPB 64. Motion compensation unit 44 may also apply one or more interpolation filters to the prediction block to calculate sub-integer pixel values ​​for use in motion estimation.

[0062] Adder 62 adds the reconstructed residual block to the motion compensated prediction block generated by motion compensation unit 44 to generate a reference block for storage in DPB 64. The reference block may then be used as a prediction block by intra BC unit 48, motion estimation unit 42, and motion compensation unit 44 to inter-predict another video block in a subsequent video frame.

[0063] Figure 3 1 is a block diagram showing an exemplary video decoder 30 according to some embodiments of the present application. The video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, and a DPB 92. The prediction processing unit 81 further includes a motion compensation unit 82, an intra-frame prediction unit 84, and an intra-frame BC unit 85. The video decoder 30 may perform the above-mentioned Figure 2 The encoding process is substantially the inverse of the decoding process described with respect to video encoder 20. For example, motion compensation unit 82 may generate prediction data based on motion vectors received from entropy decoding unit 80, and intra-prediction unit 84 may generate prediction data based on intra-prediction mode indicators received from entropy decoding unit 80.

[0064] In some examples, embodiments of the present disclosure may be dispersed in one or more of the units of the video decoder 30. For example, the intra BC unit 85 may perform embodiments of the present application alone or in combination with other units of the video decoder 30, such as the motion compensation unit 82, the intra prediction unit 84, and the entropy decoding unit 80. In some examples, the video decoder 30 may not include the intra BC unit 85, and the functions of the intra BC unit 85 may be performed by other components of the prediction processing unit 81, such as the motion compensation unit 82.

[0065] Video data memory 79 may store video data, such as an encoded video bitstream, to be decoded by other components of video decoder 30. Video data stored in video data memory 79 may be obtained, for example, from storage device 32, from a local video source (e.g., a camera), via wired or wireless network communication of video data, or by accessing a physical data storage medium (e.g., a flash drive or hard disk).

[0066] During the decoding process, the video decoder 30 receives an encoded video bitstream representing video blocks of an encoded video frame and associated syntax elements. The entropy decoding unit 80 of the video decoder 30 entropy decodes the bitstream to generate quantization coefficients, motion vectors or intra-frame prediction mode indicators, and other syntax elements. The entropy decoding unit 80 then forwards the motion vectors or intra-frame prediction mode indicators, and other syntax elements to the prediction processing unit 81.

[0067] When a video frame is encoded as an intra-prediction coded (I) frame or for intra-coded prediction blocks in other types of frames, the intra-prediction unit 84 of the prediction processing unit 81 can generate prediction data for a video block of the current video frame based on the intra-prediction mode transmitted by the signal and reference data from a previously decoded block of the current frame.

[0068] When the video frame is encoded as an inter-frame prediction coding (i.e., B or P) frame, the motion compensation unit 82 of the prediction processing unit 81 generates one or more prediction blocks for the video block of the current video frame based on the motion vector and other syntax elements received from the entropy decoding unit 80. Each of the prediction blocks can be generated from a reference frame in one of the reference frame lists. The video decoder 30 can construct the reference frame lists, i.e., List 0 and List 1, based on the reference frames stored in the DPB 92 using a default construction technique.

[0069] In some examples, when a video block is encoded according to the intra BC mode described herein, intra BC unit 85 of prediction processing unit 81 generates a prediction block for the current video block based on the block vector and other syntax elements received from entropy decoding unit 80. The prediction block may be within a reconstructed region of the same picture as the current video block as defined by video encoder 20.

[0070] Motion compensation unit 82 and / or intra BC unit 85 determine prediction information for a video block of the current video frame by parsing motion vectors and other syntax elements, and then use the prediction information to generate a prediction block for the current video block being decoded.

[0071] Motion compensation unit 82 may also perform interpolation using interpolation filters as used by video encoder 20 during encoding of the video blocks to calculate interpolated values ​​for sub-integer pixels of reference blocks. In this case, motion compensation unit 82 may determine the interpolation filters used by video encoder 20 based on the received syntax elements and use these interpolation filters to produce the prediction blocks.

[0072] Inverse quantization unit 86 inverse quantizes the quantized transform coefficients provided in the bitstream and entropy decoded by entropy decoding unit 80, using the same quantization parameters calculated by video encoder 20 for each video block in the video frame to determine the degree of quantization. Inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients to reconstruct the residual block in the pixel domain.

[0073] After the motion compensation unit 82 or the intra BC unit 85 generates a prediction block for the current video block based on the vector and other syntax elements, the adder 90 reconstructs the decoded video block for the current video block by adding the residual block from the inverse transform processing unit 88 to the corresponding prediction block generated by the motion compensation unit 82 and the intra BC unit 85. A loop filter 91 (e.g., a deblocking filter, an SAO filter, a CCSAO filter, and / or an ALF) may be located between the adder 90 and the DPB 92 to further process the decoded video block. In some examples, the loop filter 91 may be omitted, and the decoded video block may be provided directly to the DPB 92 by the adder 90. The decoded video blocks in a given frame are then stored in the DPB 92, which stores reference frames for subsequent motion compensation of the next video block. The DPB 92 or a memory device separate from the DPB 92 may also store the decoded video for later presentation on a display device (e.g., Figure 1 on a display device 34).

[0074] As described above, in the related art, there is a problem that the bit rate allocation of the encoder pair is not ideal, which affects the subjective picture quality experience.

[0075] Specifically, since the content of the transitioned image has no information correlation with the previous frame, it is difficult to use temporal correlation for encoding, so more codewords are usually required. However, existing hardware encoders cannot customize the bitrate allocation strategy in different scenarios according to demand.

[0076] Taking the multimedia codec framework VideoToolbox as an example, it is mainly used for hardware-accelerated encoding and decoding of audio and video on iOS and mac operating system platforms. It provides developers with a set of application programming interfaces (APIs) and tools, allowing applications to use the hardware acceleration function of the device to perform efficient audio and video processing.

[0077] However, the multimedia codec framework exposes limited peripheral code control interfaces, limited adjustment space, and low flexibility. In addition, the multimedia codec framework cannot recognize scenes that require high bit rates, such as screen switching, and has limited effect in adaptively adjusting bit rate allocation according to different video complexity scenes.

[0078] In order to solve at least part of the above problems, exemplary embodiments of the present disclosure provide a video encoding method, a video encoding device, a bit stream generation method, an electronic device, a computer readable storage medium, and a computer program product. Figures 1 to 10 Describe in detail.

[0079] According to an exemplary embodiment of the present disclosure, a video encoding method is provided. The video encoding method can be applied to any video encoding scenario. Figure 4 An example implementation scenario of a video encoding method according to an exemplary embodiment of the present disclosure is given.

[0080] like Figure 4 As shown, when a user uses a video application client on a user terminal (e.g., mobile phone 411, desktop computer 412, tablet computer 413, etc.) to request to watch video content from a server or video distribution platform 430 through network 420, the server or video distribution platform 430 can send the requested video content to user terminals 411, 412 and / or 413 through network 420, and the user can watch the received video content through the video application client.

[0081] In the above process, the server or video distribution platform 430 needs to encode the video content to be sent. In the encoding process, the video to be sent can be encoded according to the video encoding method of the exemplary embodiment of the present disclosure. Specifically, the server or video distribution platform 430 can divide the video to be encoded to obtain multiple video segments; it can also determine the segment encoding bit rate of each video segment according to the encoding complexity of each video segment and the encoding complexity of the video to be encoded; it can also encode the video frames in each video segment based on the segment encoding bit rate of each video segment.

[0082] It should be noted that although the above description uses a server or video distribution platform as an example, it is only an example. The executor of the video encoding method can be any electronic device, which may include, for example, smart phones, tablet computers, laptops, digital assistants, wearable devices, vehicle terminals, etc., which are physical devices that serve as hardware codecs, and may also include software that runs on physical devices as software video encoders.

[0083] It should also be noted that although the video encoding application scenario in which a user terminal requests to watch a video is described here as an example, it should be understood that the application scenario of the video encoding method according to the present disclosure is not limited to this, and it can also be applied to any other application scenario involving video encoding.

[0084] Figure 5It is a flow chart of a video encoding method according to an exemplary embodiment. As described above, according to the video encoding method of the exemplary embodiment of the present disclosure, the encoding complexity of each video segment can be analyzed to allocate encoding bit rates to each video segment, so that the encoding bit rates of each video segment can be reasonably allocated, the video encoding quality can be improved, and the subjective picture quality experience of the video can be improved.

[0085] like Figure 5 As shown, the video encoding method according to an exemplary embodiment of the present disclosure may include the following steps:

[0086] In step S510, the video to be encoded may be segmented to obtain a plurality of video segments.

[0087] In this step, the video to be encoded may be segmented in any manner. As an example, the video to be encoded may be evenly segmented into a preset number of video segments. As another example, according to an embodiment of the present disclosure, the video segmentation may be performed according to the transition scenes in the video.

[0088] Specifically, if Figure 6 As shown, the encoded video can be segmented in the following way to obtain multiple video segments:

[0089] In step S610, the intra-frame complexity of a first video frame in the video to be encoded and the inter-frame complexity of the first video frame relative to a second video frame may be determined, wherein the second video frame precedes the first video frame.

[0090] Here, the first video frame can be any video frame in the video to be encoded. For example, in the process of encoding the video to be encoded, each video frame can be taken as the first video frame in sequence according to the order in the video frame sequence, and its intra-frame complexity and inter-frame complexity can be determined to determine whether the video frame is a transition frame.

[0091] The second video frame may be any video frame before the first video frame, for example, may be a frame before the first video frame.

[0092] In this step, the inter-frame complexity can represent the information that needs to be encoded when the first video frame takes the second video frame as a reference; the intra-frame complexity can represent the information that needs to be encoded when the first video frame takes itself as a reference. The first video frame can be subjected to intra-frame prediction and inter-frame prediction in the encoding technology, and the codewords and distortion size required to encode the first video frame under the two prediction modes are calculated, thereby obtaining the above-mentioned intra-frame complexity and inter-frame complexity respectively. Here, the encoding complexity of the video frame can characterize the amount of information that needs to be encoded for the video frame, and the complexity can be represented by, for example, but not limited to, SAD (Sum of Absolute Difference) or SATD (Sum of Absolute Transformed Difference).

[0093] Here, for example, typical prediction methods in HEVC may be used but are not limited to perform intra-frame prediction and inter-frame prediction. As an example, the intra-frame prediction method may be but are not limited to 33 angle modes, direct current (DC) mode, planar mode, etc., and the inter-frame prediction method may be but are not limited to motion estimation, motion compensation, etc.

[0094] For example, Figure 7 As shown, the video encoding method according to the exemplary embodiment of the present disclosure can be implemented by a rate pre-allocation model, wherein the video frames of the video to be encoded can be subjected to complexity analysis by a complexity analysis module in the rate pre-allocation model. Specifically, intra-frame prediction and inter-frame prediction can be performed to obtain complexity information including intra-frame complexity and inter-frame complexity for subsequent scene detection.

[0095] In addition, in step S610, as an example, in the process of determining the intra-frame complexity and the inter-frame complexity, the prediction algorithm can be optimized by fast algorithm and assembly, so as to reduce the overall complexity of the bit rate allocation, thereby ensuring that the prediction algorithm maintains a high accuracy without introducing excessive computing power consumption.

[0096] Here, the fast algorithm may include an intra-frame fast algorithm and an inter-frame fast algorithm. The intra-frame fast algorithm may reduce the types of prediction modes, for example, including but not limited to reducing the prediction types to five directions: lower left, left, upper left, upper and upper right, as well as a DC mode and a Planar mode; the inter-frame fast algorithm may reduce the number of reference frames, for example, each frame may only refer to the previous frame for motion estimation.

[0097] In addition, the fast algorithm may also include reducing the resolution of the video frame. For example, the length and width of the video frame may be reduced to 1 / 2, 1 / 4 or other proportions of the original size according to a preset configuration to reduce the amount of calculation.

[0098] In addition, the fast algorithm may also include reducing the inter-frame prediction search range. For example, during the inter-frame prediction search process, in response to the inter-frame complexity of the coding block being less than a preset threshold, the inter-frame prediction search process may be skipped.

[0099] Here, assembly optimization may include intra-frame assembly optimization and inter-frame assembly optimization. Intra-frame assembly optimization may, for example, but is not limited to, performing arm64 assembly optimization on all prediction modes to increase computational parallelism. Inter-frame assembly optimization may, for example, but is not limited to, performing single instruction multiple data (SIMD) assembly optimization on operations such as pixel interpolation in motion estimation and motion compensation, which may also increase computational parallelism.

[0100] By adding fast algorithms and assembly optimization, the calculation of this method can have no obvious impact on the overall encoding complexity, significantly improving the user experience when producing and consuming videos.

[0101] In step S620, it may be determined whether the first video frame is a transition frame according to the inter-frame complexity and the intra-frame complexity of the first video frame.

[0102] In this step, it can be determined whether the first video frame is a transition frame based on the complexity difference between the inter-frame complexity and the intra-frame complexity of the first video frame. Figure 7 As shown, scene detection can be performed through the scene detection module in the bit rate pre-allocation model to determine whether the first video frame is a transition frame, thereby obtaining scene switching information. Specifically, the transition frame can be determined.

[0103] As an example, in response to the complexity difference exceeding the difference threshold, the first video frame may be determined to be a transition frame; in response to the complexity difference not exceeding the difference threshold, the first video frame may be determined to be a non-transition frame.

[0104] Specifically, when the inter-frame complexity is relatively large relative to the intra-frame complexity, it can be said that the first video frame does not obtain higher encoding performance by using the preceding second video frame as a reference compared to only referring to itself, indicating that the first video frame has a low correlation with the preceding second video frame, and thus it can be judged as a transition frame. By determining the transition frame by analyzing the complexity difference, the transition scene in the video can be detected more accurately, which serves as the basis for improving the bit rate of the transition scene, and is conducive to guiding the subsequent more reasonable allocation of the encoding bit rate.

[0105] Here, the complexity difference between the inter-frame complexity and the intra-frame complexity may be represented by, for example but not limited to, a ratio, a difference, or other mathematical methods. As an example, the complexity difference may be represented by the ratio in the following formula (1):

[0106]

[0107] Among them, inter_cost represents the inter-frame complexity, which represents the information that needs to be encoded when the first video frame uses the second video frame as a reference; intra_cost represents the intra-frame complexity, which represents the information that needs to be encoded when the first video frame uses itself as a reference. For this formula (1), the larger the ratio ratio is, the less information the first video frame can refer to from the second video frame, the greater the difference between the two frame scenes, the higher the inter-frame complexity, the lower the correlation between the first video frame and the second video frame, and the greater the possibility that the first video frame is a transition frame. However, the representation method of complexity difference is not limited to this formula (1), for example, it can also be represented by the inverse of the ratio ratio.

[0108] Here, the difference threshold may be an empirical value according to industry standards. Taking the above formula (1) as an example, according to the Scenecut decision algorithm in High Efficiency Video Coding (HEVC), a difference threshold scenecut_th may be set, and the difference threshold scenecut_th may be, for example, but not limited to, 40%. If the ratio ratio in the above formula (1) exceeds the difference threshold scenecut_th, it may be considered that the two frames of scenes are significantly different, and thus a transition may be considered to have occurred.

[0109] In such Figure 7 In the bit rate pre-allocation model shown, the video can be segmented by the video segmentation module based on the transition frames as scene switching information to obtain video segments, which can be input into the segment complexity statistics module to calculate the segment encoding bit rate of the video segments.

[0110] Return to reference Figure 6 In step S630, in response to the first video frame being a transition frame, the video to be encoded may be segmented based on the transition frame to obtain a plurality of video segments.

[0111] In this step, video segmentation can be performed at the position of the transition frame or the frame number to segment the entire video into video segments of multiple different scenes (shots). For example, the transition frame can be used as the first frame in the segmented video segment, but is not limited to it.

[0112] In this way, the video can be segmented according to the transition scenes in the video, so that the scenes in each segmented video are unified and the contents of the video frames are similar. These video frames with similar contents are more suitable for encoding using the same segment encoding bit rate, so that the bit rate can be allocated more reasonably.

[0113] Return to reference Figure 5In step S520, for each video segment, the segment encoding bit rate of each video segment can be determined according to the encoding complexity of each video segment and the encoding complexity of the video to be encoded.

[0114] In this step, when each video segment has been obtained, the coding complexity of the video segment can be analyzed, and it can be compared with the coding complexity of the entire video to be encoded, so as to allocate corresponding segment coding bit rates according to the coding complexity level of the video segment. For example, for video segments with higher coding complexity, more bit rates can be allocated; and for video segments with lower coding complexity, less bit rates can be allocated, so as to achieve reasonable bit rate allocation.

[0115] like Figure 8 As shown, the coding complexity of a video segment can be determined in the following way:

[0116] In step S810 , the coding complexity of each video frame in the video segment may be determined.

[0117] Here, the coding complexity of the video frame may be, for example, but not limited to, inter-frame complexity. For example, the information to be encoded for the current video frame with reference to the previous frame may be used as the coding complexity of the video frame. However, it is not limited to this, and may also be the inter-frame complexity predicted with reference to other video frames.

[0118] In step S820 , the normalized complexity of each video frame may be determined based on the minimum complexity and the maximum complexity among the coding complexities of the video frames in the video segment, the first coefficient, and the second coefficient.

[0119] Here, the relationship between the first coefficient and the second coefficient can be used to control the difference between the segmented encoding bit rates of each video segment. Here, since the segmented encoding bit rate of each video segment can be adjusted according to the normalized complexity of the video frame, controlling the difference in complexity of the video frame by the first coefficient and the second coefficient is equivalent to indirectly controlling the difference in segmented encoding bit rates of the video segments.

[0120] Specifically, the normalized complexity of a video frame can be obtained by normalizing the coding complexity of the video frame according to the minimum complexity and the maximum complexity of the coding complexities of each video frame. In an embodiment of the present disclosure, a first coefficient and a second coefficient can be introduced in the normalization process. The first coefficient and the second coefficient can be used to control the variance of the normalized complexity of each video frame, thereby controlling the difference in segmented coding bit rates of each video segment.

[0121] For example, any existing normalization algorithm can be used to obtain a normalization item of the video frame based on the minimum complexity and maximum complexity of the encoding complexity of each video frame. The normalization item can be weighted using a first coefficient, and the weighted normalization item can be added to the second coefficient to obtain a final normalization result.

[0122] In this example, when the first coefficient is smaller than the second coefficient (for example, the first coefficient is much smaller than the second coefficient), the normalized complexity of each video segment can be made close to the value of the second coefficient, the segment encoding bit rates of each video segment are basically the same, and the bit rate distribution difference is relatively small; when the first coefficient is larger than the second coefficient (for example, the first coefficient is much larger than the second coefficient), the bit rate allocated to the video segment with the smallest complexity (normalized complexity = second coefficient) is close to 0, and the bit rate distribution difference is relatively large.

[0123] As an example, the normalized complexity can be expressed by the following formula (2):

[0124]

[0125] in, represents the normalized complexity of the i-th video frame in the video segment, x i represents the coding complexity of the i-th video frame, x min represents the minimum complexity of the coding complexity of each video frame in the video segment, x max represents the maximum complexity of the coding complexity of each video frame in the video segment, a represents the first coefficient, b represents the second coefficient, wherein, represents the normalization term.

[0126] Although the above formula (2) is used as an example to provide an example calculation method of normalized complexity, it is not limited thereto. For example, the normalized term may also be obtained by using other existing normalization algorithms.

[0127] The above normalized complexity determination process can be performed, for example, as follows: Figure 7 The complexity normalization module shown is used to implement the normalization process, and the complexity normalization module can perform normalization processing based on the encoding complexity of each video frame calculated by the complexity analysis module to obtain the normalized complexity.

[0128] In step S830, the encoding complexity of the video segment may be determined based on the normalized complexity of each video frame in the video segment.

[0129] As an example, the average value of the normalized complexity of all video frames in the video segment can be used as the coding complexity of the video segment. Figure 7The segment complexity statistics module shown is used to implement the above, and the segment complexity statistics module can calculate the segment encoding bit rate of each video segment based on the complexity information output from the complexity normalization module and the segment information output from the segment complexity statistics module.

[0130] By determining the coding complexity of the video segments in the above manner, the difference between the coding complexities of the video segments can be adjusted, so that the distribution difference between the coding bit rates of the segments can be adjusted by adjusting the above first coefficient and second coefficient according to the actual scene or transmission conditions.

[0131] In addition, the coding complexity of the video to be encoded can be determined in a similar manner to the coding complexity of the video segment. Specifically, the coding complexity of the video to be encoded can be determined based on the normalized complexity of each video frame in the video to be encoded. For example, the average value of the normalized complexity of all video frames in the video to be encoded can be used as the coding complexity of the video to be encoded.

[0132] As an example, the segment encoding bitrate of each video segment may be determined in the following manner: according to the encoding complexity of the video segment and the encoding complexity of the video to be encoded, the original encoding bitrate of the video to be encoded is adjusted to determine the segment encoding bitrate of each video segment.

[0133] Here, the original encoding bit rate of the video to be encoded may be preset, for example, an expected encoding bit rate. The original encoding bit rate may be used as a benchmark for adjusting the bit rate of the video segment, and adjustments are made on the basis of the original encoding bit rate to obtain the segment encoding bit rates of the video segment. In this way, by making adjustments based on the original encoding bit rate to obtain the encoding bit rates of each segment, the encoding bit rates of each segment may be determined more conveniently.

[0134] In one example, the encoding bit rate of the video segment can be allocated according to the ratio of the encoding complexity of the video segment to the encoding complexity of the video to be encoded. For example, the original encoding bit rate can be weighted by using the ratio. However, the embodiments of the present disclosure are not limited to this, and the amplitude of adjustment of the original encoding bit rate, or the degree of deviation of the adjusted overall encoding bit rate relative to the original encoding bit rate, can also be controlled.

[0135] Specifically, as an example, the step of adjusting the original encoding bit rate of the video to be encoded to determine the segmented encoding bit rate of each video segment may include: adjusting the original encoding bit rate of the video to be encoded to determine the segmented encoding bit rate of each video segment according to the encoding complexity of all video frames in the current video segment, the encoding complexity of all video frames in the video to be encoded, and the offset coefficient.

[0136] Here, the offset coefficient may represent the degree of offset of the average encoding rate of the to-be-encoded video compared to the original encoding rate after the encoding rate of each video segment is adjusted.

[0137] Specifically, the offset coefficient can be introduced to control the degree of offset of the overall encoding rate after the bit rate adjustment compared to the original encoding rate. The larger the offset coefficient, the higher the overall encoding rate. As an example, the offset coefficient can be introduced when adjusting the original encoding rate by subtracting the encoding complexity of the video to be encoded from the offset coefficient.

[0138] For example, the segmented encoding bitrate of each video segment can be determined in the following manner: determine a first value according to the encoding complexity of the video segment, the encoding complexity of the video to be encoded, and the offset coefficient; determine the segmented encoding bitrate of each video segment based on the first value and the original encoding bitrate.

[0139] Here, the difference between the coding complexity of the video to be coded and the offset coefficient can be determined. The first value can be, for example, but not limited to, the ratio of the coding complexity of the video segment to the difference. The original coding bit rate can be weighted by the ratio to obtain the segment coding bit rate. Specifically, the segment coding bit rate can be obtained by the following equations (3) and (4):

[0140]

[0141] Among them, br represents the encoding bit rate of the current video segment, BR represents the original encoding bit rate, represents the coding complexity of the current video segment (e.g., segment i), Indicates the encoding complexity of the video to be encoded, represents the minimum value of the coding complexity of all video segments, offset represents the offset coefficient, and λ is a configurable coefficient that can be set according to actual needs. Indicates the first value mentioned above.

[0142] As an example, the configurable coefficient λ can be used to adjust the size of the offset coefficient to control the degree of deviation of the adjusted overall encoding bit rate compared to the original encoding bit rate. The value range of the configurable coefficient λ is, for example, but not limited to [0,1]. When λ=0, the encoding bit rates of different video segments are adjusted up and down around the original encoding bit rate; when λ=1, the encoding bit rate of the video segment with the lowest encoding bit rate (or the video segment with the lowest complexity) is the original encoding bit rate.

[0143] The above segment coding rate determination process can be carried out as follows: Figure 7The bit rate allocation module in the bit rate pre-allocation (Per-Shot bit rate allocation) model shown in the figure is implemented. The bit rate allocation module can determine the segmented encoding bit rate of the video segment based on the encoding complexity of the video segment output by the segment complexity statistics, the encoding complexity of the video to be encoded, and the preset original encoding bit rate.

[0144] Through the above strategy, relatively more codewords can be assigned to complex scenes in the video, while relatively fewer codewords can be assigned to simple scenes.

[0145] Return to reference Figure 5 In step S530, the video frames in each video segment may be encoded based on the segment encoding bit rate of each video segment.

[0146] In this step, the video segments may be encoded based on the segment encoding bit rates of the video segments, for example, using an existing encoding method.

[0147] The above describes an example process of adjusting the segment encoding bit rate of each video segment. In an embodiment of the present disclosure, considering that during the encoding process, the actual encoding bit rate may be different from the configured bit rate (such as the above-mentioned segment encoding bit rate), the actual encoding bit rate of a scene with lower complexity may be lower than the configured bit rate, resulting in part of the bit rate remaining. Therefore, as an example, in the video encoding method, a bit rate feedback mechanism can also be introduced to count the codewords consumed after the encoding of the previous frame is completed, and compare them with the codeword consumption predicted at the original encoding bit rate. If the codewords are relatively abundant, the remaining codewords are compensated for the encoding of the remaining unencoded frames, so as to better allocate the encoding bit rate and avoid wasting the preset original encoding bit rate.

[0148] Specifically, the video encoding method may further include: in response to an actual encoding bit rate of the current video segment being less than a predicted encoding codeword of the current video segment, increasing a segment encoding bit rate of a video segment subsequent to the current video segment.

[0149] As an example, Fig. 9 As shown, the segment encoding bit rate of the video segment after the current video segment can be increased by:

[0150] In step S910 , a residual codeword may be determined based on a difference between an actual encoding codeword of the current video segment and a predicted encoding codeword of the current video segment.

[0151] Here, the predicted coding codeword of the video segment can be determined based on the segment coding bit rate of the video segment and the number of video frames in the video segment. Specifically, the predicted coding codeword of the video segment can be expressed by the following formula (5):

[0152]

[0153] Among them, bits_est represents the predicted coding codeword of the video segment, br represents the segment coding bit rate of the video segment, frames represents the number of video frames in the video segment, and fps represents the frame rate of the video to be encoded.

[0154] When the video segment coding is completed, the actual coding codeword bits_out after the video segment coding can be obtained. The difference between the actual codeword bits_out and the predicted coding codeword bits_est is: bits_diff=bits_est-bits_out. The difference bits_diff can be determined as the remaining codeword.

[0155] In step S920, a rate offset for a subsequent video segment may be determined based on the remaining codewords, the number of subsequent video frames to be encoded in the video to be encoded, and the number of frames of the subsequent video segment.

[0156] Specifically, in response to the remaining codeword being greater than 0, such as the above bits_diff>0, it can be considered that the current bit rate is sufficient and the encoding bit rate of the subsequent frames to be encoded can be improved. As an example, the remaining codewords can be evenly distributed to the subsequent video frames to be encoded to determine the bit rate offset of the subsequent video segments. In this example, the bit rate offset can be expressed by the following formula (6):

[0157]

[0158] Among them, br_diff represents the bit rate offset, and frames_lef represents the number of subsequent video frames to be encoded.

[0159] However, the method for determining the bit rate offset is not limited to the above-mentioned average allocation method, and other allocation methods may also be used, for example, proportional allocation according to the complexity of the video frame to be encoded, weighting the above formula (6) according to the complexity of the video frame to be encoded, etc.

[0160] In step S930, an increased segment encoding bit rate for the subsequent video segment may be obtained based on the bit rate offset and the segment encoding bit rate of the subsequent video segment.

[0161] In this step, for example, the bit rate offset may be added to the segment encoding bit rate of the subsequent video segment to obtain an increased segment encoding bit rate.

[0162] Through the above method, a rate compensation mechanism based on the encoded bit rate can be used to count the codewords consumed after the previous frame is encoded and compare them with the predicted encoding bit rate. If the codewords are sufficient, the remaining codewords can be compensated to the unencoded frames by superimposing the bit rate offset, so that the encoding bit rate of each unencoded frame can be improved, thereby further improving the encoding quality.

[0163] In addition, the video encoding method according to the embodiments of the present disclosure can be adapted to various hardware encoders such as Video Tool Box (VTB), MediaCodec of Android platform, etc., as long as the hardware encoder supports the real-time bit rate setting interface.

[0164] The video encoding method according to the embodiment of the present invention was verified on the iOS VideoToolBox hardware encoder. Under 10 test sequences, the average bit rate remained the same, the subjective evaluation index KVQ (Kuaishou Video Quality) increased by 0.01, the picture quality improvement reached the level visible to the human eye, and at the same time, the bad cases of picture quality in some transition scenes were improved, and the overall calculation complexity was extremely low. Taking the VTB hardware encoder as the reference benchmark, the complexity increased by only less than 4%, which solved the pain point that the hardware encoders on the market could not adjust the code control to improve the picture quality according to the scene requirements.

[0165] Fig.10 A schematic diagram showing comparison of image quality before and after bit rate improvement in an application example of a video encoding method according to an exemplary embodiment is shown. Fig.10 The left picture in the figure is the video screen exported by the original encoder, and the right picture is the video screen exported by the encoder after the bit rate is adjusted by the above video encoding method. It can be observed that the overall picture quality of the video is improved.

[0166] According to the video encoding method of the embodiment of the present disclosure, a content-adaptive global bit rate allocation scheme is proposed, which can segment the video, for example, by detecting transition scenes to divide it, and adjust the encoding bit rate allocated to each video segment according to the complexity of the segment, so as to obtain better subjective picture quality while maintaining the overall encoding bit rate by more reasonably allocating codewords to scenes of different complexities.

[0167] In addition, according to the video encoding method of the embodiment of the present disclosure, under the conditions of limited hardware encoding interface, combined with the external video complexity detection model, the bit rate allocation strategy can be effectively improved, and more code words can be provided for scenes with complex pictures or transitions that are eye-sensitive and have large bit rate requirements, thereby improving the subjective viewing experience of videos in multiple scenarios.

[0168] Fig.11 is a block diagram of a video encoding device according to an exemplary embodiment. Fig.11 , the video encoding device 1100 includes a segmentation unit 1110, a determination unit 1120 and an encoding unit 1130.

[0169] The segmentation unit 1110 is configured to segment the video to be encoded to obtain a plurality of video segments.

[0170] The determining unit 1120 is configured to determine, for each video segment, a segment encoding bit rate of each video segment according to the encoding complexity of each video segment and the encoding complexity of the video to be encoded.

[0171] The encoding unit 1130 is configured to encode the video frames in each video segment based on the segment encoding bit rate of each video segment.

[0172] As an example, the segmentation unit 1110 is also configured to: determine the intra-frame complexity of a first video frame in the video to be encoded and the inter-frame complexity of the first video frame relative to a second video frame, wherein the second video frame precedes the first video frame; determine whether the first video frame is a transition frame based on the inter-frame complexity and intra-frame complexity of the first video frame; and segment the video to be encoded based on the transition frame to obtain multiple video segments.

[0173] As an example, the segmentation unit 1110 is further configured to: in response to a complexity difference between the inter-frame complexity and the intra-frame complexity of the first video frame exceeding a difference threshold, determine that the first video frame is a transition frame.

[0174] As an example, the determining unit 1120 is further configured to: determine the segment encoding rate of each video segment by adjusting the original encoding rate of the video to be encoded according to the encoding complexity of the video segment and the encoding complexity of the video to be encoded.

[0175] As an example, the determination unit 1120 is also configured to: determine the segmented encoding rate of each video segment by adjusting the original encoding rate of the video to be encoded according to the encoding complexity of the video segment, the encoding complexity of the video to be encoded, and the offset coefficient, wherein the offset coefficient represents the degree of deviation of the average encoding rate of the video to be encoded compared to the original encoding rate after the encoding rate of each video segment is adjusted.

[0176] As an example, the determination unit 1120 is also configured to: determine a first value based on the coding complexity of all video frames in the current video segment, the coding complexity of all video frames in the video to be encoded, and the offset coefficient; determine the segmented coding rate of each video segment based on the first value and the original coding rate.

[0177] As an example, the determination unit 1120 is also configured to determine the coding complexity of the video segment in the following manner: determine the coding complexity of each video frame within the video segment; determine the normalized complexity of each video frame based on the minimum complexity and the maximum complexity of the coding complexity of each video frame within the video segment, the first coefficient and the second coefficient, wherein the relationship between the first coefficient and the second coefficient is used to control the difference between the segmented coding bit rates of each video segment; determine the coding complexity of the video segment based on the normalized complexity of each video frame within the video segment.

[0178] As an example, the video encoding apparatus further includes a bit rate adjustment unit 1140, which is configured to increase the segment encoding bit rate of the video segment after the current video segment in response to the actual encoding codeword of the current video segment being less than the predicted encoding codeword of the current video segment.

[0179] As an example, the rate adjustment unit 1140 is also configured to: determine the remaining codeword based on the difference between the actual encoding codeword of the current video segment and the predicted encoding codeword of the current video segment; determine the rate offset for the subsequent video segment based on the remaining codeword, the number of subsequent video frames to be encoded in the video to be encoded, and the number of frames of the subsequent video segment; and obtain the increased segment encoding rate for the subsequent video segment based on the rate offset and the segment encoding rate of the subsequent video segment.

[0180] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0181] Fig.12 FIG. 1 is a block diagram of an electronic device according to an exemplary embodiment. Fig.12 As shown, the electronic device 1200 includes a processor 1210 and a memory 1220 for storing processor executable instructions. Here, when the processor executable instructions are executed by the processor, the processor executes the video encoding method as described in the above exemplary embodiments.

[0182] As an example, the electronic device 1200 is not necessarily a single device, but may be any collection of devices or circuits that can execute the above instructions (or instruction sets) individually or in combination. The electronic device 1200 may also be part of an integrated control system or system manager, or may be configured as a server that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0183] In the electronic device 1200, the processor 1210 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor 1210 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0184] The processor 1210 may execute instructions or codes stored in the memory 1220, which may also store data. Instructions and data may also be sent and received over a network via a network interface device, which may employ any known transmission protocol.

[0185] The memory 1220 may be integrated with the processor 1210, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 1220 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 1220 and the processor 1210 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 1210 can read files stored in the memory 1220.

[0186] In addition, the electronic device 1200 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 1200 may be connected to each other via a bus and / or a network.

[0187] In an exemplary embodiment, a computer-readable storage medium may also be provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can perform the video encoding method as described in the above exemplary embodiment. The computer-readable storage medium may be, for example, a memory including instructions. Optionally, the computer-readable storage medium may be: a read-only memory (ROM), a random access memory (RAM), a random access programmable read-only memory (PROM), an electrically erasable programmable read-only memory (EEPROM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a non-volatile memory, a CD-ROM, a CD-R, a CD+R, a CD-RW, a CD+RW, a DVD-ROM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, a DVD-RAM, a BD-ROM, a BD-R, a BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0188] In an exemplary embodiment, a computer program product may also be provided. The computer program product includes computer instructions. When the computer instructions are executed by a processor, the video encoding method as described in the exemplary embodiment above is implemented.

[0189] In an exemplary embodiment, a method for generating a bitstream may be provided. The method includes: generating a bitstream according to the video encoding method as described in the exemplary embodiment above.

[0190] In an exemplary embodiment, a computer program product may also be provided, having instructions for storing a bit stream, the bit stream comprising: video data generated according to the video encoding method as described in the above exemplary embodiments.

[0191] Fig.13 A computing environment 1310 is shown coupled to a user interface 1350. The computing environment 1310 may be part of a data processing server. The computing environment 1310 includes a processor 1320, a memory 1330, and an input / output (I / O) interface 1340.

[0192] The processor 1320 generally controls the overall operation of the computing environment 1310, such as operations associated with display, data acquisition, data communication, and image processing. The processor 1320 may include one or more processors for executing instructions to perform all or some of the steps in the above method. In addition, the processor 1320 may include one or more modules that facilitate the interaction between the processor 1320 and other components. The processor may be a central processing unit (CPU), a microprocessor, a single-chip microcomputer, a graphics processing unit (GPU), etc.

[0193] The memory 1330 is configured to store various types of data to support the operation of the computing environment 1310. The memory 1330 may include predetermined software 1332. Examples of such data include instructions for any application or method operating on the computing environment 1310, video data sets, image data, etc. The memory 1330 may be implemented by using any type of volatile or non-volatile memory device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk.

[0194] I / O interface 1340 provides an interface between processor 1320 and peripheral interface modules (e.g., keyboard, click wheel, buttons, etc.). Buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. I / O interface 1340 may be coupled to an encoder and a decoder.

[0195] In an embodiment, a non-transitory computer-readable storage medium including, for example, a plurality of programs in a memory 1330 and / or storing a bit stream generated by the above encoding method or a bit stream to be decoded by the above decoding method is also provided, wherein the plurality of programs can be executed by a processor 1320 in a computing environment 1310 to perform the above method. In one example, the plurality of programs can be executed by a processor 1320 in a computing environment 1310 to (for example, from Figure 2The video encoder 20 in the computing environment 1310 receives a bit stream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or one or more associated syntax elements, etc.), and can also be executed by the processor 1320 in the computing environment 1310 to perform the above-mentioned decoding method according to the received bit stream or data stream. In another example, the multiple programs can be executed by the processor 1320 in the computing environment 1310 to perform the above-mentioned encoding method to encode the video information (e.g., video blocks representing video frames, and / or one or more associated syntax elements, etc.) into a bit stream or data stream, and can also be executed by the processor 1320 in the computing environment 1310 to (e.g., to Figure 3 Alternatively, a non-transitory computer-readable storage medium may store the bit stream or data stream generated by the encoder (e.g., Figure 2 The video encoder 20 in FIG. 1 generates a video signal for a decoder (eg, Figure 3 A bitstream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or associated one or more syntax elements, etc.) used by the video decoder 30 in the video decoder 30 when decoding video data. The non-transitory computer-readable storage medium can be, for example, a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0196] In an embodiment, a bit stream generated by the above encoding method or a bit stream to be decoded by the above decoding method is provided. In an embodiment, a bit stream including encoded video information generated by the above encoding method or encoded video information to be decoded by the above decoding method is provided.

[0197] In an embodiment, a computing device is also provided, which includes: one or more processors (e.g., processor 1320); and a non-transitory computer-readable storage medium or memory 1330 in which multiple programs that can be executed by the one or more processors are stored, wherein the one or more processors are configured to perform the above-mentioned method when executing the multiple programs.

[0198] In an embodiment, a computer program product having instructions for storing or transmitting a bitstream is also provided, wherein the bitstream includes the encoded video information generated by the above encoding method or the encoded video information to be decoded by the above decoding method. In an embodiment, a computer program product including, for example, a plurality of programs in a memory 1330 is also provided, wherein the plurality of programs can be executed by a processor 1320 in a computing environment 1310 for performing the above method. For example, the computer program product may include a non-transitory computer-readable storage medium.

[0199] In an embodiment, the computing environment 1310 may be implemented by one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs, GPUs, controllers, microcontrollers, microprocessors, or other electronic components for performing the above methods.

[0200] In an embodiment, a method for storing a bit stream is also provided, comprising: storing the bit stream on a digital storage medium, wherein the bit stream comprises encoded video information generated by the above encoding method or encoded video information to be decoded by the above decoding method.

[0201] In an embodiment, a method for transmitting a bit stream generated by the above encoder is also provided. In an embodiment, a method for receiving a bit stream to be decoded by the above decoder is also provided.

[0202] The description of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or limited to the present disclosure. Many modifications, variations, and alternative embodiments will be apparent to one of ordinary skill in the art having the benefit of the teachings presented in the foregoing description and the associated drawings.

[0203] Unless otherwise specifically stated, the order of steps of the method according to the present disclosure is intended to be illustrative only, and the steps of the method according to the present disclosure are not limited to the order specifically described above, but can be changed according to actual conditions. In addition, at least one of the steps of the method according to the present disclosure can be adjusted, combined or deleted according to actual needs.

[0204] The examples are chosen and described in order to explain the principles of the present disclosure and to enable other persons skilled in the art to understand the various embodiments of the present disclosure and to best utilize the basic principles and various embodiments with various modifications suitable for the specific use contemplated. Therefore, it will be understood that the scope of the present disclosure is not limited to the specific examples of the disclosed embodiments, and modifications and other embodiments are intended to be included within the scope of the present disclosure.

[0205] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0206] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video encoding method, characterized in that: The video encoding method comprises: Segment the video to be encoded to obtain multiple video segments; For each video segment, determining a segment encoding bit rate of each video segment according to the encoding complexity of each video segment and the encoding complexity of the video to be encoded; The video frames in each video segment are encoded based on the segment encoding bit rate of each video segment.

2. The video encoding method according to claim 1, characterized in that The video to be encoded is segmented to obtain multiple video segments, including: Determine an intra-frame complexity of a first video frame in a video to be encoded and an inter-frame complexity of the first video frame relative to a second video frame, wherein the second video frame precedes the first video frame; Determining whether the first video frame is a transition frame according to the inter-frame complexity and the intra-frame complexity of the first video frame; In response to the first video frame being a transition frame, the to-be-encoded video is segmented based on the transition frame to obtain the multiple video segments.

3. The video encoding method according to claim 2, characterized in that: The determining, according to the inter-frame complexity and the intra-frame complexity of the first video frame, whether the first video frame is a transition frame includes: In response to a complexity difference between the inter-frame complexity and the intra-frame complexity of the first video frame exceeding a difference threshold, the first video frame is determined to be a transition frame.

4. The video encoding method according to claim 1, characterized in that: The step of determining, for each video segment, a segment encoding bit rate of each video segment according to the encoding complexity of each video segment and the encoding complexity of the video to be encoded, includes: According to the coding complexity of the video segment and the coding complexity of the video to be encoded, the original coding bit rate of the video to be encoded is adjusted to determine the segment coding bit rate of each video segment.

5. The video encoding method according to claim 4, characterized in that: The adjusting the original encoding bit rate of the video to be encoded to determine the segment encoding bit rate of each video segment includes: According to the coding complexity of the video segment, the coding complexity of the video to be coded and the offset coefficient, the original coding rate of the video to be coded is adjusted to determine the segment coding rate of each video segment, The offset coefficient represents the degree of deviation of the average encoding rate of the to-be-encoded video compared with the original encoding rate after the encoding rate of each video segment is adjusted.

6. The video encoding method according to claim 5, characterized in that: The step of adjusting the original encoding bit rate of the video to be encoded to determine the segment encoding bit rate of each video segment according to the encoding complexity of the video segment, the encoding complexity of the video to be encoded, and the offset coefficient includes: Determine a first value according to the coding complexity of all video frames in the current video segment, the coding complexity of all video frames in the video to be encoded, and the offset coefficient; Based on the first value and the original encoding bit rate, a segment encoding bit rate of each video segment is determined.

7. The video encoding method according to claim 1, characterized in that: The coding complexity of the video segment is determined by: Determining a coding complexity of each video frame within the video segment; Determine the normalized complexity of each video frame based on the minimum complexity and the maximum complexity of the encoding complexities of each video frame in the video segment, a first coefficient and a second coefficient, wherein the relationship between the first coefficient and the second coefficient is used to control the difference between the segment encoding bit rates of each video segment; The coding complexity of the video segment is determined based on the normalized complexity of each video frame in the video segment.

8. The video encoding method according to any one of claims 1 to 7, characterized in that: The video encoding method further comprises: In response to an actual encoding codeword of the current video segment being less than a predicted encoding codeword of the current video segment, a segment encoding bitrate of a video segment subsequent to the current video segment is increased.

9. The video encoding method according to claim 8, characterized in that: The step of increasing the segment encoding bit rate of the video segment after the current video segment comprises: Determining a remaining codeword based on a difference between an actual encoding codeword of the current video segment and a predicted encoding codeword of the current video segment; Determining a rate offset for the subsequent video segment based on the remaining codewords, the number of subsequent video frames to be encoded in the video to be encoded, and the number of frames of the subsequent video segment; Based on the rate offset and the segment encoding rate of the subsequent video segment, an increased segment encoding rate for the subsequent video segment is obtained.

10. A video encoding device, characterized in that: The video encoding device comprises: A segmentation unit is configured to segment the video to be encoded to obtain a plurality of video segments; A determination unit configured to determine, for each video segment, a segment encoding bit rate of each video segment according to the encoding complexity of each video segment and the encoding complexity of the video to be encoded; The encoding unit is configured to encode the video frames in each video segment based on the segment encoding bit rate of each video segment.

11. An electronic device, characterized in that: The electronic device comprises: Processor; and a memory for storing processor executable instructions, When the processor executable instructions are executed by the processor, the processor executes the video encoding method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the video encoding method according to any one of claims 1 to 9.

13. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the video encoding method according to any one of claims 1 to 9 is implemented.

14. A method for generating a bit stream, characterized in that: include: A bit stream is generated according to the video encoding method according to any one of claims 1 to 9.

15. A computer program product having instructions for storing a bit stream, characterized in that The bit stream includes: video data generated according to the video encoding method according to any one of claims 1 to 9.

Images