Method and device for determining frame type, and video coding method and device

Abstract

The invention relates to a method and a device for determining a frame type, and a video coding method and a video coding device. The method for determining the frame type comprises the following steps: determining a first residual image between a reference image frame and a predicted image frame obtained by predicting a current image frame by referring to the reference image frame; determining a quantization factor based on the first residual image, wherein the quantization factor indicates the influence of quantization processing in prediction on the reference cost of predicting the current image frame with reference to the reference image frame; determining a reference cost based on the quantization factor and the first residual image; and determining the frame type of the current image frame based on the reference cost. According to the method and the device for determining the frame type disclosed by the invention, the determined frame type is more accurate. The video coding method comprises the following steps: determining the frame type of an image frame in a video through the method for determining the frame type; the image frames in the video are coded based on the frame types of the image frames in the video, so that the video coding effect is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of video technology, and more specifically, to a method and device for determining a frame type, and a video encoding method and device. Background Art

[0002] Various electronic devices (e.g., digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smart phones, video teleconferencing devices, video streaming devices, etc.) support digital video. Electronic devices send and receive or otherwise transmit digital video data through a communication network, and / or store digital video data on a storage device. Due to the limited bandwidth capacity of the communication network and the limited storage resources of the storage device, before the video data is transmitted or stored, the video data can be compressed using a video codec according to one or more video codec standards. For example, video codec standards include general video codec (VVC), joint exploration test model (JEM), high efficiency video codec (HEVC / H.265), advanced video codec (AVC / H.264), moving picture expert group (MPEG) codec, etc. Video codecs generally use prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that utilize the inherent redundancy in video data. Video codecs are intended to compress video data into a form that uses a lower bit rate while avoiding or minimizing the degradation of video quality.

[0003] In the field of video coding, frames to be coded are divided into intra-frame coded frames, forward-predictive coded frames, and bidirectional-predictive coded frames according to the prediction method used for each frame of the video. Different frame types will affect the video compression efficiency due to the different coding methods adopted, so the determination of frame type becomes an important factor affecting coding efficiency. Frame type determination technology adaptively determines the type of image frame in the video according to the characteristics of the video content, and plays an important role in the encoder.

[0004] Open source encoders x264 and x265 usually traverse multiple frame types based on the Viterbi algorithm, calculate the reference cost of the image frame, and select the frame type with the smallest reference cost as the frame type of the image frame. However, in open source encoders such as x264 and x265, the impact of any quantization processing on the reference cost is not considered during the process of determining the frame type, resulting in low accuracy in determining the frame type. Summary of the invention

[0005] Exemplary embodiments of the present disclosure provide a method and apparatus for determining a frame type, so as to at least solve the problem of determining a frame type in the related art.

[0006] According to an exemplary embodiment of the present disclosure, a method for determining a frame type is provided, comprising: determining a first residual image between a reference image frame and a predicted image frame, wherein the predicted image frame is obtained by predicting a current image frame with reference to the reference image frame; determining a quantization factor based on the first residual image, wherein the quantization factor indicates an influence of a quantization process in the prediction on a reference cost for predicting the current image frame with reference to the reference image frame; determining the reference cost based on the quantization factor and the first residual image; and determining a frame type of the current image frame based on the reference cost.

[0007] Optionally, determining the quantization factor based on the first residual image may include: determining a first reference cost and a second reference cost for predicting the current image frame with reference to the reference image frame based on the first residual image, wherein the first reference cost is a reference cost determined by ignoring the influence of the quantization processing in the prediction on the reference cost, and the second reference cost is a reference cost determined by considering the influence of the quantization processing in the prediction on the reference cost; determining the quantization factor based on the first reference cost and the second reference cost.

[0008] Optionally, determining the first reference cost and the second reference cost for predicting the current image frame with reference to the reference image frame based on the first residual image may include: determining the first reference cost based on the first residual image; obtaining the second residual image by quantizing the first residual image; and determining the second reference cost based on the second residual image.

[0009] Optionally, determining the quantization factor based on the first reference cost and the second reference cost may include: determining the quantization factor based on a ratio of the second reference cost to the first reference cost.

[0010] Optionally, determining the quantization factor based on the first reference cost and the second reference cost may include: determining the quantization factor based on the first reference cost and the second reference cost and a first preset value and a second preset value.

[0011] Optionally, determining the quantization factor based on the first reference cost and the second reference cost and the first preset value and the second preset value may include: determining a first sum of the first reference cost and the first preset value, and a second sum of the second reference cost and the second preset value; and determining the quantization factor based on the ratio of the second sum to the first sum.

[0012] Optionally, determining the reference cost based on the quantization factor and the first residual image may include: determining the reference cost based on the quantization factor and a first reference cost.

[0013] Optionally, determining the reference cost based on the quantization factor and the first reference cost may include: determining the reference cost based on the quantization factor, video characteristic related parameters and the first reference cost, wherein the video characteristic related parameters are determined based on the characteristics of the video in which the current image frame is located.

[0014] Optionally, determining the reference cost based on the quantization factor, video characteristic related parameters and a first reference cost may include: determining the sum of the video characteristic related parameters of the reference image frame and the quantization factor; and determining the reference cost based on the product of the sum and the first reference cost.

[0015] Optionally, the step of obtaining the second residual image by performing quantization processing on the first residual image may include: for each candidate frame type among multiple candidate frame types of the current image frame, performing quantization processing corresponding to each candidate frame type among the multiple candidate frame types on the first residual image, so as to obtain a second residual image corresponding to each candidate frame type among the multiple candidate frame types, wherein the quantization processing may include a transform operation, a quantization operation, an inverse quantization operation, and an inverse transform operation, wherein the quantization operation and the inverse quantization operation may be different according to the candidate frame type.

[0016] Optionally, determining the second reference cost based on the second residual image may include: for each candidate frame type in the multiple candidate frame types, determining the second reference cost corresponding to each candidate frame type in the multiple candidate frame types based on the second residual image corresponding to each candidate frame type in the multiple candidate frame types.

[0017] Optionally, determining the quantization factor based on the first reference cost and the second reference cost may include: determining the quantization factor corresponding to each of the multiple candidate frame types based on the first reference cost and the second reference cost corresponding to each of the multiple candidate frame types, respectively.

[0018] Optionally, determining the reference cost based on the quantization factor and the first residual image may include: determining the reference cost corresponding to each of the multiple candidate frame types based on the quantization factor corresponding to each of the multiple candidate frame types and the first residual image, respectively, wherein determining the frame type of the current image frame based on the reference cost may include: determining the frame type of the current image frame from the multiple candidate frame types based on the reference cost corresponding to each of the multiple candidate frame types.

[0019] Optionally, the reference image frames of the multiple image frames including the current image frame are the same predetermined image frame.

[0020] According to an exemplary embodiment of the present disclosure, there is provided an apparatus for determining a frame type, comprising: a residual determination unit, configured to determine a first residual image between a reference image frame and a predicted image frame, wherein the predicted image frame is obtained by predicting a current image frame with reference to the reference image frame; a quantization factor determination unit, configured to determine a quantization factor based on the first residual image, wherein the quantization factor indicates an influence of a quantization process in the prediction on a reference cost for predicting the current image frame with reference to the reference image frame; a cost determination unit, configured to determine the reference cost based on the quantization factor and the first residual image; and a frame type determination unit, configured to determine the frame type of the current image frame based on the reference cost.

[0021] Optionally, the quantization factor determination unit may be configured to: determine, based on the first residual image, a first reference cost and a second reference cost for predicting the current image frame with reference to the reference image frame, wherein the first reference cost is a reference cost determined by ignoring the influence of the quantization processing in the prediction on the reference cost, and the second reference cost is a reference cost determined by considering the influence of the quantization processing in the prediction on the reference cost; determine the quantization factor based on the first reference cost and the second reference cost.

[0022] Optionally, the quantization factor determination unit may be configured to: determine a first reference cost based on a first residual image; obtain a second residual image by quantizing the first residual image; and determine a second reference cost based on the second residual image.

[0023] Optionally, the quantization factor determination unit may be configured to: determine the quantization factor based on a ratio of a second reference cost to a first reference cost.

[0024] Optionally, the quantization factor determination unit may be configured to determine the quantization factor based on a first reference cost and a second reference cost and a first preset value and a second preset value.

[0025] Optionally, the quantization factor determination unit may be configured to: determine a first sum of a first reference cost and a first preset value, and a second sum of a second reference cost and a second preset value; and determine the quantization factor based on a ratio of the second sum to the first sum.

[0026] Optionally, the cost determination unit may be configured to: determine the reference cost based on the quantization factor and a first reference cost.

[0027] Optionally, the cost determination unit may be configured to determine the reference cost based on the quantization factor, video characteristic related parameters and a first reference cost, wherein the video characteristic related parameters are determined based on characteristics of the video in which the current image frame is located.

[0028] Optionally, the cost determination unit may be configured to: determine a sum of a video characteristic related parameter of a reference image frame and the quantization factor; and determine the reference cost based on a product of the sum and a first reference cost.

[0029] Optionally, the quantization factor determination unit may be configured to: for each candidate frame type among a plurality of candidate frame types of the current image frame, perform quantization processing corresponding to each of the plurality of candidate frame types on the first residual image, respectively, to obtain a second residual image corresponding to each of the plurality of candidate frame types, wherein the quantization processing may include a transform operation, a quantization operation, an inverse quantization operation, and an inverse transform operation, wherein the quantization operation and the inverse quantization operation may be different according to the candidate frame type.

[0030] Optionally, the quantization factor determination unit may be configured to: for each candidate frame type among the multiple candidate frame types, determine a second reference cost corresponding to each candidate frame type among the multiple candidate frame types based on a second residual image corresponding to each candidate frame type among the multiple candidate frame types.

[0031] Optionally, the quantization factor determination unit may be configured to determine a quantization factor corresponding to each of the multiple candidate frame types based on a first reference cost and a second reference cost corresponding to each of the multiple candidate frame types, respectively.

[0032] Optionally, the cost determination unit may be configured to: determine the reference cost corresponding to each of the multiple candidate frame types based on the quantization factor corresponding to each of the multiple candidate frame types and the first residual image, respectively, wherein the frame type determination unit may be configured to: determine the frame type of the current image frame from the multiple candidate frame types based on the reference cost corresponding to each of the multiple candidate frame types.

[0033] Optionally, the reference image frames of the multiple image frames including the current image frame are the same predetermined image frame.

[0034] According to an exemplary embodiment of the present disclosure, a video encoding method is provided, comprising: determining a frame type of an image frame in a video by the method for determining a frame type of the present disclosure; and encoding the image frame in the video based on the frame type of the image frame in the video.

[0035] According to an exemplary embodiment of the present disclosure, a video encoding device is provided, including: a frame type acquisition unit, configured to determine the frame type of an image frame in a video by using the method for determining a frame type of the present disclosure; and an image frame encoding unit, configured to encode the image frame in the video based on the frame type of the image frame in the video.

[0036] According to an exemplary embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a method for determining a frame type or a video encoding method according to an exemplary embodiment of the present disclosure.

[0037] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor of an electronic device, the electronic device executes the method for determining a frame type or the video encoding method according to the exemplary embodiment of the present disclosure.

[0038] According to an exemplary embodiment of the present disclosure, a computer program product is provided, including a computer program / instruction. When the computer program / instruction is executed by a processor, the method for determining a frame type or a video encoding method according to an exemplary embodiment of the present disclosure is implemented.

[0039] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0040] By determining a first residual image between a reference image frame and a predicted image frame obtained by predicting the current image frame with reference to the reference image frame, and determining a quantization factor based on the first residual image, wherein the quantization factor indicates the influence of a quantization process in the prediction on a reference cost for predicting the current image frame with reference to the reference image frame, and by determining the reference cost based on the quantization factor and the first residual image, determining a frame type of the current image frame based on the reference cost, thereby making the determined frame type more accurate by introducing the quantization factor indicating the influence of a quantization process in the prediction on the reference cost for predicting the current image frame with reference to the reference image frame;

[0041] In addition, the frame types of image frames in a video are determined by using the method for determining frame types disclosed in the present invention, and the image frames in the video are encoded based on the frame types of the image frames in the video. Therefore, since more accurate frame types of the image frames are obtained by using the method for determining frame types disclosed in the present invention, more accurate frame types can be used to improve video encoding effects.

[0042] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0044] Figure 1 is a block diagram illustrating an exemplary system for encoding and decoding video blocks according to some embodiments of the present disclosure.

[0045] Figure 2 is a block diagram illustrating an exemplary video encoder according to some embodiments of the present disclosure.

[0046] Figure 3 is a block diagram illustrating an exemplary video decoder according to some embodiments of the present disclosure.

[0047] Figure 4 A flowchart of a method for determining a frame type according to an exemplary embodiment of the present disclosure is shown.

[0048] Figure 5 A flowchart of a video encoding method according to an exemplary embodiment of the present disclosure is shown.

[0049] Figure 6 A block diagram of an apparatus for determining a frame type according to an exemplary embodiment of the present disclosure is shown.

[0050] Figure 7 A block diagram of a video encoding apparatus according to an exemplary embodiment of the present disclosure is shown.

[0051] Figure 8 is a diagram illustrating a computing environment coupled with a user interface according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0052] Reference will now be made in detail to specific embodiments, examples of which are shown in the accompanying drawings. In the following detailed description, a number of non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, various alternatives may be used without departing from the scope of the claims, and the subject matter may be practiced without these specific details. For example, the subject matter presented herein may be implemented on many types of electronic devices having digital video capabilities.

[0053] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the drawings are used to distinguish objects, and are not used to describe any specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those shown in the drawings or described in the present disclosure. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0054] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0055] Figure 1 is a block diagram illustrating an exemplary system 10 for encoding and decoding video blocks in parallel according to some embodiments of the present disclosure. Figure 1 As shown in , system 10 includes a source device 12 that generates and encodes video data to be later decoded by a target device 14. Source device 12 and target device 14 may include any of a wide variety of electronic devices, including cloud servers, server computers, desktop or laptop computers, tablet computers, smart phones, set-top boxes, digital televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some implementations, source device 12 and target device 14 are equipped with wireless communication capabilities.

[0056] In some implementations, target device 14 may receive the encoded video data to be decoded via link 16. Link 16 may include any type of communication medium or device capable of moving the encoded video data from source device 12 to target device 14.

[0057] In some other embodiments, the encoded video data may be sent from the output interface 22 to the storage device 32. The encoded video data in the storage device 32 may then be accessed by the target device 14 via the input interface 28.

[0058] like Figure 1, source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Video source 18 may include sources such as, or a combination of such sources: a video capture device (e.g., a camera), a video archive containing previously captured video, a video feed interface for receiving video from a video content provider, and / or a computer graphics system for generating computer graphics data as source video.

[0059] The captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video data may be sent directly to target device 14 via output interface 22 of source device 12. The encoded video data may also (or alternatively) be stored on storage device 32 for later access by target device 14 or other devices for decoding and / or playback.

[0060] Target device 14 includes input interface 28, video decoder 30, and display device 34. Input interface 28 may include a receiver and / or a modem and receives encoded video data via link 16. The encoded video data transmitted via link 16 or provided on storage device 32 may include various syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements may be included within the encoded video data sent over a communication medium, stored on a storage medium, or stored on a file server.

[0061] The video encoder 20 and the video decoder 30 can operate according to a proprietary standard or an industry standard (e.g., VVC, HEVC, Part 10 of MPEG-4, AVC) or an extension of such a standard. It should be understood that the present application is not limited to a specific video encoding / decoding standard and can be applied to other video encoding / decoding standards. It is generally believed that the video encoder 20 of the source device 12 can be configured to encode the video data according to any of these current standards or future standards. Similarly, it is also generally believed that the video decoder 30 of the target device 14 can be configured to decode the video data according to any of these current standards or future standards.

[0062] The video encoder 20 and the video decoder 30 may be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic devices, software, hardware, firmware, or any combination thereof. When partially implemented in software, the electronic device may store instructions for the software in a suitable non-transitory computer-readable medium, and use one or more processors to execute the instructions in the hardware to perform the video encoding / decoding operations disclosed in the present disclosure. Each of the video encoder 20 and the video decoder 30 may be included in one or more encoders or decoders, and either of the encoders or decoders may be integrated as part of a combined encoder / decoder (CODEC) in the corresponding device.

[0063] Figure 2 is a block diagram illustrating an exemplary video encoder 20 according to some embodiments described in the present application. The video encoder 20 can perform intra-frame prediction coding and inter-frame prediction coding on video blocks within a video frame. Intra-frame prediction coding relies on spatial prediction to reduce or remove spatial redundancy in video data within a given video frame or picture. Inter-frame prediction coding relies on temporal prediction to reduce or remove temporal redundancy in video data within adjacent video frames or pictures of a video sequence. It should be noted that in the field of video coding and decoding, the term "frame" can be used as a synonym for the term "image" or "picture".

[0064] like Figure 2As shown in , the video encoder 20 includes a video data memory 40, a prediction processing unit 41, a decoded picture buffer (DPB) 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The prediction processing unit 41 further includes a motion estimation unit 42, a motion compensation unit 44, a segmentation unit 45, an intra-frame prediction processing unit 46, and an intra-frame block copy (BC) unit 48. In some embodiments, the video encoder 20 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and an adder 62 for video block reconstruction. A loop filter 63 such as a deblocking filter can be located between the adder 62 and the DPB 64 to filter the block boundary to remove the block effect from the reconstructed video. In addition to the deblocking filter, another loop filter (e.g., a sample adaptive offset (SAO) filter, a cross-component sample adaptive offset (CCSAO) filter, and / or an adaptive loop filter (ALF)) can also be used to filter the output of the adder 62. In some examples, the loop filter may be omitted, and the decoded video blocks may be provided directly by adder 62 to DPB 64. Video encoder 20 may take the form of a fixed or programmable hardware unit, or may be distributed among one or more of the illustrated fixed or programmable hardware units.

[0065] Video data memory 40 may store video data to be encoded by components of video encoder 20. Figure 1 Video source 18 is shown obtaining video data in video data memory 40. DPB 64 is a buffer that stores reference video data (eg, reference frames or pictures) for use by video encoder 20 in encoding the video data (eg, in intra or inter prediction coding modes).

[0066] like Figure 2 As shown in , after receiving the video data, the segmentation unit 45 within the prediction processing unit 41 segments the video data into video blocks. This segmentation may also include segmenting the video frame into strips, tiles (e.g., a set of video blocks), or other larger coding units (CUs) according to a predefined splitting structure associated with the video data, such as a quadtree (QT) structure. It should be noted that the terms "block" or "video block" used herein may be a portion of a frame or picture, in particular a rectangular (square or non-square) portion. With reference to, for example, HEVC and VVC, a block or video block may be or correspond to a coding tree unit (CTU), a CU, a prediction unit (PU), or a transform unit (TU) and / or may be or correspond to a corresponding block (e.g., a coding tree block (CTB), a coding block (CB), a prediction block (PB), or a transform block (TB)) and / or a sub-block.

[0067] The prediction processing unit 41 may select one of a plurality of feasible prediction coding modes for the current video block based on the error results (e.g., coding rate and distortion level), such as one of one or more inter-frame prediction coding modes in a plurality of intra-frame prediction coding modes. The prediction processing unit 41 may provide the resulting intra-frame prediction coding block or inter-frame prediction coding block to the adder 50 to generate a residual block, and to the adder 62 to reconstruct the coding block for subsequent use as part of a reference frame. The prediction processing unit 41 also provides syntax elements (e.g., motion vectors, intra-frame mode indicators, partition information, and other such syntax information) to the entropy coding unit 56.

[0068] To select an appropriate intra-prediction coding mode for the current video block, intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-prediction coding of the current video block in relation to one or more neighboring blocks in the same frame as the current block to be encoded to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 may perform inter-prediction coding of the current video block in relation to one or more prediction blocks in one or more reference frames to provide temporal prediction. Video encoder 20 may perform multiple encoding passes, for example, to select an appropriate coding mode for each block of video data.

[0069] In some embodiments, motion estimation unit 42 determines an inter-prediction mode for a current video frame by generating a motion vector according to a predetermined pattern within a sequence of video frames, the motion vector indicating the displacement of a video block within the current video frame relative to a prediction block within a reference video frame. Motion estimation performed by motion estimation unit 42 is the process of generating a motion vector that estimates the motion of a video block. For example, a motion vector may indicate the displacement of a video block within a current video frame or picture relative to a prediction block within a reference frame associated with a current block being encoded within the current frame. The predetermined pattern may designate a video frame in a sequence as a P frame or a B frame. Intra BC unit 48 may determine a vector (e.g., a block vector) for intra BC coding in a manner similar to the motion vector determined by motion estimation unit 42 for inter prediction, or may utilize motion estimation unit 42 to determine a block vector.

[0070] Regardless of whether the prediction block is from the same frame according to intra-frame prediction or from a different frame according to inter-frame prediction, the video encoder 20 can form a residual video block by subtracting the pixel values ​​of the prediction block from the pixel values ​​of the current video block being encoded. The pixel difference values ​​forming the residual video block may include both luma component differences and chroma component differences.

[0071] Intra-prediction processing unit 46 may encode the current block using various intra-prediction modes, for example, during separate encoding passes, and intra-prediction processing unit 46 (or, in some examples, mode selection unit) may select an appropriate intra-prediction mode to use from the tested intra-prediction modes. Intra-prediction processing unit 46 may provide information indicating the intra-prediction mode selected for the block to entropy encoding unit 56. Entropy encoding unit 56 may encode the information indicating the selected intra-prediction mode into the bitstream.

[0072] After prediction processing unit 41 determines a prediction block for the current video block via inter-prediction or intra-prediction, adder 50 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block may be included in one or more TUs and provided to transform processing unit 52. Transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform (e.g., a discrete cosine transform (DCT) or a conceptually similar transform).

[0073] The transform processing unit 52 may send the resulting transform coefficients to a quantization unit 54. The quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may also reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, the quantization unit 54 may then perform a scan on the matrix including the quantized transform coefficients. Alternatively, the entropy coding unit 56 may perform the scan.

[0074] After quantization, entropy coding unit 56 entropy encodes the quantized transform coefficients into a video bitstream using, for example, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding method or technique. The encoded bitstream may then be sent to a video bitstream such as a Figure 1 The video decoder 30 shown, or archived as Figure 1 The video frame may be stored in storage device 32 as shown for later transmission to or retrieval by video decoder 30. Entropy encoding unit 56 may also entropy encode motion vectors and other syntax elements for the current video frame being encoded.

[0075] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual video block in the pixel domain for use in generating reference blocks for predicting other video blocks. As noted above, motion compensation unit 44 may generate a motion compensated prediction block from one or more reference blocks of a frame stored in DPB 64. Motion compensation unit 44 may also apply one or more interpolation filters to the prediction block to calculate sub-integer pixel values ​​for use in motion estimation.

[0076] Adder 62 adds the reconstructed residual block to the motion compensated prediction block generated by motion compensation unit 44 to generate a reference block for storage in DPB 64. The reference block may then be used as a prediction block by intra BC unit 48, motion estimation unit 42, and motion compensation unit 44 to inter-predict another video block in a subsequent video frame.

[0077] Figure 3 1 is a block diagram showing an exemplary video decoder 30 according to some embodiments of the present application. The video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, and a DPB 92. The prediction processing unit 81 further includes a motion compensation unit 82, an intra-frame prediction unit 84, and an intra-frame BC unit 85. The video decoder 30 may perform the above-mentioned Figure 2 The encoding process is substantially the inverse of the decoding process described with respect to video encoder 20. For example, motion compensation unit 82 may generate prediction data based on motion vectors received from entropy decoding unit 80, and intra-prediction unit 84 may generate prediction data based on intra-prediction mode indicators received from entropy decoding unit 80.

[0078] In some examples, embodiments of the present disclosure may be dispersed in one or more of the units of the video decoder 30. For example, the intra BC unit 85 may perform embodiments of the present application alone or in combination with other units of the video decoder 30, such as the motion compensation unit 82, the intra prediction unit 84, and the entropy decoding unit 80. In some examples, the video decoder 30 may not include the intra BC unit 85, and the functions of the intra BC unit 85 may be performed by other components of the prediction processing unit 81, such as the motion compensation unit 82.

[0079] Video data memory 79 may store video data, such as an encoded video bitstream, to be decoded by other components of video decoder 30. Video data stored in video data memory 79 may be obtained, for example, from storage device 32, from a local video source (e.g., a camera), via wired or wireless network communication of video data, or by accessing a physical data storage medium (e.g., a flash drive or hard disk).

[0080] During the decoding process, the video decoder 30 receives an encoded video bitstream representing video blocks of an encoded video frame and associated syntax elements. The entropy decoding unit 80 of the video decoder 30 entropy decodes the bitstream to generate quantization coefficients, motion vectors or intra-frame prediction mode indicators, and other syntax elements. The entropy decoding unit 80 then forwards the motion vectors or intra-frame prediction mode indicators, and other syntax elements to the prediction processing unit 81.

[0081] When a video frame is encoded as an intra-prediction coded (I) frame or for intra-coded prediction blocks in other types of frames, the intra-prediction unit 84 of the prediction processing unit 81 can generate prediction data for a video block of the current video frame based on the intra-prediction mode transmitted by the signal and reference data from a previously decoded block of the current frame.

[0082] When the video frame is encoded as an inter-frame prediction coding (i.e., B or P) frame, the motion compensation unit 82 of the prediction processing unit 81 generates one or more prediction blocks for the video block of the current video frame based on the motion vector and other syntax elements received from the entropy decoding unit 80. Each of the prediction blocks can be generated from a reference frame in one of the reference frame lists. The video decoder 30 can construct the reference frame lists, i.e., List 0 and List 1, based on the reference frames stored in the DPB 92 using a default construction technique.

[0083] In some examples, when a video block is encoded according to the intra BC mode described herein, intra BC unit 85 of prediction processing unit 81 generates a prediction block for the current video block based on the block vector and other syntax elements received from entropy decoding unit 80. The prediction block may be within a reconstructed region of the same picture as the current video block as defined by video encoder 20.

[0084] Motion compensation unit 82 and / or intra BC unit 85 determine prediction information for a video block of the current video frame by parsing motion vectors and other syntax elements, and then use the prediction information to generate a prediction block for the current video block being decoded.

[0085] Motion compensation unit 82 may also perform interpolation using interpolation filters as used by video encoder 20 during encoding of the video blocks to calculate interpolated values ​​for sub-integer pixels of reference blocks. In this case, motion compensation unit 82 may determine the interpolation filters used by video encoder 20 based on the received syntax elements and use these interpolation filters to produce the prediction blocks.

[0086] Inverse quantization unit 86 inverse quantizes the quantized transform coefficients provided in the bitstream and entropy decoded by entropy decoding unit 80, using the same quantization parameters calculated by video encoder 20 for each video block in the video frame to determine the degree of quantization. Inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients to reconstruct the residual block in the pixel domain.

[0087] After the motion compensation unit 82 or the intra BC unit 85 generates a prediction block for the current video block based on the vector and other syntax elements, the adder 90 reconstructs the decoded video block for the current video block by adding the residual block from the inverse transform processing unit 88 to the corresponding prediction block generated by the motion compensation unit 82 and the intra BC unit 85. A loop filter 91 (e.g., a deblocking filter, an SAO filter, a CCSAO filter, and / or an ALF) may be located between the adder 90 and the DPB 92 to further process the decoded video block. In some examples, the loop filter 91 may be omitted, and the decoded video block may be provided directly to the DPB 92 by the adder 90. The decoded video blocks in a given frame are then stored in the DPB 92, which stores reference frames for subsequent motion compensation of the next video block. The DPB 92 or a memory device separate from the DPB 92 may also store the decoded video for later presentation on a display device (e.g., Figure 1 on a display device 34).

[0088] The related method determines the frame type by calculating the reference cost (also called rate-distortion cost) of the image frame of the video. However, the related method does not consider the quantization process in the process of predicting another image frame with reference to one image frame, resulting in the related method being unable to accurately estimate the reference cost and thus unable to accurately determine the frame type.

[0089] As an example, in the process of determining the frame type of the x264 / x265 encoder, two frames f i and f j Between j Reference i The reference cost cost is calculated as cost = D(f resi ),in, Here, f resi represents the residual image, D(·) is used to measure the size of the residual, usually calculated using the sum of absolute transformed differences (SATD); f j Based on f i The predicted frame.

[0090] The present disclosure introduces a quantization factor indicating the influence of the quantization process in the prediction on the reference cost of predicting the current image frame with reference to the reference image frame when determining the reference cost, so that the frame type can be accurately determined. In addition, the present disclosure can further remove the dependence of the determination of the frame type on the encoding of the referenced image frame while considering the quantization process, so that the time complexity introduced is low, thereby improving the encoding performance or compression performance.

[0091] Next, we will refer to Figures 4 to 8 A method and apparatus for determining a frame type according to exemplary embodiments of the present disclosure are described in detail.

[0092] Figure 4 A flowchart of a method for determining a frame type according to an exemplary embodiment of the present disclosure is shown.

[0093] Reference Figure 4 In step S401, a first residual image between a reference image frame and a predicted image frame is determined. Here, the predicted image frame is obtained by predicting the current image frame with reference to the reference image frame. Here, the residual image between the reference image frame and the predicted image frame can be obtained by any method, and the present disclosure does not limit this.

[0094] The reference image frame may generally be an image frame one frame before or multiple image frames before the current image frame, or an image frame one frame after the current image frame. Preferably, in the present disclosure, the reference image frame of multiple image frames may be the same predetermined image frame.

[0095] In an exemplary embodiment of the present disclosure, the reference image frame of the multiple image frames including the current image frame is the same predetermined image frame, so that when determining the frame type, it is not necessary to rely on the encoding of the surrounding image frames of the current image frame. The same predetermined image frame can be the first image frame of the video, an image frame specified in the video, etc., and the present disclosure does not limit this.

[0096] In step S402, a quantization factor is determined based on the first residual image. Here, the quantization factor indicates the influence of the quantization process in the prediction on the reference cost of predicting the current image frame with reference to the reference image frame.

[0097] In an exemplary embodiment of the present disclosure, determining a quantization factor based on a first residual image may include: determining a first reference cost and a second reference cost for predicting the current image frame with reference to the reference image frame based on the first residual image; determining the quantization factor based on the first reference cost and the second reference cost, thereby introducing the quantization factor in the process of determining the frame type. Here, the first reference cost may be a reference cost determined by ignoring the influence of the quantization process in the prediction on the reference cost, and the second reference cost may be a reference cost determined by considering the influence of the quantization process in the prediction on the reference cost.

[0098] In an exemplary embodiment of the present disclosure, determining the first reference cost and the second reference cost for predicting the current image frame with reference to the reference image frame based on the first residual image may include: determining the first reference cost based on the first residual image; obtaining the second residual image by quantizing the first residual image; determining the second reference cost based on the second residual image, thereby obtaining the second reference cost considering the quantization effect in encoding by introducing quantization processing. Here, the second residual image may represent a reconstructed image of the residual image.

[0099] In an exemplary embodiment of the present disclosure, determining a first reference cost based on a first residual image may include: inputting the first residual image into a residual function to obtain a first reference cost. In an exemplary embodiment of the present disclosure, determining a second reference cost based on a second residual image may include: inputting the second residual image into the residual function to obtain a second reference cost. Here, obtaining a reference cost through a residual function is only an example, and the present disclosure is not limited thereto. In an exemplary embodiment of the present disclosure, the reference cost may be obtained in various ways.

[0100] In an exemplary embodiment of the present disclosure, obtaining a second residual image by quantizing the first residual image may include: sequentially performing a transform operation, a quantization operation, an inverse quantization operation, and an inverse transform operation on the first residual image to obtain a second residual image. Here, the quantization process may include a transform operation, a quantization operation, an inverse quantization operation, and an inverse transform operation. In addition, the quantization process here includes a transform operation, a quantization operation, an inverse quantization operation, and an inverse transform operation, which is only an example, and the present disclosure is not limited thereto. Various quantization processes may be performed in the exemplary embodiments of the present disclosure. For example, the transform operation may be various discrete sine and cosine transform operations, and the present disclosure is not limited thereto. For example, the quantization operation may be various quantization operations such as ordinary scalar quantization, rate-distortion optimized quantization (RDOQ), and dependent quantization (DQ), and the present disclosure is not limited thereto.

[0101] In an exemplary embodiment of the present disclosure, determining the quantization factor based on the first reference cost and the second reference cost may include: determining the quantization factor based on the ratio of the second reference cost to the first reference cost, thereby obtaining the quantization factor. As an example, the quantization factor may be obtained by formula To calculate the quantization factor q. Here, f resi represents the residual image, D(·) is used to measure the residual size, Here, T(·) is and T -1 (·) represents the transformation and inverse transformation operations; Q(·) represents the sum Q -1 (·) represents quantization and dequantization operations.

[0102] In an exemplary embodiment of the present disclosure, determining the quantization factor based on the first reference cost and the second reference cost may include: determining the quantization factor based on the first reference cost and the second reference cost and the first preset value and the second preset value, thereby improving the accuracy of the quantization factor.

[0103] In an exemplary embodiment of the present disclosure, determining the quantization factor based on the first reference cost and the second reference cost and the first preset value and the second preset value may include: determining a first sum of the first reference cost and the first preset value, and a second sum of the second reference cost and the second preset value; determining the quantization factor based on the ratio of the second sum to the first sum, thereby improving the accuracy of the quantization factor. The quantization factor calculation method here is only an example, and the present disclosure is not limited thereto. In an exemplary embodiment of the present disclosure, the quantization factor can be obtained by various quantization factor calculation methods. Here, the first preset value and the second preset value may be different, and in addition, the first preset value and the second preset value may also be the same, and the present disclosure is not limited thereto. The first preset value and the second preset value may be respectively set in advance. For example, the first preset value and the second preset value may be preferred values ​​respectively set based on experience. The first preset value and the second preset value may be constants less than or equal to 1, for example, but not limited to, 0.00001-0.0009, 0.00091-0.009, 0.0091-0.09, 0.091-0.9. For example, the first preset value may be 0.001, and the second preset value may be 0.002. For example, the first preset value and the second preset value may both be 0.001. For another example, the first preset value may be 0.002, and the second preset value may be 0.001. For another example, the first preset value and the second preset value may both be 0.002. For another example, the first preset value may be 0.002, and the second preset value may be 0.003. For another example, the first preset value and the second preset value may both be 0.003.

[0104] For example, the first preset value may be 0.01, and the second preset value may be 0.02. For example, the first preset value and the second preset value may both be 0.01. For another example, the first preset value may be 0.02, and the second preset value may be 0.01. For another example, the first preset value and the second preset value may both be 0.02. For another example, the first preset value may be 0.02, and the second preset value may be 0.03. For another example, the first preset value and the second preset value may both be 0.03.

[0105] As an example, the formula Here, a represents the first preset value, b represents the second preset value, and f resi represents the residual image, D(·) is used to measure the residual size, Here, T(·) is and T -1 (·) represents the transformation and inverse transformation operations; Q(·) represents the sum Q -1 (·) represents quantization and dequantization operations.

[0106] As an example, when the first preset value and the second preset value are both 0.001, the formula To calculate the quantization factor q. Here, f resi represents the residual image, D(·) is used to measure the residual size, Here, T(·) is and T -1 (·) represents the transformation and inverse transformation operations; Q(·) represents the sum Q -1 (·) represents the quantization and dequantization operations, and the quantization factor q reflects the effect of quantization. Here, D(·) can be, for example, but not limited to, the absolute transformation error and the calculation. For example, D(f resi ) can represent the first reference cost, A second reference cost may be represented.

[0107] In the present disclosure, various transform operations, various quantization operations, various inverse quantization operations, and various inverse transform operations may be used, and the present disclosure is not limited thereto.

[0108] In the present disclosure, various other calculation methods related to the first reference cost and the second reference cost may also be used to calculate the quantization factor, and the present disclosure does not limit this.

[0109] In an exemplary embodiment of the present disclosure, obtaining a second residual image by performing quantization processing on a first residual image may include: for each candidate frame type in a plurality of candidate frame types of the current image frame, performing quantization processing corresponding to each candidate frame type in the plurality of candidate frame types on the first residual image, respectively, to obtain a second residual image corresponding to each candidate frame type in the plurality of candidate frame types. Here, the quantization processing may include a transform operation, a quantization operation, an inverse quantization operation, and an inverse transform operation, wherein the quantization operation and the inverse quantization operation may be different according to the candidate frame type. The plurality of candidate frame types may include an intra-coded frame, a forward predictive coded frame, or a bidirectional predictive coded frame.

[0110] In an exemplary embodiment of the present disclosure, determining the second reference cost based on the second residual image may include: for each candidate frame type among the multiple candidate frame types, determining the second reference cost corresponding to each candidate frame type among the multiple candidate frame types based on the second residual image corresponding to each candidate frame type among the multiple candidate frame types.

[0111] In an exemplary embodiment of the present disclosure, determining the quantization factor based on the first reference cost and the second reference cost may include: determining the quantization factor corresponding to each of the multiple candidate frame types based on the first reference cost and the second reference cost corresponding to each of the multiple candidate frame types, respectively.

[0112] In step S403, the reference cost is determined based on the quantization factor and the first residual image.

[0113] In an exemplary embodiment of the present disclosure, determining the reference cost based on the quantization factor and the first residual image may include: determining the reference cost based on the quantization factor and the first reference cost, thereby improving the accuracy of the reference cost.

[0114] In an exemplary embodiment of the present disclosure, determining the reference cost based on the quantization factor and the first reference cost may include: determining the reference cost based on the quantization factor, video characteristic related parameters and the first reference cost. In an exemplary embodiment of the present disclosure, the video characteristic related parameters may be determined based on the characteristics of the video in which the current image frame is located, thereby improving the correction effect of the reference cost by selecting appropriate video characteristic related parameters. Video characteristics are characteristics that jointly determine the quality of the video and the viewing experience, and may be, for example, but not limited to, resolution, frame rate, bit rate, etc.

[0115] In an exemplary embodiment of the present disclosure, determining the reference cost based on the quantization factor, the video characteristic related parameter and the first reference cost may include: determining the sum of the video characteristic related parameter and the quantization factor; determining the reference cost based on the product of the sum and the first reference cost, thereby improving the correction effect of the reference cost. The reference cost correction method here is only an example, and the present disclosure is not limited thereto. In the exemplary embodiments of the present disclosure, the corrected reference cost can be obtained by various reference cost correction methods.

[0116] As an example, cost′=(α+q)·D(f resi ) to calculate the reference cost. Here, cost′ represents the reference cost, α represents the video characteristic related parameter, q represents the quantization factor, D(f resi ) represents the first reference cost. Here, α is a constant. In addition, α may be a constant greater than or equal to 0 and less than or equal to 5.

[0117] In an exemplary embodiment of the present disclosure, determining the reference cost based on the quantization factor and the first residual image may include: determining the reference cost corresponding to each of the multiple candidate frame types based on the quantization factor and the first residual image corresponding to each of the multiple candidate frame types.

[0118] In step S404, the frame type of the reference image frame is determined based on the reference cost. Here, the frame type may be, for example, an intra-frame coded frame, a forward predictive coded frame, or a bidirectional predictive coded frame.

[0119] In an exemplary embodiment of the present disclosure, determining the frame type of the current image frame based on the reference cost may include: determining the frame type of the current image frame from the multiple candidate frame types based on the reference cost corresponding to each candidate frame type in the multiple candidate frame types.

[0120] Here, a reference cost may be calculated for each of the plurality of frame types, and a frame type having the lowest calculated reference cost may be determined as the frame type of the current image frame.

[0121] In the present disclosure, while considering the quantization process, it is also possible to further consider removing the reliance of the determination of the frame type on the coding of the referenced image frame, so as to introduce the quantization factor under the condition of low coding complexity and improve the coding performance. For example, by taking the predetermined original image frame as the reference image frame of all image frames, the frame type can be determined for all image frames in parallel, thereby introducing quantization under the condition of low coding complexity, removing the reliance of the determination of the frame type on the coding of the referenced image frame, and making the time complexity low.

[0122] Since video coding is intended to compress video data into a form using a lower bit rate while avoiding or minimizing the degradation of video quality, different frame types will affect the video compression efficiency due to different encoding methods adopted. According to the method for determining the frame type of an exemplary embodiment of the present disclosure, by introducing a quantization factor, the reference cost determined in the frame type determination process is made more accurate, thereby determining a more accurate frame type.

[0123] Figure 5 A flowchart of a video encoding method according to an exemplary embodiment of the present disclosure is shown.

[0124] Reference Figure 5 In step S501, the frame type of the image frame in the video is determined by the method for determining the frame type disclosed in the present invention. Here, the frame type can be, for example, an intra-frame coded frame, a forward predictive coded frame, or a bidirectional predictive coded frame. The specific process of determining the frame type of the image frame in the video by the method for determining the frame type disclosed in the present invention has been described in detail. Figure 4 The description of Figure 4 The description in is not repeated here.

[0125] In step S502, the image frames in the video are encoded based on the frame types of the image frames in the video. Here, different image frames in the video are encoded respectively for different frame types, thereby improving encoding performance, encoding compression performance, and encoding efficiency.

[0126] For example, image frames whose frame types in the video are forward predictive coding frames are encoded using a forward coding mode, image frames whose frame types in the video are bidirectional predictive coding frames are encoded using a bidirectional coding mode, and image frames whose frame types in the video are intra-frame coding frames are intra-frame encoded using an intra-frame coding mode.

[0127] The above has been combined Figure 4 and Figure 5 A method for determining a frame type and a video encoding method according to an exemplary embodiment of the present disclosure are described. Figure 6 and Figure 7 A device for determining a frame type and a unit thereof, and a video encoding device and a unit thereof according to exemplary embodiments of the present disclosure are described.

[0128] Figure 6 A block diagram of an apparatus for determining a frame type according to an exemplary embodiment of the present disclosure is shown.

[0129] Reference Figure 6 The apparatus for determining a frame type includes a residual determination unit 61, a quantization factor determination unit 62, a cost determination unit 63 and a frame type determination unit 64.

[0130] The residual determination unit 61 is configured to determine a first residual image between a reference image frame and a predicted image frame. Here, the predicted image frame is obtained by predicting the current image frame with reference to the reference image frame.

[0131] In an exemplary embodiment of the present disclosure, the reference image frames of the plurality of image frames including the current image frame are the same predetermined image frame.

[0132] The quantization factor determination unit 62 is configured to determine a quantization factor based on the first residual image. Here, the quantization factor indicates the influence of the quantization process in the prediction on the reference cost of predicting the current image frame with reference to the reference image frame.

[0133] In an exemplary embodiment of the present disclosure, the quantization factor determination unit 62 may be configured to: determine a first reference cost and a second reference cost for predicting the current image frame with reference to the reference image frame based on the first residual image; and determine the quantization factor based on the first reference cost and the second reference cost. Here, the first reference cost may be a reference cost determined by ignoring the influence of the quantization process in the prediction on the reference cost, and the second reference cost may be a reference cost determined by considering the influence of the quantization process in the prediction on the reference cost.

[0134] In an exemplary embodiment of the present disclosure, the quantization factor determination unit 62 may be configured to: determine a first reference cost based on a first residual image; obtain a second residual image by quantizing the first residual image; and determine a second reference cost based on the second residual image.

[0135] In an exemplary embodiment of the present disclosure, the quantization factor determination unit 62 may be configured to determine the quantization factor based on a ratio of the second reference cost to the first reference cost.

[0136] In an exemplary embodiment of the present disclosure, the quantization factor determination unit 62 may be configured to determine the quantization factor based on a first reference cost and a second reference cost and a first preset value and a second preset value.

[0137] In an exemplary embodiment of the present disclosure, the quantization factor determination unit 62 may be configured to: determine a first sum of a first reference cost and a first preset value, and a second sum of a second reference cost and a second preset value; and determine the quantization factor based on a ratio of the second sum to the first sum.

[0138] In an exemplary embodiment of the present disclosure, the quantization factor determination unit 62 may be configured to: for each of the multiple candidate frame types of the current image frame, perform quantization processing corresponding to each of the multiple candidate frame types on the first residual image, and obtain a second residual image corresponding to each of the multiple candidate frame types. Here, the quantization processing may include a transform operation, a quantization operation, an inverse quantization operation, and an inverse transform operation, wherein the quantization operation and the inverse quantization operation may be different according to the candidate frame type.

[0139] In an exemplary embodiment of the present disclosure, the quantization factor determination unit 62 may be configured to: for each candidate frame type among the multiple candidate frame types, determine the second reference cost corresponding to each candidate frame type among the multiple candidate frame types based on the second residual image corresponding to each candidate frame type among the multiple candidate frame types.

[0140] In an exemplary embodiment of the present disclosure, the quantization factor determination unit 62 may be configured to determine a quantization factor corresponding to each of the multiple candidate frame types based on a first reference cost and a second reference cost corresponding to each of the multiple candidate frame types, respectively.

[0141] The cost determination unit 63 is configured to determine the reference cost based on the quantization factor and the first residual image.

[0142] In an exemplary embodiment of the present disclosure, the cost determination unit 63 may be configured to determine the reference cost based on the quantization factor and a first reference cost.

[0143] In an exemplary embodiment of the present disclosure, the cost determination unit 63 may be configured to determine the reference cost based on the quantization factor, video characteristic related parameters and a first reference cost, wherein the video characteristic related parameters are determined based on the characteristics of the video in which the current image frame is located.

[0144] In an exemplary embodiment of the present disclosure, the cost determination unit 63 may be configured to: determine a sum of a video characteristic related parameter of a reference image frame and the quantization factor; and determine the reference cost based on a product of the sum and a first reference cost.

[0145] In an exemplary embodiment of the present disclosure, the cost determination unit 63 may be configured to: input the first residual image into a residual function to obtain a first reference cost; and input the second residual image into the residual function to obtain a second reference cost.

[0146] In an exemplary embodiment of the present disclosure, the cost determination unit 63 may be configured to: determine the reference cost corresponding to each of the multiple candidate frame types based on the quantization factor corresponding to each of the multiple candidate frame types and the first residual image, respectively;

[0147] The frame type determination unit 64 is configured to determine the frame type of the current image frame based on the reference cost.

[0148] In an exemplary embodiment of the present disclosure, the frame type determination unit 64 is configured to determine the frame type of the current image frame from the multiple candidate frame types based on the reference cost corresponding to each candidate frame type in the multiple candidate frame types.

[0149] Figure 7 A block diagram of a video encoding apparatus according to an exemplary embodiment of the present disclosure is shown.

[0150] Reference Figure 7 The video encoding device includes a frame type acquisition unit 71 and an image frame encoding unit 72.

[0151] The frame type acquisition unit 71 is configured to determine the frame type of the image frame in the video by the method for determining the frame type disclosed in the present invention. The specific process of determining the frame type of the image frame in the video by the method for determining the frame type disclosed in the present invention has been described in detail. Figure 4 The description of Figure 4 The description in is not repeated here.

[0152] The image frame encoding unit 72 is configured to encode the image frames in the video based on the frame types of the image frames in the video.

[0153] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0154] The above has been combined Figure 6 and Figure 7 The apparatus for determining the frame type according to an exemplary embodiment of the present disclosure is described. Figure 8 A computing environment coupled with a user interface according to some embodiments of the present disclosure is described.

[0155] Figure 8 A computing environment 810 is shown coupled to a user interface 850. The computing environment 810 may be part of a data processing server. The computing environment 810 includes a processor 820, a memory 830, and an input / output (I / O) interface 840.

[0156] The processor 820 generally controls the overall operation of the computing environment 810, such as operations associated with display, data acquisition, data communication, and image processing. The processor 820 may include one or more processors for executing instructions to perform all or some steps in the above method. In addition, the processor 820 may include one or more modules that facilitate the interaction between the processor 820 and other components. The processor may be a central processing unit (CPU), a microprocessor, a single-chip microcomputer, a graphics processing unit (GPU), etc.

[0157] The memory 830 is configured to store various types of data to support the operation of the computing environment 810. The memory 830 may include predetermined software 832. Examples of such data include instructions for any application or method operating on the computing environment 810, video data sets, image data, etc. The memory 830 may be implemented by using any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0158] The I / O interface 840 provides an interface between the processor 820 and peripheral interface modules (e.g., keyboard, click wheel, buttons, etc.). The buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. The I / O interface 840 may be coupled to an encoder and a decoder.

[0159] In an embodiment, a non-transitory computer-readable storage medium including, for example, a plurality of programs in a memory 830 and / or storing a bit stream generated by the above encoding method or a bit stream to be decoded by the above decoding method is also provided, wherein the plurality of programs can be executed by a processor 820 in a computing environment 810 to perform the above method. In one example, the plurality of programs can be executed by a processor 820 in a computing environment 810 to (for example, from Figure 2 The video encoder 20 in the computing environment 810 receives a bit stream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or one or more associated syntax elements, etc.), and can also be executed by the processor 820 in the computing environment 810 to perform the above-mentioned decoding method according to the received bit stream or data stream. In another example, the multiple programs can be executed by the processor 820 in the computing environment 810 to perform the above-mentioned encoding method to encode the video information (e.g., video blocks representing video frames, and / or one or more associated syntax elements, etc.) into a bit stream or data stream, and can also be executed by the processor 820 in the computing environment 810 to (e.g., to Figure 3Alternatively, a non-transitory computer-readable storage medium may store the bit stream or data stream generated by the encoder (e.g., Figure 2 The video encoder 20 in FIG. 1 generates a video signal for a decoder (eg, Figure 3 A bitstream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or associated one or more syntax elements, etc.) used by the video decoder 30 in the video decoder 30 when decoding video data. The non-transitory computer-readable storage medium can be, for example, a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0160] In an embodiment, a bit stream generated by the above encoding method or a bit stream to be decoded by the above decoding method is provided. In an embodiment, a bit stream including encoded video information generated by the above encoding method or encoded video information to be decoded by the above decoding method is provided.

[0161] In an embodiment, a computing device is also provided, which includes: one or more processors (e.g., processor 820); and a non-transitory computer-readable storage medium or memory 830 in which multiple programs that can be executed by the one or more processors are stored, wherein the one or more processors are configured to perform the above-mentioned method when executing the multiple programs.

[0162] In an embodiment, a computer program product having instructions for storing or transmitting a bitstream is also provided, wherein the bitstream includes coded video information generated by the above encoding method or coded video information to be decoded by the above decoding method. In an embodiment, a computer program product including, for example, a plurality of programs in a memory 830 is also provided, wherein the plurality of programs can be executed by a processor 820 in a computing environment 810 for performing the above method. For example, the computer program product may include a non-transitory computer-readable storage medium.

[0163] In an embodiment, the computing environment 810 may be implemented by one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs, GPUs, controllers, microcontrollers, microprocessors, or other electronic components for performing the above methods.

[0164] According to the method and apparatus for determining a frame type disclosed in the present invention, a first residual image between a reference image frame and a predicted image frame is determined, wherein the predicted image frame is obtained by predicting a current image frame with reference to the reference image frame, a quantization factor is determined based on the first residual image, wherein the quantization factor indicates the influence of the quantization processing in the prediction on the reference cost for predicting the current image frame with reference to the reference image frame, the reference cost is determined based on the quantization factor and the first residual image, and the frame type of the current image frame is determined based on the reference cost, thereby making the determined frame type more accurate by introducing a quantization factor indicating the influence of the quantization processing in the prediction on the reference cost for predicting the current image frame with reference to the reference image frame, thereby improving encoder performance.

[0165] In addition, according to the method and device for determining the frame type disclosed in the present invention, by using a predetermined original image frame as a reference image frame for all image frames, the frame type determination can be performed in parallel, thereby eliminating the dependence of the frame type determination on the encoding of the referenced image frame, reducing the time complexity and further improving the encoding (or compression) performance.

[0166] According to the video encoding method and device disclosed in the present invention, the frame type of the image frame in the video is determined by using the method for determining the frame type disclosed in the present invention, and the image frame in the video is encoded based on the frame type of the image frame in the video. Since a more accurate frame type of the image frame is obtained by using the method for determining the frame type disclosed in the present invention, a more accurate frame type can be used to improve the video encoding effect.

[0167] The description of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or limited to the present disclosure. Many modifications, variations, and alternative embodiments will be apparent to one of ordinary skill in the art having the benefit of the teachings presented in the foregoing description and the associated drawings.

[0168] Unless otherwise specifically stated, the order of steps of the method according to the present disclosure is intended to be illustrative only, and the steps of the method according to the present disclosure are not limited to the order specifically described above, but can be changed according to actual conditions. In addition, at least one of the steps of the method according to the present disclosure can be adjusted, combined or deleted according to actual needs.

[0169] The examples are chosen and described in order to explain the principles of the present disclosure and to enable other persons skilled in the art to understand the various embodiments of the present disclosure and to best utilize the basic principles and various embodiments with various modifications suitable for the specific use contemplated. Therefore, it will be understood that the scope of the present disclosure is not limited to the specific examples of the disclosed embodiments, and modifications and other embodiments are intended to be included within the scope of the present disclosure.

Claims

1. A method for determining a frame type, characterized in that include: Determine a first residual image between a reference image frame and a predicted image frame, wherein the predicted image frame is obtained by predicting the current image frame with reference to the reference image frame; Determining a quantization factor based on the first residual image, wherein the quantization factor indicates an effect of a quantization process in the prediction on a reference cost for predicting the current image frame with reference to the reference image frame; Determining the reference cost based on the quantization factor and the first residual image; A frame type of the current image frame is determined based on the reference cost.

2. The method according to claim 1, characterized in that The determining of the quantization factor based on the first residual image comprises: Determine, based on the first residual image, a first reference cost and a second reference cost for predicting the current image frame with reference to the reference image frame, wherein the first reference cost is a reference cost determined by ignoring the influence of quantization processing in the prediction on the reference cost, and the second reference cost is a reference cost determined by considering the influence of quantization processing in the prediction on the reference cost; The quantization factor is determined based on a first reference cost and a second reference cost.

3. The method according to claim 2, characterized in that The determining, based on the first residual image, a first reference cost and a second reference cost for predicting the current image frame with reference to the reference image frame comprises: determining a first reference cost based on the first residual image; Obtaining a second residual image by quantizing the first residual image; A second reference cost is determined based on the second residual image.

4. The method according to claim 2, characterized in that: The determining the reference cost based on the quantization factor and the first residual image includes: The reference cost is determined based on the quantization factor and a first reference cost.

5. The method according to claim 4, characterized in that The determining the reference cost based on the quantization factor and the first reference cost includes: The reference cost is determined based on the quantization factor, a video characteristic related parameter and a first reference cost, wherein the video characteristic related parameter is determined based on the characteristics of the video where the current image frame is located.

6. The method according to claim 3, characterized in that The step of obtaining the second residual image by quantizing the first residual image includes: For each of the multiple candidate frame types of the current image frame, respectively perform quantization processing corresponding to each of the multiple candidate frame types on the first residual image to obtain a second residual image corresponding to each of the multiple candidate frame types, The quantization process includes a transform operation, a quantization operation, an inverse quantization operation, and an inverse transform operation, wherein the quantization operation and the inverse quantization operation are different according to the candidate frame type.

7. The method according to claim 6, characterized in that The determining the second reference cost based on the second residual image includes: For each of the multiple candidate frame types, a second reference cost corresponding to each of the multiple candidate frame types is determined based on a second residual image corresponding to each of the multiple candidate frame types.

8. The method according to claim 7, characterized in that The determining the quantization factor based on the first reference cost and the second reference cost includes: A quantization factor corresponding to each of the multiple candidate frame types is determined based on the first reference cost and the second reference cost corresponding to each of the multiple candidate frame types.

9. The method according to claim 8, characterized in that The determining the reference cost based on the quantization factor and the first residual image includes: Determine the reference cost corresponding to each of the multiple candidate frame types based on the quantization factor corresponding to each of the multiple candidate frame types and the first residual image, Wherein, determining the frame type of the current image frame based on the reference cost includes: Based on the reference cost corresponding to each candidate frame type in the multiple candidate frame types, the frame type of the current image frame is determined from the multiple candidate frame types.

10. A device for determining a frame type, characterized in that: include: a residual determination unit configured to determine a first residual image between a reference image frame and a predicted image frame, wherein the predicted image frame is obtained by predicting the current image frame with reference to the reference image frame; a quantization factor determining unit configured to determine a quantization factor based on the first residual image, wherein the quantization factor indicates an influence of a quantization process in the prediction on a reference cost for predicting the current image frame with reference to the reference image frame; a cost determining unit configured to determine the reference cost based on the quantization factor and the first residual image; and The frame type determining unit is configured to determine the frame type of the current image frame based on the reference cost.

11. A video encoding method, comprising: Determine the frame type of the image frame in the video by the method for determining the frame type according to any one of claims 1 to 9; The image frames in the video are encoded based on their frame types.

12. A video encoding device, comprising: A frame type determination unit, configured to determine a frame type of an image frame in a video by using a method for determining a frame type according to any one of claims 1 to 9; The image frame encoding unit is configured to encode the image frames in the video based on the frame types of the image frames in the video.

13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method as claimed in any one of claims 1 to 9 or 11.

14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor of an electronic device, the electronic device is caused to execute the method as claimed in any one of claims 1 to 9 or 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 9 or 11 is implemented.

Images