Video frame processing method and device, electronic equipment and storage medium

Abstract

The invention provides a video frame processing method and device, electronic equipment and a storage medium. The video frame processing method comprises the following steps: acquiring a plurality of video frames from a video to be used for coding; determining a filtering reference frame of the current video frame based on the frame type and the coding reference relation of the current video frame to be filtered in the plurality of video frames; and performing time domain filtering of motion compensation on the current video frame based on the filtering reference frame of the current video frame.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing technology, and in particular to a video frame processing method, device, electronic device and storage medium for time domain filtering for motion compensation. Background Art

[0002] Various electronic devices (e.g., digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smart phones, video teleconferencing devices, video streaming devices, etc.) support digital video. Electronic devices send and receive or otherwise transmit digital video data through a communication network, and / or store digital video data on a storage device. Due to the limited bandwidth capacity of the communication network and the limited storage resources of the storage device, before the video data is transmitted or stored, the video data can be compressed using a video codec according to one or more video codec standards. For example, video codec standards include general video codec (VVC), joint exploration test model (JEM), high efficiency video codec (HEVC / H.265), advanced video codec (AVC / H.264), moving picture expert group (MPEG) codec, etc. Video codecs generally use prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that utilize the inherent redundancy in video data. Video codecs are intended to compress video data into a form that uses a lower bit rate while avoiding or minimizing the degradation of video quality.

[0003] In related technologies, the motion compensated temporal filter (MCTF) method is used to filter the video before encoding to remove the temporal noise of the video before encoding, thereby improving the video encoding efficiency. This method uses the classic block-based motion estimation and motion compensation method, and uses the temporal correlation of the video to perform temporal filtering on the video frames to reduce the temporal redundant information generated during the video frame reference process, thereby improving the overall encoding compression rate of the video.

[0004] In relevant coding standards (such as Versatile Video Coding (VVC) and Audio Video Coding (AVS)), the MCTF method usually uses a fixed filtering strategy (especially a fixed number of adjacent video frames) to filter the current video frame. This filtering method is very inflexible. In addition, when the filtering reference frame is significantly different from the current video frame, this filtering method may introduce unnecessary bad cases. Therefore, this traditional MCTF method lacks effective detection and judgment for the filtering reference frame. Summary of the invention

[0005] According to a first aspect of the present disclosure, a video processing method is provided, comprising: obtaining multiple video frames from a video to be used for encoding; determining a filtering reference frame of the current video frame based on a frame type and a coding reference relationship of a current video frame to be filtered among the multiple video frames; and performing motion compensated time domain filtering on the current video frame based on the filtering reference frame of the current video frame.

[0006] Optionally, the current video frame to be filtered is a video frame among a plurality of video frames used as an encoding reference frame during video encoding.

[0007] Optionally, based on the frame type and encoding reference relationship of a current video frame to be filtered among multiple video frames, determining a filtering reference frame of the current video frame includes: determining candidate filtering reference frames of the current video frame, wherein the candidate filtering reference frames include video frames that use the current video frame as encoding reference frames during video encoding; based on the frame type of the current video frame, determining at least one of the candidate filtering reference frames as a filtering reference frame of the current video frame.

[0008] Optionally, determining the candidate filtering reference frames of the current video frame includes: in response to the current video frame not being used as a long-term reference frame during video encoding, determining the candidate filtering reference frames of the current video frame based on the image group to which the current video frame belongs; and / or in response to the current video frame being used as a long-term reference frame during video encoding, determining the candidate filtering reference frames of the current video frame based on the image group to which the current video frame belongs and a subsequent image group.

[0009] Optionally, based on the frame type of the current video frame, determining at least one of the candidate filtering reference frames as the filtering reference frame of the current video frame includes: based on the current video frame being an I frame, determining all candidate filtering reference frames as the filtering reference frames of the current video frame; and / or based on the current video frame being a P frame or a B frame, according to the video display order POC of the candidate filtering reference frames, determining the candidate filtering reference frames that meet the conditions as the filtering reference frames of the current video frame.

[0010] Optionally, based on whether the current video frame is a P frame or a B frame, according to the video display order POC of the candidate filtering reference frames, determining the candidate filtering reference frame that meets the conditions as the filtering reference frame of the current video frame includes: based on the current video frame being a P frame, determining the candidate filtering reference frame that meets the first condition as the filtering reference frame of the current video frame, wherein the first condition includes: the difference between the POC of the current video frame and the POC of the candidate filtering reference frame is less than a first threshold, and / or the difference between the POC of the current video frame and the POC of the candidate filtering reference frame is greater than or equal to the first threshold and the candidate filtering reference frame is a coded reference frame; and / or based on the current video frame being a B frame, determining the candidate filtering reference frame that meets the second condition as the filtering reference frame of the current video frame, wherein the second condition includes: the difference between the POC of the current video frame and the POC of the candidate filtering reference frame is less than or equal to a second threshold, wherein the first threshold is determined based on the size of the image group to which the current video frame belongs, and / or the second threshold is determined based on the number of forward candidate filtering reference frames or backward candidate filtering reference frames of the current video frame.

[0011] Optionally, the multiple video frames are divided into multiple layers, wherein B frames in the same layer have the same second threshold, and B frames in different layers among the multiple layers have different second thresholds.

[0012] Optionally, after determining the filtering reference frame of the current video frame, the method further comprises: based on the brightness of the current video frame and the filtering reference frame, discarding the filtering reference frame that meets the brightness condition.

[0013] Optionally, based on the brightness of the current video frame and the filtered reference frame, discarding the filtered reference frame that meets the brightness condition includes: obtaining grayscale histograms of the current video frame and the filtered reference frame; determining the brightness difference between the current video frame and the filtered reference frame based on the grayscale histograms of the current video frame and the filtered reference frame; based on the brightness difference between the current video frame and the filtered reference frame, discarding the filtered reference frame that meets the brightness condition, wherein the brightness condition includes that the brightness difference between the current video frame and the filtered reference frame is greater than or equal to a third threshold, wherein the third threshold is determined based on the size of the video frame.

[0014] According to a second aspect of the present disclosure, a video frame processing device is provided, comprising: a video frame acquisition unit, configured to acquire multiple video frames from a video to be used for encoding; a reference frame determination unit, configured to determine a filtering reference frame of a current video frame based on a frame type and a coding reference relationship of a current video frame to be filtered among the multiple video frames; and a video frame filtering unit, configured to perform motion compensated time domain filtering on the current video frame based on the filtering reference frame of the current video frame.

[0015] Optionally, the current video frame to be filtered is a video frame among a plurality of video frames used as an encoding reference frame during video encoding.

[0016] Optionally, the reference frame determination unit is configured to determine a filtering reference frame of the current video frame based on a frame type and a coding reference relationship of the current video frame to be filtered among multiple video frames by the following operations: determining candidate filtering reference frames of the current video frame, wherein the candidate filtering reference frames include video frames that use the current video frame as coding reference frames during video encoding; based on the frame type of the current video frame, determining at least one of the candidate filtering reference frames as a filtering reference frame of the current video frame.

[0017] Optionally, the reference frame determination unit is configured to determine candidate filtering reference frames for the current video frame by: in response to the current video frame not being used as a long-term reference frame during video encoding, determining candidate filtering reference frames for the current video frame based on the image group to which the current video frame belongs; and / or in response to the current video frame being used as a long-term reference frame during video encoding, determining candidate filtering reference frames for the current video frame based on the image group to which the current video frame belongs and subsequent image groups.

[0018] Optionally, the reference frame determination unit is configured to determine at least one of the candidate filtering reference frames as a filtering reference frame of the current video frame based on the frame type of the current video frame through the following operations: based on the current video frame being an I frame, all candidate filtering reference frames are determined as filtering reference frames of the current video frame; and / or based on the current video frame being a P frame or a B frame, according to the video display order POC of the candidate filtering reference frames, candidate filtering reference frames that meet the conditions are determined as filtering reference frames of the current video frame.

[0019] Optionally, the reference frame determination unit is configured to determine a candidate filtering reference frame that meets the conditions as a filtering reference frame of the current video frame based on whether the current video frame is a P frame or a B frame and according to the video display order POC of the candidate filtering reference frame through the following operations: based on the current video frame being a P frame, determining a candidate filtering reference frame that meets a first condition as a filtering reference frame of the current video frame, wherein the first condition includes: a difference between the POC of the current video frame and the POC of the candidate filtering reference frame is less than a first threshold, and / or a difference between the POC of the current video frame and the POC of the candidate filtering reference frame is greater than or equal to the first threshold and the candidate filtering reference frame is a coded reference frame; and / or based on the current video frame being a B frame, determining a candidate filtering reference frame that meets a second condition as a filtering reference frame of the current video frame, wherein the second condition includes: a difference between the POC of the current video frame and the POC of the candidate filtering reference frame is less than or equal to a second threshold, wherein the first threshold is determined based on the size of the image group to which the current video frame belongs, and / or the second threshold is determined based on the number of forward candidate filtering reference frames or backward candidate filtering reference frames of the current video frame.

[0020] Optionally, the multiple video frames are divided into multiple layers, wherein B frames in the same layer have the same second threshold, and B frames in different layers among the multiple layers have different second thresholds.

[0021] Optionally, the reference frame determination unit is further configured to: after determining the filtered reference frame of the current video frame, discard the filtered reference frame that meets the brightness condition based on the brightness of the current video frame and the filtered reference frame.

[0022] Optionally, the reference frame determination unit is configured to discard the filtered reference frame that meets the brightness condition based on the brightness of the current video frame and the filtered reference frame through the following operations: obtaining grayscale histograms of the current video frame and the filtered reference frame; determining the brightness difference between the current video frame and the filtered reference frame based on the grayscale histograms of the current video frame and the filtered reference frame; and discarding the filtered reference frame that meets the brightness condition based on the brightness difference between the current video frame and the filtered reference frame, wherein the brightness condition includes that the brightness difference between the current video frame and the filtered reference frame is greater than or equal to a third threshold, wherein the third threshold is determined based on the size of the video frame.

[0023] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, prompt the at least one processor to execute the video frame processing method as described above.

[0024] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor of a filtering device / electronic device / server for video encoding, the filtering device / electronic device / server for video encoding can perform the video frame processing method as described above.

[0025] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising computer instructions, which implement the video frame processing method as described above when executed by a processor.

[0026] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: according to the frame type of the video frame in the actual coding scene, the reference relationship between different video frames and / or the GOP structure, the filtering reference frame of the MCTF can be adaptively adjusted, which can effectively improve the filtering efficiency of the MCTF while improving the filtering quality of the MCTF, thereby improving the overall coding efficiency of the video coding process. In addition, by eliminating the filtering reference frames with large texture differences or brightness differences with the current video frame to be filtered, the filtering reference frames with positive effects are screened out, further improving the subjective and objective quality of the filtered video frame, which has obvious optimization effects in the actual application of MCTF with badcases.

[0027] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0029] Figure 1 is a block diagram illustrating an exemplary system for encoding and decoding video blocks according to some embodiments of the present disclosure.

[0030] Figure 2 is a block diagram illustrating an exemplary video encoder according to some embodiments of the present disclosure.

[0031] Figure 3 is a block diagram illustrating an exemplary video decoder according to some embodiments of the present disclosure.

[0032] Figures 4 to 7 is a diagram showing an example of a GOP structure.

[0033] Figure 8 is a schematic diagram showing an example of a video frame and a filtering reference frame used in motion compensated temporal filtering.

[0034] Fig. 9 is a schematic diagram illustrating an overall environment to which a video frame processing scheme according to an exemplary embodiment of the present disclosure is applied.

[0035] Fig.10 is a flowchart illustrating a video frame processing method according to an exemplary embodiment of the present disclosure.

[0036] Fig.11 is a block diagram illustrating a video frame processing apparatus according to an exemplary embodiment of the present disclosure.

[0037] Fig.12 is a diagram illustrating a computing environment coupled with a user interface according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0038] Reference will now be made in detail to specific embodiments, examples of which are shown in the accompanying drawings. In the following detailed description, a number of non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, various alternatives may be used without departing from the scope of the claims, and the subject matter may be practiced without these specific details. For example, the subject matter presented herein may be implemented on many types of electronic devices having digital video capabilities.

[0039] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the drawings are used to distinguish objects, but not to describe any specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those shown in the drawings or described in the present disclosure.

[0040] Figure 1 is a block diagram illustrating an exemplary system 10 for encoding and decoding video blocks in parallel according to some embodiments of the present disclosure. Figure 1 As shown in , system 10 includes a source device 12 that generates and encodes video data to be later decoded by a target device 14. Source device 12 and target device 14 may include any of a wide variety of electronic devices, including cloud servers, server computers, desktop or laptop computers, tablet computers, smart phones, set-top boxes, digital televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some implementations, source device 12 and target device 14 are equipped with wireless communication capabilities.

[0041] In some implementations, target device 14 may receive the encoded video data to be decoded via link 16. Link 16 may include any type of communication medium or device capable of moving the encoded video data from source device 12 to target device 14.

[0042] In some other embodiments, the encoded video data may be sent from the output interface 22 to the storage device 32. The encoded video data in the storage device 32 may then be accessed by the target device 14 via the input interface 28.

[0043] like Figure 1 , source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Video source 18 may include sources such as, or a combination of such sources: a video capture device (e.g., a camera), a video archive containing previously captured video, a video feed interface for receiving video from a video content provider, and / or a computer graphics system for generating computer graphics data as source video.

[0044] The captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video data may be sent directly to target device 14 via output interface 22 of source device 12. The encoded video data may also (or alternatively) be stored on storage device 32 for later access by target device 14 or other devices for decoding and / or playback.

[0045] Target device 14 includes input interface 28, video decoder 30, and display device 34. Input interface 28 may include a receiver and / or a modem and receives encoded video data via link 16. The encoded video data transmitted via link 16 or provided on storage device 32 may include various syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements may be included within the encoded video data sent over a communication medium, stored on a storage medium, or stored on a file server.

[0046] The video encoder 20 and the video decoder 30 can operate according to a proprietary standard or an industry standard (e.g., VVC, HEVC, Part 10 of MPEG-4, AVC) or an extension of such a standard. It should be understood that the present application is not limited to a specific video encoding / decoding standard and can be applied to other video encoding / decoding standards. It is generally believed that the video encoder 20 of the source device 12 can be configured to encode the video data according to any of these current standards or future standards. Similarly, it is also generally believed that the video decoder 30 of the target device 14 can be configured to decode the video data according to any of these current standards or future standards.

[0047] The video encoder 20 and the video decoder 30 may be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic devices, software, hardware, firmware, or any combination thereof. When partially implemented in software, the electronic device may store instructions for the software in a suitable non-transitory computer-readable medium, and use one or more processors to execute the instructions in the hardware to perform the video encoding / decoding operations disclosed in the present disclosure. Each of the video encoder 20 and the video decoder 30 may be included in one or more encoders or decoders, and either of the encoders or decoders may be integrated as part of a combined encoder / decoder (CODEC) in the corresponding device.

[0048] Figure 2 is a block diagram illustrating an exemplary video encoder 20 according to some embodiments described in the present application. The video encoder 20 can perform intra-frame prediction coding and inter-frame prediction coding on video blocks within a video frame. Intra-frame prediction coding relies on spatial prediction to reduce or remove spatial redundancy in video data within a given video frame or picture. Inter-frame prediction coding relies on temporal prediction to reduce or remove temporal redundancy in video data within adjacent video frames or pictures of a video sequence. It should be noted that in the field of video coding and decoding, the term "frame" can be used as a synonym for the term "image" or "picture".

[0049] like Figure 2As shown in , the video encoder 20 includes a video data memory 40, a prediction processing unit 41, a decoded picture buffer (DPB) 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The prediction processing unit 41 further includes a motion estimation unit 42, a motion compensation unit 44, a segmentation unit 45, an intra-frame prediction processing unit 46, and an intra-frame block copy (BC) unit 48. In some embodiments, the video encoder 20 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and an adder 62 for video block reconstruction. A loop filter 63 such as a deblocking filter can be located between the adder 62 and the DPB 64 to filter the block boundary to remove the block effect from the reconstructed video. In addition to the deblocking filter, another loop filter (e.g., a sample adaptive offset (SAO) filter, a cross-component sample adaptive offset (CCSAO) filter, and / or an adaptive loop filter (ALF)) can also be used to filter the output of the adder 62. In some examples, the loop filter may be omitted, and the decoded video blocks may be provided directly by adder 62 to DPB 64. Video encoder 20 may take the form of a fixed or programmable hardware unit, or may be distributed among one or more of the illustrated fixed or programmable hardware units.

[0050] Video data memory 40 may store video data to be encoded by components of video encoder 20. Figure 1 Video source 18 is shown obtaining video data in video data memory 40. DPB 64 is a buffer that stores reference video data (eg, reference frames or pictures) for use by video encoder 20 in encoding the video data (eg, in intra or inter prediction coding modes).

[0051] like Figure 2 As shown in , after receiving the video data, the segmentation unit 45 within the prediction processing unit 41 segments the video data into video blocks. This segmentation may also include segmenting the video frame into strips, tiles (e.g., a set of video blocks), or other larger coding units (CUs) according to a predefined splitting structure associated with the video data, such as a quadtree (QT) structure. It should be noted that the terms "block" or "video block" used herein may be a portion of a frame or picture, in particular a rectangular (square or non-square) portion. With reference to, for example, HEVC and VVC, a block or video block may be or correspond to a coding tree unit (CTU), a CU, a prediction unit (PU), or a transform unit (TU) and / or may be or correspond to a corresponding block (e.g., a coding tree block (CTB), a coding block (CB), a prediction block (PB), or a transform block (TB)) and / or a sub-block.

[0052] The prediction processing unit 41 may select one of a plurality of feasible prediction coding modes for the current video block based on the error results (e.g., coding rate and distortion level), such as one of one or more inter-frame prediction coding modes in a plurality of intra-frame prediction coding modes. The prediction processing unit 41 may provide the resulting intra-frame prediction coding block or inter-frame prediction coding block to the adder 50 to generate a residual block, and to the adder 62 to reconstruct the coding block for subsequent use as part of a reference frame. The prediction processing unit 41 also provides syntax elements (e.g., motion vectors, intra-frame mode indicators, partition information, and other such syntax information) to the entropy coding unit 56.

[0053] To select an appropriate intra-prediction coding mode for the current video block, intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-prediction coding of the current video block in relation to one or more neighboring blocks in the same frame as the current block to be encoded to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 may perform inter-prediction coding of the current video block in relation to one or more prediction blocks in one or more reference frames to provide temporal prediction. Video encoder 20 may perform multiple encoding passes, for example, to select an appropriate coding mode for each block of video data.

[0054] In some embodiments, motion estimation unit 42 determines an inter-prediction mode for a current video frame by generating a motion vector according to a predetermined pattern within a sequence of video frames, the motion vector indicating the displacement of a video block within the current video frame relative to a prediction block within a reference video frame. Motion estimation performed by motion estimation unit 42 is the process of generating a motion vector that estimates the motion of a video block. For example, a motion vector may indicate the displacement of a video block within a current video frame or picture relative to a prediction block within a reference frame associated with a current block being encoded within the current frame. The predetermined pattern may designate a video frame in a sequence as a P frame or a B frame. Intra BC unit 48 may determine a vector (e.g., a block vector) for intra BC coding in a manner similar to the motion vector determined by motion estimation unit 42 for inter prediction, or may utilize motion estimation unit 42 to determine a block vector.

[0055] Regardless of whether the prediction block is from the same frame according to intra-frame prediction or from a different frame according to inter-frame prediction, the video encoder 20 can form a residual video block by subtracting the pixel values ​​of the prediction block from the pixel values ​​of the current video block being encoded. The pixel difference values ​​forming the residual video block may include both luma component differences and chroma component differences.

[0056] Intra-prediction processing unit 46 may encode the current block using various intra-prediction modes, for example, during separate encoding passes, and intra-prediction processing unit 46 (or, in some examples, mode selection unit) may select an appropriate intra-prediction mode to use from the tested intra-prediction modes. Intra-prediction processing unit 46 may provide information indicating the intra-prediction mode selected for the block to entropy encoding unit 56. Entropy encoding unit 56 may encode the information indicating the selected intra-prediction mode into the bitstream.

[0057] After prediction processing unit 41 determines a prediction block for the current video block via inter-prediction or intra-prediction, adder 50 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block may be included in one or more TUs and provided to transform processing unit 52. Transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform (e.g., a discrete cosine transform (DCT) or a conceptually similar transform).

[0058] The transform processing unit 52 may send the resulting transform coefficients to a quantization unit 54. The quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may also reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, the quantization unit 54 may then perform a scan on the matrix including the quantized transform coefficients. Alternatively, the entropy coding unit 56 may perform the scan.

[0059] After quantization, entropy coding unit 56 entropy encodes the quantized transform coefficients into a video bitstream using, for example, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding method or technique. The encoded bitstream may then be sent to a video bitstream such as a Figure 1 The video decoder 30 shown, or archived as Figure 1 The video frame may be stored in storage device 32 as shown for later transmission to or retrieval by video decoder 30. Entropy encoding unit 56 may also entropy encode motion vectors and other syntax elements for the current video frame being encoded.

[0060] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual video block in the pixel domain for use in generating reference blocks for predicting other video blocks. As noted above, motion compensation unit 44 may generate a motion compensated prediction block from one or more reference blocks of a frame stored in DPB 64. Motion compensation unit 44 may also apply one or more interpolation filters to the prediction block to calculate sub-integer pixel values ​​for use in motion estimation.

[0061] Adder 62 adds the reconstructed residual block to the motion compensated prediction block generated by motion compensation unit 44 to generate a reference block for storage in DPB 64. The reference block may then be used as a prediction block by intra BC unit 48, motion estimation unit 42, and motion compensation unit 44 to inter-predict another video block in a subsequent video frame.

[0062] Figure 3 1 is a block diagram showing an exemplary video decoder 30 according to some embodiments of the present application. The video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, and a DPB 92. The prediction processing unit 81 further includes a motion compensation unit 82, an intra-frame prediction unit 84, and an intra-frame BC unit 85. The video decoder 30 may perform the above-mentioned Figure 2 The encoding process is substantially the inverse of the decoding process described with respect to video encoder 20. For example, motion compensation unit 82 may generate prediction data based on motion vectors received from entropy decoding unit 80, and intra-prediction unit 84 may generate prediction data based on intra-prediction mode indicators received from entropy decoding unit 80.

[0063] In some examples, embodiments of the present disclosure may be dispersed in one or more of the units of the video decoder 30. For example, the intra BC unit 85 may perform embodiments of the present application alone or in combination with other units of the video decoder 30, such as the motion compensation unit 82, the intra prediction unit 84, and the entropy decoding unit 80. In some examples, the video decoder 30 may not include the intra BC unit 85, and the functions of the intra BC unit 85 may be performed by other components of the prediction processing unit 81, such as the motion compensation unit 82.

[0064] Video data memory 79 may store video data, such as an encoded video bitstream, to be decoded by other components of video decoder 30. Video data stored in video data memory 79 may be obtained, for example, from storage device 32, from a local video source (e.g., a camera), via wired or wireless network communication of video data, or by accessing a physical data storage medium (e.g., a flash drive or hard disk).

[0065] During the decoding process, the video decoder 30 receives an encoded video bitstream representing video blocks of an encoded video frame and associated syntax elements. The entropy decoding unit 80 of the video decoder 30 entropy decodes the bitstream to generate quantization coefficients, motion vectors or intra-frame prediction mode indicators, and other syntax elements. The entropy decoding unit 80 then forwards the motion vectors or intra-frame prediction mode indicators, and other syntax elements to the prediction processing unit 81.

[0066] When a video frame is encoded as an intra-prediction coded (I) frame or for intra-coded prediction blocks in other types of frames, the intra-prediction unit 84 of the prediction processing unit 81 can generate prediction data for a video block of the current video frame based on the intra-prediction mode transmitted by the signal and reference data from a previously decoded block of the current frame.

[0067] When the video frame is encoded as an inter-frame prediction coding (i.e., B or P) frame, the motion compensation unit 82 of the prediction processing unit 81 generates one or more prediction blocks for the video block of the current video frame based on the motion vector and other syntax elements received from the entropy decoding unit 80. Each of the prediction blocks can be generated from a reference frame in one of the reference frame lists. The video decoder 30 can construct the reference frame lists, i.e., List 0 and List 1, based on the reference frames stored in the DPB 92 using a default construction technique.

[0068] In some examples, when a video block is encoded according to the intra BC mode described herein, intra BC unit 85 of prediction processing unit 81 generates a prediction block for the current video block based on the block vector and other syntax elements received from entropy decoding unit 80. The prediction block may be within a reconstructed region of the same picture as the current video block as defined by video encoder 20.

[0069] Motion compensation unit 82 and / or intra BC unit 85 determine prediction information for a video block of the current video frame by parsing motion vectors and other syntax elements, and then use the prediction information to generate a prediction block for the current video block being decoded.

[0070] Motion compensation unit 82 may also perform interpolation using interpolation filters as used by video encoder 20 during encoding of the video blocks to calculate interpolated values ​​for sub-integer pixels of reference blocks. In this case, motion compensation unit 82 may determine the interpolation filters used by video encoder 20 based on the received syntax elements and use these interpolation filters to produce the prediction blocks.

[0071] Inverse quantization unit 86 inverse quantizes the quantized transform coefficients provided in the bitstream and entropy decoded by entropy decoding unit 80, using the same quantization parameters calculated by video encoder 20 for each video block in the video frame to determine the degree of quantization. Inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients to reconstruct the residual block in the pixel domain.

[0072] After the motion compensation unit 82 or the intra BC unit 85 generates a prediction block for the current video block based on the vector and other syntax elements, the adder 90 reconstructs the decoded video block for the current video block by adding the residual block from the inverse transform processing unit 88 to the corresponding prediction block generated by the motion compensation unit 82 and the intra BC unit 85. A loop filter 91 (e.g., a deblocking filter, an SAO filter, a CCSAO filter, and / or an ALF) may be located between the adder 90 and the DPB 92 to further process the decoded video block. In some examples, the loop filter 91 may be omitted, and the decoded video block may be provided directly to the DPB 92 by the adder 90. The decoded video blocks in a given frame are then stored in the DPB 92, which stores reference frames for subsequent motion compensation of the next video block. The DPB 92 or a memory device separate from the DPB 92 may also store the decoded video for later presentation on a display device (e.g., Figure 1 on a display device 34).

[0073] Before introducing the embodiments of the present disclosure, Figures 4 to 8 Some key terms and related technologies involved in the embodiments of the present disclosure are introduced, among which: Figures 4 to 7 is a schematic diagram showing an example of a group of pictures (Group Of Picture, GOP) structure, Figure 8 is a schematic diagram showing an example of a video frame and a filtering reference frame used in motion compensated temporal filtering.

[0074] Reference Figures 4 to 7 , Figures 4 to 6 A single GOP is shown separately, and Figure 7 Two GOPs are shown, and multiple numbered boxes represent multiple video frames, respectively, where the numbers represent the POC of the video frame, that is, the video frame marked with the number "X" is the "X" frame. In the video encoding process, multiple video frames of the video are divided into multiple layers (or levels), and usually, encoding starts from layer 0 and continues upward layer by layer.

[0075] Group Of Picture (GOP): In video encoding, encoding is usually performed in units of GOP. GOP8 can mean that 1 GOP includes 8 video frames (or picture frames) (e.g. Figure 4 and Figure 7 As shown in FIG. 1 ), GOP16 may indicate that 1 GOP includes 16 video frames (as shown in FIG. Figure 5 ), and GOP32 may indicate that 1 GOP includes 32 video frames (such as Figure 6 shown).

[0076] Video frames of different frame types may be located in different layers, for example, Figure 4 , the starting frame of the video frame of the 0th layer (e.g., the 0th frame) is an I frame or a P frame and another video frame of the 0th layer (e.g., the 8th frame) is a P frame, the video frame of the top layer is a non-(encoded) reference frame (i.e., a b frame), and the video frames of the remaining intermediate layers are all bidirectional reference frames (i.e., B frames). In addition, Figure 5 GOP16 Figure 6 GOPO32 and Figure 7 The video frames of the two GOP8 segments have similar GOP structures.

[0077] (Encoding) reference frame: In the video encoding process, after the encoding of the video frame of the lower layer is completed, the video frame of the lower layer can be used as a reference for the video frame of the same layer or the upper layer, that is, the video frame of the lower layer can be used as the encoding reference frame of the video frame of the same layer or the upper layer during video encoding. For example, Figure 4 The reference relationship in can be: after the encoding of the 0th frame is completed, the 0th frame can be used as a coding reference frame for reference of the 1st to 8th frames; after the encoding of the 8th frame is completed, the 8th frame can be used as a coding reference frame for reference of the 1st to 7th frames; after the encoding of the 8th frame is completed, the 8th frame can be used as a coding reference frame for reference of the 1st to 3rd frames and the 5th to 7th frames.

[0078] Long-Term Reference (LTR): It is a technique used to improve coding efficiency in video coding. In the GOP structure of LTR technology, video frames included in one GOP can be referenced by video frames in one or more subsequent GOPs. Figure 7 In the encoding of the first GOP8, frame 0 can be referenced by subsequent frames 1 to 8 in the GOP8. If frame 0 is a long-term reference frame, frame 0 can also be referenced by some or all of the video frames from frame 9 to frame 16 in the second GOP8, and can also be referenced by other video frames in other GOPs after frame 16.

[0079] Motion Compensated Temporal Filter (MCTF) method: It uses the classic block-based motion estimation and motion compensation method, and uses the temporal correlation of the video to perform temporal filtering on the video frames to reduce the temporal redundant information generated during the video frame reference process. Schematically, MCTF consists of two parts: motion compensation (MC) and temporal filtering (TF). Figure 8 The implementation methods of these two parts are introduced respectively.

[0080] (1) Motion Compensation (MC) part. MCTF performs block-level motion estimation (ME) based on the image blocks of the video frame. That is, the video frame is evenly divided into 8×8 (width and height are 8 pixels) image blocks, and each image block is motion estimated in the adjacent previous and next video frames (these adjacent previous and next video frames can be called filtering reference frames) to obtain the motion vector of each image block relative to each filtering reference frame.

[0081] Specifically, in the related art, the traditional MCTF filtering method adopts a fixed filtering strategy: first, determine whether the video frame needs MCTF. When the video frame satisfies POC%16==0 (i.e., POC is a multiple of 8) or POC%8==0 (i.e., POC is a multiple of 4), the video frame needs to be MCTFed, and the four video frames before and after the video frame or the two video frames before and after the video frame are used as filtering reference frames; for the remaining video frames, MCTF is not performed on them. For example, referring to Figure 8 , if the current video frame (denoted as I t ) satisfies the above conditions and therefore requires MCTF, the four video frames before and after the current video frame (denoted as I t-4 ,I t-3 ,I t-2 ,I t-1 ,I t+1 ,I t+2 ,I t+3 ,I t+4 ) is used as a filtering reference frame. For example, referring to Figures 4 to 7 , showing video frames that require MCTF or do not require MCTF determined according to the filtering strategy of the related art.

[0082] Then, block-based motion estimation is performed based on the current video frame and the filtered reference frame. t Block-level motion estimation is performed with 8 adjacent video frames to search for I tThe most similar matching block of each 8×8 image block in the 8 reference frames is the motion vector. Correspondingly, if there are 8 filtering reference frames, then I t Each 8×8 image block in has 8 MVs (i.e., MV t-4 , MV t-3 , MV t-2 , MV t-1 , MV t+1 , MV t+2 , MV t+3 , MV t+4 ).

[0083] Subsequently, for any filtered reference frame, block-based motion compensation is performed on each image block based on the motion vector of each image block to obtain a motion compensated image block corresponding to each image block, and multiple motion compensated image blocks are spliced ​​to obtain a motion compensated frame; this motion compensation process is performed for each filtered reference frame to obtain a motion compensated frame corresponding to each filtered reference frame.

[0084] (2) Temporal filtering (TF) part. Based on the motion compensation frame corresponding to each filtering reference frame, bilateral filtering is performed on the current frame in units of image blocks to obtain the temporal filtering result. Figure 8 , for I t For each 8×8 image block in the image, the 9 image blocks (I t For example, nine image blocks can be weighted with a predetermined weight {W t-4 ,W t-3 ,W t-2 ,W t-1 ,W t ,W t+1 ,W t+2 ,W t+3 ,W t+4} After multiplication, the result of time domain filtering is obtained (i.e., a filtered 8×8 image block), which replaces I t The original image block in .

[0085] As described above, the MCTF method determines whether the current video frame needs to be filtered and the number of filtering reference frames required for filtering the current video frame based on the POC size of the video frame. This fixed filtering strategy lacks flexibility. Since filtering directly affects the reference relationship between video frames in the video encoding process, and the method of making filtering decisions based on POC only follows a fixed pattern and is not adaptable, the filtering reference frames used in the traditional MCTF method may have a greater impact on the filtering effect. Therefore, the present disclosure proposes a video frame processing method, device, electronic device and storage medium for time-domain filtering for motion compensation, especially adaptive adjustment of filtering reference frames based on frame type, coding reference relationship and / or GOP structure.

[0086] The implementation environment involved in the embodiments of the present disclosure is introduced below.

[0087] Fig. 9 is a schematic diagram showing an implementation environment of the video frame processing method and device of the present disclosure, see Fig. 9 , the implementation environment includes an electronic device 901. In some embodiments, the electronic device 901 can access a network through wired or wireless communication to achieve more functions.

[0088] The electronic device 901 is used to implement at least one of the following functions for the video: filtering function, encoding function, enhancement function, denoising function, etc., which is not limited in the present disclosure. In some embodiments, taking the electronic device 901 as an example for implementing the filtering function and encoding function for the video, the electronic device 901 samples the video to be encoded, obtains multiple video frames to be encoded of the video, and after time domain filtering these video frames, encodes these filtered video frames based on the video coding standard to obtain corresponding video encoding data (also called code stream), and transmits the video encoding data to the decoding device (the decoding device can also be the electronic device 901, for example, the video decoding function and the video encoding function are respectively implemented by different functional modules of the electronic device 901). Among them, the video coding standard usually adopts a block-based hybrid coding framework, including but not limited to: High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC) and Audio Video Coding Standard 3 (AVS3), etc., which are not limited in the present disclosure embodiment.

[0089] In addition, the present disclosure does not limit the form of the electronic device 901. For example, the electronic device 601 is a terminal or a server. Taking the electronic device 901 as a terminal as an example, the terminal is at least one of a smart phone, a smart watch, a desktop computer, a laptop, a virtual reality terminal, an augmented reality terminal, a wireless terminal and a laptop computer. The terminal has a communication function and can access a wired network or a wireless network. Taking the electronic device 901 as a server as an example, the server can be an independent physical server, or a server cluster or a distributed file system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Of course, the server can also include other functional servers to provide more comprehensive and diversified services.

[0090] The following will refer to Fig.10 A video frame processing method according to an exemplary embodiment of the present disclosure is described below. Fig.10 is a flow chart showing a method for processing a video frame according to an exemplary embodiment of the present disclosure. The method may be executed in, for example, the electronic device 901 described above to implement MCTF for the video frame.

[0091] like Fig.10 As shown, in step S1001, a plurality of video frames are obtained from a video to be used for encoding. Optionally, the plurality of video frames obtained may be video frames included in one or more GOPs, for example, Figure 4 That is, the acquired multiple video frames can be divided into multiple layers (or levels).

[0092] In step S1002, based on the frame type and the coding reference relationship of the current video frame to be filtered among the multiple video frames, a filtering reference frame of the current video frame is determined.

[0093] According to an embodiment of the present disclosure, MCTF is performed on all video frames that may be used for coding reference among multiple video frames, and whether MCTF is performed is not determined based on the size of the POC of the video frame. Therefore, the current video frame to be filtered can be a video frame used as a coding reference frame during video coding among multiple video frames, that is, all video frames whose frame types are I frames, P frames, and / or B frames. In other words, because the topmost non-coding reference frame in the GOP (i.e., b frame, for example, Figure 4 The b-frames at the third layer in (b-frames at the third layer in) are not referenced by other video frames during video encoding. Therefore, only the top-most b-frames are not filtered, and all video frames except the b-frames (e.g., Figure 4The video frames from layer 0 to layer 2 in are all filtered.

[0094] Since video frames with different frame types may have different characteristics (for example, bit rate, reference relationship, etc.) and as mentioned above, filtering has an impact on the encoding reference relationship in the encoding process, the present disclosure proposes to adaptively determine the filtering reference frame of the video frame based on the frame type and encoding reference relationship of the video frame currently to be filtered.

[0095] According to an embodiment of the present disclosure, step S1002 may include: determining candidate filtering reference frames of the current video frame, wherein the candidate filtering reference frames include video frames that use the current video frame as a coding reference frame during video coding; and determining at least one candidate filtering reference frame among the candidate filtering reference frames as a filtering reference frame of the current video frame based on a frame type of the current video frame. For example, with the current video frame I to be filtered t For example, if during video encoding, video frame I t Can be one or more other video frames I t '、I t "...I t "' reference, one or more other video frames I t '、I t "...I t "'Determined as video frame I t The candidate filtering reference frame is then based on the video frame I t Frame type, one or more other video frames I t '、I t "...I t "' is determined to be used for video frame I t The filtering reference frame of the filtering reference.

[0096] This will be described in detail below. First, an embodiment in which the frame type of the current video frame is an I frame will be described.

[0097] According to an embodiment of the present disclosure, based on the current video frame being an I frame, all video frames that use the current video frame (i.e., the I frame) as a coding reference frame during video encoding are determined as filtering reference frames of the current video frame. In other words, based on the current video frame being an I frame, all candidate filtering reference frames are determined as filtering reference frames of the current video frame.

[0098] In the video encoding process, the bit rate of an I frame is higher than that of a P frame, a B frame or a b frame, and the quality of a video frame as an I frame is better. Therefore, an I frame can provide a high-quality reference for a subsequently encoded video frame during encoding. According to an embodiment of the present disclosure, when filtering an I frame, all subsequent video frames that may refer to the I frame in the encoding process are determined as filtering reference frames of the I frame.

[0099] Furthermore, as described above, in order to improve coding efficiency, a video frame (eg, an I frame or a P frame) may be used as a long-term reference frame.

[0100] According to an embodiment of the present disclosure, a long-term reference frame is determined internally by a video encoder, and using a video frame as a long-term reference frame requires satisfying the following conditions: 1) the video encoder supports the long-term reference frame (LTR) encoding technology and it is turned on; 2) the current video frame is an I frame or a P frame, that is, only I frames and P frames can be used as long-term reference frames, and B frames or b frames will not be used as long-term reference frames.

[0101] According to an embodiment of the present disclosure, in response to a current video frame as an I frame not being used as a long-term reference frame during video encoding, a candidate filtering reference frame of the current video frame is determined based on the GOP to which the current video frame belongs. Specifically, when the I frame is not used as a long-term reference frame, the I frame is only used as a reference for video frames within one GOP, and therefore, all frames within one GOP are listed as filtering reference frames of the I frame. For example, with reference to Figure 4 In GOP8, the filtering reference frame of the I frame (ie, the 0th frame) includes 8 video frames (ie, the 1st frame to the 8th frame); Figure 5 In GOP16, the filtering reference frame of the I frame (ie, the 0th frame) includes 16 video frames (ie, the 1st frame to the 16th frame); Figure 6 In GOP32, the filtering reference frame of the I frame (ie, the 0th frame) includes 32 video frames (ie, the 1st frame to the 32nd frame); the determination of the filtering reference frame of the I frame in the remaining GOPs can be deduced by analogy.

[0102] According to an embodiment of the present disclosure, in response to a current video frame as an I frame being used as a long-term reference frame during video encoding, a candidate filtering reference frame of the current video frame is determined based on the GOP to which the current video frame belongs and the subsequent GOP. The I frame as a long-term reference frame can be used as a reference for video frames within multiple GOPs, so no matter which GOP other video frames that use the I frame as a coding reference frame belong to, these other video frames are determined as filtering reference frames for the I frame, and therefore, all video frames in the GOP to which the current video frame belongs and the subsequent GOP that use the I frame as a coding reference frame are determined as candidate filtering reference frames for the current video frame. For example, Figure 7The filtering reference frames of the I frame (i.e., the 0th frame) in GOP8 include 16 video frames (i.e., the 1st to 16th frames in two GOP8s). For another example, in GOP32, if the I frame can be used as the encoding reference frame of the video frames in the two subsequent consecutive GOP32s, the filtering reference frames of the I frame (i.e., the 0th frame) include 64 video frames (i.e., the 1st to 64th frames in the two GOP32s); the filtering reference frames of the I frames of the remaining GOPs can be deduced by analogy. In this case, as long as the I frame can be listed as an encoding reference frame by the subsequent video frames, the subsequent video frames need to be determined as the filtering reference frames of the I frame.

[0103] As described above, considering the higher quality of I frames, all video frames that may refer to I frames are determined as filtering reference frames of I frames, which can avoid the influence of filtering on the reference quality between video frames, thereby improving the encoding quality. In addition, as described above, the GOP structure is also taken into account. When the I frame is not used as a long-term reference frame, only the video frames in a single GOP are considered as filtering reference frames, and when the I frame is used as a long-term reference frame, the video frames in multiple GOPs are considered as filtering reference frames. Such adaptive determination of filtering reference frames based on the GOP structure can also improve the filtering effect.

[0104] The following describes an embodiment in which the frame type of the current video frame is a P frame.

[0105] According to an embodiment of the present disclosure, based on the current video frame being a P frame, all video frames that use the current video frame (i.e., the P frame) as a coding reference frame during video encoding are determined as candidate filtering reference frames of the current video frame, and according to the POC of the candidate filtering reference frame, the candidate filtering reference frame that meets the conditions is determined as the filtering reference frame of the current video frame. That is, after the candidate filtering reference frame is determined, a useful filtering reference frame is selected from the candidate filtering reference frames according to the POC of the candidate filtering reference frame.

[0106] In the video encoding process, although the bit rate of P frame is lower than that of I frame, it is still relatively high compared with B frame and b frame. Therefore, P frame can also provide effective reference for subsequent video frames during encoding. The difference between P frame and I frame is that P frame can be used as the encoding reference frame of the video frame before the current video frame (that is, the video frame with POC less than the POC of the P frame). Therefore, based on the current video frame being a P frame, there may be a situation where the POC of the filtered reference frame is less than the POC of the current video frame.

[0107] In addition, similar to the case where an I frame is used as a long-term reference frame, according to an embodiment of the present disclosure, in response to the current video frame being a P frame not being used as a long-term reference frame during video encoding, a candidate filtering reference frame of the current video frame is determined based on the GOP to which the current video frame belongs; and in response to the current video frame being a P frame being used as a long-term reference frame during video encoding, a candidate filtering reference frame of the current video frame is determined based on the GOP to which the current video frame belongs and subsequent GOPs. That is, all video frames in the GOP to which the current video frame belongs and subsequent GOPs that use the P frame as a coding reference frame are determined as candidate filtering reference frames of the current video frame.

[0108] For example, taking GOP32 as an example, after the encoding of the P frame with a POC of 32 (i.e., the 32nd frame) used as a long-term reference frame is completed, the 32nd frame can be referenced by the B frame or b frame with a POC of 1 to 31 and the subsequent B frame, b frame or P frame with a POC of 33 to 64. Therefore, the filtering reference frame of the P frame with a POC of 32 may include 63 candidate filtering reference frames (i.e., the forward 31 video frames and the subsequent 32 video frames in another GOP32).

[0109] According to an embodiment of the present disclosure, based on the current video frame being a P frame, all candidate filtering reference frames may be determined as filtering reference frames of the current video frame.

[0110] In addition, in the present disclosure, for a video frequency reference frame used as a P frame, after determining the candidate filtering reference frames of the P frame, the candidate filtering reference frames may be further screened to eliminate some of the candidate filtering reference frames in consideration of the reference relationship of the P frame.

[0111] According to an embodiment of the present disclosure, a POC threshold N (also referred to as a first threshold in the present disclosure) for a candidate filtering reference frame of a P frame is defined, where N is a natural number greater than or equal to 0. Optionally, the value of N may be based on the size of the GOP to which the current video frame belongs. GopNum Optionally, the value of N can be determined based on GopNum / 2≤N≤GopNum. For example, in GOP32, GopNum is 32, and N can be 16. It should be understood that the value of the POC threshold N here is only an example, and this application does not limit it.

[0112] According to an embodiment of the present disclosure, for each candidate filtering reference frame of the P frame, the filtering reference frame of the current video frame is determined based on the comparison result of the difference between the POC of the current video frame and the POC of the candidate filtering reference frame (i.e., the absolute value of the difference between the two POCs) and the POC threshold N, as well as the frame type of the candidate filtering reference frame.

[0113] Optionally, define the following equation (1):

[0114] abs(POC P -POC ref )≥N (1)

[0115] Among them, POC P and POC ref They respectively represent the POC of the current video frame as a P frame and its candidate filtering reference frame, and abs() represents taking the absolute value.

[0116] If the candidate filtering reference frame of the current video frame does not satisfy equation (1) (ie, the POC difference is less than the POC threshold N), the candidate filtering reference frame is directly determined as the filtering reference frame of the P frame.

[0117] If the candidate filtering reference frame of the current video frame satisfies equation (1) (i.e., the difference of POC is greater than or equal to the POC threshold value N), the candidate filtering reference frame is eliminated based on the fact that the candidate filtering reference frame is a non-coded reference frame (i.e., the topmost b-frame in the GOP), and / or the candidate filtering reference frame is determined as the filtering reference frame of the P frame based on the fact that the candidate filtering reference frame is a coded reference frame (i.e., an I-frame, a P-frame, or a B-frame). In other words, if equation (1) is satisfied, it is necessary to determine whether the candidate filtering reference frame is a b-frame. If it is a b-frame, the candidate filtering reference frame is not used as the filtering reference frame of the P frame.

[0118] For example, refer to Figure 6 For GOP32, taking N as 16 as an example, for the current video frame (i.e., the 32nd frame) as a P frame and its candidate filtering reference frames (i.e., the 1st to 31st frames in the forward direction), since the POC difference of the 17th to 31st frames relative to the 32nd frame is less than 16, the 17th to 31st frames are directly determined as the filtering reference frames of the P frame. In addition, for the 1st to 16th frames whose POC difference relative to the 32nd frame is greater than or equal to 16, since the 1st / 3rd / 5th / 7th / 9th / 11th / 13th / 15th frames are b frames and the remaining 2nd / 4th / 6th / 8th / 10th / 12th / 14th / 16th frames are non-b frames, only the 2nd / 4th / 6th / 8th / 10th / 12th / 14th / 16th frames as non-b frames are determined as the filtering reference frames of the P frame.

[0119] Therefore, according to an embodiment of the present disclosure, based on the current video frame being a P frame, a candidate filtering reference frame that satisfies a first condition is determined as a filtering reference frame of the current video frame, wherein the first condition includes: the difference between the POC of the current video frame and the POC of the candidate filtering reference frame is less than a first threshold, and / or the difference between the POC of the current video frame and the POC of the candidate filtering reference frame is greater than or equal to the first threshold and the candidate filtering reference frame is a coded reference frame.

[0120] As described above, considering the relatively high quality of P frames, video frames that are video frames with small POC differences or non-b frames that may refer to P frames are determined as filtering reference frames for P frames, which can avoid the impact of filtering on the reference quality between video frames and improve filtering efficiency, thereby improving encoding quality. In addition, as described above, the GOP structure is also taken into account. When P frames are not used as long-term reference frames, only video frames in a single GOP are considered as filtering reference frames, and when P frames are used as long-term reference frames, video frames in multiple GOPs are considered as filtering reference frames. Such adaptive determination of filtering reference frames based on the GOP structure can also improve the filtering effect.

[0121] The following describes an embodiment in which the frame type of the current video frame is a B frame.

[0122] Similar to the case where the current video frame is a P frame, according to an embodiment of the present disclosure, based on the current video frame being a B frame, all video frames that use the current video frame (i.e., the B frame) as a coding reference frame during video encoding are determined as candidate filtering reference frames for the current video frame, and according to the POC of the candidate filtering reference frames, the candidate filtering reference frames that meet the conditions are determined as the filtering reference frames for the current video frame. That is, after the candidate filtering reference frames are determined, the filtering reference frames are selected from the candidate filtering reference frames according to the POC of the candidate filtering reference frames.

[0123] In the video coding process, usually, except for layer 0 and the top layer (e.g. Figure 6 The video frames in the remaining layers except the 5th layer in ) are all B frames.

[0124] According to an embodiment of the present disclosure, based on the current video frame being a B frame, all candidate filtering reference frames may be determined as filtering reference frames of the current video frame.

[0125] In addition, in the present disclosure, for the video frequency reference frame serving as the B frame, after the candidate filtering reference frame of the B frame is determined, the candidate filtering reference frame may be further screened to eliminate some of the candidate filtering reference frames, taking into account the reference relationship of the B frame and the layer to which the B frame belongs.

[0126] According to an embodiment of the present disclosure, a POC threshold NB (also referred to as a second threshold in the present disclosure) for a candidate filtering reference frame of a B frame is defined, and NB is a natural number greater than or equal to 0.

[0127] Optionally, the value of NB can be determined based on the number of candidate filtering reference frames in the forward direction (or referred to as forward candidate filtering reference frames) or the number of candidate filtering reference frames in the backward direction (or referred to as backward candidate filtering reference frames) of the current video frame. Optionally, the value of NB can be determined based on refNum / 2≤NB≤refNum, where refNum is determined based on the number of forward and / or backward candidate filtering reference frames of the current video frame. Optionally, the value of NB can be refNum / 2+1. It should be understood that the value of the POC threshold NB here is only an example and is not limited to this application.

[0128] For example, refer to Figure 3 In GOP32, taking the B frame with POC=16 as an example, the B frame includes 15 forward candidate filtering reference frames (i.e., the 1st to 15th frames) and 15 backward candidate filtering reference frames (i.e., the 17th to 31st frames). In this case, the value of refNum is 15; if the number of forward candidate filtering reference frames is inconsistent with the number of backward candidate filtering reference frames, the value of refNum can be the larger number of the two. According to an embodiment of the present disclosure, the value of refNum can be equal to the number of forward / backward candidate filtering reference frames of the current video frame, or can be equal to the larger value of the number of forward candidate filtering reference frames and the number of backward candidate filtering reference frames.

[0129] Since multiple video frames are divided into multiple layers, and different B frames may be located in different intermediate layers. According to an embodiment of the present disclosure, B frames located in the same layer have the same refNum value and thus have the same POC threshold NB, and B frames located in different layers of the multiple layers have different refNum values ​​and thus have different POC thresholds NB. For example, according to an embodiment of the present disclosure, in a GOP, the refNum value of a B frame of a lower layer may generally be greater than or equal to the refNum value of a B frame of a higher layer. The description here is shown only as an example, and the present disclosure is not limited thereto.

[0130] For example, refer to Figure 3 For GOP32, for the B frame with POC=8 in the second layer, the number of candidate filtering reference frames in the forward and backward directions of the B frame is 7. Therefore, the value of refNum of the B frame with POC=8 can be 7; for the B frame with POC=24 in the second layer, the value of refNum can also be 7; however, for the B frame with POC=2 / 6 / 10 / 14 / 18 / 22 / 26 / 30 in the fourth layer, the value of refNum is 1.

[0131] According to an embodiment of the present disclosure, for each candidate filtering reference frame of the B frame, the filtering reference frame of the current video frame is determined based on the comparison result of the difference between the POC of the current video frame and the POC of the candidate filtering reference frame (i.e., the absolute value of the POC difference) and the POC threshold NB.

[0132] Optionally, define the following equation (2):

[0133] abs(POC B -POC ref )>NB (2)

[0134] Among them, POC B and POC ref They respectively represent the POC of the current video frame as a B frame and the candidate filtering reference frame, and abs() represents taking the absolute value.

[0135] If the candidate filtering reference frame of the current video frame does not satisfy equation (2) (ie, the POC difference is less than or equal to the POC threshold NB), the candidate filtering reference frame is directly determined as the filtering reference frame of the B frame.

[0136] If the candidate filtering reference frame of the current video frame satisfies equation (2) (ie, the POC difference is greater than the POC threshold NB), the candidate filtering reference frame is directly eliminated.

[0137] For example, refer to Figure 6 For GOP32 of the first layer, taking the B frame with POC=16 in the first layer as an example, refNum=15 and NB=8 can be determined, and only the 8th to 15th frames among the forward candidate filtering reference frames (i.e., the 1st to 15th frames) of the B frame and the 17th to 24th frames among the backward candidate filtering reference frames (i.e., the 17th to 31st frames) of the B frame are determined as the filtering reference frames of the B frame. Similarly, for the subsequent B frames of the first layer, refNum=15 and NB=8 can be directly determined, and the filtering reference frames of the subsequent B frames can be similarly determined.

[0138] Therefore, according to an embodiment of the present disclosure, based on the current video frame being a B frame, a candidate filtering reference frame that meets the second condition is determined as the filtering reference frame of the current video frame, wherein the second condition includes: the difference between the POC of the current video frame and the POC of the candidate filtering reference frame is less than or equal to a second threshold.

[0139] As described above, considering the reference relationship of B frames and the GOP structure, determining the video frames that may refer to B frames as the filtering reference frames of B frames as the parts with smaller POC differences can avoid the influence of filtering on the reference quality between video frames and improve the filtering efficiency, thereby improving the encoding quality. In addition, combined with the threshold setting of the level of the GOP structure, the process of determining whether the candidate filtering reference frames of the B frames are retained can be simplified, thereby improving the filtering efficiency.

[0140] In addition, in the present disclosure, in order to further improve the filtering quality, after executing step S1002, the determined filtering reference frame may be further screened.

[0141] According to an embodiment of the present disclosure, after determining the filtered reference frame of the current video frame, the filtered reference frame that meets the brightness condition (for example, the brightness difference or texture difference is too large) can be discarded based on the brightness of the current video frame and the filtered reference frame.

[0142] In the present disclosure, a filtered reference frame having a large brightness difference or texture difference relative to a current video frame is discarded by using brightness difference detection based on a grayscale histogram. Here, a grayscale histogram is a tool for counting the frequency (or probability) of each grayscale level (or referred to as grayscale level) in an image.

[0143] According to an embodiment of the present disclosure, based on the brightness of the current video frame and the filtered reference frame, discarding the filtered reference frame that meets the brightness condition includes: obtaining the grayscale histogram of the current video frame and the filtered reference frame; based on the grayscale histogram of the current video frame and the filtered reference frame, determining the brightness difference between the current video frame and the filtered reference frame; based on the brightness difference between the current video frame and the filtered reference frame, discarding the filtered reference frame that meets the brightness condition.

[0144] According to an embodiment of the present disclosure, for a current video frame and its filtered reference frame, obtaining a grayscale histogram of the current video frame and the filtered reference frame includes: defining multiple grayscale levels; for each grayscale level, counting the frequency of occurrence of the grayscale level in a video frame; and obtaining a grayscale histogram of the corresponding video frame based on the statistical probability of the grayscale levels.

[0145] Optionally, for an 8-bit image, the range of multiple gray levels may be defined as 0 to 255 ((1<<8)-1); for a 10-bit image, the range of multiple gray levels may be defined as 0 to 1023 ((1<<10)-1); other bit-depth images may have similar definitions. It should be understood that gray levels may be defined in other ways, and this application does not limit this.

[0146] Optionally, the probability of grayscale occurrence can be counted based on the size of the video frame and the YUV component of the video frame. For example, taking an 8-bit bit depth image as an example, for each grayscale level i (i is a value from 0 to 255), the frequency of grayscale level i in a video frame can be counted by the following equation (3):

[0147]

[0148] Among them, width and height represent the width and height of the video frame respectively, and Y represents the Y component of the YUV three components of the video frame.

[0149] That is to say, counting gray level i is essentially counting the number of occurrences (or the number of pixels) of which the pixel value in the Y component is equal to i. For example, for gray level 10, H(10) is the number of occurrences (or the number of pixels) of which the pixel value in the Y component is equal to 10. It should be understood that any other method of obtaining a gray histogram may be used, and this application does not limit this.

[0150] For each video frame in the current video frame and its filtered reference frame, after counting the frequency of each gray level, the gray histogram of the corresponding video frame can be directly determined based on the frequency of the gray value counted. After determining the gray histogram of the current video frame and its filtered reference frame, the brightness difference or texture difference between the current video frame and each filtered reference frame can be further determined.

[0151] According to an embodiment of the present disclosure, based on the grayscale histogram H of the current video frame cur And the grayscale histogram H of the filtered reference frame ref , the brightness difference deltaH between the current video frame and the filtered reference frame can be determined by the following equation (4):

[0152]

[0153] Wherein, maxRange represents the maximum grayscale of the video frame (for example, for an image with a bit depth of 8 bits, maxRange may be 255). It should be understood that any other method may be used to represent the brightness difference, and this application does not limit this.

[0154] According to an embodiment of the present disclosure, based on the brightness difference between the current video frame and the filtered reference frame, the filtered reference frame that meets the brightness condition is discarded, wherein the brightness condition includes that the brightness difference between the current video frame and the filtered reference frame is greater than or equal to a brightness threshold (or referred to as a third threshold), wherein the brightness threshold is determined based on the size of the video frame. For example, the brightness of the filtered reference frame can be determined by the following equation (5) to determine that the brightness of the filtered reference frame meets the condition, that is, the brightness difference or texture difference between the filtered reference frame and the current video frame is large:

[0155] deltaH≥thre×width×height (5)

[0156] Wherein, width and height represent the width and height of the video frame respectively, and thre represents the brightness threshold. It should be understood that any other conditions may be used to determine whether the brightness difference or texture difference is large, and this application does not limit this.

[0157] Optionally, different brightness thresholds may be set for different frame types of video frames.

[0158] For example, the brightness threshold for I frames may be set to 0.25, and the brightness threshold for P frames and B frames may be set to 0.30. According to equation (4), the larger the value of deltaH is, the greater the brightness difference or texture difference between the filtered reference frame and the current video is.

[0159] As described above, by using the brightness of the current video frame and its filtering reference frame, the filtering reference frame with large differences in texture and brightness from the current video frame to be filtered can be detected, and the detected filtering reference frame with large differences can be discarded, thereby further improving the filtering quality of the current video frame using the filtering reference frame. Tests have shown that the brightness based on the grayscale histogram can effectively improve the subjective and objective quality of filtering, and has obvious optimization effects in practical applications with badcases for MCTF.

[0160] Return to reference Fig.10 In step S1003, motion compensated temporal filtering (MCTF) is performed on the current video frame based on the filtering reference frame of the current video frame. For example, as described above, the filtering result of the current frame is obtained by MCTF based on the filtering reference frame.

[0161] Fig.11 is a block diagram showing a video frame processing apparatus according to an exemplary embodiment of the present disclosure. It should be understood that the apparatus can be implemented in the form of software, hardware, or a combination of software and hardware in an electronic device such as a smartphone.

[0162] like Fig.11 As shown, the video frame processing device 1100 may include: a video frame acquisition unit 1110 , a reference frame determination unit 1120 and a video frame filtering unit 1130 .

[0163] The video frame acquisition unit 1110 may be configured to acquire a plurality of video frames from a video to be used for encoding.

[0164] The reference frame determining unit 1120 may be configured to determine a filtering reference frame of the current video frame based on a frame type and an encoding reference relationship of the current video frame to be filtered among the plurality of video frames.

[0165] The video frame filtering unit 1130 may be configured to perform motion compensated temporal filtering on the current video frame based on a filtering reference frame of the current video frame.

[0166] According to an embodiment of the present disclosure, the current video frame to be filtered is a video frame used as an encoding reference frame during video encoding among a plurality of video frames.

[0167] According to an embodiment of the present disclosure, the reference frame determination unit 1120 may be configured to: determine candidate filtering reference frames for the current video frame, wherein the candidate filtering reference frames include video frames that use the current video frame as encoding reference frames during video encoding; and based on the frame type of the current video frame, determine at least one of the candidate filtering reference frames as a filtering reference frame for the current video frame.

[0168] According to an embodiment of the present disclosure, the reference frame determination unit 1120 may be configured to: in response to the current video frame not being used as a long-term reference frame during video encoding, determine a candidate filtering reference frame for the current video frame based on the image group to which the current video frame belongs; and / or in response to the current video frame being used as a long-term reference frame during video encoding, determine a candidate filtering reference frame for the current video frame based on the image group to which the current video frame belongs and a subsequent image group.

[0169] According to an embodiment of the present disclosure, the reference frame determination unit 1120 may be configured to: based on the current video frame being an I frame, determine all candidate filtering reference frames as filtering reference frames of the current video frame; and / or based on the current video frame being a P frame or a B frame, determine the candidate filtering reference frames that meet the conditions as filtering reference frames of the current video frame according to the video display order POC of the candidate filtering reference frames.

[0170] According to an embodiment of the present disclosure, the reference frame determination unit 1120 may be configured to: based on the current video frame being a P frame, determine a candidate filtering reference frame that satisfies a first condition as a filtering reference frame of the current video frame, wherein the first condition includes: a difference between the POC of the current video frame and the POC of the candidate filtering reference frame is less than a first threshold, and / or a difference between the POC of the current video frame and the POC of the candidate filtering reference frame is greater than or equal to the first threshold and the candidate filtering reference frame is a coded reference frame; and / or based on the current video frame being a B frame, determine a candidate filtering reference frame that satisfies a second condition as a filtering reference frame of the current video frame, wherein the second condition includes: a difference between the POC of the current video frame and the POC of the candidate filtering reference frame is less than or equal to a second threshold, wherein the first threshold is determined based on the size of the image group to which the current video frame belongs, and / or the second threshold is determined based on the number of forward candidate filtering reference frames or backward candidate filtering reference frames of the current video frame.

[0171] According to an embodiment of the present disclosure, the multiple video frames are divided into multiple layers, wherein B frames in the same layer have the same second threshold, and B frames in different layers among the multiple layers have different second thresholds.

[0172] According to an embodiment of the present disclosure, the reference frame determination unit 1120 may be further configured to: after determining the filtered reference frame of the current video frame, discard the filtered reference frame that meets the brightness condition based on the brightness of the current video frame and the filtered reference frame.

[0173] According to an embodiment of the present disclosure, the reference frame determination unit 1120 may be configured to: obtain grayscale histograms of the current video frame and the filtered reference frame; determine the brightness difference between the current video frame and the filtered reference frame based on the grayscale histograms of the current video frame and the filtered reference frame; and discard the filtered reference frame that meets the brightness condition based on the brightness difference between the current video frame and the filtered reference frame, wherein the brightness condition includes that the brightness difference between the current video frame and the filtered reference frame is greater than or equal to a third threshold, wherein the third threshold is determined based on the size of the video frame.

[0174] The above has been referred to Figures 4 to 10 A specific operation example of the video frame processing method according to the exemplary embodiment of the present disclosure has been described accordingly, and those skilled in the art should understand that the description can also be implemented in the video frame processing device 1100 .

[0175] According to an embodiment of the present disclosure, an electronic device is also provided. The electronic device comprises: at least one processor; at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, cause the at least one processor to execute the above reference Figures 4 to 10 The video frame processing method.

[0176] According to an embodiment of the present disclosure, a computer-readable storage medium is also provided. When the instructions in the computer-readable storage medium are executed by a processor of a filtering device / electronic device / server for video encoding, the filtering device / electronic device / server for video encoding can perform the above-mentioned Figures 4 to 10 The video frame processing method.

[0177] According to an embodiment of the present disclosure, a computer program product is also provided, including computer instructions, which, when executed by a processor, implement the above-mentioned Figures 4 to 10 The video frame processing method.

[0178] Fig.12 is a diagram illustrating a computing environment coupled with a user interface according to some embodiments of the present disclosure.

[0179] Fig.12 A computing environment 1210 is shown coupled to a user interface 1250. The computing environment 1210 may be part of a data processing server. The computing environment 1210 includes a processor 1220, a memory 1230, and an input / output (I / O) interface 1240.

[0180] The processor 1220 generally controls the overall operation of the computing environment 1210, such as operations associated with display, data acquisition, data communication, and image processing. The processor 1220 may include one or more processors for executing instructions to perform all or some of the steps in the above method. In addition, the processor 1220 may include one or more modules that facilitate the interaction between the processor 1220 and other components. The processor may be a central processing unit (CPU), a microprocessor, a single-chip microcomputer, a graphics processing unit (GPU), etc.

[0181] The memory 1230 is configured to store various types of data to support the operation of the computing environment 1210. The memory 1230 may include predetermined software 1232. Examples of such data include instructions for any application or method operating on the computing environment 1210, video data sets, image data, etc. The memory 1230 may be implemented by using any type of volatile or non-volatile memory device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk.

[0182] I / O interface 1240 provides an interface between processor 1220 and peripheral interface modules (e.g., keyboard, click wheel, buttons, etc.). Buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. I / O interface 1240 may be coupled to an encoder and a decoder.

[0183] In an embodiment, a non-transitory computer-readable storage medium including, for example, a plurality of programs in a memory 1230 and / or storing a bit stream generated by the above encoding method or a bit stream to be decoded by the above decoding method is also provided, wherein the plurality of programs can be executed by a processor 1220 in a computing environment 1210 to perform the above method. In one example, the plurality of programs can be executed by a processor 1220 in a computing environment 1210 to (for example, from Figure 2The video encoder 20 in the computing environment 1210 receives a bit stream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or one or more associated syntax elements, etc.), and can also be executed by the processor 1220 in the computing environment 1210 to perform the above-mentioned decoding method according to the received bit stream or data stream. In another example, the multiple programs can be executed by the processor 1220 in the computing environment 1210 to perform the above-mentioned encoding method to encode the video information (e.g., video blocks representing video frames, and / or one or more associated syntax elements, etc.) into a bit stream or data stream, and can also be executed by the processor 1220 in the computing environment 1210 to (e.g., to Figure 3 Alternatively, a non-transitory computer-readable storage medium may store the bit stream or data stream generated by the encoder (e.g., Figure 2 The video encoder 20 in FIG. 1 generates a video signal for a decoder (eg, Figure 3 A bitstream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or associated one or more syntax elements, etc.) used by the video decoder 30 in the video decoder 30 when decoding video data. The non-transitory computer-readable storage medium can be, for example, a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0184] In an embodiment, a bit stream generated by the above encoding method or a bit stream to be decoded by the above decoding method is provided. In an embodiment, a bit stream including encoded video information generated by the above encoding method or encoded video information to be decoded by the above decoding method is provided.

[0185] In an embodiment, a computing device is also provided, which includes: one or more processors (e.g., processor 1220); and a non-transitory computer-readable storage medium or memory 1230 in which multiple programs that can be executed by the one or more processors are stored, wherein the one or more processors are configured to perform the above-mentioned method when executing the multiple programs.

[0186] In an embodiment, a computer program product having instructions for storing or transmitting a bitstream is also provided, wherein the bitstream includes the encoded video information generated by the above encoding method or the encoded video information to be decoded by the above decoding method. In an embodiment, a computer program product including, for example, a plurality of programs in a memory 1230 is also provided, wherein the plurality of programs can be executed by a processor 1220 in a computing environment 1210 for performing the above method. For example, the computer program product may include a non-transitory computer-readable storage medium.

[0187] In an embodiment, the computing environment 1210 may be implemented by one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs, GPUs, controllers, microcontrollers, microprocessors, or other electronic components for performing the above methods.

[0188] In an embodiment, a method for storing a bit stream is also provided, comprising: storing the bit stream on a digital storage medium, wherein the bit stream comprises encoded video information generated by the above encoding method or encoded video information to be decoded by the above decoding method.

[0189] In an embodiment, a method for transmitting a bit stream generated by the above encoder is also provided. In an embodiment, a method for receiving a bit stream to be decoded by the above decoder is also provided.

[0190] The description of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or limited to the present disclosure. Many modifications, variations, and alternative embodiments will be apparent to one of ordinary skill in the art having the benefit of the teachings presented in the foregoing description and the associated drawings.

[0191] Unless otherwise specifically stated, the order of steps of the method according to the present disclosure is intended to be illustrative only, and the steps of the method according to the present disclosure are not limited to the order specifically described above, but can be changed according to actual conditions. In addition, at least one of the steps of the method according to the present disclosure can be adjusted, combined or deleted according to actual needs.

[0192] The examples are chosen and described in order to explain the principles of the present disclosure and to enable other persons skilled in the art to understand the various embodiments of the present disclosure and to best utilize the basic principles and various embodiments with various modifications suitable for the specific use contemplated. Therefore, it will be understood that the scope of the present disclosure is not limited to the specific examples of the disclosed embodiments, and modifications and other embodiments are intended to be included within the scope of the present disclosure.

Claims

1. A video frame processing method, characterized in that: include: Obtaining a plurality of video frames from a video to be used for encoding; Determining a filtering reference frame of the current video frame based on a frame type and a coding reference relationship of a current video frame to be filtered among the multiple video frames; Based on the filtering reference frame of the current video frame, the current video frame is subjected to motion compensated temporal filtering.

2. The method according to claim 1, characterized in that The current video frame to be filtered is a video frame used as an encoding reference frame during video encoding among a plurality of video frames.

3. The method according to claim 1, characterized in that Determining a filtering reference frame of the current video frame based on a frame type and a coding reference relationship of a current video frame to be filtered among the multiple video frames includes: determining candidate filtering reference frames for a current video frame, wherein the candidate filtering reference frames include video frames that use the current video frame as a coding reference frame during video encoding; At least one of the candidate filtering reference frames is determined as a filtering reference frame of the current video frame based on a frame type of the current video frame.

4. The method according to claim 3, characterized in that Determine the candidate filtering reference frame of the current video frame, including: In response to the current video frame not being used as a long-term reference frame during video encoding, determining a candidate filtering reference frame for the current video frame based on a group of pictures to which the current video frame belongs; and / or In response to the current video frame being used as a long-term reference frame during video encoding, candidate filtering reference frames for the current video frame are determined based on a group of pictures to which the current video frame belongs and a subsequent group of pictures.

5. The method according to claim 3, characterized in that: Determining, based on a frame type of the current video frame, at least one of the candidate filtering reference frames as a filtering reference frame of the current video frame, comprising: Based on the current video frame being an I frame, all candidate filtering reference frames are determined as filtering reference frames of the current video frame; and / or Based on whether the current video frame is a P frame or a B frame, according to the video display order POC of the candidate filtering reference frames, the candidate filtering reference frames that meet the conditions are determined as the filtering reference frames of the current video frame.

6. The method according to claim 5, characterized in that Based on whether the current video frame is a P frame or a B frame, according to the video display order POC of the candidate filtering reference frames, determining the candidate filtering reference frames that meet the conditions as the filtering reference frames of the current video frame, including: Based on the current video frame being a P frame, determining a candidate filtering reference frame that meets a first condition as a filtering reference frame of the current video frame, wherein the first condition includes: a difference between a POC of the current video frame and a POC of the candidate filtering reference frame is less than a first threshold, and / or a difference between the POC of the current video frame and a POC of the candidate filtering reference frame is greater than or equal to the first threshold and the candidate filtering reference frame is a coding reference frame; and / or Based on the current video frame being a B frame, determining a candidate filtering reference frame that meets a second condition as a filtering reference frame of the current video frame, wherein the second condition includes: a difference between a POC of the current video frame and a POC of the candidate filtering reference frame is less than or equal to a second threshold, The first threshold is determined based on the size of the image group to which the current video frame belongs, and / or the second threshold is determined based on the number of forward candidate filtering reference frames or backward candidate filtering reference frames of the current video frame.

7. The method according to claim 6, characterized in that The plurality of video frames are divided into a plurality of layers, The B frames in the same layer have the same second threshold, and the B frames in different layers among the multiple layers have different second thresholds.

8. The method according to claim 1, characterized in that After determining the filtering reference frame of the current video frame, the method further includes: Based on the brightness of the current video frame and the filtered reference frame, the filtered reference frame that meets the brightness condition is discarded.

9. The method according to claim 8, characterized in that Based on the brightness of the current video frame and the filtered reference frame, discard the filtered reference frame that meets the brightness condition, including: Get the grayscale histogram of the current video frame and the filtered reference frame; Determining a brightness difference between the current video frame and the filtered reference frame based on the grayscale histograms of the current video frame and the filtered reference frame; Based on the brightness difference between the current video frame and the filtered reference frame, the filtered reference frame that meets the brightness condition is discarded. The brightness condition includes that a brightness difference between the current video frame and the filtered reference frame is greater than or equal to a third threshold, wherein the third threshold is determined based on a size of the video frame.

10. A video frame processing device, characterized in that: include: A video frame acquisition unit, configured to acquire a plurality of video frames from a video to be used for encoding; a reference frame determining unit configured to determine a filtering reference frame of a current video frame based on a frame type and a coding reference relationship of a current video frame to be filtered among a plurality of video frames; The video frame filtering unit is configured to perform motion compensated time domain filtering on the current video frame based on a filtering reference frame of the current video frame.

11. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, When the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the video frame processing method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of a filtering device / electronic device / server for video encoding, the filtering device / electronic device / server for video encoding is enabled to perform the video frame processing method as described in any one of claims 1 to 9.

13. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the video frame processing method according to any one of claims 1 to 9 is implemented.

Images