Video frame processing method and device, electronic equipment and storage medium

Abstract

The invention provides a video frame processing method and device, electronic equipment and a storage medium. The video frame processing method comprises the following steps: performing motion estimation on to-be-filtered image blocks in a video frame at a plurality of size levels to obtain matching blocks of the plurality of size levels from adjacent frames of the video frame; according to a comparison result of the similarity between the to-be-filtered image blocks of adjacent size levels and the matching block, updating the motion vector of the to-be-filtered image block at the smaller size level in the adjacent size levels until the motion vector of the to-be-filtered image block of the minimum size level is updated; and using the updated motion vector of the to-be-filtered image block with the minimum size level to perform time domain filtering of motion compensation on the to-be-filtered image block on the minimum size level.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing technology, and in particular to a video frame processing method, device, electronic device and storage medium for motion compensated time-domain filtering. Background Art

[0002] Various electronic devices (e.g., digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smart phones, video teleconferencing devices, video streaming devices, etc.) support digital video. Electronic devices send and receive or otherwise transmit digital video data through a communication network, and / or store digital video data on a storage device. Due to the limited bandwidth capacity of the communication network and the limited storage resources of the storage device, before the video data is transmitted or stored, the video data can be compressed using a video codec according to one or more video codec standards. For example, video codec standards include general video codec (VVC), joint exploration test model (JEM), high efficiency video codec (HEVC / H.265), advanced video codec (AVC / H.264), moving picture expert group (MPEG) codec, etc. Video codecs generally use prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that utilize the inherent redundancy in video data. Video codecs are intended to compress video data into a form that uses a lower bit rate while avoiding or minimizing the degradation of video quality.

[0003] In related technologies, a motion estimation-compensated temporal filter (MCTF) is used to filter the video before encoding to improve the video encoding efficiency. This method uses the classic block-based motion estimation and motion compensation method, and uses the temporal correlation of the video to perform temporal filtering on the video frames to reduce the temporal redundant information generated during the video frame reference process, thereby improving the overall encoding compression rate of the video.

[0004] The MCTF method uses a fixed 8×8 block size to filter the image. In some high-noise video content areas, the 8×8 block is more susceptible to noise interference, which is prone to mismatching during the motion estimation process, and then introduces additional noise during the time domain filtering process. In addition, the MCTF method does not check the search results of its motion estimation process during the filtering process, but defaults to adding all matching blocks to the filtering of the current block. In some cases, one or more of the matching blocks may be significantly different from the current image block. In this case, if these matching blocks are still used for filtering, it will have a significant negative impact on the overall filtering effect. Summary of the invention

[0005] According to a first aspect of the present disclosure, a video processing method is provided, comprising: performing motion estimation on an image block to be filtered in a video frame at multiple size levels to obtain matching blocks of the multiple size levels from adjacent frames of the video frame; updating a motion vector of the image block to be filtered at a smaller size level among the adjacent size levels according to a comparison result of similarities between the image block to be filtered and the matching block at adjacent size levels until the motion vector of the image block to be filtered at a minimum size level is updated; and performing motion compensated time-domain filtering on the image block to be filtered at the minimum size level using the updated motion vector of the image block to be filtered at the minimum size level.

[0006] According to a first aspect of the present disclosure, the two adjacent size levels include a first size level and a second size level smaller than the first size level, wherein updating the motion vector of the image block to be filtered at the smaller size level in the adjacent size levels according to the comparison result of the similarity between the image block to be filtered and the matching block of the image block to be filtered at the adjacent size levels comprises: determining a first similarity between a first image block in the video frame and a first matching block of the first image block and a second similarity between a second image block in the video frame and a second matching block of the second image block, wherein the first image block is an image block obtained by dividing the image block to be filtered at the first size level, and the second image block is an image block obtained by dividing the image block to be filtered at the second size level; in response to the comparison result between the first similarity and the second similarity indicating that the first image block is closer to the image block to be filtered than the second image block, updating the second motion vector of the second image block to the first motion vector of the first image block, and re-determining the second matching block of the second image block using the updated second motion vector; in response to the comparison result between the first similarity and the second similarity indicating that the second image block is closer to the image block to be filtered than the first image block, keeping the second motion vector of the second image block unchanged.

[0007] According to a first aspect of the present disclosure, the similarity between a first image block and a first matching block is represented by a first mean square error MSE of pixels of the first image block and the first matching block, and the similarity between a second image block and a second matching block is represented by a second MSE between the second image block and the second matching block, wherein, in response to determining that the ratio between the sum of the first MSE and the sum of the second MSE is less than a first threshold, it is determined that the first matching block is closer to the image block to be filtered than the second matching block; in response to determining that the ratio between the sum of the first MSE and the sum of the second MSE is greater than or equal to the first threshold, it is determined that the second matching block is closer to the image block to be filtered than the first matching block.

[0008] According to the first aspect of the present disclosure, the method also includes: after determining that the ratio between the sum of the first MSEs and the sum of the second MSEs is less than a first threshold, further determining the difference between the second MSEs and the difference between the second motion vectors; in response to determining that the difference between the second MSEs exceeds the second threshold and the difference between the second motion vectors exceeds a third threshold, updating the second motion vector of the second image block to the first motion vector of the first image block; in response to the difference between the second MSEs not exceeding the second threshold or the difference between the second motion vectors not exceeding the third threshold, maintaining the second motion vector of the second image block.

[0009] According to the first aspect of the present disclosure, the third threshold includes a first sub-threshold and a second sub-threshold, and when the following conditions are met, it is determined that the difference between the second motion vectors exceeds the third threshold: the ratio of the maximum X component value to the minimum X component value in the second motion vector is greater than the first sub-threshold; the ratio of the maximum Y component value to the minimum Y component value in the second motion vector is greater than the second sub-threshold.

[0010] According to the first aspect of the present disclosure, it is determined that the difference between the second MSEs exceeds the second threshold when the following condition is satisfied: the difference between the maximum second MSE and the minimum second MSE is greater than or equal to the second threshold.

[0011] According to a second aspect of the present disclosure, a video frame processing method is provided, comprising: determining a plurality of matching blocks corresponding to an image block in a video frame; respectively determining a similarity between each of the plurality of matching blocks and the image block; adjusting a filtering weight corresponding to the matching block based on the similarity of the matching block; and weighting the matching block according to the adjusted filtering weight to perform motion compensated time-domain filtering on the image block.

[0012] According to a second aspect of the present disclosure, the similarity is indicated by a structural similarity index value SSIM between the image block and the matching block, wherein adjusting the filtering weight corresponding to the matching block based on the similarity of the matching block includes: comparing the SSIM value of the matching block with a first threshold; in response to determining that the SSIM value of the matching block is less than the first threshold, determining that the matching block is not used for filtering the image block; in response to determining that the SSIM value of the matching block is greater than or equal to the first threshold, keeping the weight of the matching block unchanged.

[0013] According to a second aspect of the present disclosure, adjusting the filter weight corresponding to the matching block based on the structural similarity of the matching block further includes: in response to determining that the SSIM values ​​of all matching blocks are less than a first threshold, determining not to filter the image block.

[0014] According to the second aspect of the present disclosure, adjusting the filter weight corresponding to the matching block based on the structural similarity of the matching block also includes: in response to determining that the SSIM value of the matching block is greater than a first threshold, comparing the SSIM value of the matching block with a second threshold, the second threshold being greater than the first threshold; in response to determining that the SSIM value of the matching block is greater than or equal to the second threshold, increasing the filter weight corresponding to the matching block in proportion to the SSIM value of the matching block.

[0015] According to a third aspect of the present disclosure, a video frame processing method is provided, comprising: performing motion estimation on an image block to be filtered in a video frame at multiple size levels to obtain matching blocks of the multiple size levels from adjacent frames of the video frame; updating a motion vector of the image block to be filtered at a smaller size level among the adjacent size levels according to a comparison result of similarities between the image block to be filtered and the matching block at adjacent size levels until the motion vector of the image block to be filtered at a minimum size level is updated; determining a plurality of matching blocks corresponding to the image block to be filtered using the updated motion vector of the image block to be filtered at the minimum size level; respectively determining a similarity between each of the plurality of matching blocks and the image block to be filtered; adjusting filtering weights corresponding to the plurality of matching blocks based on the similarities of the plurality of matching blocks; and weighting the plurality of matching blocks according to the adjusted filtering weights to perform motion compensated time domain filtering on the image block to be filtered.

[0016] According to a fourth aspect of the present disclosure, a video frame processing device is provided, comprising: a motion estimation unit, configured to perform motion estimation on an image block to be filtered in a video frame at multiple size levels to obtain matching blocks of the multiple size levels from adjacent frames of the video frame; a motion vector updating unit, configured to update the motion vector of the image block to be filtered at a smaller size level among the adjacent size levels according to a comparison result of the similarity between the image block to be filtered and the matching block at adjacent size levels, until the motion vector of the image block to be filtered at a minimum size level is updated; and a filtering unit, configured to perform motion compensated time domain filtering on the image block to be filtered at the minimum size level using the updated motion vector of the image block to be filtered at the minimum size level.

[0017] According to a fourth aspect of the present disclosure, the two adjacent size levels include a first size level and a second size level smaller than the first size level, wherein the motion vector update unit is configured to: determine a first similarity between a first image block in the video frame and a first matching block of the first image block, and a second similarity between a second image block in the video frame and a second matching block of the second image block, wherein the first image block is an image block obtained by dividing the image block to be filtered at a first size level, and the second image block is an image block obtained by dividing the image block to be filtered at a second size level; in response to a comparison result between the first similarity and the second similarity indicating that the first image block is closer to the image block to be filtered than the second image block, update the second motion vector of the second image block to the first motion vector of the first image block, and re-determine the second matching block of the second image block using the updated second motion vector; in response to a comparison result between the first similarity and the second similarity indicating that the second image block is closer to the image block to be filtered than the first image block, keep the second motion vector of the second image block unchanged.

[0018] According to a fourth aspect of the present disclosure, the similarity between the first image block and the first matching block is represented by a first mean square error MSE of pixels of the first image block and the first matching block, and the similarity between the second image block and the second matching block is represented by a second MSE between the second image block and the second matching block, wherein the motion vector update unit is configured to: in response to determining that the ratio between the sum of the first MSE and the sum of the second MSE is less than a first threshold, determine that the first matching block is closer to the image block to be filtered than the second matching block; in response to determining that the ratio between the sum of the first MSE and the sum of the second MSE is greater than or equal to the first threshold, determine that the second matching block is closer to the image block to be filtered than the first matching block.

[0019] According to a fourth aspect of the present disclosure, the motion vector update unit is further configured to: after determining that the ratio between the sum of the first MSEs and the sum of the second MSEs is less than a first threshold, further determine the difference between the second MSEs and the difference between the second motion vectors; in response to determining that the difference between the second MSEs exceeds the second threshold and the difference between the second motion vectors exceeds a third threshold, update the second motion vector of the second image block to the first motion vector of the first image block; in response to the difference between the second MSEs not exceeding the second threshold or the difference between the second motion vectors not exceeding the third threshold, maintain the second motion vector of the second image block.

[0020] According to a fourth aspect of the present disclosure, the third threshold includes a first sub-threshold and a second sub-threshold, and the motion vector update unit is configured to determine that the difference between the second motion vectors exceeds the third threshold when the following conditions are met: a ratio of a maximum X component value to a minimum X component value in the second motion vector is greater than the first sub-threshold; a ratio of a maximum Y component value to a minimum Y component value in the second motion vector is greater than the second sub-threshold.

[0021] According to a fourth aspect of the present disclosure, the motion vector update unit is configured to determine that the difference between the second MSEs exceeds a second threshold when the following condition is satisfied: a difference between a maximum second MSE and a minimum second MSE is greater than or equal to the second threshold.

[0022] According to a fifth aspect of the present disclosure, a video frame processing device is provided, comprising: a matching block determination unit, configured to determine a plurality of matching blocks corresponding to an image block in a video frame; a similarity determination unit, configured to respectively determine a similarity between each of the plurality of matching blocks and the image block; a weight adjustment unit, configured to adjust a filtering weight corresponding to the matching block based on the similarity of the matching block; and a filtering unit, configured to weight the matching block according to the adjusted filtering weight to perform motion compensated time domain filtering on the image block.

[0023] According to a fifth aspect of the present disclosure, the similarity is indicated by a structural similarity index value SSIM between the image block and the matching block, wherein the weight adjustment unit is configured to: compare the SSIM value of the matching block with a first threshold; in response to determining that the SSIM value of the matching block is less than the first threshold, determine that the matching block is not used for filtering the image block; in response to determining that the SSIM value of the matching block is greater than or equal to the first threshold, keep the weight of the matching block unchanged.

[0024] According to a fifth aspect of the present disclosure, the weight adjustment unit is further configured to: in response to determining that the SSIM values ​​of all matching blocks are smaller than a first threshold, determine not to filter the image block.

[0025] According to the fifth aspect of the present disclosure, the weight adjustment unit is also configured to: in response to determining that the SSIM value of the matching block is greater than a first threshold, compare the SSIM value of the matching block with a second threshold, the second threshold being greater than the first threshold; in response to determining that the SSIM value of the matching block is greater than or equal to the second threshold, increase the filtering weight corresponding to the matching block in proportion to the SSIM value of the matching block.

[0026] According to a sixth aspect of the present disclosure, a video frame processing device is provided, comprising: a motion estimation unit configured to perform motion estimation on an image block to be filtered in a video frame at multiple size levels to obtain matching blocks of the multiple size levels from adjacent frames of the video frame; a motion vector updating unit configured to update the motion vector of the image block to be filtered at a smaller size level among the adjacent size levels according to a comparison result of the similarity between the image block to be filtered at adjacent size levels and the matching block, until the motion vector of the image block to be filtered at a minimum size level is updated; a matching block determining unit configured to determine a plurality of matching blocks corresponding to the image block to be filtered using the updated motion vector of the image block to be filtered at the minimum size level; a similarity determining unit configured to respectively determine the similarity between each of the plurality of matching blocks and the image block to be filtered; a weight adjusting unit configured to adjust the filtering weights corresponding to the plurality of matching blocks based on the similarities of the plurality of matching blocks; and a filtering unit configured to weight the plurality of matching blocks according to the adjusted filtering weights to perform motion compensated time domain filtering on the image block to be filtered.

[0027] According to the seventh aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, prompt the at least one processor to execute the method described above.

[0028] According to an eighth aspect of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is enabled to execute the method as described above.

[0029] According to a ninth aspect of the present disclosure, a computer program product is provided, wherein instructions in the computer program product are executed by at least one processor to perform the method as described above.

[0030] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: by comparing the similarity between image blocks and matching blocks at different size levels and comparing the content and position differences between matching blocks, it is possible to determine at which size level the motion vector is more suitable for motion compensation, thereby reducing matching errors and solving the problem that fixed-size matching blocks are easily interfered by noise. In addition, the coding efficiency can be improved by excluding matching blocks that are not conducive to filtering and increasing the influence of matching blocks that are conducive to filtering in filtering.

[0031] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0033] Figure 1 is a block diagram illustrating an exemplary system for encoding and decoding video blocks according to some embodiments of the present disclosure.

[0034] Figure 2 is a block diagram illustrating an exemplary video encoder according to some embodiments of the present disclosure.

[0035] Figure 3 is a block diagram illustrating an exemplary video decoder according to some embodiments of the present disclosure.

[0036] Figure 4 is an example showing video frames and reference frames used in motion compensated temporal filtering.

[0037] Figure 5 is a schematic diagram illustrating an overall environment to which a video frame processing scheme according to an exemplary embodiment of the present disclosure is applied.

[0038] Figure 6 is a flowchart illustrating a video frame processing method according to an exemplary embodiment of the present disclosure.

[0039] Figure 7 is a schematic diagram showing corresponding blocks at different size levels in multi-size motion estimation for a video frame according to an exemplary embodiment of the present disclosure.

[0040] Figure 8 is a flowchart illustrating a video frame processing method according to another exemplary embodiment of the present disclosure.

[0041] Fig. 9 is a block diagram illustrating a video frame processing apparatus according to an exemplary embodiment of the present disclosure.

[0042] Fig.10 is a block diagram illustrating a video frame processing apparatus according to another exemplary embodiment of the present disclosure.

[0043] Fig.11 is a diagram illustrating a computing environment coupled with a user interface according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0044] Reference will now be made in detail to specific embodiments, examples of which are shown in the accompanying drawings. In the following detailed description, a number of non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, various alternatives may be used without departing from the scope of the claims, and the subject matter may be practiced without these specific details. For example, the subject matter presented herein may be implemented on many types of electronic devices having digital video capabilities.

[0045] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the drawings are used to distinguish objects, but not to describe any specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those shown in the drawings or described in the present disclosure.

[0046] Figure 1 is a block diagram illustrating an exemplary system 10 for encoding and decoding video blocks in parallel according to some embodiments of the present disclosure. Figure 1 As shown in , system 10 includes a source device 12 that generates and encodes video data to be later decoded by a target device 14. Source device 12 and target device 14 may include any of a wide variety of electronic devices, including cloud servers, server computers, desktop or laptop computers, tablet computers, smart phones, set-top boxes, digital televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some implementations, source device 12 and target device 14 are equipped with wireless communication capabilities.

[0047] In some implementations, target device 14 may receive the encoded video data to be decoded via link 16. Link 16 may include any type of communication medium or device capable of moving the encoded video data from source device 12 to target device 14.

[0048] In some other embodiments, the encoded video data may be sent from the output interface 22 to the storage device 32. The encoded video data in the storage device 32 may then be accessed by the target device 14 via the input interface 28.

[0049] like Figure 1 , source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Video source 18 may include sources such as, or a combination of such sources: a video capture device (e.g., a camera), a video archive containing previously captured video, a video feed interface for receiving video from a video content provider, and / or a computer graphics system for generating computer graphics data as source video.

[0050] The captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video data may be sent directly to target device 14 via output interface 22 of source device 12. The encoded video data may also (or alternatively) be stored on storage device 32 for later access by target device 14 or other devices for decoding and / or playback.

[0051] Target device 14 includes input interface 28, video decoder 30, and display device 34. Input interface 28 may include a receiver and / or a modem and receives encoded video data via link 16. The encoded video data transmitted via link 16 or provided on storage device 32 may include various syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements may be included within the encoded video data sent over a communication medium, stored on a storage medium, or stored on a file server.

[0052] The video encoder 20 and the video decoder 30 can operate according to a proprietary standard or an industry standard (e.g., VVC, HEVC, Part 10 of MPEG-4, AVC) or an extension of such a standard. It should be understood that the present application is not limited to a specific video encoding / decoding standard and can be applied to other video encoding / decoding standards. It is generally believed that the video encoder 20 of the source device 12 can be configured to encode the video data according to any of these current standards or future standards. Similarly, it is also generally believed that the video decoder 30 of the target device 14 can be configured to decode the video data according to any of these current standards or future standards.

[0053] The video encoder 20 and the video decoder 30 may be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic devices, software, hardware, firmware, or any combination thereof. When partially implemented in software, the electronic device may store instructions for the software in a suitable non-transitory computer-readable medium, and use one or more processors to execute the instructions in the hardware to perform the video encoding / decoding operations disclosed in the present disclosure. Each of the video encoder 20 and the video decoder 30 may be included in one or more encoders or decoders, and either of the encoders or decoders may be integrated as part of a combined encoder / decoder (CODEC) in the corresponding device.

[0054] Figure 2is a block diagram illustrating an exemplary video encoder 20 according to some embodiments described in the present application. The video encoder 20 can perform intra-frame prediction coding and inter-frame prediction coding on video blocks within a video frame. Intra-frame prediction coding relies on spatial prediction to reduce or remove spatial redundancy in video data within a given video frame or picture. Inter-frame prediction coding relies on temporal prediction to reduce or remove temporal redundancy in video data within adjacent video frames or pictures of a video sequence. It should be noted that in the field of video coding and decoding, the term "frame" can be used as a synonym for the term "image" or "picture".

[0055] like Figure 2 As shown in , the video encoder 20 includes a video data memory 40, a prediction processing unit 41, a decoded picture buffer (DPB) 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The prediction processing unit 41 further includes a motion estimation unit 42, a motion compensation unit 44, a segmentation unit 45, an intra-frame prediction processing unit 46, and an intra-frame block copy (BC) unit 48. In some embodiments, the video encoder 20 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and an adder 62 for video block reconstruction. A loop filter 63 such as a deblocking filter can be located between the adder 62 and the DPB 64 to filter the block boundary to remove the block effect from the reconstructed video. In addition to the deblocking filter, another loop filter (e.g., a sample adaptive offset (SAO) filter, a cross-component sample adaptive offset (CCSAO) filter, and / or an adaptive loop filter (ALF)) can also be used to filter the output of the adder 62. In some examples, the loop filter may be omitted, and the decoded video blocks may be provided directly by adder 62 to DPB 64. Video encoder 20 may take the form of a fixed or programmable hardware unit, or may be distributed among one or more of the illustrated fixed or programmable hardware units.

[0056] Video data memory 40 may store video data to be encoded by components of video encoder 20. Figure 1 Video source 18 is shown obtaining video data in video data memory 40. DPB 64 is a buffer that stores reference video data (eg, reference frames or pictures) for use by video encoder 20 in encoding the video data (eg, in intra or inter prediction coding modes).

[0057] like Figure 2As shown in , after receiving the video data, the segmentation unit 45 within the prediction processing unit 41 segments the video data into video blocks. This segmentation may also include segmenting the video frame into strips, tiles (e.g., a set of video blocks), or other larger coding units (CUs) according to a predefined splitting structure associated with the video data, such as a quadtree (QT) structure. It should be noted that the terms "block" or "video block" used herein may be a portion of a frame or picture, in particular a rectangular (square or non-square) portion. With reference to, for example, HEVC and VVC, a block or video block may be or correspond to a coding tree unit (CTU), a CU, a prediction unit (PU), or a transform unit (TU) and / or may be or correspond to a corresponding block (e.g., a coding tree block (CTB), a coding block (CB), a prediction block (PB), or a transform block (TB)) and / or a sub-block.

[0058] The prediction processing unit 41 may select one of a plurality of feasible prediction coding modes for the current video block based on the error results (e.g., coding rate and distortion level), such as one of one or more inter-frame prediction coding modes in a plurality of intra-frame prediction coding modes. The prediction processing unit 41 may provide the resulting intra-frame prediction coding block or inter-frame prediction coding block to the adder 50 to generate a residual block, and to the adder 62 to reconstruct the coding block for subsequent use as part of a reference frame. The prediction processing unit 41 also provides syntax elements (e.g., motion vectors, intra-frame mode indicators, partition information, and other such syntax information) to the entropy coding unit 56.

[0059] To select an appropriate intra-prediction coding mode for the current video block, intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-prediction coding of the current video block in relation to one or more neighboring blocks in the same frame as the current block to be encoded to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 may perform inter-prediction coding of the current video block in relation to one or more prediction blocks in one or more reference frames to provide temporal prediction. Video encoder 20 may perform multiple encoding passes, for example, to select an appropriate coding mode for each block of video data.

[0060] In some embodiments, motion estimation unit 42 determines an inter-prediction mode for a current video frame by generating a motion vector according to a predetermined pattern within a sequence of video frames, the motion vector indicating the displacement of a video block within the current video frame relative to a prediction block within a reference video frame. Motion estimation performed by motion estimation unit 42 is the process of generating a motion vector that estimates the motion of a video block. For example, a motion vector may indicate the displacement of a video block within a current video frame or picture relative to a prediction block within a reference frame associated with a current block being encoded within the current frame. The predetermined pattern may designate a video frame in a sequence as a P frame or a B frame. Intra BC unit 48 may determine a vector (e.g., a block vector) for intra BC coding in a manner similar to the motion vector determined by motion estimation unit 42 for inter prediction, or may utilize motion estimation unit 42 to determine a block vector.

[0061] Regardless of whether the prediction block is from the same frame according to intra-frame prediction or from a different frame according to inter-frame prediction, the video encoder 20 can form a residual video block by subtracting the pixel values ​​of the prediction block from the pixel values ​​of the current video block being encoded. The pixel difference values ​​forming the residual video block may include both luma component differences and chroma component differences.

[0062] Intra-prediction processing unit 46 may encode the current block using various intra-prediction modes, for example, during separate encoding passes, and intra-prediction processing unit 46 (or, in some examples, mode selection unit) may select an appropriate intra-prediction mode to use from the tested intra-prediction modes. Intra-prediction processing unit 46 may provide information indicating the intra-prediction mode selected for the block to entropy encoding unit 56. Entropy encoding unit 56 may encode the information indicating the selected intra-prediction mode into the bitstream.

[0063] After prediction processing unit 41 determines a prediction block for the current video block via inter-prediction or intra-prediction, adder 50 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block may be included in one or more TUs and provided to transform processing unit 52. Transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform (e.g., a discrete cosine transform (DCT) or a conceptually similar transform).

[0064] The transform processing unit 52 may send the resulting transform coefficients to a quantization unit 54. The quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may also reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, the quantization unit 54 may then perform a scan on the matrix including the quantized transform coefficients. Alternatively, the entropy coding unit 56 may perform the scan.

[0065] After quantization, entropy coding unit 56 entropy encodes the quantized transform coefficients into a video bitstream using, for example, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding method or technique. The encoded bitstream may then be sent to a video bitstream such as a Figure 1 The video decoder 30 shown, or archived as Figure 1 The video frame may be stored in storage device 32 as shown for later transmission to or retrieval by video decoder 30. Entropy encoding unit 56 may also entropy encode motion vectors and other syntax elements for the current video frame being encoded.

[0066] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual video block in the pixel domain for use in generating reference blocks for predicting other video blocks. As noted above, motion compensation unit 44 may generate a motion compensated prediction block from one or more reference blocks of a frame stored in DPB 64. Motion compensation unit 44 may also apply one or more interpolation filters to the prediction block to calculate sub-integer pixel values ​​for use in motion estimation.

[0067] Adder 62 adds the reconstructed residual block to the motion compensated prediction block generated by motion compensation unit 44 to generate a reference block for storage in DPB 64. The reference block may then be used as a prediction block by intra BC unit 48, motion estimation unit 42, and motion compensation unit 44 to inter-predict another video block in a subsequent video frame.

[0068] Figure 3 1 is a block diagram showing an exemplary video decoder 30 according to some embodiments of the present application. The video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, and a DPB 92. The prediction processing unit 81 further includes a motion compensation unit 82, an intra-frame prediction unit 84, and an intra-frame BC unit 85. The video decoder 30 may perform the above-mentioned Figure 2 The encoding process is substantially the inverse of the decoding process described with respect to video encoder 20. For example, motion compensation unit 82 may generate prediction data based on motion vectors received from entropy decoding unit 80, and intra-prediction unit 84 may generate prediction data based on intra-prediction mode indicators received from entropy decoding unit 80.

[0069] In some examples, embodiments of the present disclosure may be dispersed in one or more of the units of the video decoder 30. For example, the intra BC unit 85 may perform embodiments of the present application alone or in combination with other units of the video decoder 30, such as the motion compensation unit 82, the intra prediction unit 84, and the entropy decoding unit 80. In some examples, the video decoder 30 may not include the intra BC unit 85, and the functions of the intra BC unit 85 may be performed by other components of the prediction processing unit 81, such as the motion compensation unit 82.

[0070] Video data memory 79 may store video data, such as an encoded video bitstream, to be decoded by other components of video decoder 30. Video data stored in video data memory 79 may be obtained, for example, from storage device 32, from a local video source (e.g., a camera), via wired or wireless network communication of video data, or by accessing a physical data storage medium (e.g., a flash drive or hard disk).

[0071] During the decoding process, the video decoder 30 receives an encoded video bitstream representing video blocks of an encoded video frame and associated syntax elements. The entropy decoding unit 80 of the video decoder 30 entropy decodes the bitstream to generate quantization coefficients, motion vectors or intra-frame prediction mode indicators, and other syntax elements. The entropy decoding unit 80 then forwards the motion vectors or intra-frame prediction mode indicators, and other syntax elements to the prediction processing unit 81.

[0072] When a video frame is encoded as an intra-prediction coded (I) frame or for intra-coded prediction blocks in other types of frames, the intra-prediction unit 84 of the prediction processing unit 81 can generate prediction data for a video block of the current video frame based on the intra-prediction mode transmitted by the signal and reference data from a previously decoded block of the current frame.

[0073] When the video frame is encoded as an inter-frame prediction coding (i.e., B or P) frame, the motion compensation unit 82 of the prediction processing unit 81 generates one or more prediction blocks for the video block of the current video frame based on the motion vector and other syntax elements received from the entropy decoding unit 80. Each of the prediction blocks can be generated from a reference frame in one of the reference frame lists. The video decoder 30 can construct the reference frame lists, i.e., List 0 and List 1, based on the reference frames stored in the DPB 92 using a default construction technique.

[0074] In some examples, when a video block is encoded according to the intra BC mode described herein, intra BC unit 85 of prediction processing unit 81 generates a prediction block for the current video block based on the block vector and other syntax elements received from entropy decoding unit 80. The prediction block may be within a reconstructed region of the same picture as the current video block as defined by video encoder 20.

[0075] Motion compensation unit 82 and / or intra BC unit 85 determine prediction information for a video block of the current video frame by parsing motion vectors and other syntax elements, and then use the prediction information to generate a prediction block for the current video block being decoded.

[0076] Motion compensation unit 82 may also perform interpolation using interpolation filters as used by video encoder 20 during encoding of the video blocks to calculate interpolated values ​​for sub-integer pixels of reference blocks. In this case, motion compensation unit 82 may determine the interpolation filters used by video encoder 20 based on the received syntax elements and use these interpolation filters to produce the prediction blocks.

[0077] Inverse quantization unit 86 inverse quantizes the quantized transform coefficients provided in the bitstream and entropy decoded by entropy decoding unit 80, using the same quantization parameters calculated by video encoder 20 for each video block in the video frame to determine the degree of quantization. Inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients to reconstruct the residual block in the pixel domain.

[0078] After the motion compensation unit 82 or the intra BC unit 85 generates a prediction block for the current video block based on the vector and other syntax elements, the adder 90 reconstructs the decoded video block for the current video block by adding the residual block from the inverse transform processing unit 88 to the corresponding prediction block generated by the motion compensation unit 82 and the intra BC unit 85. A loop filter 91 (e.g., a deblocking filter, an SAO filter, a CCSAO filter, and / or an ALF) may be located between the adder 90 and the DPB 92 to further process the decoded video block. In some examples, the loop filter 91 may be omitted, and the decoded video block may be provided directly to the DPB 92 by the adder 90. The decoded video blocks in a given frame are then stored in the DPB 92, which stores reference frames for subsequent motion compensation of the next video block. The DPB 92 or a memory device separate from the DPB 92 may also store the decoded video for later presentation on a display device (e.g., Figure 1 on a display device 34).

[0079] Before introducing the embodiments of the present disclosure, some key terms involved in the embodiments of the present disclosure are introduced below.

[0080] Block Partition, in the video encoding process, refers to dividing the video frame to be encoded (or image frame) into several non-overlapping image blocks (or coding units). The image block here is also the basic unit of video encoding.

[0081] Motion Vector (MV), in the process of inter-frame coding, there is a certain correlation between the scenes in different video frames at adjacent moments. Therefore, a video frame is divided into many non-overlapping image blocks, and the displacement of all pixels in the image block is assumed to be the same. Then, for each image block, within a given search range of the reference frame, the block most similar to the current image block is found according to a certain matching criterion, that is, the reference image block (or reference block, matching block). The relative displacement between the reference image block and the current image block is called the motion vector MV.

[0082] Motion Estimation (ME), the process of motion estimation is also the process of obtaining the motion vector MV. The quality of motion estimation usually determines the size of the residual of video coding, that is, it affects the efficiency of video coding. The process of motion estimation can be regarded as the process of finding matching blocks in each reference frame using matching criteria, for example, matching the current image block with all image blocks in the reference frame to obtain the best matching block. Of course, other motion estimation methods can also be used, such as pyramid motion estimation, etc., which are not limited in this disclosure.

[0083] Motion Compensation (MC) is a method to describe the difference between adjacent frames (adjacent here means adjacent in coding relationship, but not necessarily in playback order). Specifically, it describes how each block of the previous frame moves to a certain position in the current frame. This method is often used in video compression / video codecs to reduce spatial redundancy in video sequences.

[0084] The Motion Compensated Temporal Filter (MCTF) method uses the classic block-based motion estimation and motion compensation method, and uses the temporal correlation of the video to perform temporal filtering on the video frames to reduce the temporal redundant information generated during the video frame reference process. Schematically, MCTF consists of two parts: the motion compensation part and the temporal filtering part. Figure 1 The implementation methods of these two parts are introduced respectively.

[0085] (1) Motion compensation (MC) part. MCTF performs block-level motion estimation ME on the basis of the image blocks of the video frame. That is, the video frame is evenly divided into 8×8 (width and height are 8 pixels) image blocks, and each image block is motion estimated in the adjacent frames (for example, the adjacent frames are the 4 frames between the current frame and the 4 frames after it. These adjacent frames are referred to as reference frames in the following text) to obtain the motion vector of each image block relative to each reference frame. Figure 4 As shown, if the current frame (denoted as I t ) needs to be filtered, we first need to determine I t The reference frame is usually four adjacent frames before and after (denoted as I t-4 ,I t-3 ,I t-2 ,I t-1 ,I t+1 ,I t+2 ,I t+3 ,I t+4 ).For example Figure 4 In the frame with POC=8, the 4th, 5th, 6th, 7th, 9th, 10th, 11th, and 12th frames are set as reference frames, and I t Block-level motion estimation is performed with the adjacent 8 frames to search for I t The most similar matching block of each 8×8 image block in the 8 reference frames, the position offset of the matching block relative to the current block is the motion vector MV t-4 , MV t-3 , MV t-2 , MV t-1 , MV t+1 , MV t+2 , MV t+3 , MV t+4 .

[0086] In this process, MCTF adopts a pyramid motion estimation method, that is, firstly, fast motion estimation is performed on the downsampled video frame, and then motion estimation is performed on the original resolution video frame based on the motion estimation result, wherein, motion estimation is first performed with integer pixel accuracy on the original resolution video frame, and then motion estimation is performed with sub-pixel accuracy based on the integer pixel motion estimation result, and finally the motion vector of each image block is obtained; then, for any reference frame, block-based motion compensation is performed on each image block based on the motion vector of each image block, and the motion compensated image block corresponding to each image block is obtained, and multiple motion compensated image blocks are spliced ​​to obtain a motion compensated frame; this motion compensation process is performed for each reference frame to obtain a motion compensated frame corresponding to each reference frame.

[0087] (2) Temporal filtering (TF) part. Based on the motion compensation frame corresponding to each reference frame, bilateral filtering is performed on the current frame in units of image blocks to obtain the temporal filtering result. Figure 4 As shown, I t Each 8×8 image block in the image is weighted with the 8 reference blocks searched in the adjacent 8 reference frames, a total of 9 image blocks according to the predetermined weight. Figure 1 As shown, the matching block and the image block can be compared with the predetermined weight {W t-4 ,W t-3 ,W t-2 ,W t-1 ,W t ,W t+1 ,W t+2 ,W t+3 ,W t+4 ,} to obtain the result of time domain filtering, that is, a filtered 8×8 image block, which will replace I t The original image block in .

[0088] The implementation environment involved in the embodiments of the present disclosure is introduced below.

[0089] Figure 5 is a schematic diagram showing an implementation environment of the video frame processing method and device of the present disclosure, see Figure 5 , the implementation environment includes an electronic device 501. In some embodiments, the electronic device 501 can access a network through wired or wireless communication to achieve more functions.

[0090] The electronic device 501 is used to implement at least one of the following functions for the video: filtering function, encoding function, enhancement function, denoising function, etc., which is not limited in the present disclosure. In some embodiments, taking the electronic device 501 as an example of implementing the filtering function and encoding function for the video, the electronic device 501 samples the video to be encoded, obtains multiple video frames to be encoded of the video, and after time domain filtering these video frames, encodes these filtered video frames based on the video coding standard to obtain corresponding video coding data (also called code stream, bit stream), and transmits the video coding data to the decoding device (the decoding device can also be the electronic device 501, for example, the video decoding function and the video coding function are respectively implemented by different functional modules of the electronic device 501). Among them, the video coding standard usually adopts a block-based hybrid coding framework, including but not limited to: High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC) and Audio Video Coding Standard 3 (AVS3), etc., which are not limited in the present disclosure embodiment.

[0091] In addition, the present disclosure does not limit the form of the electronic device 501. For example, the electronic device 501 is a terminal or a server. Taking the electronic device 501 as a terminal as an example, the terminal is at least one of a smart phone, a smart watch, a desktop computer, a laptop, a virtual reality terminal, an augmented reality terminal, a wireless terminal and a laptop computer. The terminal has a communication function and can access a wired network or a wireless network. Taking the electronic device 501 as a server as an example, the server can be an independent physical server, or a server cluster or a distributed file system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Of course, the server can also include other functional servers to provide more comprehensive and diverse services.

[0092] The following will refer to Figure 6 The video frame processing method according to an exemplary embodiment of the present disclosure is described below. The method may be executed in, for example, the electronic device 501 described above to implement MCTF for the video frame.

[0093] like Figure 6 As shown, in step S601, motion estimation is performed on the image blocks to be filtered in the video frame at multiple size levels to obtain matching blocks of the image blocks to be filtered at the multiple size levels from adjacent frames of the video frame.

[0094] Here, the video frame is the frame where the image block currently requiring filtering is located, for example, it can be Figure 4 The tth frame I shown t The adjacent frames may be a predetermined number of frames adjacent to the video frame where the image block is located, for example, Figure 4 The tth frame I shown t The 4 frames before and after I t-4 ,I t-3 ,I t-2 ,I t-1 ,I t+1 ,I t+2 ,I t+3 ,I t+4 .

[0095] According to an embodiment of the present disclosure, motion estimation can be performed on the image blocks to be filtered at three size levels of 32×32, 16×16 and 8×8, so that matching blocks at three size levels can be obtained. That is, the video frame can be divided into 32×32, 16×16 and 8×8 image blocks respectively, and then motion estimation is performed on the image blocks of the video frame. However, it should be understood that more or fewer size levels can be used.

[0096] Figure 7 1 shows corresponding blocks of different size levels in multi-size motion estimation for a video frame according to an exemplary embodiment of the present disclosure. Figure 7 As shown, the image block L0 at the 32×32 size level can be divided into image blocks M0, M1, M2, and M3 at the 16×16 size level, and divided into image blocks S0 to S12 at the 8×8 size level. The matching block L0' corresponding to the image block L0, the matching blocks M0' to M3' corresponding to the image blocks M0 to M3, and the image blocks S0' to S12' corresponding to the image blocks S0 to S12 can be obtained in the adjacent frames by motion estimation. The motion vector corresponding to the image block at the 32×32 size level can be MV 32 , the motion vectors corresponding to the four image blocks at the 16×16 size level can be The motion vectors corresponding to the 16 image blocks at the 8×8 size level can be

[0097] In step S603, based on the comparison result of the similarity between the image block to be filtered and the matching block at adjacent size levels, the motion vector of the image block to be filtered is updated at a smaller size level among the adjacent size levels until the motion vector of the image block to be filtered at the smallest size level is updated.

[0098] Here, two adjacent size levels may include a first size level and a second size level smaller than the first size level. Accordingly, S603 may include: determining a first similarity between a first image block in the video frame and a first matching block of the first image block, and a second similarity between a second image block in the video frame and a second matching block of the second image block, wherein the first image block is an image block obtained by dividing the image block to be filtered at a first size level, and the second image block is an image block obtained by dividing the image block to be filtered at a second size level; in response to a comparison result between the first similarity and the second similarity indicating that the first image block is closer to the image block to be filtered than the second image block, updating the second motion vector of the second image block to the first motion vector of the first image block, and re-determining the second matching block of the second image block using the updated second motion vector; in response to a comparison result between the first similarity and the second similarity indicating that the second image block is closer to the image block to be filtered than the first image block, keeping the second motion vector of the second image block unchanged.

[0099] For example, in the case of using three size levels of 32×32, 16×16 and 8×8, the first similarity between the matching block L0' at the 32×32 size level and the image block L0 and the second similarity between the matching blocks M0-M3 at the 16×16 size level and the corresponding matching blocks M0'-M3' can be determined first. Since the image blocks and matching blocks at the 32×32 size level correspond to the four image blocks and matching blocks at the 16×16 size level, the first similarity at the 32×32 size level can be compared with the second similarities at the four 16×16 size levels, that is, whether the matching block at the 32×32 size level is closer to the image block to be filtered than the four matching blocks at the 16×16 size level can be determined based on the similarity comparison result. If the matching block at the 32×32 size level is closer to the image block to be filtered than the four matching blocks at the 16×16 size level, it means that a matching error may have occurred in the motion estimation at the 16×16 size level, and it is necessary to convert the four motion vectors at the 16×16 size level into the motion vector of the image block at the 32×32 size level. If the matching block at the 32×32 size level is not closer to the image block to be filtered than the four matching blocks at the 16×16 size level, the motion vector of the matching block at the 16×16 size level can be kept unchanged, and the similarity comparison as described above is continued between the image blocks at the 16×16 size level and the 8×8 size level and the matching block, so as to finally update or maintain the motion vector of the image block at the 8×8 size level.

[0100] Next, in step S605, the updated motion vector of the image block to be filtered at the minimum size level is used to perform motion compensated time domain filtering on the image block to be filtered at the minimum size level. That is, after the motion vector of the image block at the 8×8 size level is finally updated through the process of S603 (it may be updated to a motion vector at the 32×32 size level or a motion vector at the 16×16 size level, or remain unchanged), the updated motion vector can be used to obtain matching blocks of the image block at the 8×8 size level through motion compensation, and these matching blocks are used to perform filtering, that is, these matching blocks are used to perform weighted operations to obtain new image blocks to replace the original image blocks.

[0101] As described above, by comparing the similarity between image blocks and matching blocks at different size levels, it is possible to determine at which size level the motion vector is more suitable for motion compensation, thereby reducing matching errors and solving the problem that fixed-size matching blocks are susceptible to noise interference.

[0102] According to an exemplary embodiment of the present disclosure, the mean square error MSE between the pixels of the image block and the matching block can be used to represent the similarity between the image block and the matching block. The larger the MSE, the greater the difference between the image block and the matching block. It should be understood that other indicators such as the structural similarity index value (SSIM) can be used to represent the similarity between the image block and the matching block, which is not limited by the present application.

[0103] According to an exemplary embodiment of the present disclosure, when MSE is used to represent the similarity between an image block and a matching block, the similarity between a first image block and a first matching block (for example, between an image block L0 and a matching block L0' at a 32×32 size level) can be represented by a first mean square error MSE of pixels between the first image block and the first matching block, and the similarity between a second image block and a second matching block (for example, between image blocks M0 to M3 and matching blocks M0' to M3' at a 16×16 size level) can be represented by a second MSE between the second image block and the second matching block, and step S603 may include: in response to determining that a ratio between the sum of the first MSEs and the sum of the second MSEs is less than a first threshold, determining that the first matching block is closer to the image block to be filtered than the second matching block; in response to determining that a ratio between the sum of the first MSEs and the sum of the second MSEs is greater than or equal to a first threshold, determining that the second matching block is closer to the image block to be filtered than the first matching block.

[0104] The MSE between the matching block and the image block can be calculated as follows:

[0105]

[0106] Where W and H are the width and height of the image block, I1 and I2 represent the image block to be filtered and the matching block in the current image t, respectively, (i, j) represents the pixel coordinates, I1(i, j) represents the pixel value of the image block at the coordinates (i, j), and I2(i, j) represents the pixel value of the matching block at the coordinates (i, j).

[0107] In this way, if it is determined that the ratio of the sum of the first MSE to the sum of the second MSE is less than the first threshold, it can be determined that the difference between the similarity between the first image block and the first matching block and the similarity between the second image block and the second matching block exceeds the predetermined range, and it is possible that the second matching block is affected by noise and an erroneous mismatch occurs. According to an exemplary embodiment of the present disclosure, the first threshold threshold1 for comparing the first MSE of the 32×32 size level with the second MSE of the 16×16 size level can be 2.75. It should be understood that the threshold value here is not limited and can be adjusted according to actual needs and circumstances. Taking the MV update of the 16×16 image block with a 32×32 image block as an example, Figure 7 As shown, a 32×32 block corresponds to four 16×16 blocks, where {MV 32 ,MSE 32} to represent the MV and MSE of the 32×32 block, and use and To represent the MV and MSE of four 16×16 blocks.

[0108] When it is determined through similarity comparison that a matching error may have occurred in a matching block of a smaller size level, in order to more accurately determine whether an error has occurred, it is possible to further determine whether an error has occurred based on the comparison between the second matching blocks. Since the contents of adjacent images are strongly correlated, the motion vectors of adjacent image blocks and the MSE between the image block and the matching block will not differ too much. Therefore, it is possible to further determine whether a mismatch has occurred due to noise interference during the search process of the second matching block based on the motion vector and MSE of the image block.

[0109] According to an exemplary embodiment of the present disclosure, step S603 may also include: after determining that the ratio between the sum of the first MSEs and the sum of the second MSEs is less than the first threshold, further determining the difference between the second MSEs and the difference between the second motion vectors; in response to determining that the difference between the second MSEs exceeds the second threshold and the difference between the second motion vectors exceeds the third threshold, updating the second motion vector of the second image block to the first motion vector of the first image block; in response to the difference between the second MSEs not exceeding the second threshold or the difference between the second motion vectors not exceeding the third threshold, maintaining the second motion vector of the second image block.

[0110] According to an exemplary embodiment of the present disclosure, the difference between the second MSEs may be determined according to the difference between the maximum second MSE and the minimum second MSE, and the difference between the motion vectors may be determined according to the difference between the X component and the Y component of the motion vector. Specifically, the third threshold may include a first sub-threshold and a second sub-threshold, and when the following conditions are met, it is determined that the difference between the second motion vectors exceeds the third threshold: the ratio of the maximum X component value to the minimum X component value in the second motion vector is greater than the first sub-threshold; the ratio of the maximum Y component value to the minimum Y component value in the second motion vector is greater than the second sub-threshold. When the following conditions are met, it is determined that the difference between the second MSEs exceeds the second threshold: the difference between the maximum second MSE and the minimum second MSE is greater than or equal to the second threshold.

[0111] Taking four 16×16 image blocks as an example, assuming that the minimum value of the x and y components of the MV of the four 16×16 blocks is (minX, minY) and the maximum value is (maxX, maxY), the minimum value of the MSE is minMSE and the maximum value is maxMSE, it can be expressed as:

[0112]

[0113] After determining the difference between the maximum second MSE and the minimum second MSE among the four blocks at the 16×16 size level, it may be determined whether the difference is greater than or equal to a second threshold value threshold2:

[0114] maxMSE–minMSE>=threshold2

[0115] After determining the maximum X component and the maximum Y component and the minimum X component and the minimum Y component in the MVs of the four blocks at the 16×16 size level, it may be determined whether a ratio of the maximum X component value to the minimum X component value in the second motion vector is greater than a first sub-threshold threshold3-1 and whether a ratio of the maximum Y component value to the minimum Y component value in the second motion vector is greater than a second sub-threshold threshold3-2:

[0116] maxX>minX*threshold3-1

[0117] maxY>minY*threshold3-2

[0118] According to an exemplary embodiment of the present disclosure, threshold2, threshold3-1, threshold3-2 may be real numbers greater than 1, for example, threshold2=17, threshold3-1=2.0, threshold=2.0.

[0119] When the above conditions are met, it can be determined that the search result of motion estimation at the 32×32 size level is better than that at the 16×16 size level. Therefore, the MVs of 4 image blocks at the 16×16 size level (i.e., the second image blocks) can be replaced with the MVs of the image blocks at the 32×32 size level (i.e., the first image blocks), that is:

[0120]

[0121]

[0122] After updating the MVs of the image blocks at the 16×16 size level, MV update can be performed between the image blocks at the 16×16 size level and the image blocks at the 8×8 size level as described above. When updating the MVs of the image blocks at the 8×8 size level, thresholds that are the same as or different from the thresholds (i.e., the above first threshold, second threshold, and third threshold) used when updating the image blocks at the 16×16 size level can be adopted. For example, threshold1 = 2.25, while the other thresholds can remain unchanged. That is, the sum sumMSE of the first MSEs of 4 image blocks at the 16×16 size level and their matching blocks can be calculated 16 and the sum sumMSE8 of the second MSEs of 16 image blocks at the 8×8 size level and their matching blocks. According to the results of the previous operations, the MVs of the image blocks at the 16×16 size level here can be the updated MVs, that is, the MVs of the image blocks at the 32×32 size level, and their matching blocks can be the matching blocks re - located using the updated MVs. When sumMSE 16 / sumMSE8 < threshold1, the maximum and minimum values of all MSE8s, as well as the maximum x - component and maximum y - component and minimum x - component and minimum y - component in the MVs8 at the 8×8 size level, are further determined, and by determining the conditions described above, that is, MaxMSE8 - MinMSE8 >= threshold2 and maxX > minX * threshold3 - 1, maxY > minY * threshold3 - 2, it is finally determined whether to update the motion vectors of the image blocks at the 8×8 size level.

[0123] After finally determining (i.e., updating or maintaining) the MVs of the image blocks at the smallest size level (i.e., the 8×8 size), the matching blocks in the neighboring frames are obtained at the smallest size level with the updated MVs, and weighted filtering is performed using the obtained matching blocks.

[0124] In the above - mentioned way, the differences between blocks of the same level can be used to more accurately determine whether there are incorrect matches in the motion estimation process, thereby further improving the accuracy of motion estimation based on the differences between blocks of different levels.

[0125] Figure 8 is a flow chart showing a video frame processing method according to another exemplary embodiment of the present disclosure. The method can also perform temporal filtering of motion compensation based on the similarity between the matching block and the image block.

[0126] In step S801, a plurality of matching blocks corresponding to image blocks in a video frame are determined. Figure 4 As shown, the video frame I can be processed in sequence t Perform block-level motion estimation with 8 adjacent reference frames to obtain I t The most similar matching block among the 8 reference frames for each 8×8 image block in .

[0127] Next, in step S803, the similarity between each matching block in the plurality of matching blocks and the image block is determined respectively.

[0128] A variety of indicators can be used to represent the similarity between the image block and the matching block. According to an exemplary embodiment of the present disclosure, the similarity can be indicated by a structural similarity index value (SSIM) between the image block and the matching block. The SSIM can be calculated by the following equation 2:

[0129]

[0130] Among them, x represents the image block to be filtered in the current video frame, y represents the matching block corresponding to the image block x, and μ x represents the pixel average value of image block x, μ y represents the pixel average of the matching block y, represents the pixel variance of image block x, is the pixel variance of the representation, σ xy is the pixel covariance of the corresponding position of image block x and matching block y. c1=(k1L) 2 ,c2=(k2L) 2 It is a constant used to maintain stability. L is the dynamic range of pixel values. In an 8-bit image video, L=256, and in a 10-bit image video, L=1024. k1=0.01, k2=0.03. The value of SSIM is [-1,1]. The larger the SSIM value, the more similar the image block x and the matching block y are. It should be understood that the above values ​​are exemplary.

[0131] In step S805, the filter weight corresponding to the matching block is adjusted based on the similarity of the matching block.

[0132] According to an exemplary embodiment of the present disclosure, when SSIM is used to indicate the similarity of a matching block, S805 may include: comparing the SSIM value of the matching block with a first threshold; in response to determining that the SSIM value of the matching block is less than the first threshold, determining that the matching block is not used for filtering the image block; in response to determining that the SSIM value of the matching block is greater than or equal to the first threshold, keeping the weight of the matching block unchanged.

[0133] Assumptions Figure 4 The SSIM values ​​of the matching blocks in the eight reference frames shown are {SSIM t-4 ,SSIM t-3 ,SSIM t-2 ,SSIM t-1 ,SSIM t+1 ,SSIM t+2 ,SSIM t+3 ,SSIM t+4}, the SSIM values ​​of these matching blocks can be compared with the first threshold SSIMthreshold1. If it is less than the threshold, the filtering weight value of the matching block can be set to 0, that is, the matching block will not participate in the future filtering process. If the SSIM value of the matching block is greater than or equal to SSIMthreshold1, the matching block will be used for subsequent filtering. According to an exemplary embodiment of the present disclosure, SSIMthreshold1 can be 0.6.

[0134] According to an exemplary embodiment of the present disclosure, if the SSIM values ​​of all matching blocks are smaller than the first threshold, the filtering process may not be performed, that is, the pixel value of the current image block is kept unchanged.

[0135] Finally, in step S807, the matching blocks are weighted according to the adjusted filtering weights to perform motion compensated temporal filtering on the image blocks. Figure 1 The description has been given and will not be repeated here.

[0136] According to an exemplary embodiment of the present disclosure, step S805 may further include: in response to determining that the SSIM value of the matching block is greater than the first threshold, comparing the SSIM value of the matching block with a second threshold, the second threshold being greater than the first threshold; in response to determining that the SSIM value of the matching block is greater than or equal to the second threshold, increasing the filter weight corresponding to the matching block in proportion to the SSIM value of the matching block. That is, when the similarity between the matching block and the image block is greater than a certain degree, that is, when the SSIM value of the matching block is greater than the second threshold, it can be considered that the image block and the matching block have a high degree of similarity, and a higher weight can be assigned to them.

[0137] According to an exemplary embodiment of the present disclosure, the weight of the matching block may be increased in proportion to the SSIM value according to the following Equation 3:

[0138]

[0139] Where SSIMthreshold2 represents the second threshold, and r represents the gain for the weight. t-3 ≥SSIMthreshold2, then the corresponding gain r is calculated according to the above equation (3) t-3 , and superimpose it on the corresponding Figure 2 The weight w in t-3 As shown in equation (4) below:

[0140] w t-3 =w t-3 × t-3 (4)

[0141] As described above, by adjusting the filtering weight according to the similarity between the image block and the matching block, the matching block that is not conducive to filtering can be effectively excluded and the influence of the matching block suitable for filtering in filtering can be adaptively increased, thereby improving the coding efficiency.

[0142] According to an exemplary embodiment of the present disclosure, it is also possible to Figure 6 and Figure 8 The video frame processing method of the exemplary embodiment of the present disclosure can be combined to further improve the coding efficiency. The video frame processing method according to the exemplary embodiment of the present disclosure may include: performing motion estimation on the image block to be filtered in the video frame at multiple size levels to obtain the matching blocks of the multiple size levels from the adjacent frames of the video frame; updating the motion vector of the image block to be filtered at a smaller size level in the adjacent size levels according to the comparison result of the similarity between the image block to be filtered and the matching block at the adjacent size levels until the motion vector of the image block to be filtered at the minimum size level is updated; using the updated motion vector of the image block to be filtered at the minimum size level, determining multiple matching blocks corresponding to the image block to be filtered; respectively determining the similarity between each of the multiple matching blocks and the image block to be filtered; adjusting the filtering weights corresponding to the multiple matching blocks based on the similarity of the multiple matching blocks; weighting the multiple matching blocks according to the adjusted filtering weights to perform motion compensated time domain filtering on the image block to be filtered.

[0143] For example, using 32×32, 16×16, and 8×8 size levels according to Figure 6After the video frame processing method updates the motion vector of the image block at the 8×8 size level and uses the updated motion vector to re-determine the matching block, the following method can be used: Figure 8 The video frame processing method further adjusts the weights of each matching block in the filtering process to obtain a better filtering effect. Figure 6 and Figure 8 All operations and details described are applicable to this method and will not be repeated here.

[0144] Fig. 9 is a block diagram showing a video frame processing apparatus according to an exemplary embodiment of the present disclosure. It should be understood that the apparatus can be implemented in the form of software, hardware, or a combination of software and hardware in an electronic device such as a smartphone.

[0145] like Fig. 9 As shown, the video frame processing apparatus 600 may include: a motion estimation unit 910 , a motion vector updating unit 920 and a filtering unit 930 .

[0146] The motion estimation unit 910 may be configured to perform motion estimation on the image block to be filtered in the video frame at multiple size levels to obtain matching blocks at multiple size levels from neighboring frames of the video frame.

[0147] The motion vector updating unit 920 may be configured to update the motion vector of the image block to be filtered at a smaller size level among adjacent size levels according to a comparison result of the similarity between the image block to be filtered at adjacent size levels and the matching block, until the motion vector of the image block to be filtered at the smallest size level is updated.

[0148] The filtering unit 930 is configured to filter the image block to be filtered at the minimum size level using the updated motion vector of the image block to be filtered at the minimum size level.

[0149] According to an exemplary embodiment of the present disclosure, two adjacent size levels include a first size level and a second size level smaller than the first size level, wherein the motion vector update unit 920 may be configured to: determine a first similarity between a first image block in a video frame and a first matching block of the first image block, and a second similarity between a second image block in a video frame and a second matching block of the second image block, wherein the first image block is an image block obtained by dividing the image block to be filtered at the first size level, and the second image block is an image block obtained by dividing the image block to be filtered at the second size level; in response to a comparison result between the first similarity and the second similarity indicating that the first image block is closer to the image block to be filtered than the second image block, update the second motion vector of the second image block to the first motion vector of the first image block, and re-determine the second matching block of the second image block using the updated second motion vector; in response to a comparison result between the first similarity and the second similarity indicating that the second image block is closer to the image block to be filtered than the first image block, keep the second motion vector of the second image block unchanged.

[0150] According to an exemplary embodiment of the present disclosure, the similarity between the first image block and the first matching block can be represented by a first mean square error MSE of pixels between the first image block and the first matching block, and the similarity between the second image block and the second matching block can be represented by a second MSE between the second image block and the second matching block, wherein the motion vector update unit 920 can be configured to: in response to determining that the ratio between the sum of the first MSE and the sum of the second MSE is less than a first threshold, determine that the first matching block is closer to the image block to be filtered than the second matching block; in response to determining that the ratio between the sum of the first MSE and the sum of the second MSE is greater than or equal to the first threshold, determine that the second matching block is closer to the image block to be filtered than the first matching block.

[0151] According to an exemplary embodiment of the present disclosure, the motion vector update unit 920 may also be configured to: after determining that the ratio between the sum of the first MSEs and the sum of the second MSEs is less than a first threshold, further determine the difference between the second MSEs and the difference between the second motion vectors; in response to determining that the difference between the second MSEs exceeds the second threshold and the difference between the second motion vectors exceeds a third threshold, update the second motion vector of the second image block to the first motion vector of the first image block; in response to the difference between the second MSEs not exceeding the second threshold or the difference between the second motion vectors not exceeding the third threshold, maintain the second motion vector of the second image block.

[0152] According to an exemplary embodiment of the present disclosure, the third threshold may include a first sub-threshold and a second sub-threshold, and the motion vector update unit 920 may determine that the difference between the second motion vectors exceeds the third threshold when the following conditions are satisfied: a ratio of a maximum X component value to a minimum X component value in the second motion vector is greater than the first sub-threshold; a ratio of a maximum Y component value to a minimum Y component value in the second motion vector is greater than the second sub-threshold.

[0153] According to an exemplary embodiment of the present disclosure, the motion vector update unit 920 may be configured to determine that the difference between the second MSEs exceeds the second threshold when the following condition is satisfied: the difference between the maximum second MSE and the minimum second MSE is greater than or equal to the second threshold.

[0154] The above has been referred to Figure 4-Figure 7 A specific operation example of the video frame processing method according to the exemplary embodiment of the present disclosure has been described accordingly, and those skilled in the art should understand that the description can also be implemented in the video frame processing device 900 .

[0155] Fig.10 is a block diagram illustrating a video frame processing apparatus 1000 according to another exemplary embodiment of the present disclosure.

[0156] The video frame processing apparatus 1000 according to an exemplary embodiment of the present disclosure may include a matching block determining unit 1010 , a similarity determining unit 1020 , a weight adjusting unit 1030 , and a filtering unit 1040 .

[0157] The matching block determining unit 1010 may be configured to determine a plurality of matching blocks corresponding to image blocks in a video frame.

[0158] The similarity determination unit 1020 may be configured to respectively determine the similarity between each matching block in the plurality of matching blocks and the image block.

[0159] The weight adjustment unit 1030 may be configured to adjust a filter weight corresponding to the matching block based on the similarity of the matching block.

[0160] The filtering unit 1040 may be configured to weight the matching block according to the adjusted filtering weight to perform motion compensated time-domain filtering on the image block.

[0161] According to an exemplary embodiment of the present disclosure, the similarity may be indicated by a structural similarity index value SSIM between the image block and the matching block, and the SSIM value between the image block and the matching block may be calculated as in the above equation 2. The weight adjustment unit 1030 may be configured to: compare the SSIM value of the matching block with a first threshold; in response to determining that the SSIM value of the matching block is less than the first threshold, determine that the matching block is not used for filtering the image block; in response to determining that the SSIM value of the matching block is greater than or equal to the first threshold, keep the weight of the matching block unchanged.

[0162] According to an exemplary embodiment of the present disclosure, the weight adjustment unit 1030 may be further configured to: in response to determining that the SSIM values ​​of all matching blocks are smaller than the first threshold, determine not to filter the image block.

[0163] According to an exemplary embodiment of the present disclosure, the weight adjustment unit 1030 may also be configured to: in response to determining that the SSIM value of the matching block is greater than the first threshold, compare the SSIM value of the matching block with a second threshold, the second threshold being greater than the first threshold; in response to determining that the SSIM value of the matching block is greater than or equal to the second threshold, increase the filter weight corresponding to the matching block in proportion to the SSIM value of the matching block. For example, the SSIM value of the matching block may be increased as shown in the above equation 3.

[0164] The above has been referred to Figure 8 The detailed operation of the video frame processing method corresponding to the video frame processing device 1000 is described. Those skilled in the art should understand that the operation of the method can correspond to the operation of the various components of the video frame processing device 1000 and can achieve the same technical effect.

[0165] According to an exemplary embodiment of the present disclosure, a video frame processing device is also provided. The device can be combined with Fig. 9 and Fig.10The processing process of the video frame processing device is to further improve the filtering effect. According to an exemplary embodiment of the present disclosure, the video frame processing device may include: a motion estimation unit, configured to perform motion estimation on the image block to be filtered in the video frame at multiple size levels to obtain the matching blocks of the multiple size levels from the adjacent frames of the video frame; a motion vector updating unit, configured to update the motion vector of the image block to be filtered at a smaller size level in the adjacent size levels according to the comparison result of the similarity between the image block to be filtered and the matching block at the adjacent size levels, until the motion vector of the image block to be filtered at the minimum size level is updated; a matching block determination unit, configured to use the updated motion vector of the image block to be filtered at the minimum size level to determine a plurality of matching blocks corresponding to the image block to be filtered; a similarity determination unit, configured to respectively determine the similarity between each of the plurality of matching blocks and the image block to be filtered; a weight adjustment unit, configured to adjust the filtering weights corresponding to the plurality of matching blocks based on the similarity of the plurality of matching blocks; and a filtering unit, configured to weight the plurality of matching blocks according to the adjusted filtering weights to perform motion compensated time domain filtering on the image block to be filtered.

[0166] It should be understood that the above reference Figure 6 and Figure 8 The various details and processes adopted by the described video frame processing method can be applied to the video frame processing device and will not be repeated here.

[0167] Fig.11 A computing environment 1110 is shown coupled to a user interface 1150. The computing environment 1110 may be part of a data processing server. The computing environment 1110 includes a processor 1120, a memory 1130, and an input / output (I / O) interface 1140.

[0168] The processor 1120 generally controls the overall operation of the computing environment 1110, such as operations associated with display, data acquisition, data communication, and image processing. The processor 1120 may include one or more processors for executing instructions to perform all or some of the steps in the above method. In addition, the processor 1120 may include one or more modules that facilitate the interaction between the processor 1120 and other components. The processor may be a central processing unit (CPU), a microprocessor, a single-chip microcomputer, a graphics processing unit (GPU), etc.

[0169] The memory 1130 is configured to store various types of data to support the operation of the computing environment 1110. The memory 1130 may include predetermined software 1132. Examples of such data include instructions for any application or method operating on the computing environment 1110, video data sets, image data, etc. The memory 1130 may be implemented by using any type of volatile or non-volatile memory device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk.

[0170] The I / O interface 1140 provides an interface between the processor 1120 and peripheral interface modules (e.g., keyboard, click wheel, buttons, etc.). The buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. The I / O interface 1140 may be coupled to an encoder and a decoder.

[0171] In an embodiment, a non-transitory computer-readable storage medium including, for example, a plurality of programs in a memory 1130 and / or storing a bitstream generated by the above video processing method is also provided, wherein the plurality of programs can be executed by a processor 1120 in a computing environment 1110 to perform the above method. In one example, the plurality of programs can be executed by a processor 1120 in a computing environment 1110 to (for example, from Figure 2 The video encoder 20 in the computing environment 1110 receives a bit stream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or one or more associated syntax elements, etc.), and can also be executed by the processor 1120 in the computing environment 1110 to perform the above-mentioned video processing method according to the received bit stream or data stream. In another example, the multiple programs can be executed by the processor 1120 in the computing environment 1110 to perform the above-mentioned video frame processing method to encode the video information (e.g., video blocks representing video frames, and / or one or more associated syntax elements, etc.) into a bit stream or data stream, and can also be executed by the processor 1120 in the computing environment 1110 to (e.g., to Figure 3 Alternatively, a non-transitory computer-readable storage medium may store the bit stream or data stream generated by the encoder (e.g., Figure 2 The video encoder 20 in the example generates a video frame for processing by a decoder (e.g., Figure 3A bitstream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or associated one or more syntax elements, etc.) used by the video decoder 30 in the video decoder 30 when decoding video data. The non-transitory computer-readable storage medium can be, for example, a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0172] In an embodiment, a bit stream generated by the above video frame processing method is provided. In an embodiment, a bit stream including encoded video information generated by the above video frame processing method is provided.

[0173] In an embodiment, a computing device is also provided, which includes: one or more processors (e.g., processor 1120); and a non-transitory computer-readable storage medium or memory 1130 in which multiple programs that can be executed by one or more processors are stored, wherein the one or more processors are configured to perform the above-mentioned method when executing the multiple programs.

[0174] In an embodiment, a computer program product having instructions for storing or transmitting a bitstream is also provided, the bitstream including encoded video information generated by the above video processing method. In an embodiment, a computer program product including multiple programs in, for example, a memory 1130 is also provided, the multiple programs can be executed by a processor 1120 in a computing environment 1110 for performing the above method. For example, the computer program product may include a non-transitory computer-readable storage medium.

[0175] In an embodiment, the computing environment 1110 may be implemented by one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs, GPUs, controllers, microcontrollers, microprocessors, or other electronic components for performing the above methods.

[0176] The description of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or limited to the present disclosure. Many modifications, variations, and alternative embodiments will be apparent to one of ordinary skill in the art having the benefit of the teachings presented in the foregoing description and the associated drawings.

[0177] Unless otherwise specifically stated, the order of steps of the method according to the present disclosure is intended to be illustrative only, and the steps of the method according to the present disclosure are not limited to the order specifically described above, but can be changed according to actual conditions. In addition, at least one of the steps of the method according to the present disclosure can be adjusted, combined or deleted according to actual needs.

[0178] The examples are chosen and described in order to explain the principles of the present disclosure and to enable other persons skilled in the art to understand the various embodiments of the present disclosure and to best utilize the basic principles and various embodiments with various modifications suitable for the specific use contemplated. Therefore, it will be understood that the scope of the present disclosure is not limited to the specific examples of the disclosed embodiments, and modifications and other embodiments are intended to be included within the scope of the present disclosure.

Claims

1. A video frame processing method, characterized in that: include: Performing motion estimation on the image block to be filtered in the video frame at multiple size levels to obtain matching blocks at the multiple size levels from adjacent frames of the video frame; According to the comparison result of the similarity between the image block to be filtered and the matching block at adjacent size levels, the motion vector of the image block to be filtered is updated at a smaller size level among the adjacent size levels until the motion vector of the image block to be filtered at the smallest size level is updated; Using the updated motion vector of the image block to be filtered at the minimum size level, motion compensated time-domain filtering is performed on the image block to be filtered at the minimum size level.

2. The method according to claim 1, characterized in that The two adjacent size levels include a first size level and a second size level smaller than the first size level, Wherein, according to the comparison result of the similarity between the image block to be filtered and the matching block of the image block to be filtered at adjacent size levels, updating the motion vector of the image block to be filtered at a smaller size level among the adjacent size levels comprises: Determine a first similarity between a first image block in the video frame and a first matching block of the first image block, and a second similarity between a second image block in the video frame and a second matching block of the second image block, wherein the first image block is an image block obtained by dividing the image block to be filtered at a first size level, and the second image block is an image block obtained by dividing the image block to be filtered at a second size level; In response to a comparison result between the first similarity and the second similarity indicating that the first image block is closer to the image block to be filtered than the second image block, updating the second motion vector of the second image block to the first motion vector of the first image block, and re-determining the second matching block of the second image block using the updated second motion vector; In response to a comparison result between the first similarity and the second similarity indicating that the second image block is closer to the image block to be filtered than the first image block, the second motion vector of the second image block is kept unchanged.

3. The method according to claim 2, characterized in that The similarity between the first image block and the first matching block is represented by the first mean square error MSE of the pixels of the first image block and the first matching block, and the similarity between the second image block and the second matching block is represented by the second MSE between the pixels of the second image block and the second matching block. In response to determining that the ratio between the sum of the first MSE and the sum of the second MSE is less than a first threshold, determining that the first matching block is closer to the image block to be filtered than the second matching block; In response to determining that the ratio between the sum of the first MSEs and the sum of the second MSEs is greater than or equal to a first threshold, it is determined that the second matching block is closer to the image block to be filtered than the first matching block.

4. The method according to claim 3, characterized in that Also includes: After determining that the ratio between the sum of the first MSEs and the sum of the second MSEs is less than a first threshold, further determining a difference between the second MSEs and a difference between the second motion vectors; In response to determining that the difference between the second MSEs exceeds a second threshold and the difference between the second motion vectors exceeds a third threshold, updating the second motion vector of the second image block to the first motion vector of the first image block; In response to the difference between the second MSEs not exceeding the second threshold or the difference between the second motion vectors not exceeding the third threshold, the second motion vector of the second image block is maintained.

5. The method according to claim 4, characterized in that When the following conditions are met, it is determined that the difference between the second MSEs exceeds the second threshold: The difference between the maximum second MSE and the minimum second MSE is greater than or equal to the second threshold, The third threshold includes a first sub-threshold and a second sub-threshold, and when the following conditions are met, it is determined that the difference between the second motion vectors exceeds the third threshold: The ratio of the maximum X component value to the minimum X component value in the second motion vector is greater than the first sub-threshold; The ratio of the maximum Y component value to the minimum Y component value in the second motion vector is greater than the second sub-threshold.

6. A video frame processing method, characterized in that: include: determining a plurality of matching blocks corresponding to the image blocks in the video frame; respectively determining a similarity between each matching block in the plurality of matching blocks and the image block; adjusting a filter weight corresponding to the matching block based on a similarity of the matching block; The matching block is weighted according to the adjusted filtering weight to perform motion compensated time domain filtering on the image block.

7. The method according to claim 6, characterized in that The similarity is indicated by a structural similarity index value SSIM between the image block and the matching block, Wherein, adjusting the filter weight corresponding to the matching block based on the similarity of the matching block includes: Comparing the SSIM value of the matching block with a first threshold; In response to determining that the SSIM value of the matching block is less than a first threshold, determining that the matching block is not used for filtering the image block; In response to determining that the SSIM value of the matching block is greater than or equal to a first threshold, keeping the weight of the matching block unchanged.

8. The method according to claim 7, characterized in that Adjusting the filter weight corresponding to the matching block based on the structural similarity of the matching block further includes at least one of the following operations: In response to determining that the SSIM values ​​of all matching blocks are less than a first threshold, determining not to filter the image block; In response to determining that the SSIM value of the matching block is greater than a first threshold, comparing the SSIM value of the matching block with a second threshold greater than the first threshold, and in response to determining that the SSIM value of the matching block is greater than or equal to the second threshold, increasing the filtering weight corresponding to the matching block in proportion to the SSIM value of the matching block.

9. A video frame processing method, characterized in that: include: Performing motion estimation on the image block to be filtered in the video frame at multiple size levels to obtain matching blocks at the multiple size levels from adjacent frames of the video frame; According to the comparison result of the similarity between the image block to be filtered and the matching block at adjacent size levels, the motion vector of the image block to be filtered is updated at a smaller size level among the adjacent size levels until the motion vector of the image block to be filtered at the smallest size level is updated; Determine a plurality of matching blocks corresponding to the image block to be filtered using the updated motion vector of the image block to be filtered at the minimum size level; respectively determining the similarity between each matching block in the plurality of matching blocks and the image block to be filtered; adjusting filter weights corresponding to the plurality of matching blocks based on similarities of the plurality of matching blocks; The multiple matching blocks are weighted according to the adjusted filtering weights to perform motion compensated time-domain filtering on the image block to be filtered.

10. A video frame processing device, comprising: a motion estimation unit configured to perform motion estimation on the image block to be filtered in the video frame at multiple size levels to obtain matching blocks of the multiple size levels from adjacent frames of the video frame; a motion vector updating unit, configured to update the motion vector of the image block to be filtered at a smaller size level among the adjacent size levels according to a comparison result of similarities between the image block to be filtered and the matching block, until the motion vector of the image block to be filtered at a minimum size level is updated; The filtering unit is configured to use the updated motion vector of the image block to be filtered at the minimum size level to perform motion compensated time domain filtering on the image block to be filtered at the minimum size level.

11. A video frame processing device, comprising: a matching block determination unit, configured to determine a plurality of matching blocks corresponding to the image blocks in the video frame; a similarity determination unit, configured to respectively determine a similarity between each matching block in the plurality of matching blocks and the image block; A weight adjustment unit, configured to adjust a filter weight corresponding to the matching block based on a similarity of the matching block; The filtering unit is configured to weight the matching block according to the adjusted filtering weight so as to perform motion compensated time domain filtering on the image block.

12. A video frame processing device, characterized in that: include: a motion estimation unit configured to perform motion estimation on the image block to be filtered in the video frame at multiple size levels to obtain matching blocks of the multiple size levels from adjacent frames of the video frame; a motion vector updating unit, configured to update the motion vector of the image block to be filtered at a smaller size level among the adjacent size levels according to a comparison result of similarities between the image block to be filtered and the matching block, until the motion vector of the image block to be filtered at a minimum size level is updated; a matching block determining unit configured to determine a plurality of matching blocks corresponding to the image block to be filtered using the updated motion vector of the image block to be filtered at the minimum size level; a similarity determination unit, configured to respectively determine the similarity between each matching block in the plurality of matching blocks and the image block to be filtered; A weight adjustment unit, configured to adjust the filter weights corresponding to the plurality of matching blocks based on the similarities of the plurality of matching blocks; The filtering unit is configured to weight the multiple matching blocks according to the adjusted filtering weights to perform motion compensated time domain filtering on the image block to be filtered.

13. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to perform the method of any one of claims 1 to 9.

14. A computer-readable storage medium, when instructions in the computer-readable storage medium are executed by at least one processor, enables the at least one processor to perform the method according to any one of claims 1 to 9.

15. A computer program product, characterized in that The instructions in the computer program product are executed by at least one processor to perform the method according to any one of claims 1 to 9.

Images