Systems and methods for enhancing image patch prediction

Abstract

This invention proposes a system and method for enhancing image patch prediction. In some embodiments, the method and system for enhancing image patch prediction first predict pixels in a video frame to generate prediction blocks. Predicted pixels include inter-frame prediction and / or intra-frame prediction. If both prediction methods are used simultaneously, a weighted combination of inter-frame and intra-frame prediction can be used. After prediction, a deep neural network is applied to enhance the prediction of the prediction blocks and adjacent reconstructed pixels. The system can determine the number of reconstructed pixels and transmit this number to the decoder. The enhanced prediction can be subtracted from the video frame and then further processed. If block artifacts are present, the system can filter the boundaries between the predicted pixels and adjacent reconstructed pixels.

Description

Technical Field

[0001] This invention relates to the field of video compression, and mainly to methods, computer programs and systems for enhancing image patch prediction. Background Technology

[0002] Video compression standards are primarily used to reduce bandwidth and video file size while maintaining high video quality. High Efficiency Video Coding (HEVC) is an existing video compression standard that, compared to Advanced Video Coding (AVC), achieves higher data compression at the same or similar bitrate while maintaining comparable video quality. HEVC employs the Integer Discrete Cosine Transform (DCT) for varying image patch sizes and the Discrete Sine Transform (DST) for 4x4 patch sizes. Essentially, this standard compares different parts of a video frame, identifies redundant regions within a single frame or between consecutive frames, and then replaces these redundant regions with brief descriptions instead of the original pixels.

[0003] In image-block-based video coding, the system first divides the video into multiple image blocks, which can be called Largest Coding Units (LCUs) or Macroblocks (MBs). Each LCU can be further divided into smaller image blocks for further prediction and reconstruction.

[0004] Typically, prediction for each image block within a specific frame can employ inter-frame prediction and / or intra-frame prediction. Intra-frame prediction uses reconstructed information from the current frame to predict the information for the current block. Conversely, inter-frame prediction uses information from other coded frames to reconstruct the information for the current block.

[0005] Whether using inter-frame prediction or intra-frame prediction, prediction errors and artifacts can occur, especially at block boundaries. In some extreme cases, these errors and artifacts can degrade the video viewing experience. Reducing the degree of prediction or compression can reduce or mitigate these errors, but reducing the compression rate increases the bit rate, which may lead to jitter or latency when streaming video.

[0006] Reducing image patch prediction errors and artifacts while maintaining low bit rates is of great significance, thus creating an urgent need for new methods to improve video quality at low bit rates. To this end, this invention proposes a system and method for enhancing image patch prediction. Summary of the Invention

[0007] The system and method of this application relate to video compression technology, primarily referring to the implementation of image block prediction enhancement during video encoding. This system and method can reduce errors in image block prediction while maintaining a low bit rate in the encoded video frames.

[0008] In some embodiments, the image patch prediction enhancement method and system first predict pixels in a video frame to generate prediction patches. Predicted pixels include inter-frame prediction and / or intra-frame prediction. When both prediction methods are used simultaneously, a weighted combination of inter-frame and intra-frame prediction can be used. Intra-frame prediction includes angle prediction, intra-frame patch copying, and color mode operations. Inter-frame prediction includes patch segmentation, unidirectional prediction, and bidirectional prediction.

[0009] After prediction, a deep neural network is used to enhance the prediction of the predicted block and adjacent reconstructed pixels. The system determines the number of reconstructed pixels and transmits this number to the decoder. The enhanced prediction is subtracted from the video frame for further processing.

[0010] In some cases, it's advisable to first determine if block artifacts exist. If present, the system can filter the boundaries between the predicted pixel and its adjacent reconstructed pixels, as well as the boundaries of the predicted block. A low-pass filter can be used to perform these filtering operations. Finally, the system can select an enhancement mode from a predefined set of enhancement modes based on the current pixel and transmit the selected enhancement mode to the decoder.

[0011] It is worth noting that the various functions of the present invention described above can be implemented individually or in combination. The functions of the present invention will be described in more detail below with reference to the accompanying drawings. Attached Figure Description

[0012] To more clearly illustrate the present invention, some embodiments of the present invention will be described below with reference to the accompanying drawings, wherein:

[0013] Figure 1 This is an example block diagram of a system for encoding and transmitting video content according to an embodiment of the present invention;

[0014] Figure 2 This is an example logic block diagram of video encoding according to an embodiment of the present invention;

[0015] Figure 3 and Figure 4 This is an example image of predicted pixels drawn according to an embodiment of the present invention;

[0016] Figure 5 This is an example flowchart illustrating image patch prediction during video encoding, as shown in an embodiment of the present invention.

[0017] Figure 6This is a flowchart of a sub-process for pixel prediction drawn according to an embodiment of the present invention;

[0018] Figure 7 This is a flowchart of a sub-process of intra-frame prediction drawn according to an embodiment of the present invention;

[0019] Figure 8 This is a flowchart of a sub-process for inter-frame prediction drawn according to an embodiment of the present invention;

[0020] Figure 9A and Figure 9B This is a flowchart of a sub-process of the enhanced prediction method according to an embodiment of the present invention; and

[0021] Figure 10A and Figure 10B This is an example diagram of a computer system capable of intelligent quantization operations, drawn according to an embodiment of the present invention. Detailed Implementation

[0022] The present invention will be described in detail below with reference to several embodiments shown in the accompanying drawings, wherein some technical details will be described to facilitate a comprehensive understanding of the embodiments of the present invention. However, those skilled in the art can also implement the embodiments of the present invention without these specific details. On the other hand, well-known technical steps and / or structures will not be described in detail to avoid unnecessarily obscuring the present invention. The functionality and advantages of the embodiments can be better understood with reference to the following drawings and descriptions.

[0023] The accompanying drawings and descriptions below will help to understand the various aspects, functions, and advantages of exemplary embodiments of the present invention. Those skilled in the art should understand that the embodiments of the present invention described herein are presented by way of example only and are not intended to limit. All functions disclosed herein can be replaced by other functions having the same or similar purpose unless explicitly stated otherwise. Therefore, other modified embodiments also fall within the scope of the present invention and its equivalents as defined herein. Therefore, the imperative and / or sequential terms used in this specification, such as “will,” “will not,” “should,” “should not,” “must,” “must not,” “first,” “initially,” “next,” “subsequently,” “before,” “after,” “finally,” and “end,” etc., are not intended to limit the scope of the invention, as the embodiments disclosed herein are merely illustrative examples.

[0024] This invention relates to systems and methods for enhancing image patch prediction when encoding video content. Figure 1This is an example diagram of a system 100 employing High Efficiency Video Coding (HEVC). The coding standard aims to achieve the highest possible coding efficiency. Coding efficiency refers to the ability to encode video at the lowest possible bit rate while achieving the highest video quality. The encoding and transmission module 102 uses intra-frame prediction to segment the input image into multiple block regions as the first image frame, or the first frame of the random access point. Intra-frame prediction refers to using other pixels within the same frame to predict blocks or pixels within that frame. After predicting the first frame using intra-frame prediction, inter-frame prediction can be used to predict other frames. Inter-frame prediction refers to predicting the content of image blocks based on data from adjacent frames. After prediction, the image passes through a loop filter, and the final image is stored in the decoded image buffer. The image stored in the decoded image buffer can be used to predict other images.

[0025] In this system, multiple sub-components of the encoding and transmission module 102 receive the input video 110. These sub-components include a general encoder 120, a transform-scalar-quantizer 130, an intra-frame predictor 143, and an inter-frame predictor 155. The general encoder 120 generates general control data, which is formatted with a header and processed using context-adaptive binary arithmetic coding (CABAC) before being merged into the encoded bitstream. The general control data is also transmitted to the transform-scalar-quantizer 130, the intra-frame predictor 143, and the inter-frame predictor 155 (not shown in the figure).

[0026] The transform-scalar-quantizer 130 performs scaling and transform operations on the input video frames, quantizes the output data as transform coefficients, and merges them into the encoded bitstream after header formatting and CABAC processing. The output data is also transmitted to the scaling and inverse transformer 170. Transform units of different sizes can be applied to encode the prediction residuals. These transform units can use discrete cosine transform or discrete sine transform. Then, the scaling and inverse transformer 170 transmits the output data to the deblocking and filtering module 180, as well as the intra-frame predictor 143 and intra-frame estimator 145.

[0027] Intra-frame predictor 143 uses various prediction algorithms to estimate the pixel values ​​of adjacent pixels within the same frame. The output data (also called the estimate) of intra-frame predictor 143 is transmitted to intra-frame predictor 145, which uses the estimate to generate a prediction of the target pixel, i.e., generates intra-frame predicted image data. Conversely, inter-frame predictor 155 receives adjacent frame data from decoded image buffer 190 and estimates the motion between the frame and adjacent frames. The motion estimation output data is transmitted to inter-frame compensator 153 and, after header formatting and CABAC processing, is finally merged into the coded bitstream (not shown).

[0028] Inter-frame compensator 153 generates motion compensation data. Intra-frame / inter-frame selector 160 selects between the intra-frame predicted image data and the motion compensation data, and feeds the selected data back to transform-scalar-quantizer 130 and deblocking and filtering module 180 (not shown).

[0029] The deblocking and filtering module 180 generates filter control data, which is then formatted with a header and processed using CABAC before being merged into the encoded bitstream (not shown). The deblocking and filtered data (i.e., the filter control data) is also transmitted to the decoded image buffer 190. The output of the decoded image buffer 190 includes the output video 199.

[0030] refer to Figure 2 The figure illustrates an example flowchart of the logical flow and data transformation for generating a bitstream 290 from the original video 210. First, the original video 210 is divided into image blocks (or chunks), including dividing video frames into blocks of the same or different sizes. In some embodiments, the block size ranges from 4x4 to 64x64 pixels. Then, a two-dimensional discrete cosine transform (DCT) 220 operation is performed on each block. DCT can significantly reduce the amount of memory and bandwidth required for compressed video. The DCT 220 operation is performed on each frame. Whether for intra-frame coding or inter-frame coding, DCT 220 is calculated using residual values.

[0031] After running DCT220, the output data is transmitted to quantization module 230. The quantization scale code is element-wise partitioned using the quantization matrix, and each resulting element is rounded. The quantization parameters derive the step size associated with the transformed coefficients and a finite number of steps. The residual block is then reconstructed through inverse quantization 240 and inverse DCT250 operations, respectively. The resulting residual block is reassembled in the deblocking effect function based on feedback from the motion compensator 270, which performs the prediction generation operation.

[0032] Motion estimation 260 uses the deblocking output data and the original video 210, and encodes it according to the relationships between frames. Motion estimation 260 encodes the current frame data based on the changes in adjacent frames. The goal of motion estimation is to find the best match of the changed regions between two adjacent frames. The input data for motion estimation are macroblocks and search regions. Motion estimation 260 performs image block motion estimation, using a search algorithm to calculate motion vectors (MV). The most basic search method uses a full search algorithm, which can handle all pixels within the search range and finds the best block match through a cost function. The output data of motion estimation is transmitted to motion compensator 270, which will be used in the image block segmentation operation. In addition, the output data of motion estimation and the output data of the quantization step are transmitted to entropy encoder 280.

[0033] The entropy encoder 280 is a lossless data compression scheme. It creates and assigns a unique prefix code to each unique symbol in the input data. Entropy encoding is performed on the quantization result of each macroblock to generate a bitstream 290.

[0034] Image patch prediction (using inter-frame prediction, intra-frame prediction, or a combination of both) can lead to errors and artifacts, especially at patch boundaries. The system disclosed in this invention can utilize deep neural networks to enhance the predicted patches and their adjacent reconstructed pixels. Figure 3 An example of a 4x4 currently predicted pixel block is shown, where the edge of the shaded pixel 320 is the adjacent reconstructed block, i.e., the shaded pixel 310. The number of reconstructed pixels can be adaptively determined based on the block size. Alternatively, the number of reconstructed pixels can be inferred at the decoder end through defined rules. For example, for an 8x8 block, the system can use an adjacent horizontal line and a vertical line; however, for a 32x32 block, one line may not be sufficient, so two lines can be used.

[0035] Filtering can also be performed at the boundaries between the current predicted pixel and its neighboring reconstructed pixels. This filtering can resolve blocking artifacts that may occur at these boundaries. Various low-pass filters can be used, or even directly applied to the filters in the deblocking module.

[0036] refer to Figure 4 The example illustrates that the current block is further subdivided into smaller parts for prediction. As shown in the figure, the current set of predicted blocks is represented by image block 420 with a shadow pattern, and the second set of predicted blocks is represented by image block 430 with a shadow pattern. These two sets of predicted blocks together constitute the current predicted block and are adjacent to the reconstructed block, which is represented by image block 410 with a shadow pattern. In the case of subdividing the predicted block into smaller blocks, the system can also filter all block boundaries.

[0037] In some embodiments, the methods and systems of the present invention can enhance predicted pixels by feeding current pixel information into a deep neural network, using the current pixel to enhance the predicted pixels, rather than using reconstructed pixels. In some embodiments, various enhancement methods can be predefined, and then the enhancement method is selected based on the closest match to the current pixel, and the selected method is transmitted to the decoder. For example, if the difference between the original block and the predicted block is very large, the system can perform low-pass filtering on the predicted block to reduce the high-frequency energy of the predicted block. Similarly, the system can also add some offset to each predicted block to reduce the difference from the original pixel, and then transmit these offsets to the decoder. The difference between pixels refers to a measure of the minimum change between the predicted block and the original block. The purpose of transmitting this information to the decoder is to reduce the computational complexity required by the decoder, and this information helps the decoder make predictions; if this information is not sent and the decoder has to derive it itself, it will increase computational complexity. The system can also infer the optimal method for enhancing predicted pixels based on the image texture of the predicted block. If the predicted texture is very complex or has a large variance, the system may use a weak low-pass filter. Conversely, if the texture is very smooth, the system may use a strong low-pass filter.

[0038] refer to Figure 5 , Figure 5 The example process for enhancing image patch prediction is shown, which is generally identified as process 500. First, at 510, conventional inter-frame and / or intra-frame prediction techniques are used to predict pixels. Figure 6 Further details of the pixel prediction subprocess are shown. First, the video is divided into multiple image blocks 610, each of which can be further subdivided into sub-block regions. Then, in some cases, intra-frame prediction can be performed at 620. Intra-frame prediction uses reconstructed information from the current frame to predict information for the current block. Figure 7 This is a more detailed illustration of the intra-frame prediction process. First, the process performs angle prediction at 710. Angle prediction uses adjacent reconstructed pixels to predict the current block. Then, at 720, the process performs intra-frame block copying, which predicts the current block based on the reconstructed blocks in the same frame pointed to by the motion vector. Finally, at 730, the color grading mode is run. The color grading mode generates an index and color map to predict the current block.

[0039] In addition to intra-frame prediction, inter-frame prediction can also be used to predict the current block, such as... Figure 6 As shown in 630. Figure 8The detailed process of inter-frame prediction is illustrated. First, image block segmentation is performed at 810. Image block segmentation divides the block into smaller parts for individual prediction. Then, the system performs unidirectional prediction at 820. Unidirectional prediction uses a prediction from a previously coded frame, pointed to by a motion vector, to predict the current block. Finally, bidirectional prediction is performed at 830. Bidirectional prediction uses two predictions from previous coded frames, pointed to by two motion vectors, to predict the current block.

[0040] In the flowchart description of this invention, intra-frame prediction and inter-frame prediction occur consecutively, but these prediction techniques can also be used individually or in any combination for pixel prediction. If used simultaneously, they can... Figure 6 At position 640, the pixel generation results are weighted and combined. This generates a final set of reconstructed pixels for prediction enhancement, such as... Figure 5 As shown in 520.

[0041] Figure 9A This is a detailed description of a first embodiment of the sub-processes of the augmentation prediction method. First, at 910, a deep neural network is applied to the prediction block and neighboring reconstructed pixels to augment the image. At 920, the system adaptively determines the number of reconstructed pixels using a set of rules. At 930, this number of reconstructed pixels is transmitted to the decoder (or inferred at the decoder).

[0042] At position 940, the system checks for edge block artifacts. If none are present, the process can terminate. However, if block artifacts exist, the system can filter the boundary between the current predicted pixel and its neighboring reconstructed pixels at position 950. Furthermore, filtering can be applied to the boundary of the current predicted block at position 960. This marks the end of the enhancement sub-process.

[0043] Figure 9B This is an explanation of a second embodiment of a sub-process for enhancing the prediction method (which can be used as an alternative to the sub-process of the first embodiment described above). In some embodiments, this second sub-process is used to replace... Figure 9A The process. Figure 9B The sub-process begins at 915 by applying a deep neural network to enhance the predicted pixel using the current pixel. At 925, the method also selects from several predefined enhancement methods based on the current pixel. At 935, the selected predefined enhancement method is passed to the decoder, and then the sub-process ends.

[0044] Regardless of the subprocess used to predict the enhancement, the process returns to the previous state after the enhancement is complete. Figure 5 At position 530, the enhanced content is subtracted from the predicted image and processed. The subtraction and processing can follow standardized video coding steps.

[0045] The above has elucidated the systems and methods for enhancing image patch prediction. Now, let's look at the devices used to perform these functions in real time. For ease of discussion, Figure 10A and Figure 10B Each system is shown as computer system 1000, which can be used to implement the embodiments of the present invention. Figure 10A This diagram illustrates one possible physical form of the computer system 1000. Of course, the computer system 1000 can have various physical forms, ranging from printed circuit boards, integrated circuits, small handheld devices to large supercomputers. The computer system 1000 may include a monitor 1002, a display 1004, a stand 1006, a blade server containing one or more storage drives 1008, a keyboard 1010, and a mouse 1012, etc. Medium 1014 is a computer-readable medium used for transmitting data to the computer system 1000. Figure 10B This is an example block diagram of computer system 1000. System bus 1020 is connected to a variety of subsystems. Processor 1022 (also called central processing unit or CPU) is adapted to storage devices (including memory 1024). Memory 1024 includes random access memory (RAM) and read-only memory (ROM). As is well known to those skilled in the art, ROM is used for unidirectional transfer of data and instructions to the CPU, while RAM is typically used for bidirectional transfer of data and instructions. Both types of memory can include any suitable form of computer-readable medium described below. Fixed medium 1026 can also be bidirectionally adapted to processor 1022 to provide additional data storage capacity and can also include any computer-readable medium described below. Fixed medium 1026 is an auxiliary storage medium (e.g., hard disk) for storing programs, data, etc., and typically operates slower than main memory. It should be noted that, where appropriate, information retained in fixed medium 1026 can be incorporated into virtual memory in memory 1024 in a standard manner. Removable medium 1014 can take the form of any computer-readable medium described below.

[0046] Processor 1022 is also compatible with various input / output devices, such as display 1004, keyboard 1010, mouse 1012, and speaker 1030. Generally, input / output devices can be any of the following: video display, trackball, mouse, keyboard, microphone, touch-sensitive display, sensor card reader, tape or paper tape reader, tablet computer, stylus, voice or handwriting recognition device, biometric reader, motion sensor, EEG reader, or other computer, etc. Processor 1022 can also interconnect with another computer or telecommunications network using network interface 1040. It is conceivable that processor 1022 uses network interface 1040 to receive information from the network, or may output information to the network during the execution of the enhanced image patch prediction method described above. Furthermore, embodiments of the method of the present invention can run independently on processor 1022, or can also operate collaboratively with a remote CPU sharing partial processing via a network such as the Internet.

[0047] Software is typically stored in non-volatile memory and / or drive units. In fact, for large programs, it may not even be possible to store the entire program in a single memory. However, it is understood that during software execution, it can be moved to a computer-readable location suitable for processing, which, for ease of explanation, is referred to as memory, if necessary. Even when the software is moved to memory for execution, the processor typically uses hardware registers and caches to store software-related values, ideally for speeding up execution. In this document, when a software program is stated to be “implemented in a computer-readable medium,” it is assumed that the software program is stored in any known or convenient location (non-volatile memory or hardware registers, etc.). A processor is considered “configured to run a program” when at least one value related to the program is stored in a processor-readable register.

[0048] During operation, computer system 1000 can be controlled by operating system software (such as a media operating system) that includes a file management system. For example, Microsoft Corporation in Redmond, Washington... An operating system series and its associated file management system is essentially operating system software with a file management system. For example, the Linux operating system and its associated file management system are also operating system software with a file management system. File management systems are typically stored in non-volatile memory and / or drive units, allowing the processor to perform various operations required by the operating system, input and output data, and store data in memory, including storing files in non-volatile memory and / or drive units.

[0049] Some parts described in detail in this document may be presented in the form of algorithms and symbolic representations of operations on data bits within computer memory. These algorithmic descriptions and representations are the means by which those skilled in the art of data processing most effectively communicate their work to others skilled in the art. As defined herein, an algorithm is a series of self-consistent operational steps designed to achieve a desired result. These operations require physical manipulation of physical quantities. Typically, these physical quantities are represented as electrical or magnetic signals that can be stored, transmitted, combined, compared, or otherwise manipulated, but this is not always necessary. For common use and ease of interpretation, these signals are often referred to as bits, values, elements, symbols, characters, items, numbers, etc.

[0050] The algorithms and representations described herein are inherently independent of any particular computer or device. The procedural methods described herein can be implemented using various general-purpose systems, or specific embodiments can be designed for dedicated devices to run, thereby achieving greater convenience. The architectures required for these systems will be detailed below. Furthermore, the techniques described herein are not referred to in any particular programming language, and therefore various embodiments can be implemented using various programming languages.

[0051] In other embodiments, the machine may operate as a standalone device or may be interconnected (e.g., networked) with other machines. In a networked deployment, the machine may operate as a server or client machine in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

[0052] The aforementioned machines can be server computers, client computers, personal computers (PCs), tablets, laptops, set-top boxes (STBs), personal digital assistants (PDAs), cellular phones, iPhones, BlackBerry phones, processor-equipped glasses, processor-equipped headphones, virtual reality devices, processors, collaborative distributed processors, telephones, network devices, network routers, switches, or bridges, or any machine capable of executing a set of instructions (whether sequentially or otherwise) specifying the operations that the machine needs to perform.

[0053] Although machine-readable media or machine-readable storage media are shown as a single medium in exemplary embodiments, "machine-readable media" and "machine-readable storage media" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store one or more sets of instructions. "Machine-readable media" and "machine-readable storage media" should also be understood to include any medium capable of storing, encoding, or carrying a set of instructions for machine execution and enabling the machine to perform any one or more methods of the currently disclosed technical solutions.

[0054] Generally, routines that run to implement embodiments of the present invention may be implemented as part of an operating system or a particular application, component, program, object, module, or sequence of instructions (referred to as a "computer program"). A computer program typically includes one or more instructions written at different times in various memories and storage devices within a computer (or distributed across computers), and when one or more processing units or processors within (or across computers) read and execute these instructions, the computer can perform operations to implement the functions disclosed in the present invention.

[0055] Furthermore, while the embodiments described herein are set in the context of fully functional computers and computer systems, those skilled in the art will understand that various embodiments can be distributed in various forms of program products, and the disclosure of this invention applies equally to any particular type of machine or computer-readable medium in which the distribution is actually implemented.

[0056] While the invention has been described through several embodiments, variations, modifications, substitutions, and equivalents that fall within the scope of the invention are still possible. Although subsection headings have been used herein to aid in the description of the invention, these headings are illustrative only and are not intended to limit the scope of the invention. It should also be noted that many alternatives are available for implementing the methods and apparatus of the invention. Therefore, the appended claims should be considered to encompass all such variations, modifications, substitutions, and equivalents within the spirit and scope of the invention.

Claims

1. A method for enhancing image patch prediction, characterized in that, include: Predict pixels in video frames to generate prediction blocks; Deep neural networks are applied to enhance the prediction of prediction blocks and the prediction of adjacent reconstructed pixels; Determine the number of reconstructed pixels; The number of reconstructed pixels is transmitted to the decoder; Subtract the enhanced prediction from the video frames; as well as Further processing enhances the forecast.

2. The method according to claim 1, characterized in that, Predicting pixels in a video frame includes inter-frame prediction and / or intra-frame prediction.

3. The method according to claim 2, characterized in that, It also includes generating a weighted combination of inter-frame prediction and intra-frame prediction.

4. The method according to claim 1, characterized in that, It also includes determining whether block effects exist.

5. The method according to claim 4, characterized in that, Also includes: If block artifacts exist, the boundary between the predicted pixel and the adjacent reconstructed pixel is filtered.

6. The method according to claim 5, characterized in that, The boundary between the predicted pixel and the adjacent reconstructed pixel is filtered using a low-pass filter.

7. The method according to claim 5, characterized in that, Also includes: If block artifacts exist, the boundaries of the predicted blocks are filtered.

8. The method according to claim 2, characterized in that, The intra-frame prediction includes angle prediction, intra-frame block copying, and color mode operation.

9. The method according to claim 2, characterized in that, The inter-frame prediction includes image block segmentation, one-way prediction, and two-way prediction.

10. The method according to claim 1, characterized in that, Also includes: An enhancement mode is selected from a predefined set of enhancements based on the current pixel, and the selected enhancement mode is transmitted to the decoder.

11. A system for enhancing image patch prediction, characterized in that, include: The prediction module is used to predict pixels in video frames to generate prediction blocks; A server is configured to apply a deep neural network to enhance the prediction of prediction blocks and adjacent reconstructed pixels, determine the number of reconstructed pixels, and transmit the number of reconstructed pixels to a decoder; and The decoder is used to subtract the enhanced prediction from the video frames and further process the enhanced prediction.

12. The system according to claim 11, characterized in that, Predicting pixels in a video frame includes inter-frame prediction and / or intra-frame prediction.

13. The system according to claim 12, characterized in that, It also includes generating a weighted combination of inter-frame prediction and intra-frame prediction.

14. The system according to claim 11, characterized in that, The server is further configured to determine whether block effects exist.

15. The system according to claim 14, characterized in that, The server is also configured to filter the boundaries between the predicted pixel and adjacent reconstructed pixels if block artifacts are present.

16. The system according to claim 15, characterized in that, The boundary between the predicted pixel and the adjacent reconstructed pixel is filtered using a low-pass filter.

17. The system according to claim 15, characterized in that, The server is also configured to filter the boundaries of predicted blocks when block artifacts are present.

18. The system according to claim 12, characterized in that, The intra-frame prediction includes angle prediction, intra-frame block copying, and color mode operation.

19. The system according to claim 12, characterized in that, The inter-frame prediction includes image block segmentation, one-way prediction, and two-way prediction.

20. The system according to claim 11, characterized in that, The server is further configured to select an enhancement mode from a predefined set of enhancements based on the current pixel and transmit the selected enhancement mode to the decoder.

Images