System and method for encoding and decoding video-codec using artificial intelligence-based in-loop filtering model

Abstract

A video encoding method (900) designed to enhance in-loop filter processing in video codecs using an AI-based filtering module is disclosed. The method (900) includes receiving original video frames (102) and generating reconstructed frames through transformation and quantization. A series of in-loop filters are applied using an in-loop filtering module (110) to the reconstructed video frame, producing preliminary enhanced video frames. The content of the original and reconstructed frames is analyzed through content analyzer block (112) to determine content types and frequency bands, which are used to create a weight matrix representing target frequency bands. The weight matrix, alongside the preliminary enhanced video frames, is processed by a pre-trained AI-based filtering module (116) to generate enhanced reconstructed video frame (118). Finally, a bitstream is generated, encoding the weight matrix information to guide image enhancement along target frequency bands during video playback.

Description

SYSTEM AND METHOD FOR ENCODING AND DECODING VIDEO-CODEC USING ARTIFICIAL INTELLIGENCE-BASED IN-LOOP FILTERING MODEL

[0001] The present invention generally relates to video compression techniques, and more particularly relates to a system and a method for encoding and decoding video codec using Artificial Intelligence (AI) based in-loop filtering model.

[0002] Image and video compression standards, such as High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC), are widely utilized to reduce the data size of multimedia content while maintaining an acceptable level of quality. These standards employ various techniques, including transform coding and quantization, to achieve efficient compression. Transform coding converts spatial domain data into frequency domain data, which is then quantized to reduce the amount of information that needs to be encoded and transmitted. While these processes are effective in reducing file sizes, they introduce a range of compression artifacts due to the quantization of transform coefficients.

[0003] Compression artifacts, such as loss of detail, blockiness, and ringing, degrade the visual quality of the compressed content. To mitigate such issues and improve output quality, in-loop filtering technologies have been developed. In-loop filters are applied within the encoding and decoding loop to enhance the reconstructed image or video quality by reducing artifacts caused by the compression process.

[0004] HEVC and VVC standards propose several types of in-loop filters to address different kinds of compression artifacts:

[0005] Sample Adaptive Offset (SAO) filter primarily targets ringing artifacts, which often occur around edges in the image. The SAO filter is configured to smoothen the gradients around edge boundaries, thereby reducing the perceptibility of these artifacts.

[0006] Deblocking filter is designed to mitigate blockiness artifacts, which manifest as visible discontinuities along the block boundaries within a codec frame. The deblocking filter smooths these boundaries, reducing the blocky appearance of the image.

[0007] Adaptive Loop Filter (ALF) employs Wiener filtering techniques based on local image characteristics to enhance overall image quality. The filter coefficients used by ALF are determined during encoding and transmitted to the decoder along with additional signal information to ensure consistent image enhancement at the decoder side.

[0008] In addition to traditional filtering methods, AI-based techniques have been explored for in-loop filtering. The AI-driven approaches aim to learn and apply more sophisticated filtering strategies that can better adapt to varying content types. However, existing AI-based methods in video codecs have limitations, particularly in their ability to generalize across different content categories. These methods often apply uniform enhancements regardless of the specific characteristics of the content, leading to suboptimal results in diverse scenarios.

[0009] Moreover, some existing solutions involve applying frequency weighting mechanisms to specific frequency bands, primarily to enhance high-frequency components in an effort to retain fine details. While these techniques have shown promise in particular applications, such as super-resolution, they are not directly applicable to general image or video enhancement tasks. Furthermore, although there is extensive literature exploring the relationship between different frequency bands and visual perception, these studies have not successfully integrated frequency-weighting concepts into a neural network training framework for content-specific image or video enhancement.

[0010] Accordingly, there is a need to address one or more of the above-mentioned disadvantages or other shortcomings or at least mitigate them and provide a useful alternative.

[0011] This summary is provided to introduce a selection of concepts, in a simplified format, that are further described in the detailed description of the invention. This summary is neither intended to identify key or essential inventive concepts of the invention nor is it intended for determining the scope of the invention.

[0012] According to one embodiment of the present disclosure, a video encoding method for enhancing in-loop filter processing of a video codec in an artificial intelligence (AI) based in-loop filtering module is disclosed. The method includes receiving a original video frame to be encoded. The method further includes generating a reconstructed video frame by at least one of a transforming and a quantizing of a residue of the original video frame. The method includes applying a plurality of in-loop filters to the reconstructed video frame to obtain a preliminary enhanced video frame. The method includes determining at least one of a content type of a plurality of content types, and one or more frequency bands of the original video frame and the reconstructed video frame by analyzing the content of at least one of the original video frame and the corresponding reconstructed video frame. The method includes generating a weight matrix representing target frequency bands of the original video frame based on the content type and the one or more frequency bands. The method includes generating an enhanced reconstructed video frame based on at least one of the preliminary enhanced video frame and the weight matrix. The method includes generating a bitstream by encoding information about the weight matrix representing target frequency bands.

[0013] According to one embodiment of the present disclosure, a video decoding method for enhancing in-loop filter processing of a video codec in an artificial intelligence (AI) based in-loop filtering module is disclosed. The method includes receiving an bitstream of an encoded video. The method includes generating a reconstructed video frame by at least one of an inverse quantization and an inverse transformation of the residue information extracted from the bitstream. The method further includes applying a plurality of in-loop filters to the reconstructed video frame to obtain a preliminary enhanced video frame. The method further includes obtaining a weight matrix representing the target frequency bands to be enhanced for the reconstructed video frame, from at least one of the bitstream. The method further includes generating an enhanced reconstructed video frame based on at least one of the preliminary enhanced video frame and the weight matrix.

[0014] According to one embodiment of the present disclosure, a video encoding system for enhancing in-loop filter processing of a video codec in an artificial intelligence (AI) based in-loop filtering module is disclosed. The system includes a memory configured to store executable instructions, at least one processor coupled to the memory and configured to execute the instructions to receive a original video frame to be encoded. The at least one processor is configured to generate a reconstructed video frame by at least one of a transforming and a quantizing of a residue of the original video frame. The at least one processor is configured to apply a plurality of in-loop filters to the reconstructed video frame to obtain a preliminary enhanced video frame. The at least one processor is configured to determine at least one of a content type of a plurality of content types, and one or more frequency bands of the original video frame and the reconstructed video frame by analyzing the content of at least one of the original video frame and the corresponding reconstructed video frame. The at least one processor is configured to generate a weight matrix representing target frequency bands of the original video frame based on the content type and the one or more frequency bands. The at least one processor is configured to generate an enhanced reconstructed video frame based on at least one of the preliminary enhanced video frame, and the weight matrix. The AI-based In-loop filtering module is pre-trained to guide the image enhancement along the target frequency bands. The at least one processor is configured to generate a bitstream by encoding information about the weight matrix representing target frequency bands.

[0015] According to one embodiment of the present disclosure, a video decoding system for enhancing in-loop filter processing of a video codec in an artificial intelligence (AI) based in-loop filtering module is disclosed. The system includes a memory configured to store executable instructions, at least one processor coupled to the memory and configured to execute the instructions to receive an bitstream of an encoded video. The at least one processor is configured to generate a reconstructed video frame by at least one of inverse quantizing and inverse transforming the residue information extracted from bitstream. The at least one processor is configured to apply a plurality of in-loop filters to the reconstructed video frame to obtain a preliminary enhanced video frame. The at least one processor configured to obtain a weight matrix representing the target frequency bands to be enhanced for the reconstructed video frames, from at least one of the bitstream, and an encoded template. The at least one processor is configured to generate an enhanced reconstructed video frame based on at least one of the preliminary enhanced video frame, and the weight matrix. The AI-based In-loop filtering module is pre-trained to guide the image enhancement along the target frequency bands.

[0016] To further clarify the advantages and features of the present invention, a more particular description of the invention will be rendered by reference to specific embodiments thereof, which are illustrated in the appended drawings. It is appreciated that these drawings depict only typical embodiments of the invention and are therefore not to be considered limiting of its scope. The invention will be described and explained with additional specificity and detail in the accompanying drawings.

[0017] These and other features, aspects, and advantages of the present invention will become better understood when the following detailed description is read with reference to the accompanying drawings in which like characters represent like parts throughout the drawings, wherein:

[0018] Figure 1 illustrates a schematic block diagram depicting video encoding for enhancing in-loop filter processing of a video codec in an artificial intelligence (AI) based in-loop filtering module, according to an embodiment of the present invention;

[0019] Figure 2 illustrates a schematic block diagram depicting video decoding apparatus for enhancing in-loop filter processing of a video codec in an artificial intelligence (AI) based in-loop filtering module, according to an embodiment of the present invention.

[0020] Figure 3 illustrates a schematic block diagram of a content analyzer block generating a weight matrix, according to an embodiment of the present invention;

[0021] Figure 4 illustrates a block diagram depicting a content-based frequency component segregation module to obtain frequency bands to be enhanced based on the incoming content, in accordance with various embodiments of the present disclosure;

[0022] Figure 5 illustrates an example depicting a content-based frequency component segregation to obtain frequency bands to be enhanced based on specific content;

[0023] Figure 6 illustrates an example depicting a content-based frequency component segregation to obtain frequency bands to be enhanced based on an specific content;

[0024] Figure 7 illustrates an example depicting a content-based frequency component segregation to obtain frequency bands to be enhanced based on an specific content, in accordance with an embodiment of the present disclosure;

[0025] Figure 8 illustrates a method depicting video encoding for enhancing in-loop filter processing of a video codec in an artificial intelligence (AI) based in-loop filtering module, according to an embodiment of the present invention;

[0026] Figure 9 illustrates a method depicting video decoding for enhancing in-loop filter processing of a video codec in an artificial intelligence (AI) based in-loop filtering module, according to an embodiment of the present invention; and

[0027] Figure 10 illustrates an embodiment of a system, according to an embodiment of the present disclosure.

[0028] Further, skilled artisans will appreciate that elements in the drawings are illustrated for simplicity and may not have necessarily been drawn to scale. For example, the flow charts illustrate the method in terms of the most prominent steps involved to help to improve understanding of aspects of the present invention. Furthermore, in terms of the construction of the device, one or more components of the device may have been represented in the drawings by conventional symbols, and the drawings may show only those specific details that are pertinent to understanding the embodiments of the present invention so as not to obscure the drawings with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.

[0029] For the purpose of promoting an understanding of the principles of the invention, reference will now be made to the various embodiments and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the invention is thereby intended, such alterations and further modifications in the illustrated system, and such further applications of the principles of the invention as illustrated therein being contemplated as would normally occur to one skilled in the art to which the invention relates.

[0030] It will be understood by those skilled in the art that the foregoing general description and the following detailed description are explanatory of the invention and are not intended to be restrictive thereof.

[0031] Reference throughout this specification to "an aspect," "another aspect" or similar language means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, appearances of the phrase "in an embodiment," "in another embodiment" and similar language throughout this specification may, but do not necessarily, all refer to the same embodiment.

[0032] The terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process or method that comprises a list of steps does not include only those steps but may include other steps not expressly listed or inherent to such process or method. Similarly, one or more devices or sub-systems or elements or structures or components proceeded by "comprises... a" does not, without more constraints, preclude the existence of other devices or other sub-systems or other elements or other structures or other components or additional devices or additional sub-systems or additional elements or additional structures or additional components.

[0033] Figure 1 illustrates a schematic block diagram depicting video encoding apparatus 100 for enhancing in-loop filter processing of a video codec in an artificial intelligence (AI) based in-loop filtering module, according to an embodiment of the present invention.

[0034] The video encoding apparatus 100 according to an embodiment includes a predictor 103, transformer 104, a quantizer 106, an inverse quantization and transformation module 108, an in-loop filtering module 110, a content analyzer block 112, weight matrix representing target frequency bands 114, and an AI-based in-loop filtering module 116.

[0035] According to an embodiment, a original video frame 102 are processed in terms of plurality of blocks of pixels from plurality of frames. The predictor 103 predicts blocks of original video frames based on previously encoded data. The predicted blocks are subtracted from corresponding original blocks to generate residual blocks that are further sent to the transformer 104 for further processing.

[0036] According to an embodiment, before the original video frame 102 are given as the input to the transformer 104, the predictor 103 may be configured to predict the content of the video frame based on previously encoded data. A residual may be generated by subtracting the predicted block from the actual block in the current frame. This residual is then transmitted to the transformer 104 for further processing.

[0037] The residual information i.e., the difference between the actual frame data and the predicted data may be passed to the transformer 104 for further processing (e.g., through Discrete Cosine Transform or similar techniques). The prediction information, such as motion vectors in the case of inter-frame prediction, is also forwarded to the bitstream generator 120, which combines it with the residual and other coding information to create the compressed bitstream 122.

[0038] According to an embodiment, the residual signal is given as an input to the transformer 104 and the quantizer 106. The output is generated as a quantized transform coefficient for each transform unit through the　transformer 104　and the　quantizer 106. The reconstructed residual is computed by using the　inverse quantization and inverse transformation process from module 108. A reconstructed video frame is generated by adding the reconstructed residual signal to the predicted signal

[0039] Further, a series of in-loop filters is applied to the reconstructed video frame through the in-loop filtering module 110 to produce a preliminary enhanced video frame 118. The in-loop filtering module 110 may include the series of in-loop filters.

[0040] In an example Sample Adaptive Offset filter (SAO) 110a primarily targets ringing artifacts, which often occur around edges in the image. The SAO filter 110a is configured to smoothen the gradients around edge boundaries, thereby reducing the perceptibility of these artifacts.

[0041] In an example, the in-loop filter may include a Deblocking filter 110b is designed to mitigate blockiness artifacts, which manifest as visible discontinuities along the block boundaries within a codec frame. The deblocking filter 110b smooths these boundaries, reducing the blocky appearance of the image.

[0042] In an example, the in-loop filter may include Adaptive Loop Filter (ALF) 110c employs Wiener filtering techniques based on local image characteristics to enhance overall image quality. The filter coefficients used by ALF 110c are determined during encoding and transmitted to the decoder along with additional signal information to ensure consistent image enhancement at the decoder side.

[0043] Upon producing the preliminary enhanced video frame 118, the content analyzer block (CAB) 112 is configured to analyze a content of either the original video frame 102, the reconstructed video frames, or both which is used to determine a type of the content present or identifies specific frequency bands within the video frame that are crucial for enhancement.

[0044] Furthermore, based on the analyzing the content by the CAB 112 The weigh matrix generating module 114 may be configured to generate the weight matrix representing the target frequency bands within the original video frame 102 that are essential for enhancing the video quality. The weight matrix and the preliminary enhanced video frames 118 are then fed into the AI-based in-loop filtering module 116. The AI-based in-loop filtering module 116 is pretrained and may be configured to generate an enhanced video frame 118 based on the target frequency bands. The target frequency bands may be represented as crucial frequency bands. The AI-based in-loop filtering module 116 is pre-trained using the determined weight marix as at least one of an input, pixel attention, and a loss guidance during an in-loop filtering by an in-loop filtering sub-module.

[0045] Finally, the video encoding apparatus 100 may be configured to generate a bitstream 122 using the bitstream generator 120. The bitstream 122 may be configured to encode information about the weight matrix, ensuring that the target frequency bands are preserved during encoding and transmission, leading to a more efficient and higher-quality video output.

[0046] Figure 2 illustrates a schematic block diagram depicting video decoding apparatus 200 for enhancing in-loop filter processing of a video codec in an artificial intelligence (AI) based in-loop filtering module 208, according to an embodiment of the present invention.

[0047] In an embodiment, the decoding apparatus 200 may include inverse quantization and transformation module 202, a predictor 203, an in-loop filtering module 204, a weight matrix generating module 206, and an AI-based in-loop filtering module 208.

[0048] The inverse quantization and transformation module 202 receives the bitstream of encoded video frame 122. The bitstream of encoded video frame 122 may include compressed and encoded video data. Further, the bitstream 122 includes all the necessary information, such as the video frame, metadata, and additional elements like the weight matrix representing target frequency bands 114 for reconstructing the video with enhanced quality.

[0049] The first step in decoding is to reconstruct the original video frame 102 from the compressed bitstream 122. This is achieved through the inverse quantization and inverse transformation using the inverse quantization and inverse transformation module 202. The inverse quantization and inverse transformation module 202 may be configured to reconstruct the Residual signal corresponding to the current frame that is being decoded.

[0050] In an embodiment, before decoding the reconstructed video frame from the compressed bitstream 122, the predictor 103 may be configured to predict the content of the reconstructed video frame based on previously decoded data. In one embodiment, the predictor 103 may be configured to receive a current frame to be decoded (here the bitstream of encoded video frames 122) and previously decoded video frames. The predictor 203 generates a prediction signal that is added to the reconstructed residual signal to generate reconstructed frames corresponding to the encoded source video frames.

[0051] The in-loop filtering module 204 may be configured to filter the reconstructed video frame to remove artifacts (such as blocking or blurring) that may have been introduced during compression. The in-loop filtering module 204 may include sample adaptive offset filter 204a, deblocking filter 204b, and an adaptive loop filter 204c. The in-loop filtering module 204 may be configured to output a preliminary enhanced video frame 210.

[0052] Further, the weight matrix generating module 206 may be configured to obtain the weight matrix from the bitstream 122 or an encoded template. The weight matrix contains information about the target frequency bands. The target frequency bands may correspond to specific details in the video, like edges, textures, or areas with high motion, which are vital for maintaining video clarity and quality.

[0053] Upon obtaining the weight matrix, the preliminary enhanced video frames 210 and the weight matrix are then fed into an AI-based in-loop filtering module 208. The AI-based in-loop filtering module 208 is pre-trained and configured to enhance the video frames by emphasizing on the target frequency bands identified in the weight matrix. The output of the AI-based in-loop filtering module 208 is an enhanced reconstructed video frame 210. The enhanced reconstructed video frame 210 represent the final, high-quality video that is ready for playback.

[0054] Figure 3 illustrates a schematic block diagram of a content analyzer block 112 generating a weight matrix 312, according to an embodiment of the present invention.

[0055] In an embodiment, the content-specific enhancement in image processing is done using the Content Analyser Block (CAB) 112. The CAB 112 includes two modules, content-based frequency component segregation module 306, and a weight determination module 310. The CAB 112 may be configured to systematically analyze both the original image content 302 and a reconstructed image 304. The CAB 112 may be configured to segregate and identify various frequency bands within the image that require enhancement based on characteristics of the content using the frequency band segregation module 308.

[0056] In an embodiment, the CAB 112 may be configured to generate a segregated one or more frequency bands based on the original video frame 102 such that the segregated one or more frequency bands are utilized for generating the weight matrix 312. The segregated one or more frequency bands indicate an enhancement type required for the determined content type of the plurality of content types.

[0057] Further, the CAB 112 may be further configured to assign at least one of a higher or a lower weightage to a one or more frequency components of the segregated one or more frequency bands. The one or more frequency components corresponds to the one or more frequency bands that is capable of enhancing one or more content characteristics. The one or more content characteristics includes at least one of textures, film-grain, compression artefacts, and sharpness in edges.

[0058] Following the analysis, a weight determination module 310 is employed to compute a weight matrix 312. The weight matrix 312 may be configured to assign priority weights to each frequency component, effectively quantifying the relative importance of different frequency bands for enhancement. The computed weight matrix 312 is then utilized as input for the AI-based in-loop filtering module 116. The AI-based in-loop filtering module 116, which operates within the enhancement pipeline may be configured to use the weight matrix 312 as guidance whether as input, attention, or loss feedback―to perform precise and content-specific enhancements on the reconstructed image 304. The combined functionality of the CAB 112 and the weight determination module 310 ensures that the enhancement process is tailored to the specific needs of the content, resulting in improved visual quality where it is most needed.

[0059] In an embodiment, the weight determination module 310 may be configured to generate a plurality of templates in a form of a look-up table. Each entry in the look-up table relates to content type with its corresponding weight matrix 312, based on a assigned weightages to one or more frequency components corresponding to the content type. The weight determination module 310 may be configured to encode the plurality of generated templates in the form of the look-up table as a part of encoded bitstream parameter set. The weight determination module 310 may be configured to select the plurality of templates for the plurality of content types for generation of the weight matrix 312.

[0060] Figure 4 illustrates a block diagram depicting a frequency band segregation module 308 to obtain frequency bands to be enhanced based on the incoming content, in accordance with various embodiments of the present disclosure.

[0061] The frequency band segregation module 308 may be configured to segregate original content 402 of different frequency bands corresponding to different characteristics 404. The different frequency bands corresponding to different characteristics 404 includes a low frequency band 406, texture-based frequency band 408, and a high frequency band 410. The low-frequency band 406 indicates the average image characteristics that need to be preserved by the video codec. The higher frequency bands 408 indicate textures, find edges and film grain noise which make the image look subjectively pleasing to end-users. The frequency components are hard to preserve, especially in cases of high quantization video compression applications.

[0062] The frequency band segregation module 308 may be configured to generate segregated frequency components based on the given content i.e., the original content 402. The weight determination module 310 may be configured to utilise the segregated frequency components to generate weights for each frequency component that signifies the level of enhancement for each frequency band.

[0063] Figure 5 illustrates an example depicting a content-based frequency component segregation to obtain frequency bands to be enhanced based on a specific content, in accordance with an embodiment of the present disclosure.

[0064] In video enhancement scenarios, preserving film grain content 502 leads to subjectively more pleasing results for end users. However, during the compression process, film grain content 502 is often lost due to quantization. The random nature of film grain makes it particularly challenging to preserve during compression. The below technique solves the aforementioned problem.

[0065] In an embodiment, an input image i.e., a film grain 502 is fed to the content-based frequency component segregation module such as film grain content-based frequency component segregation module 504 to convert the input image to its frequency domain to segregate the frequency bands in the input that require enhancement to determine frequency domain spectrum 506.

[0066] The resultant frequency domain spectrum 506 is then passed on to the weight determination module 508 to determine frequency weight matrix. The weight determination module 508 determines a rectangular frequency-band comprising of the high frequency components are weighted more significantly, as they correspond to the frequency bands that enhance film grain noise. The generated weight matrix 510 is the output of the CAB 112 which is passed as input to the AI-based in-loop filtering module 116 of the video codec.

[0067] Figure 6 illustrates an example depicting a content-based frequency component segregation to obtain frequency bands to be enhanced based on a specific content, in accordance with an embodiment of the present disclosure.

[0068] In video enhancement scenarios, preserving textures and sharp edges leads to content that is subjectively more pleasing for end users. However, during the compression process, textures are often lost, and edges become smoothed out due to quantization. Recovering these fine details and textures is a significant challenge in the field of video compression. The below technique solves the aforementioned problem:

[0069] In an embodiment, an input image i.e., a texture / edge content 602 is fed to the content-based frequency component segregation module such as film grain content-based frequency component segregation module 604 to convert the input image to its frequency domain to segregate the frequency bands in the input that require enhancement to determine frequency domain spectrum 606.

[0070] The resultant frequency domain spectrum 606 is then passed on to the weight determination module 608 to determine frequency weight matrix 610. The weight determination module 608 may be configured to determine a diagonal frequency-band comprising of the high frequency components are weighted more significantly, as they correspond to the frequency bands that can enhance textures and sharpness in edges. The generated weight matrix 610 is the output of the CAB 112 which is passed as input to the AI-based in-loop filtering module 116 of the video codec.

[0071] Figure 7 illustrates an example depicting a content-based frequency component segregation to obtain frequency bands to be enhanced based on a specific content, in accordance with an embodiment of the present disclosure.

[0072] In video enhancement scenarios, Screen video content 702 involves a combination of graphics, animated content along with naturally occurring content. The below solution effectively enhances the quality of the screen content using the CAB 112 by weighing important frequency components:

[0073] In an embodiment, an input image i.e., a screen video content 702 is fed to the content-based frequency component segregation module such as screen content-based frequency component segregation module 704 to convert the input image to its frequency domain to segregate the frequency bands in the input that require enhancement to determine frequency domain spectrum 706.

[0074] The resultant frequency domain spectrum 706 is then passed on to the weight determination module 708 to determine frequency weight matrix 710. The weight determination module 708 may be configured to determine a cris-cross frequency-band comprising of the high frequency components are weighted more significantly, as they correspond to the frequency bands that can enhance graphics, animated content along with naturally occurring content. The generated weight matrix 710 is the output of the CAB 112 which is passed as input to the AI-based in-loop filtering module 116 of the video codec.

[0075] Figure 8 illustrates a method 800 depicting video encoding for enhancing in-loop filter processing of a video codec in an artificial intelligence (AI) based in-loop filtering module, according to an embodiment of the present invention.

[0076] In an embodiment, at step 802, the method 800 may include receiving a original video frame 102 to be encoded.

[0077] At step 804, the method 800 may include generating a reconstructed video frame by at least one of a transforming and a quantizing of a residue of the original video frame 102.

[0078] At step 806, the method 800 may include applying a plurality of in-loop filters to the reconstructed video frame to obtain a preliminary enhanced video frame 210.

[0079] At step 808, the method 800 may include determining at least one of a content type of a plurality of content types, and one or more frequency bands of the original video frame 102 and the reconstructed video frame by analyzing the content of at least one of the original video frame 102 and the corresponding reconstructed video frame.

[0080] At step 810, the method 800 may include generating a weight matrix representing target frequency bands 114 of the original video frame 102 based on the content type and the one or more frequency bands.

[0081] At step 812, the method 800 may include generating an enhanced reconstructed video frame 118 based on at least one of the preliminary enhanced video frame 118 and the weight matrix.

[0082] At step 814, the method 800 may include generating a weight matrix representing target frequency bands 114 of the original video frame 102 based on the content type and the one or more frequency bands.

[0083] Figure 9 illustrates a method 900 depicting video decoding for enhancing in-loop filter processing of a video codec in an artificial intelligence (AI) based in-loop filtering module, according to an embodiment of the present invention.

[0084] At step 902, the method 900 may include receiving an bitstream of an encoded video 122.

[0085] At step 904, the method 900 may include generating a reconstructed video frame by at least one of an inverse quantization and an inverse transformation the bitstream 122.

[0086] At step 906, the method 900 may include applying a plurality of in-loop filters to the reconstructed video frame to obtain a preliminary enhanced video frame.

[0087] At step 908, the method 900 may include obtaining a weight matrix representing the target frequency bands to be enhanced for the reconstructed video frame, from at least one of the bitstream 122 and an encoded template.

[0088] At step 910, the method 900 may include generating an enhanced reconstructed video frame 118 based on at least one of the preliminary enhanced video frame 118 and the weight matrix. For example the method 900 may include generating an enhanced reconstructed video frame 118 by inputting at least one of the preliminary enhanced video frame 118 and the weight matrix to an AI-based In-loop filtering module. The AI-based In-loop filtering module 116 may be pre-trained to guide the image enhancement along the target frequency bands.

[0089] Figure 10 illustrates an embodiment of a system 1000, according to an embodiment of the present disclosure.

[0090] As shown in Figure 10, the system 1000 includes processor 1002, a memory 1004, a storage component 1006, an input component 1008, an output component 1010, a communication interface 1012, a bus 1014, an encoder 1016, and a decoder 1018.

[0091] The processor 1002, as used herein, means any type of computational circuit that may comprise hardware elements and software elements. The processor 1002 may be embodied as a multi-core processor, a single core processor, or a combination of one or more multi-core processors and / or one or more single core processors, a distributed processing system, or the like. The processor 1002 may be a Central Processing Unit (CPU) a graphics processing unit (GPU), an accelerated processing unit (APU), an application-specific integrated circuit (ASIC), or another type of processing component.

[0092] The memory 1004 includes a non-transitory computer readable medium. The memory 1004 includes a random-access memory (RAM), a read only memory (ROM), and / or another type of dynamic or static storage device (e.g., a flash memory, a magnetic memory, and / or an optical memory) that stores information and / or instructions for use by processor 1002. The memory 1004 comprises machine-readable instructions which are executable by the processor 1002. These machine-readable instructions when executed by the processor 1002 cause the processor 1002 to perform one or more method steps of an embodiment described above.

[0093] The storage component 1006 stores information and / or software related to the operation and use of the device 1400. For example, the storage component 1006 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optic disk, and / or a solid-state disk), a compact disc (CD), a digital versatile disc (DVD), a floppy disk, a cartridge, a magnetic tape, and / or another type of non-transitory computer-readable medium, along with a corresponding drive.

[0094] The input component 1008 may be configured to receive information, such as user input. For example, the input component 1008 may include, but not be limited to, a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, and / or a microphone. The output component 1010 is configured to convey information from the system 1440 to the user or other systems, utilizing a variety of devices and technologies tailored to specific application needs. The output component 1010 may include visual output devices such as display screens (LCD, LED, OLED), projectors, and heads-up displays (HUDs) for presenting graphical or textual information. Additionally, auditory output through speakers and headphones provides audio feedback and alerts, while haptic output devices, like vibration motors in smartphones or game controllers, offer tactile feedback. Functionally, the output component 1010 serves multiple roles, including displaying graphical user interface (GUI) elements for user interaction, delivering notifications and alerts through sound, visual indicators, or vibrations, and rendering complex data visualizations like charts and graphs for easier comprehension.

[0095] In an embodiment, the output component 1010 may be configured to receive processed data from the processor 1002, which determines the information to be communicated, and the output component 1010 may access memory 1004 and storage component 1006 to retrieve and display stored information such as documents, media files, or application states. Furthermore, the output component 1010 may be configured to meet the specific requirements of different applications, such as high-resolution visual output and immersive audio for gaming systems or clear and precise data visualization and alert mechanisms for industrial control systems. Through these varied output methods, the output component 1010 ensures effective communication of information, enhancing both system functionality and user experience.

[0096] The communication interface 1012 may be configured to transmit data between the video codec system and external devices or networks. For example, the communication interface 1012 may be configured to handle the reception of the original video frame 102 from a camera module or the transmission of encoded bitstreams to a streaming server.

[0097] The bus 1014 is an internal data transfer pathway connecting various components, such as the processor 1002, the memory 1004, and the storage 1006. The bus 1014 may be configured to facilitate high-speed communication within the system 1000, enabling the transfer of video frames, metadata (e.g., weight matrices), and control signals needed for encoding or decoding operations. For instance, during processing, the bus 1014 may transmit the reconstructed video frame from the memory 1004 to the AI-based in-loop filtering module 116.

[0098] The encoder 1016 may be configured to process the original video frame 102 to generate a compressed bitstream by applying transformations, quantization, and encoding crucial information, such as the weight matrix for content-driven enhancement. For example, the encoder 1016 may encode a video with compression settings optimized for preserving textures while minimizing artifacts, based on AI-guided adjustments.

[0099] The decoder 1018 may be configured to perform the reverse operation by reconstructing the video frame from the encoded bitstream 122. The decoder 1018 may use techniques like inverse quantization and inverse transformation and applies the AI-based in-loop filtering module 116 to enhance the video quality further. In an example, during playback, the decoder 1018 may reconstruct a video with preserved details like film grain while reducing compression artifacts, based on the weight matrix embedded in the bitstream.

[0100] The present invention provides various advantages:

[0101] a. The present invention provides the method dynamically adjusts filtering based on the specific characteristics of the video content.

[0102] b. The present invention provides enhanced video frame which ensures that target frequency bands are prioritized, leading to more accurate and visually appealing reconstructed video frames.

[0103] c. The integration of a pre-trained AI module allows for real-time optimization of the filtering process, improving the overall efficiency and effectiveness of video compression.

[0104] d. The present invention provides ability to encode and transmit weight matrix information ensuring consistent video quality during playback, even under varying network conditions, making it particularly beneficial for streaming applications where bandwidth constraints are a concern.

[0105] While specific language has been used to describe the disclosure, any limitations arising on account of the same are not intended. As would be apparent to a person in the art, various working modifications may be made to the method in order to implement the inventive concept as taught herein.

[0106] The drawings and the forgoing description give examples of embodiments. Those skilled in the art will appreciate that one or more of the described elements may well be combined into a single functional element. Alternatively, certain elements may be split into multiple functional elements. Elements from one embodiment may be added to another embodiment. For example, orders of processes described herein may be changed and are not limited to the manner described herein.

Claims

1.A video encoding method (800) for enhancing in-loop filter processing of a video codec in an artificial intelligence (AI) based in-loop filtering module (116), the method comprising:receiving a original video frame (102) to be encoded;generating a reconstructed video frame by at least one of a transforming and a quantizing of a residue of the original video frame (102);applying a plurality of in-loop filters to the reconstructed video frame to obtain a preliminary enhanced video frame (118);determining at least one of a content type of a plurality of content types and one or more frequency bands of the original video frame (102) and the reconstructed video frame by analyzing the content of at least one of the original video frame (102) and the corresponding reconstructed video frame;generating a weight matrix representing target frequency bands (114) of the original video frame (102) based on the content type and the one or more frequency bands;generating an enhanced reconstructed video frame (118) based on at least one of the preliminary enhanced video frame (118) and the weight matrix; andand generating a bitstream by encoding information about the weight matrix representing target frequency bands.2.The video encoding method (800) as claimed in claim 1 further comprising:generating a segregated one or more frequency bands based on the original video frame (102) such that the segregated one or more frequency bands are utilized for generating the weight matrix, wherein the segregated one or more frequency bands indicate an enhancement type required for the determined content type of the plurality of content types.3.The video encoding method (800) as claimed in claim 2 further comprising:assigning at least one of a higher or a lower weightage to a one or more frequency components of the segregated one or more frequency bands, wherein the one or more frequency components corresponds to the one or more frequency bands that is capable of enhancing one or more content characteristics, wherein the one or more content characteristics includes at least one of textures, film-grain, compression artefacts, and sharpness in edges.4.The video encoding method (800) as claimed in claim 1, further comprising:generating a plurality of templates in a form of a look-up table, wherein each entry in the look-up table relates to content type with its corresponding weight matrix based on an assigned weightages to one or more frequency components corresponding to the content type;encoding the plurality of generated templates in the form of the look-up table as a part of encoded bitstream parameter set; andselecting the plurality of templates for the plurality of content types for generation of the weight matrix.5.The video encoding method (800) of claim 1, wherein the plurality of in-loop filters comprise of at least one of sample adaptive offset (110a), deblocking filter (110b), and adaptive loop filter (110c).6.The video encoding method (800) as claimed in claim 1, wherein the weight matrix is used to modulate the AI-based in-loop filtering module (116) to perform content-driven enhancements to preserve at least one of a texture, a film-grain, and to remove compression artefacts.7.The video encoding method (800) as claimed in claim 1, wherein the AI-based in-loop filtering module (116) is pre-trained using the determined weight matrix as at least one of an input, a pixel attention, and a loss guidance during an in-loop filtering by an in-loop filtering sub-module.8.A video decoding method (900) for enhancing in-loop filter processing of a video codec in an artificial intelligence (AI) based in-loop filtering module, the method comprising:receiving (902) a bitstream of an encoded video;generating (904) a reconstructed video frame by at least one of an inverse quantizing and an inverse transforming of the bitstream;applying a plurality of in-loop filters to the reconstructed video frame to obtain a preliminary enhanced video frame;obtaining a weight matrix representing the target frequency bands to be enhanced for the reconstructed video frame from the at least one of the bitstream and an encoded template; andgenerating an enhanced reconstructed video frame (210) based on at least one of the preliminary enhanced video frame (210) and the weight matrix.9.The video decoding method (900) as claimed in claim 8 further comprising:receiving a plurality of templates in the form of a look-up table where each entry relates to a content type with its corresponding weight matrix from an encoded bitstream parameter set.10.The video decoding method (900) as claimed in claim 9 further comprising:selecting at least one of the plurality of received templates to specify the content type and its corresponding weight matrix.11.The video decoding method (900) of claim 8, wherein the plurality of in-loop filters comprise of at least one of a sample adaptive offset (204a), a deblocking filter (204b), and an adaptive loop filter (204c).12.The video decoding method (900) as claimed in claim 8, wherein the weight matrix is used to modulate the AI-based in-loop filtering module (208) to perform content-driven enhancements to preserve at least one of a texture, a film-grain, and to remove compression artefacts.13.The video decoding method (900) as claimed in claim 8 further comprising:applying the pre-trained AI based in-loop filtering module (208) by using the determined weight matrix during an in-loop filtering, wherein the in-loop filtering is performed by the in-loop filtering sub-module.14.A video decoding system (1000) for enhancing in-loop filter processing of a video codec in an artificial intelligence (AI) based in-loop filtering module, the video encoding system (1000) comprising:a memory (1004) configured to store executable instructions;at least one processor (1002) coupled to the memory (1004) and configured to execute the instructions to:receive an bitstream of an encoded video;generate a reconstructed video frame by at least one of an inverse quantizing and an inverse transforming of the bitstream;apply a plurality of in-loop filters to the reconstructed video frame to obtain a preliminary enhanced video frame;obtain a weight matrix representing the target frequency bands to be enhanced for the reconstructed video frame from the at least one of the bitstream and an encoded template; andgenerate an enhanced reconstructed video frame (210) based on at least one of the preliminary enhanced video frame (210) and the weight matrix.15.A method for transmitting a bitstream, the method comprising:receiving a original video frame (102) to be encoded;generating a reconstructed video frame by at least one of a transforming and a quantizing of a residue of the original video frame (102);applying a plurality of in-loop filters to the reconstructed video frame to obtain a preliminary enhanced video frame (118);determining at least one of a content type of a plurality of content types and one or more frequency bands of the original video frame (102) and the reconstructed video frame by analyzing the content of at least one of the original video frame (102) and the corresponding reconstructed video frame;generating a weight matrix representing target frequency bands (114) of the original video frame (102) based on the content type and the one or more frequency bands;generating an enhanced reconstructed video frame (118) based on at least one of the preliminary enhanced video frame (118) and the weight matrix;generating a bitstream by encoding information about the weight matrix representing target frequency bands; andtransmitting the bitstream from a video encoding apparatus to a video decoding apparatus.

Images