Method and system for content-based scaling of in-loop filter based on artificial intelligence

Abstract

A method for artificial intelligence (AI)-based media encoding includes compressing an input image frame associated with an input video, generating a reconstructed image frame corresponding to the input image frame using an AI-based in-loop filter, determining an offset value based on the input image frame and the reconstructed image frame, and encoding the input image frame and the reconstructed image frame. And encoding the reconstructed image frame on the basis of the determined offset value.

Description

Technical Field

[0001] The present invention relates generally to image processing, and more particularly to methods and systems for content-based scaling of an artificial intelligence (AI) based in-loop filter. Background Art

[0002] Video compression technology is fundamental to the development of consumer video content. The basic goal of video compression technology is to reduce transmission time and reduce bandwidth requirements. With the advent of artificial intelligence (AI) technology, video codec standard bodies are looking forward to reaping the potential benefits of AI technology in video compression technology. However, implementing AI-based models (such as deep neural networks, linear regression, etc.) has a different set of obstacles.

[0003] Video compression technology includes video codec pipelines for compressing a wide variety of content, where newer content is being developed all the time, such as content with higher resolutions, content for different screens, etc. When AI-based encoding tools are introduced into the video codec pipeline, the AI-based model must first be trained and then deployed in the video codec pipeline. However, AI-based models can only be trained on limited training data sets collected at a certain point in time, thereby introducing inherent bias and variance into the AI-based model and may lead to unexpected performance and untrustworthy answers. Summary of the invention

[0004] Technical Solution

[0005] This summary is provided to introduce a selection of concepts in a simplified format that are further described in the detailed description of the invention. This summary is not intended to identify key or essential inventive concepts of the invention, nor is it intended to determine the scope of the invention.

[0006] According to one embodiment of the present disclosure, a method for media encoding based on artificial intelligence (AI) is disclosed. In an embodiment of the present disclosure, the method may include compressing an input image frame associated with an input video. In an embodiment of the present disclosure, the method may include: using an AI-based in-loop filter to generate a reconstructed image frame corresponding to the input image frame. In an embodiment of the present disclosure, the method may include determining an offset value based on the input image frame and the reconstructed image frame. In an embodiment of the present disclosure, the method may include encoding the reconstructed image frame based on the determined offset value.

[0007] According to an embodiment of the present disclosure, a method for using AI-based media decoding is disclosed. In an embodiment of the present disclosure, the method may include receiving bitstream information including offset information from an encoder. In an embodiment of the present disclosure, the method may include generating a reconstructed image frame based on the bitstream information using an AI-based in-loop filter. In an embodiment of the present disclosure, the method may include performing a scaling operation on the reconstructed image frame based on the offset information to generate a scaled image frame. In an embodiment of the present disclosure, the method may include generating an output video based on the scaled image frame.

[0008] According to an embodiment of the present disclosure, an electronic device for AI-based media encoding is disclosed. The electronic device includes a system, wherein the system includes a processor coupled to a memory and a communicator. In an embodiment of the present disclosure, the processor may be configured to compress an input image frame associated with an input video. In an embodiment of the present disclosure, the processor may be configured to generate a reconstructed image frame corresponding to the input image frame using an AI-based in-loop filter. In an embodiment of the present disclosure, the processor may be configured to determine an offset value based on the input image frame and the reconstructed image frame. In an embodiment of the present disclosure, the processor may be configured to encode the reconstructed image frame based on the determined offset value.

[0009] According to an embodiment of the present disclosure, an electronic device for AI-based media decoding is disclosed. The electronic device includes a system, wherein the system includes a processor coupled to a memory and a communicator. In an embodiment of the present disclosure, the processor may be configured to receive bitstream information including offset information from an encoder. In an embodiment of the present disclosure, the processor may be configured to generate a reconstructed image frame based on the bitstream information using an AI-based in-loop filter. In an embodiment of the present disclosure, the processor may be configured to perform a scaling operation on the reconstructed image frame based on the offset information to generate a scaled image frame. In an embodiment of the present disclosure, the processor may be configured to generate an output video based on the scaled image frame.

[0010] In order to further illustrate the advantages and features of the present invention, a more specific description of the present invention will be presented by reference to the specific embodiments of the present invention shown in the accompanying drawings. It should be understood that these drawings only depict typical embodiments of the present invention and should not be considered as limiting the scope thereof. The present invention will be described and explained with additional features and details in the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] These and other features, aspects and advantages of the present invention will be better understood when the following detailed description is read with reference to the accompanying drawings, in which like characters refer to like parts throughout the several views, and in which:

[0012] Figure 1A An example scenario for visualizing the trade-off between bias and variance in an artificial intelligence (AI) based model according to the related art is shown;

[0013] Figure 1B The limitations of AI-based models based on bias and variance learned from training data according to the related art are shown;

[0014] Figure 2A A block diagram of a conventional Versatile Video Coding (VVC) decoder according to the related art is shown;

[0015] Figure 2B A conventional VVC decoder pipeline with an AI-based in-loop filter according to the related art is shown;

[0016] Figure 3 A block diagram of a VVC encoder pipeline for AI-based media coding according to an embodiment of the present disclosure is shown;

[0017] Figure 4 A rate-distortion curve diagram for determining a rate-distortion (RD) cost value for each compressed image frame for AI-based media coding according to the related art is shown;

[0018] Figure 5 A typical architecture of a deep convolutional neural network (DCNN) associated with an AI-based in-loop filter for improving the quality of each compressed image frame for AI-based media coding according to the related art is shown;

[0019] Figure 6 An exemplary scenario for generating a set of first representative data points according to an embodiment of the present disclosure is shown;

[0020] Figure 7 An exemplary scenario for generating a set of second representative data points according to an embodiment of the present disclosure is shown;

[0021] Figure 8 An exemplary scenario of generating a pixel map for offset scaling for AI-based media coding according to an embodiment of the present disclosure is shown;

[0022] Fig. 9 An exemplary scenario for generating offset scaling according to an embodiment of the present disclosure is shown;

[0023] Fig.10 and Fig.11 Various ways of modeling a domain according to embodiments of the present disclosure are shown;

[0024] Fig.12A block diagram of a VVC decoder pipeline for AI-based media coding according to an embodiment of the present disclosure is shown;

[0025] Fig.13A A block diagram of an electronic device for AI-based media encoding according to an embodiment of the present disclosure is shown;

[0026] Fig. 13B A block diagram of an electronic device for AI-based media decoding according to an embodiment of the present disclosure is shown;

[0027] Fig.14 is a flowchart illustrating a method for AI-based media encoding according to an embodiment of the present disclosure;

[0028] Fig.15 is a flowchart illustrating a method for AI-based media compression according to an embodiment of the present disclosure; and

[0029] Fig.16 is a flowchart illustrating a method for AI-based media decoding according to an embodiment of the present disclosure.

[0030] Furthermore, those skilled in the art will appreciate that those elements in the drawings are illustrated for simplicity and may not necessarily be drawn to scale. For example, a flow chart illustrates the method in terms of the most salient steps involved to help improve understanding of the various aspects of the invention. Furthermore, with respect to the construction of the device, one or more components of the device may have been represented in the drawings by conventional symbols, and the drawings may show only those specific details relevant to understanding the embodiments of the invention so as not to obscure the drawings with details that would be readily apparent to one of ordinary skill in the art having the benefit of the description herein. DETAILED DESCRIPTION

[0031] To promote an understanding of the principles of the invention, reference will now be made to the embodiments illustrated in the drawings and specific language will be used to describe the embodiments. It will be understood, however, that no limitation of the scope of the invention is intended thereby and such changes and further modifications in the illustrated systems and such further applications of the principles of the invention as illustrated therein are deemed to be within the ordinary scope of those skilled in the art to which the invention relates.

[0032] Those skilled in the art will understand that both the foregoing general description and the following detailed description are illustrative of the present invention and are not intended to limit the present invention.

[0033] References throughout this specification to "on one hand," "on the other hand," or similar language mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the phrases "in an embodiment," "in one embodiment," "in another embodiment," and similar language appearing throughout this specification may, but do not necessarily, all refer to the same embodiment.

[0034] The terms "comprises", "comprising ..." or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process or method that includes a list of steps includes not only those steps but may also include other steps not expressly listed or inherent to such process or method. Similarly, one or more devices or subsystems or elements or structures or components followed by "comprising ..." do not exclude the presence of other devices or other subsystems or other elements or other structures or other components or additional devices or additional subsystems or additional elements or additional structures or additional components, without more constraints.

[0035] Throughout this disclosure, the expression "at least one of a, b, or c" refers to only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0036] The embodiments of this article and its various features and advantageous details are explained more fully with reference to the non-limiting embodiments shown in the drawings and described in detail in the following description. The description of known components and processing techniques is omitted so as not to unnecessarily obscure the embodiments of this article. In addition, the various embodiments described herein are not necessarily mutually exclusive, because some embodiments can be combined with one or more other embodiments to form new embodiments. Unless otherwise stated, the term "or" used herein refers to a non-exclusive "or". The examples used herein are intended only to facilitate understanding of the manner in which the embodiments of this article can be practiced, and further enable those skilled in the art to practice the embodiments of this article. Therefore, the examples should not be interpreted as limiting the scope of the embodiments of this article.

[0037] As traditional in the art, embodiments can be described and illustrated according to the block of performing one or more functions described. These blocks (which may be referred to as units or modules, etc. in this article) are physically implemented by analog or digital circuits (such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, etc.), and can be optionally driven by firmware and software. The circuit can be, for example, embodied in one or more semiconductor chips, or embodied on substrate supports such as printed circuit boards, etc. The circuit constituting the block can be implemented by dedicated hardware, or by a processor (for example, one or more programmed microprocessors and associated circuits), or by a combination of dedicated hardware that performs some functions of the block and a processor that performs other functions of the block. Without departing from the scope of the present invention, each block of the embodiment can be physically divided into two or more interacting and discrete blocks. Similarly, without departing from the scope of the present invention, the block of the embodiment can be physically combined into more complex blocks.

[0038] The accompanying drawings are used to help easily understand various technical features, and it should be understood that the embodiments presented herein are not limited by the accompanying drawings. Therefore, except for those specifically set forth in the accompanying drawings, the present disclosure should be interpreted as extending to any changes, equivalents and alternative forms. Although the terms first, second, etc. can be used herein to describe various elements, these elements should not be limited by these terms. These terms are usually only used to distinguish one element from another element.

[0039] Throughout this disclosure, the terms "desired compression level" and "desired level of compression" are used interchangeably and mean the same thing. Throughout this disclosure, the terms "input video," "input video data," and "input media" are used interchangeably and mean the same thing. The terms "reconstructed video," "output video," and "optimized compressed media" are used interchangeably and mean the same thing.

[0040] Bias and variance are two key properties of models (e.g., AI-based models, machine learning (ML) models, etc.). The bias of a model refers to how well the model can represent all potential outcomes. Variance, on the other hand, refers to how well the model's predictions are affected by small changes in the input data. The trade-off between bias and variance is a fundamental issue in models, and it is often necessary to experiment with several model types in order to find the balance that best fits the training data. Figure 1AAn example scenario for visualizing the trade-off between bias and variance in an AI-based model according to the related art is shown. A person throws one or more stones hoping to reach a point representing a target value. However, one or more stones may hit the ground at various locations that may be far from the target value. The deviation is depicted here as the gap between the target value (i.e., the actual value) and the average of the points where one or more stones land. The dotted line reflects the average of the predictions provided by one or more AI-based models, which are trained using different training data sets extracted from the same group of people for different locations where one or more stones hit the ground. Here, the dotted line can reflect the expected location where one or more stones may hit the ground according to the calculation of one or more AI-based models. However, since each AI-based model is trained with different training data, one or more stones may fall at various locations around the target value. Taking into account the changes in the location where one or more stones hit the ground, the dotted line can be used as an approximation of the expected results according to one or more AI-based models. The shorter the distance, the smaller the deviation between the average value (i.e., the dotted line) and the target value, resulting in a better model. Variance can be represented as the spread of individual points around the mean (i.e., the dashed line). Lower spread and variance indicate a more optimal model in one or more AI-based models. This is due to the fact that lower spread means a higher level of accuracy and precision in the model's predictions. By minimizing variance, the model is able to accurately predict outcomes and provide more reliable results.

[0041] Figure 1BThe limitations of AI-based models based on the bias and variance of learning from training data according to the related art are shown. In order to obtain the expected performance and credible answers of the AI-based model, the source data (ie, training data) and the target data (ie, test data) must be from the same sample space. However, in a real-time environment, the source data and the target data may belong to different sample spaces. In this case, the AI-based model may not provide the expected performance and credible response. For example, consider a scenario in which an AI-based model is trained only on the faces of Indians (limited training data) for facial classification and detection. In this case, if the trained AI-based model is run on the faces of Americans (ie, newer and unseen data), the trained AI-based model may not perform as expected. This is due to certain differences between the faces of Indians and the faces of Americans (eg, anchor point changes). Therefore, the trained AI-based model has the bias and variance of learning for the faces of Indians. Therefore, the trained AI-based model may not provide the expected performance and credible response. To solve this problem, the AI-based model must be retrained every time a new sample space is found, which may result in a large amount of and continuous manual operations.

[0042] Furthermore, since newer forms of data are constantly generated, AI-based models tend to become outdated over time and must be retrained. However, it is inconvenient to retrain AI-based models regularly. For example, video compression technology is deployed globally, and it is not desirable to frequently adjust the weights of AI-based models. Furthermore, conventional in-loop filters (ILFs) connected to AI-based models utilize image and signal processing principles to identify certain artifacts. Figure 2A and Figure 2B As shown in and discussed later in the specification, with limited options, conventional ILFs employ many modes / filters to best suit a given content. Experimenting with various modes / filters for a single input increases encoding time. Some conventional ILFs do not even use measurements as a measure of quality, but instead focus only on visible artifacts; this inherent nature of conventional ILFs makes them rigid.

[0043] Figure 2AA block diagram of a conventional general video coding (VVC) decoder according to the related art is shown. The VVC decoder is an important component of a video coding system that can efficiently compress and decompress video data. The VVC decoder is configured to provide high-quality video playback while minimizing the amount of data required for transmission and storage. One of the basic modules of the VVC decoder is context-adaptive binary arithmetic coding (CABAC), which is a complex entropy coding technique that adapts to the statistical characteristics of the input data. CABAC allows the VVC decoder to achieve a high compression ratio without sacrificing video quality. In addition to CABAC, the VVC decoder also includes several other important modules, including an inverse quantization module, an inverse transform module, a luminance-mapped chrominance scaling (LMCS) module, an in-loop filter module, an intra-frame prediction module, an inter-frame prediction module, a LMCS (forward luminance mapping) module, a combined inter-frame intra-frame prediction (CIIP) module, a decoded picture buffer (DPB) module, and a reconstructed image. The inverse quantization module and the inverse transform module are responsible for converting the quantized and transformed video data back to its original form. The LMCS module is used to adjust the chrominance and luminance components of the video data to improve the overall image quality.

[0044] In addition, the in-loop filter is configured to remove artifacts and noise that may occur during the video compression process. The in-loop filter typically includes an inverse luminance-mapped chroma scaling (LMCS), a deblocking filter (DBF), a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), and a cross-component adaptive loop filter (CC-ALF). The inverse LMCS module is configured to adjust the chroma scaling to match the luminance mapping and ensure proper color representation during video compression. DBF is used on block boundaries to eliminate block artifacts. The SAO filter is then applied to the deblocked samples. SAO is very effective in removing ring artifacts and correcting local average intensity offsets. ALF and CC-ALF perform block-based linear filtering and adaptive filtering to minimize the mean square error (MSE) between the original data (e.g., the original image) and the reconstructed data (e.g., the reconstructed image).

[0045] Finally, the DPB module and the reconstructed image module work together to store and display decoded video data. The DPB module stores previously decoded frames to achieve efficient inter-frame prediction, while the reconstructed image module generates high-quality display images from the decoded video data.

[0046] The intra-frame prediction module and the inter-frame prediction module are configured to predict the value of the video data based on spatial correlation and temporal correlation. This helps to reduce the amount of data that needs to be transmitted or stored by encoding only the difference between the predicted value and the actual value. The CIIP module combines intra-frame and inter-frame prediction techniques to further improve compression efficiency.

[0047] In a conventional VVC decoder, each in-loop filter targets a unique artifact or a predefined artifact. For example, as mentioned earlier, DBF only removes block artifacts. As a result, since there is no "intelligent" element in the conventional filtering process, the conventional filtering process cannot adjust performance based on the type of input content. In order to obtain the best performance, the conventional filtering process uses heuristics to select among multiple different filter versions (e.g., LMCS, DBF, SAO, etc.), which reduces the user experience.

[0048] Figure 2B A conventional VVC decoder pipeline with an AI-based in-loop filter according to the related art is shown. Figure 2A Compared to the block diagram in , the AI-based in-loop filter is an additional block in the conventional VVC decoder pipeline. The AI-based in-loop filter eliminates the problem of in-loop filters (i.e., each filter targets a unique artifact or a predefined artifact). However, as Figure 1B As shown, due to the limited amount of training data used by the AI-based model, the AI-based in-loop filter may add inherent bias and variation. Therefore, in scenarios where the AI-based model has not been trained, the AI-based in-loop filter may produce unexpected results and make the solution untrustworthy. In addition, since newer forms of data are constantly generated, the AI-based model may become outdated over time and must be retrained. However, it is inconvenient to retrain the AI-based model regularly.

[0049] As combined Figures 3 to 16As described, the method according to the embodiment of the present disclosure can provide a unique strategy for enhancing AI-based models, which utilizes an additional real-time module to adapt to updated and unseen data by performing content-based scaling to reduce data domain gaps. The method according to the embodiment of the present disclosure uses pixel mapping to bridge the data domain gap between updated / unseen data and training data in the AI-based in-loop filter. Therefore, the method according to the embodiment of the present disclosure is superior to conventional methods that target specific artifacts and frequently train with limited data, resulting in inherent bias and volatility in AI-based training models. The method according to the embodiment of the present disclosure can implement content-based scaling to identify pixel mappings for scaling offsets, operate in real time, do not require pre-training, and can provide greater generalization for different types of data. In addition, when compared with conventional methods, the complexity of the method according to the embodiment of the present disclosure is very low, because conventional methods deploy very complex deep neural network (DNN) models to achieve higher gains, and they are also not adapted to content-to-content basis. The method according to the embodiment of the present disclosure can solve this problem by introducing statistical regression based on video data during the encoding process and using statistical regression based on video data during the decoding process.

[0050] Referring now to the drawings, and more particularly to Figures 3 to 16 One or more embodiments of the present disclosure are shown, wherein like reference numerals denote corresponding features consistently throughout the drawings.

[0051] Figure 3 A block diagram of a Versatile Video Coding (VVC) encoder pipeline 300 for AI-based media (eg, input video) encoding is shown according to an embodiment of the present disclosure.

[0052] The VVC encoder pipeline 300 includes multiple modules. The multiple modules may include a compression module 301, an in-loop filter 302, a CABAC module 303, and an RD cost module 304. The compression module 301 is a conventional module that can receive (or identify) an input video (e.g., one or more input image frames). Upon receiving the input video, the compression module 301 may perform one or more predefined operations on the received input video, such as transform, quantization, dequantization, and inverse transform.

[0053] a. Transform: A transform operation involves converting data (e.g., input video) from its original domain representation to a different domain where the data can be more efficiently compressed and includes transform coefficients. Transform operations are typically implemented using mathematical techniques such as discrete cosine transform (DCT) or discrete wavelet transform (DWT).

[0054] b. Quantization: Once the data has been transformed, a quantization operation is performed. The quantization operation involves reducing the precision or dynamic range of the transform coefficients. The transform coefficients essentially map a continuous range of values ​​to a finite set of discrete levels. By quantizing the coefficients, information is lost, which helps in compression.

[0055] c. Inverse quantization: The inverse quantization operation is the inverse of the quantization operation. Inverse quantization involves restoring the quantized coefficients to their approximate original values, which is achieved by multiplying each quantized coefficient by an inverse quantization factor. The inverse quantization operation is performed during decompression to restore the approximate values ​​of the original transform coefficients.

[0056] d. Inverse Transform: The inverse transform operation involves converting the transformed and dequantized coefficients back to the original domain. The inverse transform operation performs the inverse of the initial transform, reconstructing the compressed data to closely resemble the original input (e.g., input video). An inverse transform operation (such as an inverse DCT or inverse DWT) effectively reverses the decorrelation and energy concentration performed during the initial transform stage.

[0057] The in-loop filter 302 may receive (or identify) compressed data (e.g., quantized data, reconstructed data) from the compression module 301. The in-loop filter 302 is configured to remove artifacts and noise that may occur during the compression operation. The in-loop filter 302 may include a LMCS 302a, a DBF 302b, a SAO filter 302c, an ALF, and a CC-ALF 302d. The functions of the various blocks (i.e., 302a, 302b, 302c, and 302d) are similar to those of FIG. Figure 2A , and these various blocks are conventional modules.

[0058] In one or more embodiments of the present disclosure, the in-loop filter 302 may include an AI-based in-loop filter 302e, a data distribution model 302f, a ground truth data distribution model 302g, a mapping range module 302h controlled by rate-distortion optimization (RDO), a pixel mapping module 302i for offset scaling, and a final AI-based in-loop filter output module 302j.

[0059] In one or more embodiments of the present disclosure, the AI-based in-loop filter 302e may be configured to apply AI technology in the loop filtering process of video encoding, such as in combination with Figure 5As described. In-loop filter modules 302a, 302b, 302c, and 302d can typically be designed using a fixed method to reduce compression artifacts (such as block artifacts and ringing artifacts). However, with the advancement of AI and machine learning, researchers have explored the use of AI-based methods to enhance the performance of in-loop filter modules 302a, 302b, 302c, and 302d. The AI-based in-loop filter 302e can use a machine learning model to learn the characteristics of video content (e.g., images input from VVC, input videos, etc.) and apply adaptive filtering techniques. Instead of relying on predefined rules or heuristics, the AI-based in-loop filter 302e can use a neural network or other AI method to analyze and process video content. The AI-based in-loop filter 302e can use an AI model and can learn from a large amount of training data to make intelligent decisions about filtering operations. By using an AI-based in-loop filter 302e, the effectiveness of artifact reduction can be improved, especially in challenging scenes that may be difficult to achieve with conventional in-loop filter modules 302a, 302b, 302c, and 302d. The AI ​​model can adapt to different content types, handle complex textures and motion, and preserve details in a better manner while reducing artifacts. Overall, the AI-based in-loop filter 302e can be designed to leverage the capabilities of AI to improve the quality of encoded video content by more adaptively and intelligently reducing compression artifacts. The encoded video content may include reconstructed video and / or one or more reconstructed image frames, such as in combination with Figure 5 As described.

[0060] In one or more embodiments of the present disclosure, the data distribution model 302f may generate a set of first representative data points, such as Figure 6 The data distribution model 302f may perform a plurality of operations given below to generate the set of first representative data points.

[0061] The data distribution model 302f may receive (or identify) one or more reconstructed image frames (e.g., one or more enhanced image frames, model output data) from the AI-based in-loop filter 302e, wherein the one or more reconstructed image frames are associated with the reconstructed video. Then, the data distribution model 302f may receive (or identify) a user input indicating a desired compression level for each of the one or more reconstructed image frames. Then, the data distribution model 302f may determine the number of segments (e.g., 5) of each of the one or more reconstructed image frames based on the received user input (or desired compression level). Then, the data distribution model 302f may fragment each of the one or more reconstructed image frames based on the determined number of segments. Then, the data distribution model 302f uses one or more distribution mechanisms (e.g., normal distribution, etc.) to analyze the content changes within each of the one or more reconstructed image frames to calculate the best segment for each of the one or more reconstructed image frames (i.e., generate the best fragmented one or more image frames). Then, the data distribution model 302f may perform a pixel binning operation on the best fragmented one or more image frames using one or more statistical mechanisms (e.g., a Gaussian mixture model (GMM)), wherein each of the best fragmented one or more image frames includes a group of pixels having unique characteristics (such as regions with similar textures, similar deviations from mean, etc.). The data distribution model 302f may perform a pixel binning operation on one or more segments (or regions) of each of the best fragmented one or more image frames to generate a set of first representative data points for each of the best fragmented one or more image frames, wherein the set of first representative data points may represent a scalar value for each of the one or more segments (or the best fragmented one or more image frames).

[0062] In one or more embodiments of the present disclosure, the real data distribution model 302g can generate a set of second representative data points, such as Figure 7 The real data distribution model 302g may perform a number of operations to generate a set of second representative data points, which are given below.

[0063] The real data distribution model 302g may receive (or identify) one or more input image frames associated with an input video (e.g., original video, uncompressed video). The real data distribution model 302g may then receive (or identify) a user input indicating a desired compression level for each of the one or more input image frames. The real data distribution model 302g may then determine the number of segments for each of the one or more input image frames based on the received user input (or desired compression level). The real data distribution model 302g may then fragment each of the one or more input image frames based on the determined number of segments. The real data distribution model 302g may then use one or more distribution mechanisms to analyze content changes within each of the one or more input image frames to calculate an optimal segment for each of the one or more input image frames (i.e., to generate optimally fragmented one or more input image frames). Then, the true data distribution model 302g may perform a pixel merging operation on the optimally fragmented one or more input image frames using one or more statistical mechanisms, wherein each of the optimally fragmented one or more input image frames includes a set of pixels (or pixel regions) having unique characteristics. The true data distribution model 302g may perform a pixel merging operation on one or more segments (or regions) of each of the optimally fragmented one or more input image frames to generate a set of second representative data points for each of the optimally fragmented one or more input image frames, wherein the set of second representative data points may represent a scalar value for each of the one or more segments (or the optimally fragmented one or more image frames).

[0064] In one or more embodiments of the present disclosure, the RDO controlled mapping range module 302h may determine the overall RD cost based on the output of the RD cost module 304 and the user input regarding the number of fragments, such as in combination with Figure 4 As described. The mapping range module 302h of RDO control can construct an objective function represented as a mathematical equation. The objective function can take inputs such as updated rate and distortion and previous RDO cost. Based on the updated RDO cost, the RD cost module 304 can make a decision to optimize the overall RD cost by minimizing the rate or distortion or both.

[0065] In one or more embodiments of the present disclosure, the pixel mapping module 302i for offset scaling may determine a data distribution dissimilarity measure between a set of first representative data points for each fragmented image frame associated with the reconstructed video and a set of second representative data points for each fragmented input image frame associated with the input video, such as in combination with Figure 8The pixel mapping module 302i for offset scaling can also determine the offset value for each fragmented image frame by utilizing the data distribution dissimilarity metric, such as in combination with Figure 8 The offset value is calculated and used at a per-segment granularity, where the number of segments is received as input from a user, and where each segment is calculated during a data distribution modeling operation (eg, a data distribution dissimilarity measure).

[0066] In one or more embodiments of the present disclosure, the pixel mapping module 302i for offset scaling can determine (or calculate) the (expected) pixel mapping in the form of a per-fragment offset using the model output data distribution, the real data distribution, and the mapping range (e.g., the mapping range controlled by RDO). The pixel mapping can be a pixel-level domain mapping based on the differences in the modeled distribution. The mapping range can be a decision to apply offset scaling to a particular fragment based on the RD cost, where the mapping range is determined using the number of user-defined fragments and the output of the RD cost module 304 (e.g., the codec RD cost).

[0067] In one or more embodiments of the present disclosure, the final AI-based in-loop filter output module 302j may perform a scaling operation on each pixel value associated with each fragmented image frame by utilizing the determined offset value, such as in combination with Fig. 9 Examples of scaling operations may include, but are not limited to, addition, multiplication, division, or exponential operations. The final AI-based in-loop filter output module 302j may further enhance the reconstructed video based on the scaling operation performed.

[0068] In one or more embodiments of the present disclosure, the encoder may send bitstream information associated with the reconstructed video to the decoder (refer to Fig.12 ), where the bitstream information may include a determined offset value (e.g., offset signaling).

[0069] In addition, the CABAC module 303 and the RD cost module 304 are two conventional modules used in the VVC encoder, which is the latest video coding standard. The CABAC module 303 is an entropy coding technique used in the VVC encoder for effectively compressing transformed and quantized coefficients. In the VVC encoder, the CABAC module 303 is configured to compress input video data and perform the following functions:

[0070] a. Context modeling: The CABAC module 303 can analyze input video data (e.g., input video, image frames, etc.) and create a statistical model for capturing the relationships and dependencies between symbols in the input video data. The CABAC module 303 can estimate the probability of different symbols appearing based on the context associated with the input video data.

[0071] b. Binary Arithmetic Coding: The CABAC module 303 can convert the probabilities into a cumulative distribution function (CDF). The CDF is used to assign binary codewords to symbols in the input video data. Binary arithmetic coding techniques efficiently encode symbols based on probabilities.

[0072] c. Bitstream Generation: The CABAC module 303 can generate a compressed bitstream that contains the coded symbols and the necessary information for the decoder to reconstruct the input video data. The compressed bitstream is then sent or stored for further processing or transmission, as described in detail below.

[0073] The RD cost module 304 is configured to determine the best trade-off between rate (ie, bit rate) and distortion (ie, visual quality), which helps the VVC encoder find the best encoding parameters for each coding unit by evaluating the rate and distortion of different encoding options. The RD cost module 304 can perform various operations listed below for encoding input video data.

[0074] a. The RD cost module 304 can estimate the number of bits required to encode data representing a particular coding option. The number of bits includes bits used to encode syntax elements, motion information, and residual data.

[0075] b. To evaluate visual quality, the RD cost module 304 can calculate the distortion by comparing the reconstructed video with the original input. Common metrics such as mean square error (MSE) or structural similarity index (SSIM) can be used for distortion measurement.

[0076] c. The RD cost module 304 can perform rate-distortion optimization by evaluating various coding options for each coding unit. The RD cost module 304 explores different modes, motion vectors, transform options, and quantization parameters to find a combination that achieves the best trade-off between rate and distortion.

[0077] d. Based on the rate and distortion calculations, the RD cost module 304 can select the encoding options that minimize the overall RD cost. These optimal options are used for the final encoding of the input video data.

[0078] By including the RD cost module 304, the VVC encoder can efficiently allocate bits based on content characteristics and perceptual importance, thereby improving video compression performance while maintaining satisfactory visual quality.

[0079] Figure 4 A rate-distortion graph 400 for determining the RD cost value of each compressed image frame for AI-based media encoding according to the related art is shown.

[0080] The RD cost module 304 may obtain user preferences for the amount of compression of the media and the quality of the compressed media. The RD cost module 304 may determine a rate (R) value (i.e., a bit rate) for each compressed image frame based on the amount (or degree) of compression, as shown in Equation 1. The RD cost module 304 may determine a distortion (D) value for each compressed image frame based on the quality of the compressed content, as shown in Equation 2.

[0081] (1)

[0082] (2)

[0083] The RD cost module 304 can determine a tuning parameter (λ), wherein the tuning parameter is configured based on current requirements (i.e., user preferences for the amount of compression of the media), and the tuning parameter can allow for different weights of rate or distortion. The RD cost module 304 can determine the RD cost value for each compressed image frame based on the R value, the D value, and the tuning parameter (λ), as shown in Equation 3.

[0084] (3) RD cost = R + D * λ

[0085] The rate-distortion graph 400 shows a plurality of rate (R) values ​​on the y-axis and a plurality of distortion (D) values ​​on the x-axis. In addition, the rate-distortion graph shows a first curve 401 of a first tuning parameter (λ1) and a second curve 402 of a second tuning parameter (λ2).

[0086] Based on the user's current requirements, the user can reduce the R value or the D value, as shown in the rate-distortion curve diagram 400. In other words, there is a trade-off between the R value and the D value. For example, as depicted at element 403, if the user wants the output of the video codec pipeline to have extremely high quality and almost no distortion, the D value can be selected to be 40.5. Then, the R value can be increased to reduce distortion. The R value is related to the second curve 402 (e.g., R value = 1.25) or to the first curve 401 (e.g., R value = 1.75). If the user wants to optimize the memory of the electronic device, the RD cost module 304 can select the second curve 402 with the lowest R value.

[0087] Figure 5A typical architecture of a deep convolutional neural network (DCNN) 500 associated with an AI-based in-loop filter 302e for AI-based media coding for improving the quality of each compressed image frame (e.g., a reconstructed image frame) according to the related art is shown.

[0088] The AI-based in-loop filter 302e receives one or more input image frames from at least one of the compression module 301, LMCS 302a, DBF 302b, SAO filter 302c, ALF, and CC-ALF 302d. The one or more input image frames may be represented by a first set of dimensions (e.g., H×W×C1) having a specific height and width with respect to the input image pixels and one or more first channels, where H represents the height of the tensor, W represents the width of the tensor, and C represents the number of channels in the tensor. The dimensions may correspond to the specific height, width, and number of channels present in the tensor of the one or more input image frames. The tensor may be constructed in such a way that it organizes the pixel values ​​of the one or more image frames in a multidimensional array. Then, the one or more input image frames may be passed through one or more DCNNs to enhance the quality of the one or more input image frames. The one or more enhanced image frames may be represented by a second set of dimensions having a specific height and weight with respect to the image pixels of the input image signal and one or more second channels (e.g., H×W×C2). In one embodiment of the present disclosure, the use of an AI-based in-loop filter 302e for quality enhancement in a video codec pipeline may allow for management of a wider range of data variations because the AI-based module may be trained to remove multiple types of artifacts (e.g., blocking artifacts, ringing artifacts, etc.). In one embodiment of the present disclosure, the AI-based in-loop filter 302e may be designed to combat various compression artifacts. The decisions of such a filter are in-loop, which may also affect decisions for future image frames. The AI-based in-loop filter 302e may pass one or more enhanced image frames to a data distribution model 302f for further processing, such as in conjunction with the Figure 6 as described.

[0089] Figure 6 An exemplary scenario 600 for generating a first set of representative data points according to an embodiment of the present disclosure is shown. An enhanced image frame is received from the AI-based in-loop filter 302e.

[0090] In step 601, the AI-based in-loop filter 302e may receive one or more input image frames (e.g., reconstructed image frames, compressed image frames) associated with the input video from at least one of the compression module 301, LMCS 302a, DBF 302b, SAO filter 302c, ALF or CC-ALF 302d. In step 602, upon receiving the one or more input image frames, the AI-based in-loop filter 302e may enhance the quality of the one or more input image frames. The AI-based in-loop filter 302e may then pass the one or more enhanced image frames to the data distribution model 302f for further processing, as described in detail below.

[0091] In steps 603-604, the data distribution model 302f may receive (or identify) one or more enhanced image frames from the AI-based in-loop filter 302e, and receive (or identify) user input to determine the number of segments for each enhanced image frame. For example, the user may specify a preferred compression trade in the form of a command line parameter.

[0092] In steps 605-606, the data distribution model 302f can analyze the content changes within each enhanced image frame by utilizing one or more distribution mechanisms to determine one or more optimally fragmented image frames, wherein each optimally fragmented image frame may include a set of pixels having unique characteristics. In an embodiment of the present disclosure, the data distribution model 302f can analyze the content changes within each enhanced image frame to generate one or more optimal fragments for each enhanced image frame. Then, the data distribution model 302f can perform pixel merging on each enhanced image frame by utilizing one or more statistical mechanisms. Then, the data distribution model 302f can fragment each enhanced image frame based on at least one of the received user input, the analyzed content changes, or the pixel merging. For example, each enhanced image frame may include four fragments or four parts (e.g., fragment-1, fragment-2, fragment-3, etc.), and each part is represented by a specific band (e.g., band-1).

[0093] At step 607, the data distribution model 302f may generate a set of first representative data points for each of the best fragmented one or more image frames, wherein the set of first representative data points may represent a vector of scalar values ​​for each of the best fragmented one or more image frames. In an embodiment of the present disclosure, the data distribution model 302f may perform pixel merging for each of the one or more fragments within each best fragmented image frame to create a set of first representative data points for each best fragmented image frame. The set of first representative data points for each best fragmented image frame may represent a scalar value for each of the one or more fragments within each best fragmented image frame.

[0094] In one embodiment, the data distribution model 302f segments each enhanced image frame using, for example but not limited to, a Gaussian mixture (GM) model.

[0095] In one embodiment, each fragmented image frame can capture different kinds of content within each enhanced image frame. Since different kinds of data changes can generate different kinds of internal data distributions, resulting in different kinds of fragments, the method according to an embodiment of the present disclosure can customize processing for multiple data changes. In addition, since the disclosed method works on the fly, no pre-training is required and it can be generalized for all kinds of input data (e.g., image frames, input videos, etc.).

[0096] Figure 7 An exemplary scenario 700 for generating a set of second representative data points according to an embodiment of the present disclosure is shown.

[0097] In steps 701 - 702 , the real data distribution model 302 g may receive (or identify) one or more input image frames associated with an input video (eg, an original video).

[0098] In steps 702-703, the real data distribution model 302g may receive (or recognize) user input to determine the number of segments for each input image frame. For example, the user may specify the segment preference in the form of a command line parameter.

[0099] In steps 704-705, the real data distribution model 302g can analyze the content changes within each input image frame by utilizing one or more distribution mechanisms to determine one or more optimally fragmented input image frames, wherein each optimally fragmented input image frame may include a set of pixels having unique characteristics. In an embodiment of the present disclosure, the real data distribution model 302g can analyze the content changes within each input image frame to generate one or more optimal fragments for each input image frame. Then, the real data distribution model 302g can perform pixel merging on each input image frame by utilizing one or more statistical mechanisms. Then, the real data distribution model 302g can fragment each input image frame based on the received user input, the analyzed content changes, and the pixel merging. For example, each input image frame includes four fragments or four parts (e.g., fragment-1, fragment-2, fragment-3, etc.), and each part is represented by a specific band (e.g., band-1).

[0100] At step 706, the true data distribution model 302g may generate a set of second representative data points for each of the best fragmented one or more input image frames, wherein the set of second representative data points may represent a vector of scalar values ​​for each of the best fragmented one or more input image frames. In an embodiment of the present disclosure, the true data distribution model 302g may perform pixel merging on each of the one or more fragments within each best fragmented input image frame to create a set of second representative data points for each best fragmented input image frame. The set of second representative data points for each best fragmented input image frame may represent a scalar value for each of the one or more fragments within each best fragmented input image frame.

[0101] In one embodiment, the true data distribution model 302g may segment each input image frame using, for example but not limited to, a GM model.

[0102] In one embodiment, the one or more statistical mechanisms utilized by the data distribution model 302f are the same as the one or more statistical mechanisms utilized by the true data distribution model 302g.

[0103] Figure 8 An exemplary scenario 800 for generating a pixel map for offset scaling for AI-based media coding according to an embodiment of the present disclosure is shown.

[0104] In one embodiment of the present disclosure, the pixel mapping module 302i for offset scaling receives (or identifies) modeled data distribution information from the data distribution model 302f and the true data distribution model 302g. One or more distribution mechanisms utilized by the two models 302f, 302g are the same, and then, the pixel mapping module 302i for offset scaling can compare the outputs of the two models 302f, 302g and determine the difference between the outputs of the two models (i.e., the data distribution model 302f and the true data distribution model 302g). The fragment information (e.g., fragment-1) is received together with a representative value of the domain (e.g., the amplitude of the pixel binning curve, such as Figure 6 and Figure 7 ) may already exist in the modeled data distribution. The pixel mapping module 302i for offset scaling may consider a segment at a specific time instance and calculate the transformation between the segment domain representative value of the corresponding segment in the real data (e.g., input image frame, fragmented input image frame) and the model output data (e.g., reconstructed image frame, enhanced image frame, fragmented image frame).

[0105] In one embodiment of the present disclosure, the pixel mapping module 302i for offset scaling may receive (or identify) mapping range information (a decision to apply offset scaling to a particular fragment (e.g., fragment-1)) from the RDO controlled mapping range module 302h. In one embodiment of the present disclosure, the pixel mapping module 302i for offset scaling may calculate the mapping range based on the objective function from the RDO controlled mapping range module 302h. The mapping range information is determined by two functions, as shown in Equation 4.

[0106] (4) Mapping range controlled by RDO = f(number of user-defined segments, codec RD cost)

[0107] In Equation 4, f may be an objective function that minimizes the total RD cost constructed by the RDO controlled mapping range module 302h.

[0108] For example, graph 801 represents distribution modeling associated with data distribution model 302f, while graph 802 represents distribution modeling associated with real data distribution model 302g. Graphs 801 and 802 depict domain information for fragment-1. The pixel mapping module 302i for offset scaling may use mathematical operations (e.g., mean operations) for offset scaling to determine offset values. The average value of the data represented on the xy axis of graph 801 is "2", while the average value of the data represented on the xy axis of graph 802 is "2.5". As a result, a domain difference average value of "0.5" is obtained. In order to overcome the domain difference average value, one or more optimization operations (e.g., addition, multiplication, etc.) may be performed, and the pixel values ​​associated with fragment-1 may be scaled to reduce distortion. The above process is performed for each fragment in the image frame. As a result, the pixel mapping module 302i for offset scaling may generate an offset map 803. The generated offset map 803 may be sent to the final AI-based in-loop filter output module 302j for further processing, such as in combination with Fig. 9 As described.

[0109] Fig. 9 An exemplary scenario 900 for generating offset scaling according to an embodiment of the present disclosure is shown.

[0110] The final AI-based in-loop filter output module 302j receives (or identifies) the generated offset map 901 from the pixel mapping module 302i for offset scaling, and receives (or identifies) the enhanced image frame 902 from the AI-based in-loop filter 302e. Upon receiving this information, the final AI-based in-loop filter output module 302j can perform offset scaling on the enhanced image frame 902 based on the generated offset map 901. As a result, the final AI-based in-loop filter output module 302j can generate a content-based scaling output.

[0111] The method according to an embodiment of the present disclosure only needs to encode each segment (or each fragment) offset in the bitstream information. The method according to an embodiment of the present disclosure may not encode the information associated with the data distribution model 302f and the real data distribution model 302g. In other words, it is not necessary to provide information about the instant modeling in the bitstream, and therefore, the disclosed method does not significantly increase the rate associated with the encoding process.

[0112] Fig.10 and Fig.11 Various ways of modeling a domain according to embodiments of the present disclosure are shown.

[0113] refer to Fig.10 :In Fig.10In the example scene shown, there is a plot 1000 representing the first three principal components. The plot 1000 visualizes pixel points in three-dimensional space. For example, there are 100 points with a quantization parameter (QP) value (i.e., 42) 1001, 500 points with a QP value 1002, and all points with a QP value 1003. In order to calculate representative points (i.e., a set of first representative data points and a set of second representative data points) using a principal component analysis (PCA) method, the pixel mapping module 302i for offset scaling can transform the pixel values ​​using a PCA method. The transformation allows the transformed values ​​to be visualized as clusters. From these clusters (i.e., 1001, 1002, and 1003), the pixel mapping module 302i for offset scaling can extract cluster centroids, which are used as representative points for that particular cluster (i.e., 1001, 1002, and 1003). The representative points capture the essential characteristics of the clusters (ie, 1001, 1002, and 1003) and can be used for further analysis or decision making process.

[0114] refer to Fig.11 : One or more graphs 1100 represent, for example, mean and standard deviation modeling of pixel differences (referred to as “differences”) between the true data distribution model 302g and the data distribution model 302f, which are related to Figure 8 When the QP value is 22, graphs 1101 and 1102 show the mean and standard deviation modeling of the pixel differences between the original data points (i.e., a set of second representative data points) and the codec reconstructed data points (i.e., a set of first representative data points). When the QP value is 32, graphs 1103 and 1104 show the mean and standard deviation modeling of the pixel differences between the original data points and the codec reconstructed data points. When the QP value is 42, graphs 1104 and 1105 show the mean and standard deviation modeling of the pixel differences between the original data points and the codec reconstructed data points.

[0115] Fig.12 A block diagram of a VVC decoder pipeline 1200 for AI-based media coding according to an embodiment of the present disclosure is shown.

[0116] The VVC decoder pipeline 1200 includes a plurality of modules. The plurality of modules may include a CABAC module 1201, an inverse quantization module 1202, an inverse transform module 1203, a LMCS 1204, an in-loop filter 1205, an intra-frame prediction module 1206, an inter-frame prediction module 1207, a CIIP module 1208, a LMCS 1209, a DPB 1210, and a reconstructed image module 1211. The CABAC module 1201, the inverse quantization module 1202, the inverse transform module 1203, the LMCS 1204, the in-loop filter 1205, the intra-frame prediction module 1206, the inter-frame prediction module 1207, the CIIP module 1208, the LMCS 1209, the DPB 1210, and the reconstructed image module 1211 are conventional modules.

[0117] The CABAC module 1201 may decompress the compressed bitstream (i.e., the bitstream information received from the VVC encoder pipeline 300) and reconstruct the original frame (i.e., the input video data):

[0118] a. Bitstream parsing: The CABAC module 1201 may receive a compressed bitstream and parse it to extract coded symbols and associated information.

[0119] b. Binary Arithmetic Decoding: The CABAC module 1201 may decode symbols from the compressed bitstream using binary arithmetic decoding. The CABAC module 1201 may assign probabilities to the decoded symbols and use the probabilities to reconstruct the original symbols associated with the encoded symbols.

[0120] c. Context Modeling Synchronization: The CABAC module 1201 may use context modeling to provide proper synchronization between the VVC encoder pipeline 300 and the VVC decoder pipeline 1200 .

[0121] In addition, the inverse quantization module 1202 and the inverse transform module 1203 are configured to convert the quantized and transformed video data back to its original form. The LMCS module 1204 is used to adjust the chrominance components and the luminance components of the video data to improve the overall image quality.

[0122] In addition, the in-loop filter 1205 may receive data from the LMCS module 1204. The in-loop filter 1205 is configured to remove artifacts and noise that may occur during the compression operation. Examples of the in-loop filter 1205 may include LMCS 1205a, DBF 1205b, SAO filter 1205c, ALF, and CC-ALF 1205d. The functions of the various blocks (1205a, 1205b, 1205c, and 1205d) are similar to Figure 2A , and these various blocks are conventional modules.

[0123] In addition, intra prediction module 1206 and inter prediction module 1207 are configured to predict the value of data based on spatial correlation and temporal correlation. CIIP module 1208 can combine both intra prediction technology and inter prediction technology to further improve compression efficiency. Finally, DPB 1210 and reconstructed image module can work together to store and display decoded video data. DPB 1210 can store previously decoded frames to achieve effective inter prediction, and reconstructed image module 1211 generates high-quality display images from decoded data (e.g., image frames from in-loop filter 1205).

[0124] In one or more embodiments of the present disclosure, the in-loop filter 1205 may include an AI-based in-loop filter 1205e, a data distribution model 1205f, a pixel mapping module 1205g for offset scaling, and a final AI-based in-loop filter output module 1205h.

[0125] In one or more embodiments of the present disclosure, the AI-based in-loop filter 1205e may be removed from the VVC encoder pipeline 300 (see Figure 3 ) receives bitstream information associated with the reconstructed video, wherein the bitstream information includes an offset value. The AI-based in-loop filter 1205e can perform various functions related to the AI-based in-loop filter 302e based on the received bitstream information.

[0126] In one or more embodiments of the present disclosure, the AI-based in-loop filter 1205e may receive (or identify) a reconstructed image frame associated with the reconstructed video from at least one of the LMCS 1204, LMCS 1205a, deblocking filter 1205b, SAO 1205c or ALF, CC-ALF 1205d. The AI-based in-loop filter 1205e may enhance the quality of the reconstructed image frame by removing artifacts from the reconstructed image frame.

[0127] In one or more embodiments of the present disclosure, the data distribution model 1205f may generate a model output data distribution for the reconstructed video. The data distribution model 1205f may generate a model output data distribution for the enhanced image frame from the AI-based in-loop filter 1205e.

[0128] In one or more embodiments of the present disclosure, the pixel mapping module 1205g for offset scaling can perform a scaling operation on each pixel value associated with each fragmented image frame of the reconstructed video by utilizing the offset value and the generated model output data distribution. In an embodiment of the present disclosure, the pixel mapping module 1205g for offset scaling can identify offset signaling from a bitstream and generate an offset map based on the offset signaling.

[0129] In one or more embodiments of the present disclosure, the final AI-based in-loop filter output module 1205h can generate an output video based on the performed scaling operation. In an embodiment of the present disclosure, the final AI-based in-loop filter output module 1205h can perform an offset scaling operation on the enhanced image frame from the AI-based in-loop filtering 1205e based on the generated offset map (or offset value, offset signaling).

[0130] Fig.13A FIG. 1 is a block diagram of an electronic device 100 for AI-based media encoding according to an embodiment of the present disclosure. In one or more embodiments of the present disclosure, the electronic device 100 may include Figure 3 Examples of the electronic device 100 may include, but are not limited to, a smart phone, a tablet computer, a personal digital assistant (PDA), an Internet of Things (IoT) device, a wearable device, and the like.

[0131] In an embodiment of the present disclosure, the electronic device 100 includes a system 101. The system 101 may include a memory 110, a processor 120, and a communicator 130.

[0132] In an embodiment of the present disclosure, the memory 110 stores instructions for AI-based media encoding as discussed throughout the present disclosure to be executed by the processor 120. The memory 110 may include a non-volatile storage element. Examples of such non-volatile storage elements may include a magnetic hard disk, an optical disk, a floppy disk, a flash memory, or an electrically programmable memory (EPROM) or an electrically erasable programmable (EEPROM) memory. In addition, in some examples, the memory 110 may be considered to be a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier or propagation signal. However, the term "non-transitory" should not be interpreted as the memory 110 being non-removable. In some examples, the memory 110 may be configured to store a larger amount of information than the memory. In some examples, the non-transitory storage medium may store data (e.g., in a random access memory (RAM) or a cache memory) that may change over time. The memory 110 may be an internal storage unit, or it may be an external storage unit of the electronic device 100, a cloud storage, or any other type of external storage.

[0133] The processor 120 communicates with the memory 110 and the communicator 130. The processor 120 is configured to execute instructions stored in the memory 110 and perform various processes for AI-based media encoding as discussed throughout the present disclosure. The processor 120 may include one or more processors, which may be a general-purpose processor (such as a central processing unit (CPU), an application processor (AP), etc.), a graphics processing unit (such as a graphics processing unit (GPU)), a visual processing unit (VPU), and / or an artificial intelligence (AI) dedicated processor (such as a neural processing unit (NPU)).

[0134] The communicator 130 is configured for internal communication between internal hardware components and for communication with external devices (eg, servers) via one or more networks (eg, radio technologies). The communicator 130 includes electronic circuits dedicated to implementing standards for wired or wireless communication.

[0135] Processor 120 is implemented by processing circuits such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, etc., and may optionally be driven by firmware. The circuits may, for example, be embodied in one or more semiconductor chips, or on a substrate support such as a printed circuit board.

[0136] In one embodiment of the present disclosure, the processor 120 includes an image compressor 121, an AI-based in-loop filter 122, a data distribution model 123, a real data distribution model 124, an RDO-controlled mapping range module 125, a pixel mapping module 126 for offset scaling, and an optimized media generator 127.

[0137] In one embodiment of the present disclosure, the image compressor 121 is associated with the compression module 301, the AI-based in-loop filter 122 is associated with the AI-based in-loop filter 302e, the data distribution model 123 is associated with the data distribution model 302f, the real data distribution model 124 is associated with the real data distribution model 302g, the RDO-controlled mapping range module 125 is associated with the RDO-controlled mapping context module 302h, the pixel mapping module 126 for offset scaling is associated with the pixel mapping module 302i for offset scaling, and the optimized media generator 127 is associated with the final AI-based in-loop filter output module 302j.

[0138] The image compressor 121 is a conventional module that can receive an input video. Upon receiving the input video, the image compressor 121 can perform one or more predefined compression operations, such as transform, quantization, dequantization, and inverse transform, on the received input video. The AI-based in-loop filter 122 can be configured to apply AI technology in the loop filtering process of video encoding. The AI-based in-loop filter 122 can enhance the performance of the in-loop filters 302a, 302b, 302c, and 302d. The AI-based in-loop filter 122 can use a machine learning model to learn the characteristics of video content (e.g., input video, etc.) and apply adaptive filtering technology. Instead of relying on predefined rules or heuristics, the AI-based in-loop filter 122 can use a neural network or other AI method to analyze and process video content. The AI-based in-loop filter 122 can use an AI model that can learn from a large amount of training data and make intelligent decisions about filtering operations. The AI-based in-loop filter 122 may generate one or more reconstructed image frames and send the one or more reconstructed image frames to the data distribution model 123 for further processing, as described in detail below.

[0139] The data distribution model 123 may generate a set of first representative data points, such as Figure 6 The data distribution model 123 may perform the steps given below to generate a set of first representative data points.

[0140] The data distribution model 123 may receive one or more reconstructed image frames (e.g., one or more enhanced image frames) from the AI-based in-loop filter 122, wherein the one or more reconstructed image frames are associated with the reconstructed video. Then, the data distribution model 123 may receive a user input indicating a desired compression level for each of the one or more reconstructed image frames. Then, the data distribution model 123 may determine the number of fragments of each of the one or more reconstructed image frames based on the received user input. Then, the data distribution model 123 may fragment each of the one or more reconstructed image frames based on the determined number of fragments. Then, the data distribution model 123 may use one or more distribution mechanisms to analyze the content changes within each fragmented image frame. Then, the data distribution model 123 may use one or more statistical mechanisms to perform a pixel merging operation on the analyzed content changes to determine the best fragmented one or more image frames, wherein each of the best fragmented one or more image frames includes a group of pixels having unique characteristics. Then, the data distribution model 123 may generate a group of first representative data points for each of the best fragmented one or more image frames.

[0141] The real data distribution model 124 can generate a set of second representative data points, such as Figure 7 The real data distribution model 124 may perform the steps given below to generate a set of second representative data points.

[0142] The real data distribution model 124 may receive one or more input image frames associated with an input video. Then, the real data distribution model 124 may receive user input indicating a desired compression level for each of the one or more input image frames. Then, the real data distribution model 124 may determine the number of segments of each of the one or more input image frames based on the received user input. Then, the real data distribution model 124 may fragment each of the one or more input image frames based on the determined number of segments. Then, the real data distribution model 124 may use one or more distribution mechanisms to analyze content changes within each fragmented input image frame. Then, the real data distribution model 124 may use one or more statistical mechanisms to perform a pixel merging operation on the analyzed content changes to determine the best fragmented one or more image frames, wherein each of the best fragmented one or more image frames includes a group of pixels having unique characteristics. Then, the real data distribution model 124 may generate a set of second representative data points for each of the best fragmented one or more image frames.

[0143] The RDO controlled mapping range module 125 may determine the total RD cost based on the output of the RD cost module 304 and the user input regarding the number of fragments, such as in combination with Figure 4 The RDO controlled mapping range module 125 may further construct an objective function to minimize the overall RD cost.

[0144] The pixel mapping module 126 for offset scaling may determine a data distribution dissimilarity metric between a set of first representative data points for each fragmented image frame associated with the reconstructed video and a set of second representative data points for each fragmented image frame associated with the input video, such as in combination with Figure 8 The pixel mapping module 126 for offset scaling may further determine the offset value for each fragmented image frame by utilizing a data distribution dissimilarity metric, such as in combination with Figure 8 as described.

[0145] The optimized media generator 127 may perform a scaling operation on each pixel value associated with each fragmented image frame using the determined offset value, such as in combination with Fig. 9 The optimized media generator 127 may further encode the reconstructed video based on the performed scaling operation.

[0146] Functions associated with various components of the electronic device 100 may be performed by non-volatile memory, volatile memory, and processor 120. One or more processors control the processing of input data according to predefined operating rules or AI models stored in non-volatile memory and volatile memory. Predefined operating rules or AI models are provided by training or learning. Here, providing by learning refers to an AI model that obtains predefined operating rules or desired characteristics by applying a learning algorithm to multiple learning data. The learning may be performed in the device itself in which the AI ​​according to the embodiment is executed, and / or may be implemented by a separate server / system. A learning algorithm is a method for training a predetermined target device (e.g., a robot) using multiple learning data to enable, allow, or control the target device to make a determination or prediction. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0147] The AI ​​model may include multiple neural network layers. Each layer has multiple weight values, and each layer performs layer operations by calculating the previous layer and operating on multiple weights. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.

[0148] Although Fig.13A Various hardware components of the electronic device 100 are shown, but it should be understood that one or more embodiments of the present disclosure are not limited thereto. In one or more embodiments of the present disclosure, the electronic device 100 may include fewer or more components. In addition, the labels or names of the components are for illustrative purposes only and do not limit the scope of the present invention. One or more components may be combined to perform the same or substantially similar functions for AI-based media encoding.

[0149] Fig. 13B FIG. 1 is a block diagram of an electronic device 1300 for AI-based media decoding according to an embodiment of the present disclosure. In one or more embodiments of the present disclosure, the electronic device 1300 includes Fig.12 Examples of the electronic device 1300 may include, but are not limited to, a smart phone, a tablet computer, a PDA, an IoT device, a wearable device, and the like.

[0150] In an embodiment, the electronic device 1300 includes a system 1301. The system 1301 may include a memory 1310, a processor 1320, and a communicator 1330.

[0151] In an embodiment, the memory 1310 stores instructions to be executed by the processor 1320 for AI-based media decoding as discussed throughout the present disclosure. The memory 1310 may include a non-volatile storage element. Examples of such non-volatile storage elements may include a magnetic hard disk, an optical disk, a floppy disk, a flash memory, or a form of EPROM or EEPROM memory. In addition, in some examples, the memory 1310 may be considered to be a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier or propagation signal. However, the term "non-transitory" should not be interpreted as the memory 1310 being non-removable. In some examples, the memory 1310 may be configured to store a larger amount of information than the memory. In some examples, the non-transitory storage medium may store data (e.g., in a RAM or cache memory) that may change over time. The memory 1310 may be an internal storage unit, or it may be an external storage unit of the electronic device 1300, a cloud storage, or any other type of external storage.

[0152] The processor 1320 communicates with the memory 1310 and the communicator 1330. The processor 1320 is configured to execute instructions stored in the memory 1310 and perform various processes for AI-based media encoding-decoding as discussed throughout the present disclosure. The processor 1320 may include one or more processors, which may be a general-purpose processor (such as a central processing unit (CPU), an application processor (AP), etc.), a graphics processing unit (such as a graphics processing unit (GPU)), a visual processing unit (VPU), and / or an artificial intelligence (AI) dedicated processor (such as a neural processing unit (NPU)).

[0153] The communicator 1330 is configured for internal communication between internal hardware components and for communication with external devices (eg, servers) via one or more networks (eg, radio technologies). The communicator 1330 includes electronic circuits dedicated to implementing standards for wired or wireless communication.

[0154] Processor 1320 is implemented by processing circuits such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, etc., and may optionally be driven by firmware. The circuits may, for example, be embodied in one or more semiconductor chips, or on a substrate support such as a printed circuit board.

[0155] In one embodiment of the present disclosure, the processor 1320 includes an AI-based in-loop filter 1322 , a data distribution model 1324 , a pixel mapping module 1326 for offset scaling, and an optimized media generator 1328 .

[0156] In one embodiment of the present disclosure, the AI-based in-loop filter 1322 is associated with the AI-based in-loop filter 1205e, the data distribution model 1324 is associated with the data distribution model 1205f, the pixel mapping module 1326 for offset scaling is associated with the pixel mapping module 1205g for offset scaling, and the optimized media generator 1328 is associated with the final AI-based in-loop filter output module 1205h.

[0157] The AI-based in-loop filter 1322 may be configured to apply AI technology in the loop filtering process of video decoding. The AI-based in-loop filter 1322 may enhance the performance of the in-loop filters 302a, 302b, 302c, and 302d. The AI-based in-loop filter 1322 may utilize a machine learning model to learn the characteristics of video content (e.g., input video, etc.) and apply adaptive filtering technology. Instead of relying on predefined rules or heuristics, the AI-based in-loop filter 1322 may use a neural network or other AI method to analyze and process video content. The AI-based in-loop filter 1322 may use an AI model that can learn from a large amount of training data and make intelligent decisions about filtering operations. The AI-based in-loop filter 1322 may generate one or more reconstructed image frames and send one or more reconstructed image frames to a data distribution model 1324 for further processing, as described in detail below.

[0158] The data distribution model 1324 can generate a set of first representative data points, such as a combination of Figure 6 The data distribution model 123 may perform the steps given below to generate a set of first representative data points.

[0159] The data distribution model 1324 may receive one or more reconstructed image frames (e.g., one or more enhanced image frames) from the AI-based in-loop filter 1322, wherein the one or more reconstructed image frames are associated with the reconstructed video. The data distribution model 1324 may then receive a user input indicating a desired compression level for each of the one or more reconstructed image frames. The data distribution model 1324 may then determine the number of segments of each of the one or more reconstructed image frames based on the received user input. The data distribution model 1324 may then fragment each of the one or more reconstructed image frames based on the determined number of segments. The data distribution model 1324 may then analyze content changes within each of the fragmented image frames using one or more distribution mechanisms. The data distribution model 1324 may then perform a pixel merging operation on the analyzed content changes using one or more statistical mechanisms to determine the best fragmented one or more image frames, wherein each of the best fragmented one or more image frames includes a pixel group having unique characteristics. The data distribution model 1324 may then generate a set of first representative data points for each of the optimally fragmented one or more image frames.

[0160] The pixel mapping module 1326 for offset scaling may perform pixel mapping for scaling operations based on offset information included in the bitstream information from the encoder, such as in conjunction with Figure 8 As described.

[0161] The optimized media generator 1328 may perform a scaling operation on the reconstructed image frame based on the pixel map to generate a scaled image frame. The optimized media generator 1328 may perform a scaling operation on each pixel value associated with each fragmented image frame by utilizing offset information (e.g., a value), such as in combination with Fig. 9 As described, the optimized media generator 1328 can generate output video based on the scaled image frames.

[0162] In addition, the optimized media generator 1328 may receive bitstream information associated with the reconstructed video from the encoder (i.e., the VVC encoder pipeline 300), wherein the bitstream information includes an offset value. The optimized media generator 1328 may generate a model output data distribution for the reconstructed video using the offset value using the data distribution model 1205f. The optimized media generator 1328 may then perform a scaling operation on each pixel value associated with each fragmented image frame of the reconstructed video using a pixel mapping module 1205g for offset scaling using the offset value and the generated model output data distribution. The optimized media generator 1328 may then generate an output video based on the performed scaling operation using a final AI-based in-loop filtering output module 1205h.

[0163] Functions associated with various components of the electronic device 100 may be performed by non-volatile memory, volatile memory, and processor 120. One or more processors control the processing of input data according to predefined operating rules or AI models stored in non-volatile memory and volatile memory. Predefined operating rules or AI models are provided by training or learning. Here, providing by learning refers to an AI model that obtains predefined operating rules or desired characteristics by applying a learning algorithm to multiple learning data. The learning may be performed in the device itself in which the AI ​​according to the embodiment is executed, and / or may be implemented by a separate server / system. A learning algorithm is a method for training a predetermined target device (e.g., a robot) using multiple learning data to enable, allow, or control the target device to make a determination or prediction. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0164] An AI model can contain multiple neural network layers. Each layer has multiple weight values, and each layer performs layer operations by calculating the previous layer and operating on multiple weights. Examples of neural networks include, but are not limited to, CNN, DNN, RNN, RBM, DBN, BRDNN, GAN, and Deep Q Network.

[0165] Although Fig. 13B Various hardware components of the electronic device 1300 are shown, but it should be understood that one or more embodiments of the present disclosure are not limited thereto. In one or more embodiments of the present disclosure, the electronic device 100 may include fewer or more components. In addition, the labels or names of the components are for illustrative purposes only and do not limit the scope of the present invention. One or more components may be combined to perform the same or substantially similar functions for AI-based media encoding-decoding.

[0166] Fig.14 is a flowchart illustrating a method 1400 for AI-based media encoding performed by an electronic device according to an embodiment of the present disclosure.

[0167] At step 1410, method 1400 includes compressing an input image associated with an input video. In an embodiment of the present disclosure, method 1400 may include compressing the input video to feed an AI-based in-loop filter, which operation may be performed in conjunction with Figure 3 The compression module 301, the in-loop filter 302 (for example, the LMCS 302a, the deblocking filter 302b, the SAO 302c, the ALF, the CC-ALF 302d), the CABAC module 303 or the RD cost module 304 are related to at least one of them.

[0168] At step 1420, the method 1400 includes generating a reconstructed image frame corresponding to the input image frame using the AI-based in-loop filter 302e, which operation can be similar to Figure 3 The output of the AI-based in-loop filter 302e is related.

[0169] At step 1430, the method 1400 includes generating an offset value based on the input image frame and the reconstructed image frame. In an embodiment of the present disclosure, the method 1400 may include generating a model output data distribution for the reconstructed video, which operation may be performed in conjunction with Figure 3 The method 1400 includes several processes for generating a model output data distribution for a reconstructed video, which are listed below. The method 1400 includes receiving one or more reconstructed image frames from an AI-based in-loop filter, wherein one or more reconstructed image frames are associated with a reconstructed video. The one or more reconstructed image frames are generated by utilizing one or more NN models. The method 1400 also includes receiving a user input indicating a desired compression level for each of the one or more reconstructed image frames. The method 1400 also includes determining the number of segments for each of the one or more reconstructed image frames based on the received user input. The method 1400 also includes fragmenting each of the one or more reconstructed image frames based on the determined number of segments. The method 1400 also includes analyzing content changes within each fragmented image frame using one or more distribution mechanisms. The method 1400 also includes performing a pixel merging operation on the analyzed content changes using one or more statistical mechanisms to determine the best fragmented one or more image frames, wherein each of the best fragmented one or more image frames includes a set of pixels having unique characteristics. The method 1400 also includes generating a set of first representative data points for each of the optimally fragmented one or more image frames.

[0170] In an embodiment of the present disclosure, method 1400 includes generating a true data distribution of an input video, which operation can be performed with Figure 3The method 1400 includes several processes for generating a real data distribution of an input video, which are listed below. The method 1400 includes receiving one or more input image frames associated with the input video. The method 1400 also includes receiving a user input indicating a desired compression level for each of the one or more input image frames. The method 1400 also includes determining the number of fragments of each of the one or more input image frames based on the received user input. The method 1400 also includes fragmenting each of the one or more input image frames based on the determined number of fragments. The method 1400 also includes using one or more distribution mechanisms to analyze content changes within each fragmented input image frame. The method 1400 also includes using one or more statistical mechanisms to perform a pixel merging operation on the analyzed content changes to determine the best fragmented one or more image frames, wherein each of the best fragmented one or more image frames includes a set of pixels having unique characteristics. The method 1400 also includes generating a set of second representative data points for each of the best fragmented one or more image frames.

[0171] In an embodiment of the present disclosure, method 1400 includes determining an offset value by comparing the model output data distribution with the true data distribution, which operation can be performed with Figure 3 The method 1400 includes several processes for determining offset values, which are listed below. The method 1400 includes determining a data distribution dissimilarity metric between a set of first representative data points for each fragmented image frame associated with the reconstructed video and a set of second representative data points for each fragmented image frame associated with the input video. The method 1400 also includes determining an offset value for each fragmented image frame by utilizing the data distribution dissimilarity metric.

[0172] At step 1440, the method 1400 includes encoding the reconstructed image frame based on the determined offset value. In an embodiment of the present disclosure, the method 1400 may include encoding the reconstructed video frame by utilizing the determined offset value, which operation may be similar to Figure 3The method 1400 includes performing a scaling operation on each pixel value associated with each fragmented image frame by using a determined offset value, wherein the scaling operation includes at least one of an addition operation, a multiplication operation, a division operation, or an exponential operation. The method 1400 also includes encoding the reconstructed video based on the scaling operation performed.

[0173] In an embodiment of the present invention, method 1400 includes sending bitstream information associated with the reconstructed video to a decoder, wherein the bitstream information includes the determined offset value.

[0174] Fig.15 1 is a flow chart illustrating a method 1500 for AI-based media compression according to an embodiment of the present disclosure. The method 1500 may be related to the method 1400.

[0175] At step 1510, method 1500 includes determining a desired compression level for input media, where the desired compression level is indicated by a user.

[0176] At step 1520, method 1500 includes compressing the input media based on the desired compression level, which may be performed in conjunction with Fig.14 Step 1410 is related to step 1420.

[0177] At step 1530, method 1500 includes determining an offset value in the compressed media relative to the input media by comparing the compressed media to the input media and a desired compression level for the input media, which operation may be similar to Fig.14 The offset value is calculated and used at a per-fragment granularity, wherein the number of fragments is received as input from a user, and wherein each fragment is calculated during a data distribution modeling operation. Method 1500 includes several processes for determining the offset value, which are listed below. Method 1500 includes modeling the model output data distribution. Method 1500 also includes modeling the real data distribution. Method 1500 also includes determining a pixel mapping using the modeled model output data distribution, the modeled real data distribution, and a mapping range. The mapping range is a decision to apply an offset scaling to a particular fragment based on an RD cost, wherein the mapping range is determined using the number of fragments defined by the user and the codec RD cost.

[0178] At step 1540, method 1500 includes obtaining optimized compressed media by adding the determined offset value to the compressed media, which operation may be performed in conjunction with Fig.14 Related to step 1440.

[0179] Fig.16 is a flow chart illustrating a method 1600 for AI-based media decoding according to an embodiment of the present disclosure.

[0180] At step 1610, the method 1600 includes receiving bitstream information including offset information from an encoder. In an embodiment of the present disclosure, the method 1600 may include receiving bitstream information associated with the reconstructed video from an encoder, wherein the bitstream information includes an offset value.

[0181] At step 1620, method 1600 includes generating a reconstructed image frame based on the bitstream information using an AI-based in-loop filter, which may be performed in conjunction with Fig.12 The method 1600 may include outputting data distribution of a model of a reconstructed video by using an offset value, which may be related to at least one of the CABAC 1201, the inverse quantization module 1202, the inverse transform module 1203, the LMCS module 1204, or the in-loop filter 1205. In an embodiment of the present disclosure, the method 1600 may include outputting data distribution of a model of a reconstructed video by using an offset value, which may be related to Fig.14 Related to step 1420.

[0182] At step 1630, method 1600 includes performing a scaling operation on the reconstructed image frame based on the offset information to generate a scaled image frame. In an embodiment of the present disclosure, method 1600 may include performing a scaling operation on each pixel value associated with each fragmented image frame of the reconstructed video by using the offset value and the generated model output data distribution, which may be performed with Fig.14 At least one of step 1420 or step 1430. The scaling operation includes at least one of an addition operation, a multiplication operation, a division operation, or an exponential operation.

[0183] At step 1640, the method 1600 includes generating an output video based on the scaled image frame, which operation may be performed with an in-loop filter (e.g., LMCS 1205a, deblocking filter 1205b, SAO 1205c, ALF, CC-ALF 1205d) or Fig.12 The method 1600 may be related to at least one of the reconstructed image modules 1211. In an embodiment of the present disclosure, the method 1600 may include generating an output video based on the performed scaling operation.

[0184] In one or more embodiments of the present disclosure, the bitstream information may include several adjustable parameters. The parameters may include an adjustable enhancement flag, an adjustable category number, and an adjustable category offset for offset scaling. For example, the adjustable enhancement flag is a binary value. When the adjustable enhancement flag is set to 0, it indicates that the bitstream information does not contain syntax elements related to the adjustable offset. Therefore, these syntax elements may not be used for reconstruction processing (see Fig.12 ). The adjustable number of categories is another parameter that can be adjusted based on the user's requirements to determine the number of categories to be considered during processing (e.g., decoding). In addition, the adjustable category offset parameter specifies a specific offset to be applied to each pixel category. These offsets can modify the characteristics of pixels within each category. By adjusting these parameters within the bitstream information, the disclosed method can adapt to different scenarios and user preferences, thereby allowing customizable and flexible processing of data.

[0185] In one or more embodiments of the present disclosure, the disclosed method provides a unique strategy for AI-based models that utilizes additional real-time modules to adapt to updated and unseen data to reduce data domain gaps, such as in combination with Figures 3 to 16 as described. The disclosed method uses pixel mapping and content-based scaling to bridge the data domain gap between updated / unseen data and training data in an AI-based in-loop filter. Therefore, the disclosed method is superior to conventional methods that target specific artifacts and train frequently using limited data, resulting in inherent bias and volatility in AI-based training models. The disclosed method can implement content-based scaling to identify pixel mappings for scaling offset values, operate in real time, do not require pre-training, and can provide greater generalization for different types of data. In addition, the complexity of the disclosed method is very low when compared to conventional methods, because conventional methods deploy very complex DNN models to achieve higher gains, but they are not adapted on a content-to-content basis. The disclosed method solves this problem by introducing statistical regression based on video data during the encoding process and using statistical regression based on video data during the decoding process.

[0186] Unlike conventional methods that target specific artifacts and rely on limited data, the disclosed method uses content-based scaling to bridge data domain gaps and provide greater generalization for different types of data. One of the key advantages of the disclosed method is its ability to identify pixel mappings for scaling offsets without pre-training. This allows instant adjustment, which is critical in a dynamic environment where data is constantly changing. In addition, compared with conventional methods that rely on complex DNN models to achieve higher gains, the disclosed method is much lower in complexity. The disclosed method also solves the problems of bias and volatility in AI-based training models by introducing statistical regression based on video data during encoding and decoding processes. This ensures that the disclosed method is adaptive on a content-to-content basis and provides more accurate predictions. In general, the disclosed method provides a powerful tool for improving AI-based models and solving challenges brought by updated and unseen data. Its ability to perform content-based scaling, bridge data domain gaps, and provide greater generalization makes the electronic device 100 an important tool for organizations seeking to utilize AI technology in many operations (e.g., encoding, decoding, etc.).

[0187] In one or more embodiments of the present disclosure, the disclosed method has the potential to enhance the efficiency of encoders and decoders in the electronic device 100, thereby reducing video recording and memory requirements in the electronic device 100. In addition, the disclosed method helps improve video quality even in bandwidth-constrained scenarios. The disclosed method can significantly improve the overall user experience and satisfaction. Its benefits are particularly important in today's world, where video content consumption has become the norm and bandwidth limitations are prevalent.

[0188] In one or more embodiments of the present disclosure, the disclosed method can enhance the Bjontegaard delta rate (BD rate). The BD rate gain measures the overall compression efficiency of the entire pipeline. In addition, the disclosed method improves visual quality by reducing distortion while keeping bandwidth requirements constant. In addition, the disclosed method can also reduce bitstream or bandwidth requirements while maintaining the same visual quality. In addition, the disclosed method can improve visual quality and also reduce bandwidth requirements. These features make the disclosed method a valuable asset in improving video streaming services.

[0189] In an embodiment of the present disclosure, a method for AI-based media encoding may include compressing an input image frame associated with an input video. In an embodiment of the present disclosure, the method may include using an AI-based in-loop filter to generate a reconstructed image frame corresponding to the input image frame. In an embodiment of the present disclosure, the method may include determining an offset value based on the input image frame and the reconstructed image frame. In an embodiment of the present disclosure, the method may include encoding the reconstructed image frame based on the determined offset value.

[0190] In an embodiment of the present disclosure, the step of determining the offset value based on the input image frame and the reconstructed image frame may include generating a model output data distribution for the reconstructed image frame. In an embodiment of the present disclosure, the step of determining the offset value based on the input image frame and the reconstructed image frame may include generating a real data distribution for the input image frame. In an embodiment of the present disclosure, the step of determining the offset value based on the input image frame and the reconstructed image frame may include determining the offset value by comparing the model output data distribution with the real data distribution.

[0191] In an embodiment of the present disclosure, the step of generating a model output data distribution for the reconstructed image frame may include identifying the number of fragments for the reconstructed image frame based on user input. In an embodiment of the present disclosure, the step of generating a model output data distribution for the reconstructed image frame may include analyzing content changes within the reconstructed image frame using one or more distribution mechanisms. In an embodiment of the present disclosure, the step of generating a model output data distribution for the reconstructed image frame may include fragmenting the reconstructed image frame based on the determined number of fragments and the analyzed content changes to generate an optimally fragmented image frame. In an embodiment of the present disclosure, the step of generating a model output data distribution for the reconstructed image frame may include performing a pixel merging operation on one or more fragments of the optimally fragmented image frame using one or more statistical mechanisms. In an embodiment of the present disclosure, the optimally fragmented image frame may include a group of pixels having unique characteristics. In an embodiment of the present disclosure, the step of generating a model output data distribution for the reconstructed image frame may include generating a group of first representative data points for the optimally fragmented image frame based on the pixel merging operation.

[0192] In an embodiment of the present disclosure, the step of generating a true data distribution for the input image frame may include identifying the number of fragments for the input image frame based on user input. In an embodiment of the present disclosure, the step of generating a true data distribution for the input image frame may include analyzing content changes within the input image frame using one or more distribution mechanisms. In an embodiment of the present disclosure, the step of generating a true data distribution for the input image frame may include fragmenting the input image frame based on the determined number of fragments and the analyzed content changes to generate an optimally fragmented input image frame. In an embodiment of the present disclosure, the step of generating a true data distribution for the input image frame may include performing a pixel merging operation on one or more fragments of the optimally fragmented input image frame using one or more statistical mechanisms. In an embodiment of the present disclosure, the optimally fragmented input image frame may include a group of pixels having unique characteristics. In an embodiment of the present disclosure, the step of generating a true data distribution for the input image frame may include generating a set of second representative data points for the optimally fragmented input image frame based on the pixel merging operation.

[0193] In an embodiment of the present disclosure, the step of determining an offset value by comparing the model output data distribution and the true data distribution may include determining a data distribution dissimilarity measure between a set of first representative data points for a fragmented image frame from the reconstructed image frame and a set of second representative data points for a fragmented input image frame from the input image frame. In an embodiment of the present disclosure, the step of determining an offset value by comparing the model output data distribution and the true data distribution may include determining the offset value based on the data distribution dissimilarity measure.

[0194] In an embodiment of the present disclosure, the step of determining the offset value by comparing the model output data distribution and the true data distribution may include determining a pixel map using the model output data distribution, the true data distribution and a mapping range in the form of a per-fragment offset.

[0195] In an embodiment of the present disclosure, the mapping range may be a decision to apply offset scaling to a specific segment based on RD cost. In an embodiment of the present disclosure, the mapping range may be determined based on the number of segments and the codec RD cost.

[0196] In an embodiment of the present disclosure, the step of encoding the reconstructed image frame may include performing a scaling operation on the reconstructed image frame based on the determined offset value to generate a scaled image frame. In an embodiment of the present disclosure, the scaling operation may include at least one of an addition operation, a multiplication operation, a division operation, or an exponential operation. In an embodiment of the present disclosure, the step of encoding the reconstructed image frame may include encoding the scaled image frame.

[0197] In an embodiment of the present disclosure, the reconstructed image frame may be generated by utilizing one or more neural network (NN) models of an AI-based in-loop filter.

[0198] In an embodiment of the present disclosure, the method may include sending bitstream information associated with the reconstructed image frame to a decoder. In an embodiment of the present disclosure, the bitstream information may include the determined offset value.

[0199] In an embodiment of the present disclosure, the offset value may be calculated and used at a per-fragment granularity.

[0200] In an embodiment of the present disclosure, a method for AI-based media decoding may include receiving bitstream information including offset information from an encoder. In an embodiment of the present disclosure, the method may include: using an AI-based in-loop filter to generate a reconstructed image frame based on the bitstream information. In an embodiment of the present disclosure, the method may include performing a scaling operation on the reconstructed image frame based on the offset information to generate a scaled image frame. In an embodiment of the present disclosure, the method may include generating an output video based on the scaled image frame.

[0201] In an embodiment of the present disclosure, the step of performing a scaling operation on the reconstructed image frame may include generating a model output data distribution for the reconstructed image frame. In an embodiment of the present disclosure, the step of performing a scaling operation on the reconstructed image frame may include performing pixel mapping for the scaling operation based on the offset information and the model output data distribution. In an embodiment of the present disclosure, the step of performing a scaling operation on the reconstructed image frame may include performing the scaling operation on the reconstructed image frame based on the pixel mapping to generate a scaled image frame.

[0202] In an embodiment of the present disclosure, the scaling operation may include at least one of an addition operation, a multiplication operation, a division operation, or an exponential operation.

[0203] A system for AI-based media encoding may include a processor operably connected to a memory and a communicator. In an embodiment of the present disclosure, the processor may be configured to compress an input image frame associated with an input video. In an embodiment of the present disclosure, the processor may be configured to generate a reconstructed image frame corresponding to the input image frame using an AI-based in-loop filter. In an embodiment of the present disclosure, the processor may be configured to determine an offset value based on the input image frame and the reconstructed image frame. In an embodiment of the present disclosure, the processor may be configured to encode the reconstructed image frame based on the determined offset value.

[0204] In an embodiment of the present disclosure, the processor may be configured to generate a model output data distribution for the reconstructed image frame. In an embodiment of the present disclosure, the processor may be configured to generate a real data distribution for the input image frame. In an embodiment of the present disclosure, the processor may be configured to determine the offset value by comparing the model output data distribution with the real data distribution.

[0205] In an embodiment of the present disclosure, the processor may be configured to identify the number of fragments for the reconstructed image frame based on user input. In an embodiment of the present disclosure, the processor may be configured to analyze the content changes within the reconstructed image frame using one or more distribution mechanisms. In an embodiment of the present disclosure, the processor may be configured to fragment the reconstructed image frame based on the determined number of fragments and the analyzed content changes to generate an optimally fragmented image frame. In an embodiment of the present disclosure, the processor may be configured to perform a pixel merging operation on one or more fragments of the optimally fragmented image frame using one or more statistical mechanisms. In an embodiment of the present disclosure, the optimally fragmented image frame may include a group of pixels having unique characteristics. In an embodiment of the present disclosure, the processor may be configured to generate a set of first representative data points for the optimally fragmented image frame based on the pixel merging operation.

[0206] In an embodiment of the present disclosure, the processor may be configured to identify the number of fragments for the input image frame based on user input. In an embodiment of the present disclosure, the processor may be configured to analyze the content changes within the input image frame using one or more distribution mechanisms. In an embodiment of the present disclosure, the processor may be configured to fragment the input image frame based on the determined number of fragments and the analyzed content changes to generate an optimally fragmented input image frame. In an embodiment of the present disclosure, the processor may be configured to perform a pixel merging operation on one or more fragments of the optimally fragmented input image frame using one or more statistical mechanisms. In an embodiment of the present disclosure, the optimally fragmented input image frame may include a group of pixels having unique characteristics. In an embodiment of the present disclosure, the processor may be configured to generate a set of second representative data points for the optimally fragmented input image frame based on the pixel merging operation.

[0207] In an embodiment of the present disclosure, the processor may be configured to determine a data distribution dissimilarity metric between a set of first representative data points for a fragmented image frame from the reconstructed image frame and a set of second representative data points for a fragmented input image frame from the input image frame. In an embodiment of the present disclosure, the processor may be configured to determine the offset value based on the data distribution dissimilarity metric.

[0208] In an embodiment of the present disclosure, the processor may be configured to determine a pixel mapping using the model output data distribution, the true data distribution, and a mapping range in the form of a per-fragment offset.

[0209] In an embodiment of the present disclosure, the mapping range may be a decision to apply offset scaling to a specific segment based on RD cost. In an embodiment of the present disclosure, the mapping range may be determined based on the number of segments and the codec RD cost.

[0210] In an embodiment of the present disclosure, the processor may be configured to perform a scaling operation on the reconstructed image frame based on the determined offset value to generate a scaled image frame. In an embodiment of the present disclosure, the scaling operation may include at least one of an addition operation, a multiplication operation, a division operation, or an exponential operation. In an embodiment of the present disclosure, the processor may be configured to encode the scaled image frame.

[0211] In an embodiment of the present disclosure, the reconstructed image frame may be generated by utilizing one or more neural network (NN) models of an AI-based in-loop filter.

[0212] In an embodiment of the present disclosure, the processor may be configured to send bitstream information associated with the reconstructed image frame to a decoder. In an embodiment of the present disclosure, the bitstream information may include the determined offset value.

[0213] In an embodiment of the present disclosure, the offset value may be calculated and used at a per-fragment granularity.

[0214] A system for AI-based media decoding may include a processor operably connected to a memory and a communicator. In an embodiment of the present disclosure, the processor may be configured to receive bitstream information including offset information from an encoder. In an embodiment of the present disclosure, the processor may be configured to generate a reconstructed image frame based on the bitstream information using an AI-based in-loop filter. In an embodiment of the present disclosure, the processor may be configured to perform a scaling operation on the reconstructed image frame based on the offset information to generate a scaled image frame. In an embodiment of the present disclosure, the processor may be configured to generate an output video based on the scaled image frame.

[0215] In an embodiment of the present disclosure, the processor may be configured to generate a model output data distribution for the reconstructed image frame. In an embodiment of the present disclosure, the processor may be configured to perform pixel mapping for the scaling operation based on the offset information and the model output data distribution. In an embodiment of the present disclosure, the processor may be configured to perform the scaling operation on the reconstructed image frame based on the pixel mapping to generate a scaled image frame.

[0216] In an embodiment of the present disclosure, the scaling operation may include at least one of an addition operation, a multiplication operation, a division operation, or an exponential operation.

[0217] According to an embodiment of the present disclosure, one or more operations described as being performed on an image frame may be performed on a video including an image frame, and similarly, one or more operations described as being performed on a video may be performed on an image frame included in the video.

[0218] The various actions, behaviors, blocks, steps, etc. in the flow chart can be performed in the order presented, in a different order, or simultaneously. In addition, in one or more embodiments of the present disclosure, some of the actions, behaviors, blocks, steps, etc. may be omitted, added, modified, skipped, etc. without departing from the scope of the present invention.

[0219] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the invention belongs.The systems, methods, and examples provided herein are illustrative only and not limiting.

[0220] Although specific language has been used to describe the subject matter, it is not intended that any limitation will be created thereby. It is apparent to those skilled in the art that various work modifications may be made to the method to realize the inventive concept taught herein. The accompanying drawings and the foregoing description provide examples of embodiments. It will be appreciated by those skilled in the art that one or more of the elements described may be well combined into a single functional element. Alternatively, some elements may be divided into a plurality of functional elements. Elements from one embodiment may be added to another embodiment.

[0221] The embodiments disclosed herein may be implemented using at least one hardware device and performing a network management function to control elements.

[0222] The foregoing description of specific embodiments will so fully reveal the general nature of the embodiments herein that others can easily modify and / or adapt various applications of such specific embodiments by applying current knowledge without departing from the general concepts, and therefore, such adaptations and modifications should and are intended to be understood as being within the meaning and scope of equivalents of the disclosed embodiments. It should be understood that the wording or terminology employed herein is for descriptive and not limiting purposes. Therefore, although the embodiments herein have been described according to preferred embodiments, those skilled in the art will recognize that the embodiments herein can be practiced by modification within the scope of the embodiments as described herein.

Claims

1. A method (1400) for artificial intelligence (AI)-based media encoding performed by an electronic device, the method (1400) comprising: compressing input image frames associated with an input video (1410); generating (1420) a reconstructed image frame corresponding to the input image frame using an AI-based in-loop filter (302e); determining (1430) an offset value based on the input image frame and the reconstructed image frame; as well as The reconstructed image frame is encoded based on the determined offset value (1440).

2. The method (1400) of claim 1, the step of determining (1430) an offset value based on the input image frame and the reconstructed image frame comprising: generating a model output data distribution for the reconstructed image frame; generating a true data distribution for the input image frame; as well as The offset value is determined by comparing the model output data distribution and the true data distribution.

3. The method (1400) of claim 2, wherein: The step of generating a model output data distribution for the reconstructed image frame comprises: identifying a number of segments for the reconstructed image frame based on a user input; analyzing content variations within the reconstructed image frame using one or more distribution mechanisms; segmenting the reconstructed image frame based on the determined number of segments and the analyzed content variation to generate an optimally segmented image frame; performing a pixel binning operation on one or more fragments of the best fragmented image frame using one or more statistical mechanisms, wherein the best fragmented image frame includes a set of pixels having unique characteristics; and A set of first representative data points for the optimally fragmented image frame is generated based on the pixel binning operation.

4. The method (1400) according to claim 2 or 3, wherein: The step of generating a true data distribution for the input image frame comprises: identifying a number of segments for the input image frame based on a user input; analyzing content changes within the input image frame using one or more distribution mechanisms; Segmenting the input image frame based on the determined number of segments and the analyzed content variation to generate an optimally segmented input image frame; performing a pixel merging operation on one or more fragments of the best fragmented input image frame using one or more statistical mechanisms, wherein the best fragmented input image frame includes a set of pixels having unique characteristics; and A set of second representative data points for the optimally segmented input image frame is generated based on the pixel binning operation.

5. The method (1400) according to any one of claims 2 to 4, wherein: The step of determining the offset value by comparing the model output data distribution with the real data distribution comprises: determining a data distribution dissimilarity measure between a first set of representative data points for a fragmented image frame from the reconstructed image frame and a second set of representative data points for a fragmented input image frame from the input image frame; and The offset value is determined based on the data distribution dissimilarity measure.

6. The method (1400) according to any one of claims 2 to 5, wherein: The step of determining the offset value by comparing the model output data distribution with the real data distribution comprises: A pixel map is determined using the model output data distribution, the true data distribution, and a mapping range in the form of per-fragment offsets.

7. The method (1400) according to any one of claims 6, wherein: The mapping range is a decision to apply the offset scaling to a specific segment based on RD cost, and wherein the mapping range is determined based on the number of segments and codec RD cost.

8. The method (1400) according to any one of claims 1 to 7, wherein: The step of encoding the reconstructed image frame (1440) includes: performing a scaling operation on the reconstructed image frame based on the determined offset value to generate a scaled image frame, wherein the scaling operation includes at least one of an addition operation, a multiplication operation, a division operation, or an exponential operation; and The scaled image frame is encoded.

9. The method (1400) according to any one of claims 1 to 8, wherein: The reconstructed image frame is generated by utilizing one or more neural network (NN) models of the AI-based in-loop filter.

10. The method (1400) according to any one of claims 1 to 9, comprising: Bitstream information associated with the reconstructed image frame is sent to a decoder, wherein the bitstream information includes the determined offset value.

11. The method (1400) according to any one of claims 1 to 10, wherein: The offset value is calculated and used at a per-fragment granularity.

12. A method (1600) for artificial intelligence (AI)-based media decoding performed by an electronic device, the method (1600) comprising: Receiving (1610) bitstream information including offset information from an encoder; generating (1620) a reconstructed image frame based on the bitstream information using an AI-based in-loop filter; performing (1630) a scaling operation on the reconstructed image frame based on the offset information to generate a scaled image frame; as well as An output video is generated (1640) based on the scaled image frames.

13. The method (1600) of claim 12, the step of performing (1630) a scaling operation on the reconstructed image frame comprising: generating a model output data distribution for the reconstructed image frame; performing pixel mapping for the scaling operation based on the offset information and the model output data distribution; as well as The scaling operation is performed on the reconstructed image frame based on the pixel map to generate a scaled image frame.

14. The method (1600) according to claim 12 or 13, wherein: The scaling operation includes at least one of an addition operation, a multiplication operation, a division operation, or an exponential operation.

15. A system (101) for artificial intelligence (AI) based media coding, wherein: The system (101) comprises: A processor (120), operatively connected to the memory (110) and the communicator (130), The processor (120) is configured to: compressing input image frames associated with an input video; Using an AI-based in-loop filter to generate a reconstructed image frame corresponding to the input image frame; determining an offset value based on the input image frame and the reconstructed image frame; and The reconstructed image frame is encoded based on the determined offset value.

Images