System and method for video encoding and transmission considering grain.

Abstract

An end-to-end system is provided that optimizes the encoding and transmission of video assets while considering grain. Grain model parameters are determined that indicate the type of grain present in the video asset. Prior to encoding the video asset, a grain reduction process is performed on the video asset according to grain reduction processing parameters that indicate how to remove grain from the video asset. After the grain reduction process, the video asset is encoded using grain-aware encoding / rate control according to video encoding / rate control parameters. The video asset is streamed and decoded. After decoding, grain synthesis is performed using grain synthesis parameters that indicate the characteristics of the grain to be added to the video asset, and the grain is reapplied to the video asset.

Description

Technical Field

[0001] ·Reference to Related Applications This application claims priority to U.S. Provisional Application No. 63 / 496,082, filed on April 14, 2023, the disclosure of which is incorporated herein by reference in its entirety. Aspects of the present disclosure relate to video encoding and transmission considering grain, and optimization between grain processing and video encoding.

Background Art

[0002] Film grain is a specific random noise within a video. Film grain may originate from the analog film shooting process or may be intentionally digitally synthesized during content production and post-production processes. In either case, the visual characteristics of the grain may be considered part of the artistic and creative intent of the content creator. Standard video encoders do not distinguish film grain from other types of noise and consume excessive bits for encoding the grain, making it very costly to preserve such creative intent in video delivery.

Summary of the Invention

[0003] As an example, an optimized encoding and transmission method for the grain of a video asset includes determining grain model parameters indicating the characteristics of the grain present in the video asset. Before encoding the video asset, it includes performing grain removal according to grain removal parameters (indicating the characteristics of grain removal from the video asset). After the grain reduction process, using grain-aware encoding / rate control, the video asset is encoded according to video encoding / rate control parameters; the video asset is streamed and decoded; after decoding, using grain synthesis parameters indicating the characteristics of the grain to be added to the video asset, grain synthesis is performed to re-impart grain to the video asset.

[0004] In one or more exemplary embodiments, the Disclosure also provides an end-to-end system for performing the above method. The system comprises one or more computing devices and is configured to determine the grain model parameters, perform a grain removal process based on grain removal parameters before encoding a video asset, then encode the video asset based on grain-aware encoding / rate control, stream and decode the encoded video asset, and after decoding, regrainize based on grain synthesis parameters to produce a video asset with re-grained.

[0005] In this specification, the terms "grain" and "film grain" are synonymous and both refer to image artifacts that have or are based on grain characteristics. Furthermore, the term "grain-aware" refers to operations performed within an end-to-end system that are configured to process grain information or are optimized for processing grain information. [Brief explanation of the drawing]

[0006] [Figure 1A] Figure 1A shows an example of an end-to-end system for video encoding and transmission that takes grain into consideration. [Figure 1B] Figure 1B shows an example of operation in an end-to-end system for video encoding and transmission that takes grain into consideration. [Figure 2] Figure 2 shows an example of data flow in the video encoding and transmission process that takes grain into consideration. [Figure 3] Figure 3 shows an example of the data flow for the grain and video quality evaluation process in video encoding and transmission that takes grain into consideration. [Figure 4]Figure 4 shows an example of the data flow of the grain removal process in video encoding and transmission that takes grain into consideration. [Figure 5] Figure 5 shows an example of the data flow for a video encoding and rate control process that takes grain into consideration. [Figure 6] Figure 6 shows an example of the data flow for a combined grain removal and encoding process that takes grain into consideration. [Figure 7] Figure 7 shows an example of the data flow for a video streaming and decoding process that takes grain into consideration. [Figure 8] Figure 8 shows an example of the data flow for the grain synthesis (re-graining) process. [Figure 9] Figure 9 shows an example of the data flow of the rendering and display process that takes grain into consideration. [Figure 10] Figure 10 shows an example of the data flow for a video quality evaluation process that takes grain into consideration, generating grain quality / fidelity evaluation, structural fidelity evaluation, and overall video quality evaluation. [Figure 11] Figure 11 shows an example of a computing device used in a video encoding and transmission system that takes grain into consideration. [Modes for carrying out the invention]

[0007] Detailed embodiments for carrying out the present invention are described below. However, it should be understood that the embodiments disclosed are merely illustrative of the present invention, and that the present invention may take various alternative forms. The drawings are not necessarily to scale, and features may be exaggerated or simplified in order to clearly show certain components. Accordingly, the specific structural and functional details disclosed herein should not be interpreted as limiting, but rather as representative grounds for enabling those skilled in the art to apply the embodiments of the present invention in various ways. As those skilled in the art will understand, various features shown in any one of the figures can be combined with features shown in other figures to constitute embodiments not explicitly shown. The combinations of features shown provide representative embodiments suitable for typical applications. However, various combinations and modifications of features based on the teachings of this disclosure may be selected depending on the specific application.

[0008] The aspects of this disclosure generally relate to visual communication applications such as video-on-demand, live video broadcasting, video conferencing, and online video games, where video assets containing analog or composite film grain are transmitted over a communication network. The presence of film grain noise in source content can require excessively large bandwidth for encoding and transmitting the video stream, otherwise resulting in perceptually unpleasant artifacts and significantly degrading the end user's perceived quality of experience (QoE). To address these issues, this disclosure relates to a system and method for performing and optimizing one or more steps in a grain-aware video transmission pipeline to reduce bandwidth requirements while maintaining and improving the end user's visual QoE.

[0009] The perceptual QoE of digital image content without film grain may relate to the perceived quality of image object features within the image content. However, image content with film grain has a psychovisual perceptual experience that differs from the perceptual quality based solely on image features. The perceptual experience of an image with grain cannot be described in the same way, with the same parameters, or with the same modeling as the perceptual experience related to image features in image content without film grain. Therefore, the perceptual QoE of an image with film grain (overall image QoE) consists of a QoE element related to the image object characteristics of the image content and another element related to the effect of film grain on the image content (grain QoE). Quality assessment based on film grain may differ from quality assessment based solely on image object characteristics.

[0010] This disclosure relates to a system and method for performing and optimizing one or more steps in an end-to-end system pipeline called grain-aware video transmission, which reduces bandwidth requirements by delivering composite film-grained video content to an end viewer while providing the end viewer with a perceptual QoE similar to that of viewing original video content with film grain, in order to address issues related to video assets that are source video with film grain. Grain-aware video transmission is configured to optimize the processing and delivery to the end viewer of video assets that include film grain. This refers to a video transmission system.

[0011] Over the past few decades, video-on-demand, live video broadcasting, video conferencing, and online gaming services have experienced phenomenal growth. Numerous studies have shown that consumers have continuously raised their expectations for better visual quality of experience (QoE), and major industry competitors have strived to maintain and improve the QoE of their audiences in their services and products. However, improving audience QoE often requires compressing video content to fit limited network capacity, significantly increasing bandwidth costs.

[0012] This problem is exacerbated when the source content contains film grain. Film grain is frequently found in high-quality video content, such as works by renowned filmmakers. This grain may originate from the analog film shooting process or it may be intentionally digitally composited during content creation or post-production. In either case, the visual texture of the grain is considered part of the artistic and creative intent of the video content creator.

[0013] Standard video encoders do not distinguish film grain from other noise and consume excessive bits to encode grain, making it extremely costly to preserve such creative intent in video distribution. As a result, when total bandwidth is limited, the number of bits available to encode other important visual information in the video is drastically reduced. This leads to perceptually annoying artifacts such as blur, macroblocking, floating, and mosquito noise, significantly degrading the end user's perceived quality of experience (QoE).

[0014] This disclosure relates to a solution that creates a perceptual visual effect similar to that of viewing an original video with film grain by applying a grain reduction process to the video before encoding and then synthesizing and re-adding the grain after decoding.

[0015] Methods exist designed for perceptual image quality evaluation and perception-motivated image encoding. These methods are typically developed under the assumption that there is a pure, original image free of noise and grain. Furthermore, they are often trained and tested on source video content that contains no noise / grain, or only negligible noise / grain. As a result, when applied to video content containing film grain, these methods often suffer significant performance degradation. Much research has also been conducted on texture modeling, texture synthesis, and image quality evaluation that considers texture. These studies are relevant when grain is considered a specific type of texture. Prior methods also exist for film grain noise reduction, grain synthesis, and their applications before and after image encoding. However, existing grain modeling, reduction, and synthesis methods are designed independently without considering encoder characteristics, the bitrate used by the encoder, and other important parameters. To address these shortcomings, this disclosure relates to a systematic, interactive, and co-optimization of grain processing and image encoding.

[0016] Figure 1A shows an example of an end-to-end system 100, which is a distribution chain including video coding that takes grain into account for transmission by a content distribution network. In the illustrated example, the video distribution chain receives a video asset 102, which may have film grain.

[0017] The video asset 102 includes, for example, live footage of current news, pre-recorded programs and movies, advertisements inserted into other video feeds, and other clips. In some cases, the video asset 102 may include only video, but in many cases, the video asset 102 further includes additional content such as audio, subtitles, metadata information describing the content and / or format of the video, and the like. The end-to-end system 100 includes the source of one or more video assets 102 and can be configured to function in cooperation with them. Generally, when a video distributor receives the source video, the distributor transmits the video asset 102 through an encoder, a transcoder, a packaging device, an origin server, a network connection, and a consumer device, and finally presents the video content to the end user.

[0018] The pre-delivery processing device 104 may be a software module that receives the video asset 102, or a unit that is a device or assembly. The pre-delivery processing device can determine aspects of the video asset 102 and be configured to prepare the video asset 102 for video encoding considering grain. Aspects of the video include, for example, detection of grain in the video asset 102, evaluation or modeling of grain in the video asset 102, reduction of grain in the video asset 102, and provision of grain parameters related to film grain in the video asset 102. Film grain refers to noise randomly distributed across the frames of the video asset 102, and this noise is characterized by various parameters such as grain size, grain density, and / or grain contrast. Video encoding considering grain refers to a process of adding film grain to the video asset and more efficiently performing the encoding of the video asset or the delivery of the encoded video asset to the content delivery network. The pre-delivery processing device can output the encoded video asset 114 for transmission by the content delivery network 106, or can output the encoded video asset for transmission by the content delivery network through external encoding.

[0019] The content distribution network 106 may be configured to transmit encoded video assets 102 to viewer devices 110. Configuration of the content distribution network may include, for example, receiving encoded video assets 114 from a pre-distribution processing unit 104, compressing and / or re-encoding the video content using one or more encoders or transcoders, and transmitting it in a format compliant with one or more standard video compression specifications. The content distribution network 106 may further use one or more packaging devices and / or origins to create segmented video files to be delivered to clients, which then combine the segments to form a continuous video stream. The content distribution network may also be configured to perform the above operations while being aware of the granularity of the content that the pre-distribution processing unit outputs to the content distribution network.

[0020] Content delivery network 106 provides the delivered and encoded video asset 116 to the post-delivery processing device 108. The post-delivery processing device 108 can be a software module, or a unit that is a device or assembly. The post-delivery processing device 108 can be configured to perform one aspect of evaluating the characteristics and restoring the grain in the delivered and encoded video asset. Aspects of the characteristic evaluation include, for example, identifying the location and parameters of the grain, the parameters used for film grain removal, the parameters used for compressing the video asset 102 after grain removal, the parameters related to the transfer of the video asset 114 encoded in a compressed format, the parameters used for decoding to restore the delivered and encoded video asset 116, the parameters generated in the synthesis and / or re-graining of the grain for output generation to the viewer device 110, and the like. The viewer device 110 may be a television, a mobile phone, or other device capable of playing back the output from the post-delivery processing device. For example, the post-delivery processing device can output the decoded and synthesized video asset 118 with grain. The post-delivery processing device can be integrated into the viewer device 110 or configured as an independent unit such as a set-top box, and provide the video asset 118 processed with decoding and synthesized grain processing to the user viewing the device. The viewer device may be a viewer device located remotely (miles or thousands of miles away) from the location where the pre-delivery processing device is installed.

[0021] The network monitor 112 may be configured to monitor various forms of video assets 102 as they pass through various stages or points on the end-to-end system 100. For example, the network monitor 112 may identify the QoE score of video assets 102 at various points on the end-to-end system 100. Components of the end-to-end system 100 may be configured to perform additional aspects related to grain-aware video encoding and grain-aware characterization and restoration in transmission, as will be discussed in detail herein. The network monitor 112 shown in Figure 1A represents a functional block in the hardware that can communicate, exchange, or query necessary and available information within the pre-delivery processing unit, post-delivery processing unit, and / or content delivery network. The network monitor can provide additional processing available to the pre-delivery processing unit and post-delivery processing unit. Processing related to the network monitor can also be performed within the pre-delivery processing unit and post-delivery processing unit devices. It should be noted that the end-to-end system 100 may be geographically distributed, and calculations may be performed in the same location or in a distributed manner.

[0022] Figure 1B shows an example of an end-to-end system configuration comprising a pre-delivery processing unit 104 configured to perform multiple operations on a received video asset 102. For example, as shown in the block representing the pre-delivery processing unit, operations include grain detection, grain evaluation, grain modeling, grain reduction, grain-aware encoding, and rate control. The pre-delivery processing unit can also be configured to hold and perform grain-aware co-optimization processing.

[0023] The post-delivery processing unit 108 is configured to perform multiple operations on the delivered encoded video asset 116. For example, as shown in the block representing the post-delivery processing unit, the operations include decoding, grain compositing, and grain-aware rendering. The post-delivery processing unit can also be configured to hold and perform quality assessment operations.

[0024] One embodiment shown in Figure 1B illustrates that grain parameters are generated within a pre-distribution processing unit and provided to the content distribution network along with the video asset 114. For example, the grain parameters generated within the pre-distribution processing unit may be generated by grain evaluation and / or grain modeling operations. The grain parameters are provided along with the encoded video asset, embedded as metadata information describing the content and / or format of the video. The encoded video asset 114 output by the pre-distribution processing unit is configured to include the encoded video asset along with metadata containing the grain parameters. The encoded video asset with metadata is distributed to the post-distribution processing unit via the content distribution network.

[0025] In another embodiment shown in Figure 1B, grain quality measurement values ​​120 are provided as feedback from the post-distribution processing unit to the pre-distribution processing unit. The grain quality measurement values ​​120 are distributed via the network monitor 112, or through network monitors in the content distribution network 106, or by other means. The post-distribution processing unit 108 can be configured to perform a video quality evaluation to generate grain quality measurement values. For example, the post-distribution processing unit decodes the distributed encoded video asset to generate a decoded video asset. The grain compositing process generates film grain using film grain parameters embedded in the metadata of the encoded video asset, and the composited film grain is added to the decoded video asset to generate a decoded and composited grain video asset. A quality evaluation can be performed on the composited grain and / or the decoded and composited grain video asset to generate a grain quality index. Other quality evaluations can also be performed to generate other quality indices such as a structural fidelity index or an overall quality index. Another quality index is a video quality index based on a video quality evaluation of the decoded and distributed encoded video asset output from the decoding process. This video quality index can be used as a video quality parameter in the pre-processing unit for distribution.

[0026] Grain-aware measurements and other quality measurements generated within the post-delivery processing unit are fed back to the pre-delivery processing unit as grain-aware parameters and other quality parameters, respectively, enabling optimization of processes within the pre-delivery processing unit. For example, grain-aware measurements / parameters are used in the pre-delivery processing unit to optimize grain-aware encoding through grain-aware encoding and rate control operations, and / or grain-aware reduction through grain-aware reduction operations, optimizing grain-aware QoE within the encoder's bitrate profile while simultaneously being suitable for the type of display in the post-delivery processing unit or remote viewing device.

[0027] Another embodiment involves using both grain-aware parameters (based on grain) and image quality parameters (based on image object characteristics) in the grain-aware joint optimization process 602 to optimally adjust the grain reduction process and grain-aware encoding process in the pre-distribution processing unit. Details of the quality evaluation and measurement disclosed above will be described later.

[0028] If the display and post-distribution processing unit are the same unit, the decoded and composited grain video asset is rendered for the display using grain-aware rendering operations. If the display device is a separate viewing device from the post-distribution processing unit, the post-distribution processing unit can output the decoded and composited grain video asset to the remote viewing device for rendering by the remote viewing device. Alternatively, the remote viewing device can expose display rendering parameters to the post-distribution processing unit in order to perform grain-aware rendering.

[0029] Existing pre-delivery and post-delivery processing units within an end-to-end system can be upgraded to enable the execution of processes that take into account the disclosed advanced grain. This upgrade may be a software upgrade or a hardware upgrade for the pre-delivery and post-delivery processing units. For example, an existing pre-delivery processing unit can be replaced with an upgraded pre-delivery processing unit, and an existing post-delivery processing unit can be replaced with an upgraded post-delivery processing unit.

[0030] This specification provides a flowchart detailing the flow of video content through various operations defined within the pre-delivery and post-delivery processing units of the target end-to-end system, as well as the interactions between these operations, for delivering video assets with film grain to viewing devices and optimizing the trade-off between grain QoE and rate distortion.

[0031] Figure 2 shows an example data flow 200 of a grain-aware video encoding and transmission process. As an example, the entire data flow 200 can be executed using components of an end-to-end system 100. As shown in the figure, a media source provides a video asset 102, and the media stream goes through several processing stages along the end-to-end system 100 before reaching the viewer device 110. The video asset 102 refers to various types of video content, as previously mentioned, and the viewer device 110 may refer to one or more consumer devices. The processes performed along the data flow 200 include grain detection 202, grain evaluation / modeling 204, grain reduction processing 206, grain-aware encoding / rate control 208, grain-aware streaming / decoding 210, grain synthesis 212 (sometimes called regrain), and grain-aware rendering / display 214. These will be explained in order.

[0032] Grain detection 202 includes one or more processes that identify and characterize the amount of grain in the video asset 102. For example, these processes are performed by the pre-delivery processing device 104. Grain detection 202 performs various statistical analyses on the video asset 102 (e.g., calculation of mean, variance, and distribution of image features) to identify the presence of film grain. In another example, frequency domain analysis may be performed to convert the video asset 102 to the frequency domain and analyze its power spectrum to identify the presence of film grain. In such an analysis, film grain is detected as a high-frequency pattern appearing as a peak in the high-frequency power spectrum. In some examples, grain detection 202 is performed by using an edge detector to identify flat or smooth areas with relatively uniform color tones within the video asset 102. For example, grain detection 202 determines a mask of the flat area to be analyzed as the grain detection result 302.

[0033] Grain evaluation / modeling 204 may include a series of processes for determining the degree of grain indicated by grain detection 202. For example, these processes are performed by the delivery preprocessor 104. Grain evaluation / modeling 204 identifies the characteristics of film grain within the video asset 102, which may be done for the entire frame or only for a masked flat region of interest. Grain evaluation / modeling 204 may be configured to provide an output showing the parameters of the grain contained in the video asset 102 (these parameters are shown more specifically as grain model parameters 304 in Figure 3).

[0034] For example, video asset 102 is denoised, and the region of the denoised video asset 102 is compared to the original video asset 102 to determine the noise parameters. For example, these grain model parameters 304 include one or more of the grain size, grain density, and / or grain contrast. In another example, film grain is modeled using root mean square (RMS) grain size, which is a numerical quantification of density heterogeneity and is equivalent to the RMS variation of optical density. In some examples, film grain can be characterized using an autoregressive model. In some examples, the intensity of film grain changes in response to signal intensity, and the grain model parameters 304 can further model these level differences, for example, as parameters of a linear function of luminance (luma).

[0035] The grain reduction process 206 is configured to reduce the grain present in the video asset 102. The grain reduction process 206 is performed in various ways by the pre-delivery processing device 104. For example, a noise reduction filter analyzes the pixels of the video asset 102 to identify the average color or brightness and / or outlier pixels. The filter then applies a smoothing algorithm that averages the pixel values ​​within the region to effectively reduce noise or grain. In another example, a machine learning model can be trained on a large dataset of images with and without grain to learn to identify and remove film grain.

[0036] Grain-aware encoding / rate control 208 performs encoding on the video asset 102 after grain reduction processing 206. For example, grain-aware encoding / rate control 208 uses the encoder of the pre-delivery processing device 104 to convert the video asset 102 into a format for transmission along the end-to-end system 100. This video encoding may take into account parameters related to the type and characteristics of grain. The encoding may also include rate control and / or quality control operations to ensure that a predetermined target bitrate and / or quality level is achieved. Examples of video encoding / rate control parameters 402 (shown in detail in Figure 4) include video bitrate, spatial resolution, frame rate or temporal resolution, encoding mode selection at video level, frame level, and local block level, and quantization step parameters at video level, frame level, and local block level.

[0037] The grain-aware streaming / decoding process 210 performs the transfer of the encoded video asset 114 along the end-to-end system 100, and the decoding of the video asset after transfer. Thus, the encoded video asset 114 is streamed to the receiving end and decoded. For example, the streaming process may be performed using components of the content distribution network 106, and the decoding process may be performed by the post-distribution processing unit 108.

[0038] The grain synthesis 212 includes one or more processes performed by the post-delivery processing unit 108 to decode the delivered encoded video asset 116, add grain, and generate a decoded and synthesized grain video asset 118. This added grain is consistent with the grain evaluated by the grain evaluation / modeling 204 and / or the grain reduced by the grain reduction process 206. For example, the grain synthesis 212 may be configured to add noise characterized by various parameters such as grain size, grain density, and / or grain contrast. As an example, these parameters are the same as, or consistent with, the grain model parameters 304 (see Figure 3) determined by the grain evaluation / modeling 204 and / or added to the region defined by the grain detection result 302 (see Figure 3) determined by the grain detection 202.

[0039] Grain-aware rendering / display 214 may include providing the decoded and composited grain video asset 118 to the viewer device 110. By reincorporating the grain after compression, transmission, and decoding, it becomes possible to more efficiently encode the video asset 102 in the absence of film grain while maintaining the visual quality of the decoded and composited grain video asset 118 at the viewer device 110.

[0040] Figure 3 shows an exemplary data flow 300 for the process of evaluating grain and video quality along grain-aware video encoding and transmission. Similar to data flow 200, operations performed along data flow 300 include grain detection 202, grain evaluation / modeling 204, grain reduction processing 206, grain-aware encoding / rate control 208, grain-aware streaming / decoding 210, grain synthesis 212, and grain-aware rendering / display 214.

[0041] Furthermore, in the data flow 300, grain-aware video encoding and transmission may be monitored or evaluated at multiple stages or checkpoints. This includes grain detection 202 or classification results generated by the grain detection process 202, grain-aware model parameters 304 generated by grain-aware evaluation / modeling 204, video quality evaluation and rate distortion performance analysis results (comparing video streams before video encoding and decoding) generated from video quality evaluation / rate distortion analysis 306, and one or more grain quality evaluations 310, such as grain quality / fidelity evaluation, structural similarity evaluation, and overall video quality evaluation results, generated by analyzing the video after grain synthesis 212.

[0042] Grain detection 202 can publish the grain detection result 302 generated by the grain detection 202 process. As described above, the grain detection result 302 may include parameters such as a mask for the region where grains are identified.

[0043] Grain evaluation / modeling 204 exposes grain model parameters 304 generated by grain evaluation / modeling 204. As mentioned above, grain model parameters 304 may include parameters such as grain size, grain density, and / or grain contrast, as well as variations in grain model parameters 304 in response to differences in signal levels.

[0044] Furthermore, a video quality evaluation / rate distortion analysis 306 can be performed to generate video quality measurement / rate distortion (RD) performance parameters 308. Video quality evaluation and rate distortion may include a comparison between the video stream before video encoding and the same video stream after decoding. For example, the video quality measurement / RD performance parameters 308 may show the difference in perceived quality between the transmitted and decoded video asset 102 and the video asset 102 before encoding.

[0045] As an example, the network monitor 112 can perform evaluation and analysis. Examples of video quality evaluation methods include full-reference or subtractive-reference methods such as peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), video quality model (VQM), SSIMPLUS, and / or video multi-method evaluation fusion (VMAF) measurement, which use the video after grain reduction processing 206 and before encoding as a reference. Examples of video encoding and rate control parameters include video bitrate, spatial resolution, frame rate or temporal resolution, encoding mode selection at the video level, frame level, and local block level, and quantization step parameters at the video level, frame level, and local block level. Examples of image quality-oriented rate distortion analysis, rate control, and optimization include SSIM-based rate distortion optimization (RDO) and analysis.

[0046] The network monitor 112 can also perform one or more grain quality evaluations 310 on the decoded and synthesized grain video asset 118 after grain synthesis 212. These grain quality evaluations 310 may generate grain quality / fidelity evaluation parameters 312. These grain quality evaluations 310 may include grain quality / fidelity evaluation, structural fidelity evaluation, and / or overall video quality evaluation. Grain quality / video fidelity can be a measure / parameter that enables the ability to distinguish between the grain quality parameters of the video asset 102 before processing by the pre-delivery processing unit and the grain quality parameters of the decoded and synthesized grain video asset generated by the post-delivery processing unit. On the other hand, video quality indicates the relationship between the preference for viewing the pre-delivery video asset 102 before processing by the pre-delivery processing unit on a display and the preference for viewing the decoded and synthesized grain video asset 118 generated by the post-delivery processing unit on a display.

[0047] Figure 4 shows an exemplary data flow 400 for the grain reduction process 206 along grain-aware video encoding and transmission. Similar to data flow 300, operations performed along data flow 400 include grain detection 202, grain evaluation / modeling 204, grain reduction process 206, grain-aware encoding / rate control 208, grain-aware streaming / decoding 210, grain synthesis 212, grain-aware rendering / display 214, video quality evaluation / rate distortion analysis 306, and grain quality evaluation 310. In this example, the grain reduction process 206 obtains information by monitoring video assets 102 processed along an end-to-end system 100.

[0048] As mentioned above, the grain evaluation / modeling 204 generates grain model parameters 304, the video quality evaluation / rate distortion analysis 306 generates video quality measurement / RD performance parameters 308, and the grain quality evaluation 310 generates grain quality / fidelity evaluation parameters 312.

[0049] Furthermore, in dataflow 400, grain-aware encoding / rate control 208 may expose its video encoding / rate control parameters 402. For example, video encoding / rate control parameters 402 may be provided to grain-aware streaming / decoding 210. As mentioned above, video encoding / rate control parameters 402 may include parameters such as video bitrate, spatial resolution, frame rate or temporal resolution, encoding mode selection at video level, frame level, and local block level, and quantization step parameters at video level, frame level, and local block level.

[0050] The grain reduction process 206 is performed using one or more of the following inputs: grain evaluation results and grain model parameters 304, video encoding / rate control parameters 402, video quality measurement / RD performance parameters 308, and grain quality / fidelity evaluation parameters 312. For example, using the grain model parameter 304 allows the grain reduction process 206 to consider aspects such as grain size, grain density, and / or grain contrast. In another example, using the video encoding / rate control parameter 402 allows the grain reduction process 206 to consider aspects such as video bitrate, spatial resolution, frame rate or temporal resolution, video level, frame level, and encoding mode selection at the local block level. In yet another example, using the grain quality / fidelity evaluation parameter 312 allows the grain reduction process 206 to consider aspects such as grain quality / fidelity, structural fidelity, and / or the overall video quality of the video asset 102. The video encoding / rate control parameter 402 and the grain quality / fidelity evaluation parameter 312 can be used to enable the grain reduction process 206 to control the level / type of grain to be reduced in order to achieve the best QoE for the final viewer while consuming the minimum bandwidth. These parameters can be jointly optimized with the final viewer's QoE and the overall required bandwidth as the overall target function.

[0051] As another example, the results and parameters of the grain reduction process 206 can be used as feedback to the grain evaluation / modeling 204 to further improve the grain evaluation / modeling 204 process. For example, the results of noise reduction filters and machine learning model adjustments for the first frame of video asset 102 can be provided as input to the grain evaluation / modeling 204 for the second frame (e.g., the next frame) of video asset 102. This allows the grain evaluation / modeling 204 to take into account inter-frame features and to converge more quickly to the correct grain model parameters 304.

[0052] Figure 5 shows an example of a data flow 500 related to the grain-aware encoding / rate control process 208. Similar to data flows 200-400, operations performed along data flow 500 include grain detection 202, grain evaluation / modeling 204, grain reduction processing 206, grain-aware encoding / rate control 208, grain-aware streaming / decoding 210, grain synthesis 212, grain-aware rendering / display 214, video quality evaluation / rate distortion analysis 306, and grain quality evaluation 310. In this example, grain-aware encoding / rate control 208 obtains information by monitoring video assets 102 processed along the end-to-end system 100.

[0053] As shown in the figure, the grain reduction process 206 may present grain reduction parameters 502. These grain reduction parameters 502 may include, for example, parameters for a noise reduction filter (such as area size and sensitivity), average color and average brightness for identifying outlier pixels, parameters for a smoothing filter (such as area size and smoothing amount), and machine learning model configurations for film grain removal. As an example, the grain reduction parameters 502 are provided to the grain-aware encoding / rate control 208.

[0054] Furthermore, the grain synthesis 212 can expose its grain synthesis parameters 504. The grain synthesis parameters 504 may include parameters such as grain size, grain density, and / or grain contrast, as well as variations in response to differences in signal levels. For example, the grain synthesis parameters 504 may be the same as, or otherwise consistent with, the grain model parameters 304 identified before the grain reduction process 206. This allows for regraining with grain consistent with the grain removed from the video asset 102. In another example, the grain synthesis parameters 504 may include regraining parameters that are identical to or consistent with the user settings of the viewer device 110 and / or the configuration of the content delivery network 106, providing a consistent appearance throughout the decoded and synthesized grain video asset 118. For example, the grain synthesis parameters 504 are provided to the grain-aware encoding / rate control 208.

[0055] Furthermore, the grain-aware rendering / display 214 exposes its rendering / display parameters 506. The rendering / display parameters 506 may include information describing the information for delivering the decoded and composited grain-aware video asset 118 to the end user. For example, the rendering / display parameters 506 may include the screen size of the viewer device 110, the screen resolution of the viewer device 110, and the distance from the viewer to the viewer device 110. As an example, the rendering / display parameters 506 may be provided to the grain-aware encoding / rate control 208. These rendering / display parameters 506 allow the grain-aware encoding / rate control 208 to characterize the video asset 102 with respect to the characteristics of the decoded and composited grain-aware video asset 118 that will be perceived by the end user. For example, if the viewer device 110 is small, the encoding may take its screen size into consideration in order to maintain the minimum perceived quality of the video asset 102.

[0056] Grain-aware encoding / rate control 208 is performed on the video asset 102 using one or more of the following inputs: grain evaluation results and grain model parameters 304 from grain evaluation / modeling 204, grain reduction processing parameters 502 from grain reduction processing 206, grain synthesis parameters 504 from grain synthesis 212, video rendering / display parameters 506 from grain-aware rendering / display 214, video quality measurement / RD performance parameters 308, and grain-aware quality / fidelity evaluation parameters 312 from grain-aware evaluation 310.

[0057] The overall performance of grain-aware encoding is measured by a cost function that considers the final viewer's QoE and total bandwidth. For example, the cost function can be defined as a weighted sum of the scores of the video quality evaluation / rate distortion analysis 306, the grain-aware / fidelity evaluation parameter 312, and the video bitrate parameter. Grain-aware encoding / rate control 208 can be tuned to find the minimum point of the cost function influenced by these parameters. Therefore, fine-tuning grain-aware encoding / rate control 208 based on these parameters to minimize the cost function becomes a co-optimization problem. As an example, if the pre-encoding grain reduction intensity parameter 206 is set high and grain is significantly reduced, and / or if the grain quality / fidelity evaluation parameter 312 indicates good quality grain synthesis 212, then grain-aware encoding / rate control 208 will adopt a lower target bitrate and a larger quantization step parameter, resulting in improved video compression and reduced bitrate.

[0058] The overall performance of grain-aware encoding is measured by a cost function that considers the final viewer's QoE (Quality of Experience) and total bandwidth. For example, the cost function can be defined as a weighted sum of 1) a negative value of the video quality evaluation score, 2) a negative value of the grain-aware / fidelity evaluation score, and 3) the video bitrate parameter. Grain-aware encoding / rate control 208 can be adjusted to find the minimum point of the cost function, which is affected by these parameters. Therefore, fine-tuning grain-aware encoding / rate control 208 based on these parameters to minimize the cost function becomes a co-optimization problem. As an example, if the strength parameter of the pre-encoding grain reduction process 206 is strong and grain is significantly reduced, and / or the grain quality / fidelity evaluation parameter indicates good grain synthesis 212 quality, then grain-aware encoding and rate control may employ a lower target bitrate and a larger quantization step parameter, resulting in improved video compression and a lower bitrate.

[0059] For example, the encoding process may generate multiple video outputs with different spatial and temporal resolutions and different bitrates. In such cases, an encoding ladder for the video asset 102 is created through the encoding process, and grain-aware encoding / rate control 208 may be performed for each individual level of the encoding ladder, or jointly for all levels.

[0060] In another embodiment, some parts of the video encoding procedure, such as the quantization step size and target bitrate, can be optimized by jointly considering the relative changes between the grain reduction processing parameter 502, the video quality measurement / RD performance parameter 308, and the grain quality / fidelity evaluation parameter 312. This enables optimization that maximizes the final viewer's QoE while minimizing the required bandwidth.

[0061] Figure 6 shows an exemplary data flow 600 in the grain-aware joint grain reduction process 206 and encoding process. Similar to data flows 200-500, operations performed along data flow 600 include grain detection 202, grain evaluation / modeling 204, grain reduction process 206, grain-aware encoding / rate control 208, grain-aware streaming / decoding 210, grain synthesis 212, grain-aware rendering / display 214, image quality evaluation / rate distortion analysis 306, and grain evaluation 310.

[0062] In this data flow 600, grain-aware collaborative optimization 602 may be performed between grain-aware reduction 206 and grain-aware encoding / rate control 208. This may be added to achieve an optimal balance between grain-aware reduction 206 and rate distortion performance. In grain-aware collaborative optimization 602, it is desirable to retain or reconstruct the original grain as much as possible while reducing the bandwidth required for video encoding. The two extreme cases in this dilemma are: 1) removing all grain and drastically reducing the required bandwidth, but also dramatically lowering the end-user's QoE; and 2) retaining all grain, resulting in very good end-user QoE, but also requiring very large bandwidth. Therefore, there is a compromise to be achieved in between, and all of these parameters may be involved in determining that compromise. This balance can be achieved by utilizing one or more of the following input elements: grain evaluation results and grain model parameters 304, grain reduction processing parameters 502, video encoding / rate control parameters 402, video quality measurement / rate distortion performance parameters 308, and grain quality / fidelity evaluation parameters 312.

[0063] As an example, the optimization objective function is defined as a weighted sum of 306 parameters for video quality evaluation / RD analysis, 310 parameters for grain quality evaluation, a structural fidelity evaluation parameter, and an overall video quality evaluation parameter. The optimization method involves searching all combinations of the grain reduction processing parameter 502, the video encoding parameters of the video encoding / rate control parameter 402 (e.g., including quantization step and encoding mode selection), and the rate control parameters of the video encoding / rate control parameter 402 (including target bitrate for video segments, video frames, and encoding blocks within each video frame) to find the selection that generates the maximum value of the optimization objective function.

[0064] In one embodiment, the grain-aware co-optimization process 602 is performed using an interactive process between the grain reduction process 206 and the grain-aware encoding / rate control 208, and can be performed iteratively through multiple iterations. For example, the grain reduction process parameter 502 for the current iteration can be adjusted using the video encoding / rate control parameter 402 and the video quality evaluation result from the previous iteration, and the video encoding and rate control parameters for the current iteration can be determined using the grain reduction process parameter 502.

[0065] In another embodiment, the grain-aware / fidelity evaluation parameter 312 can be used to simultaneously adjust both the grain-aware reduction 206 and the grain-aware corresponding encoding / rate control 208 control parameters.

[0066] In another embodiment, the grain evaluation / modeling 204 process of video asset 102 may also include determining grain quality metrics for the video asset received by the pre-delivery processing unit. These grain quality metrics may be of the same type as the quality metrics determined by the post-delivery processing unit. The quality metrics determined in grain evaluation / modeling 204 are provided as grain quality parameters to the grain-aware co-optimization process 602, along with grain model parameters 304. The grain-aware co-optimization can optimize video encoding and rate control parameters using the grain quality metrics / parameters before processing by the pre-delivery processing unit and after processing by the post-delivery processing unit. For example, the grain-aware / fidelity evaluation parameter 312 in Figure 6 can be used in combination with the fidelity quality parameter determined for video asset 102 from the grain-aware evaluation / modeling 204 operation to optimize video encoding and rate control parameters.

[0067] Figure 7 shows an example data flow 700 in the process of grain-aware video streaming and decoding. Similar to data flows 200-600, operations performed along data flow 700 include grain detection 202, grain evaluation / modeling 204, grain reduction processing 206, grain-aware encoding / rate control 208, grain-aware streaming / decoding 210, grain synthesis 212, grain-aware rendering / display 214, video quality evaluation / rate distortion analysis 306, and grain evaluation 310. In this example, the grain-aware streaming / decoding process 210 obtains information by monitoring the video asset 102 being processed along the end-to-end system 100. These additional inputs enable the grain-aware streaming / decoding process 210 to recognize grain-aware aspects of the encoded video asset 114 during streaming.

[0068] For example, grain-aware streaming / decoding processing 210 may be performed using one or more of the following inputs: grain evaluation results and grain model parameters 304, grain reduction processing parameters 502, video encoding / rate control parameters 402, grain synthesis parameters 504, video rendering / display parameters 506, video quality measurement / RD performance parameters 308, and grain quality / fidelity evaluation parameters 312. Here, the streaming scenario is similar to the encoding scenario, but differs in that the optimal solution must be selected from a more limited set of choices (multiple encoding derived levels or levels within an encoding ladder). The basic principle is largely the same: to find the optimal encoding level that minimizes overall bandwidth usage while maximizing the end user's QoE.

[0069] In one embodiment, only one level of the video asset 102 in the encoding ladder is streamed and decoded by the decoder. This level is jointly determined by the granularity model parameter 304, the image quality measurement / RD performance parameter 308, and the granularity quality / fidelity evaluation parameter 312.

[0070] Figure 8 shows an exemplary data flow 800 in the grain synthesis 212 or regrainization process. Similar to data flows 200-700, operations performed along data flow 800 include grain detection 202, grain evaluation / modeling 204, grain reduction processing 206, grain-aware encoding / rate control 208, grain-aware streaming / decoding 210, grain synthesis 212, grain-aware rendering / display 214, video quality evaluation / rate distortion analysis 306, and grain evaluation 310. In this data flow 800, grain synthesis 212 may be enhanced to receive additional inputs from other components. These additional inputs enable grain synthesis 212 to recognize grain-related aspects of the encoded video asset 114 during streaming.

[0071] As shown in the figure, the grain-aware streaming / decode 210 exposes streaming / decode parameters 802. These streaming / decode parameters 802 include, for example, the codec used for streaming, the encoder settings used for encoding generation, the bitrate used for streaming, and the resolution of the video asset 102. As an example, the streaming / decode parameters 802 are provided to the grain synthesis 212.

[0072] Grain synthesis 212 (or regrainization) is performed using one or more of the following inputs: grain evaluation results and grain model parameters 304, grain reduction processing parameters 502, video encoding / rate control parameters 402, video rendering / display parameters 506, video quality measurement / RD performance parameters 308, grain quality / fidelity evaluation parameters 312, and streaming / decode parameters 802. The final appearance of the grain is controlled by grain synthesis parameter 504. As with video encoding, quality measurement, and rendering, grain synthesis parameter 504 needs to be adjusted so that an appropriate level of grain is generated.

[0073] In one embodiment, the grain blending parameter 504 (e.g., grain intensity) is determined by jointly utilizing the grain model parameter 304 and the video encoding / rate control parameter 402, and is adjusted by jointly utilizing the video quality evaluation and rate distortion performance parameter and the grain quality / fidelity evaluation parameter 312. For example, if the video encoding quantization is strong or the bitrate is low, the intensity of the grain blending 212 can be increased to produce a visual masking effect on overcompressed or oversmoothed areas within the video frame. As another example, if the grain quality / fidelity evaluation parameter 312 indicates that the quality of the decoded and blended grain video asset 118 is low, the intensity of the grain blending 212 can be reduced to avoid generating even lower quality grain.

[0074] Figure 9 shows an example of dataflow 900 in the process of grain-aware rendering / display 214. Similar to dataflows 200-800, operations performed along dataflow 900 include grain detection 202, grain evaluation / modeling 204, grain reduction processing 206, grain-aware encoding / rate control 208, grain-aware streaming / decoding 210, grain synthesis 212, grain-aware rendering / display 214, video quality evaluation / rate distortion analysis 306, and grain evaluation 310. In this dataflow 900, grain-aware rendering / display 214 may be enhanced to receive additional inputs from other components. These additional inputs enable grain-aware rendering / display 214 to recognize grain-related aspects of the encoded video asset 114 during streaming.

[0075] Grain-aware rendering / display 214 may be performed using one or more of the following inputs: grain evaluation results and grain model parameters 304, grain reduction processing parameters 502, video encoding / rate control parameters 402, grain synthesis parameters 504, video quality evaluation / RD analysis 306, grain quality / fidelity evaluation parameters 312, and streaming / decode parameters 802.

[0076] Depending on the specific configuration and deployment, the grain model parameter 304 and grain blending parameter 504 may or may not be available on the viewer device 110. If available, the rendering / display parameter 506 is determined using the grain model and grain blending parameter 504 to blend grain that is visually similar to the original image. Regraining may occur after decoding and on the video rendering device. Depending on the rendering / display parameter 506 (e.g., screen type, resolution, size, brightness), the regraining algorithm of the grain blending 212 adjusts the resulting grain characteristics (e.g., size, density, contrast) to maximize the final viewer's QoE.

[0077] Figure 10 shows an example of dataflow 1000 in a grain-aware video quality evaluation process, which generates grain-aware / fidelity measurements, structural fidelity measurements, and overall video quality measurements. Similar to dataflows 200-900, operations performed along dataflow 1000 include grain detection 202, grain evaluation / modeling 204, grain reduction processing 206, grain-aware encoding / rate control 208, grain-aware streaming / decoding 210, grain synthesis 212, grain-aware rendering / display 214, video quality evaluation / rate distortion analysis 306, and grain quality evaluation 310.

[0078] In this data flow 1000, the grain quality evaluation 310 is performed in multiple steps, each step generating a different useful evaluation for the image after grain synthesis 212 (re-grained image). These operations include: (A) grain-aware fidelity / quality evaluation, which evaluates the fidelity or quality of the synthesized grain using grain-aware model parameters 304 as a reference; (B) structural fidelity evaluation, which evaluates the structural details in the re-grained image content and utilizes parameters for image quality evaluation and rate distortion analysis between pre- and post-encoded images; and (C) overall image quality evaluation, which utilizes both grain fidelity evaluation and structural fidelity evaluation, and their interaction. These evaluations provide quality information for the final rendered image from three perspectives (grain fidelity, structural fidelity, and overall image quality). Each evaluation examines a different quality aspect and provides the corresponding image QoE information as described above. Overall quality is relatively important as it is closest to the end user and reflects their QoE. Grain fidelity and structural fidelity are two important components of overall quality, separate from other factors, because they are characteristics that respond most sensitively to the human visual system (HVS).

[0079] Figure 11 shows an example of a computing device 1102 used in an end-to-end system 100 for grain-aware video encoding transmission. Referring to Figure 11 and Figure 1A-10, the devices and modules described herein may be examples of such a computing device 1102. As shown in the figure, the computing device 1102 includes a processing unit 1104 operably connected to storage 1106, a network device 1108, an output device 1110, and an input device 1112. It should be noted that this is merely an example, and computing devices 1102 with more, fewer, or different components may be used.

[0080] The processing unit 1104 may include one or more integrated circuits that implement the functions of a central processing unit (CPU) and / or a graphics processing unit (GPU). In some examples, the processing unit 1104 is a system-on-a-chip (SoC) that integrates the functions of the CPU and GPU. The SoC may also include other components, such as storage 1106 and network devices 1108, in a single integrated device. In other examples, the CPU and GPU are connected to each other via peripheral connectivity devices such as Peripheral Component Interconnect (PCI) Express or other appropriate peripheral data connections. As an example, the CPU is a commercially available central processing unit that implements an instruction set such as x86, ARM, Power, or Interconnected Pipelined Stageless Microprocessor (MIPS) instruction set family.

[0081] Regardless of the details, during operation, the processing unit 1104 executes stored program instructions obtained from the storage unit 1106. Therefore, the stored program instructions include software that controls the operation of the processing unit 1104 to perform the operations described herein. The storage unit 1106 may include both non-volatile memory and volatile memory devices. Non-volatile memory includes solid-state memory such as NAND flash memory, magnetic and optical storage media, or other suitable data storage devices that retain data when the system is inactive or power is lost. Volatile memory includes static and dynamic random access memory (RAM) that stores program instructions and data during the operation of the end-to-end system 100.

[0082] The GPU may include hardware and software for displaying at least two-dimensional (2D) and optionally three-dimensional (3D) graphics to the output device 1110. The output device 1110 may include a graphical or visual display device, such as an electronic display screen, a projector, a printer, or other suitable device for reproducing a graphical display. As another example, the output device 1110 may include an audio device, such as a speaker or headphones. As yet another example, the output device 1110 may include a tactile device, such as a mechanically raised device that can be configured to display, for example, Braille or other physical output that can provide information to the user by touch.

[0083] The input device 1112 may include any of a variety of devices that enable the computing device 1102 to receive control input from a user. Examples of suitable input devices for receiving human interface input include keyboards, mice, trackballs, touchscreens, voice input devices, and graphics tablets.

[0084] Each network device 1108 can include any of a variety of devices that enable sending and receiving data from external devices over the network. Examples of suitable network devices 1108 include Ethernet interfaces, Wi-Fi transceivers, cellular transceivers, or Bluetooth or Bluetooth Low Energy (BLE) transceivers, ultra-wideband (UWB) transceivers, or other network adapters and peripheral interconnect devices, which are useful for receiving data from other computers or external data storage devices and for efficiently receiving large amounts of data.

[0085] The processes, methods, or algorithms disclosed herein are available / implementable for processing units, controllers, or computers, which may include existing programmable electronic control units or dedicated electronic control units. Similarly, the processes, methods, or algorithms are stored as data and instructions and are executable by controllers or computers, and this may take many forms, including, but not limited to, information permanently stored on non-writable storage media such as read-only memory (ROM) devices, and information variably stored on writable storage media such as floppy disks, magnetic tapes, compact disks (CDs), RAM devices, and other magnetic and optical media. The processes, methods, or algorithms may also be implemented as software executable objects. Alternatively, the processes, methods, or algorithms may be embodied in whole or in part using appropriate hardware components, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers, other hardware components or devices, or combinations of hardware, software, and firmware components.

[0086] While exemplary embodiments have been described above, these embodiments are not intended to describe all possible forms included in the claims. The terminology used in the specification is for illustrative purposes only, not limitation, and it should be understood that various modifications are possible without departing from the spirit and scope of the disclosure. As noted above, features of various embodiments can be combined to form further embodiments of the invention not expressly described or illustrated. Various embodiments may be described as offering advantages or being preferable to other embodiments or prior art examples with respect to one or more desired characteristics, but those skilled in the art will recognize that one or more features or characteristics may be compromised to achieve desired overall system characteristics that depend on the particular application and embodiment. These attributes include, but are not limited to, strength, durability, lifecycle, marketability, appearance, packaging, size, maintainability, weight, manufacturability, and ease of assembly. Therefore, even embodiments described as being undesirable to other embodiments or prior art examples with respect to one or more characteristics are not outside the scope of the disclosure and may be desirable in a particular application.

[0087] The above embodiments are illustrative and not intended to encompass all possible forms of the invention. Rather, the terminology used in this specification is for illustrative purposes only, not limitation, and it should be understood that various modifications are possible without departing from the spirit and scope of the invention. Furthermore, it is possible to combine features from various embodiments to form further embodiments of the invention.

Claims

1. A method for optimizing video encoding and transmission that takes into account the grain of video assets, The steps include determining grain model parameters that represent the grain characteristics present in the aforementioned video asset, The steps include: performing grain reduction processing on the video asset before encoding it, according to grain reduction processing parameters that indicate how to remove grain from the video asset; After the grain reduction process, the video asset is encoded using grain-aware encoding / rate control according to the video encoding / rate control parameters. The steps include streaming and decoding the aforementioned video asset, A method comprising the steps of: after decoding, performing grain synthesis using grain synthesis parameters that indicate the characteristics of the grain to be applied to the video asset, and then reapplying the grain to the video asset.

2. The steps include: performing a video quality evaluation / rate distortion (RD) analysis that compares the video asset before encoding with the video asset after decoding, and generating video quality measurement values / RD performance parameters from the video quality evaluation / RD analysis; The steps include: performing at least one grain quality evaluation on the video asset after grain synthesis to generate grain quality / fidelity evaluation parameters; The method according to claim 1, further comprising at least one of the steps of controlling at least a portion of rendering the video asset to an end user by grain reduction processing, grain-aware encoding / rate control, streaming and decoding, grain synthesis, and / or grain-aware rendering / display, using the video quality measurement value / RD performance parameter and / or grain quality / fidelity evaluation parameter.

3. The method according to claim 2, further comprising the step of rendering the video asset to the end user by rendering / displaying the grain, wherein the rendering / displaying the grain is provided with information based on at least one of the grain model parameters, the grain reduction processing parameters, the video encoding / rate control parameters, the grain synthesis parameters, the video quality evaluation / RD analysis, and the grain quality / fidelity evaluation parameters.

4. The steps include determining rendering / display parameters that indicate the rendering of the video asset to the end user, The method according to claim 2, further comprising the step of determining the grain quality / fidelity evaluation parameters using the rendering / display parameters.

5. The grain model parameter represents at least one of grain size, grain density, and / or grain contrast, and the grain reduction process includes setting the grain reduction process using the grain model parameter. The video encoding / rate control parameter indicates at least one of the video bitrate, spatial resolution, frame rate or temporal resolution, encoding mode selection at each video / frame / local block level, and quantization step parameter at each video / frame / local block level, and the grain reduction process includes setting the grain reduction process using the video encoding / rate control parameter, and / or The method according to claim 2, wherein the grain quality / fidelity evaluation parameter includes at least one of grain quality / fidelity evaluation, structural fidelity evaluation, and / or overall quality evaluation, and the grain reduction process includes setting the grain reduction process using the grain quality / fidelity evaluation parameter.

6. The method according to claim 1, wherein the encoding / rate control considering grain includes the step of encoding the video asset using the video encoding / rate control parameters and the grain reduction processing parameters after the grain reduction processing.

7. The method according to claim 1, further comprising the step of performing grain-aware joint optimization between the grain reduction process and the grain-aware encoding / rate control, and balancing the grain reduction performance of the grain reduction process and the rate distortion performance of the grain-aware encoding / rate control.

8. The steps include adjusting the grain reduction processing parameters in the current frame using the video encoding / rate control parameters and video quality measurement / RD performance parameters from the previous frame of the video asset, and / or The method according to claim 7, further comprising at least one step of simultaneously adjusting control parameters in both the grain reduction process and the grain-aware encoding / rate control using the grain quality / fidelity evaluation parameters.

9. The method according to claim 1, further comprising the step of streaming the level of the video asset in an encoding ladder based on the grain model parameters, the video quality measurement / RD performance parameters, and the grain quality / fidelity evaluation parameters by grain-aware streaming / decoding.

10. The aforementioned grain synthesis, The steps include: determining the grain synthesis parameters using the grain model parameters and the video encoding / rate control parameters together; and further adjusting the grain synthesis parameters using the video quality evaluation / RD performance parameters and the grain quality / fidelity evaluation parameters together; If the video encoding of the video asset is strongly quantized and the bitrate of the video asset is low, the grain intensity in the grain blending parameter is increased in order to create a visual masking effect in the overcompressed or oversmoothed regions of the video asset, and / or The method according to claim 1, further comprising the step of reducing the grain intensity in the grain synthesis parameter to avoid adding low-quality grain when the grain quality / fidelity evaluation parameter indicates a low-quality video asset.

11. The method according to claim 1, further comprising the step of enabling the grain composite to adjust the grain composite parameters with respect to the perceptual characteristics of the video asset, such that the rendering / display considering the grain provides the rendering / display parameters to the grain composite, and the rendering / display parameters include size, resolution, and / or viewer distance from the viewing device.

12. Grain-aware video quality evaluation / rate distortion analysis, A step of generating grain quality / fidelity measurements of the aforementioned video asset, A step of generating a structural fidelity measurement of the video asset, and / or The method according to claim 1, comprising at least one of the steps of generating an overall video quality measurement of the video asset.

13. An end-to-end system that optimizes video encoding and transmission while considering the grain of video assets, Equipped with at least one computing device, The computing device in question Determine the grain model parameters that represent the grain characteristics present in the aforementioned video asset. Prior to encoding the video asset, a grain reduction process is performed on the video asset according to grain reduction processing parameters that indicate a manner in which grain is removed from the video asset. After the grain reduction process, the video asset is encoded using grain-aware encoding / rate control according to the video encoding / rate control parameters. The aforementioned video asset is streamed and decoded. An end-to-end system configured to perform grain synthesis using grain synthesis parameters that indicate the characteristics of the grain to be applied to the video asset after the decoding, thereby reapplying grain to the video asset.

14. The aforementioned at least one computing device further, A video quality evaluation / rate distortion (RD) analysis is performed to compare the video asset before encoding and after decoding, and video quality measurement values / RD performance parameters are generated by the said video quality evaluation / RD analysis. Perform at least one grain quality evaluation on the video asset after grain synthesis to generate grain quality / fidelity evaluation parameters, and / or The end-to-end system according to claim 13, configured to control some or all of the rendering of the video asset to the end user by grain reduction processing, grain-aware encoding / rate control, streaming and decoding, grain synthesis, and / or grain-aware rendering / display, using the video quality measurement value / RD performance parameter and / or grain quality / fidelity evaluation parameter.

15. The aforementioned at least one computing device further, The end-to-end system according to claim 14, wherein the video asset is rendered to the end user by rendering / displaying with respect to grain, and the rendering / displaying with respect to grain is configured to provide information based on at least one of the grain model parameter, the grain reduction processing parameter, the video encoding / rate control parameter, the grain synthesis parameter, the video quality evaluation / RD analysis, and the grain quality / fidelity evaluation parameter.

16. The aforementioned at least one computing device further, Determine rendering / display parameters that indicate the rendering of the video asset to the end user. The end-to-end system according to claim 14, configured to determine the grain quality / fidelity evaluation parameters using the rendering / display parameters.

17. The grain model parameter represents at least one of grain size, grain density, and / or grain contrast, and the grain reduction process includes setting the grain reduction process using the grain model parameter. The video encoding / rate control parameter indicates at least one of the video bitrate, spatial resolution, frame rate or temporal resolution, encoding mode selection at each video / frame / local block level, and quantization step parameter at each video / frame / local block level, and the grain reduction process includes setting the grain reduction process using the video encoding / rate control parameter, and / or The end-to-end system according to claim 13, wherein the grain quality / fidelity evaluation parameter includes at least one of grain quality / fidelity evaluation, structural fidelity evaluation, and / or overall quality evaluation, and the grain reduction process includes setting the grain reduction process using the grain quality / fidelity evaluation parameter.

18. The end-to-end system according to claim 13, wherein the grain-aware encoding / rate control includes encoding the video asset using the video encoding / rate control parameters and the grain reduction processing parameters after the grain reduction processing.

19. The aforementioned at least one computing device further, The end-to-end system according to claim 13, wherein joint optimization considering grain is performed between the grain reduction process and the encoding / rate control that considers grain, and the system is configured to balance the grain reduction performance of the grain reduction process and the rate distortion performance of the encoding / rate control that considers grain.

20. The aforementioned at least one computing device further, Using the video encoding / rate control parameters and video quality measurement / RD performance parameters from the previous frame, the grain reduction processing parameters in the current frame are adjusted, and / or The end-to-end system according to claim 19, configured to simultaneously adjust control parameters in both the grain reduction process and the grain-aware encoding / rate control using the grain quality / fidelity evaluation parameters.

21. The end-to-end system according to claim 13, wherein at least one computing device is further configured to stream the level of the video asset in an encoding ladder based on the grain model parameter, the video quality measurement / RD performance parameter, and the grain quality / fidelity evaluation parameter by grain-aware streaming / decoding.

22. The aforementioned grain synthesis, The grain synthesis parameter is determined by jointly using the grain model parameter and the video encoding / rate control parameter, and further the grain synthesis parameter is adjusted by jointly using the video quality evaluation / RD performance parameter and the grain quality / fidelity evaluation parameter, and if the video encoding of the video asset is strongly quantized and the bitrate of the video asset is low, the grain intensity in the grain synthesis parameter is increased to produce a visual masking effect in the overcompressed or oversmoothed region, and / or The end-to-end system according to claim 13, further comprising reducing the grain intensity in the grain synthesis parameter to avoid adding low-quality grain when the grain quality / fidelity evaluation parameter indicates a low-quality video asset.

23. The end-to-end system according to claim 13, wherein the grain-aware rendering / display provides the rendering / display parameters to the grain composite, the rendering / display parameters including size, resolution, and / or viewer distance from the viewing device, so that the grain composite can adjust the grain composite parameters with respect to the perceptual characteristics of the video asset.

24. Grain-aware video quality evaluation / rate distortion analysis, To generate grain quality / fidelity measurements of the aforementioned video assets, To generate a structural fidelity measurement of the aforementioned video asset, and / or, The end-to-end system according to claim 13, comprising at least one of generating an overall video quality measurement of the video assets.

25. A processing method for distributing a source video asset having film grain via a content distribution network and optimizing the grain QoE of the image displayed on a viewing device within the bitrate range of the encoder, The video asset is received, and the video asset is the source video asset, The pre-distribution processing device generates an encoded video asset, the pre-distribution processing device reduces the film grain of the video asset, determines at least one film grain parameter, encodes the grain-reduced video asset, and embeds the at least one film grain parameter as metadata to generate the encoded video asset. The steps include: distributing the encoded video asset to a remote viewing device via the content distribution network; The post-distribution processing device decodes the distributed encoded video asset to generate a decoded video asset, synthesizes film grain using the embedded at least one film grain parameter, and adds the synthesized film grain to the decoded video asset to generate a decoded and grain-synthesized video asset. The steps include: performing a grain quality evaluation on the decoded and grain-composited video asset, and determining a grain quality measurement value using the post-distribution processing device based on the at least one film grain parameter and the parameters of the remote viewing device; A method comprising the steps of: feeding back the grain quality measurement value as a grain quality parameter to the pre-distribution processing device; the pre-distribution processing device performing film grain reduction processing on the video asset using the grain quality parameter; optimizing the encoding of the grain-reduced video asset within the bitrate profile of the remote viewing device's encoder; and optimizing the grain QoE of the decoded and grain-composited video asset displayed on the remote viewing device.

26. The method according to claim 25, wherein the pre-distribution processing device includes the steps of performing grain evaluation, grain modeling, grain reduction processing, and encoding rate control on the video asset.

27. The method according to claim 25, further comprising the step of generating grain model parameters for film grain reduction processing of the video asset.

28. The method according to claim 25, further comprising the steps of the post-distribution processing device performing a video quality evaluation of the decoded video asset to generate video quality parameters, performing a rate distortion analysis of the decoded video asset to generate rate distortion performance parameters, and providing the video quality parameters and the rate distortion performance parameters to the pre-distribution processing device for further optimization of encoding of the video asset having film grain.

29. The method according to claim 28, wherein both the video quality parameter and the grain quality parameter are used in a joint optimization process to adjust the grain reduction process and the encoding that takes grain into consideration.

30. An end-to-end system that receives a video asset having film grain, processes the video asset for distribution to a remote viewing device, and optimizes the grain QoE of the displayed image at the encoder bitrate on the viewing device, A pre-distribution processing unit that reduces the film grain of the video asset, determines at least one film grain parameter, encodes the grain-reduced video asset with an encoder, and embeds the at least one film grain parameter as metadata to generate an encoded video asset; A content network that receives and distributes the aforementioned encoded video assets, The post-distribution processing unit includes: receiving the encoded video asset from the content distribution network; decoding the encoded video asset to generate a decoded video asset; synthesizing film grain using the embedded at least one film grain parameter; adding the synthesized film grain to the decoded video asset to generate a decoded and grain-synthesized video asset; and performing a grain quality evaluation of the decoded and grain-synthesized video asset to determine the grain quality measurement value. The end-to-end system is configured such that the post-distribution processing unit is configured to feed back the grain quality measurement values ​​as grain quality parameters to the pre-distribution processing unit, the pre-distribution processing unit uses the grain quality parameters to perform film grain reduction processing on the video asset, optimizes the encoding of the grain-reduced video asset within the bitrate profile of the remote viewing device's encoder, and optimizes the grain QoE of the decoded and grain-composited video asset displayed on the remote viewing device.

31. A pre-delivery processing unit for processing video assets having film grain for optimized encoding, A process for receiving the aforementioned video asset and grain quality parameters, A process to reduce the film grain of the aforementioned video asset and determine at least one film grain parameter, A process to encode the grain-reduced video asset and embed the at least one film grain parameter as metadata to generate an encoded video asset, A process for outputting the encoded video asset to a content distribution network, A pre-delivery processing unit is configured to optimize the encoded video asset using the grain quality parameter, which is a grain quality measurement value output from a post-delivery processing unit, and to perform at least one of the following processes: decoding and grain quality evaluation of the grain-composited video asset, which are determined by the post-delivery processing unit based on the encoded video asset received from the content delivery network.

32. The pre-delivery processing unit according to claim 31, further configured to optimize encoder rate control and the grain quality measurement values.

33. A post-delivery processing unit that receives output from a content delivery network, which is a delivered encoded video asset having at least one film grain parameter as metadata, The process involves decoding the aforementioned encoded video asset to generate a decoded video asset, A process to synthesize grain using the decoded video asset and the embedded at least one film grain parameter, and to add the synthesized grain to the decoded video asset to generate a decoded and grain-synthesized video asset, A process to perform a grain quality evaluation on the aforementioned synthetic grain and determine the grain quality measurement value, A process for outputting the aforementioned grain quality measurement values ​​as grain quality parameters, A post-delivery processing unit is configured to receive the grain quality parameters into a pre-delivery processing unit and to perform at least one of the following processes: generate an encoded video asset that optimizes the grain QoE of the decoded and grain-composited video asset to be displayed.

Images