Film particle decomposition and processing in transport system

Abstract

In some embodiments, source content is received. A method decomposes source content into a plurality of components. The constituent parts comprise first film particle content and second film particle content. And processing the first film particle content, wherein the first film particle content is processed by an encoder system or a film particle synthesis system. The method communicates information of the first film particle content processed by the encoder system or the film particle synthesis system to the client device. The second film particle content is not communicated to the client device.

Description

[Technical Field]

[0001] This invention generally relates to content processing. [Background Technology]

[0002] Film grain can be a prominent feature in video production, such as in videos shot with traditional film cameras (e.g., programs or films produced in the film industry). Film grain can be a visually pleasing noise that can be used to express artistic intent. However, including film grain in video streamed to client devices over a network can present technical challenges, such as the potentially high bitrate required to encode video containing film grain. This leads to high bandwidth demands and may be incompatible with streaming environments. To save bandwidth, film grain can be removed from the video before it is streamed to client devices. However, viewers may still be dissatisfied with the visual quality of video without film grain.

[0003] One technique involves synthesizing film grain on the client device to add it to decoded video frames, thereby mimicking the film grain in the source video. This process reintroduces film grain into the video. However, film grain synthesis can be a challenging task, making it difficult to accurately replicate the original film grain in the source video. Furthermore, film grain synthesis can add unwanted effects to video frames, such as new visual artifacts. [Summary of the Invention]

[0004] One aspect of the present invention provides a method comprising: receiving source content; decomposing the source content into a plurality of components, wherein one of the plurality of components includes first film grain content and second film grain content; processing the first film grain content, wherein the first film grain content is processed by an encoder system or a film grain synthesis system; and transmitting information of the first film grain content processed by the encoder system or the film grain synthesis system to a client device, wherein the second film grain content is not transmitted to the client device. [Attached Image Description]

[0005] The accompanying drawings are for illustrative purposes and are intended only to provide possible structural and operational embodiments of the disclosed inventive systems, apparatuses, methods, and computer program products. These drawings do not limit any changes in form and detail that may be made by those skilled in the art without departing from the spirit and scope of the disclosed embodiments.

[0006] Figure 1 A simplified system for generating film grain for video, according to some embodiments, is shown.

[0007] Figure 2 An example of configuration #1 according to some implementation methods is shown.

[0008] Figure 3 An example of configuration #2 according to some implementation methods is shown.

[0009] Figure 4 Configuration #3 according to some implementation methods is shown.

[0010] Figure 5 A simplified flowchart of a method for performing a film grain decomposition process according to some embodiments is shown.

[0011] Figure 6 A simplified flowchart of a code-based decomposition process according to some implementations is shown.

[0012] Figure 7 An example of a prediction network that can be used to perform decomposition according to some implementations is shown.

[0013] Figure 8 A second example of determining encoder-friendly film grain and synthetic-friendly film grain according to some implementation methods is shown.

[0014] Figure 9 Examples of using frequency domain selection components are shown according to some implementation methods.

[0015] Figure 10 An example of a computing device according to some implementation methods is illustrated.

Detailed Implementation Methods

[0016] This document describes techniques for video processing systems. In the following description, numerous embodiments and specific details are set forth for purposes of explanation to provide a thorough understanding of some embodiments. Some embodiments as defined by the claims may include some or all of the features in these examples, either alone or in combination with other features described below, and may also include variations and equivalents of the features and concepts described herein.

[0017] System Overview

[0018] The system uses a film grain decomposition process to improve the encoding, transmission, and display of content such as video. This system adaptively decomposes film grain in source content into different components, such as encoding-friendly film grain, synthesis-friendly film grain, and encoding-unfriendly / synthesis-unfriendly film grain. Encoding-friendly film grain can be encoded using the regular video encoding pipeline of the transmission system. That is, grain-free content (e.g., clean content) and encoding-friendly film grain can be encoded together by the encoder and then decoded together. Due to its encoding-friendly nature, film grain is preserved through the encoding process.

[0019] Synthesized film grain can be modeled by a server-side film grain compositing system and then synthesized by a film grain compositing system on the client device. Unfriendly encoding / unfriendly synthesis film grain may be unfriendly encoding film grain, unfriendly synthesis film grain, or both. Unfriendly encoding / unfriendly synthesis film grain may be excluded from transmission, for example, it may not be encoded in the encoding process or synthesized in the film grain compositing system process. On the client device, if friendly encoding film grain, friendly synthesis film grain, or both are received, they may be used to generate the final decoded content displayed on the client device. For example, clean content and friendly encoding film grain are combined with friendly synthesis film grain to produce the final content. In other implementations, only clean content and friendly encoding film grain are used to produce the final content. In yet another implementation, only clean content and friendly synthesis film grain are used to produce the final content.

[0020] Using the above decomposition, the encoding, transmission, and display of content can be improved. For example, by eliminating unfriendly encoding or composite film grain, encoded content can use a lower bitrate. For instance, encoding unfriendly film grain might require a very high bitrate to preserve the film grain throughout the encoding process. If the bitrate is not high enough, the encoder might naturally remove the film grain during encoding. Therefore, if the system wants to preserve film grain in the decoded video, a higher bitrate is needed. For example, if film grain content and clean content are encoded at 10 megabits per second (Mbps), the encoder might encode the video, but the decoded video might not contain film grain. However, at 20 Mbps, the decoded video might contain film grain. However, a higher bitrate might require more bandwidth for transmission. Using a higher bitrate can place a significant burden on the encoder and system resources. Furthermore, the transmission process might also be improved because available bandwidth may be limited in some cases. This could lead to rebuffering during playback because there isn't enough content available for playback on the client device.

[0021] Furthermore, excluding composite grain artifacts can improve the final content display on client devices. For example, composite grain artifacts can create visual clutter within the composited grain or may deviate from the original artistic intent. When composite grain artifacts are not used, these artifacts may not be present in the final decoded video. Similarly, when composite grain artifacts are not used, film grains that differ from the original film grain may also be absent.

[0022] system

[0023] Figure 1 A simplified system 100 for generating film grain for video is illustrated according to some embodiments. System 100 includes a server system 102 that provides a service for transmitting content to a client device 104. Although the discussion may use both server system 102 and client device 104 to generate film grain, the described process can be used in different ways. For example, film grain composition may be performed only on server system 102, only on client device 104, or distributed across different devices in different ways.

[0024] Content providers may operate content delivery system 106, providing content delivery services that allow entities to request and receive media content. Content providers can use content delivery system 106 to coordinate the distribution of media content to clients. Media content can be of different types, such as video, audio, etc. Video can be programs, such as movies, television programs, advertisements, etc. Server system 102 may receive source video containing different types of content, such as video, audio, or other types of content information. Encoder system 108 may encode the source video to create an encoded version of the source video, which is then transmitted as a video bitstream to client device 104. Although video transmission from content delivery system 106 to client device 104 is shown, content delivery system 106 may use a content delivery network (not shown) to transmit video to client device 104. Although source video is described, any content that may contain film grain, such as images, can be used. Different types of encoders can be used, such as encoders using different encoding specifications.

[0025] Client device 104 may include various computing devices, such as smartphones, living room devices, televisions, set-top boxes, tablets, etc. Client device 104 includes interface 116 and media player 110 for playing content, such as video. At client device 104, decoder system 112 receives the video bitstream and decodes it into decoded video frames. The decoded video frames are displayed by media player 110 on interface 116.

[0026] Film grain can be present in some videos, such as television programs and movies shot with traditional film cameras. Film grain can be a visible texture or pattern appearing in video shot on film. Film grain can also manifest as noise in video. Retaining film grain from the source video in the video bitstream can be challenging for several reasons. For example, the bitrate of the video bitstream may increase when film grain is present in the original source video. Furthermore, the randomness of film grain in the source video can cause random variations in the bitrate, as the frame bitrate increases when film grain is encountered, which can affect the transmission of the video bitstream to the client device 104. This randomness can affect the playback experience, as changes in the bitrate during playback can lead to rebuffering. Additionally, the randomness of film grain in video makes it difficult to predict when (e.g., which frames) and where (e.g., where within a frame) film grain will appear in the source video using prediction schemes in video coding specifications. This can lead to inefficient compression.

[0027] In film grain synthesis, film grain can be removed from the source video before encoding, and the video bitstream may not include film grain from the source video when sent from server system 102 to client device 104. Furthermore, digital cameras may not produce film grain in the video, or frames of the source video may not contain film grain, but the system may still add film grain to the video. Given the above, the video bitstream may not include film grain from the source video; however, film grain synthesis system 114-1 can model film grain using film grain parameters, and film grain synthesis system 114-2 can synthesize film grain that can be added to the decoded video frames. However, conventional film grain synthesis can encounter problems. Typically, the parameters of the film grain model may be used to describe the film grain that should be added to the decoded video. While these models may add film grain to the decoded video frames, the reconstructed film grain may be undesirable; for example, the film grain may differ significantly from the original artistic intent in the original source video, or it may introduce visual artifacts not present in the source video.

[0028] To overcome the aforementioned drawbacks, the system uses film grain decomposition to break down film grain into distinct components. In some implementations, these components include coded friendly film grain, composite friendly film grain, and coded unfriendly / composite unfriendly film grain. While these components are described, other components may also be used. Coding friendly film grain might be film grain that can be encoded at a bit rate below a threshold without being removed. Composite friendly film grain might be composited while preserving the original artistic intent or might not introduce artifacts. Coding unfriendly film grain might require encoding at a bit rate above a threshold to appear in the decoded video. Composite unfriendly film grain might be composited but without preserving the original artistic intent or might introduce artifacts. In film grain composition, composite unfriendly film grain may include artifacts, which might be annoying or unpleasant repetitive patterns. Artifacts may also be the result of unstable film grain composition in the temporal domain. For example, in some frames, the composited film grain may be heavy, while in other frames, the composited film grain may be very light. This can also lead to an unpleasant appearance.

[0029] Content analyzer 118 may analyze the source content to be encoded. Content analyzer 118 may decompose the source content into different components. These components may include clean content without grain, and components with different grain patterns, including encoded grain-friendly grain, synthetic grain-friendly grain, and encoded / synthesized grain-unfriendly grain. As mentioned above, encoded / synthesized grain-unfriendly grain may be encoded grain-unfriendly grain, synthetic grain-unfriendly grain, or both encoded grain-unfriendly and synthetic grain-unfriendly grain.

[0030] The source content can be processed using different configurations. The use case configuration system 120 may determine the configurations used to process different components. This determination may be dynamic, based on characteristics of the source content. For example, the use case configuration system 120 may use characteristics of the source content to determine the configuration. In other implementations, the use case configuration system 120 may use settings to determine the configuration. For example, the source content may be associated with a configuration. The source content is then processed using the selected configuration.

[0031] The following describes different configurations for processing the components of film grain.

[0032] Configuration #1

[0033] Figure 2 An example of configuration #1 according to some implementations is shown. In configuration #1, a higher bit rate is enabled to encode encoder-friendly film grain, and film grain synthesis is supported to synthesize synthetic friendly film grain at client device 104. Configuration #1 includes a server side associated with server system 102 and a client side associated with client device 104.

[0034] Content analyzer 118 receives source video. The source video may be video that will be provided and encoded on a content delivery service, but it may also be used for other purposes. The video can be encoded in multiple different profiles for a profile ladder, where these profiles may be encoded at different bitrates or qualities, or both.

[0035] Content analyzer 118 can analyze the content and output a decomposition of the source video. For example, at 202, it outputs clean content and encoded friendly film grain c+g1. At 204, it outputs encoded unfriendly / synthesized unfriendly film grain g2. At 206, it outputs synthesized friendly film grain g3. Here, each component may be an image containing relevant information. For example, clean content and encoded friendly film grain are images containing frame content with encoded friendly film grain. Synthesized friendly film grain is an image with synthesized friendly film grain. Encoded unfriendly / synthesized unfriendly film grain is an image with encoded unfriendly / synthesized unfriendly film grain. Images can be output for each frame of the source content and processed as described herein.

[0036] Configuration #1 may retain some film grain, which can preserve artistic film grain in the decoded video. For example, some film grain can be encoded, and some film grain can be modeled and composited using film grain compositing. Using film grain compositing can help reduce the bit rate required to encode film grain by removing some film grain from the encoding process.

[0037] In this process, encoder system 108 can encode clean content and encoder-friendly film grain into encoded video. The resulting encoded video includes both clean content and encoder-friendly film grain.

[0038] Film grain compositing system 114-1 can model composite friendly film grain using film grain parameters that describe composite friendly film grain. Film grain parameters describe the film grain that should be composited and added to the decoded video at client device 104. Film grain parameters may describe various characteristics, such as the brand model of the source content, film grain size, density, and amplitude, parameters of the autoregressive film grain generator, parameters of the frequency film grain descriptor, or other characteristics. Film grain parameters can be conveyed via video bitstream, transmitted via sidechannels, or self-derived by film grain compositing system 114-2. Encoded video and film grain parameters can be transmitted from the server side to the client side. Different methods can be used to represent the video bitstream.

[0039] On the client side, the encoded video is passed to the decoder system 112, which decodes the encoded video into decoded frames. Different decoders may be used in this process, such as different decoders using different encoding specifications. This results in a decoded video with clean content and encoder-friendly film grain (c+g1).

[0040] Film grain synthesis system 114-2 receives film grain parameters and decoded video. Then, film grain synthesis system 114-2 can synthesize composite friendly film grain. During the synthesis process, film grain synthesis system 114-2 can collect features from the decoding process, such as syntactic features from the video bitstream and / or decoder-related features from decoder system 112. Syntactic features and decoder-related features are generally referred to as decoding features. Syntactic features can refer to information about the video bitstream, such as information that needs to be decoded. Examples of syntactic features include quantization parameters, frame type (I / B / P frame), motion vectors, etc. Syntactic features can be sent in the video bitstream or via a separate control channel. Decoder-related features can be features determined based on the decoding of the video bitstream by decoder system 112. Examples of decoder-related features include block partitioning results, quantization residual distribution (e.g., the error distribution between the original frame and the quantized version of the frame), etc. Although a particular feature is described as part of a syntactic feature or a decoder-related feature, the same feature may be found in both syntactic features and decoder-related features, such as quantization parameters, frame type, etc.

[0041] The film grain synthesis system 114-2 outputs synthesized friendly film grains g3. The synthesis of film grains depends on parameters determined from film grain modeling. The film grain synthesis system 114-2 can synthesize film grains based on film grain parameters and features.

[0042] The combination of decoded video and composite-friendly film grain produces decoded video with composite film grain (s' = c + g1 + g3). Composite-unfriendly / encoded unfriendly film grain g2 may not be encoded or composited. Here, encoded unfriendly / composite-unfriendly film grain may be discarded and not included in the video with composite film grain s'. Excluding encoded unfriendly / composite-unfriendly film grain may result in a lower bitrate for the encoded video than when encoded unfriendly film grain was encoded. Furthermore, excluding composite-unfriendly film grain may remove film grain that causes artifacts or does not look like the original film grain.

[0043] Configuration #2

[0044] Figure 3 An example of configuration #2 according to some implementations is shown. In configuration #2, a relatively low bit rate can be used, and film grain synthesis is supported. A lower bit rate can be used because film grain may not be encoded by the encoder system 108. That is, a higher bit rate may be needed to fully reproduce film grain in the decoded video.

[0045] Here, the content analyzer 118 decomposes the source video into clean content c at 302, synthesizes unfriendly film grain g2 at 304, and synthesizes friendly film grain g3 at 306. The encoder system 108 can encode the clean content into encoded video. The film grain synthesis system 114-1 can model the synthesized friendly film grain as film grain parameters. Synthesized unfriendly film grain can be discarded. In this case, there may be no encoder-unfriendly film grain. However, if encoder-unfriendly film grain exists, it can also be discarded.

[0046] Encoded video and film grain parameters can be transmitted from the server side to the client side. Decoder system 112 decodes the encoded video into a decoded video with clean content c. Film grain compositing system 114-2 uses the film grain parameters to compose composite-friendly film grain g3. The clean content is combined with the composite film grain to output a video with composite film grain (s' = c + g3). In this configuration, because encoder system 108 does not encode encoder-unfriendly film grain, the available bitrate is lower than in configuration #1. For example, for 1080p video, configuration #1 might use 6.5 Mbps and 5.5 Mbps for the two corresponding profiles. However, for configuration #2, bitrates of 3 Mbps and 1 Mbps might be used. As mentioned above, encoding encoder-unfriendly film grain may require a higher bitrate. However, composite-friendly film grain can be synthesized, and film grain can be included in the video. This film grain may be closer to the original artistic intent, less prone to artifacts, and film grain can still be included in the displayed video.

[0047] Configuration #3

[0048] Figure 4 Configuration #3, according to some implementations, is shown. In configuration #3, a relatively high bitrate is used, and film grain needs to be preserved in the encoded video. Film grain synthesis can be disabled at client device 104.

[0049] Content analyzer 118 can decompose the source video into clean content and encoder-friendly film grain c+g1 at 402. At 404, encoder-unfriendly film grain and film grain / synthesized unfriendly film grain g2 are shown. Encoder system 108 encodes the clean content and encoder-friendly film grain. Encoder-unfriendly film grain can be discarded, and film grain synthesis is not performed. Synthesized unfriendly film grain may not exist, but if it does, it can also be discarded. Decoder system 112 receives the encoded video and decodes it into a decoded video that includes the clean content and encoder-friendly film grain c+g1.

[0050] In the process described above, encoder-unfriendly film grain and composite-unfriendly film grain can be discarded. This eliminates encoder-unfriendly film grain that might require high bitrate encoding. Furthermore, composite-unfriendly film grain that may not represent the original artistic intent or may introduce artifacts is not composited. However, encoder-friendly film grain is encoded to preserve certain aspects of the film grain in the original source video.

[0051] In different configurations, the source content is broken down into components. Different methods for breaking down source content into different components will be described below. Although the processes that can be used to break down source content will be described below, the implementation is not limited to these processes.

[0052] Film grain decomposition

[0053] Figure 5 A simplified flowchart 500 is shown of a method for performing a film grain decomposition process according to some embodiments. At 502, server system 102 receives source content. The source content can be any content to be encoded.

[0054] At position 504, content analyzer 118 analyzes the source content. The content analyzer can analyze the source content to determine its characteristics. Some characteristics may include content type; for example, the content may be categorized into different classes. For example, categories might include cartoons, movies, action movies, etc. Content analysis may include spatial complexity analysis, time complexity analysis, color, brightness, content type (such as animation, sports, etc.), or analysis of other characteristics.

[0055] At point 506, use case configuration system 120 determines the configuration. For example, use case configuration system 120 might select one of the configurations discussed above based on characteristics of the source content or other features. For instance, a particular configuration might be more suitable for the characteristics of the source content. However, the configuration might not be entirely determined by the content. For example, features such as specified requirements, network conditions (e.g., available bandwidth), and whether the client side supports film grain composition might be considered. If the client side does not support film grain composition, only configuration #3 is used. If the client side supports film grain composition, configurations #1 and #2 might be used. For example, system 100 might calculate the space and time complexity of the content. If the complexity is high, the content requires a higher bitrate to achieve good quality. Therefore, use case configuration system 120 might choose not to encode film grain as configuration #3. This allows more bits to be allocated for encoding complex content. If the content is relatively simple, or film grain is an important feature of the content, use case configuration system 120 might choose configuration #1.

[0056] At 508, the content analyzer 118 generates components from the decomposition of the source content. The process of generating components may use different methods. For example, encoding-based film grain decomposition may be used to decompose the source content into encoding-friendly and encoding-unfriendly film grains. Alternatively, pre-analysis-based film grain decomposition may be performed. This decomposition may decompose the source content into encoding-friendly, encoding-unfriendly, synthetic-friendly, or synthetic-unfriendly film grains. In some implementations, encoding-based film grain decomposition may be used for configuration #1 or configuration #3. Pre-analysis-based film grain decomposition may be used for configuration #1, configuration #2, or configuration #3. These two processes may be used individually or in combination. These processes will be described in more detail below.

[0057] At point 510, system 100 uses the configuration processing component. The processing may be performed as described above.

[0058] The following will describe different types of decomposition, but the implementation methods are not limited to these decomposition processes.

[0059] Encoding-based decomposition

[0060] Figure 6 A simplified flowchart 600 of an encoding-based decomposition process according to some embodiments is shown. At 602, content analyzer 118 receives source video. At 604, content analyzer 118 determines whether the source video contains film grain. Content analyzer 118 may use different processes to determine whether the source video contains film grain. Film grain detection can be performed directly on the source video. For example, film grain detection can be performed using a denoised representation of the source content with noise removed. The original source content and the denoised representation can be analyzed to determine differences. These differences can indicate whether the source content contains film grain; for example, differences can be analyzed, and if differences exist, it indicates the presence of film grain. A threshold can be applied to determine the amount of noise required to determine the presence of film grain. Furthermore, content analyzer 118 may receive input indicating whether the source video contains film grain; for example, an identifier can be set to indicate whether the source video contains film grain. For example, a value "1" indicates the presence of film grain, and a value "0" indicates the absence of film grain. Additionally, the content analyzer 118 may use a predictive network to analyze the source content to determine whether the content contains film grain.

[0061] At 606, if the source content does indeed contain film grain, the encoder system 108 encodes the source content. Encoding the source content generates encoded source content. The encoded source content can be encoded using setting S0. In some implementations, different settings may be used to encode the source content. For example, different settings may be based on a profile of the source content that should be encoded. In some implementations, settings S1 to Sn can be used as a profile ladder. The available profiles can be limited based on the bitrate. For example, profiles S1 to Sx with bitrates below a threshold can be used. In some examples, settings S1 to Sx may include 1080p (1920x1080) resolution, where the target bitrates for S1 to Sx are 8500kbps, 7000kbps, 5500kbps, and 3000kbps, respectively. Encoding S0 may be a setting with a bitrate higher than that of settings S1 to Sx. This encoding may be first-pass encoding, which removes unfriendly film grain. An example of setting S0 could be a target encoding bitrate of 30 Mbps for 1080p resolution. A high bitrate can remove encoding-unfriendly film grain because film grain is typically a high-frequency signal. When encoding, the encoded video may remove some of the high-frequency signals from the film grain. Here, the encoder system 108 uses a high bitrate for encoding to maintain quality in subsequently encoded source content, where the encoded source content does not include encoder-unfriendly film grain. Even when encoding at very high bitrates, the encoder cannot retain encoder-unfriendly film grain. Retaining encoder-unfriendly film grain through the encoding process is limited by the bitrate available for encoding due to the bandwidth constraints of content delivery.

[0062] At 608, encoder system 108 then encodes the encoded source content. For example, the encoded source content might be encoded using settings S1 to Sx. This encoded source content does not contain encoder-friendly film grain. Therefore, this process can remove the unfriendly film grain in configurations #1 and #3.

[0063] If the source content does not contain film grain, then at 610, the encoder system 108 encodes the source content to generate the source content components. For example, the source content is encoded using settings S1 to Sx.

[0064] Based on pre-analysis of film grain decomposition

[0065] The second process can use a prediction network to determine the components. Figure 7An example of a prediction network 702, which can be used to perform decomposition according to some embodiments, is shown. The prediction network 702 receives source content. The prediction network 702 can be trained to remove encoded / synthetic unfriendly film grain g2. Furthermore, the prediction network 702 can be trained to decompose the source content into clean content plus encoder-friendly film grain c+g1 and synthetic-friendly film grain g3. Other components, such as clean content only, can also be determined.

[0066] The prediction network 702 can be trained based on supervised learning, where it takes as input samples of source content and analyzes the predicted components to determine whether the output components are similar to the true values. The parameters of the prediction network 702 can be adjusted based on the differences. In some implementations, the collected training dataset contains components including clean content, clean content with encoder-friendly film grain, encoder-unfriendly film grain, synthetic-friendly film grain, and synthetic-unfriendly film grain. The prediction network 702 is then trained to predict these components.

[0067] Other methods can be used to determine encoder-friendly film grain and synthetic-friendly film grain. Figure 8 A second example of determining encoder-friendly film grain and synthesized friendly film grain according to some implementations is shown. Source content containing film grain c+g is input to a denoiser 802. The denoiser 802 may remove the film grain, resulting in clean content c. The clean content is then removed from the source content, resulting in film grain g.

[0068] The prediction network 702 then analyzes the film grain. The prediction network 702 can be trained to decompose the film grain into multiple components, such as encoder-friendly film grain g1, encoder-unfriendly / synthetic-unfriendly film grain g2, and synthetic-friendly film grain g3. The encoder-friendly film grain is then added to the denoised video, resulting in clean content plus encoder-friendly film grain c+g1.

[0069] In the third process, digital signal processing can be used to decompose the source content. Only a portion of the frequency domain subbands may have a significant impact on the subjective quality of film grain as perceived by a human viewer. The source content within the film grain can be divided into several subbands. Appropriate subbands can be selected, which may include encoder-friendly and composite-friendly film grain. The remaining subbands may include encoder-unfriendly and composite-unfriendly film grain. Figure 9 Examples of using frequency domain selection components are shown according to some implementations. At 902, a steerable pyramid can be used to transform the source content into different subbands in the frequency domain. In some implementations, the steerable pyramid 902 may be a spatial domain steerable pyramid.

[0070] The input to the controllable pyramid 902 can be source content or film grain extracted from the source content. This example can use source content. Subbands are shown at 904. Some subbands may be associated with different types of film grain. In some implementations, high-frequency subbands may include encoder-unfriendly film grain because high-frequency signals can be challenging for encoding the encoder system 108. At 906, this subband may be classified as encoding-unfriendly film grain and synthesized unfriendly film grain g2. Encoding-friendly film grain may come from less frequent subbands. For example, at 908, this subband is classified as clean content plus encoding-friendly film grain c+g1. Lower-frequency film grain may be easier to generate using film grain synthesis. At 910, these subbands are classified as synthesized-friendly film grain g3.

[0071] Although a controllable pyramid has been described, other methods can be used to generate subbands, such as the fast Fourier transform, Gaussian pyramid, frequency domain steerable pyramid, or Laplacian pyramid.

[0072] in conclusion

[0073] Therefore, different processes can be used to break down the source content into its components. These components can then be encoded or composited in different configurations. Using decomposition and configuration can potentially improve the encoding / decoding process by using a lower bitrate, which may also improve the transmission process. Furthermore, the display of the content can be improved by excluding film grain that does not represent the original artistic intent or introduces artifacts.

[0074] system

[0075] Figure 10An example of a computing device according to some embodiments is illustrated. According to various embodiments, a system 1000 suitable for implementing the embodiments described herein includes a processor 1001, a memory module 1003, a storage device 1005, an interface 1011, and a bus 1015 (e.g., a PCI bus or other interconnection). The system 1000 may operate as a variety of devices, such as any other device or service described herein. Although a particular configuration is described, various alternative configurations may exist. The processor 1001 can perform the operations described herein. Instructions for performing such operations may be embodied in the memory module 1003, on one or more non-transitory computer-readable media, or on some other storage device. Various specially configured devices may also be used to replace or supplement the processor 1001. The memory module 1003 may be random access memory (RAM) or other dynamic storage devices. The storage device 1005 may include a non-transitory computer-readable storage medium that stores information, instructions, or combinations thereof, such as instructions that, when executed by the processor 1001, configure or enable the processor 1001 to perform one or more operations of the methods described herein. Bus 1015 or other communication components may support information communication within system 1000. Interface 1011 may be connected to bus 1015 and configured to send and receive data packets over a network. Examples of supported interfaces include, but are not limited to: Ethernet, Fast Ethernet, Gigabit Ethernet, Frame Relay, cable, Digital Subscriber Line (DSL), Token Ring, Asynchronous Transfer Mode (ATM), High-Speed ​​Serial Interface (HSSI), and Fiber Distributed Data Interface (FDDI). These interfaces may include ports suitable for communication with appropriate media. They may also include a dedicated processor and / or volatile RAM. The computer system or computing device may include or communicate with a monitor, printer, or other suitable display device to provide the user with any results mentioned herein.

[0076] Any disclosed implementation may be embodied in various types of hardware, software, firmware, computer-readable media, and combinations thereof. For example, some of the techniques disclosed herein may be implemented, at least in part, through non-transitory computer-readable media containing program instructions, state information, etc., to configure a computing system to perform the various services and operations described herein. Examples of program instructions include machine code (such as code generated by a compiler) and higher-level code that can be executed by an interpreter. Instructions may be embodied in any suitable language, such as Java, Python, C++, C, HTML, any other markup language, JavaScript, ActiveX, VBScript, or Perl. Examples of non-transitory computer-readable media include, but are not limited to: magnetic media such as hard disks and magnetic tapes; optical media such as flash memory, optical discs (CDs), or digital versatile optical discs (DVDs); magneto-optical media; and other hardware devices such as read-only memory (“ROM”) devices and random access memory (“RAM”) devices. Non-transitory computer-readable media may be any combination of such storage devices.

[0077] In the foregoing description, various techniques and mechanisms are described in the singular for clarity. However, it should be noted that, unless otherwise stated, some implementations include multiple iterations of a technique or multiple instances of a mechanism. For example, a system may use a processor in various contexts, but multiple processors may be used while still remaining within the scope of this disclosure, unless otherwise stated. Similarly, various techniques and mechanisms may be described as including connections between two entities. However, a connection does not necessarily mean a direct, unobstructed connection, as various other entities (e.g., bridges, controllers, gateways, etc.) may reside between two entities. Some implementations may be embodied in a non-transitory computer-readable storage medium for use by or in conjunction with an instruction execution system, apparatus, system, or machine. The computer-readable storage medium contains instructions for controlling a computer system to perform the methods described in some implementations. The computer system may include one or more computing devices. The instructions, when executed by one or more computer processors, may be configured or operable to perform the instructions described in some implementations.

[0078] As used in the description herein and throughout the appended claims, “a” and “the” include plural expressions unless the context clearly indicates otherwise. Furthermore, as used in the description herein and throughout the appended claims, “in” means both “in” and “on” unless the context clearly indicates otherwise.

[0079] The foregoing description illustrates various implementation methods and aspects of how some implementation methods can be carried out. The above implementation methods and embodiments should not be considered as the only possible implementation methods, and are presented to illustrate the flexibility and advantages of some implementation methods as defined by the appended claims. Based on the foregoing disclosure and the appended claims, other arrangements, implementations, practices, and equivalents may be employed without departing from the scope of the invention as defined by the claims.

Claims

1. A method comprising: Receive source content; The source content is decomposed into multiple components, wherein the components of the multiple components include first film grain content and second film grain content; Processing the first film grain content, wherein the first film grain content is processed by an encoder system or a film grain synthesis system; as well as The information of the first film grain content processed by the encoder system or the film grain synthesis system is transmitted to the client device, wherein the second film grain content is not transmitted to the client device.

2. The method of claim 1, wherein decomposing the source content into a plurality of components comprises: The source content is decomposed into content without film grain and the first film grain content.

3. The method according to claim 2, wherein: The encoder system encodes the content without film grain and the content with the first film grain, and The second film grain content is not processed through the encoder system or the film grain synthesis system.

4. The method according to claim 3, wherein: The source content is decomposed into third film grain content, and The third film grain content is processed by the film grain synthesis system.

5. The method according to claim 4, wherein the second film grain content is encoder-unfriendly film grain content or synthesized unfriendly film grain content.

6. The method of claim 1, wherein decomposing the source content into a plurality of components comprises: The source content is decomposed into content without film grain.

7. The method according to claim 6, wherein: The encoder system encodes the film-grain-free content, and The first film grain content is processed by the film grain synthesis system.

8. The method of claim 7, wherein the second film grain content is encoder-unfriendly film grain content or synthesized unfriendly film grain content.

9. The method of claim 1, wherein decomposing the source content into a plurality of components comprises: The source content is decomposed into content without film grain and the first film grain content, wherein the content without film grain and the first film grain content are encoded by the encoder system.

10. The method of claim 9, wherein the second film grain content is not processed by the encoder system or the film grain synthesis system.

11. The method of claim 10, wherein the second film grain content is encoder-unfriendly film grain content or synthesized unfriendly film grain content.

12. The method of claim 1, wherein decomposing the source content into a plurality of components comprises: The source content is encoded using the encoder system with a first setting, wherein the first setting generates encoded content that includes the first film grain content but excludes the second film grain content.

13. The method of claim 12, wherein processing the first film grain content comprises: The encoded content, including the content of the first film grain, is encoded using a set of settings that do not include the first setting. as well as The encoded content is transmitted to the client device.

14. The method of claim 1, wherein decomposing the source content into a plurality of components comprises: The source content is input into the prediction network; as well as The prediction network predicts the content of the first film grain and the content of the second film grain.

15. The method of claim 1, wherein decomposing the source content into a plurality of components comprises: Multiple sub-bands are generated based on different frequencies of the source content; as well as Select the first film grain content from one or more first sub-bands of the plurality of sub-bands, and select the second film grain content from one or more second sub-bands of the plurality of sub-bands.

16. The method according to claim 1, wherein: The decoder system decodes the content without film grain and the information of the first film grain content to form the decoded content without film grain and the decoded content of the first film grain. The third film grain content is synthesized by the film grain synthesis system of the client device, and The decoded content without film grain and the decoded first film grain content are combined with the third film grain content to form decoded content, which is displayed on the client device.

17. The method according to claim 1, wherein: The decoder system decodes the information of the source content without film grain to form the decoded content without film grain. The film grain synthesis system of the client device synthesizes the information of the first film grain content to form the synthesized first film grain content, and The synthesized first film grain content is combined with the decoded film grain-free content on the client device to form decoded content, which is displayed on the client device.

18. The method according to claim 1, wherein: The decoder system decodes the source content without film grain and the information of the first film grain content to form decoded content, which is displayed on the client device.

19. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions that, when executed by a computing device, enable the computing device to perform the following operations: Receive source content; The source content is decomposed into multiple components, wherein the components of the multiple components include first film grain content and second film grain content; Processing the first film grain content, wherein the first film grain content is processed by an encoder system or a film grain synthesis system; as well as The information of the first film grain content processed by the encoder system or the film grain synthesis system is transmitted to the client device, wherein the second film grain content is not transmitted to the client device.

20. An apparatus comprising: One or more computer processors; as well as A computer-readable storage medium including instructions for controlling the one or more computer processors to perform the following operations: Receive source content; The source content is decomposed into multiple components, wherein the components of the multiple components include first film grain content and second film grain content; Processing the first film grain content, wherein the first film grain content is processed by an encoder system or a film grain synthesis system; as well as The information of the first film grain content processed by the encoder system or the film grain synthesis system is transmitted to the client device, wherein the second film grain content is not transmitted to the client device.

Images