Method and apparatus for film particle analysis and synthesis

Abstract

A method is disclosed that includes obtaining (S800) an image. And a film particle pattern vector representing characteristics of film particles contained in the obtained image. A pattern vector is obtained (S802) with the obtained image as an input using a first neural network (S), which is a network trained to output a film particle pattern vector from the image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of European Application No. 23305106.9 filed on January 27, 2023 and European Application No. 23306526.7 filed on September 15, 2023, which are incorporated herein by reference in their entirety. Technical Field

[0003] At least one of the present embodiments generally relates to methods and apparatus for analyzing and synthesizing grainy images. Background Art

[0004] To achieve high compression efficiency, image and video coding schemes typically employ prediction and transforms to exploit spatial and temporal redundancy in video content. Typically, intra-frame or inter-frame prediction is used to exploit intra-frame or inter-frame picture correlations. The difference between the original block and the predicted block (usually expressed as a prediction error or prediction residual) is then transformed, quantized, and entropy coded. To reconstruct the video, the compressed data is decoded through the inverse process of entropy coding, quantization, transform, and prediction.

[0005] Film grain is a specific type of noise that appears in video and is both highly pleasing and unique. Essentially, film grain is the result of the physical process of exposing and developing analog film stock. Originally, film grain was noise caused by the photographic process—more precisely, sensor noise—which was naturally present and unique to each analog camera. With the advent of digital cameras, this sensor noise disappeared during the capture phase, but is now added to content afterward to recreate the film texture. The random nature of this noise makes it difficult to compress using traditional decoding tools. Common encoding tool parameters (such as those selected for low bitrates) typically remove film grain. High bitrates are required to preserve and recreate film grain of sufficient quality, which contradicts the encoding / decoding goal of saving bits when encoding content. To overcome this encoder filtering issue, film grain modeling is typically performed before the encoding stage, and during the decoding stage, during the so-called compositing step, the film grain model is used to add the film grain back into the reconstructed video.

[0006] Even for content that originally did not contain film grain, adding film grain during the decoding stage can help mask artifacts or distortions that may appear in the content, particularly as a result of the encoding / decoding process. Summary of the Invention

[0007] In one embodiment, a method for analyzing film grain in an image is disclosed. The method obtains a film grain pattern vector representing characteristics of film grain contained in the image using a first neural network, the first neural network being a network trained to output a film grain pattern vector from the image.

[0008] A method for reconstructing a grainy image by using an obtained image and an obtained film grain pattern vector as inputs using a second neural network, the second neural network being a network trained to output a grainy image from the film grain pattern vector and the image.

[0009] In another example, a film-grain pattern vector is obtained by using another neural network with the latent code as input. The pattern vector can be used by another neural network configured to output a grainy image from the image.

[0010] Lightweight versions of different networks have also been proposed. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 illustrates a block diagram of a system in which aspects of the present embodiments may be implemented;

[0012] Figure 2 illustrates a block diagram of an embodiment of a video encoder;

[0013] Figure 3 illustrates a block diagram of an embodiment of a video decoder;

[0014] Figure 4 A flowchart depicting a method for analysis and synthesis of film grain implemented in the context of Versatile Video Coding (VVC);

[0015] Figure 5 The variation of the scaling factor according to the intensity level is illustrated;

[0016] Figure 6 Illustration of a film grain model defined in the frequency domain for a given intensity range;

[0017] Figure 7 The graph shows the variation of film grain standard deviation with intensity;

[0018] Figure 8A illustrates a block diagram of a system in which aspects and embodiments may be implemented;

[0019] Figure 8B Describes the architecture of the residual unit of a neural network;

[0020] Figure 8C illustrates a block diagram of a system in which aspects and embodiments may be implemented;

[0021] Figure 9A depicts a flow chart of a method for analyzing film grain of an image according to an example;

[0022] Figure 9B depicts a flow chart of a method for synthesizing film grain in an image according to an example;

[0023] Figure 10A depicts a flow chart of an encoding method according to an example;

[0024] Figure 10B depicts a flowchart of a decoding method according to an example;

[0025] Figure 11A illustrates a block diagram of a system in which aspects and embodiments may be implemented;

[0026] Figure 11B illustrates a block diagram of a system in which aspects and embodiments may be implemented;

[0027] Figure 12 depicts a flow chart of a method for adding film grain to an image according to an example;

[0028] Figure 13A A flowchart illustrating a method for jointly learning neural network parameters is shown;

[0029] Figure 13B A flow chart illustrating another method for jointly learning neural network parameters; and

[0030] Figure 14 A flowchart illustrating the complete cycle of sequentially applying the mapping network, synthesis network, and style network. DETAILED DESCRIPTION

[0031] This application describes various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described in detail, and at least in order to illustrate the various characteristics, they are generally described in a manner that may sound restrictive. However, this is for the purpose of describing clearly and does not limit the application or scope of those aspects. In fact, all of the different aspects can be combined and interchanged to provide further aspects. In addition, the aspects can also be combined and interchanged with aspects described in earlier submissions.

[0032] The various aspects described and contemplated in this application may be implemented in many different forms. Figure 1 、 2 and 3 provide some embodiments, but other embodiments are contemplated, and Figure 1 、 2The discussion of 3 does not limit the breadth of implementation. At least one of the aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the methods described, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the methods described.

[0033] Various methods are described herein, and each method in the method includes one or more steps or actions for realizing the method. Unless the correct operation method requires the steps or actions of a specific order, the order and / or use of specific steps and / or actions can be modified or combined. In addition, in various embodiments, terms such as "first", "second" etc. can be used to modify elements, components, steps, operations etc., such as, for example, "first decoding" and "second decoding". Unless otherwise required, using such terms does not mean the sequencing of the operation to modification. Therefore, in this example, the first decoding does not need to be performed before the second decoding, and can occur in, for example, before, during, or in the time period overlapping with the second decoding.

[0034] The present aspects are not limited to VVC (Versatile Video Coding), ECM (Enhanced Compression Model) or HEVC (High Efficiency Video Coding), and can be applied to, for example, other standards and recommendations (whether pre-existing or developed in the future) and extensions of any such standards and recommendations (including VVC, ECM and HEVC). Unless otherwise indicated or technically excluded, the various aspects described in this application can be used alone or in combination.

[0035] In this application, the terms "reconstruction" and "decoding" may be used interchangeably, the terms "encoding" or "encoded" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image," "picture," and "frame" may be used interchangeably. Typically, but not necessarily, the term "reconstruction" is used at the encoder side, while "decoding" is used at the decoder side. Hereinafter, the terms "grain" and "film grain" may be used interchangeably.

[0036] Figure 1A block diagram illustrating an example of a system in which various aspects and embodiments can be implemented is illustrated. System 100 can be implemented as a device including the various components described below, and is configured to perform one or more aspects of the various aspects described in this application. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances and servers. The elements of system 100 can be implemented individually or in combination in a single integrated circuit, multiple ICs and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed across multiple ICs and / or discrete components. In various embodiments, system 100 is coupled to other systems or to other electronic devices via, for example, a communication bus or by dedicated input ports and / or output ports. In various embodiments, system 100 is configured to implement one or more aspects of the various aspects described in this application.

[0037] The system 100 includes at least one processor 110, which is configured to execute instructions loaded therein for implementing various aspects described in the present application. The processor 110 may include embedded memory, input and output interfaces, and various other circuits as known in the art. The system 100 includes at least one memory 120 (e.g., a volatile memory device and / or a non-volatile memory device). The system 100 includes a storage device 140, which may include non-volatile memory and / or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, magnetic disk drive, and / or optical disk drive. As non-limiting examples, the storage device 140 may include an internal storage device, an attached storage device, and / or a network-accessible storage device.

[0038] The system 100 includes an encoder / decoder module 130, which is configured to process data to provide encoded video or decoded video, for example, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents a module (or modules) that may be included in a device to perform encoding and / or decoding functions. As is well known, a device may include one or both of an encoding module and a decoding module. Additionally, the encoder / decoder module 130 may be implemented as a separate element of the system 100, or may be incorporated into the processor 110 as a combination of hardware and software as known to those skilled in the art.

[0039] Program code to be loaded onto the processor 110 or the encoder / decoder 130 to perform various aspects described herein may be stored in the storage device 140 and subsequently loaded onto the memory 120 for execution by the processor 110. According to various embodiments, one or more of the processor 110, the memory 120, the storage device 140, and the encoder / decoder module 130 may store one or more of the various entries during execution of the processes described herein. Such stored entries may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from processing equations, formulas, operations, and operational logic.

[0040] In some embodiments, memory internal to processor 110 and / or encoder / decoder module 130 is used to store instructions and provide working memory for processing during encoding or decoding. However, in other embodiments, memory external to a processing device (e.g., the processing device may be processor 110 or encoder / decoder module 130) is used for one or more of these functions. The external memory may be memory 120 and / or storage device 140, such as dynamic volatile memory and / or non-volatile flash memory. In some embodiments, the external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, fast external dynamic volatile memory (such as RAM) is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard developed by the Joint Video Experts Team JVET).

[0041] Input to the elements of system 100 may be provided through various input devices as indicated in block 105. Such input devices include, but are not limited to: (i) a radio frequency (RF) section that receives an RF signal transmitted over the air, for example, by a broadcaster; (ii) a component (COMP) input terminal (or a set of COMP input terminals); (iii) a universal serial bus (USB) input terminal; and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 1 Other examples not shown include composite video.

[0042] In various embodiments, the input device of block 105 has associated corresponding input processing elements as known in the art. For example, the RF part can be associated with elements applicable to the following: (i) selecting the desired frequency (also referred to as selecting a signal, or limiting the signal band to a frequency band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower frequency band to select the signal band that (for example) can be referred to as a channel in certain embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired data packet stream. The RF part of various embodiments includes one or more elements for performing these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF part can include a tuner that performs various functions in these functions, including, for example, down-converting the received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or down-converting to baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal transmitted by wired (for example, cable) medium, and by filtering, down-conversion and filtering to desired frequency band again to perform frequency selection.Various embodiments rearrange the order of (and other) element described above, remove some elements in these elements, and / or add other elements of execution similar or different functions.Adding element can be included in and inserts element between existing element, for example, inserts amplifier and analog to digital converter.In various embodiments, the RF part comprises antenna.

[0043] Additionally, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 100 to other electronic devices across the USB and / or HDMI connections. It will be appreciated that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, for example, within a separate input processing IC or within the processor 110, as desired. Similarly, various aspects of USB or HDMI interface processing may be implemented, for example, within a separate interface IC or within the processor 110, as desired. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 110 and the encoder / decoder 130, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on an output device.

[0044] The various elements of system 100 may be provided within an integrated housing within which the various elements may interconnect and transfer data therebetween using a suitable connection arrangement 115 (e.g., an internal bus as known in the art, including an I2C bus, wiring, and printed circuit boards).

[0045] System 100 includes a communication interface 150 that enables communication with other devices via a communication channel 190. Communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 190. Communication interface 150 may include, but is not limited to, a modem or a network card, and communication channel 190 may be implemented, for example, within a wired and / or wireless medium.

[0046] In various embodiments, a Wi-Fi network, such as IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers), is used to stream data to the system 100. The Wi-Fi signals of these embodiments are received through a communication channel 190 and a communication interface 150 suitable for Wi-Fi communication. The communication channel 190 of these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments provide streaming data to the system 100 using a set-top box that delivers data through an HDMI connection of the input block 105. Still other embodiments provide streaming data to the system 100 using an RF connection of the input block 105. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0047] The system 100 can provide output signals to various output devices, including a display 165, speakers 175, and other peripheral devices 185. The display 165 of various embodiments includes, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. The display 165 can be used in a television, a tablet computer, a laptop computer, a cellular phone (mobile phone), or other devices. The display 165 can also be integrated with other components (for example, as in a smartphone), or be separate (for example, an external monitor for a laptop computer). In various examples of embodiments, the other peripheral devices 185 include one or more of a stand-alone digital video disk (or digital versatile disk) (DVR, for both terms), a disk player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 185 that provide functions based on the output of the system 100. For example, a disk player performs the function of playing the output of the system 100.

[0048] In various embodiments, control signals are communicated between the system 100 and the display 165, speakers 175, or other peripheral devices 185 using signaling such as AV.Link, CEC, or other communication protocols that enable device-to-device control with or without user intervention. Output devices can be communicatively coupled to the system 100 via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, output devices can be connected to the system 100 using a communication channel 190 via the communication interface 150. In electronic devices such as televisions, the display 165 and speakers 175 can be integrated into a single unit with the other components of the system 100. In various embodiments, the display interface 160 includes a display driver, such as a timing controller (TCon) chip.

[0049] For example, if the RF portion of input 105 is part of a separate set-top box, the display 165 and speaker 175 may alternatively be separate from one or more of the other components. In various embodiments where the display 165 and speaker 175 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0050] The embodiments may be executed by computer software implemented by the processor 110, or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. As a non-limiting example, the memory 120 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology (such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory). As a non-limiting example, the processor 110 may be of any type suitable for the technical environment and may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0051] Figure 2 An example video encoder 200 is illustrated, such as a VVC (Versatile Video Coding) encoder. Figure 2 It is also possible to illustrate an encoder in which the VVC standard is improved or an encoder that adopts a technique similar to VVC.

[0052] Before being encoded, the video sequence may undergo a pre-encoding process (201), for example, applying a color transform to the input color picture (e.g., converting from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components to obtain a more resilient to compression signal distribution (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and attached to the bitstream.

[0053] In encoder 200, a picture is encoded by encoder elements as described below. The picture to be encoded is segmented (202) and processed in units of, for example, CUs (coding units). For example, each unit is encoded using intra or inter mode. When a unit is encoded in intra mode, it performs intra prediction (260), for example using an intra prediction tool such as decoder-side intra mode derivation (DIMD). In inter mode, motion estimation (275) and compensation (270) are performed. The encoder decides (205) which of intra mode or inter mode to use to encode the unit and indicates the intra / inter decision by, for example, a prediction mode flag. For example, the prediction residual is calculated by subtracting (210) the predicted block from the original image block.

[0054] The prediction residual is then transformed (225) and quantized (230). Video coding standards such as High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), and Enhanced Compression Model (ECM6.0) support different types of block transforms, such as DCT (Discrete Cosine Transform) or DST (Discrete Sine Transform), which have been designed for square or rectangular blocks. These transforms are typically applied separately to blocks of prediction residuals obtained after intra-frame or inter-frame prediction.

[0055] The quantized transform coefficients, along with motion vectors and other syntax elements such as picture segmentation information, are entropy encoded (245) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., encode the residual directly without applying the transform or quantization process. In cases where the input picture contains some film grain, encoder quantization has the effect of removing the grain.

[0056] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The image block is reconstructed by combining (255) (e.g., adding) the decoded prediction residual and the prediction block. A loop filter (265) is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset) / ALF (adaptive loop filter) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (280).

[0057] Figure 3 FIGURE 3 illustrates a block diagram of an example video decoder 300. In the decoder 300, the bitstream is decoded by decoder elements as described below. The video decoder 300 generally performs the same operations as described above. Figure 2 The decoding process is the opposite of the encoding process described in .

[0058] Encoder 200 also typically performs video decoding as part of encoding the video data.

[0059] In particular, the input to the decoder includes a video bitstream, which may be generated by the video encoder 200. The bitstream is first entropy decoded (330) to obtain transform coefficients, prediction mode, motion vectors, and other encoding information. Picture segmentation information indicates how the picture is segmented. Thus, the decoder can divide (335) the picture according to the decoded picture segmentation information. The transform coefficients are dequantized (340) and inverse transformed (350) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (355) (e.g., added) to reconstruct the image block. The prediction block can be obtained (370) from intra-frame prediction (360) or motion compensated prediction (i.e., inter-frame prediction) (375). A loop filter (365) is applied to the reconstructed image. The filtered image is stored at a reference picture buffer (380). Note that for a given picture, the contents of the reference picture buffer 380 on the decoder 300 side are the same as the contents of the reference picture buffer 280 on the encoder 200 side for the same picture.

[0060] The decoded picture may further undergo post-decoding processing (385), such as an inverse color transform (e.g., from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping that performs the inverse of the remapping process performed in the pre-encoding process (201). The post-decoding processing may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0061] Film grain modeling

[0062] Film grain is often a desired feature in video production, creating a natural look and feel and helping to express the creative intent. Film grain is a pleasing noise that enhances the natural look of video content. It was originally produced during the physical process of exposing and developing photographic film. However, digital sensors do not undergo such a process and therefore do not have film grain. This produces a noise-free digital video, whose perfection, sharp and distinct edges and monotonous areas may deteriorate the viewer's subjective experience. Therefore, re-graining video (i.e., adding film grain to video) can improve the visual experience, and content creators often use it before distributing content. This is particularly true in the film industry, where many creators turn to the technique of adding film grain to video content to add texture and warmth to their videos, or sometimes to create a sense of nostalgia (for example, if the narrative requires it, to depict a previous era). In addition, film grain can be used to mask compression artifacts, even if it was not originally present in the source video.

[0063] When added by content creators, film grain is adapted to the different characteristics of the content: the intensity of the pixel values, the presence and type of texture, and the saturation of the pixel values. They also choose different grain patterns depending on the look they want to create. Furthermore, the grain characteristics are adapted based on the color component being considered, resulting in colored film grain. Thus, the addition of film grain is a content-dependent, pixel- or region-dependent process that considers the intensity, texture, and saturation characteristics of the content, as well as the characteristics of the grain pattern.

[0064] Film grain modeling has become part of modern video coding standards since the introduction of supplementary enhancement information for film grain in H.264 / AVC. In particular, film grain analysis and synthesis are used in the compression standard Versatile Video Coding (VVC), also known as ITU-T H.266 and ISO / IEC 23090-3. In the context of video encoding / decoding, Figure 4A flow chart of a method for analysis and synthesis of film grain implemented in the context of VVC is depicted. Information on film grain is conveyed as metadata via SEI messages (an acronym for "Supplementary Enhancement Information") specified, for example, by Versatile Supplementary Enhancement Information (VSEI, also known as ITU-T Recommendation H.274 and ISO / IEC 23002-7). Grain analysis takes as input an original frame with grain and a corresponding filtered frame without film grain. The filtered frame is first obtained via pre-processing (e.g., via a denoising step) at step S400. Homogeneous regions can then be extracted from the filtered frame, as they will be used as a mask to estimate grain parameters in the grain layer in step S402, which are simply calculated as the difference between the original frame and its filtered version. In state-of-the-art versions of grain estimation, the extraction of homogeneous regions is important, as edges and texture can affect the estimate of film grain intensity and pattern. In step S404, the filtered video and grain parameters ( Figure 4 In another embodiment, the film grain parameters obtained from the configuration file may be encoded. In a variant, the film grain parameters are obtained directly by the decoder (e.g. from a storage medium) without encoding. These film grain parameters may be encoded, for example, in the case where the input image is grainless and therefore step S402 is bypassed. The encoding step outputs encoded data (e.g. in the form of a bitstream and SEI messages), which includes the filtered video and grain parameters as metadata. The encoded data is transmitted, for example, to a decoder. The decoding step S406 outputs the filtered video and grain parameters. From the decoded data, i.e. the filtered video and grain parameters, a decoded video with grain may be reconstructed at step S408, where the film grain is synthesized and added back to the filtered video. Common techniques for inserting noise / film grain consist in applying a pattern / noise that is unique to each camera in a normal image, but with an intensity level that depends on the input signal. Thus, as Figure 5 As shown in , different scaling factors are applied to the film grain intensity depending on the intensity of the content. However, for a given block in a given frame, a constant film grain intensity level is usually chosen. Those blocks are uniform, i.e., they have the same size (e.g., 8x8).

[0065] Additionally, depending on the artistic look being targeted, the film grain intensity also depends on the texture present in the area.

[0066] Figure 6 An example of a grain model defined in the DCT (acronym for Discrete Cosine Transform) domain of the luminance layer (denoted as Y layer) is depicted. Figure 6In the above example, consider a block of 16x16 pixels. The 16x16 patch of particles in the Y space domain follows a model based on a random Gaussian distribution in the DCT domain and is applied only in a sub-portion of the 16x16 patch. The sub-portion of the patch is defined by the vertical / horizontal cutoff frequency also defined in the DCT domain. The three characteristics for modeling particles in the DCT domain are the variance σ of the noise 2 and two cutoff frequencies that limit the area where noise is present. The cutoff frequencies define the particle shape. If they are equal, the particles are round. The DC component is set to null in order to obtain centered patchy particles in the spatial domain. 2 The value of depends on the average Y value of the patches to which the grain has to be added, so that the synthesis of film grain is area-dependent. Figure 7 An example of the evolution of the grain variance with respect to the intensity level in the filtered Y layer is shown. For a given shot, the cutoff frequency value should be constant.

[0067] In AV1 (AO Media Video 1), the film grain pattern / template is modeled with an autoregressive process and its intensity is represented by a piecewise linear function that includes the grain intensity for each Y, Cb, and Cr component. The model is signaled into the coded data (e.g., SEI message). At the decoding step, the film grain model is decoded from the coded data. In the re-noising step (i.e., synthesis stage), a 64x64 luminance film grain template is generated using the decoded film grain model. After piecewise linear scaling, the film grain extracted from the 64x64 film grain template (32x32 luminance blocks) is applied on the basis of 32x32 luminance blocks to reconstruct the picture with film grain. Film grain addition is applied over the entire image. The block size is fixed and does not dynamically adapt to the region.

[0068] Outside the context of video compression, film grain generation methods have also been proposed. Such methods make it possible to add grain to digital content without having to perform a prior analysis. In an example, a parametric model based on texture statistics can be used for film grain generation. Given a grain template image, the latter is decomposed into a steerable pyramid, i.e. a linear, multi-scale and multi-orientation image transform. Each scale and orientation of the pyramid is analyzed with respect to several statistical texture features including minimum and maximum grayscale values ​​and correlation of sub-bands. The synthesis starts with random noise, which ensures high spatiotemporal variation. The method produces synthetic grain that matches the template well, while random noise-based methods inherently provide realistic spatial and temporal variation. However, the resulting synthesis is not content-adaptive and is therefore uniform on any image, regardless of the underlying image characteristics (e.g. structure and texture).

[0069] In another example, a stochastic model can be used to approximate the physical reality of film grain. To model film grain, a Boolean non-homogeneous model is used to mimic the analog photography process as closely as possible. This model corresponds to a uniformly distributed disk using a Poisson process with variable intensity, which determines the amount of grain relative to the local image grayscale. A single Gaussian kernel is used to model the grain rendering method using a Monte Carlo simulation that performs simultaneous filtering and discretization of the film grain model. A wide range of grain sizes and intensities can be generated by varying the model's parameters, including the mean grain radius and its standard deviation. Larger values ​​for these parameters emphasize the graininess of the rendered result.

[0070] In another example, a deep learning-based approach (hereinafter referred to as deep1) is used, where the film grain synthesis task is considered as the transformation of a given grain-free input image to a corresponding grainy output image. As an example, a U-Net with residual blocks can be used for grain generation, where the U-Net is trained with a discriminator in a conditional GAN ​​(generative adversarial network) framework. The synthesis model is conditioned on both the input image and some expected grain intensity level, which allows the synthesis to be content-dependent with controllable intensity levels.

[0071] The film grain generation methods mentioned above make it possible to generate only a limited set of grain types that vary only in size and intensity. Therefore, these methods do not provide a diversity of the generated grain types. More generally, methods for both estimating film grain parameters in an encoding / decoding context and then synthesizing / generating film grain are limited in the diversity of film grain characteristics that they can estimate and then re-synthesize in the content. In fact, as implemented in VVC or AV1, the texture and color saturation characteristics of the content are not modeled during the analysis step. Moreover, a single grain pattern is considered and only some variations in the size of this pattern are considered. Furthermore, only uniform areas are considered during the estimation, but grains are also present in non-uniform areas and have different characteristics.

[0072] During the compositing step, film grain is typically applied per block on the input content, with variations in intensity that depend only on the intensity level of each block. In some examples, texture can also be taken into account by some information from the encoding / decoding process, such as the size of the encoded block, since it is driven by the level of texture in each area. In AV1, the block size is fixed, i.e., the frame is not split into blocks of different sizes depending on its underlying texture. Therefore, in AV1, texture is not taken into account during the compositing step. In short, the fidelity of the original film grain is not always respected. In addition, the diversity of the generated film grain is limited.

[0073] The deep1 method makes it possible to generate more accurate and diverse film grains thanks to a learning process that automatically captures more of the characteristics of film grain. However, the information describing the film grain characteristics is contained in the model itself, which means that no parameters representing film grain are available. In other words, there is no compact representation of film grain. The weights of the learned model are required to generate film grain. Furthermore, both learned models are driven by a film grain level map to set the expected intensity level, which also limits the diversity of the output. Therefore, the diversity of the learned grain does not represent the diversity of grain styles that can be generated by content creators. Style can be referred to as the look and feel of film grain.

[0074] In the following embodiments, film grain generation and synthesis are therefore improved by providing a complete, compact, and flexible representation of film grain as it exists in content generated by content creators. Such a representation enables the synthesis of film grain that adheres to some fidelity of the original ground truth film grain (if available), and it also provides a wide variety of film grain that represents the diversity of film grain generated by content creators.

[0075] The proposed method makes it possible to extract film grain characteristics (grain intensity and pattern, depending on content such as intensity, texture, saturation, etc.) from grainy images and model these film grain characteristics, summarizing these characteristics in the form of a film grain pattern vector. A grainy image is an image containing film grain. Hereinafter, the terms "film grain pattern vector" and "pattern vector" will be used interchangeably. A film grain pattern vector therefore represents all the stylistic information for a given film grain. The pattern vector can be used to synthesize film grain while maintaining the fidelity of the original film grain (if any). When the synthesized film grain is perceptually similar to the original film grain, the fidelity of the original film grain is maintained. Furthermore, the pattern vector makes it possible to generate a wide variety of film grain types. The pattern vector can also be encoded into the encoded data (e.g., in the bitstream as auxiliary information). In other words, one of the proposed methods makes it possible to generate a number of film grain patterns that are perceptually similar and as diverse as those manually generated by content creators and artists. More precisely, the method maintains the fidelity of some existing grain reference (if any). Furthermore, the approach makes it possible to represent a wide variety of grain patterns in a compact manner that can be easily transmitted in video compression use cases.

[0076] To this end, a deep learning-based approach is used. This approach employs a neural network that learns through federated learning. This approach uses a style vector to model film grain characteristics—a complete, flexible, and compact representation of these characteristics. Thanks to a style encoder (also known as a style neural network or style network), this style vector is constructed from a reference frame containing some grain, or, if no grainy reference image is available, from a random latent code, thanks to a mapping network. Based on this style vector, film grain is then synthesized for any content using a synthesis network that adheres to the fidelity of some ground-truth film grain (when available). This allows for the random generation of a wide variety of grain patterns when no ground truth (i.e., no reference) is available. For cases where ground truth is available, it improves the fidelity of the actual film grain synthesis process in the encoding / decoding chain. In other words, the synthesized film grain is perceptually close to the original film grain (also known as the ground-truth film grain). For cases where no ground truth is available, it enables the generation of diverse film grains, which will expand the ability to mask artifacts, such as those generated from some encoding / decoding processes. The fidelity to the ground truth film grain and the diversity of grain styles are achieved by using a specific loss function in the learning phase (also known as the training phase) of the neural network.

[0077] In practice, the parameters of the neural network are learned first. In other words, the neural network is trained to learn its parameters. Once the parameters of the neural network are learned, the network can be effectively used to perform tasks such as Figure 8A to Figure 1 0, or for film grain synthesis as depicted in Figures 11-12. Inference refers to the effective use of a neural network defined by learned parameters. In other words, inference applies a trained neural network and uses it to reason about results. Inference occurs after training because it requires a trained neural network.

[0078] Figure 8A A block diagram of a system 7 in which various aspects and embodiments may be implemented is illustrated. This example may be referred to as reference-guided synthesis because an image (e.g., a grainy image) is used as a reference. In effect, the synthesized grain is expected to resemble the grain present in the input image.

[0079] System 7 includes a trained style network S (700) and a trained synthesis network G (702).

[0080] In operation, the trained style network S (700) (also called style encoder S) receives an input image x_ref and outputs a style vector s_ref. Given an input grainy image x_ref, the style encoder S first extracts the grain properties in the form of a style vector s_ref: s_ref = S(x_ref).

[0081] Hence, the style vector s_ref faithfully represents the film grain properties, such as the grain pattern and its size, randomness, intensity, color, etc. Such properties may also include how the film grain adapts to the original content, e.g., to content properties such as its intensity, saturation, texture, etc. Therefore, using a vector as a general representation to be inferred directly from grainy frames is particularly well designed for the task. In the context of video compression and transmission, the vector will also be easier to encode and transmit compared to a mapping or compared to the weights of the network. The architecture of the style network is based on a convolutional neural network (CNN) with residual blocks and a final fully connected layer, e.g., with an output dimension of 64, which is the dimension of the style vector. In the specific example mentioned in Table 1, the style network S may comprise a first 1x1 convolutional layer, 6 residual units (whose architecture is depicted in Figure 8B ), followed by a Leaky ReLu layer, a 4x4 kernel convolutional layer, another Leaky ReLu, and finally a fully connected layer of size 64. The style network S can take as input an image with granules x_ref (e.g., size 256x256x3) and output a style vector of dimension s_ref (e.g., dimension 64).

[0082]

[0083] Table 1

[0084] Therefore, given a grainy image of size 256x256x3, the style network S is able to derive a film grain style vector of size 64. Figure 8B As depicted above, the residual unit architecture may include a normalization layer, a Leaky ReLu layer, a convolutional layer with a kernel of 3x3, a resampling layer, another normalization layer, followed by a second Leaky ReLu layer and a second convolutional layer (e.g., with a kernel of 3x3). In parallel, a residual connection consisting of a convolutional layer (e.g., with a kernel of 3x3) and a resampling layer may be set between the input and output. In other examples, the dimensions of the input granular image / output style network and the architecture of the network may vary.

[0085] The specificity of this film grain analysis / modeling module is the nature of its input / output. In fact, it takes as input a grainy image (rather than a pair of grainy and filtered images as in the state-of-the-art techniques), and it outputs a film grain pattern vector (rather than a finite set of different parameters).

[0086] In an example, the pattern vector is conditioned to contain spatially organized information, ie, the first n bits are related to the particle pattern, the next m bits are related to the particle size, etc., where n and m are positive integers.

[0087] Thanks to the style encoder S, the filtering and edge detection steps performed in some particle analysis methods can be bypassed. The style encoder S is configured to extract particle properties as a style vector from the granular image alone (without prior filtering). The output style vector can later be used by the synthesis network G to reconstruct the particles with maximum fidelity to the original particles, for example, from a particle-free image given as input.

[0088] Based on the input image y_src and the style vector, the synthesis network G (702) is able to generate film grain in the form of a grain map, which enables the output image The reconstruction is the addition of the input image y_src and the film grain map generated according to the style vector. In a variant example, instead of adding the grain map, a grain map whose values ​​are in the range zero to one is multiplied pixel by pixel by the input image y_src.

[0089] The synthesis network (or generator) G is based on an encoder-decoder architecture with residual blocks, where the style vector is injected into all upsampling blocks using AdaIN, which is a normalization method that aligns the mean and variance of the content features with the mean and variance of the style features. In the specific example mentioned in Table 2, the synthesis network (or generator) G may include a first convolutional layer (e.g., with kernel 1x1), followed by (e.g., four) downsampling blocks, (e.g., four) intermediate blocks, and (e.g., four) upsampling blocks, for example, all of which inherit pre-activated residual units (ResBlk), and for example a final 1x1 convolutional layer.

[0090] The residual unit architecture consists of Figure 8B Description. The normalization (NORM) layers and resampling (RESAMPLE) layers differ depending on whether ResBlk is in a downsampling block, an intermediate block, or an upsampling block. For downsampling blocks, the resampling layers correspond to average pooling 2D layers. For upsampling blocks, they consist of classic upsampling layers with interpolation. For the normalization layers, instance normalization is used in the downsampling block and the first two intermediate blocks, while adaptive instance normalization is used in the last two intermediate blocks and the upsampling block. The synthesis network G can take as input an image without particles x_src (e.g., of size 256x256x3) and a style vector s (e.g., of dimension 64).

[0091] It can output particles (For example, an image of size 256x256x3).

[0092]

[0093] Table 2

[0094] In another embodiment, the synthesis network can output a value to be further applied to x_src to generate a particle The particle map of the image (size 256x256x3).

[0095] In another embodiment, the synthesis network can output a particle (e.g., of size 256x256x3) and its particle map (e.g., of size 256x256x3). In this case, the last convolutional layer of the network outputs a particle map with dimensions of 256x256x3, which is added to x_src to output a particle map with dimensions of 256x256x3.

[0096] The synthesis network G generates a specific grain style corresponding to the style vector and mixes it into the input image in Contains x ref Perceive similar particles.

[0097] The style vector produced by the style encoder is independent of the content. In fact, the synthesis network G is responsible for src The underlying content of the granule is used to adapt and scale the granule style.

[0098] Figure 8C A block diagram of a system 8 in which various aspects and embodiments may be implemented is illustrated. This example may also be referred to as reference-guided synthesis because an image (e.g., a grainy image) is used as a reference. In practice, the synthesized particles are expected to be similar to the particles present in the input image. The system 8 includes an encoder (800), a trained style network S (802), a decoder (804), and a trained synthesis network G (806). The trained style network S (802) and the synthesis network G (806) may be the same as the trained style network S (700) and the synthesis network G (702), respectively.

[0099] In operation, the encoder (800) (e.g., referring to Figure 2The disclosed encoder) receives a granular image x_ref on one of its inputs. The granular image may be of YUV type. In an example, the granular image is converted to an RGB image before being used as input to the style encoder. This conversion step is optional. In another example, the color space of the input granular image after or before the conversion may be replaced by another color space such as Lab, HSV. In another embodiment, only the Y component of the granular image is transmitted as input to the style encoder. The encoder encodes the image into encoded data, for example into a bitstream. The trained style network S (802) (also referred to as style encoder S) receives the input image x_ref (possibly after color conversion) and outputs a style vector s_ref. Given an input granular image x_ref corresponding to a grainless image x_src, the style encoder S first extracts the granular properties in the form of a style vector s_ref: s_ref = S(x_ref).

[0100] The style vector s_ref is then encoded by the encoder (800). For example, the style vector may be encoded in an SEI message (SEI stands for "Supplemental Enhancement Information") that is added to the bitstream containing the encoded image. This encoded data (the image and its associated style vector) may then be transmitted to a decoder (804), e.g. Figure 3 The decoder can reconstruct the encoded image and further reconstruct the style vector. Based on the reconstructed image x_rec and the style vector, the synthesis network G (806) can generate film grain in the form of a grain map, which enables the output image to be reconstructed as the addition of the decoded image x_rec and the film grain map generated based on the style vector. In a variant example, instead of adding the grain map, the grain map whose values ​​are in the range of zero and one is multiplied pixel by pixel by the decoded image.

[0101] The synthesis network G generates a specific grain style corresponding to the style vector and mixes it into the input image in Contains x ref Perceive similar particles.

[0102] The style vector produced by the style encoder is content-independent. In fact, the synthesis network G is responsible for adapting and scaling the granular style according to the underlying content of the input.

[0103] In the examples disclosed below, some optional color conversion of the image can occur before the style network S, before the encoder, and / or after the decoder to match the color space required as input to the style network S, encoder, and / or synthesis network G.

[0104] Figure 9A A flow chart of a method for analyzing film grain of an image is depicted, according to an example.

[0105] In step S800, an image is obtained. The image can be obtained from a storage medium or from a network.

[0106] At step S802, a film grain pattern vector is obtained by using a first neural network (i.e., a pattern network S) with the obtained image as input. The film grain pattern vector represents the characteristics of film grain contained in the obtained image. The first neural network is a network trained to output a film grain pattern vector from an image.

[0107] Figure 9B Depicted is a flow chart of a method for synthesizing film grain in an image, according to an example.

[0108] At step S804 , an image and a film grain pattern vector are obtained, for example, from a storage medium or from a network.

[0109] At step S806, a grainy image is obtained by using a second neural network (i.e., synthesis network G) with the obtained image and the obtained film grain pattern vector as input, the second neural network being a network trained to output a grainy image from the film grain pattern vector and the image.

[0110] Figure 10A A flow chart of an encoding method according to an example is depicted.

[0111] In step S900 , an image, such as an RGB or YUV image, is obtained. The image can be obtained from a storage medium or from a network.

[0112] At step S902, a film grain pattern vector is obtained by using a first neural network (i.e., a pattern network S) with the received image as input. The film grain pattern vector represents the characteristics of film grain contained in the obtained image. The first neural network is a network trained to output a film grain pattern vector from an image.

[0113] At step S904 , the obtained image and film grain pattern vector are encoded in the encoded data.

[0114] Figure 10B A flow chart of a decoding method according to an example is depicted.

[0115] At step S1000, encoded data representing an image and a film grain pattern vector is obtained, wherein the film grain pattern vector represents characteristics of film grain. The encoded data may be obtained from a storage medium or transmitted from an encoder via a wired or wireless network.

[0116] At step S1002 , an image and a film grain pattern vector are decoded from the encoded data.

[0117] At step S1004, a grainy image is obtained by using a second neural network (i.e., synthesis network G) with the decoded image and the film grain pattern vector as input, the second neural network being a network trained to output a grainy image from the film grain pattern vector and the image.

[0118] Figure 11A Illustrated is a block diagram of a system 9 in which aspects and embodiments may be implemented. This example may be referred to as a potential guided synthesis, as no reference is available.

[0119] System 9 includes a trained mapping network M (905) and a trained synthesis network G (906), which can be the same as in systems 7 and 8. The mapping network M is configured to generate a style vector s_lat based on a random latent vector (also called a latent code) as input. More precisely, the mapping network is configured to transform a random vector (also called a latent code) sampled from a Gaussian distribution into a style vector so as to generate a variety of grain types. The style vector s_lat is in turn used as input to the synthesis network G to generate grain on an input image (e.g., on a grain-free image). When no reference grainy image is available, a set of style vectors can be defined and stored in a database and further used for film grain generation. However, the diversity of film grain types will be limited to the stored vectors. In contrast, by using the mapping network, more diversity in grain styles is generated.

[0120] In operation, the synthesis network G (906) is able to reconstruct the output image from the input image x_src and the style vector s_lat in is the input image x_src to which the film grain generated using the style vector s_lat is added. In the example, the image x_src is an RGB image. In another embodiment, the color space of the input image can be replaced by another color space such as Lab, YUV, or HSV.

[0121] As in systems 7 and 8, the style vector s_lat produced by the mapping network M (905) is content-independent. In practice, the synthesis network G is responsible for adapting and scaling the granular style according to the underlying content of the input.

[0122] Based on the input image x_src and the style vector s_lat, the synthesis network G (906) can generate film grain in the form of a grain map, which enables the output image to be reconstructed as the addition of the input image and the film grain map generated based on the style vector. In a variant example, instead of adding the grain map, the grain map, whose values ​​are in the range of zero to one, is multiplied pixel by pixel by the decoded image.

[0123] In system 9, no reference grainy image is used; instead, a mapping network M transforms a random latent code z from a standard Gaussian distribution Z into a style vector s_lat: s_lat = M(z). The mapping network M can be based on a multilayer perceptron (MLP) and can take as input, for example, a latent code (e.g., of dimension 16) randomly sampled from a standard Gaussian distribution. In a specific example, the mapping network M can be a multilayer perceptron (MLP) of (e.g., eight) fully connected layers (e.g., of dimension 512). The dimensions of the latent code, hidden layers, and style code can be set to 16, 512, and 64, respectively, for example. In this case, the mapping network M outputs a style vector s_lat of dimension 64. Since film grain is random noise that does not necessarily follow a Gaussian distribution, the mapping network enables mapping latent codes sampled from a Gaussian distribution to style vectors to generate more accurate and diverse grain patterns. Thus, the mapping network learns the distribution and diversity of film grain present in the training dataset. Due to joint learning, it also learns to output a style vector that is close to the style vector output by the style network S of system 8. In another example, the latent code z can be taken from a non-standard Gaussian distribution.

[0124] In yet another example, the input and output dimensions of the mapping network and its architecture can vary. However, the output dimensions should match the output dimensions of the style encoder and the dimensions of the style vector used as input to the generator.

[0125] Figure 11B A block diagram of a system 10 is shown in which various aspects and embodiments may be implemented. This example may also be referred to as potential-guided synthesis because no reference is available. System 10 is a variant of system 9 in which the image used as input by the trained synthesis network G (906) is an image reconstructed by a decoder (903). Thus, system 10 includes an encoder (901) and a decoder (903).

[0126] In operation, the synthesis network G (906) is able to reconstruct the output image from the input image x_rec and the style vector s_lat in is the input image x_rec to which the film grain generated using the style vector s_lat is added. In the example, the image x_rec is an RGB image. In another embodiment, the color space of the input image can be replaced by another color space such as Lab, YUV, HSV. The input image x_rec is decoded by a decoder (903) (e.g., a computer configured to decode the received encoded data and reconstruct an image therefrom). Figure 3 The received coded data can be obtained by the encoder (901) (for example, Figure 2The encoder (901) is configured to receive an image on one of its inputs x_ref and to encode the image into coded data, for example into a bitstream.

[0127] As in systems 7 to 9, the style vector s_lat produced by the mapping network M (905) is content-independent. In practice, the synthesis network G is responsible for adapting and scaling the granular style according to the underlying content of the input.

[0128] Based on the input image and the style vector s_lat, the synthesis network G (906) can generate film grain in the form of a grain map, which enables the output image to be reconstructed as the sum of the input image and the film grain map generated based on the style vector. In a variant example, instead of adding the grain map, the grain map, whose values ​​are in the range of zero to one, is multiplied pixel by pixel by the decoded image.

[0129] Figure 12 Depicted is a flow chart of a method for adding film grain to an image, according to an example.

[0130] At step S1200, an image is obtained, such as an RGB or YUV image. The image can be obtained from a storage medium or from a network. In a specific example, the image is an image decoded from encoded data by a decoder. The decoded image can optionally be converted to match the color space required as input to the synthesis network G.

[0131] At step S1202, a film grain pattern vector representing characteristics of film grain is obtained by using a third neural network (i.e., mapping network M) with the latent code as input, where the third neural network is a network trained to output a film grain pattern vector from the latent code.

[0132] At step S1204, a grainy image is obtained by applying a fourth neural network (i.e., synthesis network G) to the received image and film grain pattern vector. The fourth neural network is a network trained to output a grainy image from the film grain pattern vector and the image. The fourth neural network is identical to the second neural network.

[0133] The neural network parameters (e.g., weights and biases) of the style, mapping, and synthesis neural networks S, M, and G used by systems 8 and 9 are estimated during the training (also called learning) phase.

[0134] To this end, the neural networks S, M and G are trained (i.e., the neural network parameters are learned) on a dataset of a large number of images. In the example, a dataset of 148,694 particle-free images is used. For each particle-free image, the dataset also includes 5 different granular versions. Classically, the learning phase consists of a forward pass and a backward pass. The forward pass represents the direction of flow from "input" to "output". The backward pass represents the direction of flow from "output" to "input", during which the gradient of the loss function is backpropagated. The purpose of the backward pass is to distribute the total error back to the network in order to update the parameters in order to minimize the cost function (loss function). The update is determined by the gradient of the cost function with respect to those parameters. The parameters are updated in such a way that when the next forward pass utilizes the updated parameters, the total error is reduced by a certain margin (until a minimum is reached).

[0135] During the learning phase of the neural networks S, M, and G, an additional neural network (i.e., the discriminator D) is used to adversarially train the synthetic network G. In an example, the discriminator can be based on a PatchGAN architecture with, for example, a 30×30 receptive field. PatchGAN takes an image as input and classifies whether each, for example, 70×70 patch in the image is fake or real, rather than providing a single probability for the entire input image.

[0136] Since film grain synthesis is a subjective task, a direct objective loss function cannot be defined to be minimized during training of the synthesis network. Therefore, all four networks are trained in a cGAN framework conditioned on the style vector used for film grain synthesis and generation. More precisely, all four networks S, M, G, and D are jointly trained in two consecutive forward steps: first a reference-guided forward step, followed by a potential-guided forward step, each followed by a backward step. This joint learning ensures both fidelity to the original grain in the reference-guided synthesis and diversity in the generated realistic grain styles in the potential-guided synthesis.

[0137] Figure 13AA flowchart of a method for jointly learning neural network parameters is illustrated. To this end, a large and diverse film grain dataset, hereinafter referred to as the FilmGrainStyle740k dataset, is used. The dataset consists of a set of images, including one image without grain and up to five corresponding images with five different types of film grain. This dataset represents the artistic intent of the content creator. Such sets of corresponding grainy and grain-free images will allow for driven reference-guided synthesis. The dataset is generated from 148K (K stands for thousand) original clean images (also referred to as grain-free images) from different and diverse image datasets, including 400 images from the Berkeley Segmentation Dataset (BSD), 4744 images from the Waterloo Exploration Database, 900 images from the DIV2K dataset, 2650 images from the Flick2K dataset, and 140,000 images from the Konstanz Artificial Distortion Image Quality Set (KADIS-700k). Five different types of film grain representing different camera types are considered, resulting in different grain patterns. Images with grain were generated from 148,694 original images to adjust the grain pattern to the underlying content based on three properties: intensity, texture, and saturation. A total of 240 different grain combinations / styles were generated by varying the parameters according to Table 3 below.

[0138] pattern strength Saturation Texture total 5 6 4 2 240

[0139] Table 3 - Number of values ​​for each parameter of each particle / content characteristic and the total number of resulting combinations.

[0140] For each grain-free image, 5 grainy versions are generated so that all 240 grain styles are represented in a fair way in the dataset. The dataset contains a total of 740K grainy images.

[0141] In the example, the Adam optimizer with β1=0 and β2=0.99 is used to train the neural network by optimizing the loss function described below. The batch size is set to 4, with 50K training iterations. The weights of all networks are initialized using, for example, He initialization. The learning rate is an important hyperparameter because, in addition to the instability of training GANs, jointly training four modules also brings additional challenges. Therefore, the learning rate is chosen to balance the learning of each network with the learning of the other networks and to ensure stability during the learning process between the synthesizer and the discriminator. In the example, they are fixed to 3e-4 for G, 5e-5 for S, 5e-5 for D, and 1e-4 for M. The synthesis network G counts 33M parameters (M stands for million), the style network counts 20M parameters, the mapping network counts 2M parameters, and the discriminator counts 3M parameters. To train the model, two grainy images (x_ref1, x_ref2) are selected for each grainless image x_src. The training set, validation set, and test set include 146K, 480, and 960 grain-free images randomly selected from the dataset, respectively.

[0142] The learning method consists of two consecutive forward steps, each followed by a backward pass to update the neural network parameters.

[0143] In step S1300, the parameters of the networks G and D are updated by minimizing the following first global loss function: In the variant, the first global loss function is defined as or

[0144] In the example, is calculated using both x_ref1 and x_ref2, while other basic loss functions are calculated only using x_ref1. Therefore, in the first forward pass, the style network S, networks G and D are used as input together with the triplet of images (x_src, x_ref1, x_ref2) to generate an output, based on which a loss value can be calculated. The triplet of images includes the particle-free image x_src, the first granular version of the particle-free image x_ref1 and the second granular version of the particle-free image x_ref2. In the following backward pass, the parameters of networks G and D are updated. In a variant, the parameters of networks S, G and D are updated during the first backward pass. In other examples, any combination of updates of network parameters can be performed, such as updating S and D, or S and G, or updating S twice and G, etc.

[0145] In step S1302, the parameters of the networks S, M, G, and D are updated by minimizing the following second global loss function:

[0146] In the example, is calculated using both z1 and z2, while the other basic loss functions are calculated only using z1. Therefore, in the second forward pass, the style network S, the mapping network M, the networks G and D are applied with the triple (x_src, z1, z2) as input to generate an output based on which the loss value can be calculated. For example, the latent codes z1 and z2 are randomly sampled from a standard Gaussian distribution. In the following backward pass, the parameters of all four networks are updated. In another variant, only some networks can be updated, for example, those networks that were not updated during the first forward pass. The networks can be updated once, twice, etc. Any other combination of network parameter updates can be performed.

[0147] Steps S1300 and S1302 may be repeated with each set of (x_src, x_ref1, x_ref2, z1, z2), where (x_src, x_ref1, x_ref2) may be derived from a dataset and (z1, z2) may be randomly sampled from, for example, a standard Gaussian distribution.

[0148] Learning can be done with input images in RGB, YUV, or any other color space. In another example, the style network is trained on only one color component (e.g., the Y component of the input image), while the synthesis network G is trained to generate particles in the full-color image.

[0149] In another example, the learning phase includes only the first forward step. In this case, only networks S, G, and D are trained for use in the inference phase.

[0150] In another example, the learning phase includes only the second forward pass step without the style encoder S. In this case, only networks M, G, and D are trained for use in the inference phase.

[0151] The following further defines examples of the various loss functions mentioned above.

[0152] Against target

[0153] The synthesis network G learns to generate synthetic granular images G(x src ,s):

[0154]

[0155] By classifying the output of the synthesis network as real or fake, the discriminator forces it to output images containing perceptually similar real particles. However, it does not enforce any diversity in the generated particles, nor their fidelity to a particular reference. These constraints are addressed in the following style (fidelity) and diversity objectives.

[0156] Style Target

[0157] In order to force the synthesis network G to utilize the style vector s when generating grainy images, a style reconstruction loss is used. The latter ensures that the style vector corresponding to the generated image is close to the style vector used to generate it. In the reference-guided scheme, the style reconstruction loss is then defined as follows:

[0158]

[0159] In the potential guidance scheme, in addition to forcing the synthesis network to use the style vector, this style reconstruction loss ensures that the style vectors output by both the mapping network and the style encoder have the same properties and are therefore interpreted similarly by the synthesis network. The above equation is transposed into:

[0160]

[0161] Diversity Goals:

[0162] To further enforce diversity in the generated particles, two diversity losses are defined according to either the reference-guided or the latent-guided scheme. In the latter, to force the mapping network M to produce different style vectors given two different latent codes z1 and z2, and to force the synthesis network G to produce corresponding diverse images, the diversity loss is defined to ensure the cosine similarity between z1 and z2 and their respective style vectors s by computing the ratio lat1 and s lat2 The cosine similarity between is the same:

[0163]

[0164] Since z1 and z2 are randomly sampled, if they are equal, ∈ (e.g., ∈ = 10 -8 ) is added to the denominator to prevent division by zero.

[0165] Similarly, to encourage both diversity and fidelity in the reference-guided scheme, another diversity loss is defined to ensure that the two reference images x ref1 and x ref2 Style vector s ref1 and s ref2 The cosine similarity between them and their composite versions and Style vector and The cosine similarity between is the same:

[0166]

[0167] In the proposed diversity loss, one minus cosine ratio is minimized, which converges to 1 in case of two similar style vectors and converges to 0 in case of two different style vectors correctly reproduced by the style encoder.

[0168] Content Retention Goals:

[0169] To preserve content and avoid introducing distortion or color shift, the synthesis network in the reference-guided scheme is further constrained by defining two content-preserving losses: ref and between pixel-wise loss, and x ref and The feature-wise loss between the final feature maps of , which are passed through the pre-trained VGG16 network:

[0170]

[0171] The proposed methods are frame-based. However, they can be easily applied to other media, such as 360° content and video. For the latter, a simple frame-by-frame processing can be envisioned first. Several training and subsequent learning networks can be trained based on different image categories (i.e., outdoor images, images with people, game images, UI images, etc.), so that the granular style vectors are refined for each category.

[0172] In by Figure 13B In another example shown, the synthesis network outputs a granular image that is the sum of the original image and the generated particle map, and the discriminator takes as input a concatenation of a particle-free image and its corresponding particle map (obtained by the difference between the clean image and the particle map) and compares it with a concatenation of the particle-free image and a reference particle map, which is calculated as the difference between the reference particle image and the particle-free image. The concatenation includes adding three dimensions to the input channels, for example, increasing the number of input channels from 3 (for the particle-free image) to 6 (with the additional channel of the particle map).

[0173] In another embodiment, the grain map is not necessarily RGB and may have dimension 1 (grey).A single channel map may be added to each channel of the image.

[0174] Explicitly decoupling the particle map from the granular image and giving it to the discriminator allows the discriminator to better classify between real and fake particle maps, thereby enhancing its performance.

[0175] therefore, Figure 13BA flowchart of a method for jointly learning neural network parameters according to the above example is illustrated, wherein the discriminator takes as input a concatenation of a particle-free image and a reference particle map, and a concatenation of a particle-free image and a synthesized particle map. Figure 13A Compared, the same dataset and loss target can be used, and the same input and output can be used for synthesis, mapping, and style encoder networks. In addition, the discriminator takes a grain-free image (x_src) and is calculated as The concatenation of the predicted particle map and the particle-free image (x_src) and the reference particle map calculated as (x_ref1-x_src) are taken as input.

[0176]

[0177] Table 4 - FID scores for different configurations of the discriminator input

[0178] Table 4 shows the Fréchet Inception Distance (FID) scores for different input configurations of the discriminator: 1) Configuration 1: granular image (clean image + particle map), 2) Configuration 2: particle map only, and 3) Configuration 3: concatenation of a clean image and its corresponding particle map. Configuration 3 improves the quality of the generated particle image due to better separation of synthetic particle and clean image features.

[0179] Reference-guided synthesis of qualitative results

[0180] To evaluate fidelity to the reference grain, different ground-truth grainy images were sampled from the test set. These images were derived from the same original grain-free content but had different grain patterns. A style encoder S was used to extract a style vector from each of these grainy images. The style vector was then fed into a synthesizer network G to reapply grain to the original clean, grain-free image. The obtained results first show that the synthesized images contain different grain patterns, demonstrating that the style encoder provides different style vectors to the synthesis network. Furthermore, the output is perceptually very close to its corresponding reference. This means that both the style encoder and the synthesis network work together to produce accurate output: the style encoder is able to extract grain properties regardless of content, and the synthesis network is able to correctly interpret the output style vector. In addition to visual comparison, COSINE similarity can be calculated between the style vectors of different ground-truth and synthesized grainy images. When the grain patterns are similar, the COSINE similarity score is very high (approximately 0.9), while when the grain patterns differ in some or all properties, including spatial distribution, intensity, and pattern, the COSINE similarity score is very low (approximately 0.3).

[0181] To verify that the style vector contains only grain characteristics, a style vector is first extracted for a given reference grainy image (from the test set). This style vector is then used to synthesize grain again, but in a different grain-free image. Qualitative comparison of the synthesized images shows that they contain the same grain pattern. However, the latter has its content adapted by the synthesizer G. Furthermore, when examining the grain map, no content from the grainy image used as a reference is present. Therefore, the style vector only contains information about grain.

[0182] A very valuable feature of the proposed style vector is that, apart from the grain style, it contains no content-related information. Therefore, when it is compressed and transmitted in a video compression scheme, no redundant information related to the underlying content is transmitted. A style vector of dimension 64 appears to be sufficient to encapsulate the desired grain properties, however, its dimension can ultimately be reduced.

[0183] Additional constraints on the properties of the style vector can be added to the style encoder in order to facilitate its compression and transmission, e.g.

[0184] Some energy-aware constraints, e.g. related to the size of the pattern vector and / or some bitrate constraints;

[0185] Some robustness constraints on the encoding / decoding of the style vector. This can be done by including the encoding / decoding steps in the framework and adding a loss to ensure that the style vector is not modified by these two steps;

[0186] ● Constrain some robustness against synthetic grainy images. Also, the encoding / decoding process during training can be added to the overall training framework, and the same loss can be used in particular in the reference-guided scheme.

[0187] The film-grain synthesis method used in System 8 was compared with a VVC implementation and a depth-based synthesis implementation, Deep1, on grainy images selected from the test set. For a fair comparison, the VVC analysis and synthesis method were performed outside the encoding / decoding steps of the process. In the case of Deep1, since no analysis step was available, all possible different intensity levels were tested, and the intensity level that was closer to the reference was selected for synthesis.

[0188] Results show that VVC is not always able to synthesize grain accurately. This may be due to several reasons, including the fact that the analysis is limited to homogeneous regions and the limited set of parameters used to represent grain properties. The Deep1 output is close to the corresponding ground truth in terms of intensity, which is manually selected to meet the reference intensity. However, the distribution of grains based on the input content texture is not adhered to. This may be due to two main reasons: Deep1 is only conditioned on the intensity level of the grains, and the spatial distribution of grains is learned from another, less rich and less representative dataset. For a fairer comparison, Deep1 should have been trained on the FilmGrainStyle740k dataset, however, this is not possible because it is only conditioned on the grain intensity level, while a set of several parameters defines each grainy image in the new dataset. In contrast, the film-grain synthesis approach used in Systems 7 and 9 provides better visual results and higher fidelity by learning a content-independent grain style, thanks to the joint work of the style encoder and synthesis network. This content-independent grain style is further adapted to the input content during synthesis. The quality of the output in terms of fidelity is due to the use of both style reconstruction and diversity losses to constrain both networks.

[0189] Another feature of the style encoder S is that it performs analysis across the entire input image (combined homogeneous / textured regions), unlike state-of-the-art models that only perform analysis on homogeneous regions. Following the same principle, the synthesis network is able to generate and blend grain with a given content based on its texture, saturation, and intensity, rather than simply scaling the grain pattern based on the pixel’s intensity level, as it does in AV1 and VVC.

[0190] Synthetic qualitative results of potential guidance

[0191] Given a specific grain style, the synthesis network is responsible for adapting that style to the underlying content on some grain-free images while blending it with the generated grain. To demonstrate this feature, a single latent code from Z is sampled, mapped into a style vector using the mapping network M, and the resulting style vector is used to add grain to different images using the synthesis network. The results show that although the same style vector is used and indeed grainy images have the same style, they also differ based on their underlying content because the grain distribution and appearance are adapted to it. For example, textured areas contain different grain characteristics than uniform areas, etc. The synthesis network is therefore the module responsible for blending the grain style to the content, making the style vector content-independent and the synthesis content-adaptive.

[0192] In existing methods, to first add and blend grain to content that does not have any film grain, a default grain pattern and a default scaling function are defined and used (without adaptation for the given content). Figure 11A In the system 9 depicted above, the synthesis network is able to generate and mix particles in a content-adaptive manner based solely on the style vector output by the mapping network.

[0193] The mapping network M produces diverse particle styles that represent the content of the dataset. To evaluate the diversity of the particle styles output by the mapping network, we simply sample different latent codes from Z, map them to style vectors, and then use them for particle generation by the synthesis network.

[0194] The resulting images show that the generated particles are different between images. Furthermore, particle diversity is not limited to intensity levels, but also includes diversity in particle pattern, size, and spatial distribution. Therefore, due to the style diversity loss, the mapping network is able to generate different style vectors given different latent codes without the help of any reference grainy images.

[0195] In terms of diversity, the film-grain synthesis method used in System 9 is compared with VVC, AV1, and Deep1. In the case of Deep1, it is impossible to generate diverse grain patterns; only the intensity can be varied. In contrast, the AV1 and VVC implementations are able to generate diverse grain styles by varying the parameters of the grain pattern. By manually changing these parameters, different types of grain are obtained. The results reported here give some indication of the range of diversity achievable by AV1 and VVC. Both AV1 and VVC can generate different grain patterns. However, the diversity is limited to the grain pattern, while the diversity provided by the proposed method covers several characteristics, such as the size, intensity, and pattern of the grain. Furthermore, the scaling of grain to the underlying content in VVC or AV1 is based solely on the content pixel values, while the film-grain synthesis method used in System 9 blends and scales grain not only based on pixel intensity but also on other low-level and high-level characteristics of the content, such as its saturation and texture. As for Deep1, it is only able to reproduce the single grain pattern on which it was trained.

[0196] The goal of jointly training the mapping network and the style encoder is to force both networks to produce style vectors that are interpreted in the same way by the synthesis network. Figure 14 Schematic diagram. The first latent guided synthesis is operated, i.e., according to the style vector (derived from the random latent code). A new style vector is extracted from the synthesized grainy image, which is again used in the second step of reference guided synthesis to generate another synthesized grainy image. Such an experiment represents a complete cycle, in which all networks M, S and G are used one after another, as shown in Figure 14As depicted above, the outputs of both latent-guided synthesis and reference-guided synthesis are compared. The results show that the outputs of latent-guided synthesis and reference-guided synthesis are perceptually very close. This means that the mapping network and style encoder are jointly learned to generate similar style vectors due to the style reconstruction loss.

[0197] Quantitative results

[0198] In reference-guided synthesis, to evaluate the quality of the synthesized grainy image, two quality metrics, PSNR and SSIM, are used. Although not well-tailored to evaluate the quality of the output grain, they will assess similarity in terms of the underlying content and ensure that no distortion is added, as GANs are known to generate noise.

[0199] In reference-guided synthesis, maximum fidelity to the reference is sought. However, such fidelity should not be understood at the pixel level, but rather in terms of grain pattern. To assess fidelity to the grain pattern, the cosine similarity metric is used. It is calculated between the corresponding style vectors, as output by the style encoder, from the reference image and its synthesized version.

[0200] In both reference-guided and latent-guided synthesis, the diversity of the synthesized grainy images is an important feature. Frechet inception distance (FID) and learned perceptual image patch similarity (LPIPS) are used to evaluate the latter. FID measures the difference between two sets of images. In the latent-guided scheme, each grain-free image from the test set is first converted into four grainy images using 4 latent vectors randomly sampled from a standard Gaussian distribution. The FID is then calculated between each pair of (reference-synthesized) images by comparing their feature vectors extracted from the last average pooling layer of Inception-V3 pre-trained on ImageNet. LPIPS measures the diversity of the generated images using the L1 distance between features extracted from AlexNet pre-trained on ImageNet. For any test grain-free image, the average LPIPS distance between all pairs (6) of each set of 4 latent-guided synthesized grainy images is first calculated. Finally, the average distance over all test images is reported. For the reference-guided scheme, FID is calculated in a similar way, where each clean image is associated with 4 grainy synthesized versions using 4 reference images instead of 4 latent codes.

[0201]

[0202] Table 5 - Quantitative comparison of film grain synthesis.

[0203] Table 5 summarizes the quantitative comparisons. For the reference-guided scheme, the proposed method is compared with VVC and Deep1, where VVC is tested while omitting the encoding / decoding steps. For Deep1, to simulate the analysis step, granular outputs are computed for all intensity levels, and the granular output closest to the ground truth is retained, favoring Deep1 in the comparison. Despite being based on a GAN and potentially introducing distortion, the proposed method delivers comparable quality to the VVC solution in terms of PSNR and SSIM. In contrast to the proposed method and VVC, Deep1 achieves lower quality values ​​because it does not consider the spatial distribution of particles. In terms of fidelity, the proposed method outperforms all methods, demonstrating its ability to correctly extract particle properties. Deep1 again achieves the lowest fidelity, although this may be due to its training dataset containing only a single pattern. In terms of diversity, the proposed method again outperforms the other methods, as it better reproduces the diversity present in the dataset. Note that the dataset was constructed using software used by artists, making it representative of artistic content. VVC ranks second due to its limited set of parameters for modeling the grain pattern, followed by Deep1, which is limited to a single grain pattern.

[0204] The proposed method is compared with AV1, VVC and Deep1 in the potential guidance scheme. For AV1 (respectively VVC), 16 (respectively 7) grainy images are generated, one for each set of available grainy parameters for each image in the test set (for VVC, 7 sets of parameters are manually selected to maximize diversity). For Deep1, 5 grainy images are generated for each test image, one for each intensity level. The proposed method again outperforms all other methods in terms of quality and diversity. The FID scores show that the diversity of grain patterns achieved by the proposed method is close to the diversity of the test set. The high FID value for VVC may be due to some odd grain patterns that are sometimes created and can be considered as distortion.

[0205] The above framework (called deep version) uses four modules: discriminator, style encoder, synthesis network and mapping network. During the learning (also called training) phase, four modules can be used, such as Figure 13A and Figure 13B During the inference phase, only three modules may be used, e.g., the discriminator is not used at the inference step. In the example deep version, these three modules use a total of approximately 55M parameters, as shown in Table 6.

[0206]

[0207] Table 6 - Detailed architecture of the synthesis network, style encoder, and mapping network

[0208] In the context of video compression, the style encoder will likely run on the encoder side, which is less constrained in terms of available resources. In contrast, the mapping and synthesis networks will likely run on the decoder side, i.e., more likely on resource-constrained devices. The current number of parameters of these networks may be problematic for deployment on any device with low memory and processing power, as it may lead to high energy consumption and memory starvation.

[0209] Therefore, a lightweight version of the framework is proposed that contains fewer parameters (e.g., 92% fewer parameters) and fewer floating-point operations (e.g., 86% fewer) than its corresponding deep version, while still achieving the same quality in terms of fidelity and the same diversity in film grain styles.

[0210] Thus, the complexity of the framework is reduced, for example, the energy cost of its training and inference phases is reduced, while maintaining equivalent performance. In the example, each module / network is simplified independently, for example using ablation studies. The performance of each module is then evaluated in the framework, where the other modules still have their deep architecture, i.e., the architecture mentioned for the deep version and illustrated in Table 6. The result of the simplification is a collection of a lightweight synthesis network, a lightweight style encoder, and a lightweight mapping network.

[0211] The lightweight architecture of the style encoder (referred to as Style_NoRes_64) can include a 1x1 kernel convolution layer (#channel_out=8), followed by (e.g., six) convolution blocks, a LeakyReLu layer, a 4x4 kernel convolution layer, another LeakyReLu, and finally a fully connected layer of, for example, size 8. Each convolution block includes average pooling 2D, a 3x3 kernel convolution layer, an instance normalization layer, and a LeakyRELU. The dimensions of the output channels in the convolution layers can be equal to 16, 32, 64, 64, 64, 64. The above architecture is an example lightweight architecture of the style encoder.

[0212] The lightweight architecture of the synthesis network (called Synthesis_NoRes_256) can include a first convolutional layer with a kernel of 1x1 (#channel out = 32), followed by (e.g., four) downsampling blocks, (e.g., four) intermediate blocks, and (e.g., four) upsampling blocks, and a final convolutional layer with a 1x1 kernel. Each downsampling block consists of an instance normalization layer, a LeakyReLU layer, a convolutional layer with a kernel of 3x3, and 2D average pooling. The dimensions of the output channels in these blocks are 64, 128, 256, 256. The intermediate block architecture is the same as that of the downsampling block, but without average pooling. The dimensions of the output channels in these blocks can be 256, 256, 256, 256. Each upsampling block includes an adaptive instance normalization layer, a LeakyReLU layer, an upsampling layer, and a convolutional layer with a kernel of 3x3. The dimensions of the output channels in these blocks can be 256, 128, 64, 32. The above architecture is an example lightweight architecture of the synthesis network.

[0213] A lightweight architecture for the mapping network (referred to as Mapping_128_4) can include (e.g., five) fully connected layers (e.g., dimension 128) of a multi-level perceptron (MLP). The dimensions of the latent code, hidden layer, and style code can be set to 16, 128, and 8, respectively. The above architecture is an example lightweight architecture for the mapping network.

[0214] Therefore, a fully end-to-end lightweight version of the framework is proposed, in which the three aforementioned lightweight architectures can be selected for each module respectively, i.e., Mapping_128_4 as the mapping network, Style_NoRes_64 as the style encoder, and Synthesis_NoRes_256 as the synthesis network. Table 7 compares the results obtained by the deep and lightweight versions of the framework, which clearly shows that similar performance can be achieved with a 92% reduction in the number of parameters and an 86% reduction in floating-point operations. Compared to the deep version, the proposed fully lightweight framework makes it possible to synthesize particles using less energy (e.g., in terms of memory and processing load) while achieving the same or similar quality performance in terms of fidelity to the original particles and the diversity of the generated particle styles.

[0215] In the following experiments, the discriminator is chosen to take a grain-free image (x_src) and compute The concatenation of the predicted particle maps as well as the particle-free image (x_src) and the reference particle map calculated as (x_ref1–x_src) are taken as input.

[0216]

[0217] Table 7 - Quantitative comparison between deep and lightweight versions of the proposed framework

[0218] In video compression, the style vector can be extracted at the encoder side, transmitted with the bitstream, and then further used at the decoder side to synthesize back the original particles. This means that the more compact the style vector, the easier it is to encode and transmit it, and the lower the transmission cost. To this end, the dimension of the style vector can be reduced, for example, while maintaining the same fidelity as the original particles and the ability to generate diverse particle styles. To this end, the fidelity and diversity of generated images with different style vector dimensions (32, 16, 8 and 4) are evaluated while maintaining the same lightweight version of the above framework. A quantitative evaluation of both the reference guided synthesis results and the potential synthesis results is reported in Table 8. Comparable performance is obtained using different style vector dimensions on the potential guided synthesis task. For quantitative evaluation of diversity, for each particle-free image, four random latent codes are sampled to generate four granular versions. Performance evaluation is limited to these four generated versions.

[0219] However, in terms of fidelity in the reference-guided synthesis task, a decrease in COSINE similarity scores was observed with smaller style vector sizes, and a sudden increase in LPIPS values ​​was observed with a style vector dimension of 4. Furthermore, six different attributes were used, with their values ​​varied, to create the training and test datasets. In this example, the dimensionality of the style vector can therefore be reduced to 8, which is sufficient to model the properties of the film grain style.

[0220]

[0221] Table 8: Quantitative performance of the lightweight framework for different dimensions of style vectors

[0222] In this example, the style vector dimension is therefore set to 8. Corresponding modifications are made in the output layers of the mapping network and the style encoder, as well as at the input layer of the synthesis network, to account for this new dimension. This style vector of dimension 8 can be applied to two versions: a lightweight version and a deep version. Hereinafter, the deep version of the framework will be referred to as the baseline.

[0223] As indicated above, the lightweight version of each module (network) is obtained independently, for example, using ablation studies. In other words, the performance of each module is evaluated in the framework with its deep architecture (e.g., the architecture mentioned for the deep version in Table 6) for the other modules. The results of the ablation study are presented below. The network that gives the best trade-off between quality and complexity for each module is selected for inclusion in the fully lightweight version of the framework.

[0224] In this example, the deep version of the style encoder is replaced by the following lightweight architecture from the full framework, while keeping the mapping and synthesis networks as in the deep version of the framework: Two different and consecutive simplifications of the architecture are explored. First, the dimensionality of the feature maps is reduced (named StyleEncoder_Res_DimFeatureMaps in Table 9). Then, the residual blocks are replaced with simple convolutional layers (named StyleEncoder_NoRes_DimFeatureMaps). Table 9 summarizes the different tested style encoder networks and their corresponding performance. It appears that exploiting the different dimensionality of the feature maps achieves comparable scores across different metrics. Similar behavior is observed when replacing the residual blocks with simple convolutional layers. Furthermore, replacing the layers of the residual blocks with convolutional layers (StyleEncoder_NoRes_64) significantly reduces complexity while improving performance on the potential-guided synthesis task. Even with a 100x reduction in the number of parameters and approximately 200x reduction in the number of GFLOPs, the style encoder still accurately extracts the granularity parameters, suggesting that the baseline style encoder is over-parameterized and still has room for further reduction.

[0225]

[0226] Table 9: Quantitative evaluation of different style encoder architectures. The underlined ones are the baseline and the selected simplified networks.

[0227] For a lightweight version of the framework, you can choose StyleEncoder_NoRes_64 as the lightweight version and use it because it achieves a good trade-off in terms of performance and the number of floating-point operations.

[0228] In this example, the deep version of the synthesis network is replaced by the following lightweight architecture from the full framework, while keeping the mapping network and style encoder as in the deep version of the framework: The dimensionality of the feature maps (Synthesis_Res_DimFeatureMaps) is reduced before replacing the residual blocks with simple convolutional layers. The network is further simplified by similarly changing the feature map dimensions (Synthesis_NoRes_DimFeatureMaps). Table 10 summarizes the different synthesis networks and their corresponding performance. The worst scores are achieved using feature maps of size 64 (with or without residual blocks). Moreover, for Synthesis_NoRes_64, the COSINE similarity score is close to 1 even when the synthesis is unreliable. This may stem from the malfunctioning style encoder, which outputs the same style vector for each input grainy image, resulting in all style vectors being similar, thus achieving a COSINE similarity score of 1. On the other hand, a relatively low COSINE similarity score can be observed for Synthesis_NoRes_256 compared to the baseline, but better JSD-NSS and LPIPS scores. Therefore, it is preferable to rely not only on the similarity of style vectors but also on the similarity of corresponding synthetic images when evaluating fidelity.

[0229]

[0230] Table 10: Quantitative evaluation of different synthetic network architectures. The underlined ones are the baseline and the selected simplified architectures.

[0231] For both reference-guided synthesis and latent-guided synthesis, the best scores are obtained with feature maps of size 256 (with or without residual blocks). In summary, the overall performance deteriorates when reducing the dimensionality of the feature maps, while replacing the residual blocks with simple convolutional layers does not seem to significantly affect the performance. Comparison of the outputs of the reference-guided synthesis from the baseline and the network with feature maps of size 256 shows that both lightweight versions of the synthesis network are able to generate particle maps that are close to the reference map. In the example, four randomly selected latent codes are fed to the mapping network, which converts them into four different style vectors, which are further used by the synthesis network. From this experiment, a fairly good diversity of the outputs of the synthesis network can be observed. Therefore, for the lightweight version of the framework, Synthesis_NoRes_256 can be used as it achieves a good trade-off in terms of performance and the number of floating-point operations.

[0232] In this example, the deep version of the mapping network is replaced by the following lightweight architecture from the full framework, while keeping the synthesis network and style encoder as in the deep version of the framework: The number of nodes in each fully connected layer is reduced, which does not change the overall architecture of the model but significantly reduces network complexity. In a second step, the number of layers is also reduced. Table 11 summarizes the different mapping network variations tested (Mapping_NbNode_NbLayers) and their corresponding performance. First, the framework's performance does not appear to be affected when changing the mapping network architecture. Second, all networks achieve very similar scores on the potential-guided synthesis task, with the best performance achieved by Mapping_256_5 and Mapping_128_4. This tends to confirm that not all nodes are useful; most of them are redundant in the deep architecture or do not contribute much. The results show that reducing the number of nodes / layers does not strongly affect diversity. This lightweight version achieves a good trade-off between diversity / quality and complexity and was chosen to build the end-to-end lightweight framework.

[0233]

[0234] Table 11: Quantitative evaluation of different mapping network architectures. The underlined ones are the baseline and the selected simplified architectures.

[0235] Therefore, for a lightweight version of the framework, Mapping_128_4 can be used as it offers a good trade-off in terms of performance versus the number of floating point operations.

[0236] The present aspects are not limited to ECM, VVC or HEVC, and may be applied to, for example, other standards and recommendations, and extensions of any such standards and recommendations. Unless otherwise indicated or technically excluded, the various aspects described in this application may be used alone or in combination.

[0237] Various numerical values ​​are used in this application. The specific values ​​are for illustrative purposes, and the described aspects are not limited to these specific values.

[0238] Various implementations involve decoding. As used herein, "decoding" may encompass, for example, all or part of a process performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also includes or alternatively includes processes performed by the decoder of the various implementations described herein, such as obtaining a film grain pattern vector and reconstructing a grainy image based on the film grain pattern vector.

[0239] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding, and in another embodiment, "decoding" refers to the entire reconstructed picture process including entropy decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or generally to a broader decoding process will be clear based on the context of the specific description and is considered to be well understood by those skilled in the art.

[0240] Various implementations relate to encoding. In a manner similar to the discussion above regarding "decoding," "encoding," as used in this application, can encompass, for example, all or part of a process performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, such as segmentation, differential encoding, transforms, quantization, and entropy encoding. In various embodiments, such processes also include or alternatively include processes performed by the encoder of the various implementations described herein, such as obtaining a film grain pattern vector and reconstructing a grainy image based on the film grain pattern vector.

[0241] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or generally to a broader encoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0242] This disclosure has described various pieces of information that can be transmitted or stored, such as syntax, for example. This information can be packaged or arranged in a variety of ways, including, for example, those commonly found in video standards, such as placing the information in an SPS (sequence parameter set), a PPS (picture parameter set), a NAL unit (network abstraction layer), a header (e.g., a NAL unit header or a slice header), or an SEI message. Other approaches are also available, including, for example, those commonly found in system-level or application-level standards, such as placing the information in one or more of the following:

[0243] a. SDP (Session Description Protocol), a format for describing multimedia communication sessions, used for the purpose of session announcement and session invitation, e.g. as described in RFCs and used in conjunction with RTP (Real-time Transport Protocol) transport.

[0244] b. DASH MPD (Media Presentation Description) descriptor, e.g. as used in DASH and transmitted over HTTP, a descriptor is associated with a representation or a set of representations to provide additional characteristics to the content representation.

[0245] c. RTP header extension, eg as used during RTP streaming.

[0246] d. ISO Base Media File Format, eg as used in OMAF, and uses boxes, which are object-oriented building blocks defined by a unique type identifier and a length (also called "atoms" in some specifications).

[0247] e. HLS (HTTP Live Streaming) manifest transmitted over HTTP. A manifest may be associated with a version or set of versions of content, for example, to provide characteristics of the version or set of versions.

[0248] When the figures are presented as flow charts, it should be understood that they also provide block diagrams of the corresponding apparatus. Similarly, when the figures are presented as block diagrams, it should be understood that they also provide flow charts of the corresponding methods / processes.

[0249] Some embodiments relate to rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is often considered, often within computational complexity constraints. Rate-distortion optimization is often expressed as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solving the rate-distortion optimization problem. For example, an approach can be based on extensive testing of all coding options (including all considered modes or coding parameter values), with a complete evaluation of their coding costs and the associated distortion of the reconstructed signal after encoding and decoding. Faster approaches can also be used to avoid coding complexity, particularly by calculating approximate distortion based on a prediction or prediction residual signal rather than a reconstructed residual signal. A hybrid of these two approaches can also be used, such as by using approximate distortion for only some of the possible coding options and full distortion for others. Other approaches only evaluate a subset of the possible coding options. More generally, many approaches employ any of a variety of techniques to perform optimization, but optimization does not necessarily require a complete evaluation of both coding costs and associated distortion.

[0250] The implementations and aspects described herein can be implemented in, for example, a method or process, a device, a software program, a data stream, or a signal. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the features discussed can also be implemented in other forms (e.g., a device or program). The device can be implemented in, for example, appropriate hardware, software, and firmware. The method can be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes communication equipment, such as, for example, a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other equipment that facilitates the transmission of information between end users.

[0251] Reference to "one embodiment" or "an embodiment" or "an implementation" or "an implementation" and other variations thereof means that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation" and any other variations thereof in various places throughout this application are not necessarily all referring to the same embodiment.

[0252] Additionally, the present application may refer to “determining” various pieces of information. Determining information may include, for example, one or more of: estimating information, calculating information, predicting information, or retrieving information from a memory.

[0253] Furthermore, the present application may involve "accessing" various pieces of information. Accessing information may include, for example, one or more of: receiving information, retrieving information (e.g., retrieving information from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0254] Additionally, the present application may involve "receiving" various pieces of information. As with "accessing," receiving is intended to be a broad term. Receiving information may include, for example, one or more of: accessing information or retrieving information (e.g., retrieving information from a memory). Furthermore, "receiving" is generally referred to in one way or another during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0255] It is to be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," use of any of the following " / ," "and / or," and "at least one of..." is intended to encompass selecting only the first-listed option (A), or only the second-listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such wording is intended to encompass selecting only the first-listed option (A), or only the second-listed option (B), or only the third-listed option (C), or only the first-listed option and the second-listed option (A and B), or only the first-listed option and the third-listed option (A and C), or only the second-listed option and the third-listed option (B and C), or all three options (A, B, and C). As will be apparent to one of ordinary skill in this and related arts, this can be extended to as many items as listed.

[0256] Furthermore, as used herein, the term "signaling" refers to, among other things, indicating something to a corresponding decoder. For example, in some embodiments, an encoder signals a specific one of the film grain parameters, such as a pattern vector. Thus, in embodiments, the same parameters are used at both the encoder and decoder sides. Thus, for example, an encoder can transmit (explicitly signal) specific parameters to a decoder so that the decoder can use the same specific parameters. Conversely, if the decoder already has the specific parameters along with other parameters, signaling can be used without transmitting them (implicitly signaling), allowing only the decoder to know and select the specific parameters. By avoiding transmitting any actual functionality, bit savings are achieved in various embodiments. It should be understood that signaling can be implemented in a variety of ways. For example, in various embodiments, information is signaled to a corresponding decoder using one or more syntax elements, flags, and so on. While the verb form of the term "signaling" has been used above, the term "signal" can also be used herein as a noun.

[0257] As will be apparent to one of ordinary skill in the art, implementations can generate a variety of signals formatted to carry information that can be stored or transmitted, for example. The information can include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal can be formatted to carry a bitstream of the described embodiments. Such a signal can be formatted as, for example, an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal can be, for example, analog or digital information. As is well known, signals can be transmitted over a variety of different wired or wireless links. The signal can be stored on a processor-readable medium.

[0258] A number of embodiments have been described above. The features of these embodiments may be provided individually or in any combination across the various claim categories and types.

[0259] A method is disclosed, comprising:

[0260] obtaining an image; and

[0261] A film grain pattern vector representing characteristics of film grains contained in the obtained image is obtained by using a first neural network (S) with the obtained image as input, the first neural network being a network trained to output a film grain pattern vector from an image.

[0262] A method is disclosed, comprising:

[0263] obtaining an image and a film grain pattern vector, the film grain pattern vector representing characteristics of film grain; and

[0264] A grainy image is reconstructed by using a second neural network (G) with the obtained image and the obtained film grain pattern vector as input, the second neural network being a network trained to output a grainy image from the film grain pattern vector and the image.

[0265] A method is disclosed, comprising:

[0266] Obtain an image;

[0267] obtaining a film grain pattern vector representing characteristics of film grain by using a third neural network with the latent code as input, the third neural network being a network trained to output the film grain pattern vector from the latent code;

[0268] A grainy image is reconstructed by using a fourth neural network on the received image and the film grain pattern vector, the fourth neural network being a network trained to output a grainy image from the film grain pattern vector and the image.

[0269] In an example, the second neural network and the fourth neural network are the same neural network, and the method comprises jointly learning the first neural network, the second neural network, and optionally the third neural network from a dataset using a discriminator neural network, the dataset comprising N different grain-free images and, for each of the N grain-free images, M grainy versions, where N and M are integers.

[0270] In the example, N=148694, and M=5.

[0271] In an example, jointly learning the first neural network, the second neural network, and optionally the third neural network using a discriminator neural network includes: for a triplet of a grain-free image, a first grainy version of the grain-free image, and a second grainy version:

[0272] a first forward pass comprising using the first neural network with the first grainy version as input to output a first film grain pattern vector and with the second grainy version as input to output a second film grain pattern vector, using the second neural network with the grainless image and the first film grain pattern vector as input to output a first grainy image, and using the grainless image and the second film grain pattern vector as input to output a second grainy image, using a discriminator neural network with the first grainy version as input and with the first grainy image as input, using the first neural network with the first grainy image as input to output a third film grain pattern vector, and with the second grainy image as input to output a fourth film grain pattern vector; and deriving a first global loss function from the outputs;

[0273] A first backward pass is used to update at least one of the first neural network, the second neural network, and the discriminator neural network by backpropagating the first global loss function.

[0274] In an example, jointly learning the first neural network, the second neural network, and the third neural network using the discriminator neural network further includes: for a triplet of a grain-free image, a first grainy version of the grain-free image, and a second grainy version:

[0275] a second forward pass comprising using a third neural network with the first latent code as input to output a fifth film grain pattern vector and with the second latent code as input to output a sixth film grain pattern vector, using a second neural network with the grainless image and the fifth film grain pattern vector as input to randomly sample the first latent code and the second latent code to output a third grainy image, and with the grainless image and the sixth film grain pattern vector as input to output a fourth grainy image, using a discriminator neural network with the first grainy version as input and with the third grainy image as input, using the first neural network with the third grainy image as input to output a seventh film grain pattern vector, and with the fourth grainy image as input to output an eighth film grain pattern vector; and deriving a second global loss function from the outputs;

[0276] The second backward pass is used to update the first neural network, the second neural network, the third neural network and the discriminator neural network by back-propagating the second global loss function.

[0277] In the example, a first forward and backward pass is repeated with new triplets, followed by a second forward and backward pass, until all possible triplets in the dataset have been used.

[0278] In an example, the first global loss function is a sum of basic first loss functions, and at least one first basic loss function is defined to ensure that the cosine similarity between the first film grain pattern vector and the second film grain pattern vector is close to the cosine similarity between the third film grain pattern vector and the fourth film grain pattern vector.

[0279] In an example, at least one second basis loss function is defined to ensure that the first grainy image is close to the first grainy version.

[0280] In an example, the second global loss function is a sum of basic second loss functions, wherein the basic second loss function includes at least one basic second loss function, and the at least one basic second loss function is defined to ensure that the cosine similarity between the first latent code and the second latent code is close to the cosine similarity between the fifth film grain pattern vector and the sixth film grain pattern vector.

[0281] The style network (or style encoder) S is configured to output a film grain style vector from an input image, enabling film grain analysis and modeling given only a grainy image (without prior filtering), enabling film grain analysis in the entire input image (not only in homogeneous regions), enabling strong fidelity to some grainy reference image due to the high modeling power of deep learning models, enabling a rich yet compact and easily transferable representation of grain properties, enabling a content-independent style representation (so that when transferred, redundant information from the content is not transferred again).

[0282] The mapping network M is configured to output a film grain style vector from the latent code, enabling a style vector to be provided when no reference is available, enabling diversity in output grain styles, and enabling an output style vector that is compatible with a style vector output by a style encoder.

[0283] The synthesis network G is configured to generate a grain style based on some input style vector and blend it to a given input image in a content-adaptive manner.

[0284] The joint learning of the above networks outperforms the state-of-the-art performance for particle analysis and synthesis, especially in compression chains, without the need for any pre-filtering or edge detection steps.

[0285] The proposed loss function used in the joint learning ensures the fidelity and diversity of grain styles. Film grain analysis is not conditioned on the grain map since it directly takes the grain image as input.

Claims

1. A method comprising: obtaining (S800) an image; as well as A film grain pattern vector representing characteristics of film grains contained in an obtained image is obtained (S802, 700) by using a first neural network (S) with the obtained image as input, the first neural network being a network trained to output a film grain pattern vector from an image.

2. The method according to claim 1, comprising: obtaining (S804) another image; A grainy image is reconstructed (S806, 702) by using a second neural network (G) with the other image and the obtained film grain pattern vector as input, the second neural network being a network trained to output a grainy image from the film grain pattern vector and the image.

3. A method comprising: obtaining ( S804 ) an image and a film grain pattern vector, the film grain pattern vector representing characteristics of film grain; as well as A grainy image is reconstructed (S806) using a second neural network (G) with the obtained image and the obtained film grain pattern vector as input, the second neural network being a network trained to output a grainy image from the film grain pattern vector and the image.

4. A method comprising: obtaining (S1200) an image; obtaining (S1202) a film grain pattern vector representing characteristics of film grain by using a third neural network (M) with the latent code as input, the third neural network being a network trained to output the film grain pattern vector from the latent code; A grain image is reconstructed (S1204) by using a second neural network (G) on the received image and the film grain pattern vector, the second neural network being a network trained to output a grainy image from the film grain pattern vector and the image.

5. A method comprising jointly learning a first neural network (S) and a second neural network (G) from a dataset using a discriminator neural network (D), the dataset comprising N different grain-free images and, for each of the N grain-free images, M grainy versions of each grain-free image, N and M being integers, the first neural network being a network trained to output a film grain pattern vector from the image, and the second neural network (G) being a network trained to output a grainy image from the film grain pattern vector and the image. The method according to claim 5 , wherein N=148694 and M=5.

7. The method according to any one of claims 5 to 6, wherein jointly learning the first neural network (S) and the second neural network (G) using the discriminator neural network (D) comprises: For a triplet of a grain-free image (x_src), a first grainy version of the grain-free image (x_ref1), and a second grainy version (x_ref2): A first forward pass, which includes using a first neural network (S) with the first grainy version (x_ref1) as input to output a first film grain pattern vector (s_ref1), and with the second grainy version (x_ref2) as input to output a second film grain pattern vector (s_ref2), using a second neural network (G) with the grainless image (x_src) and the first film grain pattern vector (s_ref1) as input to output a first grainy image and uses the grainless image (x_src) and the second film grain pattern vector (s_ref2) as input to output a second grainy image Use the discriminator neural network (D) to extract the first granular version (x_ref1) and the first granular image As input, a discriminator neural network (D) is used with a first cascade of a particle-free image (x_src) and a predicted particle map and a second cascade of a particle-free image (x_src) and a reference particle map, wherein the predicted particle map is calculated as a first granular image and a grain-free image (x_src), and wherein the reference grain map is calculated as the difference between the first grainy version (x_ref1) and the grain-free image (x_src), using a first neural network (S) to Outputs a third film grain pattern vector as input And with a second grainy image Outputs the fourth film grain pattern vector as input and deriving a first global loss function from the output; A first backward pass for updating at least one of the first neural network (S), the second neural network (G), and the discriminator neural network (D) by backpropagating the first global loss function.

8. The method of claim 7, wherein jointly learning the first neural network (S) and the second neural network (G) using the discriminator neural network (D) comprises: The first neural network (S), the second neural network (G), and the third neural network (M) are jointly learned using a discriminator neural network (D), and the third neural network is a network trained to output a film grain pattern vector from a latent code.

9. The method of claim 8, wherein jointly learning the first neural network (S), the second neural network (G), and the third neural network (M) using the discriminator neural network (D) further comprises: A second forward pass, which includes using a third neural network (M) with the first latent code as input to output a fifth film grain pattern vector (s_lat1), and using the second latent code as input to output a sixth film grain pattern vector (s_lat2), and using a second neural network (G) with the grainless image (x_src) and the fifth film grain pattern vector (s_lat1) as input to randomly sample the first latent code and the second latent code to output a third grainy image And takes the grainless image (x_src) and the sixth film grain pattern vector (s_lat2) as input to output a fourth grainy image Use the discriminator neural network (D) to extract the first granular version (x_ref1) and the third granular image As input, a discriminator neural network (D) is used with a first cascade of a particle-free image (x_src) and a predicted particle map and a second cascade of a particle-free image (x_src) and a reference particle map, wherein the predicted particle map is calculated as a first granular image and a grain-free image (x_src), and wherein the reference grain map is calculated as the difference between the first grainy version (x_ref1) and the grain-free image (x_src), using a first neural network (S) to generate a third grainy image Outputs the seventh film grain pattern vector as input And with the fourth granular image Outputs the eighth film grain pattern vector as input and deriving a second global loss function from the output; A second backward pass for updating at least one of the first neural network (S), the second neural network (G), the third neural network (M), and the discriminator neural network (D) by backpropagating a second global loss function.

10. The method of claim 9, wherein a first forward and backward pass is repeated with a new set of values ​​(x_src, x_ref1, x_ref2, z1, z2) followed by a second forward and backward pass.

11. The method according to any one of claims 7 to 10, wherein The first global loss function is a sum of basic first loss functions, at least one first basic loss function is defined to ensure that the cosine similarity between the first film grain pattern vector and the second film grain pattern vector (s_ref1, s_ref2) is close to the cosine similarity between the third film grain pattern vector and the fourth film grain pattern vector The cosine similarity between .

12. The method according to claim 11, wherein At least another first basic loss function is defined to ensure a first grainy image Close to the first granular version (x_ref1).

13. The method according to claim 9 or claim 10, wherein: The second global loss function is the sum of basic second loss functions, wherein the basic second loss function includes at least one basic second loss function, and the at least one basic second loss function is defined to ensure that the cosine similarity between the first latent code and the second latent code (z1, z2) is close to the cosine similarity between the fifth film grain pattern vector and the sixth film grain pattern vector (s_lat1, s_lat2).

14. An apparatus comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to perform the method according to claim 1. 15 . An apparatus comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to perform the method according to claim 3 . 16 . An apparatus comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to perform the method according to claim 4 .

17. A computer program comprising program code instructions for implementing the method of any one of claims 1 to 13 when executed by a processor.

18. A computer-readable storage medium having stored thereon instructions for implementing the method according to any one of claims 1 to 13.

Images