Method and apparatus for film particle modeling

Abstract

A method and apparatus for selecting a region for film particle modeling is provided. A reconstructed image is obtained by encoding an input image. At least one region of a reconstructed image in encoding an input image is selected based on at least one encoding parameter of the region, and film particle parameters are determined from the selected at least one region and the input image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to European application number 22305671.4 filed on May 5, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present embodiments generally relate to video compression, distribution and rendering, and more particularly to film grain modeling.The present embodiments relate to methods and apparatus for detecting uniform regions for film grain modeling. Background Art

[0004] To achieve high compression efficiency, image and video coding schemes typically use prediction and transform to exploit spatial and temporal redundancy in video content. Typically, intra-frame or inter-frame prediction is used to exploit intra- or inter-picture correlations, and then the difference between the original block and the predicted block (usually expressed as a prediction error or prediction residual) is transformed, quantized, and entropy encoded. To reconstruct the video, the compressed data is decoded through the inverse process corresponding to entropy coding, quantization, transform, and prediction.

[0005] Film grain is a special type of noise that appears in videos and is very pleasing and very unique. Essentially, film grain is the result of the physical process of exposure and development of analog film. Film grain was originally a noise that was created as a result of the photographic process and consisted of sensor noise, which was naturally present and unique to each analog camera. With the advent of the digital camera era, this sensor noise disappeared during the capture phase, but is now added to content in post in order to reproduce the film style. The random nature of this noise makes it difficult to compress using traditional encoding tools. Common parameters of encoding tools, such as those selected for low bitrates, often eliminate film grain. High bitrates are required to maintain and reconstruct film grain with sufficient quality, which runs counter to the encoding / decoding goal of saving bits when encoding content. To overcome this encoder filtering problem, film grain modeling is usually performed before the encoding phase and used during the so-called compositing step of the decoding phase to add film grain back to the reconstructed video. Summary of the invention

[0006] According to one aspect, a method for selecting a region for film grain modeling is provided. The method includes: obtaining a reconstructed image by encoding an input image; selecting the region based on at least one encoding parameter of at least one region of the reconstructed image when encoding the input image; and determining film grain parameters based on the selected at least one region and the input image.

[0007] According to another aspect, an apparatus for selecting a region for film grain modeling is provided. The apparatus includes one or more processors operable to: obtain a reconstructed image by encoding an input image; select at least one region of the reconstructed image based on at least one encoding parameter of the region when encoding the input image; and determine film grain parameters based on the selected at least one region and the input image.

[0008] One or more embodiments also provide a computer program comprising instructions, wherein the instructions, when executed by one or more processors, cause the one or more processors to perform a method for selecting a region for film grain modeling according to any embodiment described herein. One or more embodiments also provide a computer readable storage medium having stored thereon instructions for selecting a region for film grain modeling according to the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A block diagram of a system is shown in which aspects of the present embodiments may be implemented.

[0010] Figure 2 A block diagram of an embodiment of a video encoder is shown.

[0011] Figure 3 A block diagram of an embodiment of a video decoder is shown.

[0012] Figure 4 An example of block partitioning is shown.

[0013] Figure 5 An example of a method for film grain encoding / decoding is shown.

[0014] Figure 6 An example of a method for encoding / decoding video content using film grain modeling is shown.

[0015] Figure 7 An example of a method for encoding / decoding video content using film grain modeling according to an embodiment is shown.

[0016] Figure 8 An example of a method for determining film grain parameters according to an embodiment is shown.

[0017] Fig. 9 An example of a method for determining a region for determining film grain parameters according to an embodiment is shown.

[0018] Fig.10 Examples of partition-driven block selection are shown for QP22 (top) and QP (32) bottom.

[0019] Fig.11Examples of block selection based on intra-mode driving in planar mode (top) and DC mode (bottom) are shown.

[0020] Fig.12 An example of residual driven block selection based on code block flag values ​​is shown.

[0021] Fig.13 An example of cost-driven block selection based on a bpp cost less than a given value is shown.

[0022] Fig.14 A block diagram of a system in which aspects of the present embodiment may be implemented is shown according to another embodiment.

[0023] Fig.15 Two remote devices are shown communicating over a communication network according to an example of the present principles.

[0024] Fig.16 A signal syntax according to an example of the present principles is shown. DETAILED DESCRIPTION

[0025] The present application describes a number of aspects, including tools, features, embodiments, models, schemes, etc. Many of these aspects are described as having specificity, and are usually described in a manner that may sound restrictive, at least to show the various features. However, this is for the purpose of describing clarity and does not limit the application or scope of these aspects. In fact, all different aspects can be combined and interchanged to provide further aspects. In addition, these aspects can also be combined and interchanged with the aspects described in the previous submission documents.

[0026] The aspects described and contemplated in this application can be implemented in many different forms. Figure 1 , 2 and 3 provide some embodiments, but other embodiments are contemplated, and with respect to Figure 1 , 2 The discussion of 3 does not limit the breadth of the embodiments. At least one aspect generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media storing instructions for encoding or decoding video data according to any of the described methods, and / or computer-readable storage media storing a bitstream generated according to any of the described methods.

[0027] In this application, the terms "reconstruction" and "decoding" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image", "picture" and "frame" may be used interchangeably.

[0028] Various methods are described herein, and each method includes one or more steps or actions for implementing the method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions can be modified or combined. In addition, in various embodiments, terms such as "first", "second" and the like can be used to modify elements, components, steps, operations, etc., such as "first decoding" and "second decoding". Unless specifically required, the use of these terms does not mean that the modified operations are sorted. Therefore, in this example, the first decoding does not need to be performed before the second decoding, and can, for example, occur before, during, or in a time period overlapping with the second decoding.

[0029] The present aspects are not limited to VVC or HEVC, and may be applied, for example, to other standards and recommendations (whether existing or developed in the future) and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described in this application may be used alone or in combination.

[0030] Figure 1 A block diagram of an example of a system that can implement various aspects and embodiments is shown. System 100 can be embodied as a device including various components described below, and is configured to perform one or more aspects described in this application. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, notebook computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances and servers. The elements of system 100 can be embodied as a single integrated circuit, multiple ICs and / or discrete components, either alone or in combination. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed on multiple ICs and / or discrete components. In various embodiments, system 100 is coupled to other systems or other electronic devices in a communication manner, such as by a communication bus or by a dedicated input and / or output port. In various embodiments, system 100 is configured to implement one or more aspects described in this application.

[0031] The system 100 includes at least one processor 110 configured to execute instructions loaded therein for implementing, for example, the various aspects described in the present application. The processor 110 may include embedded memory, input-output interfaces, and various other circuits known in the art. The system 100 includes at least one memory 120 (e.g., a volatile memory device and / or a non-volatile memory device). The system 100 includes a storage device 140, which may include a non-volatile memory and / or a volatile memory, including but not limited to an EEPROM, a ROM, a PROM, a RAM, a DRAM, an SRAM, a flash memory, a magnetic disk drive, and / or an optical disk drive. As non-limiting examples, the storage device 140 may include an internal storage device, an additional storage device, and / or a network accessible storage device.

[0032] The system 100 includes an encoder / decoder module 130, which is configured to, for example, process data to provide encoded video or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents a module that may be included in a device to perform encoding and / or decoding functions. As is well known, a device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 130 may be implemented as a separate element of the system 100, or may be incorporated into the processor 110 as a combination of hardware and software known to those of ordinary skill in the art.

[0033] Program code to be loaded onto the processor 110 or the encoder / decoder 130 to perform various aspects described herein may be stored in the storage device 140 and subsequently loaded onto the memory 120 for execution by the processor 110. According to various embodiments, one or more of the processor 110, the memory 120, the storage device 140, and the encoder / decoder module 130 may store one or more of a variety of items during execution of the processes described herein. These stored items may include, but are not limited to, input video, decoded video or a portion thereof, bitstreams, matrices, variables, and intermediate or final results from processing of equations, formulas, operations, and operational logic.

[0034] In some embodiments, memory inside the processor 110 and / or the encoder / decoder module 130 is used to store instructions and provide processing working memory required during encoding or decoding. However, in other embodiments, memory outside the processing device (for example, the processing device can be the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory can be a memory 120 and / or a storage device 140, such as a dynamic volatile memory and / or a non-volatile flash memory. In multiple embodiments, the external non-volatile flash memory is used to store the operating system of the TV. In at least one embodiment, a fast external dynamic volatile memory (such as RAM) is used as a working memory for video encoding and decoding operations (for example, MPEG-2 (MPEG refers to the Moving Picture Experts Group, MPEG-2 is also called ISO / IEC 13818, 13818-1 is also called H.222, and 13818-2 is also called H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2) or VVC (Versatile Video Coding, a new standard being developed by the Joint Video Experts Group JVET)).

[0035] Input to the elements of system 100 may be provided through a variety of input devices as shown in block 105. Such input devices include, but are not limited to: (i) a radio frequency (RF) section that receives an RF signal transmitted over the air by, for example, a broadcaster; (ii) a component (COMP) input terminal (or a set of COMP input terminals); (iii) a universal serial bus (USB) input terminal; and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 1 Other examples not shown include composite video.

[0036] In various embodiments, the input device of block 105 has associated respective input processing elements known in the art. For example, the RF section may be associated with elements adapted to (i) select a desired frequency (also referred to as selecting a signal or bandwidth limiting a signal to a frequency band), (ii) downconvert the selected signal, (iii) again bandwidth limit a narrower frequency band to select (for example) a signal frequency band (which may be referred to as a channel in some embodiments), (iv) demodulate the downconverted and bandwidth limited signal, (v) perform error correction, and (vi) demultiplex to select a desired data packet stream. The RF section of various embodiments includes one or more elements to perform these functions, such as a frequency selector, a signal selector, a bandwidth limiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs a variety of these functions, including, for example, downconverting a received signal to a lower frequency (e.g., an intermediate frequency or near baseband frequency) or baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal transmitted by wired (for example, cable) medium, and filter, down-convert and filter to the desired frequency band again to perform frequency selection. Various embodiments rearrange the order of above-mentioned (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding element can include inserting element between existing elements, for example inserting amplifier and analog / digital converter. In various embodiments, the RF part includes antenna.

[0037] In addition, the USB and / or HDMI terminals may include respective interface processors for connecting the system 100 to other electronic devices via the USB and / or HDMI connections. It should be appreciated that aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, for example, within a separate input processing IC or within the processor 110 as desired. Similarly, aspects of USB or HDMI interface processing may be implemented, for example, within a separate interface IC or within the processor 110 as desired. The demodulated, error-corrected, and demultiplexed streams are provided to a variety of processing elements, including, for example, the processor 110 and the encoder / decoder 130, which operate in conjunction with memory and storage elements to process the data streams as desired for presentation on an output device.

[0038] The various elements of the system 100 may be disposed within an integrated housing. Within the integrated housing, the various elements may be interconnected and transmit data between each other using suitable connection means 115 (eg, an internal bus known in the art, including an I2C bus, wiring, and printed circuit boards).

[0039] The system 100 includes a communication interface 150 that enables communication with other devices via a communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to send and receive data through the communication channel 190. The communication interface 150 may include, but is not limited to, a modem or a network card, and the communication channel 190 may be implemented, for example, within a wired and / or wireless medium.

[0040] In various embodiments, a Wi-Fi network (e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers)) is used to stream data to the system 100. The Wi-Fi signals of these embodiments are received through a communication channel 190 and a communication interface 150 suitable for Wi-Fi communication. The communication channel 190 of these embodiments is typically connected to an access point or router that provides access to an external network (including the Internet) to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box to provide streaming data to the system 100, which transmits data through the HDMI connection of the input block 105. Other other embodiments use the RF connection of the input block 105 to provide streaming data to the system 100. As described above, various embodiments provide data in a non-streaming manner. In addition, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0041] The system 100 can provide output signals to a variety of output devices, including a display 165, a speaker 175, and other peripherals 185. The display 165 of various embodiments includes, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. The display 165 can be used for a television, a tablet computer, a laptop computer, a cell phone (mobile phone), or other devices. The display 165 can also be integrated with other components (for example, as in a smartphone) or be separate (for example, an external display of a laptop computer). In examples of various embodiments, other peripherals 185 include one or more of a stand-alone digital video disc (or digital versatile disc) (DVR, for both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripherals 185 that provide functions based on the output of the system 100. For example, a disc player performs the function of playing the output of the system 100.

[0042] In various embodiments, control signals are communicated between the system 100 and the display 165, speakers 175, or other peripheral devices 185 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. Output devices may be communicatively coupled to the system 100 via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, output devices may be connected to the system 100 through the communication interface 150 using the communication channel 190. The display 165 and speakers 175 may be integrated into a single unit with other components of the system 100 in an electronic device, such as a television. In various embodiments, the display interface 160 includes a display driver, such as a timing controller (TCon) chip.

[0043] Display 165 and speaker 175 may also be separate from one or more other components, for example, if the RF portion of input 105 is part of a separate set-top box. In various embodiments where display 165 and speaker 175 are external components, output signals may be provided through dedicated output connections including, for example, an HDMI port, a USB port, or a COMP output.

[0044] The embodiments may be performed by computer software implemented by the processor 110 or by hardware or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 120 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory as non-limiting examples. The processor 110 may be of any type suitable for the technical environment and may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture as non-limiting examples.

[0045] Figure 2 An encoder 200 is shown. Variations of the encoder 200 are contemplated, but for clarity, only the encoder 200 is described below, rather than describing all contemplated variations.

[0046] In some embodiments, Figure 2 Also shown are encoders that improve the HEVC standard or the VVC standard or encoders that adopt techniques similar to HEVC or VVC, such as encoders being developed by JVET (Joint Video Exploration Team).

[0047] Prior to encoding, the video sequence may undergo pre-encoding processing (201), such as applying a color transform to an input color picture (e.g., converting from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of input picture components to obtain a signal distribution more amenable to compression (e.g., using histogram equalization of color components), or resizing the picture (e.g., downscaling). Metadata may be associated with the pre-processing and appended to the bitstream.

[0048] In encoder 200, a picture is encoded by encoder elements as described below. The picture to be encoded is divided (202) and processed in units such as CUs. Each unit is encoded using, for example, intra or inter mode. When the unit is encoded in intra mode, it performs intra prediction (260). In inter mode, motion estimation (275) and compensation (270) are performed. The encoder decides (205) which of intra mode or inter mode to use to encode the unit, and indicates the intra / inter decision by, for example, a prediction mode flag. The encoder can also mix (263) intra prediction results and inter prediction results, or mix results from different intra / inter prediction methods. For example, the prediction residual is calculated by subtracting (210) the prediction block from the original image block.

[0049] The motion refinement module (272) uses the already available reference image to refine the motion field of the block without referring to the original block. The motion field for a region can be considered as the set of motion vectors for all pixels in the region. If the motion vectors are based on sub-blocks, the motion field can also be represented as the set of motion vectors for all sub-blocks in the region (all pixels within a sub-block have the same motion vector, and the motion vector may vary between different sub-blocks). If a single motion vector is used for the region, the motion field for the region can also be represented by a single motion vector (the motion vector is the same for all pixels in the region).

[0050] The prediction residual is then transformed (225) and quantized (230). The quantized transform coefficients, along with motion vectors and other syntax elements, are entropy encoded (245) to output a bitstream. The encoder may skip the transform and apply quantization directly to the untransformed residual signal. The encoder may bypass the transform and quantization, i.e., encode the residual directly without applying the transform or quantization process.

[0051] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are inverse quantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual and the predicted block are combined (255) to reconstruct the image block. A loop filter (265) is applied to the reconstructed image to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (280).

[0052] Figure 3 A block diagram of a video decoder 300 is shown. In the decoder 300, the bitstream is decoded by the decoder elements as described below. The video decoder 300 generally performs the same Figure 2 The decoding process is the reverse of the encoding process described in . The encoder 200 also typically performs video decoding as part of the encoding of the video data.

[0053] Specifically, the input to the decoder includes a video bitstream, which may be generated by the video encoder 200. The bitstream is first entropy decoded (330) to obtain transform coefficients, motion vectors, and other encoding information. Picture partition information indicates how the picture is partitioned. Thus, the decoder can segment (335) the picture according to the decoded picture partition information. The transform coefficients are inverse quantized (340) and inverse transformed (350) to decode the prediction residual. The decoded prediction residual and the predicted block are combined (355) to reconstruct the image block.

[0054] The predicted block may be obtained (370) by intra prediction (360) or motion compensated prediction (i.e., inter prediction) (375). The decoder may blend (373) intra prediction results and inter prediction results, or blend results from multiple intra / inter prediction methods. Prior to motion compensation, the motion field may be refined (372) using already available reference pictures. A loop filter (365) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (380).

[0055] The decoded picture may be further subjected to post-decoding processing (385), such as inverse color conversion (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping that performs the inverse of the remapping process performed in the pre-encoding process (201), or resizing the reconstructed picture (e.g., upscaling). The post-decoding processing may use metadata derived in the pre-encoding processing and signaled in the bitstream.

[0056] Block partition structure

[0057] In VVC, a picture is split or divided into multiple non-overlapping coding tree units (CTUs). For luma samples, the CTU size in VVC can be set to 128×128. Each coding tree unit (CTU) is either processed as a coding unit or split into multiple coding units (CUs) through one or more recursive quadtree (QTBT) partitions, followed by one or more recursive multi-type tree partitions, such as binary tree (BT) or ternary tree (TT). The latter can be horizontal binary tree partitioning (SPLIT_BT_HOR), vertical binary tree partitioning (SPLIT_BT_VER), horizontal ternary tree partitioning (SPLIT_TT_HOR) or vertical ternary tree partitioning (SPLIT_TT_VER), such as Figure 4 As shown. The CTU dual tree for intra-coded slices is described on the basis of the block partitioning structure, which separates the coding trees for luminance and chrominance. The partitions (QTBT, BT or TT) applied to each CTU and resulting in one or more CUs have square and rectangular shapes. They can better adapt to the local picture content characteristics and features.

[0058] By increasing the CTU size to 128×128 for luma, coding efficiency can be improved, but at the expense of increased complexity. The size of each CU is related to the information present in that CU, or more precisely, to the rate-distortion (RD) cost of the CU. The rate-distortion (RD) cost of a candidate block can be calculated as follows:

[0059] CostRD=D+Lambda(QP)×R

[0060] Where D is the distortion of the current block and its coding parameters (e.g. L2, i.e., Euclidean distance), R is the correlation rate (expressed in the number of coded bits), and Lambda (QP) is the Lagrangian parameter derived from the quantization parameter QP. High QP generates fewer but larger CUs than small QP. After the division is completed in the encoding stage, multiple features are available for intra frames and each CU as follows. The intra prediction mode indicates which prediction mode is used, such as planar or angular prediction. The code block flag (CBF) indicates whether a residual is added on top of the prediction. The CBF flag is encoded per component. The encoder can extract the bit cost per pixel (bpp) during the encoding stage. DCT transform coefficients can also be used, which are the values ​​of the transformed and quantized coefficients. The importance flag and the last xy can be used. These parameters signal which coding groups within the transform unit contain the coefficients.

[0061] Film Grain Modeling

[0062] Modern video compression and distribution systems may provide a mechanism to remove film grain before and / or during compression and automatically add it back on the user side in a controllable manner using parametric models. Figure 5 A block diagram showing an example of film grain usage in a video coding framework is shown. Typically, an encoder removes (500) original film grain from a video and estimates film grain model parameters based on uniform region selection (501) and original content analysis (502).

[0063] For film grain estimation, uniform regions in the image are detected. Common techniques for detecting uniform regions include applying some pre-filtering, such as denoisers or edge detection, outside the encoding process. This is an overhead for the encoding stage, as uniform region detection is an additional process performed after pre-processing the frame (to remove film grain from the input video) and encoding the frame. It consumes CPU and memory.

[0064] exist Figure 5 In the illustrated framework, content without film grain is encoded (503) and transmitted to a decoder, which decodes the transmitted content (504) and replaces (505) the removed film grain with a synthetic replacement that visually approximates the removed film grain.

[0065] In this way, information about film grain is communicated as additional metadata, e.g. Figure 6 As shown in the embodiment, Figure 6 A block diagram showing an example of film grain usage in a video coding framework according to another example. Figure 5 As described above, film grain preservation has two stages: modeling (region selection and analysis) and compositing.

[0066] The embodiments presented herein focus on modeling, and more precisely on granular analysis of / noise extraction from raw content. Figure 6 As shown, the grain analysis uses an original frame with grain (input video) and a corresponding filtered frame without film grain (filtered video). The filtered frame is obtained by a denoising step (600, preprocessing). Flat areas are then extracted (601) from the filtered frame, as they will be used as a mask for estimating (602) grain parameters in the grain layer, which are determined as the difference between the original frame and its filtered version. The filtered frame is encoded (603).

[0067] After estimation, the grain parameters (FG parameters) are sent with the compressed video bitstream, for example via a SEI message (FG SEI).After decoding (604), film grain is synthesized (605) based on the film grain parameters and added back to the reconstructed video frame.

[0068] To determine film grain parameters, denoising and flat area detection are used in grain analysis.

[0069] Many approaches have addressed the problem of film grain removal, from using the video encoder as a denoiser to Wiener-type filters, nonlinear denoising using total variation minimization, and multi-hypothesis motion compensated filtering. MCTF (Motion Compensated Temporal Filtering), a video filtering tool in VVC, is also an effective denoiser. After applying denoising, flat areas are estimated so that only flat / uniform / smooth areas of the picture are used in the estimation, as edges and textures affect the estimation of film grain intensity and pattern. To determine the smooth areas of the picture, the Canny edge detector is usually applied to the denoised image at different scales, followed by a dilation operation.

[0070] It can be seen that the above approach relies on an additional process to extract the film grain parameters, which may not be suitable for very low complexity / latency encoders.

[0071] exist Figure 5 and Figure 6 In the example shown, the workflow of a conventional solution can be summarized as follows. Film grain analysis and extraction is done outside the codec workflow. Film grain parameters are sent (usually using SEI messages) to the decoder. The compositor puts the film grain back on the reconstructed image for display.

[0072] According to an embodiment, a method and apparatus for determining film grain parameters using encoding process results is provided. This allows fast and reliable film grain parameter extraction to be performed by selecting regions based on encoding parameters obtained during encoding of input frames.

[0073] In an embodiment, only intra-coded frames are considered for region selection.

[0074] In another embodiment, the region is selected based on coding unit (CU) partitioning of the intra frame during encoding.

[0075] In some embodiments, the flat area can be selected by analyzing relevant information of each CU from CTU brightness, for example, considering at least one or a combination of the following parameters: QP, coding unit (CU) size, intra-frame prediction mode (MPI, DC, planar, angular), code block flag (CBF) OFF or ON, bit per pixel (BPP) cost, Last_xy_sig, DCT coefficient distribution.

[0076] In this way, the detection of uniform areas for film grain determination is accelerated and improved.

[0077] Figure 7An example of a method for encoding / decoding video content using film grain modeling according to an embodiment is shown. In this embodiment, steps 602-605 are Figure 6 The steps described in are similar to those described in , except that the encoding of the input frame (603) replaces Figure 6 In the embodiment presented here, the encoder is used as a denoiser. In other words, the encoding step of the encoder is used to remove film grains of the input frame.

[0078] Furthermore, in this embodiment, region detection (700) is driven by encoding information provided at the encoder for encoding the input frame. Thus, an additional process for region detection is avoided.

[0079] In an embodiment, for detecting flat / homogeneous areas, only the partitions for each CU and the intra frames where the information is more relevant are considered.In an embodiment, only the luma component is used for the analysis.

[0080] When encoding an input frame, each CTU is either processed as a CU or split into multiple CUs through a recursive QT and then a recursive binary ternary tree (BTT), also known as a multi-type tree (MTT). As mentioned above, the rate-distortion (RD) cost of the candidate block is driven by the quantization parameter QP. QP defines the quantization level.

[0081] At high bitrates and high PSNRs, the QP value is low, while at low bitrates and low PSNRs, the QP value is high.

[0082] For QP values ​​above a given QP value, e.g., QP>=22, film grain is filtered and not preserved when encoding the input frame. Thus, in this embodiment, the encoding can be viewed as a demister and can be used to estimate a filtered version of the input frame without film grain. In this case, no external demister is needed.

[0083] In another embodiment, when the QP value is higher than a given QP value, such as QP<22, film grain is no longer removed from the input frame when encoding the input frame. Therefore, the input frame still needs to be pre-processed to filter and remove film grain before encoding the input frame. The denoiser can be any denoiser external to the video encoder tool or an internal denoiser, such as image filtering with or without temporal information, such as the MCTF tool of VVC. Therefore, in this embodiment, Figure 7 The method shown in includes an additional film grain removal pre-processing step. However, the region selection (700) is still based on at least one of the encoding parameters obtained for a CU of the input frame when encoding the input frame.

[0084] Figure 8An example of a method for determining film grain parameters according to an embodiment is shown. At 800, a reconstructed image is obtained by encoding an input image. As described above, the input image may be processed to remove film grains before encoding, or may not be processed, depending on the QP used to encode the input image. At 801, at least one region of the reconstructed image is selected based on at least one encoding parameter of the region when encoding the input image. At 802, film grain parameters are determined based on the selected at least one region and the input image.

[0085] Several embodiments of the region selection process are described below. The embodiments can be used alone or in combination.

[0086] Partition drive block selection

[0087] The block size selection is highly dependent on the QP value; higher and lower QP values ​​mean choosing larger and smaller block sizes, respectively. Lower QP values ​​preserve image details, while higher QP values ​​provide higher compression ratios at the expense of image quality loss. If the cost is not satisfactory for a certain CU, the CU is split. The higher the QP, the larger the CU. Uniform areas with low costs should be located in CUs with large surfaces.

[0088] Fig.10 Examples of quadtree partitioning obtained for QP22 (top) and QP (32) bottom are shown. It can be seen that for the same region in the frame, the quadtree partitioning is not the same depending on the QP value.

[0089] In this embodiment, all small CUs are discarded (i.e., not selected), and only large CUs are analyzed. Small CUs should be understood as CUs whose size is smaller than a given CU size. Similarly, large CUs should be understood as CUs whose size is larger than a given CU size. For example, in order to obtain a relevant model of film grain, a CU block size of 8×8 is too small. 32×32 and 16×16 or 64×64 and 32×32 and 16×16 can be considered good choices.

[0090] When determining film grain from a large CU, if the block size is not 64 × 64, block scaling to 64 × 64 is applied. Experimental results show that at least 4 64 × 64 blocks are required to estimate film grain parameters well.

[0091] Intra-mode driven block selection

[0092] In this embodiment, intra-mode coding information is used to check whether the CU is encoded using directional information. In VVC, intra-mode coding contains 6 most probable modes. Planar mode, which is always the most probable mode (MPM), is encoded first using a separate flag. DC and angular modes are encoded using a list of the remaining five MPMs derived from the intra-modes of neighboring blocks. If the intra-mode is encoded using an angular mode, it means that the block contains coded directional propagation.

[0093] In this embodiment, only CUs encoded using at least one of the DC mode or the planar mode are selected, which correspond to flat blocks, making it more likely that the region is located in a uniform area. Fig.11 shows the planar mode (mode 0, Fig.11 top) and DC mode (mode 1, Fig.11 Example of intra-mode driven block selection (bottom). Blocks selected based on this benchmark are Fig.11 In these examples, 37% of the CUs are in planar mode and 12% are in DC mode.

[0094] Depending on the intra prediction modes provided by the encoder and the video compression standard used to encode the video, other intra prediction modes may be used as long as they correspond to prediction modes for flat regions.

[0095] Residual-driven block selection

[0096] In another embodiment, the code block flag (CBF) is considered to select the CU. The CBF is used to indicate whether the encoding result of the CU contains a non-zero residual. If the CBF is equal to 0 (OFF), the residual coefficients of the CU do not need to be encoded. This means that the coefficients are all 0, and the area can be considered to be well predicted. When combined with the above embodiment (intra-frame mode), such areas with a CBF of 0 can be regarded as having no details or uniform. Fig.12 An example of residual driven block selection based on code block flag values ​​is shown. The blocks selected based on this benchmark are Fig.12 In this example, 10% of the CUs have their CBF set to the OFF value.

[0097] Cost-driven block selection

[0098] Another relevant information that can be used for block selection is the bit per pixel (BPP) cost of each CU. The coding cost of texture areas is higher than that of uniform areas. In this embodiment, CUs with low BPP costs are selected, that is, CUs with bpp costs below a given value are selected. For example, BPP<0.015 can be a given value for detecting uniform areas. Fig.13An example of cost-driven block selection based on a BPP cost below 0.015 is shown. Blocks selected based on this benchmark are Fig.13 Shown in light grey.

[0099] Coefficient distribution driven block selection

[0100] In this embodiment, the coefficient distribution of the transformed coefficients in the block is considered for region selection. The distribution of DCT coefficients or the energy of the first n DCT or transformed coefficients also provides some information about uniform regions. If most of the DCT coefficients are in the low frequencies, it means that there is no energy or not much energy in the high frequencies, i.e. there is no detail in this CU and the CU can be regarded as a uniform region. The energy percentage in the first n coefficients can be given by the following formula:

[0101] Let X be the DCT coefficient of the CU, and set a threshold for the energy, for example, considering 90% of the energy: EnergyTh=0.9.

[0102] The total energy of a block is determined by summing the energies of all coefficients, for example using L2:

[0103] Xsort = sort(abs(X))

[0104] energyTot=normL2(Xsort).

[0105] The sort() function sorts all the DCT coefficients of the CU from the least frequent to the most frequent. The abs() function returns the absolute value of the coefficient. The normL2() function calculates the square root of the sum of the squares of the elements. It provides the energy of the Xsort vector.

[0106] Then, loop over all coefficients from low frequency to high frequency, and for each coefficient c, check whether the sum of the energies of all coefficients from the first coefficient to coefficient c relative to the total energy of the block is below the energy threshold:

[0107] coeffIdx=1

[0108] while normL2(Xsort(1:coeffIdx))) / energyTot <EnergyTh:

[0109] coeffIdx=coeffIdx+1

[0110] This function returns coeffIdx, the index of the DCT coefficient where 90% of the energy can be found in the lower DCT coefficients before this DCT coefficient. If coeffIdx is low, it means that the energy is in the first few / low frequencies, i.e. this CU is a homogeneous region. coeffIdx is considered low when it is below a given percentage of the number of coefficients in the block.

[0111] In a variant, the last_xy_sig flag, which provides similar information, can be used to determine the coefficient distribution. The flag corresponds to the position (x, y) of the last valid DCT coefficient in the CU. If (x, y) is close to the upper left corner position in the CU, the energy of the DCT coefficient is concentrated in the low frequency, that is, the CU is a uniform area. In the same manner as above, when the position of the last valid coefficient in the CU is before the given position, the position of the last valid coefficient is considered to be close to the upper left corner position in the CU. The given position may depend on the size of the CU.

[0112] The above-described embodiments for selecting an area for determining film grain parameters may be used alone or in combination of two or more embodiments.

[0113] Fig. 9 An example of a method for determining a region for determining film grain parameters according to an embodiment is shown.

[0114] When encoding a CU of a frame, the size of the CU is checked using the partitioning of the frame at 900. In response to determining that the size of the CU is less than a given size, for example, if the CU size is less than 16×16, the CU is not selected and the process ends for the CU. Checking the size of the CU may include checking whether the CU is a square CU, and when the CU is square, checking whether the CU is less than a given size.

[0115] In another variant, checking the size may further include, when the CU is not a square, that is, the CU is a rectangle, checking whether the minimum dimension of the CU is less than a given value. For example, for a rectangular CU, checking whether the minimum dimension of the width and height of the CU is less than a given value, such as 16, 32, or 64.

[0116] If it is determined at 900 that the size of the CU is greater than or equal to the given size value, the intra-mode prediction used to encode the CU is checked at 901. In response to determining that the intra-mode prediction is not a DC mode or a planar mode, the CU is not selected and the process ends for the CU. Otherwise, the CU is selected.

[0117] At 902, it is checked whether a residual is encoded for the current CU. For example, the value of CBF is checked. If it is determined that the value of CBF is 0 (OFF, i.e., no residual is encoded for the CU), the CU is selected and the process ends.

[0118] Otherwise (CBF value is 1, ON), depending on the variant, other coding parameters may be checked for the CU.

[0119] At 903, check last_xy_flag. If last_xy_flag is close to the upper left corner of the CU, the CU is selected. Otherwise, the CU is not selected and the process ends.

[0120] In another variation, at 904, the energy of the first N coefficients of the CU is checked, and if it is determined that the energy of the first N coefficients of the CU is greater than or equal to a given amount of the energy of the entire CU (considering all coefficients of the CU), the CU is selected, where N is a given number of coefficients considered at low frequencies. N may depend on the CU size.

[0121] In another variation, at 905 , the bit per pixel cost of the CU is checked, and if the BPP cost of the CU is determined to be below a given value, the CU is selected.

[0122] In another variation, if the CU is not selected from the previous determination, the determinations at 903 , 904 , and 905 may be performed subsequently.

[0123] Fig.14 A block diagram of a system in which aspects of the present embodiment may be implemented is shown according to another embodiment. Fig.14 An embodiment of an apparatus 1400 for determining film grain parameters or for selecting a region for determining film grain parameters according to any of the embodiments described herein is shown.

[0124] The apparatus includes a processor 1410 and may be interconnected to a memory 1420 through at least one port. The processor 1410 and the memory 1420 may also have one or more additional external connections interconnected.

[0125] The processor 1410 is also configured using any of the embodiments described herein to: obtain a reconstructed image by encoding an input image; select at least one region of the reconstructed image based on at least one encoding parameter of the region when encoding the input image; and determine a film grain parameter based on the selected at least one region and the input image. For example, the processor 1410 is configured using a computer program product including code instructions for implementing any of the embodiments described herein.

[0126] exist Fig.15 In the embodiment shown, in a transmission environment between two remote devices A and B via a communication network NET, device A includes a processor associated with a memory RAM and a ROM, which is configured to implement the same as Figure 1 , 2, and the video encoding method described in relation to 7, and the device B comprises a processor associated with a memory RAM and a ROM, which is configured to implement the video encoding method described in relation to Figure 1 , 3 , 7 and the video decoding method described in relation to the above. According to an embodiment, the device A is further configured to determine film grain parameters or select an area for determining film grain parameters, such as Figure 4-13 In some embodiments, device B is further configured to synthesize film grain on the received video using the received film grain metadata.

[0127] According to an example, the network is a broadcast network adapted to broadcast / transmit encoded video and film grain metadata from device A to decoding devices including device B.

[0128] Fig.16 An example of the syntax of a signal transmitted via a packet-based transport protocol is shown. Each transmitted data packet P includes a header H and a payload PAYLOAD. In some embodiments, the payload PAYLOAD may include encoded video data and film grain metadata encoded according to any of the above embodiments.

[0129] Various implementations involve decoding. "Decoding" as used in this application may encompass, for example, all or part of a process performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also includes or alternatively includes a process performed by a decoder of the various embodiments described in this application.

[0130] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding, and in another embodiment, "decoding" refers to the entire image reconstruction process including entropy decoding. Whether the phrase "decoding process" is intended to specifically refer to a subset of operations or to generally refer to a broader decoding process will become clear based on the context of the specific description and is believed to be well understood by those of ordinary skill in the art.

[0131] Various implementations involve encoding. Similar to the above discussion about "decoding", "encoding" as used in this application can cover, for example, all or part of the processes performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as partitioning, differential encoding, transforms, quantization, and entropy encoding. In various embodiments, such processes also include or alternatively include processes performed by an encoder of various embodiments described in this application, such as determining resampling filter coefficients, resampling decoded images.

[0132] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or generally to a broader encoding process will become clear based on the context of the specific description and is believed to be well understood by those of ordinary skill in the art.

[0133] It should be noted that the syntax elements used in this article are descriptive terms. Therefore, they do not exclude the use of other syntax element names.

[0134] This disclosure describes, for example, a variety of information that can be transmitted or stored, such as syntax. The information can be packaged or arranged in a variety of ways, including, for example, ways commonly used in video standards, such as placing the information in an SPS, PPS, NAL unit, header (e.g., a NAL unit header or a slice header), or SEI message. Other ways can also be used, including, for example, ways commonly used by system-level or application-level standards, such as placing the information in one or more of the following:

[0135] a. SDP (Session Description Protocol), a format for describing multimedia communication sessions, used for session announcements and session invitations, e.g. described in RFCs and used in conjunction with RTP (Real-time Transport Protocol) transport.

[0136] b. DASH MPD (Media Presentation Description) descriptor, used, for example, in DASH and transmitted over HTTP, a descriptor is associated with a representation or a set of representations to provide additional characteristics to the content representation.

[0137] c. RTP header extension, such as used during RTP streaming.

[0138] d. ISO base media file formats, such as used in OMAF, and use boxes, which are object-oriented building blocks defined by a unique type identifier and a length, also called "atoms" in some specifications.

[0139] e. HLS (HTTP Live Streaming) manifests transmitted over HTTP. For example, a manifest can be associated with a version or set of versions of the content to provide characteristics of the version or set of versions.

[0140] When a diagram is presented as a flow chart, it should be understood that it also provides a block diagram of the corresponding device. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flow chart of the corresponding method / process.

[0141] Some embodiments relate to rate-distortion optimization. Specifically, in the encoding process, a balance or trade-off between rate and distortion is usually considered, usually taking into account the limitation of computational complexity. Rate-distortion optimization is usually expressed as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different methods to solve the rate-distortion optimization problem. For example, these methods can be based on extensive testing of all coding options, including all considered modes or coding parameter values, and a complete evaluation of their coding costs and the associated distortion of the reconstructed signal after encoding and decoding. Faster methods can also be used to save coding complexity, in particular, to calculate approximate distortion based on prediction or prediction residual signals rather than reconstructed signals. The two methods can also be mixed, for example, only using approximate distortion for some possible coding options and using complete distortion for other coding options. Other schemes only evaluate a subset of possible coding options. More generally, many schemes use any of a variety of techniques to perform optimization, but optimization is not necessarily a complete evaluation of coding costs and associated distortions.

[0142] The embodiments and aspects described herein can be implemented as, for example, methods or processes, devices, software programs, data streams or signals. Even if only discussed in the context of a single embodiment (e.g., discussed only as a method), the implementation of the discussed features can also be implemented in other forms (e.g., devices or programs). The device can be implemented as, for example, suitable hardware, software, and firmware. The method can be implemented as, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device, such as a computer, a mobile phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate information communication between end users.

[0143] References to "an embodiment" or "an embodiment" or "an implementation" or "an implementation" and other variations thereof mean that the particular features, structures, characteristics, etc. described in connection with the embodiment are included in at least one embodiment. Therefore, the phrases "in an embodiment" or "in an embodiment" or "in an implementation" or "in an implementation" and any other variations appearing in multiple places in the present application do not necessarily all refer to the same embodiment.

[0144] Furthermore, the present application may involve “determining” a variety of information. Determining information may include one or more of the following, such as estimating information, calculating information, predicting information, or retrieving information from memory.

[0145] Furthermore, the present application may involve "accessing" various information. Accessing information may include one or more of the following, such as receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0146] Furthermore, the present application may involve "receiving" a variety of information. Like "accessing," receiving is intended to be a broad term. Receiving information may include one or more of the following, such as accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" generally involves in some way during an operation such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0147] It should be understood that the use of any of the following " / ", "and / or", and "at least one", such as in the case of "A / B", "A and / or B", and "at least one of A and B", is intended to cover selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C", such wording is intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). This can be extended to as many items as listed as will be apparent to one of ordinary skill in this and related arts.

[0148] In addition, as used herein, the term "signal" refers, among other things, to indicating something to a corresponding decoder. For example, in some embodiments, an encoder signals a specific one of a plurality of transforms, coding modes, or flags. Thus, in one embodiment, the same transform, parameter, or mode is used on the encoder side and the decoder side. Thus, for example, an encoder may transmit (explicitly signal) a specific parameter to a decoder so that the decoder may use the same specific parameter. Conversely, if the decoder already has the specific parameter as well as other parameters, it may be signaled (implicitly signaled) without transmission to simply allow the decoder to know and select the specific parameter. Bit savings are achieved in various embodiments by avoiding transmission of any actual functionality. It should be understood that signaling may be accomplished in a variety of ways. For example, in various embodiments, one or more grammatical elements, flags, etc. are used to signal information to a corresponding decoder. Although the verb form of the word "signal" is mentioned above, the word "signal" may also be used as a noun here.

[0149] As will be appreciated by those of ordinary skill in the art, embodiments may generate a variety of formatted signals to carry information that may be, for example, stored or transmitted. The information may include, for example, instructions for executing a method or data generated by one of the embodiments. For example, a signal may be formatted to carry a bitstream of the embodiment. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, a signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

[0150] A number of embodiments have been described above. The features of these embodiments may be provided individually or in any combination, across a variety of claim categories and types.

Claims

1. A method comprising: Obtaining a reconstructed image by encoding the input image; selecting at least one region of the reconstructed image based on at least one encoding parameter of the region when encoding the input image; as well as Film grain parameters are determined based on the selected at least one region and the input image.

2. An apparatus comprising one or more processors, wherein the one or more processors are operable to: Obtaining a reconstructed image by encoding the input image; selecting at least one region of the reconstructed image based on at least one encoding parameter of the region when encoding the input image; and Film grain parameters are determined based on the selected at least one region and the input image.

3. The method according to claim 1 or the apparatus according to claim 2, wherein the region is a coding unit, and the at least one coding parameter is at least one or a combination of two or more of the following: a size of the coding unit for encoding the coding unit; an intra prediction mode for encoding the coding unit; a code block flag indicating whether encoding of the coding unit provides a non-zero residual; a per-pixel bit cost for encoding the coding unit; The position of the last significant coefficient in the coding unit; An index of a first coefficient for which a given percentage of the energy of coefficients of the coding unit is obtained by coefficients preceding the first coefficient. 4 . The method or apparatus of claim 3 , wherein the at least one region is selected in response to determining that a size of the at least one region is greater than or equal to a given coding unit size.

5. The method or apparatus of any one of claims 3 and 4, wherein the at least one region is selected in response to determining that the intra prediction mode is a DC mode or a planar mode.

6. The method or apparatus of any one of claims 3 to 5, wherein the at least one region is selected in response to determining that the code block flag indicates that encoding of the coding unit provides a zero residual.

7. A method or apparatus according to any one of claims 3 to 6, wherein the at least one region is selected in response to determining that the bit per pixel cost is greater than or equal to a given value.

8. A method or apparatus according to any one of claims 3 to 7, wherein the at least one region is selected in response to determining that a distance between an upper left position in the coding unit and a position of the last significant coefficient in the coding unit is less than or equal to a given value.

9. A method or apparatus according to any one of claims 3 to 8, wherein the at least one region is selected in response to determining that the distance between the upper left position in the coding unit and the position of the first coefficient in the coding unit is less than or equal to a given value, wherein for the first coefficient, a given percentage of the energy is provided by a coefficient located between the upper left position and the first coefficient.

10. A method according to any one of claims 1 or 3 to 9, or an apparatus according to any one of claims 2 to 9, wherein the input image is encoded as an intra picture.

11. The method according to any one of claims 1 or 3 to 10, or the apparatus according to any one of claims 2 to 10, wherein the at least one encoding parameter of the region is obtained by encoding a luminance component of the input image.

12. A method according to any one of claims 1 or 3 to 11, or an apparatus according to any one of claims 2 to 11, wherein obtaining the reconstructed image comprises removing film grain from the input image.

13. The method or apparatus of claim 12, wherein in response to determining that a value of a quantization parameter used to encode the input image is greater than or equal to a given value, film grain is removed from the input image by encoding the input image with the quantization parameter.

14. The method or apparatus of claim 12, wherein in response to determining that the value of the quantization parameter used to encode the input image is less than a given value, film grain is removed from the input image by denoising the input image using an external denoiser or an image filtering tool of a video encoder.

15. A computer-readable storage medium having stored thereon instructions for causing one or more processors to execute the method according to any one of claims 1 or 3 to 14.

16. A computer program product comprising instructions, wherein when the program is executed by one or more processors, the instructions cause the one or more processors to perform the method according to any one of claims 1 or 3 to 14.

Images