Film grain processing method and communication device

Abstract

This disclosure provides a film grain processing method, apparatus, and storage medium, which, in the field of image processing technology, can more accurately and flexibly simulate the visual effect of film grain. The method includes the steps of: obtaining a decoded image from a bitstream; obtaining a plurality of film grain parameters and first instruction information, wherein the first instruction information is used to instruct a scheme for adding film grain to the decoded image using the plurality of film grain parameters in combination; and adding film grain to the decoded image based on the plurality of film grain parameters and the first instruction information.

Description

Technical Field

[0001] (Cross-Reference to Related Application) The present disclosure claims priority to Chinese Patent Application No. 202311128154.4 filed on August 30, 2023, the entire content of which is incorporated herein by reference.

[0002] The present disclosure relates to the field of image processing technology, in particular to a film grain processing method and communication equipment .

Background Art

[0003] Film grain technology is currently one of the technologies discussed and studied by international video coding and decoding standardization organizations. The film grain modeling and synthesis methods defined in different standards are not completely identical, and the levels of support for film grain are also different 。

Summary of the Invention

Means for Solving the Problem

[0004] Embodiments of the present disclosure provide a film grain processing method, apparatus and storage medium, which can simulate the visual effect of film grain more accurately and flexibly. In one aspect, there is provided a film grain processing method applied to a decoding side, obtaining a decoded image from a bitstream; obtaining a plurality of sets of film grain parameters and first indication information, wherein the first indication information is used to indicate a scheme for adding film grain to the decoded image by using the plurality of sets of film grain parameters in combination; adding film grain to the decoded image based on the plurality of sets of film grain parameters and the first indication information.

[0005] In another aspect, there is provided a film grain processing method applied to an encoding side, Steps to obtain the source image, The steps include: performing noise reduction on the source image to obtain a noise-reduced image, The process includes determining a set of film grain parameters and a first instruction information based on a source image and a denoising image, wherein the first instruction information is used to instruct a scheme for adding film grain to a decoded image using the set of film grain parameters in combination.

[0006] Also In another embodiment, a film grain processing apparatus is provided that is applied to the decoding side, the film grain processing apparatus including an acquisition module and an additional module. The acquisition module is used to obtain the decoded image from the bitstream. The acquisition module is further used to acquire multiple sets of film grain parameters and first instruction information. The first instruction information is used to instruct a scheme to add film grain to the decoded image using multiple sets of film grain parameters in combination. The additional module is used to add film grain to the decoded image based on multiple sets of film grain parameters and first instruction information.

[0007] In another embodiment, a film grain processing apparatus is provided that is applied to the encoding side, the film grain processing apparatus including an acquisition module, a processing module and a determination module. The acquisition module is used to acquire the source image. The processing module is used to perform noise reduction on the source image and obtain a noise-reduced image. The decision module is used to determine multiple sets of film grain parameters and first instruction information based on the source image and the denoising image. The first instruction information is used to instruct a scheme to add film grain to the decoded image using multiple sets of film grain parameters in combination.

[0008] In another embodiment, a communication device is provided, the communication device including memory and a processor, the memory and the processor being coupled, the memory being used to store computer programs, and the processor executing computer programs, any of the aforementioned Described in the manner This enables a film grain processing method.

[0009] In another embodiment, a computer-readable storage medium is provided, the computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by the processor, the aforementioned arbitrary Described in the manner This enables a film grain processing method.

[0010] In another embodiment, a computer program product is provided, which includes computer program instructions, and when these instructions are executed by a processor, the film grain processing method described in any of the above embodiments is realized. The embodiments of this disclosure provide a film grain processing method that allows for the addition of film grain to a decoded image based on a scheme that uses multiple sets of film grain parameters and multiple sets of film grain parameters in combination to add film grain to the decoded image. In this way, compared to the fact that different standards or private protocols of each manufacturer in the related technology define only methods for performing film grain simulation for a single film grain model, the method of simulating complex film grain characteristics in a video image using multiple sets of film grain parameters (corresponding to multiple film grain models) provided by the embodiments of this disclosure can achieve the objective of simulating the visual effect of film grain more accurately and flexibly, thereby providing viewers with a better viewing experience. At the same time, a method of synthesizing film grain based on multiple film grain models increases the robustness of the film grain synthesis process, thereby avoiding problems such as synthesis algorithms becoming dysfunctional due to the high computational cost of a single model.

[0011] To more clearly explain the technical concepts in the embodiments of this disclosure, the drawings necessary for describing some embodiments of this application are briefly described below. Needless to say, the drawings in the following description are only drawings of some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these. [Brief explanation of the drawing]

[0012] [Figure 1] This figure shows the architecture of an encoding / decoding system according to some embodiments of the present disclosure. [Figure 2] This figure shows the architecture of another encoding / decoding system according to some embodiments of the present disclosure. [Figure 3] This figure shows the architecture of yet another encoding / decoding system according to some embodiments of the present disclosure. [Figure 4] This figure shows the architecture of yet another encoding / decoding system according to some embodiments of the present disclosure. [Figure 5] This is a flowchart of a film grain processing method according to some embodiments of the present disclosure. [Figure 6] This is a flowchart of another film grain processing method according to some embodiments of the present disclosure. [Figure 7] This is a flowchart of yet another film grain processing method according to some embodiments of the present disclosure. [Figure 8] This is a flowchart of yet another film grain processing method relating to some embodiments of the present disclosure. [Figure 9] This is a flowchart of yet another film grain processing method relating to some embodiments of the present disclosure. [Figure 10]It is a diagram showing the architecture of still another encoding / decoding system according to some embodiments of the present disclosure. [Figure 11] It is a diagram showing the configuration of a film grain processing apparatus according to some embodiments of the present disclosure. [Figure 12] It is a diagram showing the configuration of another film grain processing apparatus according to some embodiments of the present disclosure. [Figure 13] It is a diagram showing the configuration of a film grain processing apparatus according to some embodiments of the present disclosure. Mode for Carrying Out the Invention

[0013] Hereinafter, the technical solutions of the present disclosure will be described clearly and completely in conjunction with the drawings of the present disclosure. Apparently, the described embodiments are merely some rather than all of the embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of the present disclosure.

[0014] It should be noted that, in the present disclosure, phrases such as "exemplary" or "for example" are used to serve as an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in the present disclosure should not be construed as superior or more advantageous than other embodiments or designs. To be precise, the use of phrases such as "exemplary" or "for example" is intended to present the related concepts in a concrete manner.

[0015] Hereinafter, the terms "first" and "second" are only used for description, and it should not be understood that they explicitly or implicitly indicate relative importance or implicitly indicate the number of the targeted technical feature. Therefore, the feature defined by "first" and "second" may explicitly or implicitly include one or more of said features.

[0016] In this disclosure, unless otherwise specified, " / " means "or". For example, A / B can mean A or B. In this specification, "and / or" merely describes the relationship between the related subjects and indicates that three relationships may exist. For example, A and / or B can mean three situations: A exists alone, A and B exist simultaneously, or B exists alone. Also, "at least one" means one or more, and "multiple" means two or more.

[0017] Film grain is a physical process that occurs during the exposure and development of photographic film (exposure and development of silver halide crystals) and is widely present in film and television content. However, digital sensors do not go through the aforementioned process, so they produce noise-free digital video, and film grain is absent. Many content creators, especially in the film industry, widely consider film grain to be a pleasant noise that enhances the natural appearance of video content. Therefore, adding film noise back into video to improve the visual experience is usually considered part of the creative intent.

[0018] Video image compression relies heavily on predictions in the temporal, spatial, or cross-component domains, but because film grain is inherently random, encoding it with typical video image encoding tools is difficult. One solution to preserve film grain is to use small quantization parameters during video compression to better preserve detail. However, for video applications such as adaptive streaming media and broadcast, this approach requires a relatively high bitrate. Another solution is to model the film grain on the encoding side to obtain film grain parameters, and then reuse those parameters on the decoding side to generate film grain and add it to the decoded video image. This allows the full potential of video image compression techniques to be realized.

[0019] For example, the High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC) series standards, developed by the Joint Video Experts Team (JVET) under the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC) and the International Telecommunication Union (ITU), utilize Film Grain Characteristic (FGC) and Supplemental Enhancement Information (SEI) to specify the film grain parameters used when synthesizing film grain during video or image data decoding and rendering, thereby achieving the goal of preserving film grain.

[0020] Referring to Figure 1, this diagram shows the architecture of the encoding and decoding system used in the video encoding and decoding standard developed by JVET. As shown in Figure 1, this system includes both an encoding side and a decoding side.

[0021] Here, the encoding side includes an encoder, a noise reduction module, and a film grain modeling module.

[0022] An encoder is used to compress and encode an input source video to obtain an encoded video bitstream.

[0023] The noise reduction module is used to perform preprocessing and noise reduction on the input source video to obtain a noise-reduced video. Here, the noise-reduced video includes a video image sequence after film grain has been removed or attenuated.

[0024] The film grain modeling module is used to parameterize the film grain model based on the difference between the denoised video and the source video, and to obtain the film grain parameters (i.e., the parameter values ​​of the film grain model).

[0025] Here, the film grain parameters may be transmitted to the decoding side along with the encoded video bitstream, or they may be transmitted to the decoding side by an external transmission method.

[0026] The decoding side includes a decoder and a film grain model.

[0027] A decoder is used to decode an input encoded video bitstream and obtain the decoded video stream.

[0028] The film grain model is used to synthesize simulated film grain according to the input film grain parameters.

[0029] Furthermore, simulated film gray n By adding it to the decoded video stream, a decoded and reconstructed video with simulated film grain can be obtained.

[0030] In some embodiments, there are implementation methods for encoding and decoding system architectures other than the one shown in Figure 1. The main difference lies on the encoding side. For example, the encoding side does not encode the source video image sequence to obtain an encoded video bitstream, but rather encodes the denoised video image sequence to obtain an encoded video. Alternatively, if the source video does not contain film grain, the encoding side transmits predetermined film grain parameters to the decoding side to perform film grain synthesis. In this case, even if film grain is not present in the source video, the decoding side can add film grain to the video to achieve a certain effect.

[0031] At the 29th JVET Conference, proposal JVET-AC0199 presented the results of film grain quality evaluation obtained using a frequency-based method and an auto-regressive-based method, respectively, and stated the following points: (1) If the source video has heavy film grain, the synthesized film grain will be coarse in size and have a clear repeating pattern. (2) If the source video has light / low film grain, the effect of the synthesized film grain is good. (3) Frequency-based film grain synthesis (FGS) methods and autoregressive FGS methods show different results under different circumstances.

[0032] As can be seen from this, single-model-based FGS methods lack flexibility and accuracy, making it difficult to obtain relatively good simulation results for complex film grains.

[0033] In response to the technical challenges described above, embodiments of this disclosure provide a film grain processing method, the idea of ​​which is to add film grain to a decoded image based on multiple sets of film grain parameters and the combined use of multiple sets of film grain parameters. Thus, compared to the fact that only methods for performing film grain simulation for a single film grain model are defined in different standards or private protocols of each manufacturer in the related technology, the method of simulating complex film grain characteristics in a video image using multiple sets of film grain parameters (corresponding to multiple film grain models) provided by embodiments of this disclosure can achieve the objective of simulating the visual effect of film grain more accurately and flexibly, thereby providing viewers with a better viewing experience.

[0034] At the same time, a method of synthesizing film grain based on multiple film grain models increases the robustness of the film grain synthesis process, thereby avoiding problems such as synthesis algorithms becoming dysfunctional due to the high computational cost of a single model.

[0035] The encoding and decoding systems provided by embodiments of this disclosure will be described in detail below with reference to the drawings.

[0036] Referring to Figure 2, this figure shows the architecture of an encoding / decoding system according to some embodiments of the present disclosure. As shown in Figure 2, this encoding / decoding system 100 includes an encoding side 110 and a decoding side 120.

[0037] Here, the encoding side 110 generates an encoded video bitstream. Therefore, the encoding side 110 is also called a video encoder. The decoding side 120 may decode the encoded video bitstream generated by the encoding side. Therefore, the decoding side 120 is also called a video decoder. Various embodiments of the encoding side 110, the decoding side 120, or both may include one or more processors and memory coupled to the one or more processors. The memory includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other media that can be used to store desired program code in the form of computer-accessible instructions or data structures, as described herein.

[0038] In some embodiments, the encoding side 110 and the decoding side 120 may include a variety of devices, including desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, telephone handsets such as so-called "smartphones," televisions, cameras, display devices, digital media players, video game consoles, in-vehicle computers, or similar devices.

[0039] In some embodiments, the decoding side 120 may receive an encoded video bitstream from the encoding side 110 via a link. The link may include one or more media or devices that can move the encoded video data from the encoding side to the decoding side 120. In one example, the link may include one or more communication media that enable the encoding side 110 to directly transmit the encoded video bitstream to the decoding side in real time. In this example, the encoding side 110 may modulate the encoded video bitstream according to a communication standard (e.g., a wireless communication protocol) and transmit the modulated video stream to the decoding side 120. The one or more communication media may include wireless and / or wired communication media, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The one or more communication media may form part of a packet-based network, such as a local area network, a wide area network, or a global network (e.g., the Internet). The one or more communication media may include routers, switches, base stations, or other devices that facilitate communication from the encoding side 110 to the decoding side 120.

[0040] In some embodiments, as shown in Figure 3, the encoding side 110 includes an encoder, a noise reduction module, and a film grain modeling module.

[0041] An encoder is used to compress and encode an input source video to obtain an encoded video bitstream.

[0042] The noise reduction module is used to perform preprocessing and noise reduction on the input source video to obtain a noise-reduced video. Here, the noise-reduced video includes a video image sequence after film grain has been removed or attenuated.

[0043] The film grain modeling module is used to parameterize the film grain model and obtain film grain parameters (i.e., parameter values ​​for the film grain model). Multiple film grain modeling modules can be used to obtain multiple sets of film grain parameters.

[0044] Here, the film grain parameters may be transmitted to the decoding side 120 along with the encoded video bitstream, or they may be transmitted to the decoding side 120 by an external transmission method.

[0045] In some embodiments, as shown in Figure 4, the decoding side 120 includes a decoder and multiple film grain modeling modules.

[0046] A decoder is used to decode an input encoded video bitstream and obtain the decoded video stream.

[0047] A film grain model is used to synthesize simulated film grain according to the input film grain parameters. Multiple film grain models each synthesize multiple simulated film grains.

[0048] In some embodiments, a decoded and reconstructed video with simulated film grain may be obtained by sequentially adding multiple simulated film grain parameters to the decoded video stream.

[0049] In some embodiments, multiple simulated film gray n By fusing according to predetermined weights, fused film grain may be obtained and added to the decoded video stream to obtain a decoded and reconstructed video with simulated film grain.

[0050] To ensure that it is understood, the application scenarios of the embodiments of this disclosure are not limited. The system architectures and service scenarios described in the embodiments of this disclosure are for the purpose of more clearly illustrating the technical solutions of the embodiments of this disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of this disclosure. Those skilled in the art will know that, as network architectures evolve and new service scenarios emerge, the technical solutions provided by the embodiments of this disclosure are equally applicable to similar technical problems.

[0051] The film grain treatment method provided by the embodiments of this disclosure will be described in detail below.

[0052] Based on the decoding side 120 shown in Figure 4, the Disclosure further provides a film grain processing method. Referring to Figure 5, Figure 5 is a flowchart of the film grain processing method according to the Disclosure, which can be performed by the decoding side 120. This method includes the following steps.

[0053] S201 retrieves the decoded image from the bitstream.

[0054] Here, bitstream (bitstream) data rate Data Rate (BR) refers to the amount of data a video file uses per unit of time, and is also called bitrate. For example, a bitstream may be an image bitstream, or a transmission stream or media file containing an image bitstream.

[0055] In some embodiments, step S201 above may be implemented to obtain a decoded image by decoding the bitstream. Exemplaryly, the decoding method may be one specified in a video decoding standard, but the embodiments of this disclosure are not limited thereto.

[0056] For example, the decoded image may include at least one of the following: an image sequence consisting of multiple consecutive frames, a single image, an image of one frame within an image sequence, or a partial image region within an image of one frame.

[0057] To ensure understanding, considering the randomness and diversity present in film grain effects in video images, the film grain processing methods provided by the embodiments of this disclosure can be adaptively applied to image sequences, single images, or partial image regions within a single frame. For example, film grain in a complete image sequence may be synthesized using one or more film grain models. The number of film grain models employed may depend on the complexity of the film grain, the simulation effect of the models, the computational complexity of the models, and so on. In short, using multiple film grain models to synthesize film grain is a reliable and effective solution for more accurately and flexibly simulating the visual effects of actual film grain.

[0058] In some embodiments, the bitstream includes a first SEI message, which includes first instruction information, second instruction information, third instruction information, fourth instruction information, and at least one of multiple sets of film grain parameters. Here, the second instruction information is used to indicate whether to support the addition of film grain based on multiple film grain models. The third instruction information is used to indicate whether to cancel the persistence of a previous first SEI message. The fourth instruction information is used to indicate whether the current first SEI message is persistent. As can be seen, multiple sets of film grain parameters may be stored within the same SEI message.

[0059] In some other embodiments, the bitstream includes multiple second SEI messages, where each second SEI message includes a set of film grain parameters. As can be seen, multiple sets of film grain parameters may be stored in different SEI messages.

[0060] Here, the bitstream further includes a first SEI message, which includes at least one of the following: first instruction information, second instruction information, third instruction information, fourth instruction information, multiple fifth instruction information, and multiple sixth instruction information. Here, the second instruction information is used to indicate whether to support the addition of film grain based on multiple film grain models. The third instruction information is used to indicate whether to cancel the persistence of the previous first SEI message. The fourth instruction information is used to indicate whether the current first SEI message has persistence. Each fifth instruction information is used to indicate whether the second SEI message corresponding to the fifth instruction information is included in the first SEI message. Each sixth instruction information is used to indicate the processing order of the second SEI messages corresponding to the sixth instruction information.

[0061] S202 acquires multiple sets of film grain parameters and first instruction information. The first instruction information is used to instruct a scheme to add film grain to the decoded image using multiple sets of film grain parameters in combination.

[0062] In some embodiments, the multiple sets of film grain parameters and the first instruction information may be included in the bitstream. Alternatively, the multiple sets of film grain parameters may be pre-configured parameters, and the first instruction information may be pre-configured instruction information.

[0063] In some embodiments, each set of film grain parameters among multiple sets of film grain parameters includes at least one of a film grain model parameter and a film grain processing parameter.

[0064] Here, the film grain model parameters may include a film grain model identifier. The film grain processing parameters are sets of film grain processing parameters corresponding to the film grain synthesis method indicated by the film grain model parameters. Exemplaryly, the film grain synthesis method may be a method defined in any public standard or private protocol, but the embodiments of this disclosure are not limited thereto.

[0065] In some embodiments, Film grain processing parameters may also be called film grain synthesis parameters or film grain modeling parameters.

[0066] For example, if the film grain model identifier indicates the adoption of a film grain synthesis method based on frequency filtering, the above film grain processing parameters may include the luminance bit depth parameter fg_bit_depth_luma_minus8, the chroma bit depth parameter fg_bit_depth_chroma_minus8, the image color gamut fg_colour_primaries corresponding to the film grain, the color space fg_matric_coeffs, the transformation function fg_transfer_characteristics, the blending mode parameter fg_blending_mode_id for the film grain and the decoded image, the film grain scaling parameter fg_log2_scale_factor, the number of models for intensity intervals of the film grain for different color components fg_num_model_values_minus1[c], and the model values ​​fg_comp_model_value[c][i][j] corresponding to each intensity interval of the film grain for different color components.

[0067] In some embodiments, multiple sets of film grain parameters further include the number of film grain models. Exemplarily, the number of film grain models included in N sets of film grain parameters is N, where N is an integer greater than or equal to 1.

[0068] In some embodiments, the first instruction information includes at least one of a joint scheme of multiple film grain models, the importance of each of the multiple film grain models, and a mode for adding the film grain to the decoded image.

[0069] In some embodiments, the joint scheme of multiple film grain models includes a first scheme and / or a second scheme.

[0070] Here, the first scheme includes generating film grains corresponding to the film grain parameters of each set based on those set of film grain parameters, and then adding the film grains corresponding to the film grain parameters of each set to the decoded image.

[0071] For illustrative purposes, suppose that multiple sets of film grain parameters include a first set of film grain parameters and a second set of film grain parameters. In this case, film grain 1 is generated based on the first set of film grain parameters, film grain 2 is generated based on the second set of film grain parameters, and then film grain 1 and film grain 2 are added to the decoded image.

[0072] The second scheme includes generating film grains corresponding to each set of film grain parameters based on those set of film grain parameters, fusing the film grains corresponding to each of the multiple sets of film grain parameters to obtain a fused film grain, and adding the fused film grain parameters to the decoded image.

[0073] Here, the method for fusing the film grains corresponding to each of the multiple sets of film grain parameters may employ an additive mode, a multiplicative mode, or any other arbitrary calculation mode.

[0074] For illustrative purposes, suppose that multiple sets of film grain parameters include a first set of film grain parameters and a second set of film grain parameters. In this case, film grain 1 is generated based on the first set of film grain parameters, film grain 2 is generated based on the second set of film grain parameters, then film grain 1 and film grain 2 are fused to obtain a fused film grain, and then the fused film grain is added to the decoded image.

[0075] In some embodiments, the importance of each of the multiple film grain models refers to the weighting coefficient in the joint scheme of film grains generated by each film grain model.

[0076] In some embodiments, the mode used to add film grain to the decoded image may be an additive mode, a multiplicative mode, or any other arbitrary calculation mode.

[0077] In some embodiments, as shown in Figure 6, this method may further include the following steps S300 to S301 before step S202.

[0078] S300, acquires second instruction information. This second instruction information is used to indicate whether to support the addition of film grain based on multiple film grain models.

[0079] One possible implementation is to include the second instruction information in the Video Parameter Set (VPS), which is used to indicate whether the video corresponding to the VPS supports the addition of film grain based on multiple film grain models.

[0080] Another possible implementation is to include the second instruction information in the Sequence Parameter Set (SPS), which is used to indicate whether the image sequence corresponding to the SPS supports the addition of film grain based on multiple film grain models.

[0081] Also Another possible implementation involves including the second instruction information in the Picture Parameter Set (PPS), which is used to indicate whether the decoded image of a single frame corresponding to the PPS supports the addition of film grain based on multiple film grain models.

[0082] moreover Another possible implementation involves including the second instruction information in the Adaptive Parameter Set (APS), which is used to indicate whether the decoded image of a single frame corresponding to the APS supports the addition of film grain based on multiple film grain models.

[0083] moreover Another possible implementation is to include the second instruction information in a Supplemental Enhancement Information (SEI) message, which is used to indicate whether the decoded image of a single frame corresponding to the SEI message supports the addition of film grain based on multiple film grain models.

[0084] S301, based on the second instruction information, it is determined whether to acquire multiple sets of film grain parameters and the first instruction information.

[0085] In some embodiments, if the second instruction indicates that the addition of film grain based on multiple film grain models is supported, multiple sets of film grain parameters and the first instruction are obtained. If the second instruction indicates that the addition of film grain based on multiple film grain models is not supported, multiple sets of film grain parameters and the first instruction are not obtained, and the film grain may be added using a single film grain model.

[0086] Note that the execution order shown in Figure 6 is merely illustrative and does not limit the film grain processing method provided by the embodiments of this disclosure. The embodiments of this disclosure do not limit the execution order between step S201 and steps S300 to S301. For example, step S201 may be performed first, followed by steps S300 to S301. Alternatively, steps S300 to S301 may be performed first, followed by step S201. Alternatively, steps S201 and S300 to S301 may be performed simultaneously.

[0087] S203 adds film grain to the decoded image based on multiple sets of film grain parameters and first instruction information.

[0088] In some embodiments, step S203 may be implemented as the following steps.

[0089] Step a1: Multiple film grains corresponding to the decoded image are generated based on multiple sets of film grain parameters.

[0090] In some embodiments, film grain corresponding to the decoded image is generated based on the film grain model parameters and film grain processing parameters included in each set of film grain parameters.

[0091] Note that one set of film grain parameters corresponds to one film grain model, and multiple film grain models may be independent of each other or may have dependencies.

[0092] For example, the processing of multiple film grain models may be done in parallel. For instance, multiple film grain models may synthesize their corresponding film grains simultaneously, but there are no restrictions on the order.

[0093] For example, the processing order of multiple film grain models may be parallel. For instance, there may be a predetermined processing order among the multiple film grain models, and the desired film grain synthesis effect must be obtained by applying the specified multiple film grain models sequentially.

[0094] Step a2: Based on the first instruction information, multiple film grains are added to the decoded image.

[0095] In some embodiments, film grain is added to the decoded image according to a joint scheme of multiple film grain models, the importance of each of the multiple film grain models, and the mode in which the film grain is added to the decoded image.

[0096] For example, if two film grain models are applied to the same image and the processing processes for the two film grain models are in parallel, then film grain 1 is generated based on film grain model 1, and film grain 2 is generated based on film grain model 2. Subsequently, the weight coefficients for film grain 1 are determined according to the importance of film grain model 1, and the weight coefficients for film grain 2 are determined according to the importance of film grain model 2. Finally, fusion is performed using a weighted addition mode according to film grain 1, the weight coefficients for film grain 1, film grain 2, and the weight coefficients for film grain 2 to obtain fused film grain, which is then added to the decoded image.

[0097] Furthermore, the method for adding film grain synthesized by multiple film grain models to the decoded image may employ an additive mode, a multiplicative mode, or any other arbitrary calculation mode. Alternatively, the method for adding film grain synthesized by multiple film grain models to the decoded image may employ a mixed mode, which is a combination of multiple calculation modes.

[0098] The embodiments of this disclosure provide a film grain processing method, and it is understood that film grain can be added to a decoded image based on a scheme that adds film grain to the decoded image by using multiple sets of film grain parameters and multiple sets of film grain parameters in combination. Thus, compared to the fact that different standards or private protocols of each manufacturer in the related technology define only a method for performing film grain simulation for a single film grain model, the method of simulating complex film grain characteristics in a video image using multiple sets of film grain parameters (corresponding to multiple film grain models) provided by the embodiments of this disclosure can achieve the objective of simulating the visual effect of film grain more accurately and flexibly, thereby providing viewers with a better viewing experience.

[0099] At the same time, a method of synthesizing film grain based on multiple film grain models increases the robustness of the film grain synthesis process, thereby avoiding problems such as synthesis algorithms becoming dysfunctional due to the high computational cost of a single model.

[0100] Based on the encoding side 110 shown in Figure 3, the Disclosure further provides a film grain processing method. Referring to Figure 7, which is a flowchart of the film grain processing method according to the Disclosure, this film grain processing method can be performed by the encoding side 110. This method includes the following steps.

[0101] S401, retrieve the source image.

[0102] For example, the source image may be an image sequence consisting of multiple consecutive frames, a single image, an image of one frame within an image sequence, or a partial image region within an image of one frame.

[0103] S402. Noise reduction processing is performed on the source image to obtain a noise-reduced image.

[0104] Here, the denoised video includes a sequence of video images after film grain has been removed or attenuated.

[0105] For example, the noise reduction process described above may be a spatial domain noise reduction method, a transformation domain noise reduction method, or a coordinated filtering method for the spatial domain and the transformation domain, but the embodiments of this disclosure are not limited thereto.

[0106] S403, based on the source image and the denoising image, multiple sets of film grain parameters and first instruction information are determined. The first instruction information is used to instruct a scheme to add film grain to the decoded image using multiple sets of film grain parameters in combination.

[0107] In some embodiments, each set of multiple sets of film grain parameters includes at least one of the following: a film grain model parameter and a film grain processing parameter.

[0108] In some embodiments, multiple sets of film grain parameters further include the number of film grain models.

[0109] In some embodiments, the first instruction information described above includes at least one of a joint scheme of multiple film grain models, the importance of each of the multiple film grain models, and a mode for adding film grain to the decoded image.

[0110] In some embodiments, the multiple sets of film grain parameters include at least one set of film grain parameters. As shown in Figure 8, the first set of film grain parameters may be determined by the following steps.

[0111] S501 determines the film grain of the source image based on the difference between the source image and the denoised image.

[0112] For example, the difference between the source image and the denoised image can be reflected by the difference between corresponding pixel points in the source image and the denoised image.

[0113] S502 constructs the first film grain model based on the film grain of the source image.

[0114] In some embodiments, step S502 may be implemented to construct a first film grain model based on the film grain of a source image and a predetermined film grain modeling method.

[0115] For example, the specified film grain modeling method described above may be a frequency-based film grain modeling method or a film grain modeling method based on autoregression coefficients.

[0116] In some embodiments, a first film grain model is constructed based on the film grain characteristics of the source image. Exemplary film grain characteristics include, but are not limited to, film grain intensity and film grain scaling parameters.

[0117] Here, the film grain characteristics may be those of the entire source image, or they may be those of different regions within the source image.

[0118] S503, based on the first film grain model, the first set of film grain parameters is determined.

[0119] As an example, we perform film grain parameterization on the first film grain model to obtain the first set of film grain parameters.

[0120] In some embodiments, the multiple sets of film grain parameters include at least a second set of film grain parameters. As shown in Figure 9, the second set of film grain parameters may be determined by the following steps.

[0121] In S504, if the difference between the film grain of the source image and the film grain output by the first film grain model is greater than a threshold, the residual film grain in the source image is determined.

[0122] To make it clear, if the difference between the film grain in the source image and the film grain output by the first film grain model is greater than a threshold, it indicates that the first film grain model cannot accurately simulate the film grain in the source image. Therefore, to achieve the goal of simulating the visual effect of film grain more accurately and flexibly, it is necessary to construct multiple film grain models.

[0123] In some embodiments, the residual film grain in the source image is determined according to the difference between the film grain in the source image and the film grain output by the first film grain model.

[0124] S505 constructs a second film grain model based on the film grain in the source image.

[0125] In some embodiments, a second film grain model is constructed based on the film grain characteristics of the residual film grain.

[0126] S506, based on the second film grain model, the second set of film grain parameters is determined.

[0127] As an example, we perform film grain parameterization on the second film grain model to obtain a second set of film grain parameters.

[0128] In some embodiments, after step S506, the method further includes the step of determining whether the first and second film grain models can accurately simulate the film grain of the source image. If the first and second film grain models cannot accurately simulate the film grain of the source image, the method further includes the step of constructing a third film grain model and then repeatedly making determinations until the constructed film grain model can accurately simulate the film grain of the source image.

[0129] To ensure clarity, the embodiments of this disclosure do not limit the number of film grain models. If a constructed film grain model cannot accurately simulate the film grain of the source image, one or more additional film grain models may be reconstructed.

[0130] In some embodiments, the method further includes the step of transmitting a second instruction information to the decoding end. The second instruction information is used to indicate whether to support the addition of film grain based on multiple film grain models.

[0131] As one possible implementation, after step S403 above, this method further includes the step of transmitting a bitstream to the decoding side. The bitstream includes encoding information of the source image, multiple sets of film grain parameters, and first instruction information.

[0132] To make it easier to understand, the encoding information of the source image, multiple sets of film grain parameters, and first instruction information can be written to the bitstream and transmitted to the decoding side along with the bitstream, thereby saving channel resources.

[0133] As another possible implementation, after step S403 above, this method further includes the step of transmitting a bitstream, multiple sets of film grain parameters, and first instruction information to the decoding side. The bitstream includes encoded information of the source image.

[0134] To make it understandable, the bitstream, source image encoding information, multiple sets of film grain parameters, and first instruction information can be transmitted separately, thereby enabling independent scheduling.

[0135] There are several methods for encoding an image and generating a bitstream; for example, methods specified in the H.26x series standard may be used, but the embodiments of this disclosure are not limited to these.

[0136] To facilitate understanding, the following will explain, in the form of an example, the film grain processing method applied to the encoding side.

[0137] As an example, as shown in Figure 10, the film grain processing method applied to the encoding side may be implemented as follows:

[0138] Step b1: Obtain the film grain 1 of the source image according to the difference between the source image and the denoised image.

[0139] Step b2: Perform the first film grain modeling according to the film grain characteristics of film grain 1 to obtain film grain model 1.

[0140] Step b3: Determine the first set of film grain parameters based on film grain model 1.

[0141] Step b4: Determine whether Film Grain Model 1 can accurately simulate the film grain of the source image.

[0142] For example, if the difference between the film grain of the source image and the film grain simulated by film grain model 1 is greater than a predetermined threshold, it is determined that film grain model 1 cannot accurately simulate the film grain of the source image.

[0143] Step b5: If the film grain model 1 cannot accurately simulate the film grain of the source image, the amount of film grain remaining in the source image is determined according to the difference between the film grain of the source image and the film grain simulated by the film grain model 1.

[0144] Step b6: A second film grain modeling is performed according to the film grain remaining in the source image to obtain film grain model 2.

[0145] Step b7: Determine the second set of film grain parameters based on film grain model 2.

[0146] Step b8: If the film grain model 2 can accurately simulate the film grain of the source image, stop modeling, write the first set of film grain parameters, the second set of film grain parameters, and the first instruction information to the bitstream, and output the bitstream.

[0147] To facilitate understanding, the information exchange between the encoding and decoding sides in the embodiments of this disclosure will be described below using SEI messages as an example.

[0148] For example, multiple sets of film grain parameters can be stored within the same SEI message.

[0149] In some embodiments, the bitstream includes a first SEI message, which includes at least one of first instruction information, second instruction information, third instruction information, fourth instruction information, and multiple sets of film grain parameters.

[0150] Here, the first instruction information is used to indicate a scheme for adding film grain to the decoded image using multiple sets of film grain parameters. The second instruction information is used to indicate whether to support the addition of film grain based on multiple film grain models. The third instruction information is used to indicate whether to cancel the persistence of the previous first SEI message. The fourth instruction information is used to indicate whether the current first SEI message has persistence.

[0151] For illustrative purposes, Table 1 below shows the syntax of the first SEI message.

[0152] [Table 1]

[0153] For example, the first instruction information may be attached to the fg_model_weight_factor (model weight coefficient) field, the fg_multi_model_joint_mode_id (multi-model joint mode identifier) ​​field, and the fg_multi_model_blending_mode_id (multi-model blending mode identifier) ​​field.

[0154] fg_model_weight_factor: Indicates the importance of each film grain model in the bitstream used in the multiple film grain models associated with this film grain multimodel SEI message. Selectively, the importance of each film grain model refers to the weight coefficient in the joint scheme of film grains generated by each film grain model.

[0155] fg_multi_model_joint_mode_id: Indicates a joint scheme of multiple film grain models. Exemplary, a joint scheme of multiple film grain models includes the first scheme and / or the second scheme, details of which are not repeated here.

[0156] fg_multi_model_blending_mode_id: Indicates the mode in which film grain is added to the decoded image. Exemplaryly, the mode in which film grain is added to the decoded image may be additive mode, multiplicative mode, or any other calculation mode.

[0157] For example, the second instruction information described above may be attached to fg_multi_model_flag (multi-model flag). Here, if fg_multi_model_flag=1, it indicates that the bitstream associated with this SEI message supports the addition of film grain based on multiple film grain models. If fg_multi_model_flag=0, it indicates that the bitstream associated with this SEI message does not support the addition of film grain based on multiple film grain models, i.e., the decoded bitstream associated with the SEI message supports only one type of film grain model.

[0158] For example, the above third instruction information may be attached to fg_multi_model_cancel_flag (multi-model cancellation flag). Here, if fg_multi_model_cancel_flag=1, it means that the film grain multi-model SEI message cancels the persistence of any preceding film grain multi-model SEI messages and closes the associated SEI function. Conversely, if fg_multi_model_cancel_flag=0, it means that the film grain multi-model information follows immediately after the SEI message.

[0159] For example, the fourth instruction information may be attached to fg_multi_model_persistence_flag (multi-model persistence flag) and is used to indicate the persistence of the current film grain multi-model SEI message. Here, if fg_multi_model_persistence_flag=0, it indicates that the current film grain multi-model SEI message applies only to the current decoded image (or sub-image). If fg_multi_model_persistence_flag=1, it indicates that the film grain multi-model SEI message applies not only to the current decoded image (or sub-image) but also persistently to subsequent decoded images (or sub-images) within the persistence range (persistence_scope).

[0160] For example, the persistence scope of a film grain multimodel SEI message may be a single access unit (AU), a coded video sequence (CVS), or unspecified, but the embodiments of this disclosure are not specifically limited thereto.

[0161] For example, multiple sets of film grain parameters may be associated with fg_multi_model_num_minus1 (multi-model number flag) and fg_model_parameters (film grain parameters).

[0162] fg_multi_model_num_minus1: This field is used to indicate the number of film grain models used by the bitstream associated with this film grain multimodel SEI message, after adding 1 to it.

[0163] fg_model_parameters: Specifies the set of film grain parameters to be used by the bitstream associated with this film grain multimodel SEI message.

[0164] Here, the film grain parameters include film grain model parameters and film grain processing parameters. Selectively, the film grain parameters may refer to film grain parameters defined in the Film grain characteristic SEI message in the H.274 standard, or to film grain parameters defined in the video compression encoding standard (AV1: AOMedia Video1) developed by the Open Media Coalition.

[0165] Example 2: An example of an SEI message based on the SEI processing order.

[0166] In some other embodiments, the bitstream includes a plurality of second SEI messages, where each second SEI message includes a set of film grain parameters. To make it clear, in the embodiments of the present disclosure, a co-synthesis scheme for multiple film grain models can be realized by storing each set of film grain parameters for synthesizing film grains in separate SEI messages and processing each SEI message instructing film grain synthesis sequentially according to a predetermined SEI processing order.

[0167] The method for storing a single set of film grain parameters in an SEI message may employ any public standard or private protocol, but the embodiments of this disclosure are not limited thereto.

[0168] In some embodiments, the bitstream further includes a first SEI message, the first SEI message including at least one of a first instruction, a second instruction, a third instruction, a fourth instruction, a plurality of fifth instruction, and a plurality of sixth instruction.

[0169] Here, the first instruction is used to indicate a method for adding film grain to the decoded image using multiple sets of film grain parameters. The second instruction is used to indicate whether to support the addition of film grain based on multiple film grain models. The third instruction is used to indicate whether to cancel the persistence of the previous first SEI message. The fourth instruction is used to indicate whether the current first SEI message has persistence. Each fifth instruction is used to indicate whether the second SEI message corresponding to the fifth instruction is included in the first SEI message. Each sixth instruction is used to indicate the processing order of the second SEI messages corresponding to the sixth instruction.

[0170] For illustrative purposes, Table 2 below shows the syntax of the first SEI message.

[0171] [Table 2]

[0172] Here, fg_multi_model_num_minus1 is the multi-model number flag, and after adding 1 to this field, it is used to indicate the number of film grain models used by the bitstream associated with this film grain multi-model SEI message. Each SEI message for instructing film grain synthesis may correspond to one film grain model, i.e., one set of film grain parameters.

[0173] For example, fg_model_sei_wrapping_flag[i] (i.e., the i-th fifth instruction) is used to indicate whether the i-th second SEI message corresponding to the i-th fifth instruction is included in the first SEI message. Here, if fg_model_sei_wrapping_flag[i]=1, it indicates that the i-th second SEI message corresponding to the i-th fifth instruction is included in the first SEI message, i.e., that the film grain parameters of this pair are included in the corresponding fg_model_sei_message(). If fg_model_sei_wrapping_flag[i]=0, it indicates that the i-th second SEI message corresponding to the i-th fifth instruction information is located outside the first SEI message, meaning that the SEI message storing the film grain parameters for this pair is indexed by fields such as fg_model_sei_prefix_flag[i], fg_model_sei_payload_type[i], fg_model_num_prefix_bytes[i], and fg_model_prefix_byte[i][j].

[0174] For example, fg_model_sei_wrapping_flag[i] (i.e., the i-th sixth instruction) is used to indicate the processing order of the i-th second SEI message corresponding to the i-th sixth instruction.

[0175] Exemplary, fg_model_weight_factor[i] is used to indicate the importance of the i-th 2SEI message. Exemplary, the importance of the i-th 2SEI message can be expressed as the importance of each film grain model employed by the bitstream associated with the i-th 2SEI message. Selectively, the importance of each film grain model refers to the weight coefficient in the joint scheme of film grains generated by each film grain model.

[0176] Furthermore, if the first instruction information does not include fg_model_weight_factor, that is, if there is no distinction of weight coefficients or importance among multiple film grain models, the objective of the joint synthesis process of multiple film grain models can also be achieved by directly using the SEI processing order and SEI message formulated by the JVET standardization organization.

[0177] To ensure clarity, the field information shown in Table 2 can be freely combined and adjusted as needed depending on the specific application scenario.

[0178] For example, referring to Table 1 above, the first SEI message may further include fg_multi_model_cancel_flag (multi-model cancellation flag) accompanied by third instruction information, and fg_multi_model_persistence_flag (multi-model persistence flag) accompanied by fourth instruction information.

[0179] If fg_multi_model_cancel_flag=1, it means that the film grain multimodel SEI message cancels the persistence of any preceding film grain multimodel SEI messages and closes the associated SEI function. Conversely, if fg_multi_model_cancel_flag=0, it means that film grain multimodel information follows immediately.

[0180] If fg_multi_model_persistence_flag=0, it indicates that the current film grain multimodel SEI message applies only to the current decoded image (or sub-image). If fg_multi_model_persistence_flag=1, it indicates that the film grain multimodel SEI message applies not only to the current decoded image (or sub-image) but also persistently to subsequent decoded images (or sub-images) within the persistence range (persistence_scope).

[0181] The above has described the invention relating to embodiments of this disclosure, primarily from a methodological standpoint. To make it clear, a film grain processing apparatus includes at least one corresponding hardware structure and software module that perform each of the above functions. A person skilled in the art will readily recognize, by considering the combination of the units and algorithmic steps of each example described in the embodiments disclosed herein, that embodiments of this disclosure can be implemented in hardware or in a combination of hardware and computer software. Whether a function is performed by hardware or by computer software driving hardware depends on the specific application and design constraints of the invention. A person skilled in the art may use different methods to implement the described functions for each specific application, but such implementations are not considered to exceed the scope of embodiments of this disclosure. 。

[0182] Some embodiments of this disclosure allow for the division of a film grain processing apparatus into functional modules according to embodiments of the above method. For example, each functional module may be divided according to each function, or two or more functions may be integrated into one functional module. The integrated functional module may be implemented using hardware or using software functional modules. Note that the division of modules in the embodiments of this disclosure is illustrative and merely a logical functional division, and other division methods may be possible in actual application. Below, an example of dividing each functional module according to each function will be described.

[0183] Figure 11 shows the configuration of a film grain processing apparatus according to an embodiment of the present disclosure. This film grain processing apparatus is applied to the decoding side and can perform the film grain processing method provided by the method embodiment described above. As shown in Figure 11, the film grain processing apparatus 600 includes an acquisition module 601 and an addition module 602. In some other embodiments, the film grain processing apparatus 600 further includes a determination module 603.

[0184] The acquisition module 601 is used to acquire the decoded image from the bitstream.

[0185] The acquisition module 601 is further used to acquire multiple sets of film grain parameters and first instruction information. The first instruction information is used to instruct a scheme to add film grain to the decoded image using multiple sets of film grain parameters in combination.

[0186] The additional module 602 is used to add film grain to the decoded image based on multiple sets of film grain parameters and first instruction information.

[0187] In some embodiments, each set of multiple sets of film grain parameters includes at least one of the following: a film grain model parameter and a film grain processing parameter.

[0188] In some embodiments, multiple sets of film grain parameters further include the number of film grain models.

[0189] In some embodiments, the first instruction information includes at least one of a joint scheme of multiple film grain models, the importance of each of the multiple film grain models, and a mode for adding the film grain to the decoded image.

[0190] In some embodiments, the joint scheme of multiple film grain models includes a first scheme and / or a second scheme.

[0191] In some embodiments, the first scheme includes generating film grains corresponding to each set of film grain parameters based on those set of film grain parameters, and adding the film grains corresponding to each set of film grain parameters to the decoded image.

[0192] In some embodiments, the second scheme includes generating film grains corresponding to each set of film grain parameters based on those set of film grain parameters, fusing the film grains corresponding to each of the multiple sets of film grain parameters to obtain a fused film grain, and adding the fused film grain parameters to the decoded image.

[0193] In some embodiments, the acquisition module 601 is further used to acquire second instruction information before acquiring multiple sets of film grain parameters and first instruction information. The second instruction information is used to indicate whether to support the addition of film grain based on multiple film grain models. The decision module 603 is used to determine, based on the second instruction information, whether to acquire multiple sets of film grain parameters and first instruction information.

[0194] In some embodiments, the second instruction information is included in the video parameter set VPS and is used to indicate whether the video corresponding to VPS supports the addition of film grain based on multiple film grain models. Alternatively, the second instruction information is included in the sequence parameter set SPS and is used to indicate whether the image sequence corresponding to SPS supports the addition of film grain based on multiple film grain models. Alternatively, the second instruction information is included in the picture parameter set PPS and is used to indicate whether the decoded image of a single frame corresponding to PPS supports the addition of film grain based on multiple film grain models. Alternatively, the second instruction information is included in the adaptive parameter set APS and indicates whether the decoded image of a single frame corresponding to APS supports the addition of film grain based on multiple film grain models. Alternatively, the second instruction information is included in the supplemental enhancement information SEI message and is used to indicate whether the decoded image of a single frame corresponding to the SEI message supports the addition of film grain based on multiple film grain models.

[0195] In some embodiments, the bitstream includes a first SEI message, which includes first instruction information, second instruction information, third instruction information, fourth instruction information, and at least one of a set of film grain parameters. Here, the second instruction information is used to indicate whether to support the addition of film grain based on multiple film grain models. The third instruction information is used to indicate whether to cancel the persistence of a previous first SEI message. The fourth instruction information is used to indicate whether the current first SEI message has persistence.

[0196] In some other embodiments, the bitstream includes multiple second SEI messages, where each second SEI message includes a set of film grain parameters.

[0197] In some embodiments, the bitstream further includes a first SEI message, which includes at least one of the following: first instruction information, second instruction information, third instruction information, fourth instruction information, a plurality of fifth instruction information, and a plurality of sixth instruction information. Here, the second instruction information is used to indicate whether to support the addition of film grain based on a plurality of film grain models. The third instruction information is used to indicate whether to cancel the persistence of a previous first SEI message. The fourth instruction information is used to indicate whether the current first SEI message has persistence. Each fifth instruction information is used to indicate whether the second SEI message corresponding to the fifth instruction information is included in the first SEI message. Each sixth instruction information is used to indicate the processing order of the second SEI messages corresponding to the sixth instruction information.

[0198] Figure 12 shows the configuration of a film grain processing apparatus according to an embodiment of the present disclosure. This film grain processing apparatus is applied to the encoding side and can perform the film grain processing method provided by the method embodiment described above. As shown in Figure 12, the film grain processing apparatus 700 includes an acquisition module 701, a processing module 702, and a determination module 703. In some other embodiments, the film grain processing apparatus 700 further includes a transmission module 704.

[0199] The acquisition module 701 is used to acquire the source image.

[0200] The processing module 702 is used to perform noise reduction processing on the source image and obtain a noise-reduced image.

[0201] The decision module 703 is used to determine multiple sets of film grain parameters and first instruction information based on the source image and the denoising image. The first instruction information is used to instruct a scheme to add film grain to the decoded image using multiple sets of film grain parameters in combination.

[0202] In some embodiments, the transmitting module 704 is used to transmit a bitstream to the decoding side. The bitstream includes encoding information of the source image, multiple sets of film grain parameters, and first instruction information.

[0203] In some embodiments, the transmitting module 704 is used to transmit a bitstream, multiple sets of film grain parameters, and first instruction information to the decoding side. The bitstream contains encoded information of the source image.

[0204] In some embodiments, each set of film grain parameters among multiple sets of film grain parameters includes at least one of a film grain model parameter and a film grain processing parameter.

[0205] In some embodiments, multiple sets of film grain parameters further include the number of film grain models.

[0206] In some embodiments, multiple sets of film grain parameters include at least one set of film grain parameters. The first set of film grain parameters may be determined by determining the film grain of the source image based on the difference between the source image and the denoising image, constructing a first film grain model based on the film grain of the source image, and determining the first set of film grain parameters based on the first film grain model.

[0207] In some embodiments, the multiple sets of film grain parameters further include a second set of film grain parameters. The determination module 703 is further used to determine residual film grain in the source image when the difference between the film grain in the source image and the film grain output by the first film grain model is greater than a threshold. The determination module 703 is further used to construct a second film grain model based on the residual film grain in the source image. The determination module 703 is further used to determine a second set of film grain parameters based on the second film grain model.

[0208] In some embodiments, the first instruction information includes at least one of a joint scheme of multiple film grain models, the importance of each of the multiple film grain models, and a mode for adding the film grain to the decoded image.

[0209] In some embodiments, the transmitting module 704 is used to transmit second instruction information to the decoding end. The second instruction information is used to indicate whether to support the addition of film grain based on multiple film grain models.

[0210] In some embodiments, the second instruction information is included in the video parameter set VPS and is used to indicate whether the video corresponding to VPS supports the addition of film grain based on multiple film grain models. Alternatively, the second instruction information is included in the sequence parameter set SPS and is used to indicate whether the image sequence corresponding to SPS supports the addition of film grain based on multiple film grain models. Alternatively, the second instruction information is included in the picture parameter set PPS and is used to indicate whether the decoded image of a single frame corresponding to PPS supports the addition of film grain based on multiple film grain models. Alternatively, the second instruction information is included in the adaptive parameter set APS and indicates whether the decoded image of a single frame corresponding to APS supports the addition of film grain based on multiple film grain models. Alternatively, the second instruction information is included in the supplemental enhancement information SEI message and is used to indicate whether the decoded image of a single frame corresponding to the SEI message supports the addition of film grain based on multiple film grain models.

[0211] In some embodiments, the bitstream includes a first SEI message, which includes first instruction information, second instruction information, third instruction information, fourth instruction information, and at least one of a set of film grain parameters. Here, the second instruction information is used to indicate whether to support the addition of film grain based on multiple film grain models. The third instruction information is used to indicate whether to cancel the persistence of a previous first SEI message. The fourth instruction information is used to indicate whether the current first SEI message has persistence.

[0212] In some other embodiments, the bitstream includes multiple second SEI messages, where each second SEI message includes a set of film grain parameters.

[0213] In some embodiments, the bitstream further includes a first SEI message, which includes at least one of the following: first instruction information, second instruction information, third instruction information, fourth instruction information, a plurality of fifth instruction information, and a plurality of sixth instruction information. Here, the second instruction information is used to indicate whether to support the addition of film grain based on a plurality of film grain models. The third instruction information is used to indicate whether to cancel the persistence of a previous first SEI message. The fourth instruction information is used to indicate whether the current first SEI message has persistence. Each fifth instruction information is used to indicate whether the second SEI message corresponding to the fifth instruction information is included in the first SEI message. Each sixth instruction information is used to indicate the processing order of the second SEI messages corresponding to the sixth instruction information.

[0214] When the functions of the above-described integrated module are to be implemented in hardware form, an embodiment of the present disclosure provides one possible structure of the communication device referred to in the above-described embodiment. As shown in Figure 13, this communication device 800 includes a processor 802 and a bus 804. Optionally, the communication device may further include a memory 801. Optionally, the communication device 800 may further include a communication interface 803.

[0215] The processor 802 can implement or run various exemplary logic blocks, modules, and circuits as described with reference to embodiments of this disclosure. The processor 802 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field-programmable gate array, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The processor 802 can also implement combinations of computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and so on.

[0216] The communication interface 803 is used to connect to other devices via a communication network. This communication network may be Ethernet®, a wireless access network, or a wireless local area network (WLAN).

[0217] The memory 801 may be, but is not limited to, a read-only memory (ROM) or other type of static storage device such as a static storage device capable of storing information and instructions, a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a magnetic disk storage medium, another magnetic storage device, or any other medium that accompanies or stores desired program code in the form of instructions or data structures and is accessible by a computer.

[0218] In one possible implementation, memory 801 may exist independently of processor 802, and memory 801 may be connected to processor 802 via bus 804 and used to store instructions or program code. When processor 802 calls and executes instructions or program code stored in memory 801, the film grain processing method provided by embodiments of this disclosure can be realized. In another possible implementation, memory 801 may be integrated with processor 802.

[0219] Bus 804 may be an Extended Industry Standard Architecture (EISA) bus, among others. Bus 804 can be classified into address buses, data buses, control buses, etc. For ease of representation, it is shown as a single thick line in Figure 13, but this does not mean that it represents only one bus or one type of bus.

[0220] Some embodiments of this disclosure provide a computer-readable storage medium (e.g., a non-temporary computer-readable storage medium) which stores computer program instructions, and when these computer program instructions are executed on a computer, the computer is instructed to execute a film grain processing method as described in any of the embodiments described above.

[0221] Exemplary examples of computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes), optical discs (e.g., compact discs (CDs), digital versatile discs (DVDs), etc.), smart cards, and flash memory devices (e.g., Erasable Programmable Read-Only Memory (EPROMs), cards, sticks, or key drives, etc.). The various computer-readable storage media described in this disclosure may represent one or more devices and / or other machine-readable storage media for storing information. The term “machine-readable storage media” may include, but is not limited to, wireless channels and various other media that can store, contain, and / or accompany instructions and / or data.

[0222] Embodiments of this disclosure provide a computer program product including instructions. When this computer program product is executed on a computer, it causes the computer to execute a film grain processing method as described in any of the embodiments described above.

[0223] The above description represents only specific embodiments of the Disclosure, and the scope of protection of the Disclosure is not limited thereto. Any modifications or substitutions that a person skilled in the art can easily conceive of within the technical scope of the Disclosure shall be included within the scope of protection of the Disclosure. Accordingly, the scope of protection of the Disclosure shall be governed by the scope of rights set forth in the claims.

Claims

1. A film grain processing method applied to the decoding side, wherein the method is: The steps include obtaining a decoded image from a bitstream, A step of acquiring multiple sets of film grain parameters and first instruction information, wherein the first instruction information is used to instruct a scheme for adding film grain to the decoded image using the multiple sets of film grain parameters in combination. The step of adding film grain to the decoded image based on the plurality of sets of film grain parameters and the first instruction information, method.

2. Each set of film grain parameters among the aforementioned multiple sets of film grain parameters includes at least one of the film grain model parameters and film grain processing parameters. The method according to claim 1.

3. The aforementioned set of film grain parameters further includes the number of film grain models. The method according to claim 2.

4. The first instruction information includes at least one of a joint scheme of multiple film grain models, the importance of each of the multiple film grain models, and a mode for adding the film grain to the decoded image. The method according to claim 1.

5. The aforementioned joint scheme of multiple film grain models includes a first scheme and / or a second scheme, The method according to claim 4.

6. The first scheme is, Based on the film grain parameters of each set, a film grain corresponding to the film grain parameters of that set is generated. This includes adding film grains corresponding to each set of film grain parameters to the decoded image, The method according to claim 5.

7. The aforementioned second scheme is, Based on the film grain parameters of each set, a film grain corresponding to the film grain parameters of that set is generated. The process involves fusing film grains corresponding to each of the aforementioned multiple sets of film grain parameters to obtain a fused film grain, This includes adding the fused film grain parameters to the decoded image, The method according to claim 5.

8. Before acquiring multiple sets of film grain parameters and first instruction information, the method, A step of obtaining second instruction information, wherein the second instruction information is used to indicate whether or not to support the addition of film grain based on multiple film grain models, The process further includes the step of determining whether to acquire the plurality of film grain parameters and the first instruction information based on the second instruction information, The method according to claim 1.

9. The second instruction information is included in the video parameter set VPS and is used to indicate whether the video corresponding to the VPS supports the addition of film grain based on the multiple film grain models, or The second instruction information is included in the sequence parameter set SPS and is used to indicate whether the image sequence corresponding to the SPS supports the addition of film grain based on the plurality of film grain models, or The second instruction information is included in the picture parameter set PPS and is used to indicate whether the decoded image of one frame corresponding to the PPS supports the addition of film grain based on the multiple film grain models, or The second instruction information is included in the adaptive parameter set APS and is used to indicate whether the decoded image of one frame corresponding to the APS supports the addition of film grain based on the plurality of film grain models, or The second instruction information is included in the supplemental enhancement information SEI message and is used to indicate whether the decoded image of one frame corresponding to the SEI message supports the addition of film grain based on the multiple film grain models. The method according to claim 8.

10. The bitstream includes a first SEI message, the first SEI message includes a first instruction information, a second instruction information, a third instruction information, a fourth instruction information, and at least one of the multiple sets of film grain parameters, the second instruction information is used to indicate whether to support the addition of film grain based on multiple film grain models, the third instruction information is used to indicate whether to cancel the persistence of a previous first SEI message, and the fourth instruction information is used to indicate whether the current first SEI message is persistent. The method according to claim 1.

11. The bitstream includes a plurality of second SEI messages, each of which includes a set of film grain parameters. The method according to claim 1.

12. The bitstream further includes a first SEI message, the first SEI message including at least one of the first instruction information, second instruction information, third instruction information, fourth instruction information, a plurality of fifth instruction information, and a plurality of sixth instruction information, wherein the second instruction information is used to indicate whether to support the addition of film grain based on a plurality of film grain models, the third instruction information is used to indicate whether to cancel the persistence of a previous first SEI message, the fourth instruction information is used to indicate whether the current first SEI message is persistent, each of the fifth instruction information is used to indicate whether a second SEI message corresponding to the fifth instruction information is included in the first SEI message, and each of the sixth instruction information is used to indicate the processing order of the second SEI messages corresponding to the sixth instruction information. The method according to claim 11.

13. A film grain processing method applied to the encoding side, wherein the method is: Steps to obtain the source image, The steps include: performing noise reduction processing on the aforementioned source image to obtain a noise-reduced image; The process includes determining a plurality of film grain parameters and a first instruction information based on the source image and the denoising image, wherein the first instruction information is used to instruct a scheme for adding film grain to a decoded image using the plurality of film grain parameters in combination. method.

14. The aforementioned method, The step of transmitting a bitstream to the decoding side, further comprising the step of the bitstream including encoded information of the source image, the plurality of sets of film grain parameters, and the first instruction information, The method according to claim 13.

15. The aforementioned method, A step of transmitting a bitstream, a plurality of sets of film grain parameters, and the first instruction information to the decoding side, further comprising the step of the bitstream including encoded information of the source image, The method according to claim 13.

16. Each set of film grain parameters among the aforementioned multiple sets of film grain parameters includes at least one of the film grain model parameters and film grain processing parameters. The method according to claim 13.

17. The aforementioned set of film grain parameters further includes the number of film grain models. The method according to claim 16.

18. The aforementioned set of multiple film grain parameters includes at least one set of film grain parameters, and the aforementioned set of film grain parameters is The film grain of the source image is determined based on the difference between the source image and the denoised image. Based on the film grain of the aforementioned source image, a first film grain model is constructed. Determined by determining the first set of film grain parameters based on the first film grain model, The method according to claim 13.

19. The aforementioned set of multiple film grain parameters further includes a second set of film grain parameters, and the second set of film grain parameters is The residual film grain in the source image is determined when the difference between the film grain of the source image and the film grain output by the first film grain model is greater than a threshold. Based on the residual film grain in the aforementioned source image, a second film grain model is constructed. The second set of film grain parameters is determined based on the second film grain model. The method according to claim 18.

20. The first instruction information includes at least one of a joint scheme of multiple film grain models, the importance of each of the multiple film grain models, and a mode for adding the film grain to the decoded image. The method according to claim 13.

21. The aforementioned method, A step of transmitting a second instruction information to the decoding end, the second instruction information being used to indicate whether to support the addition of film grain based on a plurality of film grain models, The method according to claim 13.

22. The second instruction information is included in the video parameter set VPS and is used to indicate whether the video corresponding to the VPS supports the addition of film grain based on the multiple film grain models, or The second instruction information is included in the sequence parameter set SPS and is used to indicate whether the image sequence corresponding to the SPS supports the addition of film grain based on the plurality of film grain models, or The second instruction information is included in the picture parameter set PPS and is used to indicate whether the decoded image of one frame corresponding to the PPS supports the addition of film grain based on the multiple film grain models, or The second instruction information is included in the adaptive parameter set APS and is used to indicate whether the decoded image of one frame corresponding to the APS supports the addition of film grain based on the plurality of film grain models, or The second instruction information is included in the supplemental enhancement information SEI message and is used to indicate whether the decoded image of one frame corresponding to the SEI message supports the addition of film grain based on the multiple film grain models. The method according to claim 21.

23. The bitstream includes a first SEI message, the first SEI message includes a first instruction information, a second instruction information, a third instruction information, a fourth instruction information, and at least one of the multiple sets of film grain parameters, the second instruction information is used to indicate whether to support the addition of film grain based on multiple film grain models, the third instruction information is used to indicate whether to cancel the persistence of a previous first SEI message, and the fourth instruction information is used to indicate whether the current first SEI message is persistent. The method according to claim 14.

24. The bitstream includes a plurality of second SEI messages, each of which includes a set of film grain parameters. The method according to claim 14.

25. The bitstream further includes a first SEI message, the first SEI message including at least one of the first instruction information, second instruction information, third instruction information, fourth instruction information, a plurality of fifth instruction information, and a plurality of sixth instruction information, wherein the second instruction information is used to indicate whether to support the addition of film grain based on the plurality of film grain models, the third instruction information is used to indicate whether to cancel the persistence of a previous first SEI message, the fourth instruction information is used to indicate whether the current first SEI message is persistent, each of the fifth instruction information is used to indicate whether a second SEI message corresponding to the fifth instruction information is included in the first SEI message, and each of the sixth instruction information is used to indicate the processing order of the second SEI messages corresponding to the sixth instruction information. The method according to claim 24.

26. A communication device including memory and a processor, wherein the memory and the processor are coupled, the memory is used to store instructions that can be executed by the processor, and when the processor executes the instructions, the film grain processing method according to any one of claims 1 to 25 is realized. Communication device.

27. A computer-readable storage medium, The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by an electronic device, the electronic device is instructed to execute the film grain processing method according to any one of claims 1 to 25. A computer-readable storage medium.

Images