Film grain processing method and apparatus, and storage medium

Abstract

The present disclosure relates to the technical field of image processing, and provides a film grain processing method and apparatus, and a storage medium, capable of simulating the visual effect of film grains more accurately and flexibly. The method comprising: acquiring a decoded image from a bitstream; acquiring a plurality of groups of film grain parameters and first indication information, wherein the first indication information is used for indicating a scheme of jointly using the plurality of groups of film grain parameters to add film grains to the decoded image; and adding film grains to the decoded image on the basis of the plurality of groups of film grain parameters and the first indication information.

Description

Film grain processing method, device and storage medium

[0001] This disclosure claims priority to Chinese patent application No. 202311128154.4, filed on August 30, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates to the field of image processing technology, and in particular to a film grain processing method, device, and storage medium. Background Art

[0003] Film grain technology is currently under discussion and research within international video codec standards organizations. The film grain modeling and synthesis methods defined in different standards vary, as does the degree of support for film grain.

[0004] Related technical research has shown that film grain synthesized based on a single film grain model is coarse and imprecise, with a noticeable repeating template effect. For images with coarse film grain, synthesizing based on a single film grain model can cause problems such as inability to start.

[0005] Summary of the Invention

[0006] The embodiments of the present disclosure provide a film grain processing method, device, and storage medium, which can simulate film grain visual effects more accurately and flexibly.

[0007] In one aspect, a film grain processing method is provided, which is applied to a decoding end and includes:

[0008] Get decoded image from code stream;

[0009] Acquire multiple sets of film grain parameters and first indication information, where the first indication information is used to indicate a scheme of adding film grain to a decoded image by jointly using the multiple sets of film grain parameters;

[0010] Film grain is added to the decoded image based on the multiple sets of film grain parameters and the first indication information.

[0011] In another aspect, a film grain processing method is provided, which is applied to an encoding end and includes:

[0012] Get the source image;

[0013] Perform denoising on the source image to obtain a denoised image;

[0014] Based on the source image and the denoised image, multiple sets of film grain parameters and first indication information are determined, where the first indication information is used to indicate a scheme of jointly using the multiple sets of film grain parameters to add film grain to the decoded image.

[0015] In another aspect, a film grain processing device is provided, which is applied to a decoding end and includes:

[0016] The acquisition module is used to obtain the decoded image from the code stream;

[0017] The acquisition module is further configured to acquire multiple sets of film grain parameters and first indication information, wherein the first indication information is configured to indicate a scheme of adding film grain to the decoded image by jointly using the multiple sets of film grain parameters;

[0018] The adding module is configured to add film grains to the decoded image based on the multiple sets of film grain parameters and the first indication information.

[0019] In another aspect, a film grain processing device is provided, which is applied to an encoding end and includes:

[0020] An acquisition module, used for acquiring a source image;

[0021] A processing module, used for performing denoising on the source image to obtain a denoised image;

[0022] The determination module is configured to determine, based on the source image and the denoised image, multiple sets of film grain parameters and first indication information, wherein the first indication information is configured to indicate a scheme of adding film grain to the decoded image by jointly using the multiple sets of film grain parameters.

[0023] On the other hand, a communication device is provided, comprising: a memory and a processor; the memory and the processor are coupled; the memory is used to store a computer program; and the processor implements the film grain processing method of any of the above embodiments when executing the computer program.

[0024] In another aspect, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the film grain processing method of any of the above embodiments is implemented.

[0025] In another aspect, a computer program product is provided. The computer program product includes computer program instructions. When the computer program instructions are executed by a processor, the film grain processing method described in any one of the above embodiments is implemented.

[0026] The disclosed embodiments provide a film grain processing method capable of adding film grain to a decoded image based on multiple sets of film grain parameters and a scheme for combining these multiple sets of film grain parameters to add film grain to the decoded image. Thus, compared to related art methods that use only a single film grain model to simulate film grain, as defined in various standards or vendor-specific protocols, the disclosed embodiments provide a method for more accurately and flexibly simulating the complex film grain characteristics of video images using multiple sets of film grain parameters (corresponding to multiple film grain models), thereby providing viewers with a better viewing experience.

[0027] At the same time, the method of synthesizing film grains based on multiple film grain models can improve the robustness of the film grain synthesis process, thereby avoiding problems such as failure of the synthesis algorithm due to the large computational complexity of a single model. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the present disclosure, the following briefly introduces the drawings required for use in some embodiments of the present disclosure. Obviously, the drawings described below are only drawings of some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0029] FIG1 is a schematic diagram of an architecture of a coding and decoding system provided by some embodiments of the present disclosure;

[0030] FIG2 is a second schematic diagram of the architecture of a coding and decoding system provided by some embodiments of the present disclosure;

[0031] FIG3 is a third schematic diagram of the architecture of a coding and decoding system provided by some embodiments of the present disclosure;

[0032] FIG4 is a fourth schematic diagram of the architecture of a coding and decoding system provided by some embodiments of the present disclosure;

[0033] FIG5 is a flowchart of a film grain processing method provided by some embodiments of the present disclosure;

[0034] FIG6 is a second flowchart of a film grain processing method provided by some embodiments of the present disclosure;

[0035] FIG7 is a third flowchart of a film grain processing method provided by some embodiments of the present disclosure;

[0036] FIG8 is a fourth flowchart of a film grain processing method provided by some embodiments of the present disclosure;

[0037] FIG9 is a fifth flowchart of a film grain processing method provided by some embodiments of the present disclosure;

[0038] FIG10 is a fifth schematic diagram of the architecture of a coding and decoding system provided by some embodiments of the present disclosure;

[0039] FIG11 is a first structural diagram of a film grain processing device provided by some embodiments of the present disclosure;

[0040] FIG12 is a second structural diagram of a film grain processing device provided by some embodiments of the present disclosure;

[0041] FIG13 is a schematic structural diagram of a communication device provided in some embodiments of the present disclosure. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions of this disclosure in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this disclosure, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of this disclosure without making any creative efforts shall fall within the scope of protection of this disclosure.

[0043] It should be noted that in this disclosure, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this disclosure as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0044] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the quantity of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features.

[0045] Throughout this disclosure, unless otherwise specified, " / " represents "or." For example, A / B can represent either A or B. "And / or" in this document simply describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exists simultaneously, or B exists alone. Furthermore, "at least one" refers to one or more, and "a plurality" refers to two or more.

[0046] Film grain is created during the physical process of photographic film exposure and development (the exposure and development of silver halide crystals) and is widely present in film and television content. However, digital sensors do not undergo this process, resulting in noise-free digital video without film grain. Many content creators, particularly those in the film industry, consider film grain to be a pleasant noise that enhances the natural look of video content. Therefore, re-adding film noise to video to improve the visual experience is often considered part of the creative intent.

[0047] Because video image compression relies heavily on predictions in the temporal, spatial, or cross-component domains, and film grain is inherently random, film grain is difficult to encode using typical video image coding tools. To preserve film grain, one solution is to use smaller quantization parameters during video compression to better preserve detail. However, for video applications such as adaptive streaming and broadcasting, this approach requires a relatively high bitrate (bitrate). Another solution is to model film grain at the encoder to obtain film grain parameters. These parameters are then used at the decoder to generate film grain and add it to the decoded video image. This approach fully realizes the potential of video image compression technology.

[0048] For example, the High Efficiency Video Coding (HEVC) standard (H.265) and the Versatile Video Coding (VVC) standard (H.266) developed by the Joint Video Experts Team (JVET) of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC) and the International Telecommunication Union (ITU) use film grain characteristics (FGC) and supplemental enhancement information (SEI) to indicate film grain parameters used to synthesize film grain when decoding and rendering video or image data, thereby preserving film grain.

[0049] Refer to Figure 1, which is a schematic diagram of the framework of the codec system used in the video codec standard formulated by JVET. As shown in Figure 1, the system includes an encoding end and a decoding end.

[0050] The encoding end includes an encoder, a denoising module, and a film grain modeling module.

[0051] The encoder is used to compress and encode the input source video to obtain an encoded video stream.

[0052] The denoising module is used to pre-process and denoise the input source video to obtain a denoised video, wherein the denoised video includes a video image sequence after film grain has been removed or attenuated.

[0053] The film grain modeling module is used to parameterize the film grain model based on the difference between the denoised video and the source video to obtain film grain parameters (i.e., parameter values ​​of the film grain model).

[0054] The film grain parameters may be transmitted to the decoding end along with the encoded video stream; or may be transmitted to the decoding end through an external transmission method.

[0055] The decoding end includes a decoder and a film grain model.

[0056] The decoder is used to decode the input encoded video stream to obtain a decoded video stream.

[0057] Film grain model, used to synthesize simulated film grain based on input film grain parameters.

[0058] Furthermore, by adding simulated film grain parameters to the decoded video stream, a decoded reconstructed video with simulated film grain can be obtained.

[0059] In some embodiments, in addition to the codec system framework shown in FIG1 , there are other implementations of the codec system framework, the main difference being in the encoding end. For example, instead of encoding the source video image sequence to obtain an encoded video stream, the encoding end may encode the denoised video image sequence to obtain an encoded video. Alternatively, when the source video does not contain film grains, the encoding end may transmit predetermined film grain parameters to the decoding end for film grain synthesis. In this case, even if film grains do not exist in the source video, film grains may be added to the video at the decoder end to achieve a certain effect.

[0060] At the 29th JVET meeting, proposal JVET-AC0199 provided film grain quality assessment results obtained using both a frequency-based method and an auto-regressive-based method, mentioning:

[0061] (1) When the source video has heavy film grain, the synthesized film grain has obvious repetitive patterns in the coarse grain.

[0062] (2) When the source video has relatively light / low film grain, the synthesized film grain effect is better;

[0063] (3) The frequency-based Film Grain Synthesis (FGS) method and the autoregressive-based FGS method show different results in different situations.

[0064] It can be seen that the FGS method based on a single model has poor flexibility and accuracy, and it is difficult to produce a good simulation effect on complex film grains.

[0065] To address the above technical issues, embodiments of the present disclosure provide a film grain processing method. The method employs multiple sets of film grain parameters, or their combined use, to add film grain to the decoded image. Compared to related art methods that use only a single film grain model to simulate film grain, as defined in various standards or vendor-specific protocols, the method provided in the present disclosure utilizes multiple sets of film grain parameters (corresponding to multiple film grain models) to simulate the complex film grain characteristics of video images. This method achieves a more accurate and flexible simulation of film grain visual effects, thereby providing viewers with a better viewing experience.

[0066] At the same time, the method of synthesizing film grains based on multiple film grain models can improve the robustness of the film grain synthesis process, thereby avoiding problems such as failure of the synthesis algorithm due to the large computational complexity of a single model.

[0067] The following is a detailed introduction to the encoding and decoding system provided by the embodiments of the present disclosure in conjunction with the accompanying drawings.

[0068] 2 , which is a schematic diagram of the architecture of a coding and decoding system provided by an embodiment of the present disclosure. As shown in FIG2 , the coding and decoding system 100 includes: an encoding end 110 and a decoding end 120 .

[0069] The encoding end 110 generates an encoded video stream. Therefore, the encoding end 110 can be referred to as a video encoding device. The decoding end 120 can decode the encoded video stream generated by the encoding end. Therefore, the decoding end 120 can be referred to as a video decoding device. Various implementations of the encoding end 110, the decoding end 120, or both can include one or more processors and a memory coupled to the one or more processors. The memory can include, but is not limited to, RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store desired program code in the form of computer-accessible instructions or data structures, as described herein.

[0070] In some embodiments, the encoding end 110 and the decoding end 120 may include various devices, including desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, telephone handsets such as so-called "smart" phones, televisions, cameras, display devices, digital media players, video game consoles, car computers, or the like.

[0071] In some embodiments, decoding end 120 may receive an encoded video stream from encoding end 110 via a link. The link may include one or more media or devices capable of moving the encoded video data from the encoding end to decoding end 120. In one example, the link may include one or more communication media that enable encoding end 110 to transmit the encoded video stream directly to the decoding end in real time. In this example, encoding end 110 may modulate the encoded video stream according to a communication standard (e.g., a wireless communication protocol) and transmit the modulated video stream to decoding end 120. The one or more communication media may include wireless and / or wired communication media, such as radio frequency (RF) spectrum or one or more physical transmission lines. The one or more communication media may form part of a packet-based network, such as a local area network, a wide area network, or a global network (e.g., the Internet). The one or more communication media may include routers, switches, base stations, or other devices that facilitate communication from encoding end 110 to decoding end 120.

[0072] In some embodiments, as shown in FIG3 , the encoding end 110 includes an encoder, a denoising module, and multiple film grain modeling modules.

[0073] The encoder is used to compress and encode the input source video to obtain an encoded video stream.

[0074] The denoising module is used to pre-process and denoise the input source video to obtain a denoised video, wherein the denoised video includes a video image sequence after film grain has been removed or attenuated.

[0075] The film grain modeling module is used to parameterize the film grain model and obtain film grain parameters (i.e., parameter values ​​of the film grain model). Multiple film grain modeling modules can obtain multiple sets of film grain parameters.

[0076] The film grain parameters may be transmitted to the decoding end 120 along with the encoded video stream; or may be transmitted to the decoding end 120 via an external transmission method.

[0077] In some embodiments, as shown in FIG. 4 , the decoding end 120 includes a decoder and a plurality of film grain models.

[0078] The decoder is used to decode the input encoded video stream to obtain a decoded video stream.

[0079] Film grain models are used to synthesize simulated film grain based on input film grain parameters. Multiple film grain models can be used to synthesize multiple simulated film grains.

[0080] In some embodiments, multiple simulated film grains may be sequentially added to a decoded video stream to obtain a decoded reconstructed video with simulated film grains.

[0081] In some embodiments, multiple simulated film grain parameters may be fused according to preset weights to obtain fused film grains, and the fused film grains may be added to a decoded video stream to obtain a decoded reconstructed video with simulated film grains.

[0082] It is understandable that the application scenarios of the embodiments of the present disclosure are not limited. The system architecture and business scenarios described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will appreciate that with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present disclosure are equally applicable to similar technical problems.

[0083] The following is a detailed introduction to the film grain processing method provided by the embodiment of the present disclosure.

[0084] Based on the decoding end 120 shown in FIG4 , the present disclosure further provides a film grain processing method, as shown in FIG5 . FIG5 is a flow chart of a film grain processing method provided by the present disclosure. The film grain processing method can be executed by the decoding end 120 and includes the following steps:

[0085] S201: Obtain a decoded image from a code stream.

[0086] The data rate refers to the data flow used by a video file per unit time, also known as the bit rate. For example, the data rate can be an image data stream or a transport stream or media file including an image data stream.

[0087] In some embodiments, the above step S201 can be implemented as follows: decoding the code stream to obtain a decoded image. For example, the decoding method can adopt the method specified in the video decoding standard, which is not limited in the embodiment of the present disclosure.

[0088] Exemplarily, the decoded image may include at least one of the following: an image sequence consisting of multiple consecutive frames of images, a single image, a frame of image in the image sequence, or a partial image area in a frame of image.

[0089] It is understandable that, given the randomness and diversity of film grain effects in video images, the film grain processing methods provided by the embodiments of the present disclosure can be adaptively applied to image sequences, single images, or partial image regions within a frame. For example, the film grain in a complete image sequence can be synthesized using a single or multiple film grain models. The number of film grain models used can depend on the complexity of the film grain, the model simulation effect, the model computational complexity, and other factors. In short, in order to more accurately and flexibly simulate the visual effects of realistic film grain, using multiple film grain models to jointly synthesize film grain is a reliable and effective solution.

[0090] In some embodiments, a codestream includes a first SEI message, and the first SEI message includes at least one of the following: first indication information, second indication information, third indication information, fourth indication information, and multiple sets of film grain parameters; wherein the second indication information indicates whether film grain addition based on multiple film grain models is supported; the third indication information indicates whether to cancel the persistence of the previous first SEI message; and the fourth indication information indicates whether the current first SEI message is persistent. It is understood that multiple sets of film grain parameters can be stored in the same SEI message.

[0091] In some other embodiments, the codestream includes multiple second SEI messages, wherein each second SEI message includes a set of film grain parameters. It is understood that the multiple sets of film grain parameters can be stored in different SEI messages.

[0092] The code stream also includes a first SEI message, and the first SEI message includes at least one of the following: first indication information, second indication information, third indication information, fourth indication information, multiple fifth indication information, and multiple sixth indication information; wherein the second indication information is used to indicate whether to support adding film grains based on multiple film grain models; the third indication information is used to indicate whether to cancel the persistence of the previous first SEI message; the fourth indication information is used to indicate whether the first SEI message of this time is persistent; each fifth indication information is used to indicate whether the second SEI message corresponding to the fifth indication information is located in the first SEI message; and each sixth indication information is used to indicate the processing order of the second SEI messages corresponding to the sixth indication information.

[0093] S202: Acquire multiple sets of film grain parameters and first indication information, where the first indication information is used to indicate a scheme of adding film grain to a decoded image by jointly using the multiple sets of film grain parameters.

[0094] In some embodiments, the multiple sets of film grain parameters and the first indication information may be included in a bitstream; or, the multiple sets of film grain parameters may be preconfigured parameters, and the first indication information may be preconfigured indication information.

[0095] In some embodiments, each set of film grain parameters in the plurality of sets of film grain parameters includes at least one of the following: a film grain model parameter, and a film grain processing parameter.

[0096] The film grain model parameters may include a film grain model identifier. The film grain processing parameters are sets of film grain processing parameters corresponding to the film grain synthesis method indicated by the film grain model parameters. Exemplarily, the film grain synthesis method may employ a method defined by any public standard or proprietary protocol, and this disclosure is not limited thereto.

[0097] Optionally, the film grain processing parameters may also be referred to as film grain synthesis parameters or film grain modeling parameters.

[0098] Exemplarily, when the film grain model identifier indicates that a frequency filtering-based film grain synthesis method is adopted, the above-mentioned film grain processing parameters may include a luminance bit depth parameter fg_bit_depth_luma_minus8, a chroma bit depth parameter fg_bit_depth_chroma_minus8, an image color gamut fg_colour_primaries corresponding to the film grains, a color space fg_matric_coeffs, a transfer function fg_transfer_chara cteristics, a blending mode parameter fg_blending_mode_id for film grains and decoded images, a film grain scaling parameter fg_log2_scale_factor, a model number parameter fg_num_model_values_minus1[c] for the intensity intervals of different color components of the film grains, and a model value fg_comp_model_value[c][i][j] corresponding to each intensity interval of the film grains for different color components.

[0099] In some embodiments, the multiple sets of film grain parameters further include the number of film grain models. Exemplarily, the number of film grain models included in the N sets of film grain parameters is N, where N is an integer greater than or equal to 1.

[0100] In some embodiments, the first indication information includes at least one of the following: a combination scheme of multiple film grain models, an importance level of each of the multiple film grain models, and a mode of adding film grains to the decoded image.

[0101] In some embodiments, the combined solution of the plurality of film grain models includes the first solution and / or the second solution.

[0102] The first solution includes: generating film grains corresponding to each set of film grain parameters based on the set of film grain parameters; and then adding the film grains corresponding to each set of film grain parameters into the decoded image.

[0103] For example, assuming that the multiple sets of film grain parameters include a first set of film grain parameters and a second set of film grain parameters, film grain 1 is generated based on the first set of film grain parameters; film grain 2 is generated based on the second set of film grain parameters; and film grain 1 and film grain 2 are then added to the decoded image.

[0104] The second solution includes: generating film grain corresponding to each set of film grain parameters based on the set of film grain parameters; fusing the film grains corresponding to multiple sets of film grain parameters to obtain fused film grain; and adding the fused film grain to the decoded image.

[0105] The method for fusing the film grains corresponding to the multiple sets of film grain parameters may adopt an addition mode, a multiplication mode or any other calculation mode.

[0106] For example, assuming that the multiple sets of film grain parameters include: a first set of film grain parameters and a second set of film grain parameters, film grain 1 is generated based on the first set of film grain parameters; film grain 2 is generated based on the second set of film grain parameters; film grain 1 and film grain 2 are then fused to obtain fused film grain, and the fused film grain is then added to the decoded image.

[0107] In some embodiments, the importance of each of the multiple film grain models refers to the weight coefficient of the film grain generated by each film grain model in the joint solution.

[0108] In some embodiments, the mode of adding film grains to the decoded image may adopt an addition mode, a multiplication mode, or any other calculation mode.

[0109] In some embodiments, as shown in FIG6 , before the above step S202 , the method may further include the following steps S300 - S301 .

[0110] S300: Obtain second indication information, where the second indication information is used to indicate whether adding film grains based on multiple film grain models is supported.

[0111] As a possible implementation manner, the second indication information is included in a video parameter set (VPS), and is used to indicate whether the video corresponding to the VPS supports adding film grains based on multiple film grain models.

[0112] As another possible implementation, the second indication information is included in a sequence parameter set (SPS), and is used to indicate whether the image sequence corresponding to the SPS supports adding film grains based on multiple film grain models.

[0113] As another possible implementation, the second indication information is included in a picture parameter set (PPS), and is used to indicate whether a decoded image frame corresponding to the PPS supports adding film grains based on multiple film grain models.

[0114] As another possible implementation manner, the second indication information is included in an adaptation parameter set (APS), and is used to indicate whether a decoded image frame corresponding to the APS supports adding film grains based on multiple film grain models.

[0115] As another possible implementation, the second indication information is included in a supplemental enhancement information (SEI) message, and is used to indicate whether a decoded image frame corresponding to the SEI message supports adding film grains based on multiple film grain models.

[0116] S301: Determine whether to obtain multiple sets of film grain parameters and first indication information based on second indication information.

[0117] In some embodiments, when the second indication information is used to indicate that adding film grain based on multiple film grain models is supported, multiple sets of film grain parameters and the first indication information are obtained; when the second indication information is used to indicate that adding film grain based on multiple film grain models is not supported, multiple sets of film grain parameters and the first indication information are not obtained, and further, a single film grain model can be used to add film grain.

[0118] It should be noted that the execution order shown in FIG6 is merely an example and does not limit the film grain processing method provided by the present embodiment. The present embodiment does not limit the execution order of step S201 and steps S300-S301. For example, step S201 may be executed first and then steps S300-S301; alternatively, steps S300-S301 may be executed first and then step S201; or, steps S201 and S300-S301 may be executed simultaneously.

[0119] S203: Add film grains to the decoded image based on the multiple sets of film grain parameters and the first instruction information.

[0120] In some embodiments, the above step S203 can be implemented as the following steps:

[0121] Step a1: Generate multiple film grains corresponding to the decoded image based on multiple sets of film grain parameters.

[0122] In some embodiments, film grain corresponding to the decoded image is generated based on the film grain model parameters and the film grain processing parameters included in each set of film grain parameters.

[0123] It should be noted that a set of film grain parameters corresponds to a film grain model, and multiple film grain models can be independent of each other or have dependencies.

[0124] For example, the processing of multiple film grain models can be performed in parallel. For example, multiple film grain models can be used to synthesize their corresponding film grains simultaneously without any order restriction.

[0125] For example, the processing order of multiple film grain models may also be serial. For example, there is a predefined processing order between multiple film grain models, and multiple specified film grain models need to be applied in sequence to achieve the desired film grain synthesis effect.

[0126] Step a2: Add a plurality of film grains to the decoded image based on the first indication information.

[0127] In some embodiments, film grain is added to the decoded image based on a combination of multiple film grain models, respective importance of the multiple film grain models, and a mode for adding film grain to the decoded image.

[0128] Exemplarily, when two film grain models are applied to the same image and the processing of the two film grain models is parallel, film grain 1 can be generated based on film grain model 1, and film grain 2 can be generated based on film grain model 2; then, the weight coefficient of film grain 1 is determined according to the importance of film grain model 1, and the weight coefficient of film grain 2 is determined according to the importance of film grain model 2; finally, film grain 1, the weight coefficient of film grain 1, film grain 2, and the weight coefficient of film grain 2 are fused using a weighted summation mode to obtain the fused film grain, and the fused film grain is added to the decoded image.

[0129] Furthermore, the method for adding the film grain synthesized from multiple film grain models to the decoded image may adopt an additive mode, a multiplicative mode, or any other calculation mode. Alternatively, the method for adding the film grain synthesized from multiple film grain models to the decoded image may adopt a hybrid mode that is a combination of multiple calculation modes.

[0130] It will be appreciated that the film grain processing method provided in the embodiments of the present disclosure can add film grain to the decoded image based on multiple sets of film grain parameters and the scheme of using multiple sets of film grain parameters in combination to add film grain to the decoded image. Thus, compared to the related art methods of simulating film grain using only a single film grain model defined in different standards or manufacturer-specific protocols, the method provided in the embodiments of the present disclosure using multiple sets of film grain parameters (corresponding to multiple film grain models) to simulate the complex film grain characteristics in video images can achieve the goal of more accurately and flexibly simulating the visual effects of film grain, thereby providing viewers with a better viewing experience.

[0131] At the same time, the method of synthesizing film grains based on multiple film grain models can improve the robustness of the film grain synthesis process, thereby avoiding problems such as failure of the synthesis algorithm due to the large computational complexity of a single model.

[0132] Based on the encoding end 110 shown in FIG3 , the present disclosure further provides a film grain processing method, as shown in FIG7 . FIG7 is a flow chart of a film grain processing method provided by the present disclosure. The film grain processing method can be performed by the encoding end 110 and includes the following steps:

[0133] S401: Acquire a source image.

[0134] Exemplarily, the source image may be an image sequence consisting of a plurality of consecutive frames of images; or a single image; or a frame of image in the image sequence; or a partial image region in a frame of image.

[0135] S402: Perform denoising processing on the source image to obtain a denoised image.

[0136] The denoised image is an image after film grains are removed or attenuated.

[0137] Exemplarily, the above-mentioned denoising process may be a spatial domain denoising method, a transform domain denoising method, or a spatial domain and transform domain collaborative filtering method, etc., which is not limited in the embodiments of the present disclosure.

[0138] S403 : Determine multiple sets of film grain parameters and first indication information based on the source image and the denoised image, where the first indication information is used to indicate a scheme of adding film grain to the decoded image by jointly using the multiple sets of film grain parameters.

[0139] In some embodiments, each set of film grain parameters in the plurality of sets of film grain parameters includes at least one of the following: a film grain model parameter, a film grain processing parameter.

[0140] In some embodiments, the multiple sets of film grain parameters further include the number of film grain models.

[0141] In some embodiments, the first indication information includes at least one of the following: a combination scheme of multiple film grain models, an importance level of each of the multiple film grain models, and a mode for adding film grains to the decoded image.

[0142] In some embodiments, the plurality of sets of film grain parameters include at least a first set of film grain parameters; as shown in FIG8 , the first set of film grain parameters may be determined by the following steps:

[0143] S501 : Determine the film grain of the source image based on the difference between the source image and the denoised image.

[0144] Exemplarily, the difference between the source image and the denoised image can be reflected by the difference between corresponding pixels in the source image and the denoised image.

[0145] S502: Construct a first film grain model based on the film grain of the source image.

[0146] In some embodiments, the above step S502 may be implemented as follows: constructing a first film grain model based on the film grain of the source image and a preset film grain modeling method.

[0147] Illustratively, the preset film grain modeling method may be a frequency-based film grain modeling method or an autoregressive coefficient-based film grain modeling method.

[0148] In some embodiments, a first film grain model is constructed based on film grain characteristics of film grain of a source image, whereby the film grain characteristics include, but are not limited to, film grain intensity, film grain scaling parameters, and the like.

[0149] The film grain characteristics may be the film grain characteristics of the entire source image; or, the film grain characteristics may be the film grain characteristics of different regions in the source image.

[0150] S503: Determine a first set of film grain parameters based on the first film grain model.

[0151] Illustratively, the first film grain model is parameterized to obtain a first set of film grain parameters.

[0152] In some embodiments, the plurality of sets of film grain parameters include at least a second set of film grain parameters; as shown in FIG9 , the second set of film grain parameters may be determined by the following steps:

[0153] S504 : When the difference between the film grain of the source image and the film grain output by the first film grain model is greater than a threshold, determine the residual film grain in the source image.

[0154] It can be understood that when the difference between the film grain of the source image and the film grain output by the first film grain model is greater than the threshold, it means that the first film grain model cannot accurately simulate the film grain in the source image. Therefore, it is necessary to construct multiple film grain models to achieve the purpose of simulating the film grain visual effect more accurately and flexibly.

[0155] In some embodiments, the residual film grain in the source image may be determined based on the difference between the film grain in the source image and the film grain output by the first film grain model.

[0156] S505: Construct a second film grain model based on the residual film grains in the source image.

[0157] In some embodiments, a second film grain model is constructed based on film grain characteristics of the residual film grain.

[0158] S506 : Determine a second set of film grain parameters based on the second film grain model.

[0159] Illustratively, film grain parameterization is performed on the second film grain model to obtain a second set of film grain parameters.

[0160] In some embodiments, after step S506, the method further includes determining whether the first film grain model and the second film grain model can accurately simulate the film grain of the source image. If the first film grain model and the second film grain model cannot accurately simulate the film grain of the source image, the method further includes constructing a third film grain model, and then repeatedly determining the constructed film grain model until the constructed film grain model can accurately simulate the film grain of the source image.

[0161] It is understandable that the embodiments of the present disclosure do not limit the number of film grain models. If the constructed film grain model cannot accurately simulate the film grain of the source image, one or more film grain models may be constructed again.

[0162] In some embodiments, the above method further includes: sending second indication information to the decoding end, where the second indication information is used to indicate whether adding film grains based on multiple film grain models is supported.

[0163] As a possible implementation, after step S403, the method further includes: sending a code stream to a decoding end, where the code stream includes coding information of the source image, multiple sets of film grain parameters, and first indication information.

[0164] It is understandable that the coding information of the source image, multiple sets of film grain parameters and the first indication information can be written into the bitstream and transmitted to the decoding end together with the bitstream, thereby saving channel resources.

[0165] As another possible implementation, after step S403, the method further includes: sending a code stream, multiple sets of film grain parameters, and first indication information to a decoding end, where the code stream includes encoding information of the source image.

[0166] It is understandable that the code stream, the encoding information of the source image, the multiple sets of film grain parameters and the first indication information can be sent separately, so that independent scheduling can be achieved.

[0167] It should be noted that there are many methods for encoding an image to generate a code stream. For example, the method specified in the H.26x series of standards may be used, and the embodiments of the present disclosure are not limited thereto.

[0168] For ease of understanding, the film grain processing method applied to the encoding end is described below in the form of an example.

[0169] Exemplarily, as shown in FIG10 , the film grain processing method applied to the encoding end can be implemented as follows:

[0170] Step b1: Obtain film grain 1 of the source image based on the difference between the source image and the denoised image.

[0171] Step b2: Perform a first film grain modeling based on the film grain characteristics of film grain 1 to obtain film grain model 1.

[0172] Step b3: Based on the film grain model 1, determine a first set of film grain parameters.

[0173] Step b4: Determine whether the film grain model 1 can accurately simulate the film grain of the source image.

[0174] Illustratively, when the difference between the film grain of the source image and the film grain simulated by the film grain model 1 is greater than a preset threshold, it is determined that the film grain model 1 is unable to accurately simulate the film grain of the source image.

[0175] Step b5: If the film grain model 1 cannot accurately simulate the film grain of the source image, determine the residual film grain in the source image based on the difference between the film grain of the source image and the film grain simulated by the film grain model 1.

[0176] Step b6: Perform a second film grain modeling based on the film grains remaining in the source image to obtain film grain model 2.

[0177] Step b7: Based on film grain model 2, determine a second set of film grain parameters.

[0178] Step b8: If the film grain model 2 can accurately simulate the film grain of the source image, stop modeling, write the first set of film grain parameters, the second set of film grain parameters, and the first indication information into the bitstream, and output the bitstream.

[0179] For ease of understanding, the information interaction between the encoding end and the decoding end in the embodiment of the present disclosure is described below using SEI message as an example.

[0180] Example 1: Multiple sets of film grain parameters are stored in the same SEI message.

[0181] In some embodiments, the code stream includes a first SEI message, and the first SEI message includes at least one of the following: first indication information, second indication information, third indication information, fourth indication information, and multiple sets of film grain parameters.

[0182] Among them, the first indication information is used to indicate the scheme of jointly using multiple sets of film grain parameters to add film grain to the decoded image; the second indication information is used to indicate whether to support adding film grain based on multiple film grain models; the third indication information is used to indicate whether to cancel the persistence of the previous first SEI message; and the fourth indication information is used to indicate whether the current first SEI message is persistent.

[0183] Illustratively, the syntax of the first SEI message is shown in Table 1 below.

[0184] Table 1

[0185] Exemplarily, the first indication information may be carried in the fg_model_weight_factor (model weight factor) field, the fg_multi_model_joint_mode_id (multi-model joint mode identifier) ​​field, and the fg_multi_model_blending_mode_id (multi-model blending mode identifier) ​​field.

[0186] fg_model_weight_factor: Indicates the importance of each of the multiple film grain models used in the codestream associated with this Film Grain Multi-Model SEI message. Optionally, the importance of each film grain model can be the weight coefficient of the film grain generated by each film grain model in the joint solution.

[0187] fg_multi_model_joint_mode_id: indicates a joint scheme of multiple film grain models. Exemplarily, the joint scheme of multiple film grain models includes the first scheme and / or the second scheme, which will not be described in detail here.

[0188] fg_multi_model_blending_mode_id: indicates the mode for adding film grain to the decoded image. Exemplarily, the mode for adding film grain to the decoded image may adopt an addition mode, a multiplication mode, or any other calculation mode.

[0189] Exemplarily, the second indication information may be carried in fg_multi_model_flag (multi-model flag). When fg_multi_model_flag = 1, it indicates that the code stream associated with the SEI message supports adding film grain based on multiple film grain models; when fg_multi_model_flag = 0, it indicates that the code stream associated with the SEI message does not support adding film grain based on multiple film grain models, that is, the code stream to be decoded associated with the SEI message supports only one film grain model.

[0190] Exemplarily, the third indication information may be carried in fg_multi_model_cancel_flag (multi-model cancel flag). When fg_multi_model_cancel_flag = 1, it indicates that the film grain multi-model SEI message cancels the persistence of any previous film grain multi-model SEI message and disables the related SEI functions. Conversely, when fg_multi_model_cancel_flag = 0, it indicates that the film grain multi-model information immediately follows.

[0191] Exemplarily, the fourth indication information may be carried in fg_multi_model_persistence_flag (multi-model persistence flag) to indicate the persistence of the current film grain multi-model SEI message. When fg_multi_model_persistence_flag = 0, it indicates that the current film grain multi-model SEI message is only applicable to the current decoded picture (or sub-picture). When fg_multi_model_persistence_flag = 1, it indicates that the film grain multi-model SEI message is applicable not only to the current decoded picture (or sub-picture), but also to subsequent decoded pictures (or sub-pictures) within the persistence range (peisistence_scope).

[0192] For example, the persistence scope (peisistence_scope) of the film grain multi-model SEI message can be an access unit (AU), a coded video sequence (CVS), or unspecified. The embodiments of the present disclosure do not impose specific restrictions on this.

[0193] For example, multiple sets of film grain parameters can be carried in fg_multi_model_num_minus1 (multi-model number flag) and fg_model_parameters (film grain parameters).

[0194] fg_multi_model_num_minus1: This field is incremented by 1 to indicate the number of film grain models used in the codestream associated with this film grain multi-model SEI message.

[0195] fg_model_parameters: indicates each set of film grain parameters used by the codestream associated with this film grain multi-model SEI message.

[0196] The film grain parameters include film grain model parameters and film grain processing parameters. Optionally, the film grain parameter indication may be the film grain parameters defined in the Film Grain Characteristic SEI message in the H.274 standard or the film grain parameters defined in the video compression coding standard launched by the Alliance for Open Media (AOMedia Video1, AV1).

[0197] Example 2: SEI message example based on SEI processing order.

[0198] In other embodiments, the codestream includes multiple second SEI messages, each of which includes a set of film grain parameters. It is understood that in the disclosed embodiments, each set of film grain parameters for synthesizing film grain can be stored in a separate SEI message, and each SEI message indicating film grain synthesis can be processed sequentially according to a pre-set SEI processing order to implement a joint synthesis solution for multiple film grain models.

[0199] The method of storing a single set of film grain parameters in an SEI message may adopt any public standard or private protocol, which is not limited in the embodiments of the present disclosure.

[0200] In some embodiments, the code stream further includes a first SEI message, and the first SEI message includes at least one of the following: first indication information, second indication information, third indication information, fourth indication information, multiple fifth indication information, and multiple sixth indication information.

[0201] Among them, the first indication information is used to indicate a scheme of jointly using multiple sets of film grain parameters to add film grain to the decoded image; the second indication information is used to indicate whether to support adding film grain based on multiple film grain models; the third indication information is used to indicate whether to cancel the persistence of the previous first SEI message; the fourth indication information is used to indicate whether the first SEI message of this time is persistent; each fifth indication information is used to indicate whether the second SEI message corresponding to the fifth indication information is located in the first SEI message; and each sixth indication information is used to indicate the processing order of the second SEI message corresponding to the sixth indication information.

[0202] Exemplarily, the syntax of the first SEI message is shown in Table 2 below.

[0203] Table 2

[0204] Among them, fg_multi_model_num_minus1 is the multi-model number flag. After adding 1, this field is used to indicate the number of film grain models used in the codestream associated with this film grain multi-model SEI message. Each SEI message used to indicate film grain synthesis can correspond to a film grain model, that is, a set of film grain parameters.

[0205] Exemplarily, fg_model_sei_wrapping_flag[i] (ie, the i-th fifth indication information) is used to indicate whether the i-th second SEI message corresponding to the i-th fifth indication information is located in the first SEI message. Among them, when fg_model_sei_wrapping_flag[i]=1, it indicates that the i-th second SEI message corresponding to the i-th fifth indication information is located in the first SEI message, that is, the group of film grain parameters is included in the corresponding fg_model_sei_message(); when fg_model_sei_wrapping_flag[i]=0, it indicates that the i-th second SEI message corresponding to the i-th fifth indication information is located outside the first SEI message, that is, the SEI message storing the group of film grain parameters will be indexed by fields such as fg_model_sei_prefix_flag[i], fg_model_sei_payload_type[i], fg_model_num_prefix_bytes[i], and fg_model_prefix_byte[i][j].

[0206] Exemplarily, fg_model_sei_processing_order[i] (i.e., the i-th sixth indication information) is used to indicate the processing order of the i-th second SEI message corresponding to the i-th sixth indication information.

[0207] Exemplarily, fg_model_weight_factor[i] is used to indicate the importance of the i-th second SEI message. Exemplarily, the importance of the i-th second SEI message can be represented by the importance of each film grain model used in the codestream associated with the i-th second SEI message. Optionally, the importance of each film grain model can be the weight coefficient of the film grain generated by each film grain model in the joint solution.

[0208] It should be noted that when the first indication information does not include fg_model_weight_factor, that is, when there is no weight coefficient or importance difference between multiple film grain models, the SEI processing order (SEI processing order) and SEI message (SEI message) formulated by the JVET standard organization can also be directly used to achieve the purpose of joint synthesis processing of multiple film grain models.

[0209] It is understandable that the field information shown in Table 2 can be freely combined or appropriately adjusted according to specific application scenarios.

[0210] Exemplarily, referring to Table 1 above, the first SEI message may further include: fg_multi_model_cancel_flag (multi-model cancellation flag) carrying the third indication information, and fg_multi_model_persistence_flag (multi-model persistence flag) carrying the fourth indication information.

[0211] When fg_multi_model_cancel_flag = 1, it indicates that the film grain multi-model SEI message cancels the persistence of any previous film grain multi-model SEI message and turns off the related SEI function. Conversely, when fg_multi_model_cancel_flag = 0, it indicates that the following is the film grain multi-model information.

[0212] When fg_multi_model_persistence_flag = 0, it indicates that the current film grain multi-model SEI message is only applicable to the current decoded picture (or sub-picture). When fg_multi_model_persistence_flag = 1, it indicates that the film grain multi-model SEI message is applicable not only to the current decoded picture (or sub-picture), but also to subsequent decoded pictures (or sub-pictures) within the persistence scope (peisistence_scope).

[0213] The above mainly introduces the solution of the embodiment of the present disclosure from the perspective of method. It can be understood that in order to realize the above functions, the film grain processing device includes at least one of the hardware structure and software modules corresponding to each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiment disclosed herein, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiment of the present disclosure.

[0214] It is understood that, in order to implement the aforementioned functions, the film grain processing device includes hardware structures and / or software modules corresponding to each function. Those skilled in the art will readily appreciate that, in conjunction with the algorithmic steps of the various examples described in the embodiments of this disclosure, the present disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is implemented in hardware or in a hardware-driven manner by computer software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this disclosure.

[0215] The disclosed embodiments can divide the film grain processing device into functional modules based on the aforementioned method embodiments. For example, each functional module can be divided into corresponding functional modules, or two or more functions can be integrated into a single functional module. These integrated modules can be implemented in either hardware or software. It should be noted that the module division in the disclosed embodiments is illustrative and represents only one logical functional division. In actual implementation, other division methods are possible. The following explanation uses the example of dividing each functional module into corresponding functional modules.

[0216] Figure 11 is a schematic diagram of the structure of a film grain processing device provided in an embodiment of the present disclosure. This device, applied to a decoder, can execute the film grain processing method provided in the aforementioned method embodiment. As shown in Figure 11 , the film grain processing device 600 includes an acquisition module 601 and an addition module 602. In other embodiments, the film grain processing device 600 also includes a determination module 603.

[0217] The acquisition module 601 is configured to acquire a decoded image from a bitstream.

[0218] The acquisition module 601 is further configured to acquire multiple sets of film grain parameters and first indication information, where the first indication information is used to indicate a scheme of jointly using the multiple sets of film grain parameters to add film grain to the decoded image.

[0219] The adding module 602 is configured to add film grains to the decoded image based on multiple sets of film grain parameters and the first indication information.

[0220] In some embodiments, each set of film grain parameters in the plurality of sets of film grain parameters includes at least one of the following: a film grain model parameter, and a film grain processing parameter.

[0221] In some embodiments, the multiple sets of film grain parameters further include the number of film grain models.

[0222] In some embodiments, the first indication information includes at least one of the following: a combination scheme of multiple film grain models, an importance level of each of the multiple film grain models, and a mode of adding film grains to the decoded image.

[0223] In some embodiments, the combined solution of the plurality of film grain models includes the first solution and / or the second solution.

[0224] In some embodiments, the first solution includes: generating film grain corresponding to each set of film grain parameters based on the set of film grain parameters; and adding the film grain corresponding to each set of film grain parameters to the decoded image.

[0225] In some embodiments, the second solution includes: generating film grain corresponding to each set of film grain parameters based on the set of film grain parameters; fusing the film grains corresponding to multiple sets of film grain parameters to obtain fused film grain; and adding the fused film grain to the decoded image.

[0226] In some embodiments, the acquisition module 601 is further configured to acquire second indication information before acquiring multiple sets of film grain parameters and the first indication information, where the second indication information is configured to indicate whether film grain addition based on multiple film grain models is supported; and the determination module 603 is configured to determine whether to acquire multiple sets of film grain parameters and the first indication information based on the second indication information.

[0227] In some embodiments, the second indication information is included in the video parameter set VPS, for indicating whether the video corresponding to the VPS supports adding film grains based on multiple film grain models; or, the second indication information is included in the sequence parameter set SPS, for indicating whether the image sequence corresponding to the SPS supports adding film grains based on multiple film grain models; or, the second indication information is included in the picture parameter set PPS, for indicating whether a frame of decoded image corresponding to the PPS supports adding film grains based on multiple film grain models; or, the second indication information is included in the adaptation parameter set APS, for indicating whether a frame of decoded image corresponding to the APS supports adding film grains based on multiple film grain models; or, the second indication information is included in the supplemental enhancement information SEI message, for indicating whether a frame of decoded image corresponding to the SEI message supports adding film grains based on multiple film grain models.

[0228] In some embodiments, the code stream includes a first SEI message, and the first SEI message includes at least one of the following: first indication information, second indication information, third indication information, fourth indication information, and multiple sets of film grain parameters; wherein the second indication information is used to indicate whether to support adding film grain based on multiple film grain models; the third indication information is used to indicate whether to cancel the persistence of the previous first SEI message; and the fourth indication information is used to indicate whether the current first SEI message is persistent.

[0229] In some embodiments, the codestream includes a plurality of second SEI messages; wherein each second SEI message includes a set of film grain parameters.

[0230] In some embodiments, the code stream also includes a first SEI message, and the first SEI message includes at least one of the following: first indication information, second indication information, third indication information, fourth indication information, multiple fifth indication information, and multiple sixth indication information; wherein the second indication information is used to indicate whether to support adding film grains based on multiple film grain models; the third indication information is used to indicate whether to cancel the persistence of the previous first SEI message; the fourth indication information is used to indicate whether the first SEI message of this time is persistent; each fifth indication information is used to indicate whether the second SEI message corresponding to the fifth indication information is located in the first SEI message; and each sixth indication information is used to indicate the processing order of the second SEI messages corresponding to the sixth indication information.

[0231] Figure 12 is a schematic diagram of the structure of another film grain processing device provided in an embodiment of the present disclosure. This device, applied to an encoder, can execute the film grain processing method provided in the aforementioned method embodiment. As shown in Figure 12, the film grain processing device 700 includes an acquisition module 701, a processing module 702, and a determination module 703. In other embodiments, the film grain processing device 700 also includes a sending module 704.

[0232] The acquisition module 701 is used to acquire a source image.

[0233] The processing module 702 is configured to perform denoising on the source image to obtain a denoised image.

[0234] The determination module 703 is configured to determine, based on the source image and the denoised image, multiple sets of film grain parameters and first indication information, where the first indication information is used to indicate a scheme of adding film grain to the decoded image by jointly using the multiple sets of film grain parameters.

[0235] In some embodiments, the sending module 704 is configured to send a code stream to a decoding end, where the code stream includes coding information of a source image, multiple sets of film grain parameters, and first indication information.

[0236] In some embodiments, the sending module 704 is further configured to send a code stream, multiple sets of film grain parameters, and first indication information to the decoding end, where the code stream includes encoding information of the source image.

[0237] In some embodiments, each set of film grain parameters in the plurality of sets of film grain parameters includes at least one of the following: a film grain model parameter, and a film grain processing parameter.

[0238] In some embodiments, the multiple sets of film grain parameters further include the number of film grain models.

[0239] In some embodiments, the multiple sets of film grain parameters include at least a first set of film grain parameters; the first set of film grain parameters may be determined by: determining the film grain of the source image based on a difference between the source image and the denoised image; constructing a first film grain model based on the film grain of the source image; and determining the first set of film grain parameters based on the first film grain model.

[0240] In some embodiments, the multiple sets of film grain parameters also include a second set of film grain parameters; the determination module 703 is further configured to determine residual film grain in the source image when the difference between the film grain of the source image and the film grain output by the first film grain model is greater than a threshold; construct a second film grain model based on the residual film grain in the source image; and determine a second set of film grain parameters based on the second film grain model.

[0241] In some embodiments, the first indication information includes at least one of the following: a combination scheme of multiple film grain models, an importance level of each of the multiple film grain models, and a mode of adding film grains to the decoded image.

[0242] In some embodiments, the sending module 704 is further configured to send second indication information to the decoding end, where the second indication information is configured to indicate whether adding film grains based on multiple film grain models is supported.

[0243] In some embodiments, the second indication information is included in the video parameter set VPS, for indicating whether the video corresponding to the VPS supports adding film grains based on multiple film grain models; or, the second indication information is included in the sequence parameter set SPS, for indicating whether the image sequence corresponding to the SPS supports adding film grains based on multiple film grain models; or, the second indication information is included in the picture parameter set PPS, for indicating whether a frame of decoded image corresponding to the PPS supports adding film grains based on multiple film grain models; or, the second indication information is included in the adaptation parameter set APS, for indicating whether a frame of decoded image corresponding to the APS supports adding film grains based on multiple film grain models; or, the second indication information is included in the supplemental enhancement information SEI message, for indicating whether a frame of decoded image corresponding to the SEI message supports adding film grains based on multiple film grain models.

[0244] In some embodiments, the code stream includes a first SEI message, and the first SEI message includes at least one of the following: first indication information, second indication information, third indication information, fourth indication information, and multiple sets of film grain parameters; wherein the second indication information is used to indicate whether to support adding film grain based on multiple film grain models; the third indication information is used to indicate whether to cancel the persistence of the previous first SEI message; and the fourth indication information is used to indicate whether the current first SEI message is persistent.

[0245] In some embodiments, the codestream includes a plurality of second SEI messages; wherein each second SEI message includes a set of film grain parameters.

[0246] In some embodiments, the code stream also includes a first SEI message, and the first SEI message includes at least one of the following: first indication information, second indication information, third indication information, fourth indication information, multiple fifth indication information, and multiple sixth indication information; wherein the second indication information is used to indicate whether to support adding film grains based on multiple film grain models; the third indication information is used to indicate whether to cancel the persistence of the previous first SEI message; the fourth indication information is used to indicate whether the first SEI message of this time is persistent; each fifth indication information is used to indicate whether the second SEI message corresponding to the fifth indication information is located in the first SEI message; and each sixth indication information is used to indicate the processing order of the second SEI messages corresponding to the sixth indication information.

[0247] In the case of implementing the functions of the above-mentioned integrated modules in hardware, the embodiments of the present disclosure provide a possible structure of the communication device involved in the above-mentioned embodiments. As shown in Figure 13, the communication device 800 includes: a processor 802 and a bus 804. Optionally, the communication device 800 may also include a memory 801; optionally, the communication device 800 may also include a communication interface 803.

[0248] Processor 802 may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the embodiments of this disclosure. Processor 802 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field-programmable gate array, or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the embodiments of this disclosure. Processor 802 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, or a combination of a DSP and a microprocessor.

[0249] The communication interface 803 is used to connect to other devices via a communication network, such as Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0250] The memory 801 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0251] As one possible implementation, memory 801 can exist independently of processor 802. Memory 801 can be connected to processor 802 via bus 804 to store instructions or program code. When processor 802 calls and executes the instructions or program code stored in memory 801, the film grain processing method provided in the embodiments of the present disclosure can be implemented. In another possible implementation, memory 801 can also be integrated with processor 802.

[0252] Bus 804 can be an Extended Industry Standard Architecture (EISA) bus, etc. Bus 804 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, FIG13 shows only one thick line, but this does not mean that there is only one bus or only one type of bus.

[0253] Some embodiments of the present disclosure provide a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium) having computer program instructions stored therein. When the computer program instructions are executed on a computer, the computer executes the film grain processing method as described in any of the above embodiments.

[0254] Exemplarily, the above-mentioned computer-readable storage media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes, etc.), optical disks (e.g., compact disks (CDs), digital versatile disks (DVDs), etc.), smart cards, and flash memory devices (e.g., erasable programmable read-only memories (EPROMs), cards, sticks, or key drives, etc.). The various computer-readable storage media described in the present disclosure may represent one or more devices and / or other machine-readable storage media for storing information. The term "machine-readable storage medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0255] An embodiment of the present disclosure provides a computer program product comprising instructions. When the computer program product is run on a computer, the computer is enabled to execute the film grain processing method described in any one of the above embodiments.

[0256] The above is only a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or replacements within the technical scope disclosed in the present disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A film grain processing method, characterized in that: Applied to a decoding end, the method comprises: Get the decoded image from the bitstream; Acquire multiple sets of film grain parameters and first indication information, wherein the first indication information is used to indicate a scheme of adding film grain to the decoded image by jointly using the multiple sets of film grain parameters; Film grains are added to the decoded image based on the multiple sets of film grain parameters and the first indication information.

2. The method according to claim 1, characterized in that Each group of film grain parameters in the plurality of groups of film grain parameters includes at least one of the following: a film grain model parameter and a film grain processing parameter.

3. The method according to claim 2, characterized in that The multiple groups of film grain parameters also include the number of film grain models.

4. The method according to claim 1, characterized in that: The first indication information includes at least one of the following: a combination scheme of multiple film grain models, the importance of each of the multiple film grain models, and a mode of adding the film grains into the decoded image.

5. The method according to claim 4, characterized in that The combined scheme of the plurality of film grain models comprises a first scheme and / or a second scheme.

6. The method according to claim 5, characterized in that The first solution includes: Based on each set of film grain parameters, generate film grain corresponding to the set of film grain parameters; Film grain corresponding to each set of film grain parameters is added to the decoded image.

7. The method according to claim 5, characterized in that The second solution includes: Based on each set of film grain parameters, generate film grain corresponding to the set of film grain parameters; Merging the film grains corresponding to the multiple groups of film grain parameters to obtain merged film grains; The fused film grain is added to the decoded image.

8. The method according to claim 1, characterized in that Before acquiring multiple sets of film grain parameters and first indication information, the method further includes: Acquire second indication information, where the second indication information is used to indicate whether adding film grains based on multiple film grain models is supported; Based on the second indication information, it is determined whether to obtain the plurality of groups of film grain parameters and the first indication information.

9. The method according to claim 8, characterized in that The second indication information is included in the video parameter set VPS, and is used to indicate whether the video corresponding to the VPS supports adding film grains based on the multiple film grain models; or, The second indication information is included in a sequence parameter set SPS, and is used to indicate whether the image sequence corresponding to the SPS supports adding film grains based on the multiple film grain models; or, The second indication information is included in the picture parameter set PPS, and is used to indicate whether a decoded picture frame corresponding to the PPS supports adding film grains based on the multiple film grain models; or, The second indication information is included in an adaptive parameter set APS, and is used to indicate whether a frame of decoded image corresponding to the APS supports adding film grains based on the multiple film grain models; or The second indication information is included in a supplemental enhancement information SEI message, and is used to indicate whether a frame of decoded image corresponding to the SEI message supports adding film grains based on the multiple film grain models.

10. The method according to claim 1, characterized in that The code stream includes a first SEI message, and the first SEI message includes at least one of the following: the first indication information, the second indication information, the third indication information, the fourth indication information, and the multiple groups of film grain parameters; wherein the second indication information is used to indicate whether to support adding film grains based on multiple film grain models; the third indication information is used to indicate whether to cancel the persistence of the previous first SEI message; and the fourth indication information is used to indicate whether the current first SEI message has persistence.

11. The method according to claim 1, characterized in that: The code stream includes a plurality of second SEI messages; wherein each of the second SEI messages includes a set of film grain parameters.

12. The method according to claim 11, characterized in that The code stream also includes a first SEI message, which includes at least one of the following: the first indication information, the second indication information, the third indication information, the fourth indication information, multiple fifth indication information, and multiple sixth indication information; wherein the second indication information is used to indicate whether to support adding film grains based on multiple film grain models; the third indication information is used to indicate whether to cancel the persistence of the previous first SEI message; the fourth indication information is used to indicate whether the first SEI message this time is persistent; each of the fifth indication information is used to indicate whether the second SEI message corresponding to the fifth indication information is located in the first SEI message; and each of the sixth indication information is used to indicate the processing order of the second SEI messages corresponding to the sixth indication information.

13. A film grain processing method, characterized in that: Applied to the encoding end, the method includes: Get the source image; Performing denoising processing on the source image to obtain a denoised image; Based on the source image and the denoised image, a plurality of sets of film grain parameters and first indication information are determined, wherein the first indication information is used to indicate a scheme of adding film grain to a decoded image by jointly using the plurality of sets of film grain parameters.

14. The method according to claim 13, characterized in that The method further comprises: A code stream is sent to the decoding end, where the code stream includes the encoding information of the source image, the multiple groups of film grain parameters, and the first indication information.

15. The method according to claim 13, characterized in that The method further comprises: Sending a code stream, the multiple groups of film grain parameters, and the first indication information to the decoding end, the code stream including encoding information of the source image.

16. The method according to claim 13, characterized in that Each group of film grain parameters in the plurality of groups of film grain parameters includes at least one of the following: a film grain model parameter and a film grain processing parameter.

17. The method according to claim 16, characterized in that The multiple groups of film grain parameters also include the number of film grain models.

18. The method according to claim 13, characterized in that The plurality of sets of film grain parameters include at least a first set of film grain parameters; the first set of film grain parameters may be determined by: determining film grain of the source image based on a difference between the source image and the denoised image; constructing a first film grain model based on film grain of the source image; Based on the first film grain model, the first set of film grain parameters is determined.

19. The method according to claim 18, characterized in that The plurality of sets of film grain parameters also include a second set of film grain parameters; the second set of film grain parameters is determined by: determining residual film grain in the source image if a difference between film grain in the source image and film grain output by the first film grain model is greater than a threshold; constructing a second film grain model based on residual film grain in the source image; Based on the second film grain model, the second set of film grain parameters is determined.

20. The method according to claim 13, characterized in that The first indication information includes at least one of the following: a combination scheme of multiple film grain models, the importance of each of the multiple film grain models, and a mode of adding the film grains into the decoded image.

21. The method according to claim 13, characterized in that The method further comprises: Sending second indication information to the decoding end, where the second indication information is used to indicate whether adding film grains based on multiple film grain models is supported.

22. The method according to claim 21, characterized in that The second indication information is included in the video parameter set VPS, and is used to indicate whether the video corresponding to the VPS supports adding film grains based on the multiple film grain models; or, The second indication information is included in a sequence parameter set SPS, and is used to indicate whether the image sequence corresponding to the SPS supports adding film grains based on the multiple film grain models; or, The second indication information is included in the picture parameter set PPS, and is used to indicate whether a decoded picture frame corresponding to the PPS supports adding film grains based on the multiple film grain models; or, The second indication information is included in an adaptive parameter set APS, and is used to indicate whether a frame of decoded image corresponding to the APS supports adding film grains based on the multiple film grain models; or The second indication information is included in a supplemental enhancement information SEI message, and is used to indicate whether a frame of decoded image corresponding to the SEI message supports adding film grains based on the multiple film grain models.

23. The method according to claim 14, characterized in that The code stream includes a first SEI message, and the first SEI message includes at least one of the following: the first indication information, the second indication information, the third indication information, the fourth indication information, and the multiple groups of film grain parameters; wherein the second indication information is used to indicate whether to support adding film grains based on multiple film grain models; the third indication information is used to indicate whether to cancel the persistence of the previous first SEI message; and the fourth indication information is used to indicate whether the current first SEI message has persistence.

24. The method according to claim 14, characterized in that The code stream includes a plurality of second SEI messages; wherein each of the second SEI messages includes a set of film grain parameters.

25. The method according to claim 24, characterized in that The code stream also includes a first SEI message, which includes at least one of the following: the first indication information, the second indication information, the third indication information, the fourth indication information, multiple fifth indication information, and multiple sixth indication information; wherein the second indication information is used to indicate whether to support adding film grains based on multiple film grain models; the third indication information is used to indicate whether to cancel the persistence of the previous first SEI message; the fourth indication information is used to indicate whether the first SEI message this time is persistent; each of the fifth indication information is used to indicate whether the second SEI message corresponding to the fifth indication information is located in the first SEI message; and each of the sixth indication information is used to indicate the processing order of the second SEI messages corresponding to the sixth indication information.

26. A communication device, characterized in that: include: Memory and processor; The memory is coupled to the processor; The memory is used to store instructions executable by the processor; When the processor executes the instructions, the film grain processing method according to any one of claims 1 to 25 is performed.

27. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the film grain processing method according to any one of claims 1 to 25.

Images