Video coding using refresh scheme

Through the combination of frame-by-frame evaluation and refreshing scheme, the decoding delay and image quality degradation caused by inter-frame mode encoding are solved, ensuring the stability and visual effect of video quality, and reducing the emergence of visual artifacts.

CN119096538APending Publication Date: 2024-12-06INTOPIX
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380036998.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-04-26
Filing Date
2023-04-26
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Existing video encoding techniques may lead to decoding delays and image quality when using inter-mode encoding in long continuous frame sequences, especially in static scenarios, where the application of refresh schemes may lead to sudden degradation of visual artifacts and image quality.

Method used

Evaluate frame by frame whether the region in a sequence of continuous frames should be encoded in intra- or inter-frame mode, and if necessary, apply a refresh scheme to force encoding in intra-frame mode, combined with higher resolution quantization to avoid image quality degradation and Visual illusion.

Benefits of technology

It effectively avoids decoding delay and image quality degradation caused by inter-frame mode encoding, ensures the stability and visual effect of video quality, especially in static scenes, and alleviates the emergence of visual artifacts.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119096538A_ABST
    Figure CN119096538A_ABST
Patent Text Reader

Abstract

A sequence of frames is encoded in the following manner. An assessment is made on at least one region of a sequence of successive frames on a frame-by-frame basis whether the region should be encoded in intra mode or in inter mode. A refresh scheme is applied that often causes the region to be forcibly encoded in intra mode for one of several consecutive frames, irrespective of the evaluation made. A region in a sequence of consecutive frames is quantized such that for a frame, the region is quantized at a higher resolution if the region is forcibly encoded in intra mode and in contrary to the made evaluation than if the region is encoded in one of intra mode and inter mode according to the made evaluation, compared to if the region is encoded in one of intra mode and inter mode according to the made evaluation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] One aspect of the invention relates to a method for encoding a frame sequence to which a refresh scheme is applied. For example, the method can be used in a camera device for encoding a captured frame sequence. The camera device can then store the encoded frame sequence, or can send the sequence constituting a video to another device, or the camera device can both store the encoded frame sequence and send the sequence constituting a video to another device. Other aspects of the invention relate to a video encoder, a computer program for a video encoder, a data set comprising an encoded frame sequence, and a video decoder. Background Art

[0002] There are numerous video coding techniques that apply intra-mode coding as well as inter-mode coding. A frame or at least a region in the frame encoded in intra-mode does not reference another frame. Thus, intra-mode coding produces a self-contained coded frame or coded region. In contrast, a frame or at least a region in the frame encoded in inter-mode references one or more previously encoded frames. Typically, a differential representation of the region to be encoded is generated. The differential representation corresponds to the difference between the region to be encoded on the one hand and the corresponding region in the decoded version of the previously encoded frame on the other hand. Thus, inter-mode coding produces a coded frame or coded region that is intrinsically linked to one or more previously encoded frames.

[0003] Video encoding may involve evaluating whether a frame or region in a frame should be encoded in intra-mode or in inter-mode. This evaluation may be based on one or more criteria, which typically include as a criterion the minimum amount of data produced by the encoding. If intra-mode is expected to produce the minimum amount of data, this mode may be applied. Conversely, if inter-mode is expected to produce the minimum amount of data, this mode may be applied. Typically, if there are relatively large differences between the frame or region in the frame to be encoded and a frame that can be referenced, the evaluation will best retain intra-mode. For example, this is often the case if there are changes in the scene, or a scene where many changes occur. If there are relatively small differences between the frame or region in the frame to be encoded and a frame that can be referenced, the evaluation will best retain inter-mode. For example, this is often the case in a quiet scene (similar to a still picture) or in a static background in the scene.

[0004] If intra-mode or inter-mode is applied according to the evaluation as described above, it may happen that there is a relatively long continuous sequence of frames encoded in inter-mode. Therefore, this continuous sequence of frames will not include any frames encoded in intra-mode. The same applies to regions in the continuous sequence of frames. For example, if the region is part of a static background, the region will be encoded in inter-mode throughout the continuous sequence of frames.

[0005] If there is a relatively long sequence of continuous frames encoded in inter-frame mode, the following decoding delay problem may arise in itself. A decoder that starts decoding during this relatively long sequence of continuous frames will not be able to produce correctly decoded video as long as the sequence lasts. That is, the decoder must first decode the frames that have been encoded in intra-frame mode in order to obtain a valid reference (a valid decoding starting point). Therefore, it may take a relatively long time before the decoder can produce correctly decoded video. This decoding delay problem also occurs if an area in a relatively long sequence of continuous frames is specially encoded in inter-frame mode. There will be no frames in the continuous sequence of frames whose areas are encoded in intra-frame mode.

[0006] A refresh scheme can be applied to solve the decoding delay problem described above. Basically, the refresh scheme often forces one frame or at least one region in a frame to be forced to be encoded in intra-frame mode for one of several consecutive frames. That is, although an assessment is made that the frame or region in the frame should be encoded in inter-frame mode, the frame or region in the frame is encoded in intra-frame mode. Therefore, the refresh scheme ensures that references appear frequently in a sequence of consecutive frames. Therefore, this avoids excessive delays in obtaining correctly decoded video.

[0007] Patent publication WO2020249790A1 describes certain aspects discussed above. Specifically, the patent disclosure describes a coding component that evaluates whether the following condition applies to a portion of a linearly transformed frame. When a portion of a linearly transformed frame is encoded in an inter-frame mode, an encoded version of the portion of the linearly transformed frame is obtained, which includes a smaller amount of data than the encoded version obtained by encoding the portion of the linearly transformed frame in an intra-frame mode. If the condition applies, the coding component encodes the portion of the linearly transformed frame in the inter-frame mode. If the condition is not applicable, the portion of the linearly transformed frame is encoded in the intra-frame mode.

[0008] Patent publication WO2020249790A1 also describes that the encoding component periodically encodes each linearly transformed frame portion in intra-frame mode. This can be considered as a refresh mechanism that allows relatively good image quality to be quickly achieved at the decoding end. The video decoder can be said to be able to quickly lock the encoded video stream provided by the video encoder containing the encoding component. Moreover, this also allows more robust video transmission. If part of the encoded video stream is lost due to, for example, an error, the affected area of ​​the frame can be restored relatively quickly. Summary of the invention

[0009] There is a need for a video encoding technique that allows for improvements in video quality.

[0010] The present invention considers the following. A refresh scheme as described above may produce visual artifacts. Assume that there is a continuous sequence of frames to which no refresh scheme is applied when encoding in inter-frame mode. As mentioned above, the continuous sequence of frames may represent a quiet scene or even a static scene. In the continuous sequence of frames, each frame encoded in inter-frame mode may provide refinement so that the image quality gradually increases or at least remains at a relatively high level.

[0011] Now assume that a refresh scheme is applied so that the frames in the continuous frame sequence described above are forced to be encoded in intra-frame mode. This encoding of the intra-frame mode potentially generates a relatively large amount of data. Typically, a frame or an area in a frame is encoded so as to generate an amount of data within the budget of the amount of data. In order to keep the amount of data generated by encoding in the intra-frame mode within the budget, it may be necessary to quantize the frames that are forced to be encoded in the intra-frame mode at a relatively low resolution. Therefore, the decoded version of the frame may have relatively poor quality. As a result, the image quality may suddenly drop at a certain point in the scene, which may be a relatively quiet scene or an almost static scene. This may constitute a visual illusion, especially due to the unexpected nature of the quality degradation in the scene.

[0012] When a frame is encoded in intra mode based on an assessment that intra mode should indeed be applied, the aforementioned visual artifacts need not occur. Then, the frame is usually very different from one or more previous frames because, for example, there are some considerable changes in the scene, or there is a change of scene. In that case, the quantization with a relatively low resolution is usually less noticeable than in the case described above, in which a frame is forced to be encoded in intra mode, while the frame would otherwise be encoded in inter mode.

[0013] The foregoing also applies to regions in a sequence of consecutive frames. The region may be part of a static background, for example. Then, for each frame, the evaluation will indicate that the region should be encoded in inter-frame mode. In addition to one or more frames that are forced to be encoded in intra-frame mode due to the application of a refresh scheme, the inter-frame mode will actually be applied. Here again, this will usually cause the region to be quantized at a relatively low resolution for these frames in order to comply with the data budget. Therefore, a sudden drop in image quality may occur, which may constitute a local visual artifact in this case.

[0014] According to an aspect of the present invention as defined in claim 1, there is provided a method of encoding a frame sequence so as to obtain an encoded frame sequence, the method comprising: - evaluating, frame by frame, for at least one region in a sequence of consecutive frames, whether the region should be coded in intra mode or in inter mode; - applying a refreshing scheme which causes said region to be forced to be coded in said intra-mode often for one of several consecutive frames, independently of the evaluation made; and - quantizing the region in the sequence of consecutive frames so that, for a frame, the region is quantized at a higher resolution than if the region were encoded in one of intra-frame mode and inter-frame mode according to the evaluation made, when the region is forced to be encoded in the intra-frame mode and contrary to the evaluation made.

[0015] According to another aspect of the present invention as defined in claim 12, there is provided a video encoder adapted to encode a sequence of frames in order to obtain an encoded sequence of frames, the video encoder comprising: -Encoding mode application components, including: - a coding mode evaluation module adapted to evaluate, frame by frame, for at least one region in a sequence of consecutive frames, whether the region should be coded in intra mode or in inter mode; and - a refresh scheme application module adapted to frequently cause said region to be forced to be coded in said intra-mode for one of several consecutive frames, independently of the evaluation made; and - a quantization component adapted to quantize said region in said sequence of consecutive frames so that, for a frame, said region is quantized at a higher resolution than if said region were encoded in one of intra-frame mode and inter-frame mode according to the evaluation made, in the case where said region is forced to be encoded in said intra-frame mode and contrary to the evaluation made.

[0016] According to a further aspect of the invention as defined in claim 13, there is provided a computer program for a video encoder, said computer program comprising a set of instructions enabling said video encoder to perform said method as defined above.

[0017] In each of these aspects, a frame or region in a frame that would otherwise be encoded in inter-frame mode but is forced to be encoded in intra-frame mode is quantized at a relatively high resolution. In other words, such a frame or such a region is given priority in quantization. This helps to avoid that the frame or the region has a significantly lower image quality than the previous frame or the region in the previous frame. Therefore, a sudden significant drop in image quality can be avoided, which therefore avoids the aforementioned visual artifacts, or at least mitigates such artifacts. Therefore, the present invention allows for an improvement in video quality.

[0018] There are various ways to give the aforementioned priority in quantization. One basic option is to adapt the quantization control scheme based on achieving a target amount of data. Priority processing can be integrated into such a quantization control scheme as is. Another basic option is to pre-quantize a frame or a region in a frame, and the resolution of the pre-quantization depends on whether the frame or the region in the frame would originally be encoded in inter-frame mode but is forced to be encoded in intra-frame mode. Compared with other frames or other regions, the quantization resolution is higher for frames or regions in frames that are forced to be encoded in intra-frame mode instead of inter-frame mode. Frames or regions in frames that are forced to be encoded in intra-frame mode instead of inter-frame mode can also not undergo any quantization, while other frames or other regions are quantized. This selective prequantization also provides priority processing.

[0019] According to an additional aspect of the invention as defined in claim 14, there is provided a data set comprising: - a sequence of coded frames obtained by the method according to the method as defined above; - an indication as to whether a region in a frame is forced to be coded in said intra mode despite an assessment that said region should be coded in said inter mode.

[0020] According to another additional aspect of the present invention as defined in claim 15, there is provided a video decoder adapted to decode a sequence of coded frames obtained by a method according to the above definition, wherein the sequence of coded frames is comprised in a data set, the data set comprising an indication as to whether a region in a frame is forced to be coded in the intra mode despite an assessment that the region should be coded in the inter mode, the video decoder comprising: - a dequantization component adapted to dequantize said region in dependence on said indication as to whether said region is forced to be coded in said intra mode despite an assessment that said region should be coded in said inter mode.

[0021] These additional aspects allow for an efficient implementation of the prioritization mentioned above. Furthermore, these additional aspects may also allow prioritization to be implemented with existing coding techniques in an efficient manner, in particular in terms of efficient use of data. JPEG XS is an example of an existing coding technique to which it is applicable. JPEG XS is defined in the ISO document "ISO / IEC 21122-1:2022 - Information technology - JPEG XS Low delay lightweight image coding system - Part 1: Core coding system". JPEG XS includes efficient techniques for specifying data quantization that has been applied by a video encoder. A relatively small amount of data can specify the data quantization that has been applied, which enables a video decoder to correctly apply dequantization. The aforementioned indication of whether a region is forced to be encoded in intra-frame mode rather than in inter-frame mode may require an additional flag. However, the indication allows maintaining an efficient technique for specifying data quantization in JPEG-XS.

[0022] For illustrative purposes, some embodiments of the present invention are described in detail with reference to the accompanying drawings. In this description, additional features will be presented, some of which are defined in the dependent claims, and advantages will be apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a schematic block diagram of a video encoder.

[0024] Figure 2 is a schematic block diagram of a quantization component in a video encoder.

[0025] Figure 3 is a schematic block diagram of a video decoder.

[0026] Figure 4 is a schematic block diagram of the entropy decoding and dequantization components in a video decoder. DETAILED DESCRIPTION

[0027] Figure 1 A video encoder 100 is schematically shown. Figure 1 A schematic block diagram of a video encoder 100 is provided. The video encoder 100 may be included, for example, in a communication device (such as a smartphone) that can capture video or store video or both. The video encoder 100 may enable the communication device to effectively transmit the video to another communication device via a communication channel that may be provided by a network.

[0028] Figure 1The video encoder 100 shown in is an innovative and improved version of the video encoder 100 proposed in the patent publication WO2020249790A1 mentioned above as background technology. For convenience, the latter video encoder 100 will be referred to as the prior art video encoder hereinafter. Figure 1 The video encoder 100 shown in FIG. 1 includes a set of similar functional entities. These functional entities include a linear transformation component 101 and an encoding component 102. The linear transformation component 101 may include a reversible color transformation module 103 and a wavelet transformation module 104. The encoding component 102 may include a coding mode selection component 105, a quantization component 106, an entropy coding module 107, a data packaging module 108, a rate allocation module 109, a dequantization module 110, a reconstruction module 111 and a reference frame buffer component 112.

[0029] Basically, the video encoder 100 can encode a frame sequence 113 that can constitute a video so as to obtain an encoded frame sequence 114. Therefore, the encoded frame sequence 114 can constitute an encoded video that includes a smaller amount of data than the video in its original form. The video encoder 100 basically operates in a similar manner to a prior art video encoder. Therefore, unless otherwise stated, the description of the prior art video encoder in patent publication WO2020249790A1 also applies to Figure 1 The video encoder 100 shown in FIG.

[0030] exist Figure 1 In the video encoder 100 shown in , the coding mode selection component 105 replaces the coding mode selection module in the prior art video encoder. Nevertheless, the coding mode selection component 105 can operate in a similar manner in essence. That is, for a continuous sequence of linearly transformed frames, the coding mode selection component 105 evaluates the sub-band frame portion with a specific position in the linearly transformed frame on a frame-by-frame basis. For the relevant linearly transformed frame, the evaluation involves whether the sub-band frame portion should be encoded in intra-frame mode or in inter-frame mode. As with the prior art video decoder, the evaluation can be based on comparing the total intra-frame mode GCLI value with the total inter-frame mode GCLI value. Therefore, the coding mode selection component 105 can include a coding mode evaluation module similar to the coding mode evaluation module in the prior art video encoder. However, the coding mode selection component 105 does not need to provide a GCLI encoding value as an output.

[0031] The coding mode selection component 105 may also apply a refresh scheme, which often targets one of several consecutive linearly transformed frames, causing a subband frame portion with a specific position in the linearly transformed frame to be forced to be encoded in intra-frame mode, regardless of the evaluation made. This aspect has also been described in detail in patent disclosure WO2020249790A1. Therefore, the coding mode component may include a specific refresh scheme application module.

[0032] The coding mode selection component 105 provides an evaluation flag 115 and a refresh flag 116 associated with the sub-band frame portion. The evaluation flag 115 indicates an evaluation that has been made: whether the sub-band frame portion should be encoded in intra mode or in inter mode. The refresh flag 116 indicates whether the sub-band frame portion has been forced to be encoded in intra mode.

[0033] Figure 1 The first innovative improvement of the video encoder 100 shown in relates to a quantization component 106 that replaces the quantization module in the prior art video encoder. Like the quantization module in the prior art video encoder, the quantization component 106 can receive a sub-band frame portion or a differential representation of the sub-band frame portion depending on whether the intra mode or the inter mode has been applied, respectively. The quantization module further receives an evaluation flag 115 and a refresh flag 116 from the encoding mode selection component 105.

[0034] Figure 2 The quantization component 106 is schematically shown. The quantization component 106 includes a pre-quantization module 201 and a controlled quantization module 202. In the present embodiment, the pre-quantization module 201 includes a group of shifters 203-206, a multiplexer 207 and a GCLI extractor 208. The group of shifters 203-206 includes a first shifter 203, a second shifter 204, a third shifter 205 and a fourth shifter 206.

[0035] The basic working principle of the prequantization module 201 is as follows. The group of shifters 203-206 receives samples included in the subband frame portion or its differential representation provided by the coding mode selection component 105. The first shifter 203 does not shift the samples. The second shifter 204 shifts the samples to the right by two positions. This is equivalent to truncating the two least significant bits. The third shifter 205 and the fourth shifter 206 shift the samples to the right by three positions. This is equivalent to truncating the three least significant bits.

[0036] The multiplexer 207 selects one of the shifters depending on the evaluation flag 115 and the refresh flag 116 provided by the coding mode selection component 105. If the evaluation flag 115 indicates that the estimated inter-frame mode should be applied and the refresh flag 116 indicates that the sub-band frame portion is forced to be coded in the intra-frame mode, the multiplexer 207 selects the first shifter 203. In that case, the prequantization module 201 does not truncate samples of the sub-band frame portion.

[0037] If the evaluation flag 115 indicates that it has been evaluated that the inter mode should apply and the refresh flag 116 indicates that encoding in intra mode is not mandatory, the multiplexer 207 selects the second shifter 204. In that case, the prequantization module 201 truncates the two least significant bits of the samples of the subband frame portion.

[0038] If the evaluation flag 115 indicates that the evaluated intra mode should apply and the refresh flag 116 indicates that forced encoding in intra mode applies, the multiplexer 207 selects the third shifter 205. In that case, the prequantization module 201 truncates the three least significant bits of the samples of the subband frame portion.

[0039] If the evaluation flag 115 indicates that it has been evaluated that intra mode should apply and the refresh flag 116 indicates that encoding in intra mode is not mandatory, the multiplexer 207 selects the fourth shifter 206. In that case, the prequantization module 201 also truncates the three least significant bits of the samples of the subband frame portion.

[0040] The following table shows the operation of the pre-quantization module 201:

[0041] The table includes four rows representing four encoding conditions. Each of the four rows uniquely represents one of the four encoding conditions. The table includes a right column representing the quantization settings applied in the encoding conditions. In this example, there are three different quantization settings.

[0042] The first of the four rows represents a first coding condition. Under the first coding condition, the subband frame portion is forced to be encoded in intra mode despite an evaluation that the inter mode should apply. Since it is the refresh scheme that indicates forced encoding in intra mode, the first coding condition may be referred to as an inter refresh coding condition. In this example, the inter refresh coding condition is signaled together by setting both the evaluation flag 115 and the refresh flag 116 to true (1). The first quantization setting applies to the inter refresh coding condition, according to which the subband frame portion is not quantized.

[0043] The second of the four rows represents a second coding condition. Under the second coding condition, the subband frame portion is encoded in inter-frame mode based on the evaluation made. Therefore, under this coding condition, the refresh scheme does not indicate forced encoding in intra-frame mode. Therefore, the second coding condition can be simply referred to as an inter-frame coding condition. In this example, the inter-frame coding condition is signaled together by setting the evaluation flag 115 and the refresh flag 116 to true (1) and false (0), respectively. The second quantization setting is applicable to the inter-frame coding condition, according to which the subband frame portion is quantized by a significant factor equal to four (4). Therefore, relative to the inter-frame refresh coding condition discussed above, the pre-quantization module 201 applies quantization with a four (4) times lower resolution under the inter-frame coding condition.

[0044] The third of the four rows represents a third coding condition. Under the third coding condition, the subband portion is encoded in intra mode according to the evaluation made, while the refresh scheme indicates mandatory encoding in intra mode. Therefore, under this coding condition, the evaluation is consistent with what the refresh scheme indicates. The third coding condition can be called an intra refresh coding condition. In this example, the intra refresh coding condition is signaled together by setting the evaluation flag 115 and the refresh flag 116 to false (0) and true (1), respectively. The third quantization setting is applicable to the intra refresh coding condition, according to which the subband frame portion is quantized by a significant factor equal to eight (8). Therefore, relative to the inter-frame refresh coding condition discussed above, the pre-quantization module 201 applies quantization with a resolution eight (8) times lower under the intra refresh coding condition, and relative to the inter-frame coding condition discussed above, the pre-quantization module 201 applies quantization with a resolution two (2) times lower under the intra refresh coding condition.

[0045] The fourth (last) of the four rows represents a fourth coding condition. Under the fourth coding condition, the subband portion is encoded in intra-frame mode according to the evaluation made, and the refresh scheme does not indicate mandatory encoding in intra-frame mode. Therefore, the fourth coding condition can be referred to as the intra-frame coding condition for short. In this example, the intra-frame coding condition is signaled together by setting both the evaluation flag 115 and the refresh flag 116 to false (0). The third quantization setting is also applicable under the intra-frame coding condition. In fact, there is no need to quantize the subband frame portion differently depending on whether the evaluation of the subband frame portion being encoded in intra-frame mode meets the indication of the refresh scheme. Therefore, any subband frame portion encoded in intra-frame mode is quantized by a significant factor equal to eight (8).

[0046] Therefore, although it is evaluated that the inter-frame mode should be applicable, under the inter-frame refresh coding condition in which the sub-band frame portion is forced to be coded in the intra-frame mode, the prequantization module 201 does not quantize the sub-band frame portion. Under other coding conditions in which coding is performed according to the evaluation made, the prequantization module 201 quantizes the sub-band frame at a given resolution. Therefore, the sub-band frame portion that is forced to be coded in the intra-frame mode and is contrary to the evaluation made is quantized at a higher resolution than the case where the sub-band frame portion is encoded according to the evaluation made, where the evaluation is one of the following two: intra-frame mode and inter-frame mode.

[0047] Under inter-frame coding conditions, the pre-quantization module 201 quantizes the sub-band frame portion with a slightly higher resolution than that under intra-frame refresh condition coding and intra-frame coding conditions. That is, if encoding is performed according to the evaluation made, the resolution when encoding in inter-frame mode is higher than the resolution when encoding in intra-frame mode. As discussed above, the evaluation will generally indicate that the inter-frame mode should be applicable to encoding static areas with less or even no changes. Accordingly, the evaluation will generally indicate that the intra-frame mode should be applicable to encoding dynamic areas with more changes. Quantizing dynamic areas with more changes at a relatively low resolution will not significantly affect the perceived image quality. On the contrary, quantizing static areas with less or even no changes at a higher resolution compared to dynamic areas with many changes can improve the perceived image quality. Therefore, quantizing areas encoded in inter-frame mode according to the evaluation made at a higher resolution will generally help improve the perceived image quality compared to areas encoded in intra-frame mode according to the evaluation made.

[0048] The controlled quantization module 202 in the quantization component 106 may operate in a manner similar to that of a controlled quantization module in a prior art video encoder. Basically, the controlled quantization module 202 quantizes the pre-quantized sub-band frame portion or its differential representation at a resolution set by the rate allocation module 109. The rate allocation module 109 applies a control scheme that ensures that the amount of data generated by encoding a frame or a portion of a frame is within the budget. Therefore, the resolution of the quantization depends on the control scheme.

[0049] The prequantization module 201 and the controlled quantization module 202 jointly perform overall quantization. Although it is evaluated that the inter-frame mode should be applicable, the prequantization module 201 gives priority to the sub-band frame portion that is forced to be encoded in the intra-frame mode in the overall quantization. That is, in the overall quantization, the resolution can be at least two (2) times higher under the inter-frame refresh coding condition compared to other coding conditions. Specifically, in the example discussed above, the resolution under the inter-frame refresh coding condition can be approximately four (4) times higher than under the inter-frame coding condition, and approximately eight (8) times higher than under the intra-frame refresh coding condition and the intra-frame coding condition. As explained above, this priority processing allows preventing visual artifacts that may occur if priority processing is not given. However, it should be noted that the embodiment presented here is only one of many possible embodiments for giving priority processing in quantization.

[0050] Figure 1 The second innovative improvement of the video encoder 100 shown in relates to providing a specification 117 of a refresh scheme applied by the video encoder 100 together with the encoded frame sequence 114. In the present embodiment, the data packaging module 108 receives the specification from the encoding mode selection component 105. The data packaging module 108 embeds the specification 117 of the refresh scheme into an output video stream including the encoded frame sequence 114. The output video stream can be considered as a data set. For example, the video output stream can be stored in a memory and then in the form of a video file, which can also be considered as a data set.

[0051] In an embodiment, a specification 117 of a refresh scheme may be included in a frame header, which may also be referred to as a picture header. The specification may include several fields. One field may specify a refresh rate, which may be an integer corresponding to a periodicity expressed in a number of frames, at which refresh should be performed. Another field may define a frame index, which may also be an integer corresponding to a unique sequence number that identifies the position of a frame in a sequence of frames. The specification 117 of this refresh scheme may be supplemented with individual refresh offset values ​​for various sub-band types. These refresh offset values ​​may be included in a table having an entry for each sub-band type and containing refresh offset values ​​for the sub-band type. This table may be predefined and stored in a memory or another type of circuit within the video encoder 100, or may be stored in a complementary video decoder.

[0052] For a given subband type, the aforementioned parameters, refresh rate, frame index and refresh offset value may determine whether a subband frame portion is forced to be coded in intra-frame mode. A subband frame portion may have a group index that defines the position of the subband frame portion within the subband frame. The following rules may then be applied. If {([group index] + [frame index] + [refresh offset value]) modulo [refresh rate]} is equal to 0, then the subband frame portion must be forced to be coded in intra-frame mode, otherwise it is not forced to be coded in intra-frame mode.

[0053] In another embodiment, the specification 117 of the refresh scheme may be included in a tag after the frame header. Thus, this tag, like the frame header, appears only once per frame. The tag may include several fields. One field may include a table specifying respective refresh offset values ​​for various sub-band types. Another field may include a table specifying respective refresh rates for various sub-band types. Thus, the two tables specify the following two parameters for each sub-band type: refresh offset value and refresh rate.

[0054] For a given subband type, the aforementioned parameters, the refresh offset value, and the refresh rate can determine whether the subband frame part is forced to be encoded in intra-frame mode. The following rules can then be applied. If {([group index]-[refresh offset value]) modulo [1<<refresh rate]} is equal to 0, the subband frame part must be forced to be encoded in intra-frame mode, otherwise it is not forced to be encoded in intra-frame mode. As mentioned above, the group index defines the position of the subband frame part in the subband frame. The symbol [1<<refresh rate] means that the binary number one (1) is shifted to the left by the number of positions corresponding to the refresh rate. Therefore, if the refresh rate is a positive integer, [1<<refresh rate] is equivalent to [2 raised to the power of the refresh rate].

[0055] A general advantage of the specification 117 of the refresh scheme provided together with the coded frame sequence 114 is that there is no need to provide a coding mode flag for each coded sub-band frame portion. That is, the specification 117 of the refresh scheme indicates the sub-band frame portion that is forced to be encoded in the intra-frame mode. Specifically, the specification 117 of the refresh scheme includes a set of parameters, which are generally applied to many separately encoded sub-band frame portions and can cover multiple coded frames, and for each coded sub-band frame portion, indicate whether the coded sub-band frame portion has been forced to be encoded in the intra-frame mode. Therefore, there is no need to provide a coding mode flag for the coded sub-band frame portion that has been forced to be encoded in the intra-frame mode. This can save signaling data and leave more space for other data (especially the coded sub-band frame portion). In turn, this can help to obtain a better video quality and data bandwidth ratio.

[0056] The data packing module 108 may also include an evaluation flag 115 in the above-mentioned data set, which is associated with the sub-band frame portion to which the evaluation flag 115 belongs. Therefore, although the inter-frame mode should be applicable after evaluation, an indication is provided in conjunction with the specification 117 of the refresh scheme as to whether the sub-band frame portion is forcibly encoded in the intra-frame mode. As explained above, such an indication allows for an efficient implementation of the priority treatment mentioned above. The embodiments described here are examples of such efficient implementation. In addition, the aforementioned indication may also allow (particularly in terms of efficient use of data) the above-mentioned priority treatment to be implemented in an efficient manner with existing coding techniques.

[0057] Figure 3 The video decoder 300 is schematically shown, which is similar to the video decoder 300 described above with reference to FIG. Figure 1 and Figure 2 The video decoder 300 described above is complementary. Figure 3 A schematic block diagram of a video decoder 300 is provided. Figure 3 The video decoder 300 shown in is an innovative and improved version of the video decoder 300 presented in the patent publication WO2020249790A1 mentioned above. For convenience, the latter video decoder 300 is referred to as the prior art video decoder hereinafter. Figure 3 The video decoder 300 shown in FIG. 3 comprises a set of similar functional entities, including an unpacking module 301 , an entropy decoding and dequantization component 302 , a decoding reconstruction module 303 , a decoding buffer memory 304 , and an inverse wavelet and color transformation module 305 .

[0058] Basically, the video decoder 300 can receive and decode Figure 1 The video encoder 100 shown in FIG. 1 generates an encoded frame sequence 114. The video encoder 100 operates substantially in a similar manner to a prior art video decoder. Therefore, unless otherwise indicated, the description of the prior art video decoder in patent publication WO2020249790A1 also applies to Figure 3 The video decoder 300 shown in FIG.

[0059] The video decoder 300 operates substantially as follows. The unpacking module 301 performs operations that are the inverse of the operations performed by the packing module described above. Thus, the unpacking module 301 retrieves various types of data, including coded sub-band frame portions, a specification of a refresh scheme 117, and respective evaluation flags 115 associated with respective coded sub-band frame portions. The unpacking module 301 may also derive respective refresh flags 116 associated with respective coded sub-band frame portions based on the specification of the refresh scheme 117.

[0060] Figure 4An entropy decoding and dequantization component 302 is schematically shown which is different from the entropy decoding and dequantization modules in prior art video decoders. Figure 4 A schematic block diagram of an entropy decoding and dequantization component 302 is provided. The entropy decoding and dequantization component 302 includes an entropy decoding module 401, a main dequantization module 402, and a post dequantization module 403. The post dequantization module 403 includes a group of shifters 404-407 and a multiplexer 408. The group of shifters 404-407 includes a first shifter 404, a second shifter 405, a third shifter 406, and a fourth shifter 407.

[0061] The entropy decoding module 401 and the main dequantization module 402 can operate in a similar manner to the entropy decoding and dequantization modules in the prior art video decoder. If encoding is performed in intra-frame mode, the main dequantization module 402 provides each partially dequantized sub-band frame portion; if encoding is performed in inter-frame mode, the main dequantization module 402 provides a differential representation of the partially dequantized sub-band frame portion. For convenience, these are collectively referred to as partially dequantized frame portions.

[0062] The post-dequantization module 403 operates substantially as follows. The group of shifters 404-407 receives samples included in the partially dequantized frame portion. The first shifter 404 does not shift the sample. The second shifter 405 shifts the sample to the left by two bits. In effect, this provides a replacement for the two least significant bits that have been truncated. The third shifter 406 and the fourth shifter 407 shift the sample to the left by three bits. In effect, this provides a replacement for the three least significant bits that have been truncated.

[0063] The multiplexer 408 selects one of the shifters depending on the evaluation flag 115 and the refresh flag 116. These flags belong to the partially dequantized frame portion whose samples are applied to the set of shifters 404-407. If the evaluation flag 115 indicates that the inter mode should have been applied to the relevant sub-band frame portion and the refresh flag 116 indicates that the sub-band frame portion has been forced to be coded in the intra mode, the multiplexer 408 selects the first shifter 404. In that case, the post-quantization module 403 does not change the samples of the partially dequantized frame portion.

[0064] If the evaluation flag 115 indicates that inter mode should have been applied to the relevant subband frame portion and the refresh flag 116 indicates no forced encoding in intra mode, the multiplexer 408 selects the second shifter 405. In that case, the post dequantization module 403 provides zeros as a replacement for the two least significant bits that have been truncated.

[0065] If the evaluation flag 115 indicates that the intra mode should have been applied and the refresh flag 116 indicates that forced encoding in the intra mode has been applied to the relevant subband frame portion, the multiplexer 408 selects the third shifter 406. In that case, the post-dequantization module 403 provides zeros as a replacement for the three least significant bits that have been truncated.

[0066] If the evaluation flag 115 indicates that intra mode should have been applied and the refresh flag 116 indicates no forced encoding in intra mode, the multiplexer 408 selects the fourth bit shifter 407. In that case, the post dequantization module 403 provides zeros as a replacement for the three least significant bits that have been truncated.

[0067] The following table shows the operation of the post-dequantization module 403:

[0068] Thus, in effect, the post-dequantization module 403 inverts the relevant weights associated with quantization given by the pre-quantization module 201 in the video encoder 100. In a sense, the pre-quantization module 201 in the video encoder 100 can be viewed as an equalizer that enhances areas in the frame that may cause visual artifacts if quantized in a conventional manner. Conversely, the post-dequantization module 403 in the video decoder 300 can be viewed as an inverse equalizer that neutralizes the aforementioned enhancements in dequantization.

[0069] Notes The embodiments described above with reference to the accompanying drawings are presented in a schematic manner. The present invention can be implemented in many different ways. In order to illustrate this, some alternative embodiments are briefly pointed out.

[0070] There are many different ways to implement the coding technology according to the present invention. In the embodiments presented above, the coding technology is applied to the video coding scheme described in the patent disclosure WO2020249790A1, in which inter-frame mode coding is performed in the transform domain. In this context, the region in the frame corresponds to a subband frame portion having a specific position in each frame in a continuous frame sequence in the transform domain. The coding technology can also be applied to a video coding scheme in which inter-frame mode coding is performed only in the spatial, pixel domain. In this scheme, the linear transformation is performed after the inter-frame mode coding instead of before the inter-frame mode coding. In this context, the region in the frame corresponds to a portion of the frame having a specific position (such as the upper left corner, for example) in each frame in a continuous frame sequence. Therefore, the coding technology according to the present invention can be applied in such an environment, in which an evaluation is made for each region in the frame as to whether the region should be encoded in intra-frame mode or in inter-frame mode, and a refresh scheme is applied, which causes a set of regions in the frame to be forced to be encoded in intra-frame mode, regardless of the evaluation made. Optionally, a specification of a refresh scheme may be provided, which identifies a set of regions in a frame that are mandatory to be coded in intra mode.

[0071] There are many different ways to implement video encoding according to the present invention, which involves giving priority in quantization. In the embodiment presented above, priority is given by pre-quantization. In other embodiments, priority can be integrated in the quantization control scheme to meet the budget of data volume. In such an embodiment, pre-quantization is not required. In the embodiment presented above, pre-quantization is completed by bit shifting. In other embodiments, quantization techniques other than bit shifting can be used, such as quantization techniques involving rounding to the nearest quantization value.

[0072] In general, there are many different ways to implement video encoding and video decoding according to the present invention. Any module in the presented embodiments can be implemented by electronic circuits that can be dedicated or programmable, or by a properly programmed processor, or a combination of both the electronic circuit and the processor. The computer program can define one or more operations that have been described with reference to the presented embodiments. In this regard, Figure 1-Figure 4 The schematic block diagrams of can also each be regarded as a flowchart representing at least in part such a computer program and representing a method that a processor can perform when running the computer program. Figure 2 The prequantization module 201 in the block diagram of can be regarded as representing a prequantization step. Similarly, other modules can be regarded as representing steps of the method.

[0073] Two innovative improvements have been proposed. The first innovative improvement relates to giving priority in quantization when the inter-frame mode would have been used but the intra-frame mode is forced to be used. The first innovative improvement is the subject of the attached claims. The second innovative improvement relates to including in the data set the specification of a refresh scheme and a sequence of encoded frames to which the refresh scheme has been applied. The second innovative improvement does not require the use of the first innovative improvement, and vice versa. That is, although the two innovative improvements can be applied together in the embodiments described above, the two innovative improvements are independent of each other. In addition, the second innovative improvement can be applied to video coding schemes different from the video coding scheme described in patent disclosure WO2020249790A1. The comments made above also apply.

[0074] The comments made above indicate that the embodiments described with reference to the accompanying drawings illustrate the invention, rather than limiting it. The invention may be implemented in a variety of alternative ways within the scope of the appended claims. All changes within the meaning and equivalent range of the claims will be included within the scope of the claims. The reference signs in any claim should not be interpreted as limiting the claim. The verb "comprise" used in a claim does not exclude the presence of other elements or other steps than those listed in the claim. The same applies to similar verbs such as "comprise" and "contain". An element mentioned in the singular in a claim relating to a product does not exclude that the product may include a plurality of such elements. Similarly, a step mentioned in the singular in a claim relating to a method does not exclude that the method may include a plurality of such steps. The mere fact that each dependent claim defines each additional feature does not exclude a combination of additional features other than those reflected in the claim.

Claims

1. A method for encoding a frame sequence to obtain an encoded frame sequence, the method comprising: - evaluating, frame by frame, for at least one region in a sequence of consecutive frames, whether the region should be coded in intra mode or in inter mode; - applying a refreshing scheme which, for one of several consecutive frames, always causes said region to be forced to be coded in said intra-mode, independently of the evaluation made; as well as - quantizing the region in the sequence of consecutive frames so that, for a frame, the region is quantized at a higher resolution than if the region was encoded based on the assessment made, when the region is forced to be encoded in the intra-frame mode despite an assessment made that the region in the frame should be encoded in the inter-frame mode.

2. The method for encoding a frame sequence according to claim 1, wherein: In the case where the region is encoded in the inter mode in dependence on the evaluation made, the region is quantized at a higher resolution than in the case where the region is encoded in the intra mode in dependence on the evaluation made.

3. The method for encoding a frame sequence according to any one of claims 1 and 2, wherein: Quantifying the area includes: - in the case where the region is forcibly coded in the intra mode despite an assessment that the region should be coded in the inter mode, fixedly quantizing the region at a higher resolution than if the region were coded according to the assessment made; and - controllably quantizing the region at a resolution dependent on a control scheme that ensures that the amount of data generated by encoding the frame or a portion of the frame is within a budget.

4. The method for encoding a frame sequence according to claim 3, wherein: Fixedly quantifying the area includes: - in case said region is forced to be coded in said intra mode despite an assessment that said region should be coded in said inter mode, quantizing said region at a first predetermined resolution; - in case said region is encoded in said inter-mode according to the evaluation made, quantizing said region at a second predetermined resolution; and - in case said region is encoded in said intra mode according to the evaluation made, quantizing said region at a third predetermined resolution; The first predetermined resolution is higher than the second predetermined resolution and the first predetermined resolution is higher than the third predetermined resolution.

5. The method for encoding a frame sequence according to claim 4, wherein: The second predetermined resolution is higher than the third predetermined resolution.

6. The method for encoding a frame sequence according to any one of claims 4 and 5, wherein: The first predetermined resolution, the second predetermined resolution, and the third predetermined resolution have a ratio of an integer power of 2 with respect to each other.

7. The method for encoding a frame sequence according to any one of claims 3 to 6, wherein: Fixedly quantizing the region includes truncating samples belonging to the region.

8. The method for encoding a frame sequence according to claim 7, wherein: - if said region is forced to be coded in said intra mode despite an assessment that said region should be coded in said inter mode, said samples are not truncated; - if said region is encoded in said inter mode according to the evaluation made, the two least significant bits are truncated; as well as - If the region is encoded in intra mode according to the evaluation made, the three least significant bits are truncated.

9. The method for encoding a frame sequence according to any one of claims 1 to 8, the method comprising: For a frame being encoded, an indication is provided as to whether the region in the frame is forced to be encoded in the intra mode despite an assessment that the region should be encoded in the inter mode.

10. The method for encoding a frame sequence according to claim 9, wherein: The instructions include: - a flag for indicating, based on said evaluation, whether said region should be coded in said intra mode or in said inter mode; and - Specification of said refresh scheme to be applied.

11. The method for encoding a frame sequence according to claim 9, wherein: The instructions include: - a first flag for indicating, according to said evaluation, whether said region should be coded in said intra mode or in said inter mode; and - a second flag for indicating whether the region has been coded in the intra mode or in the inter mode.

12. A video encoder adapted to encode a sequence of frames so as to obtain an encoded sequence of frames, the video encoder comprising: -Encoding mode selection component, including: - a coding mode evaluation module adapted to evaluate, frame by frame, for at least one region in a sequence of consecutive frames, whether the region should be coded in intra mode or in inter mode; and a refresh scheme application module adapted to always cause, for one of several consecutive frames, said region to be forced to be coded in said intra-mode, independently of the evaluation made; and - a quantization component adapted to quantize said region in said sequence of consecutive frames such that, for a frame, said region is quantized at a higher resolution than if said region were encoded according to an assessment made, in a case where said region is forced to be encoded in said intra-frame mode despite an assessment made that said region in said frame should be encoded in said inter-frame mode.

13. A computer program for a video encoder, the computer program comprising a set of instructions enabling the video encoder to perform the method according to any one of claims 1 to 11.

14. A data set comprising: - a coded frame sequence obtained by a method according to any one of claims 1 to 11; - an indication as to whether a region in a frame is forced to be coded in said intra mode despite an assessment that said region should be coded in said inter mode.

15. A video decoder adapted to decode a coded frame sequence obtained by the method according to any one of claims 1 to 11, wherein: the sequence of coded frames being comprised in a data set, the data set comprising an indication as to whether a region in a frame is forced to be coded in the intra mode despite an assessment being made that the region should be coded in the inter mode, The video decoder comprises: - a dequantization component adapted to dequantize said region in dependence on said indication as to whether said region is forced to be coded in said intra mode despite an assessment that said region should be coded in said inter mode.