Video coding using refresh scheme
By using a refresh scheme in video encoding technology to encode areas in frames in intra mode, the decoding delay problem caused by long continuous frame sequences is solved, the video decoding speed and robustness are improved, and within a given data volume budget is provided. Improved video quality.
Patent Information
- Application Number
- CN202380036997.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-26
- Filing Date
- 2023-04-26
- Publication Date
- 2024-11-22
AI Technical Summary
Existing video encoding techniques may cause decoding delay problems in long continuous frame sequences, because the decoder needs to decode frames encoded in intra-mode mode to obtain the correct reference frame, resulting in too long video decoding time.
Using the refresh scheme, the set of areas in the frame is encoded in intra-mode, ensuring frequent reference frames appearing in consecutive frame sequences, thereby avoiding decoding delays, and forcing the set of areas encoded in intra-mode through the specification identification of the refresh scheme. Reduce the number of bits of the encoding mode flag and improve encoding efficiency.
Effectively avoid decoding delay problems, improves the speed and robustness of video decoding, saves bit count, and allows improved video quality within a given data volume budget.
Smart Images

Figure CN119013970A_ABST
Abstract
Description
Technical Field
[0001] One aspect of the invention relates to a method for encoding a frame sequence to which a refresh scheme is applied. For example, the method can be used in a camera device for encoding a captured frame sequence. The camera device can then store the encoded frame sequence, or can send this sequence constituting a video to another device, or the camera device can both store the encoded frame sequence and send this sequence constituting a video to another device. Other aspects of the invention relate to a video encoder, a computer program for a video encoder, a data set comprising an encoded frame sequence, and a video decoder. Background Art
[0002] There are numerous video coding techniques that apply intra-mode coding as well as inter-mode coding. A frame or at least a region in the frame encoded in intra-mode does not reference another frame. Thus, intra-mode coding produces a self-contained coded frame or coded region. In contrast, a frame or at least a region in the frame encoded in inter-mode references one or more previously encoded frames. Typically, a differential representation of the region to be encoded is generated. The differential representation corresponds to the difference between the region to be encoded on the one hand and the corresponding region in the decoded version of the previously encoded frame on the other hand. Thus, inter-mode coding produces a coded frame or coded region that is intrinsically linked to one or more previously encoded frames.
[0003] Video encoding may involve evaluating whether a frame or region in a frame should be encoded in intra-mode or in inter-mode. This evaluation may be based on one or more criteria, which typically include as a criterion the minimum amount of data produced by the encoding. If intra-mode is expected to produce the minimum amount of data, this mode may be applied. Conversely, if inter-mode is expected to produce the minimum amount of data, this mode may be applied. Typically, if there are relatively large differences between the frame or region in the frame to be encoded and a frame that can be referenced, the evaluation will best retain intra-mode. For example, this is often the case if there are changes in the scene, or a scene where many changes occur. If there are relatively small differences between the frame or region in the frame to be encoded and a frame that can be referenced, the evaluation will best retain inter-mode. For example, this is often the case in a quiet scene (similar to a still picture) or in a static background in the scene.
[0004] If intra-mode or inter-mode is applied according to the evaluation as described above, it may happen that there is a relatively long continuous sequence of frames encoded in inter-mode. Therefore, this continuous sequence of frames will not include any frames encoded in intra-mode. The same applies to regions in the continuous sequence of frames. For example, if the region is part of a static background, the region will be encoded in inter-mode throughout the continuous sequence of frames.
[0005] If there is a relatively long sequence of continuous frames encoded in inter-frame mode, the following decoding delay problem may arise in itself. A decoder that starts decoding during this relatively long sequence of continuous frames will not be able to produce correctly decoded video as long as the sequence lasts. That is, the decoder must first decode the frames that have been encoded in intra-frame mode in order to obtain a valid reference (a valid decoding starting point). Therefore, it may take a relatively long time before the decoder can produce correctly decoded video. This decoding delay problem also occurs if an area in a relatively long sequence of continuous frames is specially encoded in inter-frame mode. There will be no frames in the continuous sequence of frames whose areas are encoded in intra-frame mode.
[0006] A refresh scheme can be applied to solve the decoding delay problem described above. Basically, the refresh scheme often forces one frame or at least one region in a frame to be forced to be encoded in intra-frame mode for one of several consecutive frames. That is, although an assessment is made that the frame or region in the frame should be encoded in inter-frame mode, the frame or region in the frame is encoded in intra-frame mode. Therefore, the refresh scheme ensures that references appear frequently in a sequence of consecutive frames. Therefore, this avoids excessive delays in obtaining correctly decoded video.
[0007] Patent publication WO2020249790A1 describes certain aspects discussed above. Specifically, the patent disclosure describes a coding component that evaluates whether the following condition applies to a portion of a linearly transformed frame. When a portion of a linearly transformed frame is encoded in an inter-frame mode, an encoded version of the portion of the linearly transformed frame is obtained, which includes a smaller amount of data than the encoded version obtained by encoding the portion of the linearly transformed frame in an intra-frame mode. If the condition applies, the coding component encodes the portion of the linearly transformed frame in an inter-frame mode. If the condition is not applicable, the portion of the linearly transformed frame is encoded in an intra-frame mode. A coding mode flag is provided, which indicates whether the portion of the linearly transformed frame is encoded in an intra-frame mode or in an inter-frame mode.
[0008] Patent publication WO2020249790A1 also describes that the encoding component periodically encodes each linearly transformed frame portion in intra-frame mode. This can be considered as a refresh mechanism that allows relatively good image quality to be quickly achieved at the decoding end. The video decoder can be said to be able to quickly lock the encoded video stream provided by the video encoder containing the encoding component. Moreover, this also allows more robust video transmission. If part of the encoded video stream is lost due to, for example, an error, the affected area of the frame can be restored relatively quickly. Summary of the invention
[0009] There is a need for a video encoding technique that allows for improved video quality within a given budget for the amount of encoded video data.
[0010] According to one aspect of the present invention as defined in claim 1, there is provided a method for encoding a frame sequence to obtain an encoded frame sequence, the method comprising: -Evaluate for each region in a frame whether the region should be coded in intra mode or in inter mode; - applying a refreshing scheme that causes a set of regions in a frame to be forced to be coded in intra mode, regardless of the evaluation made; and - Provides the specification of a refresh scheme that identifies the set of regions in a frame that are forced to be coded in intra mode.
[0011] According to another aspect of the invention as defined in claim 3, there is provided a video encoder adapted to encode a sequence of frames in order to obtain an encoded sequence of frames, the video encoder comprising: -Encoding mode selection component, including: - a coding mode evaluation module adapted to evaluate, for each region in a frame, whether the region should be coded in intra mode or in inter mode; and - a refresh scheme application module adapted to cause a set of regions in a frame to be forced to be coded in intra mode, independently of the evaluation made; and - a data packing module adapted to provide a data set comprising a sequence of coded frames and comprising a specification of a refresh scheme, the specification identifying a set of regions in the frames that are mandatory to be coded in intra mode.
[0012] According to a further aspect of the invention as defined in claim 4, there is provided a computer program for a video encoder, the computer program comprising a set of instructions enabling the video encoder to perform the method defined above.
[0013] According to a further aspect of the present invention as defined in claim 5, there is provided a data set comprising a sequence of encoded frames obtained by the method as defined above; the data set comprises a specification of a refresh scheme, which identifies a set of areas in the frame that are forced to be encoded in intra-frame mode.
[0014] According to another aspect of the present invention as defined in claim 6, there is provided a video decoder suitable for decoding a sequence of encoded frames obtained by the method as defined above, the decoder being suitable for decoding areas included in a set in intra-frame mode using the specification of a refresh scheme, the specification identifying a set of areas in a frame that are compulsorily encoded in intra-frame mode.
[0015] In each of these aspects, the specification of the refresh scheme is such that the coding mode flag does not need to be used for areas in the frame that are forced to be encoded in intra-frame mode. That is, the specification identifies these areas as being decoded in intra-frame mode. The specification of the refresh scheme may require fewer bits than the number of coding flags for areas that are forced to be encoded in intra-frame mode. Therefore, since the coding mode flag requires at least one bit, the specification of the refresh scheme can be more efficient than using the coding mode flag for areas that are forced to be encoded in intra-frame mode. Therefore, bit savings can be achieved. The bits that become available can be used to represent video data in a coded form with higher precision. This is particularly true when the claimed invention is used in an environment where it is necessary to distinguish between areas that are forced to be encoded in intra-frame mode and areas that are not forced to be encoded in intra-frame mode. Therefore, the present invention allows for improved video quality within a given budget for the amount of encoded video data.
[0016] For illustrative purposes, some embodiments of the present invention are described in detail with reference to the accompanying drawings. In this description additional features will appear, some of which are defined in the dependent claims, and advantages will be apparent. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 is a schematic block diagram of a video encoder.
[0018] Figure 2 is a schematic block diagram of a quantization component in a video encoder.
[0019] Figure 3 is a schematic block diagram of a video decoder.
[0020] Figure 4 is a schematic block diagram of the entropy decoding and dequantization components in a video decoder. DETAILED DESCRIPTION
[0021] Figure 1 A video encoder 100 is schematically shown. Figure 1 A schematic block diagram of a video encoder 100 is provided. The video encoder 100 may be included, for example, in a communication device (such as a smartphone) that can capture video or store video or both. The video encoder 100 may enable the communication device to effectively transmit the video to another communication device via a communication channel that may be provided by a network.
[0022] Figure 1 The video encoder 100 shown in is an innovative and improved version of the video encoder 100 proposed in the patent disclosure WO2020249790A1 mentioned above as background technology. For convenience, the latter video encoder 100 will be referred to as the prior art video encoder hereinafter. Figure 1 The video encoder 100 shown in FIG. 1 includes a set of similar functional entities. These functional entities include a linear transformation component 101 and an encoding component 102. The linear transformation component 101 may include a reversible color transformation module 103 and a wavelet transformation module 104. The encoding component 102 may include a coding mode selection component 105, a quantization component 106, an entropy coding module 107, a data packaging module 108, a rate allocation module 109, a dequantization module 110, a reconstruction module 111 and a reference frame buffer component 112.
[0023] Basically, the video encoder 100 can encode a frame sequence 113 that can constitute a video so as to obtain an encoded frame sequence 114. Therefore, the encoded frame sequence 114 can constitute an encoded video that includes a smaller amount of data than the video in its original form. The video encoder 100 basically operates in a similar manner to a prior art video encoder. Therefore, unless otherwise stated, the description of the prior art video encoder in patent publication WO2020249790A1 also applies to Figure 1 The video encoder 100 shown in FIG.
[0024] exist Figure 1 In the video encoder 100 shown in , the coding mode selection component 105 replaces the coding mode selection module in the prior art video encoder. Nevertheless, the coding mode selection component 105 can operate in a substantially similar manner. That is, for a continuous sequence of linearly transformed frames, the coding mode selection component 105 evaluates the sub-band frame portions with specific positions in the linearly transformed frames on a frame-by-frame basis. For the relevant linearly transformed frames, the evaluation involves whether the sub-band frame portions should be encoded in intra-frame mode or in inter-frame mode. As with the prior art video decoder, the evaluation can be based on comparing the total intra-frame mode GCLI value with the total inter-frame mode GCLI value. Therefore, the coding mode selection component 105 can include a coding mode evaluation module similar to the coding mode evaluation module in the prior art video encoder. However, the coding mode selection component 105 does not need to provide a GCLI encoding value as an output.
[0025] The coding mode selection component 105 may also apply a refresh scheme, which often targets one of several consecutive linearly transformed frames, causing a subband frame portion with a specific position in the linearly transformed frame to be forced to be encoded in intra-frame mode, regardless of the evaluation made. This aspect has also been described in detail in patent disclosure WO2020249790A1. Therefore, the coding mode component may include a specific refresh scheme application module.
[0026] The coding mode selection component 105 provides an evaluation flag 115 and a refresh flag 116 associated with the sub-band frame portion. The evaluation flag 115 indicates an evaluation that has been made: whether the sub-band frame portion should be encoded in intra mode or in inter mode. The refresh flag 116 indicates whether the sub-band frame portion has been forced to be encoded in intra mode.
[0027] Figure 1 The first innovative improvement of the video encoder 100 shown in relates to a quantization component 106 that replaces the quantization module in the prior art video encoder. Like the quantization module in the prior art video encoder, the quantization component 106 can receive a sub-band frame portion or a differential representation of the sub-band frame portion depending on whether the intra mode or the inter mode has been applied, respectively. The quantization module further receives an evaluation flag 115 and a refresh flag 116 from the encoding mode selection component 105.
[0028] Figure 2 The quantization component 106 is schematically shown. The quantization component 106 includes a pre-quantization module 201 and a controlled quantization module 202. In the present embodiment, the pre-quantization module 201 includes a group of shifters 203-206, a multiplexer 207 and a GCLI extractor 208. The group of shifters 203-206 includes a first shifter 203, a second shifter 204, a third shifter 205 and a fourth shifter 206.
[0029] The basic working principle of the prequantization module 201 is as follows. The group of shifters 203-206 receives samples included in the subband frame portion or its differential representation provided by the coding mode selection component 105. The first shifter 203 does not shift the samples. The second shifter 204 shifts the samples to the right by two positions. This is equivalent to truncating the two least significant bits. The third shifter 205 and the fourth shifter 206 shift the samples to the right by three positions. This is equivalent to truncating the three least significant bits.
[0030] The multiplexer 207 selects one of the shifters depending on the evaluation flag 115 and the refresh flag 116 provided by the coding mode selection component 105. If the evaluation flag 115 indicates that the estimated inter-frame mode should be applied and the refresh flag 116 indicates that the sub-band frame portion is forced to be coded in the intra-frame mode, the multiplexer 207 selects the first shifter 203. In that case, the prequantization module 201 does not truncate samples of the sub-band frame portion.
[0031] If the evaluation flag 115 indicates that it has been evaluated that the inter mode should apply and the refresh flag 116 indicates that encoding in intra mode is not mandatory, the multiplexer 207 selects the second shifter 204. In that case, the prequantization module 201 truncates the two least significant bits of the samples of the subband frame portion.
[0032] If the evaluation flag 115 indicates that the evaluated intra mode should apply and the refresh flag 116 indicates that forced encoding in intra mode applies, the multiplexer 207 selects the third shifter 205. In that case, the prequantization module 201 truncates the three least significant bits of the samples of the subband frame portion.
[0033] If the evaluation flag 115 indicates that it has been evaluated that intra mode should apply and the refresh flag 116 indicates that encoding in intra mode is not mandatory, the multiplexer 207 selects the fourth shifter 206. In that case, the prequantization module 201 also truncates the three least significant bits of the samples of the subband frame portion.
[0034] The following table shows the operation of the pre-quantization module 201:
[0035] The table includes four rows representing four encoding conditions. Each of the four rows uniquely represents one of the four encoding conditions. The table includes a right column representing the quantization settings applied in the encoding conditions. In this example, there are three different quantization settings.
[0036] The first of the four rows represents a first coding condition. Under the first coding condition, the subband frame portion is forced to be encoded in intra mode, although it is evaluated that the inter mode should be applicable. Since it is the refresh scheme that indicates forced encoding in intra mode, the first coding condition can be called an inter-frame refresh coding condition. In this example, the inter-frame refresh coding condition is signaled together by setting both the evaluation flag 115 and the refresh flag 116 to true (1). The first quantization setting is applicable to the inter-frame refresh coding condition, according to which the subband frame portion is not quantized.
[0037] The second of the four rows represents a second coding condition. Under the second coding condition, the subband frame portion is encoded in inter-frame mode based on the evaluation made. Therefore, under this coding condition, the refresh scheme does not indicate mandatory encoding in intra-frame mode. Therefore, the second coding condition can be simply referred to as an inter-frame coding condition. In this example, the inter-frame coding condition is signaled together by setting the evaluation flag 115 and the refresh flag 116 to true (1) and false (0), respectively. The second quantization setting is applicable to the inter-frame coding condition, according to which the subband frame portion is quantized by a significant factor equal to four (4). Therefore, relative to the inter-frame refresh coding condition discussed above, the pre-quantization module 201 applies quantization with a four (4) times lower resolution under the inter-frame coding condition.
[0038] The third of the four rows represents a third coding condition. Under the third coding condition, the subband portion is encoded in intra mode according to the evaluation made, while the refresh scheme indicates mandatory encoding in intra mode. Therefore, under this coding condition, the evaluation is consistent with what the refresh scheme indicates. The third coding condition can be called an intra refresh coding condition. In this example, the intra refresh coding condition is signaled together by setting the evaluation flag 115 and the refresh flag 116 to false (0) and true (1), respectively. The third quantization setting is applicable to the intra refresh coding condition, according to which the subband frame portion is quantized by a significant factor equal to eight (8). Therefore, relative to the inter-frame refresh coding condition discussed above, the pre-quantization module 201 applies quantization with a resolution eight (8) times lower under the intra refresh coding condition, and relative to the inter-frame coding condition discussed above, the pre-quantization module 201 applies quantization with a resolution two (2) times lower under the intra refresh coding condition.
[0039] The fourth (last) of the four rows represents a fourth coding condition. Under the fourth coding condition, the subband portion is encoded in intra-frame mode according to the evaluation made, and the refresh scheme does not indicate mandatory encoding in intra-frame mode. Therefore, the fourth coding condition can be referred to as the intra-frame coding condition for short. In this example, the intra-frame coding condition is signaled together by setting both the evaluation flag 115 and the refresh flag 116 to false (0). The third quantization setting is also applicable under the intra-frame coding condition. In fact, there is no need to quantize the subband frame portion differently depending on whether the evaluation of the subband frame portion being encoded in intra-frame mode meets the indication of the refresh scheme. Therefore, any subband frame portion encoded in intra-frame mode is quantized by a significant factor equal to eight (8).
[0040] Therefore, although it is evaluated that the inter-frame mode should be applicable, under the inter-frame refresh coding condition in which the sub-band frame portion is forced to be coded in the intra-frame mode, the prequantization module 201 does not quantize the sub-band frame portion. Under other coding conditions in which coding is performed according to the evaluation made, the prequantization module 201 quantizes the sub-band frame at a given resolution. Therefore, the sub-band frame portion that is forced to be coded in the intra-frame mode and is contrary to the evaluation made is quantized at a higher resolution than the case where the sub-band frame portion is encoded according to the evaluation made, where the evaluation is one of the following two: intra-frame mode and inter-frame mode.
[0041] Under inter-frame coding conditions, the pre-quantization module 201 quantizes the sub-band frame portion with a slightly higher resolution than under intra-frame refresh coding conditions and intra-frame coding conditions. That is, if encoding is performed according to the evaluation made, the resolution when encoding in inter-frame mode is higher than the resolution when encoding in intra-frame mode. As discussed above, the evaluation will generally indicate that the inter-frame mode should be applicable to encoding static areas with less or even no changes. Accordingly, the evaluation will generally indicate that the intra-frame mode should be applicable to encoding dynamic areas with more changes. Quantizing dynamic areas with more changes at a relatively low resolution will not significantly affect the perceived image quality. On the contrary, quantizing static areas with less or even no changes at a higher resolution compared to dynamic areas with many changes can improve the perceived image quality. Therefore, quantizing areas encoded in inter-frame mode according to the evaluation made at a higher resolution will generally help improve the perceived image quality compared to areas encoded in intra-frame mode according to the evaluation made.
[0042] The controlled quantization module 202 in the quantization component 106 may operate in a manner similar to that of a controlled quantization module in a prior art video encoder. Basically, the controlled quantization module 202 quantizes the pre-quantized sub-band frame portion or its differential representation at a resolution set by the rate allocation module 109. The rate allocation module 109 applies a control scheme that ensures that the amount of data generated by encoding a frame or a portion of a frame is within the budget. Therefore, the resolution of the quantization depends on the control scheme.
[0043] The prequantization module 201 and the controlled quantization module 202 jointly perform overall quantization. Although it is evaluated that the inter-frame mode should be applicable, the prequantization module 201 allows the sub-band frame portion that is forced to be encoded in the intra-frame mode to be given priority in the overall quantization. That is, in the overall quantization, the resolution can be at least two (2) times higher under the inter-frame refresh coding condition compared to other coding conditions. Specifically, in the example discussed above, the resolution under the inter-frame refresh coding condition can be approximately four (4) times higher than under the inter-frame coding condition, and approximately eight (8) times higher than under the intra-frame refresh coding condition and the intra-frame coding condition. As explained above, this priority processing allows the prevention of visual artifacts that may occur if priority processing is not given. However, it should be noted that the embodiment presented here is only one of many possible embodiments for giving priority processing in quantization.
[0044] Figure 1The second innovative improvement of the video encoder 100 shown in relates to providing a specification 117 of a refresh scheme applied by the video encoder 100 together with the encoded frame sequence 114. In the present embodiment, the data packaging module 108 receives the specification from the encoding mode selection component 105. The data packaging module 108 embeds the specification 117 of the refresh scheme into an output video stream including the encoded frame sequence 114. The output video stream can be considered as a data set. For example, the video output stream can be stored in a memory and then in the form of a video file, which can also be considered as a data set.
[0045] In an embodiment, a specification 117 of a refresh scheme may be included in a frame header, which may also be referred to as a picture header. The specification may include several fields. One field may specify a refresh rate, which may be an integer corresponding to a periodicity expressed in a number of frames, at which refresh should be performed. Another field may define a frame index, which may also be an integer corresponding to a unique sequence number that identifies the position of a frame in a sequence of frames. The specification 117 of this refresh scheme may be supplemented with individual refresh offset values for various sub-band types. These refresh offset values may be included in a table having an entry for each sub-band type and containing refresh offset values for the sub-band type. This table may be predefined and stored in a memory or another type of circuit within the video encoder 100, or may be stored in a complementary video decoder.
[0046] For a given subband type, the aforementioned parameters, refresh rate, frame index and refresh offset value may determine whether a subband frame portion is forced to be coded in intra-frame mode. A subband frame portion may have a group index that defines the position of the subband frame portion within the subband frame. The following rules may then be applied. If {([group index] + [frame index] + [refresh offset value]) modulo [refresh rate]} is equal to 0, then the subband frame portion must be forced to be coded in intra-frame mode, otherwise it is not forced to be coded in intra-frame mode.
[0047] In another embodiment, the specification 117 of the refresh scheme may be included in a tag after the frame header. Thus, this tag, like the frame header, appears only once per frame. The tag may include several fields. One field may include a table specifying respective refresh offset values for various sub-band types. Another field may include a table specifying respective refresh rates for various sub-band types. Thus, the two tables specify the following two parameters for each sub-band type: refresh offset value and refresh rate.
[0048] For a given subband type, the aforementioned parameters, the refresh offset value, and the refresh rate can determine whether the subband frame part is forced to be encoded in intra-frame mode. The following rules can then be applied. If {([group index]-[refresh offset value]) modulo [1<<refresh rate]} is equal to 0, the subband frame part must be forced to be encoded in intra-frame mode, otherwise it is not forced to be encoded in intra-frame mode. As mentioned above, the group index defines the position of the subband frame part in the subband frame. The symbol [1<<refresh rate] means that the binary number one (1) is shifted to the left by the number of positions corresponding to the refresh rate. Therefore, if the refresh rate is a positive integer, [1<<refresh rate] is equivalent to [2 raised to the power of the refresh rate].
[0049] A general advantage of the specification 117 of the refresh scheme provided together with the coded frame sequence 114 is that there is no need to provide a coding mode flag for each coded sub-band frame portion. That is, the specification 117 of the refresh scheme indicates the sub-band frame portion that is forced to be encoded in the intra-frame mode. Specifically, the specification 117 of the refresh scheme includes a set of parameters, which are generally applied to many separately encoded sub-band frame portions and can cover multiple coded frames, and for each coded sub-band frame portion, indicate whether the coded sub-band frame portion has been forced to be encoded in the intra-frame mode. Therefore, there is no need to provide a coding mode flag for the coded sub-band frame portion that has been forced to be encoded in the intra-frame mode. This can save signaling data and leave more space for other data (especially the coded sub-band frame portion). In turn, this can help to obtain a better video quality and data bandwidth ratio.
[0050] The data packing module 108 may also include an evaluation flag 115 in the above-mentioned data set, which is associated with the sub-band frame portion to which the evaluation flag 115 belongs. Therefore, although the inter-frame mode should be applicable after evaluation, an indication is provided in conjunction with the specification 117 of the refresh scheme as to whether the sub-band frame portion is forcibly encoded in the intra-frame mode. As explained above, such an indication allows for an efficient implementation of the priority treatment mentioned above. The embodiments described here are examples of such efficient implementation. In addition, the aforementioned indication may also allow (particularly in terms of efficient use of data) the above-mentioned priority treatment to be implemented in an efficient manner with existing coding techniques.
[0051] Figure 3 The video decoder 300 is schematically shown, which is similar to the video decoder 300 described above with reference to FIG. Figure 1 and Figure 2 The video decoder 300 described above is complementary. Figure 3 A schematic block diagram of a video decoder 300 is provided. Figure 3The video decoder 300 shown in is an innovative and improved version of the video decoder 300 presented in the patent publication WO2020249790A1 mentioned above. For convenience, the latter video decoder 300 is referred to as the prior art video decoder hereinafter. Figure 3 The video decoder 300 shown in FIG. 3 comprises a set of similar functional entities, including an unpacking module 301 , an entropy decoding and dequantization component 302 , a decoding reconstruction module 303 , a decoding buffer memory 304 , and an inverse wavelet and color transformation module 305 .
[0052] Basically, the video decoder 300 can receive and decode Figure 1 The video encoder 100 shown in FIG. 1 generates an encoded frame sequence 114. The video encoder 100 operates substantially in a similar manner to a prior art video decoder. Therefore, unless otherwise indicated, the description of the prior art video decoder in patent publication WO2020249790A1 also applies to Figure 3 The video decoder 300 shown in FIG.
[0053] The video decoder 300 operates substantially as follows. The unpacking module 301 performs operations that are the inverse of the operations performed by the packing module described above. Thus, the unpacking module 301 retrieves various types of data, including coded sub-band frame portions, a specification of a refresh scheme 117, and respective evaluation flags 115 associated with respective coded sub-band frame portions. The unpacking module 301 may also derive respective refresh flags 116 associated with respective coded sub-band frame portions based on the specification of the refresh scheme 117.
[0054] Figure 4 An entropy decoding and dequantization component 302 is schematically shown which is different from the entropy decoding and dequantization modules in prior art video decoders. Figure 4 A schematic block diagram of an entropy decoding and dequantization component 302 is provided. The entropy decoding and dequantization component 302 includes an entropy decoding module 401, a main dequantization module 402, and a post dequantization module 403. The post dequantization module 403 includes a group of shifters 404-407 and a multiplexer 408. The group of shifters 404-407 includes a first shifter 404, a second shifter 405, a third shifter 406, and a fourth shifter 407.
[0055] The entropy decoding module 401 and the main dequantization module 402 may operate in a similar manner to the entropy decoding and dequantization modules in the prior art video decoder. If encoded in intra-frame mode, the main dequantization module 402 provides the respective partially dequantized sub-band frame portions; if encoded in inter-frame mode, the main dequantization module 402 provides the differential representation of the partially dequantized sub-band frame portions. For convenience, these will be collectively referred to as partially dequantized frame portions.
[0056] The post-dequantization module 403 operates substantially as follows. The group of shifters 404-407 receives samples included in the partially dequantized frame portion. The first shifter 404 does not shift the sample. The second shifter 405 shifts the sample to the left by two bits. In effect, this provides a replacement for the two least significant bits that have been truncated. The third shifter 406 and the fourth shifter 407 shift the sample to the left by three bits. In effect, this provides a replacement for the three least significant bits that have been truncated.
[0057] The multiplexer 408 selects one of the shifters depending on the evaluation flag 115 and the refresh flag 116. These flags belong to the partially dequantized frame portion whose samples are applied to the set of shifters 404-407. If the evaluation flag 115 indicates that the inter mode should have been applied to the relevant sub-band frame portion and the refresh flag 116 indicates that the sub-band frame portion has been forced to be coded in the intra mode, the multiplexer 408 selects the first shifter 404. In that case, the post-quantization module 403 does not change the samples of the partially dequantized frame portion.
[0058] If the evaluation flag 115 indicates that inter mode should have been applied to the relevant subband frame portion and the refresh flag 116 indicates no forced encoding in intra mode, the multiplexer 408 selects the second shifter 405. In that case, the post dequantization module 403 provides zeros as a replacement for the two least significant bits that have been truncated.
[0059] If the evaluation flag 115 indicates that the intra mode should have been applied and the refresh flag 116 indicates that forced encoding in the intra mode has been applied to the relevant subband frame portion, the multiplexer 408 selects the third shifter 406. In that case, the post-dequantization module 403 provides zeros as a replacement for the three least significant bits that have been truncated.
[0060] If the evaluation flag 115 indicates that intra mode should have been applied and the refresh flag 116 indicates that encoding in intra mode is not forced, the multiplexer 408 selects the fourth bit shifter 406. In that case, the post dequantization module 403 provides zeros as a replacement for the three least significant bits that have been truncated.
[0061] The following table shows the operation of the post-dequantization module 403:
[0062] Thus, in effect, the post-dequantization module 403 inverts the relevant weights associated with quantization given by the pre-quantization module 201 in the video encoder 100. In a sense, the pre-quantization module 201 in the video encoder 100 can be viewed as an equalizer that enhances areas in the frame that may cause visual artifacts if quantized in a conventional manner. Conversely, the post-dequantization module 403 in the video decoder 300 can be viewed as an inverse equalizer that neutralizes the aforementioned enhancements in dequantization.
[0063] Notes The embodiments described above with reference to the accompanying drawings are presented only by way of illustration. The present invention can be implemented in many different ways. To illustrate this, some alternative embodiments are briefly pointed out.
[0064] There are many different ways to implement the coding technology according to the present invention. In the embodiments presented above, the coding technology is applied to the video coding scheme described in patent disclosure WO2020249790A1, in which inter-frame mode coding is performed in the transform domain. In this context, the region in the frame corresponds to a subband frame portion having a specific position in each frame in a sequence of consecutive frames in the transform domain. The coding technology can also be applied to video coding schemes that perform inter-frame mode coding only in the spatial, pixel domain. In this scheme, the linear transform is performed after the inter-frame mode coding, rather than before the inter-frame mode coding. In this context, the region in the frame corresponds to a portion of the frame having a specific position (such as the upper left corner as an example) in each frame in a sequence of consecutive frames.
[0065] In general, there are many different ways to implement video encoding and video decoding according to the present invention. Any module in the presented embodiments can be implemented by electronic circuits that can be dedicated or programmable, or by a properly programmed processor, or a combination of both the electronic circuit and the processor. The computer program can define one or more operations that have been described with reference to the presented embodiments. In this regard, Figure 1-Figure 4 The schematic block diagrams of can also each be regarded as a flow chart representing at least in part such a computer program, and representing a method that a processor can perform when running the computer program. Figure 2 The prequantization module 201 in the block diagram of can be regarded as representing a prequantization step. Similarly, other modules can be regarded as representing steps of the method.
[0066] Two innovative improvements have been proposed. The first innovative improvement relates to giving priority in quantization when the inter-frame mode would have been used but the intra-frame mode is forced to be used. The second innovative improvement relates to including in the data set the specification of a refresh scheme and a sequence of encoded frames to which the refresh scheme has been applied. The second innovative improvement is the subject of the attached claims. The second innovative improvement does not require the use of the first innovative improvement, and vice versa. That is, although the two innovative improvements can be applied together in the embodiments described above, the two innovative improvements are independent of each other. In addition, the first innovative improvement can be applied to a video coding scheme different from the video coding scheme described in patent disclosure WO2020249790A1. The comments made above also apply.
[0067] The comments made above indicate that the embodiments described with reference to the accompanying drawings illustrate the invention, rather than limiting it. The invention may be implemented in a variety of alternative ways within the scope of the appended claims. All changes within the meaning and equivalent range of the claims will be included within the scope of the claims. The reference signs in any claim should not be interpreted as limiting the claim. The verb "comprise" used in a claim does not exclude the presence of other elements or other steps than those listed in the claim. The same applies to similar verbs such as "comprise" and "contain". An element mentioned in the singular in a claim relating to a product does not exclude that the product may include a plurality of such elements. Similarly, a step mentioned in the singular in a claim relating to a method does not exclude that the method may include a plurality of such steps. The mere fact that each dependent claim defines each additional feature does not exclude a combination of additional features other than those reflected in the claim.
Claims
1. A method for encoding a frame sequence to obtain an encoded frame sequence, the method comprising: -Evaluate for each region in a frame whether the region should be coded in intra mode or in inter mode; - applying a refreshing scheme throughout the encoding of said sequence of frames, said refreshing scheme causing a set of regions in said frames to be forced to be encoded in said intra-mode, independently of said evaluation made; - providing a specification of said refresh scheme, said specification identifying said set of regions in said frame which are mandatory to be coded in said intra mode; as well as - embedding said specification of said refresh scheme in a data set comprising said encoded sequence of frames.
2. The method for encoding a frame sequence according to claim 1, the method comprising: In addition to the specification of the refresh scheme, an assessment flag is embedded in the data set, the assessment flag being associated with a region in a frame, the assessment flag indicating an assessment that has been made whether the region should be encoded in the intra mode or in the inter mode.
3. The method for encoding a frame sequence according to any one of claims 1 and 2, wherein: Each frame is provided with its own specification, which is provided in association with the frame.
4. A video encoder adapted to encode a sequence of frames in order to obtain an encoded sequence of frames, the video encoder comprising: -Encoding mode selection component, including: - a coding mode evaluation module adapted to evaluate, for each region in a frame, whether the region should be coded in intra mode or in inter mode; and a refresh scheme application module adapted to apply a refresh scheme throughout the encoding of said sequence of frames, said refresh scheme causing a set of regions in said frames to be forcibly encoded in said intra-frame mode, independently of said evaluation made; and - a data packing module adapted to provide a data set comprising said sequence of coded frames and comprising a specification of said refresh scheme, said specification identifying said set of areas in said frames that are mandatory to be coded in said intra-frame mode.
5. A computer program for a video encoder, the computer program comprising a set of instructions enabling the video encoder to perform the method according to any one of claims 1 to 3.
6. A data set comprising a sequence of encoded frames obtained by the method according to any one of claims 1 to 3; the data set comprises the specification of the refresh scheme identifying the set of regions in the frames that are forced to be encoded in the intra-frame mode.
7. A video decoder adapted to decode a sequence of coded frames obtained by a method according to any one of claims 1 to 3, the decoder adapted to decode the areas included in the set in intra-frame mode using the specification of the refresh scheme, the specification identifying the set of areas in the frame that are compulsorily encoded in the intra-frame mode.
Citation Information
Patent Citations
Encoder and method of encoding a sequence of frames
WO2020249790A1
