Decoding and coding based on adaptive intra-refresh mechanism

Abstract

To provide decoding and encoding methods based on an adaptive intra-frame refreshing mechanism, and to provide a related device.SOLUTION: Embodiments of the present invention belong to a field of video encoding and decoding. In the embodiments of the present invention, an adaptive intra-frame video refreshing extension ID and virtual boundary location mark information are newly added to extended data in a bit stream of the current frame. In this way, since the virtual boundary location mark information can indicate the location of a virtual boundary, and the virtual boundary is at least used for distinguishing between a refreshed area and a non-refreshed area in the current frame, when dividing the refreshed area of the current frame, an encoding terminal can arbitrarily set the virtual boundary based on actual demands, and indicate the location of the virtual boundary by means of the virtual boundary location mark information, thereby effectively improving the flexibility of area division.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] FIELD OF THE INVENTION Embodiments of the present invention relate to the field of video encoding and decoding, and in particular to a method and related device for encoding and decoding based on an adaptive intra-refresh mechanism. [Background technology]

[0002] To solve the problem of I-frames having too large a bitrate and therefore requiring a long time to decode them, the intra-refresh mechanism was accordingly developed. The basic principle of the intra-refresh mechanism is to distribute the bitrate of one I-frame across several P-frames. During the encoding process, a complete image frame is divided into N forced intra-regions based on a refresh period, and N intra-refresh frames for the image frame are sequentially encoded. Each intra-refresh frame includes one of the N forced intra-regions, and the encoding mode of the N forced intra-regions is forced intra-mode, while the encoding mode of the other regions in each intra-refresh frame is allowed to adopt inter-mode. In this way, the bitrate of each intra-refresh frame is not only reduced relative to the bitrate of the I-frame, but also the bitrate of each intra-refresh frame is relatively stable. However, the process of activating the intra-refresh mechanism based on the conventional syntax is somewhat tedious. Summary of the Invention [Problem to be solved by the invention]

[0003] The embodiments of the present invention provide a decoding and encoding method and related devices based on an adaptive intra-refresh mechanism, which can improve the flexibility of application of the division of the intra-refresh mechanism. [Means for solving the problem]

[0004] In one aspect, a decoding method based on an adaptive intra-refresh mechanism is provided, comprising: receiving a bitstream of a current frame; if extension data is present in the bitstream of the current frame and an adaptive intra-refresh video extension ID is attached to the extension data, obtaining virtual boundary position marking information attached to the extension data, wherein the virtual boundary position marking information is used to indicate a position of a virtual boundary, and the virtual boundary is used to distinguish at least a refreshed area from an unrefreshed area in the current frame; and determining whether the current frame supports adaptive intra-refresh technology based on the virtual boundary position marking information.

[0005] Based on the above method, in one possible implementation form, the method further includes a step of determining that the current frame does not support the adaptive intra-refresh technology if the adaptive intra-refresh video extension ID is not present in the extension data of the bitstream of the current frame.

[0006] Based on the above method, in one possible implementation form, the virtual boundary position marking information includes first virtual boundary position marking information and second virtual boundary position marking information, wherein the first virtual boundary position marking information is used to indicate the pixel position of the virtual boundary in the horizontal direction of the current frame, and the second virtual boundary position marking information is used to indicate the pixel position of the virtual boundary in the vertical direction of the current frame.

[0007] Based on the above method, in one possible implementation form, the first virtual boundary position marking information is the x-coordinate of the virtual boundary in the horizontal direction of the current frame, the second virtual boundary position marking information is the y-coordinate of the virtual boundary in the vertical direction of the current frame, the refreshed area is a rectangular area with its origin at the coordinate (0,0) of the upper left corner of the image, its width being the x-coordinate and its height being the y-coordinate, and the unrefreshed area is an area other than the refreshed area in the current frame.

[0008] Based on the above method, in one possible implementation form, the step of determining whether the current frame supports adaptive intra-refresh technology based on the virtual boundary position marking information includes the step of determining that the current frame supports adaptive intra-refresh technology if the value of the first virtual boundary position marking information and the value of the second virtual boundary position marking information are both greater than 0, and the step of determining that the current frame does not support adaptive intra-refresh technology if the value of the first virtual boundary position marking information and / or the value of the second virtual boundary position marking information are equal to 0.

[0009] Based on the above method, in one possible implementation form, the method further includes a step of determining that the current frame is a random access point if the current frame supports adaptive intra refresh technology and the frame previous to the current frame in decoding order does not support adaptive intra refresh technology, wherein the previous frame not supporting adaptive intra refresh technology includes that an adaptive intra refresh video extension ID is not present in the extended data of the previous frame, or that an adaptive intra refresh video extension ID is present in the extended data of the previous frame but the value of the first virtual boundary position marking information and / or the second virtual boundary position marking information of the previous frame is equal to 0.

[0010] Based on the above method, in one possible implementation form, the method further includes a step of determining that the current frame is a random access recovery point if it determines that the current frame supports adaptive intra-refresh technology, and determines that the pixel position of the virtual boundary is equal to or exceeds the right boundary of the current frame based on first virtual boundary position marking information of the current frame, and determines that the pixel position of the virtual boundary is equal to or exceeds the lower boundary of the current frame based on second virtual boundary position marking information of the current frame.

[0011] Based on the above method, in one possible implementation, the precision of the first virtual boundary position marking information is the width of one large coding unit (LCU), and the precision of the second virtual boundary position marking information is the height of one of the LCUs.

[0012] Based on the above method, in one possible implementation form, if the current frame is used for random access, before decoding the current frame, the method further includes a step of obtaining a valid sequence header and a step of decoding the current frame based on information attached to the sequence header.

[0013] In another aspect, there is provided an encoding method based on an adaptive intra-refresh mechanism, comprising the steps of: determining whether a current frame supports adaptive intra-refresh technology; and if the current frame supports adaptive intra-refresh technology, adding an adaptive intra-refresh video extension ID and virtual boundary position marking information of the current frame to extension data of the current frame, wherein the virtual boundary position marking information is used to indicate a position of a virtual boundary, and the virtual boundary is used to distinguish at least a refreshed area from an un-refreshed area in the current frame.

[0014] Based on the above method, in one possible implementation form, the virtual boundary position marking information includes first virtual boundary position marking information and second virtual boundary position marking information, wherein the first virtual boundary position marking information is used to indicate the pixel position of the virtual boundary in the horizontal direction of the current frame, and the second virtual boundary position marking information is used to indicate the pixel position of the virtual boundary in the vertical direction of the current frame.

[0015] Based on the above method, in one possible implementation form, the first virtual boundary position marking information is the x-coordinate of the virtual boundary in the horizontal direction of the current frame, the second virtual boundary position marking information is the y-coordinate of the virtual boundary in the vertical direction of the current frame, the refreshed area is a rectangular area with its origin at the coordinate (0,0) of the upper left corner of the image, its width being the x-coordinate and its height being the y-coordinate, and the unrefreshed area is an area other than the refreshed area in the current frame.

[0016] Based on the above method, in one possible implementation form, if the current frame does not support adaptive intra-refresh technology, an adaptive intra-refresh video extension ID is not added to the extended data of the current frame, or an adaptive intra-refresh video extension ID is added to the extended data of the current frame, but the value of the first virtual boundary position marking information and / or the value of the second virtual boundary position marking information added to the extended data of the current frame is set to 0.

[0017] Based on the above method, in one possible implementation, if the current frame supports adaptive intra-refresh technology, the value of the first virtual boundary position marking information added to the extended data of the current frame is greater than 0, and the value of the second virtual boundary position marking information is also greater than 0.

[0018] Based on the above method, in one possible implementation form, the method further includes a step of determining, if the current frame is a random access point, that the current frame supports adaptive intra-refresh technology and that the frame preceding the current frame in encoding order does not support adaptive intra-refresh technology.

[0019] Based on the above method, in one possible implementation form, the method further includes a step of determining that when the current frame is a random access recovery point, the current frame supports adaptive intra-refresh technology, and the pixel position of the virtual boundary indicated by the first virtual boundary position marking information is equal to or exceeds the right boundary of the current frame, and the pixel position of the virtual boundary indicated by the second virtual boundary position marking information is equal to or exceeds the lower boundary of the current frame.

[0020] Based on the above method, in one possible implementation form, the accuracy of the first virtual boundary position marking information is the width of one LCU, and the accuracy of the second virtual boundary position marking information is the height of one of the LCUs.

[0021] In another aspect, a decoding device is provided, comprising: a receiving module for receiving a bitstream of a current frame; and, if extension data is present in the bitstream of the current frame and an adaptive intra-refresh video extension ID is attached to the extension data, acquiring virtual boundary position marking information attached to the extension data, the virtual boundary position marking information being used to indicate the position of a virtual boundary, and the virtual boundary being used to distinguish at least a refreshed area and an un-refreshed area in the current frame; and a processing module for determining whether the current frame supports adaptive intra-refresh technology based on the virtual boundary position marking information.

[0022] Based on the above device, in one possible implementation form, the processing module is further used to determine that the current frame does not support the adaptive intra-refresh technology if the adaptive intra-refresh video extension ID is not present in the extension data of the bitstream of the current frame.

[0023] Based on the above device, in one possible implementation form, the virtual boundary position marking information includes first virtual boundary position marking information and second virtual boundary position marking information, wherein the first virtual boundary position marking information is used to indicate the pixel position of the virtual boundary in the horizontal direction of the current frame, and the second virtual boundary position marking information is used to indicate the pixel position of the virtual boundary in the vertical direction of the current frame.

[0024] Based on the above device, in one possible implementation form, the first virtual boundary position marking information is the x-coordinate of the virtual boundary in the horizontal direction of the current frame, the second virtual boundary position marking information is the y-coordinate of the virtual boundary in the vertical direction of the current frame, the refreshed area is a rectangular area with its origin at the coordinate (0,0) of the upper left corner of the image, its width being the x-coordinate and its height being the y-coordinate, and the unrefreshed area is an area other than the refreshed area in the current frame.

[0025] Based on the above device, in one possible implementation form, the processing module is used to determine that the current frame supports adaptive intra-refresh technology if the value of the first virtual boundary position marking information and the value of the second virtual boundary position marking information are both greater than 0, and to determine that the current frame does not support adaptive intra-refresh technology if the value of the first virtual boundary position marking information and / or the value of the second virtual boundary position marking information are equal to 0.

[0026] Based on the above device, in one possible implementation form, the processing module is further used to determine that the current frame is a random access point if the current frame supports adaptive intra refresh technology and the frame previous to the current frame in decoding order does not support adaptive intra refresh technology, wherein the previous frame not supporting adaptive intra refresh technology includes: no adaptive intra refresh video extension ID is present in the extended data of the previous frame, or an adaptive intra refresh video extension ID is present in the extended data of the previous frame but the value of the first virtual boundary position marking information and / or the second virtual boundary position marking information of the previous frame is equal to 0.

[0027] Based on the above device, in one possible implementation form, the processing module is further used to determine that the current frame is a random access recovery point if it determines that the current frame supports adaptive intra-refresh technology, and determines that the pixel position of the virtual boundary is equal to or exceeds the right boundary of the current frame based on the first virtual boundary position marking information of the current frame, and determines that the pixel position of the virtual boundary is equal to or exceeds the lower boundary of the current frame based on the second virtual boundary position marking information of the current frame.

[0028] Based on the above device, in one possible implementation form, the accuracy of the first virtual boundary position marking information is the width of one LCU, and the accuracy of the second virtual boundary position marking information is the height of one of the LCUs.

[0029] Based on the above device, in one possible implementation form, the processing module is further used to obtain one valid sequence header before decoding the current frame if the current frame is used for random access, and to decode the current frame based on information attached to the sequence header.

[0030] In another aspect, an encoding device is provided that includes an encoding module for determining whether a current frame supports adaptive intra-refresh technology, wherein if the current frame supports adaptive intra-refresh technology, the encoding module is further used to add an adaptive intra-refresh video extension ID and virtual boundary position marking information of the current frame to extension data of the current frame, the virtual boundary position marking information is used to indicate the position of a virtual boundary, and the virtual boundary is used to distinguish at least a refreshed area from an un-refreshed area in the current frame.

[0031] Based on the above device, in one possible implementation form, the virtual boundary position marking information includes first virtual boundary position marking information and second virtual boundary position marking information, wherein the first virtual boundary position marking information is used to indicate the pixel position of the virtual boundary in the horizontal direction of the current frame, and the second virtual boundary position marking information is used to indicate the pixel position of the virtual boundary in the vertical direction of the current frame.

[0032] Based on the above device, in one possible implementation form, the first virtual boundary position marking information is the x-coordinate of the virtual boundary in the horizontal direction of the current frame, the second virtual boundary position marking information is the y-coordinate of the virtual boundary in the vertical direction of the current frame, the refreshed area is a rectangular area with its origin at the coordinate (0,0) of the upper left corner of the image, its width being the x-coordinate and its height being the y-coordinate, and the unrefreshed area is an area other than the refreshed area in the current frame.

[0033] Based on the above device, in one possible implementation form, the encoding module is further used to, if the current frame does not support adaptive intra-refresh technology, not add an adaptive intra-refresh video extension ID to the extended data of the current frame, or add an adaptive intra-refresh video extension ID to the extended data of the current frame but set the value of the first virtual boundary position marking information and / or the value of the second virtual boundary position marking information added to the extended data of the current frame to 0.

[0034] Based on the above device, in one possible implementation form, the encoding module is further used to, if the current frame supports adaptive intra refresh technology, make the value of the first virtual boundary position marking information added to the extended data of the current frame greater than 0, and also make the value of the second virtual boundary position marking information greater than 0.

[0035] Based on the above device, in one possible implementation form, the encoding module is further used to determine that if the current frame is a random access point, the current frame supports adaptive intra-refresh technology and the frame before the current frame in encoding order does not support adaptive intra-refresh technology.

[0036] Based on the above device, in one possible implementation form, the encoding module is further used to determine that when the current frame is a random access recovery point, the current frame supports adaptive intra-refresh technology, and the pixel position of the virtual boundary indicated by the first virtual boundary position marking information is equal to or exceeds the right boundary of the current frame, and the pixel position of the virtual boundary indicated by the second virtual boundary position marking information is equal to or exceeds the lower boundary of the current frame.

[0037] Based on the above device, in one possible implementation form, the accuracy of the first virtual boundary position marking information is the width of one LCU, and the accuracy of the second virtual boundary position marking information is the height of one of the LCUs.

[0038] In another aspect, there is provided a decoding device including a processor and a memory for storing instructions executable by the processor, the processor being configured to perform any of the steps in the decoding method based on the adaptive intra refresh mechanism described above.

[0039] In another aspect, there is provided an encoding device including a processor and a memory for storing instructions executable by the processor, the processor configured to perform any of the steps of the encoding method based on the adaptive intra refresh mechanism described above.

[0040] In another aspect, there is provided a computer-readable storage medium storing instructions that, when executed by a processor, cause any step in the decoding method based on the adaptive intra-refresh mechanism or the encoding method based on the adaptive intra-refresh mechanism to be performed.

[0041] In another aspect, there is provided a computer program product comprising instructions which, when executed on a computer, cause the computer to perform any of the steps of the decoding method based on the adaptive intra refresh mechanism or the encoding method based on the adaptive intra refresh mechanism described above. [Effects of the Invention]

[0042] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: an adaptive intra-refresh video extension ID and virtual boundary position marking information are newly added to the extended data in the bitstream of the current frame, the virtual boundary position marking information can indicate the position of a virtual boundary, and the virtual boundary is used to distinguish at least a refreshed area from an unrefreshed area in the current frame. Therefore, when dividing the refresh area of ​​the current frame, the encoding side can arbitrarily set the virtual boundary according to actual needs, and then indicate the position of the virtual boundary through the virtual boundary position marking information, thereby improving the flexibility of refresh area division; that is, the embodiments of the present invention provide an adaptive intra-refresh mechanism that realizes adaptive division of the refresh area through extended data.

[0043] In order to more clearly explain the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings that need to be used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention, and those skilled in the art can further obtain other drawings based on these drawings without expending creative efforts. [Brief explanation of the drawings]

[0044] [Figure 1] 2 is a schematic diagram of an intra-refresh mechanism provided in an embodiment of the present invention; [Figure 2] 1 is a schematic diagram of an image sequence employing an intra-refresh mechanism provided in an embodiment of the present invention; [Figure 3] 1 is a schematic diagram of an image sequence employing an intra-refresh mechanism provided in an embodiment of the present invention; [Figure 4] 1 is a flowchart of an encoding method based on an adaptive intra refresh mechanism provided in an embodiment of the present invention; [Figure 5]FIG. 2 is a schematic diagram of the area distribution of intra-refresh frames in the adaptive intra-refresh mechanism provided in an embodiment of the present invention; [Figure 6] 1 is a flowchart of a decoding method based on an adaptive intra refresh mechanism provided in an embodiment of the present invention; [Figure 7] 1 is a schematic diagram of the structure of a decoding device provided in an embodiment of the present invention; [Figure 8] 1 is a schematic diagram of the structure of an encoding device provided in an embodiment of the present invention; [Figure 9] FIG. 2 is a schematic diagram of the structure of a terminal provided in an embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0045] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the embodiments of the present invention will be described in more detail in conjunction with the drawings to make the objectives, technical solutions and advantages of the embodiments of the present invention clearer.

[0046] Before describing the method provided in the embodiment of the present invention, the application scenario of the embodiment of the present invention will be described first.

[0047] In video encoding and decoding, an image frame referenced as a reference is called an I-frame, and an I-frame is also called a keyframe. When an I-frame is encoded, the compression rate is low, and its purpose is to enable subsequent decoding to obtain an I-frame by directly decoding based on the I-frame bitstream, thereby eliminating the need to refer to other image frames. When encoding, an image frame encoded based on the difference value between the previous image frame and the current image frame is called a P-frame. The current image frame is then obtained by decoding based on the I-frame, the previous image frame obtained by decoding, and the difference value in the bitstream, so the difference value between the previous image frame and the current image frame is added to the bitstream of the encoded P-frame. Furthermore, during encoding, an image frame that is encoded based on the difference value between the previous image frame and the current image frame and the difference value between the current image frame and the subsequent image frame is called a B frame. The current image frame is obtained by decoding a subsequent I frame, the previous image frame and the subsequent image frame obtained by decoding, and the difference value in the bitstream. Therefore, the difference value between the previous image frame and the current image frame and the difference value between the current image frame and the subsequent image frame are added to the bitstream of the encoded B frame. A sequence consisting of one encoded I frame and several B frames or several P frames is called an image sequence. This image sequence is also called an IPPP or IBBB sequence.

[0048] When the decoding side accesses the IPPP or IBBB sequence in a random access manner, the bit rate of the I frame is usually much larger than that of the P frame or B frame, so the decoding time of the I frame is longer than that of the P frame or B frame. In particular, in a weak network environment, the time required to decode the I frame is longer, which may cause video delay.

[0049] To avoid video delays caused by the long decoding time of I-frames, the industry currently provides an intra-refresh mechanism. The main idea of ​​this intra-refresh mechanism is to distribute the bitrate of an I-frame across several P-frames, each of which has a small region in which the coding mode is forced intra-mode, while the coding mode of other regions is allowed to adopt inter-mode, so that the bitrate of each P-frame is ultimately much smaller than that of the original I-frame. Furthermore, the intra-refresh mechanism requires that the forced intra-regions of different P-frames do not intersect with each other, so that after several P-frames, the entire image region can be refreshed based on the forced intra-mode.

[0050] For the convenience of the following explanation, the principle of the intra-refresh mechanism will be explained in detail below.

[0051] 1 is a schematic diagram of an intra refresh mechanism provided in an embodiment of the present invention, which distributes the bit rate of an I-frame across four P-frames, each having a region that is a forced intra region, where:

number

number

number

[0052] As shown in Figure 1, a complete image frame is divided into four forced intra-refresh regions, and a refresh period includes four intra-refresh frames, each corresponding to four frames of images from left to right in Figure 1. For a first frame image in a refresh period, the image includes one intra-refresh region and an unrefreshed region. Since the first frame image is the first frame image in the refresh period, the refreshed region in the first frame image only includes the forced intra-refresh region in the current image frame. For a second frame image in a refresh period, the image includes a refreshed region, one intra-refresh region, and an unrefreshed region in the first frame image. The refreshed region of the first frame image and the intra-refresh region included in the current image frame in the second frame image are collectively referred to as the refreshed region of the second frame image. For the third frame image within a refresh period, the image includes the refreshed region in the second frame image, one intra-refresh region, and an unrefreshed region, and the refreshed region of the second frame image in the third frame image and the intra-refresh region included in the current image frame are collectively referred to as the refreshed region of the third frame image. For the fourth frame image within a refresh period, the image includes the refreshed region in the third frame image and one intra-refresh region, and the refreshed region of the third frame image in the fourth frame image and the intra-refresh region included in the current image frame are collectively referred to as the refreshed region of the fourth frame image. Because the complete image was previously divided into four forced intra-refresh regions, there are no unrefreshed regions in the fourth frame image.

[0053] The unrefreshed area is also called a dirty area, and the refreshed area of ​​the previous frame in coding order and the forced intra area of ​​the current frame are collectively called the clean area of ​​the current frame, and the boundary between them is the boundary in Figure 1. The virtual boundary in embodiments of the present invention is also a boundary for distinguishing the clean area from the dirty area, although the boundary setting method is more flexible and may be a polygonal line boundary and is not limited to a vertical boundary. In embodiments of the present invention, for any intra-refresh frame, the virtual boundary between the refreshed area and the unrefreshed area of ​​the intra-refresh frame is the boundary between the refreshed area and the unrefreshed area, and this boundary is also the boundary between the clean area and the dirty area of ​​the intra-refresh frame.

[0054] FIG. 2 is a schematic diagram of an image sequence employing an intra-refresh mechanism provided in an embodiment of the present invention. As shown in FIG. 2, four frames, I2 (I frame), P8 (P frame), P9 (P frame), and P10 (P frame), in an original image sequence (abbreviated as "original sequence" in FIG. 2) are replaced with four intra-refresh frames marked with X1, X2, X3, and X4 to obtain an image sequence based on the intra-refresh mechanism (abbreviated as "intra-refresh sequence" in FIG. 2). Each of these four intra-refresh frames has one forced intra-region, and the coding mode of the other regions is allowed to adopt inter mode. Therefore, these four intra-refresh frames can be regarded as four "large P frames" whose bitrate is approximately 1 / 4 of the bitrate of the original I2 frame. These four intra-refresh frames may also be abbreviated as "IR frames."

[0055] In the image sequence shown in Figure 2, the random access method assumes that the I2 frame in the original image sequence is the random access point, and frames after the I2 frame can be decoded independently. Therefore, the P8 frame can only reference the I2 frame, not the P7 frame. However, in an image sequence based on the intra-refresh mechanism (abbreviated as an intra-refresh sequence in Figure 2), the X1 frame is the random access point, and the inter-coded area in the unrefreshed area of ​​the X1 frame can reference the P7 frame, but may not be accurately decoded (because the P7 frame was not acquired during random access). Similarly, the unrefreshed areas of the X2 and X3 frames may not be accurately decoded. When the X1 to X4 frames are all refreshed, the inter area in the refreshed area of ​​the X4 frame is decoded based on the bitstream of the forced intra areas of the X1 to X3, and the forced intra area in the refreshed area of ​​the X4 frame is obtained based on the forced intra decoding mode. Therefore, the X4 frame without an unrefreshed area can be completely and accurately decoded. Therefore, the X1, X2, X3, and X4 frames in FIG. 2 are one refresh cycle, and since only the last frame in a refresh cycle can be completely and correctly decoded, this frame is called the Recovery Point.

[0056] In the case of random access, only the image frames at and after the recovery point are displayed to the user, but other images in the refresh cycle where the random access point is located may not be displayed because they may not be able to be decoded correctly. In the case of non-random access, all image frames are displayed to the user because they can be completely and correctly decoded.

[0057] In addition, in order to ensure the decoding function of a bitstream coded based on the intra refresh mechanism, the current intra refresh mechanism imposes the following restrictions on the intra mode / inter mode and loop filter in the decoding mode.

[0058] In condition 1, for any intra-refresh frame, a block in a refreshed area of ​​the intra-refresh frame can only be decoded by referring to a refreshed area of ​​another intra-refresh frame in the same refresh cycle, and cannot be decoded by referring to an unrefreshed area of ​​another intra-refresh frame; a co-located block in a reference frame of a current block in a TMVP (temporal motion vector prediction) mechanism must not be in an unrefreshed area of ​​the reference frame, where a co-located block is an image block that is located at the same position as the current block in the reference frame, that is, the area indicated by the MV (motion vector) information of the image block of the current image frame must not be in an unrefreshed area of ​​the reference frame. Note that, in the embodiments of the present invention, the current block and the current image block are the same concept, and for convenience of explanation, the current image block may be abbreviated as the current block.

[0059] In condition 2, for any intra-refresh frame, there is no reference restriction when decoding a block in an unrefreshed area in the intra-refresh frame.

[0060] In condition 3, for an image frame in an image sequence that is not an intra-refresh frame, the blocks of the image frame are allowed to be decoded by referring to the refreshed area of ​​an intra-refresh frame in the random access period in which they reside, but cannot be decoded by referring to the unrefreshed area of ​​an intra-refresh frame in the random access period in which they reside, and the co-located blocks in the reference frame of the current block in the TMVP mechanism must not be in the unrefreshed area of ​​the reference frame.

[0061] In condition 4, for any intra-refresh frame, the loop filter cannot cross the boundary between the refreshed and unrefreshed regions of that intra-refresh frame, i.e., it does not perform loop filter operations on the virtual boundary between the refreshed and unrefreshed regions.

[0062] The Gradual Decoding Refresh (GDR) technology in the current Versatile Video Coding (VVC) standard proposes an intra-refresh mechanism, the specific contents of which are as follows:

[0063] First, the refreshed region and the unrefreshed region are divided into a vertical division scheme. In the vertical division scheme, the virtual boundary between the refreshed region and the unrefreshed region is a boundary line parallel to the Y axis of the image coordinate system. The division boundary is aligned with the edge of the smallest CU (coding unit) (the minimum CU is 8 pixels). That is, the width of the refreshed region is an integer multiple of the width of the smallest CU, and the height of the refreshed region is the image height of the intra-refresh frame.

[0064] Next, the forced inter regions in different intra-refresh frames are divided into equal intervals, and the width of the forced inter region is defined as the total width of the image frame divided by the refresh period. The refresh period is the number of intra-refresh frames required to correctly decode and obtain a complete image frame. For example, the refresh period in FIG. 3 is 6. Here, the refresh period can be set on the encoding side. For example, the encoding side can set the refresh period using the image header parameter recovery_poc_cnt in the bitstream of the intra-refresh frame. For the image sequence shown in FIG. 3, recovery_poc_cnt=6 indicates that the refresh period in which the intra-refresh frame is located is 6. Furthermore, the refresh order does not need to be explicitly indicated in the bitstream; it defaults to left-to-right order. The decoding side decodes each intra-refresh frame in left-to-right refresh order by default.

[0065] Furthermore, for any image frame, the encoding side can further mark whether the current image frame is an intra-refresh frame based on the image header parameter gdr_pic_flag in the bitstream of the image frame. For example, gdr_pic_flag=1 indicates that the current image frame is an intra-refresh frame, and gdr_pic_flag=0 indicates that the current image frame is not an intra-refresh frame.

[0066] Furthermore, the intra-refresh mechanism in the VCC standard similarly includes the four restrictive conditions, Condition 1 to Condition 4, but the description thereof will be omitted here.

[0067] The forced intra-area in the intra-refresh mechanism of the VCC standard is a vertical stripe area with equal intervals, and it is not flexible because it cannot be divided adaptively according to the image content. Furthermore, the intra-refresh mechanism of the VCC standard does not allow a loop filter that crosses the virtual boundary between the refreshed and unrefreshed areas, which results in poor image quality on the virtual boundary.

[0068] In view of the problems encountered with the intra refresh mechanism in the VCC standard, an embodiment of the present invention provides an encoding / decoding method based on an adaptive intra refresh mechanism, which provides a more flexible forced intra region segmentation method, allowing the segmentation method to be applied at the encoding side. The method provided in the embodiment of the present invention can adaptively segment refreshed and unrefreshed regions according to image content, improving the flexibility of forced intra region segmentation. Furthermore, the virtual boundary between refreshed and unrefreshed regions can be made to coincide as closely as possible with the object boundary in the image, thereby avoiding the impact on image quality after decoding caused by the virtual boundary being unable to be loop filtered.

[0069] Since the method provided in the embodiment of the present invention can adaptively divide the refreshed area and the unrefreshed area according to the image content, the intra-refresh frame in the embodiment of the present invention may also be referred to as an adaptive intra-refresh frame, and for the convenience of the following description, the adaptive intra-refresh frame in the embodiment of the present invention will be abbreviated as an intra-refresh frame or a refresh frame.

[0070] Hereinafter, the encoding / decoding method based on the adaptive intra refresh mechanism provided in the embodiment of the present invention will be explained in detail.

[0071] 4 is a coding method based on an adaptive intra refresh mechanism provided in an embodiment of the present invention. As shown in FIG. 4, the method includes the following steps 401 and 402:

[0072] In step 401, the encoding side determines whether the current frame supports the adaptive intra-refresh technique.

[0073] During the process of a user accessing a video, there are usually random access scenes, in which the decoder does not obtain the bitstream before the random access point and needs to decode based on the bitstream after the random access point. In such scenes, in order to enable accurate decoding by the decoder, it is necessary for the bitstream after the random access point to support adaptive intra refresh technology.

[0074] Based on the above scenario, the encoding side may determine whether the current frame supports the adaptive intra refresh technology in the following manner: If the current frame is a random access point, it is determined that the current frame supports the adaptive intra refresh technology, but the frame preceding the current frame in encoding order does not support the adaptive intra refresh technology; If the current frame is a recovery point of random access, it is determined that the current frame supports the adaptive intra refresh technology; If the current frame is an image of any frame between a random access point and a recovery point of random point access, it is determined that the current frame supports the adaptive intra refresh technology.

[0075] If the current frame supports adaptive intra-refresh technology, the encoding side needs to determine the refreshed and unrefreshed areas of the current frame, so that when subsequently encoding the bitstream of the current frame, it can encode virtual boundary position marking information into the bitstream, which indicates the position of the virtual boundary that distinguishes the refreshed and unrefreshed areas of the current frame.

[0076] For ease of understanding, the virtual boundaries provided in the embodiments of the present invention will now be interpreted and explained.

[0077] The virtual boundary may be a boundary line of a refreshed region in the current frame, a boundary line of an unrefreshed region in the current frame, or a dividing line between the refreshed region and the unrefreshed region obtained by dividing the frame. These three examples of virtual boundaries are essentially the same.

[0078] In the following embodiments, the virtual boundary is described as a dividing boundary line between a refreshed region and an unrefreshed region obtained by division. That is, the virtual boundary position marking information may indicate the position of the virtual boundary between the refreshed region and the unrefreshed region in the current frame, and the position of the virtual boundary may specifically include the position of the virtual boundary in the horizontal direction of the current frame and the position of the virtual boundary in the vertical direction of the current frame.

[0079] It should be noted that the virtual boundary in the adaptive intra refresh means provided in the embodiment of the present invention is not limited to the above interpretation, and the virtual boundary may be any one of the three virtual boundary examples described above.

[0080] In step 402, if the current frame supports adaptive intra-refresh technology, the encoding side adds an adaptive intra-refresh video extension ID and virtual boundary position marking information of the current frame to the extension data of the current frame, where the virtual boundary position marking information is used to indicate the position of the virtual boundary, and the virtual boundary is used to distinguish at least the refreshed area from the unrefreshed area in the current frame.

[0081] Note that extension data is a portion of the data between the picture header and the picture data in the bitstream of the current coded frame. In some standards, extension data is also called SEI (supplemental enhancement information).

[0082] Currently, multiple video extension IDs can be indicated in the extension data, and each video extension ID can subsequently indicate several decoding parameters. Therefore, in an embodiment of the present invention, a new video extension identifier may be extended in the extension data, and the extended video extension identifier may be called an adaptive intra-refresh video extension ID. The adaptive intra-refresh video extension ID is used to inform the decoding side that the extension data further includes parameters related to the position of a virtual boundary (i.e., virtual boundary position marking information), so that the decoding side can then determine whether the current frame supports the adaptive intra-refresh mechanism and further decode the bitstream of the current frame based on the virtual boundary position marking information.

[0083] Table 1 is a summary table of syntax elements of extension data provided in an embodiment of the present invention. As shown in Table 1, the extended adaptive intra-refresh video extension ID in this embodiment of the present invention is "1110". In this scene, the extension data further includes an "air_parameters_extension" field for indicating parameters related to the position of the virtual boundary. [Table 1]

[0084] It should be noted that the above Table 1 is merely a selectable example of syntax elements of extension data provided in the embodiment of the present invention, and the embodiment of the present invention does not limit the specific expression format of these syntax elements. Furthermore, the interpretation of other syntax elements in Table 1 can refer to related standards, and the embodiment of the present invention does not limit this.

[0085] In one possible embodiment, the virtual boundary position marking information may include first virtual boundary position marking information and second virtual boundary position marking information, where the first virtual boundary position marking information is used to indicate a pixel position of the virtual boundary in the horizontal direction of the current frame, and the second virtual boundary position marking information is used to indicate a pixel position of the virtual boundary in the vertical direction of the current frame.

[0086] In this case, the syntax element air_parameters_extension() in Table 1 above may include first virtual boundary position marking information and second virtual boundary position marking information, as shown in Table 2. Table 2 is a summary table of another syntax element provided in an embodiment of the present invention, and as shown in Table 2, air_parameters_extension() includes extension_id (used to identify the adaptive intra-refresh video extension ID described above), air_bound_x (used to identify the first virtual boundary position marking information), and air_bound_y (used to identify the second virtual boundary position marking information). [Table 2]

[0087] It should be noted that the above examples of virtual boundary position marking information are merely enumerated and do not limit the function of the virtual boundary position marking information provided in the embodiments of the present invention. Marking information that can indicate the position of the virtual boundary between the refreshed area and the unrefreshed area in the horizontal direction of the current frame, and the position of the virtual boundary in the vertical direction of the current frame, are all within the scope of the virtual boundary position marking information provided in the embodiments of the present invention.

[0088] After determining whether the current frame supports the adaptive intra refresh technology, the bitstream of the current frame can be coded. Specifically, if the current frame does not support the adaptive intra refresh technology, an adaptive intra refresh video extension ID is not added to the extended data of the current frame, or an adaptive intra refresh video extension ID is added to the extended data of the current frame, but the value of the first virtual boundary position marking information or the value of the second virtual boundary position marking information added to the extended data of the current frame is set to 0.

[0089] Accordingly, if the current frame supports adaptive intra-refresh technology, an adaptive intra-refresh video extension ID and virtual boundary position marking information of the current frame are added to the extended data of the current frame, and the value of the first virtual boundary position marking information added to the extended data of the current frame is greater than 0, and the value of the second virtual boundary position marking information is also greater than 0.

[0090] For example, the first virtual boundary position marking information is the x-coordinate of the virtual boundary in the horizontal direction of the current frame, and the second virtual boundary position marking information is the y-coordinate of the virtual boundary in the vertical direction of the current frame. In such a scene, the refreshed area in the current frame is a rectangular area whose origin is the coordinate (0,0) of the upper left corner of the image, whose width is the x-coordinate and whose height is the y-coordinate, and the unrefreshed area in the current frame is an area other than the refreshed area in the current frame.

[0091] Also, for example, the value of the first virtual boundary position marking information may indicate the width of the virtual boundary in the horizontal direction of the current frame, which corresponds to the number of LCUs, and the value of the second virtual boundary position marking information may indicate the height of the virtual boundary in the vertical direction of the current frame, which corresponds to the number of LCUs.

[0092] In this case, air_bound_x in Table 2 indicates the x-coordinate of the virtual boundary between the refreshed and unrefreshed regions in the image, where the x-coordinate is in units of the width of the LCU. Air_bound_y in Table 2 indicates the y-coordinate of the virtual boundary between the refreshed and unrefreshed regions in the image, where the y-coordinate is in units of the height of the LCU.

[0093] For example, if the value of the first virtual boundary position marking information is 2 and the value of the second virtual boundary position marking information is 3, the currently refreshed area represents a rectangular area with the coordinate (0,0) of the upper left corner of the image as the origin, the width of twice the LCU as the x coordinate, and the height of three times the LCU as the y coordinate.

[0094] In such a scene, the precision of the first virtual boundary position marking information is the width of one LCU, and the precision of the second virtual boundary position marking information is the height of one LCU.

[0095] The above are optional examples for illustrative purposes only, and do not limit the scope of the boundary accuracy of the embodiments of the present invention. Here, the boundary accuracy includes the width accuracy and height accuracy of the refreshed area. Here, the width accuracy of the refreshed area is the minimum width of the refreshed area, and the actual width of the refreshed area is typically set to an integer multiple of the width accuracy. The height accuracy of the refreshed area is the minimum height of the refreshed area, and the actual height of the refreshed area is typically set to an integer multiple of the height accuracy.

[0096] 5 is a schematic diagram of an area distribution of an intra-refresh frame in the adaptive intra-refresh mechanism provided in an embodiment of the present invention. As shown in FIG. 5, the refreshed area of ​​an intra-refresh frame is a rectangular area with its upper left corner at coordinates (0,0), its width at x-coordinate AirBoundX, and its height at y-coordinate AirBoundY. The unrefreshed area of ​​the intra-refresh frame is the area other than the refreshed area in the intra-refresh frame.

[0097] That is, the intra-refresh frame in Figure 5 includes a refreshed region and an unrefreshed region, where the refreshed region may be decoded using only the refreshed region of the current image or another image, and the unrefreshed region may be decoded using either the current image or another image.

[0098] In Figure 5, PictureWidthInLcu indicates the value obtained by dividing the image width by the width of the LCU, and PictureHeightInLcu indicates the value obtained by dividing the image height by the height of the LCU. As shown in Figure 5, the value of AirBoundX should be less than or equal to PictureWidthInLcu. The value of AirBoundY should be less than or equal to PictureHeightInLcu.

[0099] Furthermore, the above AirBoundX and AirBoundY are two variables set for the first virtual boundary position marking information and the second virtual boundary position marking information in the syntax during the decoding process. These two variables are still used to indicate the x and y coordinates of the refreshed area in the image. Specifically, the values ​​of these two variables will be described in detail later on the decoding side, and will not be described in detail here.

[0100] Furthermore, if the current frame is a recovery point of random access, the encoding side determines that the current frame supports adaptive intra-refresh technology, and the pixel position of the virtual boundary indicated by the first virtual boundary position marking information is equal to or exceeds the right boundary of the current frame, and the pixel position of the virtual boundary indicated by the second virtual boundary position marking information is equal to or exceeds the bottom boundary of the current frame, so that the decoding side can then determine whether the current frame is a recovery point of random access based on the first virtual boundary position marking information and the second virtual boundary position marking information.

[0101] As described above, an adaptive intra-refresh video extension ID and virtual boundary position marking information are newly added to the extended data in the bitstream of the current frame, and the virtual boundary position marking information can indicate the position of the virtual boundary, and the virtual boundary is used to distinguish at least the refreshed area and the unrefreshed area in the current frame. Therefore, when dividing the refresh area of ​​the current frame, the encoding side can arbitrarily set the virtual boundary according to actual needs, and then indicate the position of the virtual boundary through the virtual boundary position marking information, thereby improving the flexibility of the refresh area division. That is, an embodiment of the present invention provides an adaptive intra-refresh mechanism that realizes adaptive division of the refresh area through extended data.

[0102] Hereinafter, the decoding method based on the intra refresh mechanism provided in the embodiment of the present invention will be explained in detail with reference to the embodiment shown in Fig. 6. As shown in Fig. 6, the decoding method includes steps 601 to 603.

[0103] In step 601, the bitstream of the current frame is received.

[0104] After encoding using the method shown in Figure 4, the encoding side can send the bit stream of each image to the decoding side, and the decoding side decodes the image using the embodiment shown in Figure 6.

[0105] In step 602, if extended data exists in the bitstream of the current frame and an adaptive intra-refresh video extended ID is attached to the extended data, virtual boundary position marking information attached to the extended data is obtained, and the virtual boundary position marking information is used to indicate the position of the virtual boundary, and the virtual boundary is used to distinguish between refreshed and unrefreshed areas in at least the current frame.

[0106] For the related interpretation of the adaptive intra-refresh video extension ID and the virtual boundary position marking information, reference can be made to step 401 in the encoding embodiment, and the description thereof will be omitted here.

[0107] Also, as can be seen from the embodiment shown in FIG. 4, if the adaptive intra-refresh video extension ID is not present in the extension data of the bitstream of the current frame, it is determined that the current frame does not support the adaptive intra-refresh technology.

[0108] For example, for the syntax elements shown in Table 2, if the video extension ID "1110" is not found in the extension data of the bitstream of the current frame, it is determined that the current frame does not support adaptive intra-refresh technology. In this case, the bitstream of the current frame can be decoded using another decoding method, and the embodiment of the present invention does not limit the decoding process when adaptive intra-refresh technology is not supported.

[0109] In step 603, it is determined whether the current frame supports adaptive intra-refresh technology based on the virtual boundary position marking information.

[0110] 4, if the extension data includes an adaptive intra-refresh video extension ID, it does not necessarily mean that the current frame supports the adaptive intra-refresh technology. Therefore, after obtaining the virtual boundary position marking information in step 602, it is necessary to further determine whether the current frame supports the adaptive intra-refresh technology based on the virtual boundary position marking information.

[0111] 4, the virtual boundary position marking information includes first virtual boundary position marking information and second virtual boundary position marking information, where the first virtual boundary position marking information is used to indicate the pixel position of the virtual boundary in the horizontal direction of the current frame, and the second virtual boundary position marking information is used to indicate the pixel position of the virtual boundary in the vertical direction of the current frame.

[0112] In such a scene, the implementation process of step 603 is as follows: if the value of the first virtual boundary position marking information and the value of the second virtual boundary position marking information are both greater than 0, determine that the current frame supports the adaptive intra refresh technology; accordingly, if the value of the first virtual boundary position marking information or the value of the second virtual boundary position marking information is equal to 0, determine that the current frame does not support the adaptive intra refresh technology.

[0113] For example, for the syntax elements shown in Tables 1 and 2, when decoding the extension data, it is determined whether the adaptive intra-refresh parameters air_bound_x and air_bound_y are present in the bitstream. If these two parameters are present, the value of AirBoundX of the current frame in Figure 5 is set equal to the value of air_bound_x, and the value of AirBoundY of the current frame is set equal to the value of air_bound_y. If these two parameters are not present, the value of AirBoundX of the current frame is set to 0, and the value of AirBoundY of the current frame is set to 0.

[0114] Here, AirBoundX and AirBoundY of the current frame are interpreted and explained as described above, and the explanation thereof will be omitted here.

[0115] After obtaining AirBoundX and AirBoundY of the current frame, if AirBoundX and AirBoundY are both greater than 0, it is determined that the current frame supports the adaptive intra-refresh technique. If AirBoundX and / or AirBoundY are equal to 0, it is determined that the current frame does not support the adaptive intra-refresh technique.

[0116] Furthermore, if the decoding side determines that the current frame supports the adaptive intra refresh technology and the frame preceding the current frame in the decoding order does not support the adaptive intra refresh technology, the decoding side determines that the current frame is a random access point, where the fact that the previous frame does not support the adaptive intra refresh technology includes that the adaptive intra refresh video extension ID does not exist in the extended data of the previous frame, or that the adaptive intra refresh video extension ID exists in the extended data of the previous frame but the value of the first virtual boundary position marking information and / or the second virtual boundary position marking information of the previous frame is equal to 0.

[0117] Accordingly, if it is determined that the current frame supports the adaptive intra-refresh technique, and it is determined based on the first virtual boundary position marking information of the current frame that the pixel position of the virtual boundary is equal to or exceeds the right boundary of the current frame, and it is determined based on the second virtual boundary position marking information of the current frame that the pixel position of the virtual boundary is equal to or exceeds the lower boundary of the current frame, it is determined that the current frame is a random access recovery point.

[0118] For example, for the syntax elements shown in Tables 1 and 2, if the parameters air_bound_x and air_bound_y are present in the extension data of the current frame and both values ​​are greater than 0, and the parameters air_bound_x and air_bound_y of the previous frame in the decoding order of the current frame are not present, or are present and at least one is equal to 0, then the current frame is a random access point and random access is allowed from the current frame.

[0119] If the parameters air_bound_x and air_bound_y exist in the extension data of the current frame, and the value of air_bound_x is equal to PictureWidthInLcu and the value of air_bound_y is equal to PictureHeightInLcu, then the current frame is a recovery point, and all frame images from the current frame onwards can be accurately decoded.

[0120] Furthermore, if the current frame is used for random access, the decoding side needs to obtain a valid sequence header before decoding the current frame, and decodes the current frame based on the information added to the sequence header.

[0121] Here, the valid sequence header may be a sequence header located before the bitstream of the current frame and closest to the bitstream of the current frame, or a sequence header received from the system layer. Information added to the sequence header includes sequence-level information required for the decoding process, such as the bitstream profile, level, enable switches for various technologies, image resolution, and frame rate.

[0122] In addition, in an embodiment of the present invention, the position information of the virtual boundary can be arbitrarily specified, so that the embodiment of the present invention supports horizontal, vertical, and diagonal refresh, and the refresh direction can be from the top left to the bottom right, and detailed description thereof will be omitted here.

[0123] As described above, an adaptive intra-refresh video extension ID and virtual boundary position marking information are newly added to the extended data in the bitstream of the current frame, and the virtual boundary position marking information can indicate the position of the virtual boundary, and the virtual boundary is used to distinguish at least the refreshed area and the unrefreshed area in the current frame. Therefore, when dividing the refresh area of ​​the current frame, the encoding side can arbitrarily set the virtual boundary according to actual needs, and then indicate the position of the virtual boundary through the virtual boundary position marking information, thereby improving the flexibility of the refresh area division. That is, an embodiment of the present invention provides an adaptive intra-refresh mechanism that realizes adaptive division of the refresh area through extended data.

[0124] 7 is a schematic diagram of the structure of a decoding device provided in an embodiment of the present invention. As shown in FIG. 7, the decoding device 700 includes: a receiving module 701 for receiving a bitstream of a current frame; a processing module 702 for acquiring virtual boundary position marking information added to the extended data when the extended data exists in the bitstream of the current frame and an adaptive intra-refresh video extension ID is added to the extended data, the virtual boundary position marking information being used to indicate the position of a virtual boundary, and the virtual boundary being used to distinguish between a refreshed region and an unrefreshed region in at least the current frame; The processing module 702 further determines whether the current frame supports an adaptive intra-refresh technique based on the virtual boundary position marking information.

[0125] Based on the above device, in one possible implementation, the processing module is further used to determine that the current frame does not support adaptive intra-refresh technology if an adaptive intra-refresh video extension ID is not present in the extension data of the bitstream of the current frame.

[0126] Based on the above device, in one possible implementation form, the virtual boundary position marking information includes first virtual boundary position marking information and second virtual boundary position marking information, where the first virtual boundary position marking information is used to indicate the pixel position of the virtual boundary in the horizontal direction of the current frame, and the second virtual boundary position marking information is used to indicate the pixel position of the virtual boundary in the vertical direction of the current frame.

[0127] Based on the above device, in one possible implementation form, the first virtual boundary position marking information is the x-coordinate of the virtual boundary in the horizontal direction of the current frame, the second virtual boundary position marking information is the y-coordinate of the virtual boundary in the vertical direction of the current frame, the refreshed area is a rectangular area with its origin at the coordinate (0,0) of the upper left corner of the image, its width being the x-coordinate and its height being the y-coordinate, and the unrefreshed area is an area other than the refreshed area in the current frame.

[0128] Based on the above device, in one possible implementation form, the processing module is used to determine that the current frame supports adaptive intra-refresh technology if the value of the first virtual boundary position marking information and the value of the second virtual boundary position marking information are both greater than 0, and to determine that the current frame does not support adaptive intra-refresh technology if the value of the first virtual boundary position marking information and / or the value of the second virtual boundary position marking information are equal to 0.

[0129] Based on the above device, in one possible implementation form, the processing module is further used to determine that the current frame is a random access point if the current frame supports adaptive intra refresh technology and the frame previous to the current frame in decoding order does not support adaptive intra refresh technology, where the previous frame not supporting adaptive intra refresh technology includes: no adaptive intra refresh video extension ID is present in the extended data of the previous frame, or an adaptive intra refresh video extension ID is present in the extended data of the previous frame but the value of the first virtual boundary position marking information and / or the second virtual boundary position marking information of the previous frame is equal to 0.

[0130] Based on the above device, in one possible implementation form, the processing module is further used to determine that the current frame is a random access recovery point if it determines that the current frame supports adaptive intra-refresh technology, determines that the pixel position of the virtual boundary is equal to or exceeds the right boundary of the current frame based on the first virtual boundary position marking information of the current frame, and determines that the pixel position of the virtual boundary is equal to or exceeds the lower boundary of the current frame based on the second virtual boundary position marking information of the current frame.

[0131] Based on the above device, in one possible implementation form, the precision of the first virtual boundary position marking information is the width of one LCU, and the precision of the second virtual boundary position marking information is the height of one LCU.

[0132] Based on the above device, in one possible implementation, the processing module is further used to obtain one valid sequence header before decoding the current frame if the current frame is used for random access, and to decode the current frame based on information attached to the sequence header.

[0133] As described above, an adaptive intra-refresh video extension ID and virtual boundary position marking information are newly added to the extended data in the bitstream of the current frame, and the virtual boundary position marking information can indicate the position of the virtual boundary, and the virtual boundary is used to distinguish at least the refreshed area and the unrefreshed area in the current frame. Therefore, when dividing the refresh area of ​​the current frame, the encoding side can arbitrarily set the virtual boundary according to actual needs, and then indicate the position of the virtual boundary through the virtual boundary position marking information, thereby improving the flexibility of the refresh area division. That is, an embodiment of the present invention provides an adaptive intra-refresh mechanism that realizes adaptive division of the refresh area through extended data.

[0134] It should be noted that when the decoding device provided in the above embodiments implements the decoding method based on the adaptive intra refresh mechanism, the above-mentioned division of each functional module is simply taken as an example for description, but in actual use, the above functions may be assigned to different functional modules according to needs, that is, the internal structure of the device may be divided into different functional modules to perform all or part of the above-mentioned functions. Furthermore, the decoding device provided in the above embodiments is based on the same concept as the embodiment of the decoding method based on the adaptive intra refresh mechanism, and its specific implementation process is omitted here for reference to the embodiment of the method in detail.

[0135] 8 is a schematic diagram of the structure of an encoding device provided in an embodiment of the present invention. As shown in FIG. 8, the encoding device 800 includes an encoding module 801 for determining whether a current frame supports adaptive intra-refresh technology. If the current frame supports adaptive intra-refresh technology, the encoding module is further used to add an adaptive intra-refresh video extension ID and virtual boundary position marking information of the current frame to the extension data of the current frame, where the virtual boundary position marking information is used to indicate the position of a virtual boundary, and the virtual boundary is used to distinguish at least a refreshed area from an unrefreshed area in the current frame.

[0136] Based on the above device, in one possible implementation form, the virtual boundary position marking information includes first virtual boundary position marking information and second virtual boundary position marking information, where the first virtual boundary position marking information is used to indicate the pixel position of the virtual boundary in the horizontal direction of the current frame, and the second virtual boundary position marking information is used to indicate the pixel position of the virtual boundary in the vertical direction of the current frame.

[0137] Based on the above device, in one possible implementation form, the first virtual boundary position marking information is the x-coordinate of the virtual boundary in the horizontal direction of the current frame, the second virtual boundary position marking information is the y-coordinate of the virtual boundary in the vertical direction of the current frame, the refreshed area is a rectangular area with its origin at the coordinate (0,0) of the upper left corner of the image, its width being the x-coordinate and its height being the y-coordinate, and the unrefreshed area is an area other than the refreshed area in the current frame.

[0138] Based on the above device, in one possible implementation form, the encoding module is further used to, if the current frame does not support adaptive intra-refresh technology, not add an adaptive intra-refresh video extension ID to the extended data of the current frame, or add an adaptive intra-refresh video extension ID to the extended data of the current frame but set the value of the first virtual boundary position marking information or the value of the second virtual boundary position marking information added to the extended data of the current frame to 0.

[0139] Based on the above device, in one possible implementation, the encoding module is further used to, if the current frame supports adaptive intra-refresh technology, make the value of the first virtual boundary position marking information added to the extended data of the current frame greater than 0, and also make the value of the second virtual boundary position marking information greater than 0.

[0140] Based on the above device, in one possible implementation, the encoding module is further used to determine that if the current frame is a random access point, the current frame supports adaptive intra-refresh technology and the frame before the current frame in the encoding order does not support adaptive intra-refresh technology.

[0141] Based on the above device, in one possible implementation, the encoding module is further used to determine that if the current frame is a random access recovery point, the current frame supports adaptive intra-refresh technology, and the pixel position of the virtual boundary indicated by the first virtual boundary position marking information is equal to or exceeds the right boundary of the current frame, and the pixel position of the virtual boundary indicated by the second virtual boundary position marking information is equal to or exceeds the lower boundary of the current frame.

[0142] Based on the above device, in one possible implementation form, the precision of the first virtual boundary position marking information is the width of one LCU, and the precision of the second virtual boundary position marking information is the height of one LCU.

[0143] As described above, an adaptive intra-refresh video extension ID and virtual boundary position marking information are newly added to the extended data in the bitstream of the current frame, and the virtual boundary position marking information can indicate the position of the virtual boundary, and the virtual boundary is used to distinguish at least the refreshed area and the unrefreshed area in the current frame. Therefore, when dividing the refresh area of ​​the current frame, the encoding side can arbitrarily set the virtual boundary according to actual needs, and then indicate the position of the virtual boundary through the virtual boundary position marking information, thereby improving the flexibility of the refresh area division. That is, an embodiment of the present invention provides an adaptive intra-refresh mechanism that realizes adaptive division of the refresh area through extended data.

[0144] It should be noted that when the encoding device provided in the above embodiments implements the encoding method based on the adaptive intra refresh mechanism, the above-mentioned division of each functional module is simply taken as an example, but in actual use, the above functions may be assigned to different functional modules according to needs, that is, the internal structure of the device may be divided into different functional modules to perform all or part of the above-mentioned functions. Furthermore, the encoding device provided in the above embodiments is based on the same concept as the embodiments of the encoding method based on the adaptive intra refresh mechanism, and its specific implementation process is omitted here for reference to the embodiments of the method in detail.

[0145] FIG. 9 is a schematic diagram of the structure of a terminal 900 provided in an embodiment of the present invention. The decoding device, decoding side, decoding apparatus, and encoding device, encoding side, encoding apparatus according to the above-described embodiment may all be implemented in the terminal. Specifically, the terminal 900 may be a smartphone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer, or a desktop computer. The terminal 900 may also be referred to by other names, such as a user device, a mobile terminal, a laptop terminal, or a desktop terminal. Typically, the terminal 900 includes a processor 901 and a memory 902.

[0146] The processor 901 may include one or more processing cores, such as a 4-core processor or an 8-core processor. The processor 901 may be implemented in at least one hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), or a PLA (Programmable Logic Array). The processor 901 may further include a main processor and a coprocessor. The main processor is a processor for processing data in a wake state and is also called a CPU (Central Processing Unit), and the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 901 may include a GPU (Graphics Processing Unit) for rendering and drawing content to be displayed on a display. In some embodiments, the processor 901 may further include an AI (Artificial Intelligence) processor for processing computational operations related to machine learning.

[0147] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may further include high-speed random access memory and non-volatile memory, such as one or more disk storage devices, flash memory storage devices, etc. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one instruction executed by the processor 901 to perform the decoding and encoding methods based on the adaptive intra-refresh mechanism provided in the method embodiments of the present invention.

[0148] In some embodiments, the terminal 900 optionally further includes a peripheral device interface 903 and at least one peripheral device. The processor 901, the memory 902, and the peripheral device interface 903 may be connected to one another via a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 903 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 904, a display 905, a camera component 906, an audio circuit 907, a positioning component 908, and a power source 909.

[0149] The peripheral device interface 903 is used to connect at least one peripheral device associated with I / O (Input / Output) to the processor 901 and the memory 902. In some embodiments, the processor 901, the memory 902, and the peripheral device interface 903 are integrated on the same chip or circuit board, and in some other embodiments, any one or two of the processor 901, the memory 902, and the peripheral device interface 903 may be implemented on separate chips or circuit boards, and this embodiment is not limited thereto.

[0150] The radio frequency circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also called electromagnetic signals. The radio frequency circuit 904 communicates with communication networks and other communication devices via electromagnetic signals. The radio frequency circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 904 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, an encoding / decoding chipset, a subscriber identity module card, etc. The radio frequency circuit 904 may communicate with other terminals via at least one wireless communication protocol. The wireless communication protocols include, but are not limited to, metropolitan area networks, mobile communication networks of various generations (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the radio frequency circuit 904 may further include circuitry related to NFC (Near Field Communication), although the present invention is not limited thereto.

[0151] The display 905 is used to display a user interface (UI). The UI may include graphics, text, icons, videos, and any combination thereof. If the display 905 is a touch display, the display 905 may further collect touch signals on or above the surface of the display 905. The touch signals may be input as control signals to the processor 901 for processing. In this case, the display 905 may further provide virtual buttons and / or a virtual keyboard, also referred to as soft buttons and / or a soft keyboard. In some embodiments, the display 905 may be a single display mounted on the front panel of the terminal 900. In other embodiments, the display 905 may be at least two displays mounted on different surfaces of the terminal 900 or designed to be foldable. In other embodiments, the display 905 may be a flexible display mounted on a curved or foldable surface of the terminal 900. Furthermore, the display 905 may be configured to have an irregular shape other than a rectangle, i.e., an irregular screen. The display 905 can be made using materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0152] The camera component 906 is used to capture images or videos. Optionally, the camera component 906 includes a front camera and a rear camera. Typically, the front camera is installed on the front panel of the terminal, and the rear camera is installed on the back of the terminal. In some embodiments, there are at least two rear cameras, each of which is one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera. This enables a background blur function by combining the main camera and the depth-of-field camera, a panoramic shooting and a virtual reality (VR) shooting function by combining the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments, the camera component 906 may further include a flash lamp. The flash lamp may be a single-color temperature flash lamp or a dual-color temperature flash lamp. A dual-color temperature flash lamp is a combination of a warm white flash lamp and a cool white flash lamp, and may be used to compensate for light beams at different color temperatures.

[0153] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment and convert them into electrical signals that are input to the processor 901 for processing or to the radio frequency circuit 904 for voice communication. For purposes of stereophonic sound collection or noise reduction, multiple microphones may be installed at different locations on the terminal 900. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker may be a conventional thin-film speaker or a piezoelectric ceramic speaker. If the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves that are audible to humans, but also into sound waves that are inaudible to humans for applications such as distance measurement. In some embodiments, the audio circuit 907 may further include a headphone jack.

[0154] The positioning component 908 is used to determine the current geographical location of the terminal 900 to enable navigation or LBS (Location Based Services). The positioning component 908 may be a positioning component based on the United States' GPS (Global Positioning System), the Chinese Beidou system, the Russian Glonass system, or the European Union's Galileo system.

[0155] The power source 909 is used to supply power to each component in the terminal 900. The power source 909 may be an AC power source, a DC power source, a primary battery, or a secondary battery. If the power source 909 includes a secondary battery, the secondary battery may support wired or wireless charging. The rechargeable battery may also be used to support fast charging technology.

[0156] In some embodiments, the terminal 900 further includes one or more sensors 910, including, but not limited to, an acceleration sensor 911, a gyro sensor 912, a pressure sensor 913, a fingerprint sensor 914, an optical sensor 915, and a proximity sensor 916.

[0157] The acceleration sensor 911 can detect the magnitude of acceleration on three coordinate axes of a coordinate system established by the terminal 900. For example, the acceleration sensor 911 is used to detect components of gravitational acceleration on three coordinate axes. Based on the gravitational acceleration signal collected by the acceleration sensor 911, the processor 901 can control the display 905 to display the user interface in a landscape view or a portrait view. The acceleration sensor 911 can also be used to collect game or user movement data.

[0158] The gyro sensor 912 can detect the orientation and rotation angle of the main body of the terminal 900, and in cooperation with the acceleration sensor 911, can collect the user's 3D motion relative to the terminal 900. Based on the data collected by the gyro sensor 912, the processor 901 can realize functions such as motion detection (such as changing the UI in response to a user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0159] The pressure sensor 913 may be installed on a side frame of the terminal 900 and / or below the display 905. When the pressure sensor 913 is installed on the side frame of the terminal 900, it can detect a grip signal from the user on the terminal 900, and the processor 901 distinguishes between left and right hands or performs a shortcut operation based on the grip signal collected by the pressure sensor 913. When the pressure sensor 913 is installed below the display 905, the processor 901 realizes control over operable controls on the UI screen in response to a user's pressing operation on the display 905. The operable controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0160] The fingerprint sensor 914 is used to collect a user's fingerprint, and the processor 901 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 914, or identifies the user's identity based on the fingerprint collected by the fingerprint sensor 914. Once the user's identity is identified as a trusted identity, the processor 901 allows the user to perform related confidential operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 914 may be installed on the front, back, or side of the terminal 900. If a physical button or a manufacturer's logo is installed on the terminal 900, the fingerprint sensor 914 may be integrated with the physical button or the manufacturer's logo.

[0161] The optical sensor 915 is used to collect the intensity of ambient light. In one embodiment, the processor 901 can control the display brightness of the display 905 based on the intensity of ambient light collected by the optical sensor 915. Specifically, when the intensity of ambient light is high, the display brightness of the display 905 is increased, and when the intensity of ambient light is low, the display brightness of the display 905 is decreased. In another embodiment, the processor 901 can further dynamically adjust the shooting parameters of the camera component 906 based on the intensity of ambient light collected by the optical sensor 915.

[0162] The proximity sensor 916, also called a distance sensor, is typically installed on the front panel of the terminal 900. The proximity sensor 916 is used to collect the distance between the user and the front of the terminal 900. In one embodiment, when the proximity sensor 916 detects that the distance between the user and the front of the terminal 900 is gradually decreasing, the processor 901 controls the display 905 to switch from a screen-on state to a screen-off state, and when the proximity sensor 916 detects that the distance between the user and the front of the terminal 900 is gradually increasing, the processor 901 controls the display 905 to switch from a screen-off state to a screen-on state.

[0163] Those skilled in the art will appreciate that the structure shown in FIG. 9 does not constitute a limitation on terminal 900, which may include more or fewer components than shown, or may combine certain components or employ different component arrangements.

[0164] An embodiment of the present invention further provides a non-transitory computer-readable storage medium, and when instructions in the storage medium are executed by a processor of a terminal, the terminal can perform the decoding and encoding method based on the adaptive intra-refresh mechanism provided in the above embodiment.

[0165] An embodiment of the present invention further provides a computer program product including instructions, which, when executed on a terminal, cause the terminal to perform a decoding and encoding method based on the adaptive intra-refresh mechanism provided in the above embodiment.

[0166] It will be understood by those skilled in the art that all or part of the steps for realizing the above embodiments may be performed by hardware, or may be instructed to perform the steps by a program that causes related hardware to perform the steps, and the program may be stored in a computer-readable storage medium, which may be, for example, a read-only memory, a magnetic disk, or an optical disk.

[0167] The above are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention. [Explanation of symbols]

[0168] 700 Decoding Device 701 Receiver Module 702 Processing Module 800 encoder 801 Encoding Module 900 terminals 901 processor 902 memory 903 Peripheral Device Interface 904 Radio Frequency Circuits 905 Display 906 Camera parts 907 Audio Circuit 908 Positioning parts 909 Power supply 910 Sensor 911 Acceleration Sensor 912 Gyro Sensor 913 Pressure Sensor 914 Fingerprint Sensor 915 Optical Sensor 916 Proximity Sensor

Claims

1. receiving a bitstream of a current frame; a step of acquiring virtual boundary position marking information added to the extended data when the bitstream of the current frame contains extended data and an adaptive intra-refresh video extension ID is added to the extended data, the virtual boundary position marking information being used to indicate the position of a virtual boundary, and the virtual boundary being used to distinguish at least a refreshed area from an unrefreshed area in the current frame; determining whether the current frame supports an adaptive intra refresh technique based on the virtual boundary position marking information; If the adaptive intra-refresh video extension ID does not exist in extension data of the bitstream of the current frame, determining that the current frame does not support the adaptive intra-refresh technology.

1. A decoding method based on an adaptive intra refresh mechanism, comprising:

2. the virtual boundary position marking information includes first virtual boundary position marking information and second virtual boundary position marking information; the first virtual boundary position marking information is used to indicate a pixel position of the virtual boundary in the horizontal direction of the current frame; the second virtual boundary position marking information is used to indicate a pixel position of the virtual boundary in the vertical direction of the current frame; 2. The method of claim 1 .

3. the first virtual boundary position marking information is an x-coordinate of the virtual boundary in the horizontal direction of the current frame; the second virtual boundary position marking information is a y coordinate of the virtual boundary in the vertical direction of the current frame; The refreshed area is a rectangular area whose origin is the coordinate (0, 0) of the upper left corner of the image, whose width is the x coordinate, and whose height is the y coordinate; The unrefreshed area is an area other than the refreshed area in the current frame.

3. The method of claim 2.

4. determining whether the current frame supports an adaptive intra refresh technique based on the virtual boundary position marking information, determining that the current frame supports the adaptive intra refresh technique when the value of the first virtual boundary position marking information and the value of the second virtual boundary position marking information are both greater than 0; determining that the current frame does not support the adaptive intra refresh technique when a value of the first virtual boundary position marking information and / or a value of the second virtual boundary position marking information is equal to 0; 3. The method of claim 2.

5. and determining that the current frame is a random access recovery point when determining that the current frame supports the adaptive intra-refresh technique, and determining that a pixel position of a virtual boundary is equal to or exceeds a right boundary of the current frame based on first virtual boundary position marking information of the current frame, and determining that a pixel position of a virtual boundary is equal to or exceeds a lower boundary of the current frame based on second virtual boundary position marking information of the current frame.

3. The method of claim 2.

6. The precision of the first virtual boundary position marking information is the width of one maximum coding unit (LCU); The accuracy of the second virtual boundary position marking information is the height of one LCU.

3. The method of claim 2.

7. determining whether the current frame supports an adaptive intra refresh technique; If the current frame supports the adaptive intra-refresh technology, adding an adaptive intra-refresh video extension ID and virtual boundary position marking information of the current frame to extension data of the current frame, the virtual boundary position marking information being used to indicate a position of a virtual boundary, and the virtual boundary being used to distinguish at least a refreshed area from an unrefreshed area in the current frame; If the current frame does not support the adaptive intra refresh technology, not adding the adaptive intra refresh video extension ID to the extended data of the current frame, or adding the adaptive intra refresh video extension ID to the extended data of the current frame but setting a value of the virtual boundary position marking information added to the extended data of the current frame to indicate whether the adaptive intra refresh technology is supported.

1. A coding method based on an adaptive intra refresh mechanism, comprising:

8. the virtual boundary position marking information includes first virtual boundary position marking information and second virtual boundary position marking information; the first virtual boundary position marking information is used to indicate a pixel position of the virtual boundary in the horizontal direction of the current frame; the second virtual boundary position marking information is used to indicate a pixel position of the virtual boundary in the vertical direction of the current frame; 8. The method of claim 7.

9. the first virtual boundary position marking information is an x-coordinate of the virtual boundary in the horizontal direction of the current frame; the second virtual boundary position marking information is a y coordinate of the virtual boundary in the vertical direction of the current frame; The refreshed area is a rectangular area whose origin is the coordinate (0, 0) of the upper left corner of the image, whose width is the x coordinate and whose height is the y coordinate, and the unrefreshed area is an area other than the refreshed area in the current frame.

9. The method of claim 8.

10. If the current frame does not support the adaptive intra-refresh technology, the adaptive intra-refresh video extension ID is added to the extended data of the current frame, but a value of the first virtual boundary position marking information and / or a value of the second virtual boundary position marking information added to the extended data of the current frame is set to 0; If the current frame supports the adaptive intra-refresh technique, the value of the first virtual boundary position marking information added to the extended data of the current frame is set to be greater than 0, and the value of the second virtual boundary position marking information is also set to be greater than 0.

9. The method of claim 8.

11. If the current frame is a random access point, determining that the current frame supports the adaptive intra refresh technique and that a frame preceding the current frame in coding order does not support the adaptive intra refresh technique; If the current frame is a random access recovery point, determining that the current frame supports the adaptive intra-refresh technique, and that a pixel position of a virtual boundary indicated by the first virtual boundary position marking information is equal to or exceeds a right boundary of the current frame, and that a pixel position of a virtual boundary indicated by the second virtual boundary position marking information is equal to or exceeds a bottom boundary of the current frame.

9. The method of claim 8.

12. The precision of the first virtual boundary position marking information is the width of one LCU; The accuracy of the second virtual boundary position marking information is the height of one LCU.

9. The method of claim 8.

13. a receiving module for receiving a bitstream of a current frame; When extension data exists in the bitstream of the current frame and an adaptive intra-refresh video extension ID is added to the extension data, obtaining virtual boundary position marking information added to the extension data, wherein the virtual boundary position marking information is used to indicate a position of a virtual boundary, and the virtual boundary is used to distinguish at least a refreshed area from an unrefreshed area in the current frame; determining whether the current frame supports an adaptive intra refresh technique based on the virtual boundary position marking information; a processing module for determining that the current frame does not support the adaptive intra-refresh technology if the adaptive intra-refresh video extension ID is not present in the extension data of the bitstream of the current frame. A decoding device characterized by:

14. Determining whether the current frame supports adaptive intra-refresh technology; If the current frame supports the adaptive intra-refresh technology, add an adaptive intra-refresh video extension ID and virtual boundary position marking information of the current frame to extension data of the current frame, wherein the virtual boundary position marking information is used to indicate a position of a virtual boundary, and the virtual boundary is used to distinguish at least a refreshed area from an unrefreshed area in the current frame; an encoding module for, if the current frame does not support the adaptive intra refresh technology, not adding the adaptive intra refresh video extension ID to the extended data of the current frame, or adding the adaptive intra refresh video extension ID to the extended data of the current frame, but setting a value of the virtual boundary position marking information added to the extended data of the current frame to indicate whether the adaptive intra refresh technology is supported.

1. An encoding device comprising:

15. a processor; a memory for storing instructions executable by a processor, The processor is configured to perform the steps of the method according to any one of claims 1 to 6.

10. A decoding device comprising:

16. a processor; a memory for storing instructions executable by a processor, The processor is configured to perform the steps of the method according to any one of claims 7 to 12.

10. A coding device comprising:

17. A computer-readable storage medium storing instructions that, when executed by a processor, perform the steps of the method of any one of claims 1 to 6. A computer-readable storage medium comprising:

18. A computer-readable storage medium storing instructions that, when executed by a processor, perform the steps of the method of any one of claims 7 to 12. A computer-readable storage medium comprising:

Images