Empty tile encoding in video encoding

Abstract

A bitstream processing method includes: parsing a bitstream to obtain a picture region flag from a data unit corresponding to a picture region in the bitstream, wherein the picture region includes N picture blocks, where N is an integer; and selectively generating a decoded representation of the picture region from the bitstream based on a value of the picture region flag, wherein the selective generation includes: when the value of the picture region flag is a first value, generating a decoded representation from the bitstream using a first decoding method; and when the value of the picture region flag is a second value different from the first value, generating a decoded representation from the bitstream using a second decoding method different from the first decoding method.

Description

Technical Field

[0001] This patent application generally relates to encoding and decoding of video and images. Background Art

[0002] Video coding uses compression tools to encode two-dimensional video frames into a compressed bitstream representation that is more efficient when stored or transmitted over a network. Conventional video coding techniques that use two-dimensional video frames for encoding are sometimes inefficient for representing visual information of three-dimensional visual scenes. Summary of the invention

[0003] Among other things, this patent application describes techniques for encoding and decoding digital video using null tile coding, which may be used in some embodiments to encode or decode immersive video.

[0004] The present disclosure relates to video processing and communication, and more particularly to a method and apparatus for encoding a digital video or picture to generate a bitstream, a method and apparatus for decoding a bitstream to reconstruct a digital video or picture (visual information), and a method and apparatus for extracting a bitstream to form a sub-bitstream.

[0005] In one example aspect, a method for processing a bitstream is disclosed. The method includes parsing a bitstream to obtain a picture region flag from a data unit corresponding to a picture region in the bitstream, wherein the picture region includes N picture blocks, where N is an integer; and selectively generating a decoded representation of the picture region from the bitstream based on a value of the picture region flag, wherein the selective generation includes: generating a decoded representation from the bitstream using a first decoding method when the value of the picture region flag is a first value; and generating a decoded representation from the bitstream using a second decoding method different from the first decoding method when the value of the picture region flag is a second value different from the first value.

[0006] In another aspect, a visual information processing method is disclosed. The method includes parsing a bitstream to obtain a picture region parameter from a parameter set data unit in the bitstream, wherein the picture region parameter indicates that a picture is divided into one or more picture regions; determining one or more picture regions located in the target picture region according to a target picture region; extracting one or more data units corresponding to the one or more picture regions located in the target picture region from the bitstream to form a sub-bitstream; generating a first data unit corresponding to an external picture region outside the target picture region, and setting a picture region flag in the first data unit to be equal to a first value, the first value indicating that: in the external picture region, no bits are encoded in the bitstream for the coding block; and inserting the first data unit into the sub-bitstream.

[0007] In another example aspect, a video or picture encoding method is disclosed. The method includes dividing a picture into one or more picture regions, wherein the picture region includes N picture blocks, where N is an integer, and selectively generating a bitstream from the N picture blocks based on a coding standard. The selective generation includes: when the coding standard is to encode the picture region, encoding a picture region flag corresponding to the picture region as a first value, and encoding the picture blocks in the picture region using a first coding method (186), and when the coding standard is not to encode the picture region, then encoding the picture region flag corresponding to the picture region as a second value, and encoding the picture region using a second coding method different from the first coding method.

[0008] In another example aspect, an apparatus for processing one or more bitstreams of video or pictures is disclosed.

[0009] In yet another example aspect, a computer program storage medium is disclosed. The computer program storage medium includes code stored thereon. When executed by a processor, the code causes the processor to implement the method.

[0010] These and other aspects are described in this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1A is a flow chart of an example method of bitstream processing.

[0012] Figure 1B is a flow chart of an example method of visual information processing.

[0013] Figure 1C is a flow chart of an example method for a method of processing a video or a picture.

[0014] Figure 2 is a diagram illustrating an example video or picture encoder implementing the methods of the present disclosure.

[0015] Figure 3 is a diagram showing an example of dividing a picture into tile groups.

[0016] Figure 4 is a diagram showing an example of dividing a picture into tile groups.

[0017] Figure 5 is a diagram showing an example of viewing a 360-degree omnidirectional video.

[0018] Figure 6 is a diagram showing an example of dividing a picture into picture areas.

[0019] Figure 7A-7B An example of a syntax structure in a bitstream is shown.

[0020] Figure 8 is a diagram illustrating an example video or picture decoder implementing the methods of the present disclosure.

[0021] Fig. 9 is a diagram showing an example of an extractor implementing the method in the present disclosure.

[0022] Fig.10 is a diagram illustrating a first example device including at least an example encoder described in the present disclosure.

[0023] Fig.11 is a diagram illustrating a second example device including at least an example decoder described in the present disclosure.

[0024] Fig.12 is a diagram illustrating an electronic system including a first example device and a second example device.

[0025] Fig.13A An example is shown where the tile group used for rendering is a viewport.

[0026] Fig. 13B An example of tile reorganization for frame-based compression is shown.

[0027] Fig.14 A hardware platform for implementing the techniques described in this application is shown. DETAILED DESCRIPTION

[0028] Section headings are used in this application only to improve readability and do not limit the scope of the embodiments and techniques disclosed in each section to only that section. Certain features are described using examples of the H.264 / AVC (Advanced Video Coding), H.265 / HEVC (High Efficiency Video Coding), and H.266 Versatile Video Coding (VVC) standards. However, the applicability of the disclosed techniques is not limited to H.264 / AVC or H.265 / HEVC or H.266 / VVC systems.

[0029] The present disclosure relates to video processing and communication, and more particularly, to a method and apparatus for encoding a digital video or picture to generate a bit stream, and a method and apparatus for decoding the bit stream to reconstruct the digital video or picture.

[0030] Brief Introduction

[0031] The technology for compressing digital video and pictures uses the correlation characteristics between pixel samples to eliminate redundancy in videos and pictures. The encoder can divide the picture into one or more picture areas containing multiple units. This picture area breaks the prediction dependency within the picture so that the picture area can be decoded, or at least the syntax elements corresponding to the picture area can be correctly parsed without referring to the data of another picture area in the same picture. This picture area introduced in the video coding standard helps in resynchronization after data loss, parallel processing, coding and streaming media areas of interest, packet transmission, viewport-related streaming media, etc. Examples of such picture areas include slices / slice groups in the H.264 / AVC standard, slices / tiles in the H.265 / HEVC standard, and tile groups / tiles in the H.266 / VVC standard (which is currently being developed by JVET (ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 Joint Video Experts Group)).

[0032] 360-degree omnidirectional video provides viewers with an immersive perceptual experience. A typical service using 360-degree omnidirectional video is virtual reality (VR). Other services using this video include augmented reality (AR), mixed reality (MR), and extended reality (XR). Take VR services as an example. In currently applicable solutions, 360-degree omnidirectional video in the form of spherical video is first projected onto a regular video of a rectangular picture, which is then encoded using an ordinary encoder (e.g., H.264 / AVC or H.265 / HEVC encoder) and transmitted over the network. At the destination, an ordinary decoder reconstructs the rectangular picture for rendering by a display (e.g., head-mounted device, HMD). The most popular projection methods are ERP (equirectangular projection) and cubemap projection.

[0033] In order to save transmission bandwidth, viewport-based streaming media has been developed. At the destination, the user device (such as HMD) tracks the direction in which the viewer is focused, generates current viewport information, and feeds the viewport information back to the media server. The media server extracts a sub-bitstream that covers only one or more picture areas for rendering the scene of the current viewport, and sends the sub-bitstream to the user device at the destination. From the perspective of video coding, this viewport-based streaming media can be performed with the help of slices / slice groups in the H.264 / AVC standard, slices / tiles in the H.265 / HEVC standard, and tile groups / tiles in the H.266 / VVC standard (which is currently being developed by JVET (Joint Video Experts Group of ITU-TSG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11)).

[0034] A general example of viewport-based streaming is shown below. A 360-degree omnidirectional video is projected onto a regular video using a cubemap projection. During encoding, the picture is split into 24 tile groups or tiles. If the viewer focuses on a scene such as Figure 5 , then 12 of the total 24 tile groups or tiles are required in rendering, as shown in FIG. Fig.13A As shown. Note that Fig.13A Reproduced from MPEG contribution m46538.

[0035] because Fig.13A The tile groups or tiles in do not form a rectangular picture, so a frame-based method is used to rearrange the positions of these tile groups or tiles to form a rectangular picture, such as Fig. 13B The server extracts data units corresponding to tile groups or tiles used to render the viewport, and organizes these data units to generate sub-bitstreams according to the formed rectangular picture.

[0036] The disadvantages of viewport-based streaming using a frame-based approach are as follows. Fig. 13B In the original picture in , the positions of tile groups or tiles correspond to the faces of the cubes projected by the cubemap used, which have a clear geometric mapping relationship with the areas on the surface of the sphere used to render the 360-degree omnidirectional video. After rearrangement by the frame-based method, this mapping relationship is destroyed in the compressed picture because not all tile groups or tiles follow the grid of the cube faces projected by the cubemap. One solution is that the server generates metadata specifying the rearrangement position and sends the metadata to the user device together with the sub-bitstream. The user device restores the position of the tile groups or tiles in the compressed picture to the position in the original picture, and then renders the area on the sphere of the 360-degree omnidirectional video for viewing. Obviously, the computational complexity of both the server and the user device will increase, and the metadata will consume additional transmission bandwidth as well as computing and storage resources of the network middleware.

[0037] In practice, the general problem is how to signal picture regions that are not represented in the video bitstream (e.g. Fig.13A or the dark area in 13B).

[0038] Another application scenario is video surveillance, especially when high-resolution video is used in surveillance systems. Since the content in the background area does not change often or always remains relatively constant, the actual focus is on one or more picture areas with moving objects. Therefore, the coding efficiency of surveillance video can be greatly improved by skipping the encoding of background content, which requires signaling of uncoded or skipped picture areas.

[0039] Embodiments of the present disclosure provide video or picture encoding and decoding methods, encoding and decoding devices, methods and apparatus for extracting bitstreams to form sub-bitstreams, so as to at least solve the problem of additional computational burden in bitstream extraction processing and extractors.

[0040] According to one aspect of an embodiment of the present disclosure, there is provided a coding method for processing a video or a picture, comprising:

[0041] Divide the picture into one or more picture regions, wherein the picture region includes one or more coding blocks;

[0042] Determine whether to encode the picture region; if yes, encode the picture region flag corresponding to the picture region to be equal to the first value, and encode the coding block in the picture region;

[0043] Otherwise, the picture region flag is encoded to be equal to a second value, encoding of the coding blocks in the picture region is skipped, and the values ​​of the pixels in the picture region are set to be equal to the values ​​of the co-located pixels in the reference picture of the picture region if the reference picture exists and the type of the picture region indicates inter-frame prediction, or, if the reference picture does not exist or the type of the picture region indicates intra-frame prediction, the values ​​of the pixels in the picture region are set to a value equal to a predetermined value.

[0044] According to one aspect of an embodiment of the present disclosure, there is provided a decoding method for processing a bit stream to reconstruct a video or a picture, comprising:

[0045] Parsing the bitstream to obtain a picture region flag from a data unit corresponding to a picture region in the bitstream;

[0046] If the picture region flag is equal to a first value, decoding one or more decoded blocks in the picture region;

[0047] Otherwise, if the picture region flag is equal to the second value, if the reference picture exists and the type of the picture region indicates inter-frame prediction, the value of the pixel in the picture region is set to be equal to the value of the co-located pixel in the reference picture of the picture region, or, if the reference picture does not exist or the type of the picture region indicates intra-frame prediction, the value of the pixel in the picture region is set to be equal to the predetermined value.

[0048] According to one aspect of an embodiment of the present disclosure, there is provided an extraction method for processing a bitstream to derive a sub-bitstream decodable using the above decoding method, comprising:

[0049] Parsing the bitstream to obtain a picture region parameter from a parameter set data unit in the bitstream, wherein the picture region parameter indicates partitioning the picture into one or more picture regions;

[0050] According to the target image area, determining one or more image areas located in the target image area;

[0051] extracting one or more data units corresponding to one or more picture regions located in the target picture region from the bitstream to form a sub-bitstream;

[0052] generating a first data unit corresponding to a picture region outside the target picture region, and setting a picture region flag in the first data unit to be equal to a first value, the first value indicating that there are no bits of a coding block in the picture region outside the target picture region;

[0053] A first data unit is inserted into the sub-bitstream.

[0054] Through the above method, the problem of additional computing burden of viewport-based streaming media in the related art is solved, and the effect of effectively encoding the picture area skipped in encoding is further implemented.

[0055] In the present disclosure, a video is composed of a sequence of one or more pictures. A bitstream (also referred to as a video elementary stream) is generated by an encoder that processes a video or picture. A bitstream may also be a transport stream or a media file, which is the output of a system layer processing performed on a video elementary stream generated by a video or picture encoder. Decoding a bitstream produces a video or picture. System layer processing is to encapsulate a video elementary stream. For example, a video elementary stream is compressed into a transport stream or a media file as a payload. System layer processing also includes an operation of encapsulating a transport stream or a media file into a stream for transmission or a file for storage as a payload. The data unit generated in the system layer processing is referred to as a system layer data unit. The information attached to the system layer data unit during the encapsulation of the payload in the system layer processing is referred to as system layer information, for example, a header of the system layer data unit. A sub-bitstream is obtained by extracting a bitstream through an extraction process, and the sub-bitstream contains a portion of the bits of the bitstream and one or more necessary modifications to the syntax elements. Decoding a sub-bitstream produces a video or picture, which may have a lower resolution and / or a lower frame rate than the video or picture obtained by decoding the bitstream. A video or picture obtained from a sub-bitstream may also be a region of a video or picture obtained from a bitstream.

[0056] Example 1

[0057] Figure 2is a schematic diagram of an encoder for encoding a video or a picture using the method of the present disclosure. The input of the encoder is a video, and the output is a bitstream. Since a video is composed of a series of pictures, the encoder processes the pictures one by one in a preset order (i.e., the encoding order). The encoder order is determined according to the prediction structure specified in the configuration file of the encoder. Note that the encoding order of the pictures in the video (corresponding to the decoding order of the pictures at the decoder end) can be the same as the display order of the pictures, or it can be different.

[0058] The segmentation unit 201 segments the picture in the input video according to the configuration of the encoder. Generally, the picture can be segmented into one or more maximum coding blocks. The maximum coding block is the maximum block allowed or configured in the encoding process, and is usually a square area in the picture. The picture can be segmented into one or more tiles, and the tile can contain an integer number of maximum coding blocks, or a non-integer number of maximum coding blocks. One option is that the tile may contain one or more slices. That is, the tile can be further segmented into one or more slices, and each slice can contain an integer number of maximum coding blocks, or a non-integer number of maximum coding blocks. Another option is that the slice contains one or more tiles, or the tile group contains one or more tiles. That is, one or more tiles in a specific order (such as a raster scan order of tiles) in the picture form a tile group. In addition, the tile group can also cover a rectangular area in the picture, which is represented by the position of the upper left tile and the lower right tile. In the following description, "tile group" is used as an example. The segmentation unit 201 can be configured to segment the picture using a fixed pattern. For example, the segmentation unit 201 segments the picture into tile groups, and each tile group has a single tile containing a row of the largest coding block. Another example is that the segmentation unit 201 segments the picture into multiple tiles, and forms the tiles in the picture into tile groups in a raster scan order. Alternatively, the segmentation unit 201 can also adopt a dynamic mode to segment the picture into tile groups, tiles, and blocks. For example, in order to adapt to the limitation of the maximum transmission unit (MTU) size, the segmentation unit 201 adopts a dynamic tile group segmentation method to ensure that the number of coded bits for each tile group does not exceed the MTU limit.

[0059] Figure 3 2 is a schematic diagram showing an example of partitioning a picture into tile groups. The partitioning unit 201 partitions the picture 30 having 16×8 maximum coding blocks (indicated by dotted lines) into 8 tiles 300, 310, 320, 330, 340, 350, 360 and 370. The partitioning unit 201 partitions the picture 30 into 3 tile groups. Tile group 3000 includes tile 300, tile group 3100 includes tiles 310, 320, 330, 340 and 350, and tile group 3200 includes tiles 360 and 370. Figure 3The tile groups in FIG30 are formed in a tile grid scan order.

[0060] Figure 4 2 is a schematic diagram showing an example of partitioning a picture into tile groups. The partitioning unit 201 partitions a picture 40 having 16×8 maximum coding blocks (indicated by dotted lines) into 8 tiles 400, 410, 420, 430, 440, 450, 460, and 470. The partitioning unit 201 partitions the picture 40 into 2 tile groups. Tile group 4000 includes tiles 400, 410, 440, and 450, and tile group 4100 includes tiles 420, 430, 460, and 470. Tile group 4000 is represented as an upper left tile 400 and a lower right tile 450, and tile group 4100 is represented as an upper left tile 420 and a lower right tile 470.

[0061] One or more tile groups or tiles may be referred to as picture regions. Typically, a picture is segmented into one or more tiles according to an encoder configuration file. The segmentation unit 201 sets segmentation parameters to indicate the manner in which the picture is segmented into tiles. For example, the segmentation manner may be to segment the picture into tiles of (almost) equal size. Another example is that the segmentation manner may indicate the position of tile boundaries in rows and / or columns for flexible segmentation.

[0062] The output parameter of the segmentation unit 201 indicates the segmentation method of the picture.

[0063] The prediction unit 202 determines the prediction samples of the coding blocks in the picture area. The prediction unit 202 includes a block segmentation unit 203, an ME (motion estimation) unit 204, an MC (motion compensation) unit 205, and an intra-frame prediction unit 206. The input of the prediction unit 202 is a picture area containing one or more maximum coding blocks output by the segmentation unit 201, and attribute parameters associated with the maximum coding block (for example, the position of the maximum coding block in the picture and the picture area). The prediction unit 202 divides the maximum coding block into one or more coding blocks, which can be further divided into smaller coding blocks. One or more segmentation methods can be applied, including quadtree, binary cut and ternary cut. The prediction unit 202 determines the prediction samples for the coding blocks obtained in the segmentation. Optionally, the prediction unit 202 can further divide the coding block into one or more prediction blocks to determine the prediction samples. The prediction unit 202 uses one or more pictures in the DPB (decoded picture buffer) unit 214 as a reference to determine the inter-frame prediction samples of the coding block. The prediction unit 202 may also use the reconstructed portion of the picture output by the adder 212 as a reference to derive the prediction sample of the coding block. The prediction unit 202 determines the prediction sample of the coding block and related parameters for deriving the prediction sample by, for example, using a general rate-distortion optimization (RDO) method, which are also output parameters of the prediction unit 202.

[0064] The prediction unit 202 also determines whether to skip encoding the picture region. When the prediction unit 202 determines not to skip encoding the picture region, the prediction unit 202 sets the picture region flag to be equal to the first value. Otherwise, when the prediction unit 202 determines to skip encoding the picture region, the prediction unit 202 sets the picture region flag to be equal to the second value, and the prediction unit 202 and other relevant units in the encoder (such as the transform unit 208, quantization unit 209, inverse quantization unit 210, and inverse transform unit 211) do not call the process of encoding the encoded blocks in the picture region. In the case where the picture region flag is equal to the second value, if the reference picture exists and the type of the picture region indicates inter prediction, the prediction unit 202 sets the values of the pixels in the picture region to be equal to the values of the quasi co-located pixels in the reference picture of the picture region, or if the reference picture does not exist or the type of the picture region indicates intra prediction, the values of the pixels in the picture region are set to be equal to a predetermined value. The reference picture can be the first picture in the reference picture list, for example, the picture indicated by the reference index equal to 0 in reference list 0. Optionally, the reference picture can also be the picture in the reference list with the smallest POC (Picture Order Count) difference between the current encoded pictures including the picture region. Optionally, the reference picture can be a picture selected by the prediction unit 202 from the pictures in the reference list (such as using the general RDO method), and the prediction unit 202 needs to output a reference index, which is to be encoded in the bitstream by the entropy encoding unit 215. The predetermined value can be a fixed value burned in the encoder and decoder, or calculated as 1<<(bitDepth–1), where bitDepth is the bit depth value of the pixel sample component, "<<" is the arithmetic left shift operator, and "x<<y" means that the x two's complement integer representation is arithmetically shifted by y binary digits. Optionally, the prediction unit 202 can set the values in the picture region to be equal to the predetermined value regardless of whether the reference picture of the picture region exists. When the picture region flag is equal to the second value, the prediction residual of the encoded blocks in the picture region is set to 0. That is, when the picture region flag is equal to the second value, the values of the reconstructed pixels in the picture region are set to be equal to their predicted values derived by the prediction unit 202.

[0065] The prediction unit 202 can use the general RDO method to determine whether to skip encoding the picture region. For example, when the prediction unit 202 finds that the cumulative value of the cost function for RDO counting of all the encoded blocks in the picture region is not greater than the value of the cost function in the RDO counting for skipping encoding the picture region, the prediction unit 202 determines the picture region flag as the first value; otherwise, it is determined as the second value.

[0066] Optionally, the prediction unit 202 may also determine the picture region flag value according to the encoder configuration. An example scenario is video surveillance, especially when high-resolution video is used in a surveillance system. Since the content in the background area does not change often or remains relatively constant, the actual focus is on one or more picture regions with moving objects, for example, using existing motion detection methods and algorithms. Therefore, when it is determined that the picture region contains at least a portion of a moving object in the scene, the prediction unit 202 sets the picture region flag corresponding to the picture region to be equal to a first value; otherwise, the prediction unit 202 sets the picture region flag to be equal to a second value.

[0067] Another example is in communications using 360-degree omnidirectional video (eg, video calls, video conferences, video chats, remote controls, etc.). Figure 5 is a diagram showing an example of viewing a 360-degree omnidirectional video. Figure 5 The audience in the video watches a 360-degree omnidirectional video encoded using cubemap projection. Figure 6 60 is a diagram showing an example of dividing a picture into picture regions. Picture 60 is divided into 24 picture regions, where a picture region can be a tile group or a tile. Picture regions 600, 601, 606, and 607 correspond to a first surface of a cube map, 602, 603, 608, and 609 correspond to a second surface, 604, 605, 610, and 611 correspond to a third surface, 612, 613, 618, and 619 correspond to a fourth surface, 614, 615, 620, and 621 correspond to a fifth surface, and 616, 617, 622, and 623 correspond to a sixth surface. In order to Figure 5 Rendering content at the viewport shown in FIG. 1 will use image regions 600, 603, 606, 609, 610, 611, 612, 613, 614, 615, 620, and 621 for rendering, while other image regions (in Figure 6 The prediction unit 202 will correspond to Figure 6 The picture region flag of the picture region marked in gray in FIG. 2 is set to be equal to the second value. The prediction unit 202 may directly set the prediction region flag corresponding to the picture region for rendering to be equal to the first value, or call a general RDO method to determine the prediction region flag.

[0068] The output of the prediction unit 202 includes a picture region flag. The predicted values ​​of the pixels in the picture region and other necessary parameters associated with the prediction region flag (eg, a reference index indicating a reference picture used to predict the sample) are also in the output of the prediction unit 202.

[0069] Within the prediction unit 202, the block segmentation unit 203 determines the segmentation of the coding block. The block segmentation unit 203 segments the largest coding block into one or more coding blocks, which can be further segmented into smaller coding blocks. One or more segmentation methods can be applied, including quadtree, binary cut and ternary cut. Optionally, the block segmentation unit 203 can further segment the coding block into one or more prediction blocks to determine the prediction samples. The block segmentation unit 203 can use the RDO method when determining the segmentation of the coding block. The output parameters of the block segmentation unit 203 include one or more parameters indicating the segmentation of the coding block.

[0070] The ME unit 204 and the MC unit 205 use one or more decoded pictures from the DPB 214 as reference pictures to determine the inter-frame prediction samples of the coding block. The ME unit 204 constructs one or more reference lists containing one or more reference pictures, and determines one or more matching blocks in the reference pictures for the coding block. The MC unit 205 uses the samples in the matching blocks to derive the prediction samples, and calculates the difference (i.e., residual) between the original samples and the predicted samples in the coding block. The output parameters of the ME unit 204 indicate the position of the matching block, which includes a reference list index, a reference index (refIdx), a motion vector (MV), etc., wherein the reference list index indicates the reference list containing the reference picture where the matching block is located, the reference index indicates the reference picture in the reference list containing the matching block, and the MV indicates the relative offset between the positions of the coding block and the matching block in the same coordinates, and the same coordinates are used to represent the position of the pixel in the picture. The output parameters of the MC unit 205 are the inter-frame prediction samples of the coding block, and the parameters used to construct the inter-frame prediction samples, such as the weighting parameters for the samples in the matching block, the filter type and parameters for filtering the samples in the matching block. In general, the RDO method can be jointly applied to the ME unit 204 and the MC unit 205 to obtain the best matching block in rate-distortion (RD) sensing and the corresponding output parameters of the two units.

[0071] In particular and optionally, the ME unit 204 and the MC unit 205 can use the current picture containing the coding block as a reference to obtain the intra-frame prediction samples of the coding block. In the present disclosure, intra-frame prediction means that only the data in the picture containing the coding block is adopted as a reference for deriving the prediction samples of the coding block. In this case, the ME unit 204 and the MC unit 205 use the reconstructed part in the current picture, where the reconstructed part comes from the output of the adder 212. An example is that the encoder allocates a picture buffer to (temporarily) store the output data of the adder 212. Another method of the encoder is to reserve a special picture buffer in the DPB 214 to save the data from the adder 212.

[0072] The intra prediction unit 206 uses the reconstructed portion of the current picture containing the coding block as a reference to obtain the intra prediction samples of the coding block. The intra prediction unit 206 uses the reconstructed neighboring samples of the coding block as input to a filter for deriving the intra prediction samples of the coding block, wherein the filter can be an interpolation filter (for example, for calculating the prediction samples when using angular intra prediction), a low-pass filter (for example, for calculating the DC value), or a cross-component filter to derive the prediction value of the (color) component using the encoded (color) component. In particular, the intra prediction unit 206 can perform a search operation to obtain a matching block of the coding block within the reconstructed portion of the current picture, and set the samples in the matching block as the intra prediction samples of the coding block. The intra prediction unit 206 calls the RDO method to determine the intra prediction mode (i.e., the method used to calculate the intra prediction samples of the coding block) and the corresponding prediction samples. In addition to the intra prediction samples, the output of the intra prediction unit 206 also includes one or more parameters indicating the intra prediction mode being used.

[0073] The adder 207 is configured to calculate the difference between the original sample and the predicted sample of the coding block. The output of the adder 207 is the residual of the coding block. The residual can be represented as an N×M two-dimensional matrix, where N and M are two positive integers, and N and M can have equal or different values.

[0074] The transform unit 208 takes the residual as its input. The transform unit 208 may apply one or more transform methods to the residual. From the perspective of signal processing, the transform method can be represented by a transform matrix. Optionally, the transform unit 208 may determine to use a rectangular block of the same shape and size as the coding block (in the present disclosure, a square block is a special case of a rectangular block) as the transform block of the residual. Optionally, the transform unit 208 may determine to divide the residual into a plurality of rectangular blocks (which may also include a special case where the width or height of the rectangular block is one sample), and the transform unit 208 may perform transform operations on the plurality of rectangles in sequence, for example, according to a default order (e.g., a raster scan order), a predefined order (e.g., an order corresponding to a prediction mode or a transform method), or a selected order of a plurality of candidate orders. The transform unit 208 may determine to perform a plurality of transforms on the residual. For example, the transform unit 208 first performs a core transform on the residual, and then performs a secondary transform on the coefficients obtained after completing the core transform. The transform unit 208 may determine the transform parameters using the RDO method, and the transform parameters indicate the execution mode used in the transform process applied to the residual block, for example, dividing the residual block into transform blocks, transform matrices, multiple transforms, etc. The transform parameters are included in the output parameters of the transform unit 208. The output parameters of the transform unit 208 include the transform parameters and the data obtained after the transform residual (e.g., transform coefficients), and the residual can be represented by a two-dimensional matrix.

[0075] The quantization unit 209 quantizes the data output by the transform unit 208 after the transform residue. The quantizer used in the quantization unit 209 may be one or both of a scalar quantizer and a vector quantizer. In most video encoders, the quantization unit 209 adopts a scalar quantizer. The quantization step size of the scalar quantizer is represented by a quantization parameter (QP) in the video encoder. Typically, a given mapping between QP and quantization step size is preset or predefined in an encoder and a corresponding decoder.

[0076] The value of QP (e.g., picture-level QP and / or block-level QP) may be set according to a profile applied to the encoder, or may be determined by an encoder control unit in the encoder. For example, the encoder control unit determines the quantization step size of the picture and / or block using a rate control (RC) method, and then converts the quantization step size into a QP according to a mapping between QP and quantization step size.

[0077] The control parameter of the quantization unit 209 is QP. The output of the quantization unit 209 is one or more quantized transform coefficients (ie, referred to as "levels") represented in the form of a two-dimensional matrix.

[0078] The inverse quantization 210 performs a scaling operation on the output of the quantization 209 to obtain a reconstructed coefficient. The inverse transform unit 211 performs an inverse transform on the reconstructed coefficient from the inverse quantization 210 according to the transform parameters from the transform unit 208. The output of the inverse transform unit 211 is a reconstructed residual. In particular, when the encoder determines to skip quantization when encoding a block (for example, the encoder implements an RDO method to determine whether to apply quantization to a coding block), the encoder directs the output data of the transform unit 208 to the inverse transform unit 211 by bypassing the quantization unit 209 and the inverse quantization unit 210.

[0079] The adder 212 takes the reconstructed residual and the prediction sample of the coding block from the prediction unit 202 as input, calculates the reconstructed sample of the coding block, and puts the reconstructed sample into a buffer (e.g., a picture buffer). For example, the encoder allocates a picture buffer to (temporarily) store the output data of the adder 212. Another method of the encoder is to reserve a special picture buffer in the DPB 214 to save the data from the adder 212.

[0080] The filtering unit 213 performs a filtering operation on the reconstructed picture samples in the decoded picture buffer and outputs the decoded picture. The filtering unit 213 may be composed of one filter or a plurality of cascade filters. For example, according to the H.265 / HEVC standard, the filtering unit is composed of two cascade filters (i.e., a deblocking filter and a sample adaptive offset (SAO) filter). The filtering unit 213 may include an adaptive loop filter (ALF). The filtering unit 213 may also include a neural network filter. When the reconstructed samples of all coding blocks in the picture have been stored in the decoded picture buffer, the filtering unit 213 may start filtering the reconstructed samples of the picture, which may be referred to as "picture layer filtering". Optionally, an alternative implementation of the picture layer filtering of the filtering unit 213 (referred to as "block layer filtering") is that if the reconstructed samples are not used as a reference when encoding all consecutive coding blocks in the picture, the reconstructed samples of the coding blocks in the picture are started to be filtered. Block layer filtering does not require the filtering unit 213 to maintain the filtering operation until all reconstructed samples of the picture are available, thus saving the time delay between threads in the encoder. The filtering unit 213 determines the filtering parameters by calling the RDO method. The output of the filtering unit 213 is the decoded picture samples and the filtering parameters, which include the indication information of the filter, the filter coefficients, the filter control parameters and the like.

[0081] The encoder stores the decoded picture from the filtering unit 213 in the DPB 214. The encoder may determine one or more instructions to be applied to the DPB 214, which are used to control operations on the pictures in the DPB 214, such as the time length of the pictures stored in the DPB 214, outputting the pictures from the DPB 214, etc. In the present disclosure, such instructions are referred to as output parameters of the DPB 214.

[0082] The entropy coding unit 215 performs binarization and entropy coding on one or more coding parameters of the picture, which converts the value of the coding parameter into a codeword consisting of binary symbols "0" and "1", and writes the codeword into a bitstream according to a specification or standard. The coding parameters can be divided into texture data and non-texture data. Texture data is the transformation coefficient of the coding block, and non-texture data is other data in the coding parameters except texture data, including output parameters of units in the encoder, parameter sets, headers, supplementary information, etc. The output of the entropy coding unit 215 is a bitstream that conforms to the specification or standard.

[0083] The entropy coding unit 215 encodes the prediction region flag in the output of the prediction unit 202. The entropy coding unit 215 encodes the prediction region flag and writes its coded bits into the data unit containing the header of the picture region. Figure 7A-7B An example of a syntax structure in a bitstream is shown, where Figure 7A-7BThe syntax in bold is a syntax element represented by a string of one or more bits present in the bitstream, and u(1) and ue(v) are two decoding methods whose functions are the same as those in public standards such as H.264 / AVC and H.265 / HEVC. In the present disclosure, a picture region may be a tile group, a tile, a slice, or a slice group. The entropy coding unit 215 sets the prediction region flag (i.e., Figure 7A-7B ) and other syntax elements conditional on picture_region_not_skip_flag. Note also that Figure 7A-7B There are some syntax elements in that are coded independently of the value of picture_region_not_skip_flag.

[0084] exist Fig. 7A In the picture_region_layer_rbsp(), picture_region_layer_rbsp() is a data unit containing coded bits of a picture region. picture_region_header() is a header of a picture region. A picture region flag (picture_region_not_skip_flag) is encoded in picture_region_header(). picture_region_data() contains coded bits of a coded block in a picture. In this example, when picture_region_not_skip_flag is equal to a second value (e.g., "0"), picture_region_data() is not present in picture_region_layer_rbsp(). For example, when the encoder determines that the value of picture_region_not_skip_flag is equal to 1, the encoder encodes the coded block in the picture region, and the entropy coding unit 215 writes one or more coded bits of the coded block to the bitstream; otherwise, when the encoder determines that the value of picture_region_not_skip_flag is equal to 0, the encoder skips encoding the coded block in the picture region, and the entropy coding unit 215 skips writing the coded bits of the coded block to the bitstream.

[0085] exist Figure 7B The semantics of the syntax elements in the picture region header are as follows.

[0086] Specifies that the parameter set being used specifies the value of the parameter set identifier.

[0087] picture_region_address() contains a syntax element representing the address of a picture region. For example, picture_region_address may be the address of the first coding block in the picture region. In addition, if the picture region is a tile group, picture_region_address may be the tile address of the first tile in the tile group.

[0088] Specifies the encoding type for the image region.

[0089] For example, picture_region_type equal to 0 indicates a "B" picture region, picture_region_type equal to 1 indicates a "P" picture region, and picture_region_type equal to 2 indicates an "I" picture region, where "B", "P" and "I" represent the same meanings as in H.264 / AVC and H.265 / HEVC.

[0090] Specifies the picture order count modulo MaxPicOrderCntLsb for the current picture.

[0091] Equal to 0 g specifies that the picture region is skipped. picture_region_not_skip_flag equal to 1 specifies that the picture region is not skipped.

[0092] When picture_region_not_skip_flag is equal to 0, the bits of the coding blocks in this picture region are not presented in the bitstream. The reconstructed values ​​of the coding blocks in this picture region are set equal to the corresponding prediction values ​​derived by the prediction unit 202 .

[0093] reference_picture_list() contains syntax elements for deriving a reference list for a picture region.

[0094] When picture_region_not_skip_flag is equal to 0, the reference picture may be used to derive a prediction value by the prediction unit 202. If the prediction unit 202 adopts a method of setting the prediction value of the picture region having picture_region_not_skip_flag equal to 0 to a fixed value or a predetermined value, when picture_not_skip_flag is equal to 0, reference_picture_list() does not exist in the syntax structure.

[0095] Example 2

[0096] Figure 8 is a diagram showing a decoder that uses the method in the present disclosure to decode the bit stream generated by the above encoder in Embodiment 1. The input of the decoder is the bit stream, and the output of the decoder is a decoded video or picture obtained by decoding the bit stream.

[0097] The parsing unit 801 in the decoder parses the input bit stream. The parsing unit 801 uses the entropy decoding method and the binarization method specified in the standard to convert each codeword in the bit stream consisting of one or more binary symbols (i.e., "0" and "1") into the numerical value of the corresponding parameter. The parsing unit 801 also derives the parameter value according to one or more available parameters. For example, when there is a flag in the bit stream indicating that the decoded block is the first block in the picture, the parsing unit 801 sets the address parameter indicating the address of the first decoded block of the picture area to 0.

[0098] In the input bit stream of the parsing unit 801, the grammatical structure of the picture area is as follows: Figure 7A-7B shown.

[0099] Figure 7A-7B is a diagram showing an example of a syntax structure in a bit stream, wherein Figure 7A-7B The syntax in bold is a syntax element represented by a string of one or more bits present in the bitstream, and u(1) and ue(v) are two decoding methods whose functions are the same as those in public standards such as H.264 / AVC and H.265 / HEVC. In the present disclosure, a picture region may be a tile group, a tile, a slice, or a slice group. The parsing unit 801 obtains the prediction region flag (i.e., the prediction region flag) according to the value of picture_region_not_skip_flag. Figure 7A-7B ) and other syntax elements conditional on picture_region_not_skip_flag. Note also that Figure 7A-7B There are some syntax elements in that are coded independently of the value of picture_region_not_skip_flag.

[0100] exist Fig. 7AIn the example, picture_region_layer_rbsp() is a data unit containing coded bits for a picture region. picture_region_header() is a header for a picture region. A picture region flag (picture_region_not_skip_flag) is located in picture_region_header(). picture_region_data() contains coded bits for coded blocks in a picture. In this example, when picture_region_not_skip_flag is equal to a second value (e.g., "0"), picture_region_data() is not present in picture_region_layer_rbsp().

[0101] exist Figure 7B The semantics of the syntax elements in the picture region header are as follows.

[0102] Specifies that the parameter set being used specifies the value of the parameter set identifier.

[0103] picture_region_address() contains a syntax element representing the address of a picture region. For example, picture_region_address may be the address of the first coding block in the picture region. In addition, if the picture region is a tile group, picture_region_address may be the tile address of the first tile in the tile group.

[0104] Specifies the encoding type for the image region.

[0105] For example, picture_region_type equal to 0 indicates a "B" picture region, picture_region_type equal to 1 indicates a "P" picture region, and picture_region_type equal to 2 indicates an "I" picture region, where "B", "P" and "I" represent the same meanings as in H.264 / AVC and H.265 / HEVC.

[0106] Specifies the picture order count modulo MaxPicOrderCntLsb for the current picture.

[0107] Equal to 0 Specifies that a picture region is skipped. picture_region_not_skip_flag equal to 1 specifies that a picture region is not skipped.

[0108] When picture_region_not_skip_flag is equal to 0, the bits of the coding blocks in this picture region are not presented in the bitstream. The reconstructed values ​​of the coding blocks in this picture region are set equal to the corresponding prediction values ​​derived by the prediction unit 802.

[0109] reference_picture_list() contains syntax elements for deriving a reference list for a picture region.

[0110] When picture_region_not_skip_flag is equal to 0, the reference picture can be used to derive the prediction value by the prediction unit 802. If the prediction unit 802 adopts a method of setting the prediction value of the picture region having picture_region_not_skip_flag equal to 0 to a fixed value or a predetermined value, when picture_not_skip_flag is equal to 0, reference_picture_list() does not exist in the syntax structure.

[0111] The parsing unit 801 passes the picture region flag (ie, picture_region_not_skip_flag) of the picture region to other units in the decoder to decode the picture region.

[0112] The parsing unit 801 passes one or more prediction parameters for deriving prediction samples of the decoded block to the prediction unit 802. In the present disclosure, the prediction parameters include output parameters of the partitioning unit 201 and the prediction unit 202 in the aforementioned encoder.

[0113] The parsing unit 801 passes one or more residual parameters for reconstructing the residual of the decoded block to the scaling unit 805 and the transform unit 806. In the present disclosure, the residual parameters include the output parameters of the transform unit 208 and the quantization unit 209, and one or more quantization coefficients (i.e., "levels") output by the quantization unit 209 in the aforementioned encoder.

[0114] The parsing unit 801 passes the filtering parameters to the filtering unit 808 for filtering (eg, in-loop filtering) the reconstructed samples in the picture.

[0115] The prediction unit 802 derives the prediction samples of the decoded blocks in the picture region according to the prediction parameters. The prediction unit 802 is composed of the MC unit 803 and the intra prediction unit 804. The input of the prediction unit 802 may also include the current decoded picture output from the adder 807 (which has not been processed by the filtering unit 808) and the reconstructed part of one or more decoded pictures in the DPB 809. When the picture region flag (i.e., picture_region_not_skip_flag) of the picture region is equal to the first value (i.e., "1"), the prediction unit 802 and other related units in the decoder (such as the scaling unit 805, the transform unit 806) call the process of decoding the decoded blocks in the picture region.

[0116] When the picture region flag of the picture region (i.e., picture_region_not_skip_flag) is equal to the second value (i.e., "0"), if the reference picture exists and the type of the picture region indicates inter prediction (that is, picture_region_type is equal to "B" or "P"), then the prediction unit 802 sets the values of the pixels in the picture region to be equal to the values of the co-located pixels in the reference picture of the picture region, or if the reference picture (e.g., the first picture of the coded video sequence in the decoding order) does not exist or the type of the picture region indicates intra prediction (i.e., picture_region_type is equal to "I"), then the values of the pixels in the picture region are set to be equal to a predetermined value. The reference picture can be the first picture in the reference picture list, for example, the picture indicated by the reference index equal to 0 in reference list 0. Optionally, the reference picture can also be the picture in the reference list with the smallest POC (Picture Order Count) difference among the current coded pictures containing the picture region. Optionally, the reference picture can be the picture indicated by the reference index in the reference list, where the reference index is obtained by the parsing unit 801 by parsing the bits in the data unit of the coded bits containing the picture region in the bitstream. The predetermined value can be a fixed value burned in the encoder and decoder, or is calculated as 1<<(bitDepth–1), where bitDepth is the bit depth value of the pixel sample component, "<<" is the arithmetic left shift operator, and "x<<y" means that the x two's complement integer representation is arithmetically shifted by y binary digits. Optionally, the prediction unit 802 can set the values in the picture region to be equal to the predetermined value regardless of whether the reference picture of the picture region exists. When the picture region flag (i.e., picture_region_not_skip_flag) is equal to the second value, the prediction residual of the coded block in the picture region is set to 0. That is, when the picture region flag (i.e., picture_region_not_skip_flag) is equal to the second value, the values of the reconstructed pixels in the picture region are set to be equal to the predicted values derived by the prediction unit 802, and during the process of decoding the decoded block in the picture region, the decoder does not call the scaling unit 805 and the transformation unit 806.

[0117] When the prediction parameters indicate that the prediction samples of the decoded block are derived using the inter-frame prediction mode, the prediction unit 802 constructs one or more reference picture lists using the same method as the method of the ME unit 204 in the aforementioned encoder. The reference list contains one or more reference pictures from the DPB 809. The MC unit 803 determines one or more matching blocks of the decoded block according to the reference list, reference index and MV indication in the prediction parameters, and obtains the inter-frame prediction samples of the decoded block using the same method as the method in the MC unit 205 in the aforementioned encoder. The prediction unit 802 outputs the inter-frame prediction samples as the prediction samples of the decoded block.

[0118] In particular and optionally, the MC unit 803 can use the current decoded picture containing the decoded block as a reference to obtain the intra-frame prediction sample of the decoded block. In the present disclosure, intra-frame prediction means that only the data in the picture containing the coding block is adopted as a reference for deriving the prediction sample of the coding block. In this case, the MC unit 803 uses the reconstructed part in the current picture, where the reconstructed part comes from the output of the adder 807 and is not processed by the filtering unit 808. For example, the decoder allocates a picture buffer to (temporarily) store the output data of the adder 807. Another method of the decoder is to reserve a special picture buffer in the DPB 809 to save the data from the adder 807.

[0119] When the prediction parameters indicate that the intra prediction mode is used to derive the prediction samples of the decoded block, the prediction unit 802 adopts the same method as the intra prediction unit 206 in the aforementioned encoder to determine the reference samples for the intra prediction unit 804 according to the reconstructed neighboring samples of the decoded block. The intra prediction unit 804 obtains the intra prediction mode (i.e., DC mode, planar mode, or angular prediction mode) and uses the reference samples to derive the intra prediction samples of the decoded block according to the specified process of the intra prediction mode. Note that the same derivation process of the intra prediction mode is implemented in the aforementioned encoder (i.e., the intra prediction unit 206) and the decoder (i.e., the intra prediction unit 804). In particular, if the prediction parameters indicate a matching block (including its position) in the current decoded picture (which contains the decoded block) of the decoded block, the intra prediction unit 804 uses the samples in the matching block to derive the intra prediction samples of the decoded block. For example, the intra prediction unit 804 sets the intra prediction samples to be equal to the samples in the matching block. The prediction unit 802 sets the prediction samples of the decoded block equal to the intra prediction samples output by the intra prediction unit 804 .

[0120] The decoder passes the QP (including luma QP and chroma QP) and the quantized coefficients to the scaling unit 805 for inverse quantization process to obtain the reconstructed coefficients as output. The decoder feeds the reconstructed coefficients from the scaling unit 805 and the transform parameters in the residual parameters (i.e., the transform parameters in the output of the transform unit 208 in the aforementioned encoder) to the transform unit 806. In particular, if the residual parameters indicate that scaling is skipped when decoding the block, the decoder directs the coefficients in the residual parameters to the transform unit 806 by bypassing the scaling unit 805. In particular, when picture_region_not_skip_flag is equal to 0, the decoder bypasses the scaling unit 805.

[0121] The transform unit 806 performs a transform operation on the input coefficients following the transform process specified in the standard. The transform matrix used in the transform unit 806 is the same as the transform matrix used in the inverse transform unit 211 in the encoder described above. The output of the transform unit 806 is the reconstructed residual of the decoded block. In particular, when picture_region_not_skip_flag is equal to 0, the decoder bypasses the scaling unit 805 and sets the reconstructed residual of the decoded block in the picture region (with picture_region_not_skip_flag equal to 0) to be equal to 0.

[0122] Generally, since only the decoding process is specified in the standard, from the perspective of the video coding standard, the processes and related matrices in the decoding process are specified as "transformation process" and "transformation matrix" in the standard text. Therefore, in the present disclosure, the description of the decoder names the unit that implements the transformation process specified in the standard text as a "transformation unit" to be consistent with the standard. However, based on the consideration that the decoding process is regarded as the inverse process of encoding, the unit can always be named as an "inverse transformation unit".

[0123] The adder 807 takes the reconstructed residual in the output of the transform unit 806 and the predicted samples in the output of the prediction unit 802 as input data and calculates the reconstructed samples of the decoded block. The adder 807 stores the reconstructed samples in the picture buffer. For example, the decoder allocates a picture buffer to (temporarily) store the output data of the adder 807. Another method of the decoder is to reserve a special picture buffer in the DPB 809 to save the data from the adder 807.

[0124] The decoder passes the filtering parameters from the parsing unit 801 to the filtering unit 808. The filtering parameters for the filtering unit 808 are the same as the filtering parameters in the output of the filtering unit 213 in the aforementioned encoder. The filtering parameters include one or more indication information of the filter to be used, the filter coefficients and the filtering control parameters. The filtering unit 808 performs filtering processing on the reconstructed samples of the picture stored in the decoded picture cache using the filtering parameters, and outputs the decoded picture. The filtering unit 808 may be composed of one filter or multiple cascade filters. For example, according to the H.265 / HEVC standard, the filtering unit is composed of two cascade filters (i.e., a deblocking filter and a sample adaptive offset (SAO) filter). The filtering unit 808 may include an adaptive loop filter (ALF). The filtering unit 808 may also include a neural network filter. When the reconstructed samples of all coded blocks in the picture have been stored in the decoded picture cache, the filtering unit 808 may start filtering the reconstructed samples of the picture, which may be referred to as "picture layer filtering". Optionally, an alternative implementation for picture-level filtering of the filtering unit 808 (referred to as "block-level filtering") is to start filtering reconstructed samples of a coding block in a picture if the reconstructed samples are not used as reference when decoding all consecutive coding blocks in the picture. Block-level filtering does not require the filtering unit 808 to maintain filtering operations until all reconstructed samples of the picture are available, thus saving time delays between threads in the decoder.

[0125] The decoder stores the decoded picture output by the filtering unit 808 in the DPB 809. In addition, the decoder may perform one or more control operations on the picture in the DPB 809 according to one or more instructions output by the parsing unit 801 (e.g., the time length of the picture stored in the DPB 809, the picture output from the DPB 809, etc.).

[0126] Example 3

[0127] Fig. 9 is a schematic diagram showing an example of an extractor implementing the method of the present disclosure. One of the inputs of the extractor is Figure 2 The extractor generates a bitstream generated by the aforementioned encoder in the extractor. Another input to the extractor is application data indicating one or more target picture regions for extraction. The output of the extractor is a bitstream that can be obtained by Figure 8 The sub-bitstream decoded by the aforementioned decoder in . This sub-bitstream (if further extractable) can also be the input bitstream of the extractor.

[0128] The basic function of the extractor is to form a sub-bitstream from the original bitstream. For example, a user selects an area in a high-resolution video for display on his smartphone, and the smartphone sends application data to a remote device (e.g., a remote server) or an internal processing unit (e.g., a software program installed on the smartphone) to request media data corresponding to the selected area (i.e., the target picture area). The extractor (or an equivalent processing unit) on the remote device or the internal processing unit extracts the sub-bitstream corresponding to the target picture area from the bitstream corresponding to the original high-resolution video. Another example is that the HMD (head-mounted device) detects the viewer's current viewport and requests media data to render the viewport. Similar to the previous example, the HMD also generates application data indicating the area of ​​the final rendering area (i.e., the target picture area) in the video picture that covers the detected viewport, and sends the application data to the remote device or its internal processing unit. The extractor (or an equivalent processing unit) on the remote device or the internal processing unit extracts the sub-bitstream corresponding to the target picture area from the bitstream corresponding to the video covering the rendered viewport.

[0129] In this embodiment, the example input bitstream is a bitstream generated by the aforementioned encoder by encoding a 360-degree omnidirectional video using a cubemap projection. Figure 6 60 is divided into 24 picture regions, wherein a picture region may be a tile group or a tile. Picture regions 600, 601, 606 and 607 correspond to a first surface of a cube map, 602, 603, 608 and 609 correspond to a second surface, 604, 605, 610 and 611 correspond to a third surface, 612, 613, 618 and 619 correspond to a fourth surface, 614, 615, 620 and 621 correspond to a fifth surface, and 616, 617, 622 and 623 correspond to a sixth surface.

[0130] When using viewport-based streaming, in order to Figure 5 Rendering content at the viewport shown, image regions 600, 603, 606, 609, 610, 611, 612, 613, 614, 615, 620 and 621 will be used for rendering, while other image regions (in Figure 6 gray in the figure) do not need to be used for rendering.

[0131] The parsing unit 901 parses the input bitstream to obtain a picture region parameter from one or more data units (eg, parameter set data units) in the input bitstream. The picture region parameter indicates Figure 6The parsing unit 901 puts the picture region parameters and other necessary data (such as picture width and height) for determining the target picture region for extraction into the data stream 90 , and sends the data stream 90 to the control unit 902 .

[0132] Note that data flow in the present disclosure refers to input parameters and return parameters of a function in a software implementation, data transmission on a bus, and data sharing between storage units (including data sharing between registers) in a hardware implementation.

[0133] The parsing unit 901 also parses the input bitstream and forwards other data to the forming unit 903 via the data stream 91 in the process of generating a sub-bitstream if necessary. The parsing unit 901 also includes the input bitstream in the data stream 91 .

[0134] The control unit 902 obtains the target picture area from the application data input thereto, which includes the position and size of the target picture area in the picture. The control unit 902 obtains the picture area parameters and the width and height of the picture according to the data stream 90. The control unit 902 determines the address and size of the picture area located in the target picture area according to the picture area parameters. In this example, the control unit 902 determines that the target picture area includes picture areas 600, 603, 606, 609, 610, 611, 612, 613, 614, 615, 620 and 621. The control unit 902 places the target picture area parameters indicating the above picture areas (e.g., the addresses of the picture areas in the target picture area) into the data stream 92.

[0135] The forming unit 903 receives the data streams 91 and 92, extracts the data unit corresponding to the picture area in the target picture area from the input bit stream forwarded in the data stream 91, and generates a new data unit for the picture area outside the target picture area. The forming unit 903 includes an extracting unit 904 and a generating unit 905. When the extracting unit 904 detects the data unit of the picture area in the target picture area (for example, according to the address of the picture area), the extracting unit 904 extracts the data unit. Figure 6 For example, the extraction unit 904 extracts data units of the picture areas 600, 603, 606, 609, 610, 611, 612, 613, 614, 615, 620 and 621 to form a sub-bitstream.

[0136] The generating unit 905 generates a new data unit for the picture area outside the target picture area, and inserts the new data unit into the sub-bitstream. Figure 7BThe value of picture_region_not_skip_flag for the picture region outside the target picture region is set to 0. The generation unit 905 inserts the new data unit into the same access unit, and the bitstream includes the data unit of the picture region in the target picture region. According to the syntax structure in FIG. 7 , the generation unit 905 does not generate bits for the coding blocks in the picture region outside the target picture region. That is, there are no bits for the coding blocks in the picture region outside the target picture region in the sub-bitstream.

[0137] The forming unit 903 appends the parameter sets (and other associated data units) from the input bitstream in the data stream 91 to the sub-bitstream according to the specified bitstream structure of the video coding standard. The output of the forming unit 903 is the sub-bitstream, which can be obtained by Figure 8 Decode with the aforementioned decoder in .

[0138] Furthermore, since the sub-bitstream in this example contains more than one picture region, the sub-bitstream is still extractable and can be the input of an extractor where the target set of picture regions covers a smaller viewport.

[0139] In the extractor, there is no need to use a frame-based method for rearrangement operations. After extraction, the geometric mapping relationship between the projected picture and the sphere used for rendering the 360-degree omnidirectional video remains unchanged. The server containing the extractor does not need to generate and send additional metadata specifying the rearrangement position of the frame-based method, which also saves the additional transmission bandwidth consumed by sending the metadata. The user device does not need to be equipped with the ability to process such metadata and additional storage resources, and the picture area in the compressed frame is remapped by the frame-based method to obtain the geometric mapping relationship for rendering.

[0140] Example 4

[0141] Fig.10 It shows that at least Figure 2 A schematic diagram of a first example device of an example video encoder or picture encoder is shown.

[0142] The acquisition unit 1001 captures videos and pictures. The acquisition unit 1001 may be equipped with one or more cameras for taking videos or pictures of natural scenes. Optionally, the acquisition unit 1001 may be implemented with a camera to obtain a depth video or a depth picture. Optionally, the acquisition unit 1001 may include components of an infrared camera. Optionally, the acquisition unit 1001 may be configured with a remote sensing camera. The acquisition unit 1001 may also be a device or apparatus that generates a video or picture by scanning an object using radiation.

[0143] Optionally, the acquisition unit 1001 can perform preprocessing on the video or picture, such as automatic white balance, auto focus, automatic exposure, backlight compensation, sharpening, denoising, stitching, upsampling / downsampling, frame rate conversion, virtual view synthesis, etc.

[0144] The acquisition unit 1001 may also receive a video or picture from another device or processing unit. For example, the acquisition unit 1001 may be a component unit in a transcoder. The transcoder feeds one or more decoded (or partially decoded) pictures to the acquisition unit 1001. Another example is that the acquisition unit 1001 obtains the video or picture from another device via a data link to the device.

[0145] Note that the acquisition unit 1001 can be used to capture other media information besides videos and pictures, such as audio signals. The acquisition unit 1001 can also receive artificial information, such as characters, texts, videos or pictures generated by computers, and the like.

[0146] Encoder 1002 is Figure 2 The example encoder shown or Fig. 9 The input of the encoder 1002 is the video or picture output by the acquisition unit 1001. The encoder 1002 encodes the video or picture and outputs the generated video or picture bit stream.

[0147] The storage / transmission unit 1003 receives a video or picture bitstream from the encoder 1002 and performs system layer processing on the bitstream. For example, the storage / transmission unit 1003 encapsulates the bitstream according to the transmission standard and the media file format (e.g., MPEG-2TS, ISOBMFF, DASH, MMT, etc.). The storage / transmission unit 1003 stores the transport stream or media file obtained after encapsulation in the memory or disk of the first example device, or transmits the transport stream or media file through a wired or wireless network.

[0148] Note that in addition to the video or picture bitstream from the encoder 1002, the input of the storage / transmission unit 1003 may also include audio, text, images, graphics, etc. The storage / transmission unit 1003 generates a transmission or media file by encapsulating these different types of media bitstreams.

[0149] The first example device described in this embodiment may be a device capable of generating or processing a video (or picture) bit stream in a video communication application, such as a mobile phone, a computer, a media server, a portable mobile terminal, a digital camera, a broadcasting device, a CDN (content distribution network) device, a surveillance camera, a video conferencing device, etc.

[0150] Example 5

[0151] Fig.11 It shows that at least Figure 8 A diagram of a second example device of an example video decoder or picture decoder is shown.

[0152] The receiving unit 1101 receives a video or picture bit stream by acquiring the bit stream from a wired or wireless network, by reading a memory or a disk in an electronic device, or by acquiring data from other devices via a data link.

[0153] The input of the receiving unit 1101 may also include a transport stream or a media file containing a video or picture bitstream. The receiving unit 1101 extracts the video or picture bitstream from the transport stream or the media file according to the specification of the transport or media file format.

[0154] The receiving unit 1101 outputs the video or picture bitstream and passes it to the decoder 1102. Note that in addition to the video or picture bitstream, the output of the receiving unit 1101 may also include an audio bitstream, characters, text, images, graphics, etc. The receiving unit 1101 passes the output to the corresponding processing unit in the second example device. For example, the receiving unit 1101 passes the output audio bitstream to the audio decoder in the device.

[0155] Decoder 1102 is Figure 8 The exemplary decoder implementation is shown in FIG. 1. The input of the decoder 1102 is the video or picture bitstream output by the receiving unit 1101. The decoder 1102 decodes the video or picture bitstream and outputs the decoded video or picture.

[0156] The rendering unit 1103 receives the decoded video or picture from the decoder 1102. The rendering unit 1103 presents the decoded video or picture to the audience. The rendering unit 1103 can be a component of the second example device, for example, a screen. The rendering unit 1103 can also be a device separate from the second example device (which has a data link to the second example device), for example, a projector, a monitor, a television, etc. Optionally, the rendering unit 1103 performs post-processing on the decoded video or picture before presenting it to the audience, for example, automatic white balance, automatic focus, automatic exposure, backlight compensation, sharpening, denoising, splicing, upsampling / downsampling, frame rate conversion, virtual view synthesis, etc.

[0157] Note that in addition to the decoded video or picture, the input of the rendering unit 1103 can be other media data from one or more units of the second example device, such as audio, characters, text, images, graphics, etc. The input of the rendering unit 1103 can also include artificial data, such as lines and marks drawn on a slide by a local teacher to attract attention in a distance education application. The rendering unit 1103 combines different types of media and then presents the combination to the audience.

[0158] The second example device described in this embodiment can be a device capable of decoding or processing video (or picture) bit streams in video communication applications, such as mobile phones, computers, set-top boxes, televisions, HMDs, monitors, media servers, portable mobile terminals, digital cameras, broadcasting equipment, CDN (content distribution network) equipment, monitoring, video conferencing equipment, etc.

[0159] Example 6

[0160] Fig.12 It shows that Fig.10 The first example device and Fig.11 A diagram of an electronic system of a second example device in FIG.

[0161] Service device 1201 is Fig.10 The first example device in.

[0162] The storage medium / transmission network 1202 may include internal memory resources of a device or electronic system, external memory resources accessible via a data link, and a data transmission network composed of wired and / or wireless networks. The storage medium / transmission network 1202 provides storage resources or a data transmission network for the storage / transmission unit 1003 in the service device 1201.

[0163] The destination device 1203 is Fig.11 The receiving unit 1101 in the destination device 1203 receives a video or picture bitstream, a transport stream containing a video or picture bitstream, or a media file containing a video or picture bitstream from a storage medium / transmission network 1202 .

[0164] The electronic system described in this embodiment can be a device or system that can generate, store or transmit video (or picture) bit streams and decode them in video communication applications, such as mobile phones, computers, IPTV systems, OTT systems, multimedia systems on the Internet, digital television broadcasting systems, video surveillance systems, portable mobile terminals, digital cameras, video conferencing systems, etc.

[0165] In the embodiment, specific examples in this embodiment may refer to the examples and exemplary implementation methods described in the above embodiment, and will not be explained in this embodiment.

[0166] Obviously, those skilled in the art should know that each module or each action of the present disclosure can be implemented by a general-purpose computing device, and these modules or actions can be concentrated on a single computing device, or distributed on a network formed by multiple computing devices, and can be optionally implemented by an executable program code for the computing device, so that these modules or actions can be stored in a storage device to be executed together with the computing device, and the actions shown or described can be executed in an order different from the order shown or described here in some cases, or each integrated circuit module can be formed separately, or multiple modules or actions therein can form a single integrated circuit module for implementation. Therefore, the present disclosure is not limited to any specific combination of hardware and software.

[0167] Figure 1A 1 is a flowchart of an example method 100 for bitstream processing. The method 100 includes parsing a bitstream to obtain a picture region flag from a data unit corresponding to a picture region in the bitstream (102), wherein the picture region includes N picture blocks, where N is an integer; and selectively generating a decoded representation of the picture region from the bitstream based on a value of the picture region flag (104). The selective generation step includes: generating a decoded representation from the bitstream using a first decoding method when the value of the picture region flag is a first value (106); and generating a decoded representation from the bitstream using a second decoding method different from the first decoding method when the value of the picture region flag is a second value different from the first value (108). The number N of picture blocks may be greater than 1. For example, the method 100 may be able to effectively decode multiple picture blocks (e.g., coding units CU).

[0168] Method 100 may be performed as follows Fig.11 Such a device may be included as part of a user device such as a smartphone, computer, tablet computer, or any other device capable of processing or displaying digital video content.

[0169] In some embodiments, the type of the picture region may be indicated as an inter-frame prediction coding region. Inter-frame prediction may include unidirectional (forward or predictive) prediction or bidirectional prediction (forward and backward). In this case, the second decoding method may include setting the value of the pixel in the picture region to be equal to the value of the co-located pixel in the reference picture of the picture region.

[0170] In some embodiments, the type of the picture region indicates inter prediction and the reference picture is not present, and wherein the second decoding method comprises setting the value of the pixel in the picture region equal to a predetermined value.

[0171] In some embodiments, the type of the picture region indicates intra prediction, and wherein the second decoding method comprises setting the values ​​of the pixels in the picture region to predetermined values.

[0172] In some embodiments, the first decoding method includes intra-frame decoding or inter-frame decoding using corresponding bits from the bitstream.

[0173] In some embodiments, a picture region may include picture blocks encoded using different encoding techniques. For example, a first picture block in a picture region is encoded using a different encoding mode than a second picture block in the picture region. Here, the encoding mode may be, for example, an inter-frame prediction encoding mode or an intra-frame prediction encoding mode.

[0174] Figure 1B The present invention is a flow chart of a method 150 for visual information processing. The method 150 includes parsing a bitstream to obtain a picture region parameter from a parameter set data unit in the bitstream (152), wherein the picture region parameter indicates that a picture is divided into one or more picture regions; determining one or more picture regions located in the target picture region according to a target picture region (154); extracting one or more data units corresponding to the one or more picture regions located in the target picture region from the bitstream to form a sub-bitstream (156); generating a first data unit corresponding to an external picture region outside the target picture region (158), and setting a picture region flag in the first data unit to be equal to a first value, the first value indicating that: in the external picture region, no bits in the bitstream for the coding block are coded (160); and inserting the first data unit into the sub-bitstream (162).

[0175] Method 150 may be performed as follows Fig.10 The device may be implemented in a smartphone, a laptop, a computer or another device for encoding video.

[0176] In some embodiments, the one or more image regions include a non-rectangular image region. In some embodiments, the target image region is based on a user viewport. In some embodiments, the external image region corresponds to an image region outside of an area visible to a user viewport.

[0177] Regarding methods 100 and 150, the segmentation unit 201 may be used in the step of parsing the bitstream (102 or 152). Embodiment 3 described in the present application may also be used to implement the parsing step to extract picture region parameters and extract data units from the bitstream and generate the first data unit.

[0178] Figure 1C is a flow chart of an example method 180 for processing a video or picture to generate a corresponding coded or compressed domain bitstream representation.

[0179] Method 180 may be performed as follows Fig.10 The device may be implemented in a smartphone, a laptop, a computer or another device for encoding video.

[0180] The method 180 includes dividing a picture into one or more picture regions, wherein the picture region includes N picture blocks, where N is an integer (182), and selectively generating a bitstream from the N picture blocks based on a coding standard (184). The selective generation (184) includes, when the coding standard is to encode the picture region, encoding a picture region flag corresponding to the picture region as a first value and encoding the picture blocks in the picture region using a first coding method (186), and when the coding standard is not to encode the picture region, then encoding the picture region flag corresponding to the picture region as a second value and encoding the picture region using a second coding method different from the first coding method (188).

[0181] For example, the segmentation unit 201 may be used to perform the segmentation step 182 and the steps 184, 186 or 188. For example, the entropy coding unit 215 may be used to encode the picture region flag in the bitstream.

[0182] In various embodiments, the first and second encoding methods may include intra-frame encoding or predictive encoding (unidirectional or bidirectional). In some embodiments, the picture region may include multiple picture blocks (e.g., N is greater than 1). Figure 5 As described above, during implementation of method 180 , the user's viewport may be used to determine how to encode and which image blocks to encode.

[0183] exist Figure 1A and 1C In the embodiment, steps 106, 108, 186, 188 are shown with dashed outlines because, according to some embodiments, only one of these two steps will be implemented for the encoding or decoding of a particular picture area. Typically, during an encoding or decoding operation of a video, one or the other step will be implemented, for example depending on the content details. However, it is also possible that some areas of a video or image may be encoded without using the encoding or decoding method described in detail. Figures 1A-1C The encoding technique described is used to encode the case.

[0184] In some embodiments, the video encoder apparatus may include a processor configured to implement method 180. The processor may include, or may control and use dedicated video encoding circuitry configured to perform operations such as those related to Figure 2 Those functions described.

[0185] In some embodiments, a video decoding or transcoding device may be used to implement method 100 or 150. Figure 8 The described apparatus may be used in the embodiments.

[0186] It should be understood that the techniques described in the present application can be incorporated into a video encoder device or a video decoder device to significantly improve the performance of operations for encoding or decoding video. For example, some video applications (such as virtual reality experiences or games) require real-time (or faster than real-time) encoding or decoding of video to provide a satisfactory user experience. The disclosed technology improves the performance of such applications by using picture region-based encoding or decoding techniques as described herein. For example, less than all portions of a video frame are encoded or decoded based on the user's viewpoint to allow only the video that the user will watch to be selectively encoded. In addition, picture blocks are reorganized to create picture regions in rectangular video frames to allow the use of standard rectangular frame-based video encoding tools (such as motion search, transforms, and quantization).

[0187] The above are only preferred embodiments of the present disclosure and are not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the scope of the principles of the present disclosure shall fall within the scope of protection defined by the claims attached to the present disclosure.

[0188] Industrial Applicability

[0189] From the above description, it can be known that the problem of the additional computational burden of the viewport-based streaming media in the prior art is solved, and the effect of effectively encoding the picture area skipped in the encoding is further implemented. All the shortcomings of the prior art are solved by using the aforementioned encoder to generate the original bitstream, using the extractor in this example embodiment to obtain the sub-bitstream, and using the aforementioned decoder to decode the bitstream (and the sub-bitstream).

[0190] Fig.14An example apparatus 1400 is shown that can be used to implement the encoder side or decoder side techniques described in the present application. The apparatus 1400 includes a processor 1402, which can be configured to perform the encoder side or decoder side techniques or both. The apparatus 1400 may also include a memory (not shown) for storing processor executable instructions and for storing video bitstreams and / or display data. The apparatus 1400 may include video processing circuits (not shown), such as transform circuits, arithmetic encoding / decoding circuits, data encoding techniques based on lookup tables, etc. The video processing circuits may be partially included in the processor and / or partially included in other dedicated circuits such as graphics processors, field programmable gate arrays (FPGAs), etc.

[0191] Device

[0192] The embodiments disclosed in this application and other embodiments, modules and functional operations described can be implemented in digital electronic circuits, or in computer software, firmware or hardware, including the structures disclosed in this application and their structural equivalents, or implemented in a combination of one or more of them. The disclosed embodiments and other embodiments can be implemented as one or more computer program products, that is, one or more modules of computer program instructions encoded on a computer-readable medium, for execution by a data processing device or control of the operation of the data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a storage device, a composition that affects a machine-readable propagation signal, or a combination of one or more of them. The term "data processing device" covers all devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. In addition to hardware, the device may also include code that creates an execution environment for the computer program in question (e.g., code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them). A propagation signal is an artificially generated signal, for example, a machine-generated electrical signal, an optical signal, or an electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device.

[0193] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple collaborative files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0194] The processes and logic flows described in this application can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, and the apparatus can also be implemented as special purpose logic circuits, for example, FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits).

[0195] Processors suitable for executing computer programs include, for example, general-purpose microprocessors and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include or be operably coupled to one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data to receive data from or transfer data to, or both. However, a computer need not have such a device. Computer-readable media suitable for storing computer program instructions and data include various forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM optical disks. The processor and memory may be supplemented by or incorporated into a dedicated logic circuit.

[0196] Although this patent application contains many details, these should not be interpreted as limitations on the scope of any invention or the scope of what may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this patent application in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although the above-mentioned features may be described as working in certain combinations and even initially claimed as such, in some cases, one or more features from the combination may be deleted from the combination, and the combination may involve sub-combinations or variations of sub-combinations.

[0197] Similarly, although operations are described in a particular order in the drawings, this should not be understood as requiring that the operations be performed in the particular order or sequence shown, or that all of the operations shown be performed, in order to achieve the desired results. In addition, the separation of various system components in the embodiments described in this patent application should not be understood as requiring such separation in all embodiments.

[0198] Only a few implementations and examples are described, and other implementations, enhancements, and variations may be made based on what is described and illustrated in this patent application.

Claims

1. A bit stream processing method, the method comprising: Parsing a bitstream to obtain a first picture region flag from a first data unit corresponding to a picture region in the bitstream, wherein the picture region includes N picture blocks, where N is an integer; and selectively generating a decoded representation of the picture region from the bitstream based on a value of the first picture region flag; Wherein, the selective generation includes: generating the decoded representation from the bitstream using a first decoding method in case the value of the first picture region flag is a first value; and generating the decoded representation from the bitstream using a second decoding method different from the first decoding method, in case the value of the first picture region flag is a second value different from the first value, wherein the type of the picture region indicates intra prediction, and The second decoding method includes setting the value of the pixel in the picture area to a predetermined value.

2. The method according to claim 1, wherein: The type of the picture region indicates inter prediction, and wherein the second decoding method comprises setting the value of a pixel in the picture region equal to the value of a co-located pixel in a reference picture of the picture region.

3. The method according to claim 1, wherein: The type of the picture area indicates inter-frame prediction, and a reference picture does not exist, and wherein the second decoding method includes setting the value of the pixel in the picture area equal to a predetermined value.

4. The method according to any one of claims 1 to 3, wherein: The first decoding method includes using intra-frame decoding or inter-frame decoding of corresponding bits from the bitstream.

5. The method according to any one of claims 1 to 3, wherein: N is greater than 1.

6. The method according to claim 5, wherein: A first picture block in the picture region is encoded using a coding mode different from a coding mode of a second picture block in the picture region, wherein the coding mode is an inter-frame prediction coding mode or an intra-frame prediction coding mode.

7. The method according to claim 1, further comprising: Parsing the bitstream to obtain a picture region parameter from a parameter set data unit in the bitstream, wherein the picture region parameter indicates that a picture is divided into one or more picture regions; According to the target picture area, determining one or more picture areas located in the target picture area; extracting one or more data units corresponding to the one or more picture regions located in the target picture region from the bitstream to form a sub-bitstream; generating a second data unit corresponding to an external picture region outside the target picture region, and setting a second picture region flag in the second data unit to be equal to a third value, the third value indicating that no bits are encoded in the bitstream for a coding block in the external picture region; and The second data unit is inserted into the sub-bitstream.

8. The method according to claim 7, wherein: The one or more picture regions include a non-rectangular picture region.

9. The method according to claim 7 or 8, wherein: The target image area is based on the user viewport.

10. The method according to claim 7 or 8, wherein: The external image area corresponds to an image area outside the area visible in the user's viewport.

11. A coding method for processing a video or a picture, the method comprising: Divide the image into one or more image regions, wherein the image region contains N image blocks, where N is an integer; selectively generating a bitstream from the N picture blocks based on a coding standard, Wherein, the selective generation includes: In a case where the coding standard is to encode the picture region, encoding a picture region flag corresponding to the picture region as a first value, and encoding a picture block in the picture region using a first coding method; and In a case where the coding standard is not to encode the picture area, the picture area flag corresponding to the picture area is encoded as a second value, and the picture area is encoded using a second coding method different from the first coding method, wherein the type of the picture region indicates intra prediction, and The second encoding method includes setting the values ​​of pixels in the picture area to predetermined values.

12. The method according to claim 11, wherein: The first encoding method includes intra-frame encoding.

13. The method according to claim 11, wherein: The first encoding method encodes the N picture blocks and writes the encoded bits of the N picture blocks into a bitstream.

14. The method according to claim 11, wherein: The second encoding method skips encoding the N picture blocks and writes the encoded bits of the N picture blocks into a bitstream.

15. The method according to any one of claims 11 to 14, wherein: N is greater than 1.

16. The method according to any one of claims 11 to 14, wherein: The encoding standard depends on the current viewport information of the picture.

17. A video encoder apparatus comprising a processor configured to implement the method according to any one of claims 11-16.

18. A video decoder apparatus comprising a processor configured to implement the method according to any one of claims 1-10.

19. A computer program product having code stored thereon which, when executed by a processor, causes the processor to carry out the method according to any one of claims 1 to 16.

Images