Method, apparatus and computer-readable storage medium for video decoding

By using SEI messages in video encoding and decoding, information of independent regions is provided, and the problem of low output efficiency of independent regions in the prior art is solved, early decoding and efficient resource utilization are realized, and it is suitable for applications such as 360VR.

CN116264863BActive Publication Date: 2024-11-15TENCENT AMERICA LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202280005834.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2022-06-28
Filing Date
2022-06-29
Publication Date
2024-11-15
Estimated Expiration
2042-06-29

AI Technical Summary

Technical Problem

When existing video encoding technology processes independent area output, it is difficult to efficiently decode and output independent areas, resulting in waste of resources and low encoding efficiency.

Method used

By using Supplementary Enhanced Information (SEI) messages during video encoding and decoding, offset information and size information of independent regions are provided, allowing the processing circuit to decode and output independent regions, independent of the decoding state of the entire image.

Benefits of technology

It realizes early access and decoding of independent areas, reduces resource consumption, improves coding efficiency, and supports partial area access for applications such as 360VR.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116264863B_ABST
    Figure CN116264863B_ABST
Patent Text Reader

Abstract

Aspects of the present disclosure provide methods, devices, and non-transitory computer-readable storage media for video decoding. The device includes a processing circuit configured to decode a supplemental enhancement information (SEI) message indicating independent region output information. The independent region output information indicates an independently decodable independent region in a current image. The processing circuit determines an independent region in the current image based on offset information and dimension information of the independent region in the independent region output information. The processing circuit decodes the determined independent region regardless of whether the entire current image is decoded. The processing circuit outputs the decoded independent region based on the position indicated by the offset information of the independent region and the size indicated by the dimension information.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Incorporation by Reference

[0002] This application claims priority to U.S. Patent Application No. 17 / 851,723, filed on June 28, 2022, for "Independently Encoded Region Output Supplemental Enhancement Information Messages." This U.S. patent claims priority to U.S. Provisional Application No. 63 / 216,443, filed on June 29, 2021, for "Techniques for Independently Encoded Region Output of Encoded Video Streams." The entire disclosure of the prior application is incorporated herein by reference. Technical Field

[0003] This disclosure describes embodiments generally related to video encoding. Background Art

[0004] The background description provided herein is to generally present the context of the present disclosure. Work by the presently named inventors within the scope of the work described in this background section and aspects of the description that may not be prior art at the time of filing are neither explicitly nor implicitly admitted to be prior art against the present disclosure.

[0005] Picture and / or video encoding and decoding can be performed using inter-frame image prediction with motion compensation. Uncompressed digital pictures and / or videos can include a series of images, each image having spatial dimensions of, for example, 1920×1080 luminance samples and associated chrominance samples. The series of images can have a fixed or variable image rate (also informally referred to as a frame rate), such as 60 images per second or 60Hz. Uncompressed pictures and / or videos have specific bit rate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (1920×1080 luminance sample resolution at 60Hz frame rate) requires a bandwidth of nearly 1.5Gbit / s. One hour of such video requires more than 600G bytes of storage space.

[0006] The purpose of picture and / or video encoding and decoding can be to reduce redundancy in input picture and / or video signal by compression. Compression helps to reduce the above-mentioned bandwidth and / or storage space requirements, which can be reduced by two orders of magnitude or more in some cases. Although the description herein uses video encoding / decoding as an illustrative example, the same technology can be applied to picture encoding / decoding in a similar manner without departing from the spirit of the present disclosure. Lossless compression and lossy compression can be used, as well as a combination of them. Lossless compression refers to a technology that can reconstruct an exact copy of the original signal from the compressed original signal. When lossy compression is used, the reconstructed signal may be different from the original signal, but the distortion between the original signal and the reconstructed signal is small enough so that the reconstructed signal can be used for the intended application. In terms of video, lossy compression is widely used. The amount of distortion that can be tolerated depends on the application; for example, users of some consumer streaming applications can tolerate higher distortion than users of TV step-by-step applications. The compression rate that can be achieved can reflect that higher allowable / tolerable distortion can produce higher compression rates.

[0007] Video encoders and decoders may utilize techniques from several broad categories including, for example, motion compensation, transforms, quantization, and entropy coding.

[0008] Video codec techniques may include techniques known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference images. In some video codecs, an image is spatially subdivided into blocks of samples. When all blocks of samples are encoded in intra-mode, the image may be an intra-image. Intra-images and their derivatives, such as independent decoder refresh images, may be used to reset the decoder state, so that intra-images and their derivatives may be used as the first image in an encoded video stream and video session, or as a still image. Samples of an intra-block may be transformed, and the transform coefficients may be quantized before entropy coding. Intra-frame prediction may be a technique for minimizing sample values ​​in a pre-transform domain. In some cases, the smaller the transformed DC value and the smaller the AC coefficient, the fewer bits are required to represent the block after entropy coding at a given quantization step size.

[0009] Conventional intra-frame coding, such as that known from MPEG-2 generation coding techniques, does not use intra-frame prediction. However, some newer video compression techniques include techniques for trying a data block from, for example, surrounding sample data and / or metadata that are obtained during encoding and / or decoding of spatially adjacent data blocks and before the data block in decoding order. Such techniques are hereinafter referred to as "intra-frame prediction" techniques. Note that, in at least some cases, intra-frame prediction uses only reference data from the current image being reconstructed, and not reference data from reference images.

[0010] Intra-frame prediction can take many different forms. When more than one such technique can be used in a given video coding technique, the techniques used can be encoded in intra-frame prediction mode. In some cases, a mode may have sub-modes and / or parameters, and these sub-modes and / or parameters may be encoded separately or included in a mode codeword. For a given mode, sub-mode, and / or parameter combination, which codeword is used will affect the coding efficiency gain through intra-frame prediction, which is also affected by the entropy coding technique that converts the codeword into a bitstream.

[0011] Certain intra prediction modes were introduced in H.264, improved in H.265, and further improved in newer coding techniques such as joint exploration model (JEM), versatile video coding (VVC), and benchmark set (BMS). The predictor block can be formed using neighboring sample values ​​belonging to already available samples. Sample values ​​of neighboring samples are copied into the predictor block according to the direction. A reference to the direction used can be encoded in the bitstream, or it can itself be predicted.

[0012] refer to Figure 1A , depicted at the bottom right is a subset of 9 predictor directions, of which 33 possible predictor directions are known from H.265 (corresponding to 33 of the 35 intra modes). The point (101) where the arrows converge represents the sample being predicted. The arrows represent the direction in which the sample is being predicted. For example, arrow (102) indicates that sample (101) is predicted based on one or more samples to the upper right that are at an angle of 45 degrees to the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted based on one or more samples to the lower left of sample (101) that are at an angle of 22.5 degrees to the horizontal.

[0013] Still reference Figure 1A, a square block (104) of 4×4 samples is depicted in the upper left corner (indicated by the dashed, bold lines). The block (104) includes 16 samples, each of which is labeled with "S", and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Similarly, sample S44 is the fourth sample of the block (104) in both the Y and X dimensions. Since the size of the block is 4×4 samples, S44 is located in the lower right corner. Reference samples are also shown that follow a similar numbering scheme. The reference samples are labeled with R, and their Y position (e.g., row index) and X position (column index) relative to the block (104). In H.264 and H.265, the prediction samples are adjacent to the block being reconstructed, so there is no need to use negative values.

[0014] Intra-image prediction can be performed by copying reference sample values ​​from neighboring samples occupied by the signaled prediction direction. For example, assume that the encoded video code stream includes signaling that, for the block, indicates a prediction direction consistent with arrow (102), that is, the sample is predicted based on one or more samples to the upper right of the prediction sample whose upper right corner is 45 degrees from the horizontal direction. In this case, samples S41, S32, S23 and S14 are predicted based on the same reference sample R05. Sample S44 is then predicted based on reference sample R08.

[0015] In some cases, the values ​​of multiple reference samples may be combined, such as by interpolation, in order to calculate the reference sample (particularly when the direction is not divisible by 45 degrees).

[0016] As video coding technology develops, the number of possible directions increases. In H.264 (2003), 9 different directions can be represented. In H.265 (2013) and JEM / VVC / BMS, this number increased to 33, and at the time of this disclosure, up to 65 directions can be supported. Experiments have been conducted to identify the most likely directions, and certain techniques in entropy coding are used to represent those possible directions with a small number of bits, with certain penalties for less likely directions. In addition, the direction itself can sometimes be predicted based on the adjacent directions used by adjacent and decoded blocks.

[0017] Figure 1B A schematic diagram (110) showing 65 intra prediction directions according to JEM is shown to illustrate the increasing number of prediction directions over time.

[0018] The mapping of intra-prediction direction bits in the coded video bitstream that represent the direction can be different from video coding technology to video coding technology, and can range from a simple direct mapping of prediction directions to intra-prediction modes, to codewords, to complex adaptive schemes involving most probable modes and similar techniques, for example. However, in all cases, there may be certain directions in the video content that are statistically less likely to occur than certain other directions. Since the goal of video compression is to reduce redundancy, in video coding techniques that work well, those less likely directions will be represented with a larger number of bits than more likely directions.

[0019] Motion compensation may be a lossy compression technique and may involve a technique where a block of samples from a previously reconstructed image or part thereof (reference image) is used to predict a newly reconstructed image or part thereof after being spatially shifted in a direction indicated by a motion vector (MV, hereinafter referred to as MV). In some cases, the reference image may be the same as the image currently being reconstructed. The MV may have two dimensions, X and Y, or three dimensions, the third dimension being an indication of the reference image used (the latter may indirectly be a temporal dimension).

[0020] In some video compression techniques, the MV applicable to a certain area of ​​sample data can be predicted from other MVs, for example, from those MVs in spatially adjacent reconstructions and before the MV in decoding order. Doing so can greatly reduce the amount of data required to encode the MV, thereby eliminating redundancy and increasing compression. MV prediction can work effectively, for example, because when encoding an input video signal derived from a camera (called natural video), there is a statistical likelihood that multiple regions move in similar directions, and the multiple regions are larger than the area applicable to a single MV. Therefore, in some cases, similar motion vectors derived from MVs of neighboring regions can be used for prediction. This results in the MV found for a given area being similar or identical to the MV predicted from the surrounding MVs, and after entropy coding, it can be represented with fewer bits than the number of bits used when the MV is directly encoded. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from the original signal (i.e., sample stream). In other cases, MV prediction itself may be lossy, for example, due to rounding errors when calculating prediction values ​​from some surrounding MVs.

[0021] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms provided by H.265, described herein is a technique referred to below as "spatial merging".

[0022] Referring to Figure 2, the current block (201) includes samples that the encoder has found during motion search that can be predicted based on a previous block of the same size that has been spatially shifted. Instead of encoding the MV directly, the MV can be derived from metadata associated with one or more reference images (e.g., from the nearest reference image (in decoding order)) using an MV associated with any of the five surrounding samples. The five surrounding samples are represented as A0, A1 and B0, B1, B2 (202 to 206, respectively). In H.265, MV prediction can use a predictor from the same reference image that a neighboring block is using. Summary of the invention

[0023] Various aspects of the present disclosure provide methods and apparatus for video encoding and decoding. In some examples, a device for video decoding includes a processing circuit. The processing circuit is configured to decode a supplemental enhancement information (SEI) message indicating independent region output information. The independent region output information may indicate an independent region in a current image that can be independently decoded. The processing circuit is configured to determine an independent region in the current image based on offset information and size information of the independent region in the independent region output information. The processing circuit may decode the determined independent region regardless of whether the entire current image is decoded. The processing circuit may output a decoded independent region based on a position indicated by the offset information of the independent region and a size indicated by the dimension information.

[0024] In one embodiment, the independent region includes a sub-image in the current image.

[0025] In one embodiment, the sub-image is included in another independent region of the current image, and the other independent region also includes another sub-image in the current image.

[0026] In one embodiment, the independent region is included in other independent regions.

[0027] In one embodiment, the independent region output information indicates a sub-image identifier of the sub-image in the independent region.

[0028] The SEI message may include a first flag indicating whether to signal the independent region output information in the SEI message. The processing circuit may determine the independent region based on the first flag indicating that the independent region output information is signaled in the SEI message.

[0029] In one embodiment, the independent region output information includes a second flag. The second flag having a first value indicates that the SEI message is applied only to the current image. The second flag having a second value indicates that the SEI message is applied to the current image and one or more subsequent images of the current layer in output order until a condition is met.

[0030] In one embodiment, the second flag has a second value. The processing circuit may output the independent regions in the subsequent image of the current layer in an output order based on the independent region output information.

[0031] In one embodiment, the independent region output information includes a third flag of the independent region, the third flag indicating whether to signal the cropping information of the independent region in the independent region output information.

[0032] In one embodiment, based on the third flag indicating that the cropping information of the independent region is signaled in the independent region output information, the processing circuit may determine the position and size of the output window of the independent region based on (i) the cropping information and (ii) the offset information and the size information of the independent region. The processing circuit may output the decoded independent region based on the position and size of the output window of the independent region.

[0033] In one example, decoding and outputting for display is performed before the entire current image is decoded.

[0034] Aspects of the present disclosure also provide a non-transitory computer-readable storage medium storing a program, the program being executed by at least one processor to perform the method for video decoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Further features, properties and various advantages of the subject matter of the present disclosure will become more apparent from the following detailed description and accompanying drawings. In the accompanying drawings,

[0036] Figure 1A is a schematic diagram of an exemplary subset of intra prediction modes.

[0037] Figure 1B is a diagram of exemplary intra prediction directions.

[0038] FIG. 2 shows a current block ( 201 ) and surrounding samples according to an embodiment.

[0039] Figure 3 is a schematic diagram of a simplified block diagram of a communication system (300) according to an embodiment.

[0040] Figure 4 is a schematic diagram of a simplified block diagram of a communication system (400) according to an embodiment.

[0041] Figure 5is a schematic diagram of a simplified block diagram of a decoder according to an embodiment.

[0042] Figure 6 is a schematic diagram of a simplified block diagram of an encoder according to an embodiment.

[0043] Figure 7 A block diagram of an encoder according to another embodiment is shown.

[0044] Figure 8 A block diagram of a decoder according to another embodiment is shown.

[0045] Fig. 9A An example of an independently encoded region in a current image and a cropped independent region associated with the independent region according to an embodiment of the present disclosure is shown.

[0046] Fig. 9B An example of an independent region overlapping with another independent region in a current image according to an embodiment of the present disclosure is shown.

[0047] Fig. 9C An example of an independent region of another independent region included in a current image according to an embodiment of the present disclosure is shown.

[0048] Fig.10 An exemplary syntax in a supplemental enhancement information (SEI) message including independent region output information according to an embodiment of the present disclosure is shown.

[0049] Fig.11 An example of the persistence scope of the SEI message according to an embodiment of the present disclosure is shown.

[0050] Fig.12 A flow chart outlining an encoding process according to an embodiment of the present disclosure is shown.

[0051] Fig.13 A flow chart outlining a decoding process according to an embodiment of the present disclosure is shown.

[0052] Fig.14 is a schematic diagram of a computer system according to an embodiment. DETAILED DESCRIPTION

[0053] Figure 3 A simplified block diagram of a communication system (300) according to an embodiment of the present disclosure is shown. The communication system (300) includes a plurality of terminal devices that can communicate with each other via, for example, a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). Figure 3In the example, a first pair of terminal devices (310) and (320) perform unidirectional transmission of data. For example, the terminal device (310) can encode video data (e.g., a video picture stream collected by the terminal device (310)) for transmission to other terminal devices (320) via the network (350). The encoded video data can be transmitted in the form of one or more encoded video streams. The terminal device (320) can receive the encoded video data from the network (350), decode the encoded video data to restore the video image, and display the video image based on the restored video data. Unidirectional data transmission may be common in media service applications, etc.

[0054] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of encoded video data, which bidirectional transmission can occur, for example, during a video conference. For the bidirectional transmission of data, in one example, each of the terminal devices (330) and (340) can encode video data (e.g., a video image stream captured by the terminal device) for transmission to the other terminal devices (330) and (340) via the network (350). Each of the terminal devices (330) and (340) can also receive encoded video data, which is transmitted by the other terminal device in the terminal devices (330) and (340), and can decode the encoded video data to restore the video image, and can display the video image on an accessible display device based on the restored video data.

[0055] exist Figure 3 In the example of , terminal devices (310), (320), (330) and (340) can be shown as servers, personal computers and smart phones, but the principles of the present disclosure may not be limited to this. Embodiments of the present disclosure are applicable to laptop computers, tablet computers, media players and / or dedicated video conferencing equipment. Network (350) represents any number of networks that transmit encoded video data between terminal devices (310), (320), (330) and (340), including, for example, wired (wired) and / or wireless communication networks. Communication network (350) can exchange data in circuit switching and / or packet switching channels. Representative networks include telecommunication networks, local area networks, wide area networks and / or the Internet. For the purposes of this discussion, unless explained below, the architecture and topology of network (350) may be unimportant to the operation of the present disclosure.

[0056] As an example of application of the presently disclosed subject matter, Figure 4The placement of the video encoder and video decoder in a streaming environment is shown. The subject matter of the present disclosure may be equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, and compressed video stored on digital media including CDs, DVDs, memory sticks, etc.

[0057] The streaming system may include an acquisition subsystem (413) that may include a video source (401), such as a digital camera, for creating, for example, an uncompressed video image stream (402). In one example, the video image stream (402) includes samples captured by a digital camera. The video image stream (402) is depicted as a thick line compared to the encoded video data (404) (or encoded video bitstream) to emphasize the high data volume of the video image stream (402) that can be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination of hardware and software to enable or implement aspects of the disclosed subject matter described in more detail below. Compared to the video stream (402), the encoded video data (404) (or the encoded video bitstream) is depicted as a thin line to emphasize the lower amount of data of the encoded video data (404) (or the encoded video bitstream), which can be stored on the streaming server (405) for future use. One or more streaming client subsystems, such as Figure 4 The client subsystems (406) and (408) in the streaming server (405) can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) may include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and creates an output stream of video images (411) that can be presented on a display (412) (e.g., a display screen) or other presentation device (not shown). In some streaming systems, the encoded video data (404), (407) and (409) (e.g., video code streams) can be encoded according to certain video encoding / compression standards. Examples of these standards include ITU-T Recommendation H.265. In one example, the video coding standard under development is informally referred to as Versatile Video Coding (VVC). The subject matter of the present disclosure can be used in the context of VVC.

[0058] It should be noted that the electronic devices (420) and (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0059] Figure 5A block diagram of a video decoder (510) according to an embodiment of the present disclosure is shown. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used instead of Figure 4 A video decoder (410) is shown in an example.

[0060] The receiver (531) can receive one or more encoded video sequences to be decoded by the video decoder (510); in the same or another embodiment, one encoded video sequence is received at a time, wherein decoding of each encoded video sequence is independent of other encoded video sequences. The encoded video sequence can be received from a channel (501), which can be a hardware / software link to a storage device that stores the encoded video data. The receiver (531) can receive the encoded video data and other data, such as encoded audio data and / or auxiliary data streams that can be forwarded to their respective consuming entities (not shown). The receiver (531) can separate the encoded video sequence from the other data. To prevent network jitter, a buffer memory (515) can be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) is part of the video decoder (510). In other cases, the buffer memory (515) can be external to the video decoder (510) (not shown). In other cases, there may be a buffer memory (not shown) external to the video decoder (510) for example to prevent network jitter, and another buffer memory (515) internal to the video decoder (510) for example to handle playout timing. When the receiver (531) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous network, the buffer memory (515) may not be needed or may be very small. For use over a best effort packet network such as the Internet, the buffer memory (515) may be necessary and may be relatively large and may advantageously be of an adaptive size, and may be at least partially implemented in an operating system or similar element (not shown) external to the video decoder (510).

[0061] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. The categories of these symbols include information for managing the operation of the video decoder (510) and potential information for controlling a presentation device such as a presentation device (512) (e.g., a display screen) that is not part of the electronic device (530) but may be used as Figure 5As shown, the control information for one (or more) presentation devices may be in the form of a supplemental enhancement information (SEI) message or a video usability information (VUI) parameter set fragment (not shown). The parser (520) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may be performed according to a video coding technique or standard, and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and the like. The parser (520) may extract a subgroup parameter set in at least one subgroup of video decoder pixels from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a group of pictures (Groups of Picture, GOP), a picture, a tile, a slice, a macroblock, a coding unit (Coding Unit, CU), a block, a transform unit, a prediction unit (Transform Unit, TU), and the like. The parser (520) may also extract information such as transform coefficients, quantizer parameter values, motion vectors, and the like from the coded video sequence.

[0062] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515) to create symbols (521).

[0063] The reconstruction of the symbol (521) may involve a number of different units, depending on the type of the coded video picture or portion of the coded video picture (e.g., inter- and intra-pictures, inter- and intra-blocks) and other factors. Subgroup control information parsed from the coded video sequence by the parser (520) may control which units are involved and how the units are controlled. For the sake of brevity, such subgroup control information flow between the parser (520) and the various units below is not described.

[0064] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into a number of functional units as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and may be at least partially integrated with each other. However, for the purpose of describing the disclosed subject matter, it is appropriate to conceptually subdivide into the functional units below.

[0065] The first unit is a sealer / inverse transform unit (551). The sealer / inverse transform unit (551) receives quantized transform coefficients as one (or more) symbols (521) and control information from the parser (520), including which transform to use, block size, quantization factor, quantization scaling matrix, etc. The sealer / inverse transform unit (551) can output a block including sample values, which can be input into an aggregator (555).

[0066] In some cases, the output samples of the scaler / inverse transform unit (551) may belong to an intra-coded block; that is, prediction information from a previously reconstructed image is not used, but prediction information from a previously reconstructed portion of the current image may be used. Such prediction information may be provided by an intra-image prediction unit (552). In some cases, the intra-image prediction unit (552) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information extracted from the current image buffer (558). The current image buffer (558) buffers, for example, a partially reconstructed current image and / or a fully reconstructed current image. In some cases, the aggregator (555) adds the prediction information already generated by the intra-frame prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a per-sample basis.

[0067] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to an inter-frame coded and potentially motion compensated block. In this case, the motion compensated prediction unit (553) may access the reference image memory (557) to extract samples for prediction. After the extracted samples are motion compensated according to the block-related symbols (521), these samples may be added to the output of the scaler / inverse transform unit (551) (in this case referred to as residual samples or residual signals) by the aggregator (555) to generate output sample information. The retrieval of the prediction samples by the motion compensated prediction unit (553) from the address in the reference image memory (557) may be controlled by a motion vector, which is available to the motion compensated prediction unit (553) in the form of a symbol (521), which may have, for example, X, Y and reference image components. Motion compensation may also include interpolation of sample values ​​extracted from the reference image memory (557) when using sub-sample accurate motion vectors, motion vector prediction mechanisms, etc.

[0068] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). The video compression techniques may include loop filtering techniques that are controlled by parameters in the coded video sequence (also referred to as the coded video bitstream) that are available to the loop filter unit (556) as symbols (521) from the parser (520), and that may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of the coded picture or coded video sequence, and to previously reconstructed and loop filtered sample values.

[0069] The output of the loop filter unit (556) may be a sample stream that may be output to a rendering device (512) and stored in a reference picture memory (557) for use in future inter-picture prediction.

[0070] Certain coded pictures, once fully reconstructed, may be used as reference pictures for future predictions. For example, once a coded picture corresponding to a current picture has been fully reconstructed and that coded picture has been identified as a reference picture (e.g., by a parser (520)), the current picture buffer (558) may become part of the reference picture memory (557) and a new current picture buffer may be reallocated before starting reconstruction of the next coded picture.

[0071] The video decoder (510) may perform decoding operations according to a predetermined video compression technology, such as in the ITU-T Rec. H.265 standard. The encoded video sequence may conform to the syntax specified by the video compression technology or standard used in the sense that the encoded video sequence conforms to the syntax of the video compression technology or standard and conforms to the profile recorded in the video compression technology or standard. Specifically, the profile may select certain tools from all available tools in the video compression technology or standard as the only tools available under the profile. For compliance, it is also required that the complexity of the encoded video sequence is within the range defined by the level of the video compression technology or standard. In some cases, the hierarchy may limit the maximum image size, maximum frame rate, maximum reconstruction sampling rate (e.g., measured in megasamples per second), maximum reference image size, etc. In some cases, the limits set by the hierarchy may be further limited by the Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the encoded video sequence.

[0072] In one embodiment, the receiver (531) is capable of receiving additional (redundant) data along with the encoded video. The additional data may include a portion of one (or more) encoded video sequences. The video decoder (510) may use the additional data to correctly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0073] Figure 6 A block diagram of a video encoder (603) according to an embodiment of the present disclosure is shown. The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used to replace Figure 4 The video encoder (403) in the example.

[0074] The video encoder (603) can be used to obtain the video source (601) (not Figure 6 In another example, the video source (601) is a part of the electronic device (620) to receive the video samples, and the video source can capture one (or more) video images to be encoded by the video encoder (603). In another example, the video source (601) is a part of the electronic device (620).

[0075] The video source (601) is capable of providing a source video sequence in the form of a digital video sample stream to be encoded by the video encoder (603), which digital video sample stream can have any suitable bit depth (e.g., 8 bits, 10 bits, 12 bits, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (601) can be a storage device storing previously prepared videos. In a video conferencing system, the video source (601) can be a camera that collects local image information as a video sequence. The video data can be provided as multiple individual images that are given motion when viewed in sequence. The image itself can be organized as a spatial array of pixels, where each pixel can include one or more samples depending on the sampling structure, color space, etc. used. The relationship between pixels and samples can be easily understood by those skilled in the art. The following description focuses on examples.

[0076] According to one embodiment, the video encoder (603) can encode and compress the images of the source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing the appropriate encoding speed is a function of the controller (650). In some embodiments, the controller (650) controls other functional units as described below and is functionally coupled to the other functional units. For the sake of brevity, the coupling is not described. The parameters set by the controller (650) may include relevant parameters of rate control (image skipping, quantizer, lambda value of rate-distortion optimization technology...), image size, image group (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other suitable functions related to the video encoder (603) optimized for a specific system design.

[0077] In some embodiments, the video encoder (603) is configured to operate in an encoding loop. As an oversimplified description, in one example, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input image to be encoded and one (or more) reference images) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data (because in the video compression techniques considered in the presently disclosed subject matter, any compression between the symbols and the encoded video code stream is lossless). The reconstructed sample stream (sample data) is input to a reference image memory (634). Since the decoding of the symbol stream produces bit-accurate results that are independent of the decoder location (local or remote), the contents of the reference image memory (634) are also bit-accurate between the local encoder and the remote encoder. In other words, the reference image samples "seen" by the prediction part of the encoder are exactly the same as the sample values ​​"seen" by the decoder when using prediction during decoding. This basic principle of reference image synchronicity (and the drift that occurs if synchronicity cannot be maintained, e.g. due to channel errors) is also used in some related techniques.

[0078] The operation of the "local" decoder (633) may be the same as the operation of a "remote" decoder, such as the video decoder (510), which has been described above in conjunction with Figure 5 However, briefly refer to Figure 5 , when symbols are available and the entropy encoder (645) and parser (520) can losslessly encode / decode the symbols into an encoded video sequence, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and the parser (520), may not be fully implemented in the local decoder (633).

[0079] In one embodiment, decoder techniques other than parsing / entropy decoding present in the decoder are also present in the corresponding encoder in the same or substantially the same functional form. Therefore, the disclosed subject matter focuses on decoder operation. Because encoder techniques are the reverse of the fully described decoder techniques, the description of encoder techniques can be abbreviated. A more detailed description of certain areas is provided below.

[0080] During operation, in some examples, the source encoder (630) may perform motion compensated predictive coding that predictively encodes an input image with reference to one or more previously encoded images from a video sequence designated as “reference images.” In this manner, the encoding engine (632) encodes the differences between pixel blocks of the input image and pixel blocks of one or more reference images that may be selected as one (or more) prediction references for the input image.

[0081] The local video decoder (633) can decode the coded video data that can use the picture designated as the reference picture based on the symbols created by the source encoder (630). The operation of the encoding engine (632) can advantageously be a lossy process. When the coded video data can be decoded at the video decoder ( Figure 6 When the video sequence is decoded at a remote location (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process that may be performed by the video decoder on the reference picture and may cause the reconstructed reference picture to be stored in the reference picture memory (634). In this way, the video encoder (603) may locally store a copy of the reconstructed reference picture that has common content (absent transmission errors) with the reconstructed reference picture to be obtained by the remote video decoder.

[0082] The predictor (635) can perform a prediction search for the encoding engine (632). That is, for a new image to be encoded, the predictor (635) can search the reference image memory (634) for sample data (as candidate reference pixel blocks) or vectors that can be used as appropriate prediction references for the new image, or certain metadata, such as reference image motion vectors, block shapes, etc. The predictor (635) can operate on a pixel-by-pixel basis based on sample blocks to find a suitable prediction reference. In some cases, based on the search results obtained by the predictor (635), it can be determined that the input image can have prediction references extracted from multiple reference images stored in the reference image memory (634).

[0083] The controller (650) can manage the encoding operations of the source encoder (630), including setting parameters and subgroup parameters, for example, for encoding video data.

[0084] The outputs of the above functional units may be subjected to entropy coding in an entropy encoder (645). The entropy encoder (645) converts the symbols generated by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0085] The transmitter (640) can buffer the one (or more) encoded video sequences created by the entropy encoder (645) in preparation for transmission via a channel (660), which can be a hardware / software link to a storage device that stores the encoded video data. The transmitter (640) can combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0086] The controller (650) can manage the operation of the video encoder (603). During encoding, the controller (650) can assign a specific coded picture type to each coded picture, which can affect the coding techniques that can be applied to the corresponding picture. For example, a picture can generally be assigned one of the following picture types:

[0087] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are aware of those variants of I pictures and their respective applications and features.

[0088] A predicted picture (P picture) may be a picture that can be encoded and decoded using intra prediction or inter prediction that predicts sample values ​​of each block using at most one motion vector and a reference index.

[0089] A bi-directionally predicted picture (B picture) may be a picture that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, a multi-predicted picture may use more than two reference pictures and associated metadata for reconstructing a single block.

[0090] The source picture may typically be spatially subdivided into a number of blocks of samples (e.g., 4×4, 8×8, 4×8, or 16×16 blocks per sample) and encoded on a block-by-block basis. Blocks may be predictively coded with reference to other (already coded) blocks, the block being determined by, for example, the coding allocation of the corresponding picture applied to the block. For example, blocks of an I picture may be non-predictively coded, or they may be predictively coded with reference to already coded blocks of the same picture (spatial prediction or intra prediction). Blocks of pixels of a P picture may be predictively coded with reference to a previously coded reference picture, either by spatial prediction or by temporal prediction. Blocks of a B picture may be predictively coded with reference to one or two previously coded reference pictures, either by spatial prediction or by temporal prediction.

[0091] The video encoder (603) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T Rec. H.265. In its operation, the video encoder (603) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.

[0092] In one embodiment, the transmitter (640) is capable of transmitting additional data along with the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0093] Video may be captured as multiple source images (video images) in a temporal sequence. Intra-image prediction (often abbreviated as intra-frame prediction) exploits spatial correlations in a given image, and inter-image prediction exploits (temporal or other) correlations between images. In one example, a particular image being encoded / decoded is divided into blocks, and the particular image being encoded / decoded is referred to as the current image. When a block in the current image is similar to a reference block in a reference image that has been previously encoded and statically buffered in the video, the block in the current image may be encoded by a vector called a motion vector. The motion vector points to a reference block in a reference image, and in the case of using multiple reference images, the motion vector may have a third dimension that identifies the reference image.

[0094] In some embodiments, a bi-prediction technique may be used in inter-frame image prediction. According to the bi-prediction technique, two reference images are used, for example, a first reference image and a second reference image that are both before the current image in the video in decoding order (but may be in the past and in the future, respectively, in display order). A block in the current image may be encoded by a first motion vector and a second motion vector, the first motion vector pointing to a first reference block of the first reference image and the second motion vector pointing to a second reference block of the second reference image. The block may be predicted by a combination of the first reference block and the second reference block.

[0095] In addition, merge mode techniques can be used in inter-picture prediction to improve coding efficiency.

[0096] According to some embodiments of the present disclosure, predictions such as inter-image prediction and intra-image prediction are performed in units of blocks. For example, according to the HEVC standard, images in a video image sequence are divided into coding tree units (CTUs) for compression, and the CTUs in the image have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Typically, a CTU includes three coding tree blocks (CTBs), which are a luminance CTB and two chrominance CTBs. Each CTU can be recursively divided into one or more coding units (CUs) in a quadtree. For example, a 64×64 pixel CTU can be divided into a 64×64 pixel CU, or 4 32×32 pixel CUs, or 16 16×16 pixel CUs. In one example, each CU is analyzed to determine the prediction type of the CU, such as an inter-frame prediction type or an intra-frame prediction type. The CU is divided into one or more prediction units depending on temporal and / or spatial predictability. Typically, each prediction unit includes a luma prediction block (PB) and two chroma PBs. In one embodiment, the prediction operation in encoding (encoding / decoding) is performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0097] Figure 7 A schematic diagram of a video encoder (703) according to another embodiment of the present disclosure is shown. The video encoder (703) is configured to receive a processing block (e.g., a prediction block) of sample values ​​within a current video image in a video image sequence, and encode the processing block into an encoded image that is a portion of an encoded video sequence. In one example, the video encoder (703) is used to replace Figure 4 The video encoder (403) in the example.

[0098] In the HEVC example, the video encoder (703) receives a matrix of sample values ​​for a processing block, which is, for example, a prediction block of 8×8 samples, etc. The video encoder (703) uses, for example, rate-distortion optimization to determine whether to best encode the processing block using intra mode, inter mode, or bi-prediction mode. When the processing block is to be encoded in intra mode, the video encoder (703) can encode the processing block into an encoded image using intra prediction techniques; and when the processing block is to be encoded in inter mode or bi-prediction mode, the video encoder (703) can encode the processing block into an encoded image using inter prediction or bi-prediction techniques, respectively. In some video coding techniques, the merge mode can be an inter-picture prediction sub-mode, in which the motion vector is derived from one or more motion vector predictors without benefiting from an encoded motion vector component outside the predictor. In some other video coding techniques, there may be a motion vector component applicable to the subject block. In an example, the video encoder (703) includes other components, such as a mode decision module (not shown) for determining the processing block mode.

[0099] exist Figure 7 In the example of FIG. 7 , the video encoder ( 703 ) includes Figure 7 An inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) are shown coupled together.

[0100] The inter-frame encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference image (e.g., blocks in a previous image and a subsequent image), generate inter-frame prediction information (e.g., redundant information description according to an inter-frame coding technique, motion vectors, merge mode information), and calculate an inter-frame prediction result (e.g., a predicted block) based on the inter-frame prediction information using any suitable technique. In some examples, the reference image is a decoded reference image that is decoded based on the encoded video information.

[0101] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), compare the block with blocks already encoded in the same image in some cases, generate quantization coefficients after transformation, and in some cases also generate intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques). In one example, the intra encoder (722) also calculates an intra prediction result (e.g., a predicted block) based on the intra prediction information and a reference block in the same image.

[0102] The general controller (721) is configured to determine general control data and control other components of the video encoder (703) based on the general control data. In one example, the general controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, when the mode is intra mode, the general controller (721) controls the switch (726) to select the intra mode result for use by the residual calculator (723), and controls the entropy encoder (725) to select the intra prediction information and include the intra prediction information in the bitstream; when the mode is inter mode, the general controller (721) controls the switch (726) to select the inter prediction result for use by the residual calculator (723), and controls the entropy encoder (725) to select the inter prediction information and include the inter prediction information in the bitstream.

[0103] The residual calculator (723) is configured to calculate the difference (residual data) between the received block and the prediction result selected from the intra-frame encoder (722) or the inter-frame encoder (730). The residual encoder (724) is configured to operate based on the residual data to encode the residual data to generate transform coefficients. In one example, the residual encoder (724) is configured to convert the residual data from the spatial domain to the frequency domain and generate transform coefficients. The transform coefficients are then subjected to quantization processing to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra-frame encoder (722) and the inter-frame encoder (730). For example, the inter encoder (730) can generate a decoded block based on the decoded residual data and the inter prediction information, and the intra encoder (722) can generate a decoded block based on the decoded residual data and the intra prediction information. In some examples, the decoded blocks are appropriately processed to generate a decoded image, and the decoded image can be buffered in a memory circuit (not shown) and used as a reference image.

[0104] The entropy encoder (725) is configured to format the bitstream to include the encoded blocks. The entropy encoder (725) is configured to include various information according to an appropriate standard, such as the HEVC standard. In one example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other appropriate information in the bitstream. It should be noted that according to the disclosed subject matter, when the block is encoded in the inter-frame mode or the merge sub-mode of the bi-prediction mode, there is no residual information.

[0105] Figure 8FIG. 8 is a schematic diagram of a video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive a coded image of a portion of a coded video sequence and decode the coded image to generate a reconstructed image. In an example, the video decoder (810) is used to replace Figure 4 A video decoder (410) is shown in an example.

[0106] exist Figure 8 In the example of FIG. 8 , the video decoder ( 810 ) includes Figure 8 An entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874) and an intra-frame decoder (872) are shown coupled together.

[0107] The entropy decoder (871) can be configured to reconstruct certain symbols from the encoded image, which represent syntax elements that constitute the encoded image. Such symbols may include, for example, the mode in which the block is encoded (e.g., intra-frame mode, inter-frame mode, bi-prediction mode, a merged sub-mode of the latter two or another sub-mode), prediction information (e.g., intra-frame prediction information or inter-frame prediction information) that can identify certain samples or metadata for prediction by the intra-frame decoder (872) or the inter-frame decoder (880), respectively, residual information in the form of, for example, quantized transform coefficients, etc. In one example, when the prediction mode is inter-frame or bi-prediction mode, the inter-frame prediction information is provided to the inter-frame decoder (880); and when the prediction type is an intra-frame prediction type, the intra-frame prediction information is provided to the intra-frame decoder (872). The residual information can be inverse quantized and provided to the residual decoder (873).

[0108] The inter-frame decoder (880) is configured to receive the inter-frame prediction information and generate an inter-frame prediction result based on the inter-frame prediction information.

[0109] The intra decoder (872) is configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0110] The residual decoder (873) is configured to perform inverse quantization to extract dequantized transform coefficients and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (to include quantizer parameters (Quantizer Parameter, QP)), and this information may be provided by the entropy decoder (871) (because this may only be low-amount control information, the data path is not depicted).

[0111] The reconstruction module (874) is configured to combine the residual output by the residual decoder (873) and the prediction result (output by the inter-frame or intra-frame prediction module, as appropriate) in the spatial domain to form a reconstructed block, which can be part of a reconstructed image, which in turn can be part of a reconstructed video. It should be noted that other suitable operations, such as deblocking operations, etc. can be performed to improve visual quality.

[0112] It should be noted that the video encoders (403), (603) and (703) and the video decoders (410), (510) and (810) can be implemented using any suitable technology. In one embodiment, the video encoders (403), (603) and (703) and the video decoders (410), (510) and (810) can be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603) and (603) and the video decoders (410), (510) and (810) can be implemented using one or more processors that execute software instructions.

[0113] According to an embodiment of the present disclosure, a code stream may include one or more coded video sequences (coded videosequence, CVS). CVS can be encoded independently of other CVS. Each CVS may include one or more layers, and each layer may be a representation of a video with a specific quality (e.g., spatial resolution), or a representation of a component interpretation attribute, for example, as a depth map, a transparency map, or a perspective map. In the time dimension, each CVS may include one or more access units (access units, AU). Each AU may include one or more images of different layers corresponding to the same time instance. A coded layer video sequence (coded layer video sequence, CLVS) is a layered CVS that may include a sequence of image units in the same layer. If the code stream has multiple layers, the CVS in the code stream may have one or more CLVS for each layer.

[0114] In one embodiment, the CVS includes a sequence of AUs. The sequence of AUs may include, in decoding order, an intra random access point (IRAP) AU followed by zero or more AUs that are not IRAP AUs. In one example, the zero or more AUs include all subsequent AUs, up to but not including any subsequent AU that is an IRAP AU. In one example, the CLVS includes an image sequence and associated non-video coding layer (VCL) network abstraction layer (NAL) units of the base layer of the CVS.

[0115] In one example, a picture unit includes a coded picture. A picture unit (PU) may include a set of NAL units associated with each other according to a specific classification rule. NAL units may be consecutive in decoding order. A PU may include a picture header (PH) and one or more VCL NAL units constituting a coded picture.

[0116] According to some aspects of the present disclosure, messages such as supplemental enhancement information (SEI) messages may be included in a coded bitstream, for example, to assist in decoding and / or display of the coded bitstream, or for other purposes. One (or more) SEI messages may include information that is not required for decoding (e.g., decoding samples of a coded image from a VCL NAL unit). One (or more) SEI messages may be optional for constructing luma or chroma samples through a decoding process. In some examples, SEI messages are not required to reconstruct luma or chroma samples during the decoding process. In addition, decoders that conform to video coding standards that support SEI messages do not need to process SEI messages to be conformed. For some coding standards, some SEI message information may be required to check bitstream consistency or for output timing decoder consistency. One (or more) SEI messages may be optionally processed by a conforming decoder so that the output order conforms to a certain standard (e.g., HEVC 265 or VVC). In one embodiment, one (or more) SEI messages are present in (e.g., signaled to) a bitstream.

[0117] SEI messages can contain various types of data that indicate the timing of video pictures, describe various properties of the encoded video, or how to use or enhance various properties. In some examples, SEI messages do not affect the core decoding process, but can indicate suggestions for how to post-process or display the video.

[0118] SEI messages can be used to provide additional information about the encoded bitstream, which can be used to change the presentation of the bitstream after it is decoded, or to provide information to the decoder. For example, SEI messages have been used to provide frame packing information (e.g., describing how video data is arranged in a video frame), content description (e.g., indicating that the encoded bitstream is, for example, a 360-degree video), and color information (e.g., color gamut and / or color range), etc.

[0119] In some examples, the SEI message can be used to signal the encoder that the encoded code stream includes 360-degree video or virtual reality (VR) video. The decoder can use the above information to present video data for 360-degree presentation. Alternatively, if the decoder cannot present 360-degree video, the decoder can use the above information to not present the video data.

[0120] The present disclosure includes video encoding techniques related to independently encoded region output for an encoded video stream.

[0121] Various immersive media (e.g., virtual reality (VR) videos, video-based point cloud applications) and gaming and / or e-sports streaming applications can benefit from partial access to one or more sub-images rather than full access to the entire image. VR video may also be referred to as 360VR or VR360. VR video or 360VR may refer to video captured using an omnidirectional camera. An omnidirectional camera can capture a full 360 degrees at the same time. In a VR video, the user can look around the entire scene. Compared to conventional videos, VR videos can provide an immersive and interactive experience. In one example, for a VR360 application, only a partial area corresponding to the user's current viewport is displayed on a head-mounted display or other portal device. In one example of a video-based point cloud application, it is beneficial to access content in a region to display a specific projection view from stereoscopic content, for example, a smaller amount of data is used for regional access.

[0122] In some video coding standards (e.g., VVC version 1), sub-picture segmentation functionality is supported. In one example, when the syntax element subpic_treated_as_pic_flag][i] in the sequence parameter set (SPS) is equal to 1, the boundary of the i-th sub-picture is regarded as a picture boundary, and the boundary of the i-th sub-picture can be filled for motion compensation prediction.

[0123] According to an embodiment of the present disclosure, an independently encoded region (or independent region) associated with one or more sub-images can be accessed earlier by a system (e.g., a video encoding system, a video decoder, etc.) before full decoding of the entire image is completed. An image may include an independently encoded region. The encoded data of the independently encoded region can be accessed, decoded, and / or output independently of, for example, one (or more) other encoded regions of the image. In one example, if the independently encoded region has a rectangular shape, the encoded data of the independent region can be independently accessed, decoded, and / or output. In one example, the independent region can be displayed independently of other encoded regions of the image.

[0124] A supplemental enhancement information (SEI) message may be decoded. According to an embodiment of the present disclosure, a first flag indicating whether to signal independent region output information in the SEI message may be determined based on the SEI message. In one example, the SEI message indicates independent region output information. Fig. 9A The independent region output information may indicate one or more independent regions in the current image (900). The one or more independent regions are configured to be independently decodable.

[0125] Fig. 9A An example of an independently encoded region (911) and a cropped independent region (921) associated with the independent region (911) in a current image (900) according to an embodiment of the present disclosure is shown. The current image (900) may be an encoded image. The current image (900) may include one or more sub-images (e.g., sub-images 0-5). The current image (900) may include one or more regions, wherein each of the one or more regions may include at least one sub-image. The one or more regions may include one or more independent regions and / or non-independent regions (e.g., one (or more) encoded regions that are not encoded independently of other one (or more) regions in the current image (900)).

[0126] exist Fig. 9A In the example shown, the current image (900) includes an independent region (911). The independent region (911) includes sub-images 1-2. The encoded data of the independent region (911) can be accessed, decoded and / or output independently of the other one (or more) encoded regions of the image (900). The independent region (911) can be decoded regardless of whether the entire current image (900) is decoded. The encoded data of the independent region (911) can be accessed, decoded and / or output independently of the other one (or more) sub-images (e.g., sub-image 0 and sub-images 3-5) of the image (900), and the other one (or more) sub-images (e.g., sub-image 0 and sub-images 3-5) of the image are not included in the independent region (911). The video decoder can access the independent region (911) earlier. In an example, the independent region (911) is accessed, decoded and / or output before the complete decoding of the entire image (900). In one example, the independent region (911) is accessed, decoded, and / or output without fully decoding the entire image (900). In one example, the independent region (911) is accessed, decoded, and / or output before decoding any other region (or regions) in the entire image (900). In one example, the independent region (911) is the only region in the entire image (900) that is accessed, decoded, and / or output.

[0127] In an example, if the independent region (eg, the independent region (911)) has a rectangular shape, encoded data of the independent region is independently accessed, decoded, and / or output.

[0128] An independent region (e.g., independent region (911)) may be cropped, and the cropped independent region (e.g., cropped independent region (921)) may be output or displayed. The cropped independent region (e.g., cropped independent region (921)) may be output or displayed independently of other one (or more) regions in the current image (900). In one example, the cropped independent region (e.g., cropped independent region (921)) is output or displayed instead of the independent region (e.g., independent region (911)).

[0129] Fig.10 An exemplary syntax (1000) in a supplemental enhancement information (SEI) message including independent region output information according to an embodiment of the present disclosure is shown.

[0130] The SEI message may be referred to as an independent region output (IRO) SEI message or an IRO window SEI message. The SEI message may include dimension information, sub-image identification, and / or offset information, such as the width, height, one (or more) sub-image IDs, and output (consistency) window offset for each independent region in the current image (e.g., current image (900)). When a target region (e.g., an independent region such as independent region (911)) is determined by the system interface, useful information for identifying the associated sub-image (e.g., sub-images 1-2) and new values ​​for the image size and consistency window (e.g., the window of the cropped independent region (921)) for the accessed region may be obtained from the SEI message. When the decoded image (900) includes multiple sub-images (e.g., sub-images 0-5), the SEI message may be used to output a specific rectangular region with cropping (e.g., the cropped independent region (921)).

[0131] In one example, 360-degree video in a 360 VR application is projected using a cubemap projection or cube mapping. Fig. 9AAs shown, the six faces of the cube can be rearranged onto the current image (900) with the padding area to alleviate boundary discontinuities. The current image (900) can be partitioned into six sub-images (e.g., sub-images 0-5) corresponding to the six faces. In one example, after the decoder decodes the current image with the cropping information in the SEI message (900), only a subset of sub-images 0-5 (e.g., one or two sub-images in sub-images 0-5) is output as required. In one example, before the decoder decodes the current image with the cropping information in the SEI message (900), only a subset of sub-images 0-5 (e.g., one or two sub-images in sub-images 0-5) is output as required. Reference Fig. 9A In one example, the independent region (911) includes a region (923) and a cropped independent region (921). The region (923) may include a padding region between adjacent sub-image pairs (e.g., a padding region between sub-images 0-1, a padding region between sub-images 1 and 4, and a padding region between sub-images 2 and 5) and a boundary region of the current image (900). The region (923) is removed according to the cropping information and is not output or displayed.

[0132] An independent region in the current image (900) may overlap with another independent region in the current image (900). Each independent region may overlap with each other. Since independent regions may be considered as sub-image groups, one sub-image may belong to multiple independent regions.

[0133] Fig. 9B An example of an independent region (911) overlapping with another independent region (913) in a current image (900) according to an embodiment of the present disclosure is shown. Fig. 9A Described in Fig. 9B The current image (900), sub-images 0-5, and an independent region (911) in FIG. 9 . The other independent region (913) in the current image (900) includes sub-images 1 and 4 in the current image (900). The other independent region (913) is different from the independent region (911). Sub-image 1 is included in the independent region (911) and the other independent region (913). The independent region (911) overlaps with the other independent region (913).

[0134] An independent region can nest or include another independent region. Fig. 9C An example of an independent region (911) according to an embodiment of the present disclosure is shown, which is included in the independent region (915) of the current image (900). Fig. 9A Described in Fig. 9C, a current image (900), sub-images 0-5, and an independent region (911) in FIG. 1 . An independent region (915) includes sub-images 1, 2, 4, and 5. An independent region (911) is included in an independent region (915). An independent region (915) nests an independent region (911). An independent region (915) and an independent region (911) can be accessed, decoded, and outputted independently of each other.

[0135] According to an embodiment of the present disclosure, the independent region output information may include offset information indicating the position of the independent region (911) in, for example, the current image (900), and dimension information indicating the size of the independent region (911). The position and size of the independent region (911) in the current image (900) may be determined based on the offset information and the dimension information. The independent region output information may be signaled in an SEI message. The presence of the independent region output information may be indicated by a first flag.

[0136] refer to Fig. 9A and Fig.10 , the offset information may include the upper left position (931) of the independent region (911) (e.g., Fig.10 The syntax elements ir_indepen_coded_region_left_offset[i] and ir_indepen_coded_region_top_offset[i] in the current image are used to indicate the position of the independent region (911). The integer i indicates the i-th independent region of the current image (900). Fig. 9A and Fig.10 , the size information may include the region width W (e.g., Fig.10 ) and the region height H (e.g., indicated by the syntax element ir_indepen_coded_region_width_in_luma_samples[i] in Fig.10 The syntax element ir_indepen_coded_region_height_in_luma_samples[i] in 911 is indicated to specify the width W and height H of the independent region (911), respectively. In an example, the independent region (911) is the 0th independent region of the current image (900), and the integer i is equal to 0.

[0137] An independent region (911) of the one or more independent regions may be decoded. In one example, based on the position and size of the independent region (911), for example, without cropping, the decoded independent region (911) is output. In one example, with cropping, the decoded independent region (911) is output. Reference Fig. 9A, outputting the cropped independent region (921) based on the position and size of the independent region (911) and the cropping information of the cropped independent region (921).

[0138] refer to Fig.10 , the IRO SEI message may include information about the partition output. Fig.10 , the associated picture (e.g., the current picture (900)) has a syntax element PicOutputFlag equal to 1. In one example, the boundary of the associated picture (e.g., the current picture (900)) can be padded for motion compensated prediction. The offset value of the IRO SEI message can be applied to the decoded picture (e.g., the current picture (900)) before cropping. When the i-th independent region (e.g., the independent region (911) where i is 0) is accessed and a cropping operation is to be performed, the cropping window specified by the IRO SEI message (e.g., using the syntax elements ir_output_win_left_offset[i], ir_output_win_right_offset[i], ir_output_win_top_offset[i], and ir_output_win_bottom_offset[i]) can be applied for cropping instead of applying the cropping window specified by the picture parameter set (PPS) for cropping.

[0139] A first flag in a SEI message (e.g., an IRO SEI message) may be indicated by a syntax element independent_region_output_cancel_flag. A first flag (e.g., independent_region_output_cancel_flag) having a first cancel value (e.g., a value of 1) may indicate that the IRO SEI message cancels the persistence of any previous IRO SEI message in the output order applied to the current layer of the CVS. A first flag (e.g., independent_region_output_cancel_flag) having a first cancel value (e.g., a value of 1) may indicate that independent region output information is not signaled in the IRO SEI message.

[0140] A first flag (e.g., independent_region_output_cancel_flag) with a second cancel value (e.g., a value of 0) may indicate that the independent region output information follows, for example, the first flag. A first flag (e.g., independent_region_output_cancel_flag) with a second cancel value (e.g., a value of 0) may indicate that the independent region output information is signaled in the IRO SEI message. When the decoder has a suitable output interface, the decoder may output each independent region after each independent region is completely decoded.

[0141] The independent region output information may include a second flag (e.g., independent_region_output_persistence_flag) that may specify the persistence of the IRO SEI message for the current layer. The second flag (e.g., independent_region_output_persistence_flag) having a first persistence value (e.g., 0) may specify that the IRO SEI message applies only to the currently decoded picture (e.g., picture (900)) (or the IRO SEI message applies only to the current picture (e.g., picture (900)).

[0142] A second flag (e.g., independent_region_output_persistence_flag) having a second persistence value (e.g., 1) may specify that the IRO SEI message may be applied to a current decoded picture (e.g., picture (900)) and one or more subsequent pictures of the current layer in output order until one or more conditions are met (or one or more conditions are true). In an example, a second flag (e.g., independent_region_output_persistence_flag) having a second persistence value (e.g., 1) specifies that the IRO SEI message may persist to one or more subsequent pictures (e.g., all subsequent pictures) of the current layer in output order until one or more conditions are true. The one or more conditions may include: (i) a new CLVS for the current layer starts; (ii) the codestream ends; and / or (iii) another picture in the current layer in an access unit or picture unit associated with the IRO SEI message (e.g., a different IRO SEI message) is output following the current picture (900) in output order.

[0143] According to an example of the present disclosure, the second flag (eg, independent_region_output_persistence_flag) has a second persistence value. Independent regions in subsequent images of the current layer of the CVS in output order may be output based on independent region output information, for example, based on offset information and dimension information.

[0144] The independent region output information may include a syntax element indicating the size of each decoded image (e.g., the current image (900)) of the image unit, the image unit including the IRO SEI message. The syntax element may include ir_pic_width_in_luma_samples and ir_pic_height_in_luma_samples. The syntax element ir_pic_width_in_luma_samples can specify the width of each decoded image of the image unit in units of luma samples. In an example, the syntax element ir_pic_width_in_luma_samples cannot be equal to 0. The syntax element ir_pic_width_in_luma_samples can be a multiple of Max(8, MinCbSizeY). Max(8, MinCbSizeY) is the maximum value between 8 and the MinCbSizeY value. In an example, the value of MinCbSizeY (e.g., 16, 32, 64) indicates the minimum size of the coding unit. The syntax element ir_pic_width_in_luma_samples may be less than or equal to a threshold value, for example indicated by the syntax element ir_pic_width_max_in_luma_samples or the syntax element pic_width_max_in_luma_samples. The syntax element ir_pic_width_max_in_luma_samples or the syntax element pic_width_max_in_luma_samples may indicate a maximum value of the width of each decoded picture of a picture unit.

[0145] The syntax element ir_pic_height_in_luma_samples may specify the height of each decoded image of the picture unit in units of luma samples. In an example, the syntax element ir_pic_height_in_luma_samples may not be equal to 0. The syntax element ir_pic_height_in_luma_samples may be a multiple of Max(8, MinCbSizeY). The syntax element ir_pic_height_in_luma_samples may be less than or equal to a threshold value indicated, for example, by the syntax element ir_pic_height_max_in_luma_samples or the syntax element pic_height_max_in_luma_samples. The syntax element ir_pic_height_max_in_luma_samples or the syntax element pic_height_max_in_luma_samples may indicate a maximum value for the height of each decoded image of the picture unit.

[0146] The independent region output information may include a syntax element ir_num_indepen_region_minus1 indicating the number of independently coded regions in the picture unit. The value of the syntax element ir_num_indepen_coded_region_minus1 plus 1 may specify the number of independently coded regions of the picture unit including the IRO SEI message.

[0147] The independent region output information may include a syntax element ir_subpic_id_len_minus1 indicating the length (e.g., number of bits) used to represent a sub-image identifier (ID). The value of the syntax element ir_subpic_id_len_minus1 plus 1 may specify the number of bits used to represent the length of the sub-image ID (e.g., syntax element ir_subpic_id[i][j]). The value of the syntax element ir_subpic_id_len_minus1 may be in the range of 0 to 15, inclusive.

[0148] The independent region output information may include a syntax element ir_num_subpics_minus1[i] indicating the number of sub-images associated with the i-th independently coded region. The value of the syntax element ir_num_subpics_minus1[i]plus 1 may specify the number of sub-images associated with the i-th independently coded region. Fig. 9C, the picture unit includes the current picture (900). The 0th independently coded region is the independent region (911) where i is 0. The value of the syntax element ir_num_subpics_minus1[0] is 1. The value of the syntax element ir_num_subpics_minus1[0]plus 1 is 2, where the number of sub-pictures associated with the independent region (911) is 2.

[0149] The first independently coded region is the independent region (915) where i is 1. The value of the syntax element ir_num_subpics_minus1[1] is 3. The value of the syntax element ir_num_subpics_minus1[1]plus 1 is 4, where the number of sub-pictures associated with the independent region (915) is 4.

[0150] According to an embodiment of the present disclosure, the independent region output information may indicate a sub-image ID of a sub-image (e.g., sub-image 1) in an independent region (e.g., independent region (911)). In one example, the independent region output information indicates a sub-image ID corresponding to the j-th sub-image associated with the i-th independently encoded region in the current image (900). The independent region output information may include a syntax element ir_subpic_id[i][j], which specifies the sub-image ID of the j-th sub-image associated with the i-th independently encoded region in the current image (900). The length of the syntax element ir_subpic_id[i][j] may be equal to the sum of the value of the syntax element ir_subpic_id_len_minus1 in bits and 1. In one example, a sub-image having the syntax element ir_subpic_id[i][j] may have a syntax element subpic_treated_as_pic_flag[i][j] equal to 1. The syntax element subpic_treated_as_pic_flag[i][j] equal to 1 may indicate that the boundary of the corresponding sub-picture is treated as a picture boundary and the boundary of the corresponding sub-picture may be padded for motion compensated prediction.

[0151] Also refer to Fig. 9C, the current image (900) includes a 0th independently coded region (911) (e.g., i=0) and a 1st independently coded region (915) (e.g., i=1). The 0th independently coded region (911) (e.g., i=0) includes sub-images 1-2 corresponding to j being 0 and 1, respectively. The sub-image ID of the 0th sub-image (e.g., sub-image 1) associated with the 0th independently coded region (911) is 1, where ir_subpic_id[0][0] is 1. The sub-image ID of the 1st sub-image (e.g., sub-image 2) associated with the 0th independently coded region (911) is 2, where ir_subpic_id[0][1] is 2.

[0152] The first independently coded region (915) (e.g., i=1) includes sub-images 1, 2, 4, and 5 corresponding to j being 0, 1, 2, and 3, respectively. The sub-image ID of the 0th sub-image (e.g., sub-image 1) associated with the first independently coded region (915) is 1, where ir_subpic_id[1][0] is 1. The sub-image ID of the first sub-image (e.g., sub-image 2) associated with the first independently coded region (915) is 2, where ir_subpic_id[1][1] is 2. The sub-image ID of the second sub-image (e.g., sub-image 4) associated with the first independently coded region (915) is 4, where ir_subpic_id[1][2] is 4. The sub-image ID of the third sub-image (e.g., sub-image 5) associated with the first independently coded region (915) is 5, where ir_subpic_id[1][3] is 5.

[0153] As described above, the syntax elements ir_indepen_coded_region_left_offset[i] and ir_indepen_coded_region_top_offset[i] may specify the upper left position (e.g., upper left position (931)) of the i-th independently coded region (e.g., independent region (911)). In an example, the syntax elements ir_indepen_coded_region_left_offset[i] and ir_indepen_coded_region_top_offset[i] may not be equal to 0. The syntax elements ir_indepen_coded_region_left_offset[i] and ir_indepen_coded_region_top_offset[i] may be an integer that is a multiple of Max(8, MinCbSizeY). The syntax element ir_indepen_coded_region_left_offset[i] may be less than the value of the syntax element ir_pic_width_in_luma_samples, and the syntax element ir_indepen_coded_region_top_offset[i] may be less than the value of the syntax element ir_pic_height_in_luma_amples.

[0154] As described above, the syntax element ir_indepen_coded_region_width_in_luma_samples[i] may specify the width of the i-th independently coded region. In an example, the syntax element ir_indepen_coded_region_width_in_luma_samples[i] may not be equal to 0. The syntax element ir_indepen_coded_region_width_in_luma_samples[i] may be an integer that is a multiple of Max(8, MinCbSizeY). The syntax element ir_indepen_coded_region_width_in_luma_samples[i] may be less than or equal to the value of the syntax element ir_pic_width_in_luma_samples.

[0155] As described above, the syntax element ir_indepen_coded_region_height_in_luma_samples[i] may specify the height of the i-th independently coded region. In an example, the syntax element ir_indepen_coded_region_height_in_luma_samples[i] may not be equal to 0. The syntax element ir_indepen_coded_region_height_in_luma_samples[i] may be an integer that is a multiple of Max(8, MinCbSizeY). The syntax element ir_indepen_coded_region_height_in_luma_samples[i] may be less than or equal to the value of the syntax element ir_pic_height_in_luma_samples.

[0156] According to an embodiment of the present disclosure, the independent region output information may include a third flag (e.g., ir_indepen_coded_region_output_window_present_flag[i]) of the i-th independent region (e.g., the independent region (911)) in the current image (900). The third flag (e.g., ir_indepen_coded_region_output_window_present_flag[i]) may indicate whether to signal cropping information of the i-th independent region in the independent region output information (e.g., indicating an output cropping window offset parameter).

[0157] A third flag (e.g., ir_indepen_coded_region_output_window_present_flag[i]) having a first current value (e.g., a value of 1) may indicate that cropping information (e.g., output cropping window offset parameter) of the i-th independently coded region can follow the third flag in the IRO SEI message. A third flag (e.g., ir_indepen_coded_region_output_window_present_flag[i]) having a second current value (e.g., a value of 0) may indicate that cropping information (e.g., output cropping window offset parameter) of the i-th independently coded region is not present (e.g., not included) in the IRO SEI message.

[0158] According to an embodiment of the present disclosure, the third flag indicates that cropping information (e.g., output cropping window offset parameter) of the ith independent region (e.g., independent region (911)) is signaled in the independent region output information. The position and size of the output window of the ith independent region may be determined based on (i) the cropping information and (ii) the offset information and the dimension information (e.g., indicating the position and size of the ith independent region). The ith independent region may be output based on the position and size of the output window of the ith independent region.

[0159] In one example, the output crop window offset parameters are indicated by the syntax elements ir_output_win_left_offset[i], ir_output_win_right_offset[i], ir_output_win_top_offset[i], and ir_output_win_bottom_offset[i] in the IRO SEI message. The syntax elements ir_output_win_left_offset[i], ir_output_win_right_offset[i], ir_output_win_top_offset[i], and ir_output_win_bottom_offset[i] may specify a rectangular region in image coordinates to output the i-th independently coded region. For example, the syntax elements ir_output_win_left_offset[i], ir_output_win_right_offset[i], ir_output_win_top_offset[i], and ir_output_win_bottom_offset[i] specify the left position, right position, top position, and bottom position of the rectangular region, respectively. When the third flag (eg, ir_indepen_coded_region_output_window_present_flag[i]) has a second current value (eg, a value of 0), it may be inferred that the values ​​of the syntax elements ir_output_win_left_offset[i], ir_output_win_right_offset[i], ir_output_win_top_offset[i], and ir_output_win_bottom_offset[i] are equal to 0.

[0160] Also refer to Fig. 9A, when the third flag (e.g., ir_indepen_coded_region_output_window_present_flag[0]) of the 0th independently coded region (e.g., the independent region (911)) has a first current value (e.g., 1), the output cropping window offset parameters of the independent region (911) follow the third flag in the IRO SEI message. The output cropping window offset parameters may include ir_output_win_left_offset[0], ir_output_win_right_offset[0], ir_output_win_top_offset[0], and ir_output_win_bottom_offset[0] that specify the left position, right position, top position, and bottom position of an output window (or consistency window) corresponding to a rectangular region (e.g., the cropped independent region (921)) to be output or displayed. The cropped independent region (921) specified by the output window is output or displayed.

[0161] For example, depending on the value of the second flag (eg, independent_region_output_persistence_flag), the SEI message (eg, IRO SEI message) may be applied to one or more pictures in the current layer of the CVS. Figures 9A-9C and Fig.10 , the above description may be appropriately applied to another layer of the CVS. The SEI message (eg, IRO SEI message) may be applied to one or more pictures in another layer of the CVS (eg, a layer different from the current layer).

[0162] An independent region of a first image in a first layer of a CVS may be accessed, decoded, and / or outputted independently of other regions (or regions) and / or sub-images (or sub-images) of the first image in the first layer of the CVS. An independent region of a second image in a second layer of the CVS is accessed, decoded, and / or outputted independently of other regions (or regions) and / or sub-images (or sub-images) of the second image. In one example, an independent region of a first image in a first layer is accessed, decoded, and / or outputted independently of one (or regions) and / or sub-images (or sub-images) of the second image. In one example, an independent region of a first image in a first layer is accessed, decoded, and / or outputted based on a partial image (e.g., region and / or sub-image) of the second image.

[0163] Fig.10The SEI message described in the specification can be appropriately adapted to various scenarios, and one (or more) syntax elements in the SEI message can be adjusted accordingly. One or more of the one (or more) syntax elements in the SEI message can be modified or omitted. The syntax elements can appear in the SEI message in any suitable order. One (or more) additional syntax elements can be added to the SEI message.

[0164] Fig.11 An example of a persistence scope of an SEI message (informative) according to an embodiment of the present disclosure is shown. In an example, an SEI message (e.g., an IRO SEI message) applies to an independent region output window. The persistence scope of the SEI message may indicate that the SEI message applies to a picture unit (PU) associated with the SEI message.

[0165] Fig.12 A flow chart outlining an encoding process (1200) according to an embodiment of the present disclosure is shown. In various embodiments, the process (1200) is performed by a processing circuit, such as a processing circuit in a terminal device (310), (320), (330), and (340), a processing circuit performing a video encoder function (e.g., (403), (603), (703), etc. In some embodiments, the process (1200) is implemented in software instructions, so when the processing circuit executes the software instructions, the processing circuit performs the process (1200). The process starts at (S1201) and proceeds to (S1210).

[0166] At (S1210), the image may be encoded in a code stream. Fig. 9A , an image (e.g., current image (900)) may include an independently encoded region (e.g., independent region (911)). The independently encoded region (e.g., independent region (911)) may include at least one sub-image of one or more sub-images in the current image. Figures 9A-9C Examples of images, independent regions, and one or more sub-images are described in .

[0167] At (S1220), a supplemental enhancement information (SEI) message indicating independent region output information of an image may be generated. The SEI message may be, for example Fig.10 IRO SEI message as described in Figures 9A-9C and Fig.10 As described, the independent region output information may indicate the location and size of independently encoded regions to be independently accessed, decoded, and / or output.

[0168] At (S1230), the SEI message may be included in the code stream. In one example, the encoded image and the SEI message may be transmitted to a decoder and received by the decoder, for example. The process (1200) proceeds to (S1299) and terminates.

[0169] The process (1200) can be suitably adapted to various scenarios, and the steps in the process (1200) can be adjusted accordingly. One or more of the steps in the process (1200) can be modified, omitted, repeated, and / or combined. The process (1200) can be implemented in any suitable order. One (or more) additional steps can be added.

[0170] In one example, the first flag in the SEI message (eg, Fig.10 The independent_region_output_cancel_flag described in (S1220) indicates that the independent region output information is not signaled in the SEI message. (S1220) may be modified so that the independent region output information is not included in the SEI message.

[0171] Fig.13 A flowchart outlining a decoding process (1300) according to an embodiment of the present disclosure is shown. In various embodiments, the process (1300) is performed by a processing circuit, such as a processing circuit in a terminal device (310), (320), (330) and (340), a processing circuit performing the function of a video encoder (403), a processing circuit performing the function of a video decoder (410), a processing circuit performing the function of a video decoder (510), a processing circuit performing the function of a video encoder (603), etc. In some embodiments, the process (1300) is implemented in software instructions, so when the processing circuit executes the software instructions, the processing circuit performs the process (1300). The process starts at (S1301) and proceeds to (S1310).

[0172] At (S1310), a supplemental enhancement information (SEI) message indicating independent region output information may be decoded. The independent region output information may indicate an independent region that is independently decodable in a current image;

[0173] At (S1320), the independent region in the current image may be determined based on the offset information and the dimension information of the independent region in the independent region output information. In one example, the offset information is Fig.10In one example, the dimension information is indicated by the syntax elements ir_indepen_coded_region_width_in_luma_samples[i] and ir_indepen_coded_region_height_in_luma_samples[i].

[0174] In one embodiment, the current image (e.g., current image (900)) includes one or more sub-images (e.g., sub-images 0-5). The independent region (e.g., independent region (911)) includes a sub-image (e.g., sub-image 1) of the one or more sub-images in the current image.

[0175] In one embodiment, reference Fig. 9B , the sub-image is included in another independent region (eg, independent region (913)) in the current image.

[0176] In one embodiment, reference Fig. 9C , an independent region (eg, independent region (911)) is included in another independent region (eg, independent region (915)).

[0177] In one embodiment, if Fig.10 As described in , the independent region output information indicates (e.g., indicated by the syntax element ir_subpic_id[i][j]) a sub-image identifier of a sub-image (e.g., the 0th independent region where i is 0) in an independent region (e.g., the 0th sub-image where j is 0). In an example, the value of the syntax element ir_subpic_id[0][0] is 1, and the sub-image ID is 1.

[0178] In one embodiment, the independent region output information includes a second flag (eg, Fig.10 A second flag with a first persistence value indicates that the SEI message applies only to the current picture in the current layer, for example. Fig.10 As described in , the second flag having the second current value indicates that the SEI message applies to the current picture and one or more subsequent pictures of the current layer in output order until a condition is met.

[0179] In one embodiment, the independent region output information includes a third flag (eg, Fig.10The third flag indicates whether to signal the cropping information of the independent region in the independent region output information.

[0180] At (S1330), the determined independent region may be decoded regardless of whether the entire current image is decoded.An independent region of one or more independent regions may be decoded using any suitable decoding method, such as the decoding method described in the present disclosure.

[0181] At (S1340), the decoded independent region may be output based on the position indicated by the offset information of the independent region and the size indicated by the dimension information, as shown in FIG. Fig.10 described.

[0182] In one embodiment, the second flag has a second current value. The independent region in the subsequent image of the current layer in the output order may be output based on the independent region output information.

[0183] The process (1300) proceeds to (S1399) and terminates.

[0184] The process (1300) can be suitably adapted to various scenarios, and the steps in the process (1300) can be adjusted accordingly. One or more of the steps in the process (1300) can be modified, omitted, repeated, and / or combined. The process (1300) can be implemented in any suitable order. One (or more) additional steps can be added.

[0185] In one embodiment, the SEI message includes a first flag (eg, Fig.10 independent_region_output_cancel_flag in SEI message (eg, Fig.10 (S1320) is signaled in the SEI message described in (S1330). If the first flag indicates that the independent region output information is signaled in the SEI message, the independent region in the current image can be determined based on the offset information and dimension information of the independent region in the independent region output information. In one embodiment, the first flag indicates that the independent region output information is not signaled in the SEI message. (S1320) can be omitted. (S1330) and (S1340) can be modified so that the current image is decoded and output.

[0186] In one embodiment, based on the third flag indicating that the cropping information of the independent region is signaled in the independent region output information, the position and size of the output window of the independent region may be determined based on (i) the cropping information and (ii) the offset information and the dimension information of the independent region. The decoded independent region may be output based on the position and size of the output window of the independent region (e.g., the cropped independent region (921)).

[0187] In one embodiment, decoding and outputting for display is performed before the entire current image is decoded.

[0188] According to an embodiment of the present disclosure, a portion of an image of a current image (e.g., one or more sub-images, independent regions) can be accessed, decoded, and / or outputted independently of other one (or more) regions and / or one (or more) sub-images in the current image. Each independent region in the current image can be accessed independently. Each independent region in the current image can be decoded independently. Each independent region in the current image can be output independently. Each independent region in the current image can be displayed independently. The sub-image or independent region in the current image can have any suitable shape and / or any suitable size. In one example, the sub-image or independent region in the current image has a rectangular shape. The current image can be accessed, decoded, and / or outputted partially or regionally (e.g., only the independent region (911)).

[0189] The embodiments of the present disclosure may be used alone or in combination in any order. In addition, each of the method (or embodiment), encoder, and decoder may be implemented by a processing circuit (e.g., one or more processors, or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium.

[0190] The above techniques may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. Fig.14 A computer system (1400) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0191] Computer software may be encoded using any suitable machine code or computer language that may be subjected to assembly, compilation, linking or similar mechanisms to create code comprising instructions that may be executed directly by one or more computer central processing units (CPUs), one or more graphics processing units (GPUs), etc., or through decoding, microcode execution, etc.

[0192] These instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, IoT devices, and the like.

[0193] Fig.14 The components of the computer system (1400) shown in the example are exemplary in nature and are not intended to suggest any limitation on the scope of use or functionality of computer software implementing the disclosed embodiments. The configuration of the components should also not be interpreted as having any dependency or requirement related to any one component or combination of components shown in the exemplary embodiment of the computer system (1400).

[0194] The computer system (1400) may include certain human-computer interface input devices. The human-computer interface input devices may respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), olfactory input (not depicted). The human-computer interface devices may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and videos (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0195] Input human-computer interface devices may include one or more of the following (only one of each is depicted): keyboard (1401), mouse (1402), touchpad (1403), touch screen (1410), data gloves (not shown), joystick (1405), microphone (1406), scanner (1407), camera (1408).

[0196] The computer system (1400) may also include certain human-computer interface output devices. These human-computer interface output devices may stimulate one or more human user senses through, for example, tactile output, sound, light, smell / taste. Such human-computer interface output devices may include tactile output devices (e.g., tactile feedback of a touch screen (1410), a data glove (not shown), or a joystick (1405), but may also be a tactile feedback device that is not used as an input device), audio output devices (e.g., speakers (1409), headphones (not depicted)), visual output devices (e.g., screens (1410) including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities, some of which are capable of outputting two-dimensional visual outputs or outputs exceeding three dimensions through devices such as stereoscopic image output, virtual reality glasses (not depicted), holographic displays and smoke boxes (not depicted), and printers (not depicted).

[0197] The computer system (1400) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (1420) with CD / DVD etc. media (1421), thumb drives (1422), removable hard drives or solid-state drives (1423), traditional magnetic media such as tapes and floppy disks (not depicted), and dedicated ROM / ASIC / PLD based devices such as security dongles (not depicted), etc.

[0198] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.

[0199] The computer system (1400) may also include an interface (1454) to one or more communication networks (1455). The network may be, for example, wireless, wired, optical. The network may also be a local area network, a wide area network, a metropolitan area network, a vehicle and industrial network, a real-time network, a delay-tolerant network, etc. Network examples include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., television wired or wireless wide area digital networks including cable TV, satellite TV and terrestrial broadcast television, and vehicle and industrial television including CANBus, etc. Some networks typically require an external network interface adapter attached to some common data port or peripheral bus (1449) (e.g., a USB port of the computer system (1400)); other network interfaces are typically integrated into the kernel of the computer system (1400) by attaching to a system bus as described below (e.g., an Ethernet interface attached to a PC computer system or a cellular network interface to a smartphone computer system). The computer system (1400) can use any of these networks to communicate with other entities. This communication may be one-way receive only (e.g., broadcast television), one-way send only (e.g., CANbus to certain CANbus devices), or bidirectional (e.g., to other computer systems using a local area network or wide area digital network). As described above, certain protocols and protocol stacks may be used on each of these networks and network interfaces.

[0200] The above-mentioned human-machine interface device, human-machine accessible storage device, and network interface may be attached to the kernel (1440) of the computer system (1400).

[0201] The core (1440) may include one or more central processing units (CPUs) (1441), graphics processing units (GPUs) (1442), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (1443), hardware accelerators for certain tasks (1444), and graphics adapters (1450), etc. These devices and read-only memory (ROM) (1445), random access memory (1446), internal mass storage (1447) such as internal non-user accessible hard drives, SSDs, etc. can be connected through a system bus (1448). In some computer systems, the system bus (1448) can be accessed in the form of one or more physical plugs to allow expansion by additional CPUs, GPUs, etc. Peripheral devices can be attached directly to the core's system bus (1448) or to the core's system bus (1448) through a peripheral device bus (1449). In one example, a screen (1410) can be connected to a graphics adapter (1450). The architecture of the peripheral bus includes PCI, USB, etc.

[0202] The CPU (1441), GPU (1442), FPGA (1443) and accelerator (1444) can execute certain instructions, which can be combined to form the above-mentioned computer code. The computer code can be stored in ROM (1445) or RAM (1446). Transition data can also be stored in RAM (1446), while permanent data can be stored in, for example, internal mass storage (1447). Fast storage and retrieval of any memory device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (1441), GPUs (1442), mass storage (1447), ROM (1445), RAM (1446), etc.

[0203] The computer readable medium may have computer codes for performing various operations implemented by the computer. The medium and computer codes may be those specially designed and constructed for the purposes of the present disclosure, or they may be of a type well known and available to those skilled in the art of computer software.

[0204] As a non-limiting example, a computer system having an architecture (1400), particularly a kernel (1440), can provide functionality due to the execution of software contained in one or more tangible computer-readable media by one (or more) processors (including CPUs, GPUs, FPGAs, accelerators, etc.). Such computer-readable media can be media associated with the user-accessible mass storage described above, as well as certain memories of the non-temporary kernel (1440), such as a kernel internal mass storage (1447) or ROM (1445). Software implementing various embodiments of the present disclosure can be stored in such a device and executed by the kernel (1440). Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software can enable the kernel (1440), particularly the processors therein (including CPUs, GPUs, FPGAs, etc.), to perform specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM (1446) and modifying these data structures according to software-defined processes. Additionally or alternatively, a computer system may function due to logic hardwired or otherwise embodied in circuitry (e.g., accelerator (1444)), which may operate in place of or in conjunction with software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic and vice versa. Where appropriate, references to computer-readable media may include circuitry (e.g., an integrated circuit (IC)) storing software for execution, circuitry embodying logic for execution, or both. The present disclosure includes any suitable combination of hardware and software.

[0205] Appendix A: Acronyms

[0206] JEM: Joint Exploration Mode

[0207] VVC: Versatile Video Coding

[0208] BMS: Benchmark Set

[0209] MV: Motion Vector

[0210] HEVC: High Efficiency Video Coding

[0211] SEI: Supplemental Enhancement Information

[0212] VUI: Video Availability Information

[0213] GOPs: Group of Pictures

[0214] TUs: Transformation Units

[0215] PU: Picture Unit

[0216] CTUs: Coding Tree Units

[0217] CTBs: Coding Tree Blocks

[0218] PBs: prediction blocks

[0219] HRD: Hypothetical Reference Decoder

[0220] SNR: Signal to Noise Ratio

[0221] CPUs: Central Processing Units

[0222] GPUs: Graphics Processing Units

[0223] CRT: cathode ray tube

[0224] LCD: Liquid Crystal Display

[0225] OLED: Organic Light Emitting Diode

[0226] CD: compact disc

[0227] DVD: Digital Video Disc

[0228] ROM: Read Only Memory

[0229] RAM: Random Access Memory

[0230] ASIC: Application-Specific Integrated Circuit

[0231] PLD: Programmable Logic Device

[0232] LAN: Local Area Network

[0233] GSM: Global System for Mobile Communications

[0234] LTE: Long Term Evolution

[0235] CANBus: Controller Area Network Bus

[0236] USB: Universal Serial Bus

[0237] PCI: Peripheral Component Interconnect

[0238] FPGA: Field Programmable Gate Array

[0239] SSD: Solid State Drive

[0240] IC: Integrated Circuit

[0241] CU: Coding Unit

[0242] RD: Rate Distortion

[0243] Although the present disclosure has described some exemplary embodiments, there are changes, permutations and various alternative equivalents that fall within the scope of the present disclosure. Therefore, it is understood that those skilled in the art will be able to design many systems and methods, although these systems and methods are not explicitly shown or described herein, these systems and methods embody the principles of the present disclosure and therefore belong to the spirit and scope of the present disclosure.

Claims

1. A method for video decoding in a video decoder, characterized in that include: decoding a supplemental enhancement information (SEI) message indicating independent region output information, the independent region output information indicating an independently decodable independent region in a current picture; the independent region output information comprising a second flag, the second flag having a first value indicating that the SEI message applies to the current picture, and the second flag having a second value indicating that the SEI message applies to the current picture and one or more subsequent pictures of a current layer in output order until a condition is satisfied; determining the independent region in the current image based on the offset information and the dimension information of the independent region in the independent region output information; Decoding the determined independent region, regardless of whether the entire current image is decoded; as well as The decoded independent region is output based on the position indicated by the offset information and the size indicated by the dimension information of the independent region.

2. The method according to claim 1, characterized in that The independent region includes a sub-image in the current image.

3. The method according to claim 2, characterized in that The sub-image is included in another independent region of the current image, and the other independent region also includes another sub-image in the current image.

4. The method according to claim 3, characterized in that The independent region is included in the other independent regions.

5. The method according to claim 2, characterized in that: The independent region output information indicates a sub-image identifier of the sub-image in the independent region.

6. The method according to claim 1, characterized in that The SEI message comprises a first flag indicating whether the independent region output information is signaled in the SEI message, and The determining the independent region includes determining the independent region based on the first flag, the first flag indicating that the independent region output information is signaled in the SEI message.

7. The method according to claim 1, characterized in that the second flag has the second value, and The method further includes outputting the independent regions in a subsequent image of the current layer in the output order based on the independent region output information.

8. The method according to any one of claims 1 to 6, characterized in that The independent region output information includes a third flag of the independent region, and The third flag indicates whether to signal cropping information of the independent region in the independent region output information.

9. The method according to claim 8, characterized in that Based on the third flag indication, the cropping information of the independent region is signaled in the independent region output information, The method further includes determining a position and a size of an output window of the independent region based on (i) the cropping information and (ii) the offset information and the dimension information of the independent region, and The outputting includes outputting the decoded independent region based on the position and the size of the output window of the independent region.

10. The method according to any one of claims 1 to 6, characterized in that The decoding and the outputting for display are performed before the entire current image is decoded.

11. A device for video decoding, characterized in that: include: A memory for storing program instructions; A processor, configured to call the program instructions stored in the memory to implement the method according to any one of claims 1 to 10.

12. A device for video decoding, characterized in that: include: A first decoding module is configured to decode a supplemental enhancement information SEI message indicating independent region output information, wherein the independent region output information indicates an independent region in a current image that can be independently decodable; the independent region output information includes a second flag, the second flag having a first value indicates that the SEI message is applied to the current image, and the second flag having a second value indicates that the SEI message is applied to the current image and one or more subsequent images of a current layer in an output order until a condition is met; a determination module configured to determine the independent region in the current image based on the offset information and the dimension information of the independent region in the independent region output information; a second decoding module configured to decode the determined independent region regardless of whether the entire current image is decoded; as well as An output module is configured to output the decoded independent region based on the position indicated by the offset information and the size indicated by the dimension information of the independent region.

13. A non-transitory computer-readable storage medium, characterized in that: A program is stored, and the program is executed by at least one processor to perform the method according to any one of claims 1 to 10.

14. A method for processing a video code stream, characterized in that: The video code stream is decoded according to the method according to any one of claims 1-10.

15. A computer program product, characterized in that The computer program product includes computer executable instructions stored in a computer readable storage medium; Wherein, when the processor of the electronic device reads the computer executable instructions from the computer readable storage medium and executes the computer executable instructions, the method described in any one of claims 1 to 10 is implemented.