Video encoding method, encoder, and program

Abstract

To provide a method and an apparatus for controlling intra-inter prediction for decoding or encoding a video sequence.SOLUTION: A method determines whether a neighboring block of the current block is coded using an intra-inter prediction mode, on the basis of determining that the neighboring block is coded using the intra-inter prediction mode, performs intra-mode coding of the current block using an intra-prediction mode associated with the intra-inter prediction mode, sets a prediction mode flag associated with the neighboring block, obtains a context value on the basis of the prediction mode flag associated with the neighboring block, and performs entropy coding using the obtained context value for the prediction mode flag associated with the current block, which indicates that the current block is intra-coded.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 62 / 767.473, filed November 14, 2018, and U.S. Application No. 16 / 454.294, filed June 27, 2019, in the United States Patent and Trademark Office, the entire contents of which are incorporated herein by reference.

[0002] Methods and apparatus according to embodiments relate to video coding, and more particularly to methods and apparatus for improved context design for entropy coding of prediction modes and coding block flags (CBFs). [Background technology]

[0003] 1A shows the intra-prediction modes used in High Efficiency Video Coding (HEVC). In HEVC, there are a total of 35 intra-prediction modes, among which mode 10 (101) is a horizontal mode, mode 26 (102) is a vertical mode, and mode 2 (103), mode 18 (104), and mode 34 (105) are diagonal modes. These intra-prediction modes are signaled by three Most Probable Modes (MPMs) and the remaining 32 modes.

[0004] For versatile video coding (VVC), some coding unit syntax tables are shown below. If the slice type is not intra and skip mode is not selected, the flag pred_mode_flag is signaled and encoded using only one context (e.g., variable pred_mode_flag). The syntax table for some coding units is as follows: [Table 1]

[0005] 1B, in VVC, there are a total of 87 intra-prediction modes, among which mode 18 (106) is the horizontal mode, mode 50 (107) is the vertical mode, and mode 2 (108), mode 34 (109), and mode 66 (110) are diagonal modes. Modes 1 to 10 (111) and modes 67 to 76 (112) are called Wide-Angle Intra Prediction (WAIP) modes.

[0006] For the chrominance component of an intra-coding block, the encoder selects the optimal chrominance prediction mode among five modes, including planar mode (mode index 0), DC mode (mode index 1), horizontal mode (mode index 18), vertical mode (mode index 50), and diagonal mode (mode index 66), and selects a direct copy of the intra-prediction mode of the associated luma component, i.e., DM mode. Table 1 below shows the mapping between chrominance intra-prediction directions and intra-prediction mode numbers. [Table 2] Summary of the Invention [Problem to be solved by the invention]

[0007] To avoid overlapping modes, the four modes other than the DM mode are assigned based on the intra-prediction mode of the associated luma component. If the intra-prediction mode number of the chroma component is 4, the intra-prediction direction of the luma component is applied to generate the intra-prediction samples of the chroma component. If the intra-prediction mode number of the chroma component is not 4 but is the same as the intra-prediction mode number of the luma component, the intra-prediction direction 66 is applied to generate the intra-prediction samples of the chroma component.

[0008] Multi-hypothesis intra-inter prediction combines one intra prediction and one merge indexed prediction, i.e., an intra-inter prediction mode. A merge coding unit (CU) signals a flag for the merge mode, and if the flag is true, selects an intra mode from the intra candidate list. For the luma component, the intra candidate list is obtained from four intra prediction modes, including DC mode, planar mode, horizontal mode, and vertical mode. Depending on the block shape, the size of the intra candidate list may be three or four. If the width of the CU is greater than twice the height of the CU, the horizontal mode is removed from the intra candidate list; if the height of the CU is greater than twice the width of the CU, the vertical mode is removed from the intra candidate list. A weighted average is used to combine one intra prediction mode selected by the intra mode index and one merge indexed prediction selected by the merge index. For the chroma component, no additional signaling is required and DM is always used.

[0009] The weights for combining predictions are explained as follows: If DC mode or planar mode is selected, or the width or height of the coding block (CB) is less than 4, equal weights are applied. For those CBs whose width or height is 4 or more, if horizontal / vertical mode is selected, first divide one CB vertically / horizontally into four equal-area regions. For each region, a corresponding (w_intra i ,w_inter i), where i ranges from 1 to 4, and (w_intra1,w_inter1)=(6,2), (w_intra2,w_inter2)=(5,3), (w_intra1,w_inter3)=(3,5), and (w_intra4,w_inter4)=(2,6). (w_intra1,w_inter1) corresponds to the region closest to the reference sample, and (w_intra4,w_inter4) corresponds to the region farthest from the reference sample. The combined prediction is then calculated by summing the two weighted predictions and right-shifting them by 3 bits. The intra prediction mode of the predictor's intra hypothesis can also be saved for subsequent intra-mode coding of neighboring CBs if they are intra-coded. [Means for solving the problem]

[0010] According to an embodiment, a method for controlling intra-inter prediction for decoding or encoding a video sequence, executed by at least one processor, includes the steps of: determining whether neighboring blocks of a current block are coded using an intra-inter prediction mode; and, based on the determination that the neighboring blocks are coded using the intra-inter prediction mode, performing intra-mode coding of the current block using an intra-prediction mode associated with the intra-inter prediction mode, setting a prediction mode flag associated with the neighboring block, obtaining a context value based on the set prediction mode flag associated with the neighboring block, and performing entropy coding using the obtained context value for the prediction mode flag associated with the current block, which indicates that the current block is intra-coded.

[0011] According to an embodiment, an apparatus for controlling intra-inter prediction for decoding or encoding a video sequence includes at least one memory configured to store computer program code, and at least one processor configured to access the at least one memory and operate based on the computer program code. The computer program code includes: first determination code configured to cause the at least one processor to determine whether neighboring blocks of a current block are coded using an intra-inter prediction mode; execution code configured to cause the at least one processor to perform intra-mode coding of the current block using an intra-prediction mode associated with the intra-inter prediction mode based on the determination that the neighboring blocks are coded using the intra-inter prediction mode; and setting code configured to cause the at least one processor to perform operations of setting a prediction mode flag associated with the neighboring block based on the determination that the neighboring blocks are coded using the intra-inter prediction mode, obtaining a context value based on the set prediction mode flag associated with the neighboring block, and performing entropy coding using the obtained context value for the prediction mode flag associated with the current block indicating that the current block is intra-coded.

[0012] According to an embodiment, a non-transitory computer-readable storage medium having stored thereon instructions causes at least one processor to perform the following operations: determining whether a neighboring block of a current block is coded using an intra-inter prediction mode; and, based on determining that the neighboring block is coded using the intra-inter prediction mode, performing intra-mode coding of the current block using an intra-prediction mode associated with the intra-inter prediction mode, setting a prediction mode flag associated with the neighboring block; obtaining a context value based on the set prediction mode flag associated with the neighboring block; and performing entropy coding using the obtained context value for the prediction mode flag associated with the current block, which indicates that the current block is intra-coded. [Brief explanation of the drawings]

[0013] [Figure 1A] FIG. 1 is a diagram of intra prediction modes in HEVC. [Figure 1B] FIG. 1 is a diagram illustrating intra prediction modes in VVC. [Figure 2] 1 is a simplified block diagram of a communication system according to an embodiment. [Figure 3] FIG. 2 is a diagram of an arrangement of video encoders and video decoders in a streaming environment, according to an embodiment. [Figure 4] FIG. 2 is a functional block diagram of a video decoder according to an embodiment. [Figure 5] 1 is a functional block diagram of a video encoder according to an embodiment; [Figure 6] 3 is a diagram of a current block and its neighboring blocks, according to an embodiment. [Figure 7] 1 is a flowchart illustrating a method for controlling intra-inter prediction for decoding or encoding a video sequence according to an embodiment. [Figure 8]1 is a simplified block diagram of an apparatus for controlling intra-inter prediction for decoding or encoding a video sequence according to an embodiment; [Figure 9] 1 is an illustration of a computer system suitable for implementing embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0014] FIG. 2 is a simplified block diagram of a communication system (200) according to an embodiment. The communication system (200) may include at least two terminals (210-220) interconnected via a network (250). In the case of one-way data transmission, a first terminal (210) can encode video data locally and transmit it to another terminal (220) via the network (250). The second terminal (220) can receive the encoded video data of the other terminal from the network (250), decode the encoded video data, and display the decoded video data. One-way data transmission may be common in media service applications, etc.

[0015] 2 shows a second pair of terminals (230, 240) provided to support bidirectional transmission of encoded video, such as may occur during a video conference. For bidirectional data transmission, each terminal (230, 240) can encode video data captured locally and transmit it to the other terminal over the network (250). Each terminal (230, 240) can also receive encoded video data transmitted from the other terminal, decode the encoded video data, and display the decoded video data on a local display device.

[0016] In FIG. 2 , the terminals (210-240) are shown as servers, personal computers, and smartphones, but the principles of the embodiments are not limited thereto. The embodiments apply to laptop computers, tablets, media players, and / or dedicated videoconferencing equipment. The network (250) represents any number of networks, including, for example, wired and / or wireless communication networks, for transmitting encoded video data between the terminals (210-240). The communication network (250) can exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of discussion in this application, the architecture and topology of the network (250) may not be important to the operation of the embodiments, unless otherwise described herein.

[0017] 3 is a diagram of a configuration of video encoders and decoders in a streaming environment, according to an embodiment. The disclosed subject matter may equally be applied to other applications for supporting video, including, for example, video conferencing, digital TV, and storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0018] The streaming system may include a capture subsystem (313), which may include, for example, a video source (301) (e.g., a digital camera) for creating an uncompressed video sample stream (302). The sample stream (302) is depicted as a thick line to emphasize its larger data volume compared to an encoded video bitstream, and the sample stream (302) may be processed by an encoder (303) coupled to the capture device (301). The encoder (303) may include hardware, software, or a combination thereof to realize or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video bitstream (304) is depicted as a thin line to emphasize its smaller data volume compared to the sample stream, and the encoded video bitstream (304) may be stored on a streaming server (305) for future use. One or more streaming clients (306, 308) can access the streaming server (305) to retrieve replicas (307, 309) of the encoded video bitstream (304). The client (306) can include a video decoder (310) for decoding the incoming replica (307) of the encoded video data and creating an outgoing video sample stream (311) that is rendered to a display (312) or other rendering device (not shown). In some streaming systems, the video bitstreams (304, 307, 309) can be encoded according to several video encoding / compression standards. Examples of these standards include the ITU-T H.265 proposal. A video coding standard informally referred to as VVC is under development. The disclosed subject matter may be applied in the context of VVC.

[0019] FIG. 4 is a functional block diagram of a video decoder (310) according to an embodiment.

[0020] The receiver (410) can receive one or more codec video sequences to be decoded by the decoder (310), and in the same or other embodiments, can receive one coded video sequence at a time, with the decoding of each coded video sequence being independent of the other coded video sequences. The coded video sequences can be received from a channel (412), which can be a hardware / software link to a storage device for storing the coded video data. The receiver (410) can receive coded video data and other data, such as coded audio data and / or auxiliary data streams, which can be forwarded to respective using entities (not shown). The receiver (410) can separate the coded video sequences from other data. To address network jitter, a buffer memory (415) can be coupled between the receiver (410) and the entropy decoder / parser (hereinafter referred to as the "parser"). If the receiver 410 receives data from a store-and-forward device or an isochronous real-time network with sufficient bandwidth and controllability, it may not require a buffer memory 415, or the buffer 615 may be small. To maximize utilization of packet networks such as the Internet, a buffer memory 415 may be required, and the buffer memory may be relatively large and advantageously of an adaptive size.

[0021] The video decoder (310) may include a parser (420) to reconstruct symbols (421) based on the entropy-encoded video sequence. These symbol categories include information for managing the operation of the video decoder (310) and potential information for controlling a rendering device, such as a display (312), which may not be part of the decoder but may be coupled to it, as shown in FIG. 4. Control information for the rendering device may be in the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (420) performs parsing / entropy decoding on the received coded video sequence. The coding of the coded video sequence may be based on a video coding technique or standard and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, context-sensitive or non-context-sensitive arithmetic coding, etc. The parser extracts a subgroup parameter set for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group, including groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The entropy decoder / parser may further extract information such as transform coefficients, quantizer parameter (QP) values, motion vectors, etc. from the coded video sequence.

[0022] The parser (420) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (415) to create symbols (421). The parser can receive the encoded data and selectively decode particular symbols (421). The parser can also determine whether a particular symbol (421) is to be provided to the motion compensation prediction unit (453), the scaler / inverse transform unit (451), the intra prediction unit (452), or the loop filter (454).

[0023] Depending on the coded video picture or some type of coded video picture (e.g., inter-picture vs. intra-picture, inter-block vs. intra-block), as well as other factors, the reconstruction of the symbols (421) may involve several different units. The units involved and the manner in which they are involved are controlled by subgroup control information parsed by the parser (420) from the coded video sequence. For the sake of brevity, the flow of such subgroup control information between the parser (420) and the following units is not described.

[0024] In addition to the functional blocks already mentioned, the decoder (310) may be conceptually subdivided into several functional units, as described below. In an actual implementation operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of illustrating the disclosed subject matter, a conceptual subdivision into the following functional units is appropriate:

[0025] The first unit is a scalar / inverse transform unit (451), which receives quantized transform coefficients as symbols (421) and control information from the parser (420), including the transform scheme to be used, the block size, the quantization factor, the quantization scaling matrix, etc. It can output blocks containing sample values ​​that can be input to the aggregator (455).

[0026] In some cases, the output samples of the scaler / inverse transform (451) may belong to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra prediction unit (452). In some cases, the intra prediction unit (452) uses surrounding already reconstructed information obtained from the current (partially reconstructed) picture (456) to generate blocks of the same size and shape as the block it is reconstructing. In some cases, the aggregator (455) adds, based on each sample, the prediction information generated by the intra prediction unit (452) to the output sample information provided by the scaler / inverse transform unit (451).

[0027] In other cases, the output samples of the scaler / inverse transform unit (451) may belong to an inter-coded and potentially motion-compensated block. In such cases, the motion-compensated prediction unit (453) can access the reference picture buffer (457) to obtain samples for prediction. After performing motion compensation on the obtained samples based on the symbols (421) belonging to the block, these samples can be added by the aggregator (455) to the output of the scaler / inverse transform unit (referred to as residual samples or residual signals in this case) to generate output sample information. The addresses in the reference picture memory from which the motion-compensated prediction unit extracts the prediction samples may be controlled by a motion vector, which may be used by the motion-compensated prediction unit in the form of a symbol (421), which may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​obtained from the reference picture memory when a sub-sample accurate motion vector is used, a motion vector prediction mechanism, etc.

[0028] The output samples of the aggregator (455) may be processed by various loop filtering techniques in a loop filter unit (454). The video compression techniques may include in-loop filtering techniques controlled by parameters included in the coded video bitstream, which may be applied to the loop filter unit (454) as symbols (421) from the parser (420); however, the video compression techniques may also respond to meta-information obtained during decoding of a coded picture or a previous portion (in decoding order) of the coded video sequence, or to previously constructed loop-filtered sample values.

[0029] The output of the loop filter unit (454) may be a sample stream, which may be output to the rendering device (312) and stored in a reference picture buffer (456) for use in future inter-picture prediction.

[0030] Once a coded picture is fully reconstructed, it is used as a reference picture for future prediction. Once a coded picture is fully reconstructed and the coded picture is recognized as a reference picture (e.g., by the parser (420)), the current reference picture (456) becomes part of the reference picture buffer (457), allowing new current picture memory to be reallocated before reconstruction of the subsequent coded picture begins.

[0031] The video decoder (310) may perform decoding operations based on a predetermined video compression technology, such as that documented in the ITU-T H.265 proposal. The encoded video sequence conforms to the syntax specified by the video compression technology or standard used, in the sense that the encoded video sequence conforms to the syntax of the video compression technology or standard explicitly specified in its profile, for example, in a video compression technology document or standard. Compliance also requires that the complexity of the encoded video sequence be within a range limited by the level of the video compression technology or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the level may be further limited by the specification of a hypothetical reference decoder (HRD) and HRD buffer management metadata signaled in the encoded video sequence.

[0032] In embodiments, the receiver (410) can receive additional (redundant) data along with the encoded video. The additional data is included as part of the encoded video sequence. The additional data is utilized by the video decoder (310) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0033] FIG. 5 may be a functional block diagram of a video encoder (303) according to an embodiment.

[0034] The encoder (303) can receive video samples from a video source (301) (not part of the encoder), which can capture video images that are encoded by the encoder (303).

[0035] The video source (301) may provide a source video sequence in the form of a digital video sample stream to be encoded by the encoder (303). The digital video sample stream may include any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any suitable color space (e.g., BT.601 Y CrCB, RGB, etc.), and any suitable sampling configuration (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media services system, the video source (301) may be a storage device for storing previously prepared video. In a video conferencing system, the video source (301) may be a camera for capturing local image information as a video sequence. The video data may be provided as multiple individual pictures that, when viewed sequentially, impart motion. The pictures themselves may be organized as a spatial pixel array, where each pixel may contain one or more samples, depending on the sampling configuration, color space, etc. used. The relationship between pixels and samples is readily apparent to those skilled in the art. The following description focuses on samples.

[0036] According to an embodiment, the encoder (303) encodes pictures of a source video sequence in real time or under any other time constraint required by the application, and compresses them as an encoded video sequence (543). Running at an appropriate encoding rate is one of the functions of the controller (550). The controller controls and is functionally coupled to other functional units, described below. For simplicity, coupling is not shown. Parameters set by the controller may include rate control-related parameters (picture skip, quantizer, lambda value for rate-distortion optimization techniques, etc.), picture size, group of pictures (GOP) placement, maximum motion vector search range, etc. Those skilled in the art will readily recognize other functions of the controller (550) for optimizing the video encoder (303) for a particular system design.

[0037] Some video encoders operate in a "coding loop," which is readily apparent to those skilled in the art. As a highly simplified explanation, the encoding loop may include an encoding portion of an encoder (530) (hereafter referred to as the "source encoder") (responsible for creating symbols based on the input picture to be encoded and reference pictures), and a (local) decoder (533) embedded in the encoder (303), which reconstructs the symbols to create sample data that will also be created by a (remote) decoder (because, in the video compression techniques considered in this disclosure, any compression between the symbols and the encoded video bitstream is lossless). This reconstructed sample stream is input to a reference picture memory (534). Decoding the symbol stream produces results that are bit-accurate regardless of the decoder location (local or remote), so the contents of the reference picture buffer are bit-accurate between the local encoder and the remote encoder. That is, the reference picture samples "seen" by the prediction part of the encoder are exactly the same sample values ​​"seen" by the decoder when it uses the prediction during decoding. The basic principle of such reference picture synchrony (and the drift that occurs when synchrony cannot be maintained, e.g., due to channel error) is well known to those skilled in the art.

[0038] The operation of the "local" decoder (533) may be the same as the operation of the "remote" decoder (310), described in detail above with reference to Figure 4. However, with brief reference to Figure 4, if symbols are available and the entropy coder (545) and parser (420) can losslessly encode / decode the symbols into an encoded video sequence, the entropy decoding portion of the decoder (310), including the channel (412), receiver (410), buffer memory (415), and parser, need not be fully implemented in the decoder (533).

[0039] In this case, it can be observed that any decoder technique other than analysis / entropy decoding present in the decoder necessarily exists in the corresponding encoder in essentially the same functional form. The description of the encoder technique can be simplified because the encoder technique and the fully described decoder technique are mutually inverse. A more detailed description is only necessary in a few places and is provided below.

[0040] As part of its operation, the source encoder (530) may perform motion-compensated predictive coding, which performs predictive coding on an input frame with reference to one or more previously coded frames from the video sequence designated as "reference frames." In this manner, the encoding engine (532) may code differences between pixel blocks of the input frame and pixel blocks that may be selected as reference frames for the input frame's predicted reference.

[0041] The decoder (533) decodes previously encoded video data of frames designated as reference frames based on the codes created by the source encoder (530). The operation of the encoding engine (532) may advantageously be a lossy process. When the encoded video data is decoded by a video decoder (not shown in FIG. 4), the reconstructed video sequence may be a replica of the source video sequence, typically with some errors. The decoder (533) may copy the decoding process that may be performed on the reference frames by the video decoder and store the reconstructed reference frames in a reference picture cache (534). In this way, the video encoder (303) may locally store replicas of reconstructed reference frames that have common content with reconstructed reference frames obtained by a remote video decoder (in the absence of transmission errors).

[0042] The predictor (535) performs a prediction search for the coding engine (532). That is, for a new frame to be coded, the predictor (535) may search the reference picture memory (534) for sample data (as candidate reference pixel blocks) that can be used as suitable prediction references for the new picture, or for specific metadata, such as the reference picture's motion vectors, block shape, etc. The predictor (535) can find suitable prediction references by operating pixel block by pixel block based on the sample blocks. In some cases, the input picture may have prediction references obtained from multiple reference pictures stored in the reference picture memory (534), as determined from the search results obtained by the predictor (535).

[0043] The controller (550) can manage the encoding operations of the source encoder (530), including, for example, setting parameters and subgroup parameters for encoding the video data.

[0044] The output of all of these functional units may be entropy coded in an entropy coder (545) that performs lossless compression on the symbols produced by the various functional units, converting them into an encoded video sequence, using techniques known to those skilled in the art (e.g., Huffman coding, variable length coding, arithmetic coding, etc.).

[0045] The transmitter (540) can buffer the encoded video sequence created by the entropy encoder (545) in preparation for transmission over a communication channel (560), which may be a hardware / software link to a storage device for storing the encoded video data. The transmitter (540) can merge the encoded video data from the source encoder (530) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (sources not shown).

[0046] The controller (550) can manage the operation of the encoder (303). During encoding, the controller (550) may assign each encoded picture a particular encoded picture type, which may affect the encoding technique applied to the corresponding picture. For example, pictures are typically assigned as one of the following frame types:

[0047] An intra picture (I-picture) may be a picture that is coded and decoded without using any other frame in the sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, "Independent Decoder Refresh" pictures. Those skilled in the art are aware of these variations of I-pictures and their corresponding uses and characteristics.

[0048] A predicted picture (P picture) may be a picture that is coded and decoded using intra- or inter-prediction to predict the sample values ​​of each block using at most one motion vector and reference index.

[0049] A bidirectionally predictive picture (B-picture) may be a picture that is encoded and decoded using intra- or inter-prediction to predict the sample values ​​of each block using at most two motion vectors and reference indices. Similarly, a multiple-prediction picture may use more than two reference images and associated metadata to reconstruct a single block.

[0050] A source picture is typically spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and may be coded block by block. A block may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to the block's corresponding picture. For example, a block of an I-picture may be non-predictively coded or predictively coded with reference to coded blocks of the same picture (spatial prediction or intra prediction). A pixel block of a P-picture may be non-predictively coded via spatial or temporal prediction with reference to one previously coded reference picture. A block of a B-picture may be non-predictively coded via spatial or temporal prediction with reference to one or two previously coded reference pictures.

[0051] The video encoder (303) may perform encoding operations based on a given video encoding technique or standard, for example, the ITU-T H.265 proposal. During operation, the video encoder (303) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancies in the input video sequence. Thus, the encoded video data may conform to a syntax specified by the video encoding technique or standard being used.

[0052] In some embodiments, the transmitter (540) can transmit additional data along with the encoded video. The source encoder (530) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other types of redundant data such as redundant pictures and slices, Supplemental Enhancement Information (SEI) messages, Video Usability Information (VUI) parameter set segments, etc.

[0053] In the prior art, only one context is used to encode the flag pred_mode_flag for indicating whether a block is intra-coded or inter-coded, rather than the value of a flag applied to a neighboring block. Also, if a neighboring block is encoded by an intra-inter prediction mode, it may be more effective to predict the neighboring block using a combination of intra and inter prediction modes, and therefore, for context design signaling the flag pred_mode_flag, it may be more effective to consider whether the neighboring block is encoded by an intra-inter prediction mode.

[0054] The embodiments described herein may be used alone or in combination in any order. Below, the flag pred_mode_flag indicates whether the current block is intra-coded or inter-coded.

[0055] FIG. 6 is a diagram of a current block and its neighboring blocks, according to an embodiment.

[0056] 6, there is shown a current block 610, a top neighboring block 620, and a left neighboring block 630 of the current block 610. The top neighboring block 620 and the left neighboring block 630 each have a width of 4 and a height of 4.

[0057] In an embodiment, information on whether neighboring blocks (e.g., the top neighboring block (620) and the left neighboring block (630)) are coded using an intra prediction mode, an inter prediction mode, or an intra-inter prediction mode is used to obtain a context value for entropy coding the flag pred_mode_flag of a current block (e.g., the current block (610)). In particular, if a neighboring block is coded using an intra-inter prediction mode, the associated intra prediction mode is applied to intra-mode coding of the current block and / or deriving the MPM, but when deriving a context value for entropy coding the flag pred_mode_flag of the current block, the neighboring block is considered to be an inter-coded block, even though intra prediction was used for the neighboring block.

[0058] In one example, the associated intra prediction mode of an intra-inter prediction mode is always a planar mode.

[0059] In another example, the associated intra prediction mode of an intra-inter prediction mode is always DC mode.

[0060] In yet another example, the associated intra-prediction mode aligns with the intra-prediction mode applied in the intra-inter-prediction mode.

[0061] In an embodiment, when encoding neighboring blocks (e.g., the top neighboring block (620) and the left neighboring block (630)) using an intra-inter prediction mode, the associated intra prediction mode is applied to the intra mode encoding of the current block (e.g., the current block (610)) and / or the derivation of the MPM, and the neighboring blocks are also considered to be intra-coded blocks when deriving a context value for entropy encoding the flag pred_mode_flag of the current block.

[0062] In one example, the associated intra prediction mode of an intra-inter prediction mode is always a planar mode.

[0063] In another example, the associated intra prediction mode of an intra-inter prediction mode is always DC mode.

[0064] In yet another example, the associated intra prediction mode aligns with the intra prediction mode applied in the intra-inter prediction mode.

[0065] In one embodiment, the context index or value is incremented by 2, 0 and 1 if the neighboring blocks are coded using intra prediction mode, inter prediction mode and inter-intra prediction mode, respectively.

[0066] In another embodiment, if the neighboring blocks are coded using intra prediction mode, inter prediction mode and inter-intra prediction mode respectively, the context index or value is incremented by 1, 0 and 0.5 respectively, and the final context index is rounded to the nearest integer.

[0067] After incrementing the context index or value for all neighboring blocks of the current block to determine a final context index, the average context index can be determined by dividing the determined final context index by the number of neighboring blocks and rounding to the nearest integer. Based on the determined average context index, a flag pred_mode_flag is set to indicate whether the current block is intra-coded or inter-coded, and arithmetic coding is performed to encode the pred_mode_flag of the current block.

[0068] In an embodiment, information on whether the current block (e.g., current block (610)) is coded using an intra prediction mode, an inter prediction mode, or an inter-intra prediction mode is used to obtain one or more context values ​​for entropy coding the CBF of the current block.

[0069] In one embodiment, three separate contexts (e.g., variables) are used to entropy code the CBF: one context is used when the current block is coded using an intra prediction mode, one context is used when the current block is coded using an inter prediction mode, and one context is used when the current block is coded using an intra-inter prediction mode. The three separate contexts may be applied only to code the luma CBF, only to code the chroma CBF, or only to code both the luma and chroma CBFs.

[0070] In other embodiments, two separate contexts (e.g., variables) are used to entropy code the CBFs: one context is used when the current block is coded using an intra-prediction mode, and one context is used when the current block is coded using an inter-prediction mode or an intra-inter-prediction mode. The two separate contexts may be applied only to code the luma CBF, only to code the chroma CBF, or only to code both the luma and chroma CBFs.

[0071] In yet another embodiment, two separate contexts (e.g., variables) are used to entropy code the CBFs: one context is used when the current block is coded using an intra-prediction mode or an intra-inter-prediction mode, and one context is used when the current block is coded using an inter-prediction mode. The two separate contexts may be applied only to code the luma CBF, only to code the chroma CBF, or only to code both the luma and chroma CBFs.

[0072] 7 is a flowchart illustrating a method (700) for controlling intra-inter prediction for decoding or encoding a video sequence, according to an embodiment. In some implementations, one or more of the processing blocks in FIG. 7 may be performed by the decoder (310). In some implementations, one or more of the processing blocks in FIG. 7 may be performed by another device or group of devices (e.g., the encoder (303)) separate from or including the decoder (310).

[0073] 7, in a first block (710), the method (700) includes determining whether a neighboring block of the current block is coded using an intra-inter prediction mode. Based on a determination (710—NO) that the neighboring block is not coded using an intra-inter prediction mode, the method (700) ends.

[0074] Based on the determination (710—YES) that the neighboring block is coded using an intra-inter prediction mode, in a second block (720), the method (700) includes a step of performing intra-mode coding of the current block using an intra-prediction mode associated with the intra-inter prediction mode.

[0075] In a third block (730), the method (700) includes setting a prediction mode flag associated with the neighboring block.

[0076] In a fourth block (740), the method (700) includes obtaining a context value based on the set prediction mode flag associated with the neighboring block.

[0077] In a fifth block (750), the method (700) includes using the obtained context value to perform entropy coding of a prediction mode flag associated with the current block, which indicates that the current block is intra-coded.

[0078] The method (700) further includes a step of performing derivation of the MPM of the current block using an intra prediction mode associated with the intra inter prediction mode based on determining (710—YES) that the neighboring block is coded using the intra inter prediction mode.

[0079] The intra prediction mode associated with the intra-inter prediction mode may be a planar mode, a DC mode, or an intra prediction mode applied in the intra-inter prediction mode.

[0080] Setting a prediction mode flag associated with the neighboring block may include setting a prediction mode flag associated with the neighboring block to indicate that the neighboring block is intra-coded.

[0081] Setting a prediction mode flag associated with the neighboring block may include setting a prediction mode flag associated with the neighboring block to indicate that the neighboring block is inter-coded.

[0082] The method (700) may further include the steps of determining whether the neighboring block is coded in an intra prediction mode, an inter prediction mode, or an intra-inter prediction mode; incrementing a context index of a prediction mode flag associated with the current block by 2 based on the determination that the neighboring block is coded in an intra prediction mode, incrementing the context index by 0 based on the determination that the neighboring block is coded in an inter prediction mode, and incrementing the context index by 1 based on the determination that the neighboring block is coded in an intra-inter prediction mode; determining an average context index based on the incremented context index and the number of neighboring blocks of the current block; and setting the prediction mode flag associated with the current block based on the determined average context index.

[0083] The method may further include the steps of determining whether the neighboring block is coded using an intra prediction mode, an inter prediction mode, or an intra-inter prediction mode; incrementing a context index of a prediction mode flag associated with the current block by 1 based on the determination that the neighboring block is coded using an intra prediction mode, incrementing the context index by 0 based on the determination that the neighboring block is coded using an inter prediction mode, and incrementing the context index by 0.5 based on the determination that the neighboring block is coded using an intra-inter prediction mode; determining an average context index based on the incremented context index and the number of neighboring blocks of the current block; and setting the prediction mode flag associated with the current block based on the determined average context index.

[0084] Although Figure 7 illustrates example blocks of method 700, in some implementations, method 700 may include additional, fewer, different, or differently arranged blocks than those depicted in Figure 7. Additionally or alternatively, two or more of the blocks of method 700 may be performed in parallel.

[0085] The proposed methods may also be implemented by processing circuitry (e.g., one or more processors, or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium to perform one or more of the proposed methods.

[0086] FIG. 8 is a simplified block diagram of an apparatus (800) for controlling intra-inter prediction for decoding or encoding a video sequence, according to an embodiment.

[0087] 8, the device 800 includes a first determination code 810, an execution code 820, and a setting code 830. The device 800 may further include an increment code 840 and a second determination code 850.

[0088] A first decision code (810) is arranged to cause at least one processor to determine whether a neighboring block of the current block is coded by an intra-inter prediction mode.

[0089] Executable code (820) is arranged to cause at least one processor to perform intra-mode encoding of a current block using an intra-prediction mode associated with the intra-inter prediction mode based on determining that the neighboring block is encoded using the intra-inter prediction mode.

[0090] The setting code (830) is configured to cause at least one processor, based on determining that the neighboring block is coded using an intra-inter prediction mode, to perform the following operations: set a prediction mode flag associated with the neighboring block; obtain a context value based on the set prediction mode flag associated with the neighboring block; and perform entropy coding of a prediction mode flag associated with the current block using the obtained context value, wherein the prediction mode flag indicates that the current block is intra-coded.

[0091] The execution code (820) may further be configured to cause at least one processor to perform, based on determining that the neighboring block is coded using an intra-inter prediction mode, derivation of a most probable mode (MPM) for the current block using an intra-prediction mode associated with the intra-inter prediction mode.

[0092] The intra prediction mode associated with the intra-inter prediction mode may be a planar mode, a DC mode, or an intra prediction mode applied in the intra-inter prediction mode.

[0093] The setting code (830) may further be arranged to cause the at least one processor to set a prediction mode flag associated with the neighboring block to indicate that the neighboring block is intra-coded.

[0094] The setting code (830) may further be arranged to cause the at least one processor to set a prediction mode flag associated with the neighboring block to indicate that the neighboring block is inter-coded.

[0095] The first decision code (810) may be further configured to cause at least one processor to determine whether the neighboring block is coded in an intra prediction mode, an inter prediction mode, or an intra-inter prediction mode. The increment code (840) may be configured to cause at least one processor to increment a context index of a prediction mode flag associated with the current block by 2 based on determining that the neighboring block is coded in an intra prediction mode, increment the context index by 0 based on determining that the neighboring block is coded in an inter prediction mode, and increment the context index by 1 based on determining that the neighboring block is coded in an intra-inter prediction mode. The second decision code (850) may be further configured to cause the at least one processor to determine an average context index based on the incremented context index and the number of neighboring blocks of the current block. The setting code (830) may be further configured to cause the at least one processor to set a prediction mode flag associated with the current block based on the determined average context index.

[0096] The first decision code (810) may be further configured to cause at least one processor to determine whether the neighboring block is coded in an intra prediction mode, an inter prediction mode, or an intra-inter prediction mode. The increment code (840) may be configured to cause at least one processor to perform the following steps: incrementing a context index of a prediction mode flag associated with the current block by 1 based on determining that the neighboring block is coded in an intra prediction mode; incrementing the context index by 0 based on determining that the neighboring block is coded in an inter prediction mode; and incrementing the context index by 0.5 based on determining that the neighboring block is coded in an intra-inter prediction mode. The second decision code (850) may be configured to cause the at least one processor to determine an average context index based on the incremented context index and the number of neighboring blocks of the current block. The setting code (830) may be further configured to cause the at least one processor to set a prediction mode flag associated with the current block based on the determined average context index.

[0097] The above techniques may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media.

[0098] FIG. 9 is a diagram of a computer system (900) suitable for implementing an embodiment.

[0099] Computer software can be encoded in any suitable machine code or computer language, which can be subjected to mechanisms such as assembling, compiling, linking, etc. to create code containing instructions that can be executed directly by a computer central processing unit (CPU), graphics processing unit (GPU), etc., or interpreted, executed by microcode, etc.

[0100] The instructions may be executed by various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0101] 9 are exemplary in nature and are not intended to imply limitations on the scope or functionality of the computer software used to implement each embodiment. Neither the arrangement of components should be interpreted as having any dependency or requirement related to any one or combination of components illustrated in the exemplary embodiment of computer system 900.

[0102] The computer system 900 may include several human interface input devices. Such human interface input devices may respond to one or more human user inputs, such as tactile input (e.g., keystrokes, slides, data glove movements), audio input (e.g., voice, hand claps), visual input (e.g., posture), and olfactory input (not shown). Human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from static image capture devices), and video (e.g., two-dimensional video, three-dimensional video including stereo video).

[0103] The human interface input devices may include one or more of a keyboard (901), a mouse (902), a touchpad (903), a touch panel (910), a data glove (904), a joystick (905), a microphone (906), a scanner (907), and an image capture device (908) (only one of each is shown).

[0104] The computer system (900) may also include several human interface output devices. These human interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. These human interface output devices may include haptic output devices (e.g., touchscreens (910), haptic feedback devices such as data gloves (904), or joysticks (905), although some haptic feedback devices may not be used as input devices), audio output devices (e.g., speakers (909), headphones (not shown)), visual output devices (e.g., screens (910), including cathode ray tube (CRT) screens, liquid crystal display (LCD) screens, plasma screens, and organic light-emitting diode (OLED) screens, each of which may or may not have touchscreen input and haptic feedback capabilities, some of which may be capable of outputting two-dimensional visual output or three-dimensional or higher-dimensional output, such as by means of stereoscopic graphics output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0105] The computer system (900) may further include human-accessible storage devices and associated media, including optical media including CD / DVD ROM / RW (920) with media such as CD / DVD (921), thumb drives (922), removable hard drives or solid state drives (923), conventional magnetic media such as magnetic tape and floppy disks (not shown), dedicated ROM / ASIC / PLD-based devices such as security dongles (not shown), and the like.

[0106] Additionally, those skilled in the art should understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transitory signals.

[0107] The computer system 900 may further include an interface to one or more communications networks. The networks may be, for example, wireless, wired, optical, etc. The networks may also be local area, wide area, metropolitan, vehicular, industrial, real-time, delay-tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks (including Global System for Mobile Communications (GSM), Third Generation (3G), Fourth Generation (4G), Fifth Generation (5G), Long Term Evolution (LTE), etc.), television wired or wireless wide area digital networks (including cable television, satellite television, and terrestrial television), vehicular, and industrial (including CAN Bus), etc. Some networks typically require an external network interface adapter connected to a specific general-purpose data port or peripheral bus 949 (e.g., a Universal Serial Bus (USB) port on the computer system 900), while other networks are typically integrated into the core of the computer system 900 by connecting to a system bus, described below (e.g., integrated into an Ethernet interface on a PC computer system or a cellular network interface on a smartphone computer system). Using any of these networks, the computer system (900) can communicate with other entities. Such communication may be one-way receive only (e.g., broadcast television), one-way transmit only (e.g., a CANbus to a CANbus device), or two-way (e.g., reaching another computer system over a local-area or wide-area digital network). Specific protocols and protocol stacks may be used for each of these networks and network interfaces, as described above.

[0108] The above human interface devices, human-accessible storage devices, and network interfaces may be connected to the core 940 of the computer system 900.

[0109] The core (940) includes one or more central processing units (CPUs) (941), graphics processing units (GPUs) (942), specialized programmable processing units in the form of field programmable gate arrays (FPGAs) (943), hardware accelerators (944) for certain tasks, etc. These devices, along with read-only memory (ROM) (945), random access memory (RAM) (946), and internal mass storage devices (e.g., internal non-user-accessible hard disk drives, solid-state drives (SSDs), etc.) (947), are connected via a system bus (948). In some computer systems, access to the system bus (948) can be provided in the form of one or more physical plugs, allowing expansion with additional CPUs, GPUs, etc. Peripheral devices can be connected to the core's system bus (948) directly or via a peripheral bus (949). Peripheral bus architectures include Peripheral Component Interconnect (PCI), USB, etc.

[0110] The CPU (941), GPU (942), FPGA (943), and accelerator (944) can execute several instructions, which, when combined, constitute the computer code described above. The computer code may be stored in ROM (945) or RAM (946). Temporary data may also be stored in RAM (946), while permanent data may be stored, for example, in an internal mass storage device (947). A cache memory can provide fast storage and retrieval from any of the storage devices, and the cache memory may be closely associated with one or more of the CPU (941), GPU (942), mass storage device (947), ROM (945), RAM (946), etc.

[0111] The computer-readable media may have computer code thereon for performing various computer-implemented operations, and the media and computer code may be specially designed and constructed for the purposes of the embodiments, or they may be of the type well known and available to those skilled in the art of computer software.

[0112] By way of example, and not limitation, the computer system 900 having the architecture, and in particular the core 940, may provide functionality through the execution of software by a processor (including a CPU, GPU, FPGA, accelerator, etc.) embodied in one or more tangible computer-readable media. Such computer-readable media may include the user-accessible mass storage media discussed above, as well as some of the core's (940) storage devices that are non-transitory, such as the core's internal mass storage device 947 or ROM 945. Software for implementing various embodiments is stored on such devices and executed by the core 940. Depending on particular needs, the computer-readable media may include one or more memory devices or chips. The software causes the core 940, and in particular the processor (including a CPU, GPU, FPGA, etc.) therein, to perform specific processes or portions of specific processes, including defining data structures stored in RAM 946 and modifying those data structures based on the software-defined processes, as described herein. Additionally or alternatively, the computer system may provide functionality through logic implemented in hardwired or otherwise circuitry (e.g., accelerator 944), which may operate in place of or in conjunction with software to perform particular processes or portions of particular processes described herein. Where appropriate, references to software may include logic, and conversely, references to logic may include software. Where appropriate, references to computer-readable media may include circuitry (e.g., integrated circuits (ICs)) on which software is stored for execution, circuitry embodying logic for execution, or both. Embodiments include any appropriate combination of hardware and software.

[0113] While this disclosure has previously described several exemplary embodiments, there are alterations, substitutions, and various substitute equivalents that fall within the scope of this disclosure. It should thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope.

Claims

1. A video encoding method performed by an encoder, The steps include setting a first prediction mode flag pred_mode_flag associated with the adjacent block above or to the left of the current block to indicate whether the adjacent block is in inter-prediction mode or intra-prediction mode, The steps include entropy encoding the first prediction mode flag pred_mode_flag into the video bitstream, A step of setting a decision flag, the decision flag being used to make a final determination whether the adjacent block above or to the left of the current block is encoded in inter-prediction mode or intra-inter-prediction mode, wherein the decision flag being true indicates that the adjacent block is encoded in the intra-inter-prediction mode. The steps include entropy encoding the decision flag into the video bitstream, The steps include: deriving a context value for the second prediction mode flag pred_mode_flag associated with the current block based on the first prediction mode flag pred_mode_flag associated with the adjacent block; The steps include: using the derived context value to entropy encode the second prediction mode flag pred_mode_flag associated with the current block into the video bitstream; The steps include transmitting the aforementioned video bitstream, Methods that include...

2. A video encoding method performed by an encoder, The steps include setting a first prediction mode flag pred_mode_flag associated with the adjacent block above or to the left of the current block to indicate whether the adjacent block is in inter-prediction mode or intra-prediction mode, The steps include entropy encoding the first prediction mode flag pred_mode_flag into the video bitstream, A step of setting a decision flag, the decision flag being used to make a final determination whether the adjacent block above or to the left of the current block is encoded in inter-prediction mode or intra-inter-prediction mode, wherein the decision flag being true indicates that the adjacent block is encoded in the intra-inter-prediction mode. The steps include entropy encoding the decision flag into the video bitstream, The steps include: deriving a context value for the second prediction mode flag pred_mode_flag associated with the current block based on the first prediction mode flag pred_mode_flag associated with the adjacent block; The steps include: using the derived context value to entropy encode the second prediction mode flag pred_mode_flag associated with the current block into the video bitstream; The steps include storing the aforementioned video bitstream in a computer-readable medium, Methods that include...

3. The method according to claim 1 or 2, wherein the planar mode is always associated with the intrainter prediction mode.

4. Based on the final determination that the adjacent block is encoded by the intrainterpretation mode, the step of performing intrapredictive coding of the current block using the planar mode associated with the intrainterpretation mode, The method according to claim 3, further comprising:

5. The method according to claim 3 or 4, further comprising the step of deriving the most likely mode (MPM) of the current block using the planar mode associated with the intrainter prediction mode, based on the final determination that the adjacent block is encoded by the intrainter prediction mode.

6. The step of deriving the aforementioned context value is: Based on the determination that the adjacent block is encoded by intra-prediction mode, the step of incrementing the context value by 2, Based on the determination that the adjacent block is encoded by interprediction mode, the step of incrementing the context value by 0, Based on the determination that the adjacent block is encoded by the intrainterpretation mode, the step of incrementing the context value by 1, The method further includes, The steps include determining the average context index based on the context value after increment and the number of adjacent blocks to the current block, The steps include setting the second prediction mode flag based on the determined average context index, The method according to any one of claims 1 to 5, further comprising:

7. The step of deriving the context value relating to the second prediction mode flag is: Based on the determination that the adjacent block is encoded by intra-prediction mode, the step of incrementing the context value by 1, Based on the determination that the adjacent block is encoded by interprediction mode, the step of incrementing the context value by 0, Based on the determination that the adjacent block is encoded by the intrainterpretation mode, the step of incrementing the context value by 0.5, The method further includes, The steps include determining the average context index based on the context value after increment and the number of adjacent blocks to the current block, The steps include setting the second prediction mode flag based on the determined average context index, The method according to any one of claims 1 to 5, further comprising:

8. At least one memory location that stores the program, At least one processor connected to the memory, Includes, The program is configured to cause the at least one processor to execute the method described in any one of claims 1 to 7. Encoder.

9. A program for causing a computer to perform the method described in any one of claims 1 to 7.

Images