Video encoding method, encoder, and program

Abstract

To provide a method and an apparatus for controlling intra-inter prediction for decoding or encoding a video sequence.SOLUTION: A method determines whether a neighboring block of the current block is coded using an intra-inter prediction mode, on the basis of determining that the neighboring block is coded using the intra-inter prediction mode, performs intra-mode coding of the current block using an intra-prediction mode associated with the intra-inter prediction mode, sets a prediction mode flag associated with the neighboring block, obtains a context value on the basis of the prediction mode flag associated with the neighboring block, and performs entropy coding using the obtained context value for the prediction mode flag associated with the current block, which indicates that the current block is intra-coded.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] [Cross - Reference to Related Applications] This application claims priority to U.S. Provisional Patent Application No. 62 / 767,473, filed on November 14, 2018, and U.S. Application No. 16 / 454,294, filed on June 27, 2019, with the United States Patent and Trademark Office, the entire contents of which are incorporated herein by reference.

[0002] The methods and apparatuses corresponding to the embodiments relate to video coding, and in particular, to methods and apparatuses for improved context design for entropy - coding prediction modes and coding block flags (CBFs).

Background Art

[0003] FIG. 1A shows intra - prediction modes used in High Efficiency Video Coding (HEVC). In HEVC, there are a total of 35 intra - prediction modes. In these intra - prediction modes, mode 10 (101) is the horizontal mode, mode 26 (102) is the vertical mode, and modes 2 (103), mode 18 (104), and mode 34 (105) are diagonal modes. These intra - prediction modes are signaled by three most probable modes (MPMs) and the remaining 32 modes.

[0004] Regarding Versatile Video Coding (VVC), some coding unit syntax tables are shown below. When the slice type is not intra and the skip mode is not selected, the flag pred_mode_flag is signaled and the flag is encoded using only one context (e.g., the variable pred_mode_flag). The syntax tables of some coding units are as follows.

Table 1

[0005] Referring to Figure 1B, in VVC there are a total of 87 intra-prediction modes. Of these intra-prediction modes, mode 18 (106) is a horizontal mode, mode 50 (107) is a vertical mode, and modes 2 (108), 34 (109), and 66 (110) are diagonal modes. Modes 1 through 10 (111) and modes 67 through 76 (112) are called Wide-Angle Intra Prediction (WAIP) modes.

[0006] For the chromaticity component of the intracoding block, the encoder selects the optimal chromaticity prediction mode from five modes, including planar mode (mode index 0), DC mode (mode index 1), horizontal mode (mode index 18), vertical mode (mode index 50), and diagonal mode (mode index 66), and also selects a direct copy of the intra-prediction mode for the associated luminance component, i.e., the DM mode. Table 1 below shows the mapping between the intra-prediction direction of chromaticity and the intra-prediction mode number. [Table 2] [Overview of the project] [Problems that the invention aims to solve]

[0007] To avoid overlapping modes, the four modes other than DM mode are assigned based on the intra-prediction mode of the associated luminance (Luma) component. If the intra-prediction mode number for the chromatic component is 4, the intra-prediction direction for the luminance component is applied to the generation of intra-prediction samples for the chromatic component. If the intra-prediction mode number for the chromatic component is not 4 but the same as the intra-prediction mode number for the luminance component, the intra-prediction direction 66 is applied to the generation of intra-prediction samples for the chromatic component.

[0008] Multi-hypothesis intrainter prediction combines one intra prediction and one merge-indexed prediction, i.e., an intrainter prediction mode. In the merge coding unit (CU), a flag is transmitted as a signal for the merge mode; if the flag is true, an intra mode is selected from the intra candidate list. For the luminance component, the intra candidate list is obtained from four intra prediction modes, including DC mode, planar mode, horizontal mode, and vertical mode, and the size of the intra candidate list may be 3 or 4 depending on the block shape. If the width of the CU is greater than twice the height of the CU, the horizontal mode is removed from the intra candidate list, and if the height of the CU is greater than twice the width of the CU, the vertical mode is removed from the intra candidate list. A weighted average is used to combine one intra prediction mode selected by the intra mode index and one merge-indexed prediction selected by the merge index. For the chromaticity component, DM is always used without requiring additional signaling.

[0009] The weights for combining predictions are explained as follows: If DC mode or planar mode is selected, or if the width or height of a coding block (CB) is less than 4, equal weights are applied. For these CBs, if the width or height is 4 or greater, and horizontal / vertical mode is selected, first divide one CB vertically / horizontally into four equal-area regions. For each region, the corresponding (w_intra i ,w_inter iApply the weight set shown as ), where i is from 1 to 4, and (w_intra1,w_inter1)=(6,2), (w_intra2,w_inter2)=(5,3), (w_intra1,w_inter3)=(3,5), (w_intra4,w_inter4)=(2,6). (w_intra1,w_inter1) corresponds to the region closest to the reference sample, and (w_intra4,w_inter4) corresponds to the region furthest from the reference sample. Then, the combined prediction is calculated by summing the two weighted predictions and right-shifting by 3 bits. Additionally, the intra-prediction mode of the predictor's intra-hypothesis can be saved for subsequent intra-mode coding of adjacent CBs if those CBs are intra-coded. [Means for solving the problem]

[0010] According to one embodiment, a method for controlling intrainterpretation for decoding or encoding a video sequence is performed by at least one processor. The method includes the steps of: determining whether an adjacent block of the current block is encoded by an intrainterpretation mode; and, based on the determination that the adjacent block is encoded by an intrainterpretation mode, performing intramode encoding of the current block using the intrapretation mode associated with the intrainterpretation mode, setting a prediction mode flag associated with the adjacent block, obtaining a context value based on the set prediction mode flag associated with the adjacent block, and using the obtained context value to perform entropy coding on the prediction mode flag associated with the current block, indicating that the current block is intra-encoded.

[0011] According to one embodiment, a device for controlling intrainterpretation for decoding or encoding a video sequence includes at least one memory arranged to store computer program code, and at least one processor that accesses the at least one memory and is arranged to operate based on the computer program code. The computer program code includes a first determination code arranged to cause at least one processor to determine whether adjacent blocks of the current block are encoded by intrainterpretation mode, an execution code arranged to cause at least one processor to perform intramode encoding of the current block using the intraprediction mode associated with the intrainterpretation mode, based on the determination that adjacent blocks are encoded by intrainterpretation mode, and a setting code arranged to cause at least one processor to perform the operation of setting a prediction mode flag associated with an adjacent block, based on the determination that adjacent blocks are encoded by intrainterpretation mode, obtaining a context value based on the set prediction mode flag associated with the adjacent block, and using the obtained context value to perform entropy coding on the prediction mode flag associated with the current block, indicating that the current block is intra-encoded.

[0012] According to the embodiment, the instruction is stored in a non-temporary computer-readable storage medium, and the instruction causes at least one processor to perform the following steps: determine whether the adjacent block of the current block is encoded by an intra-interpredictive mode; and, based on the determination that the adjacent block is encoded by an intra-interpredictive mode, use the intrapredictive mode associated with the intra-interpredictive mode to perform intra-mode encoding of the current block, set a predictive mode flag associated with the adjacent block, obtain a context value based on the set predictive mode flag associated with the adjacent block, and use the obtained context value to perform entropy encoding on the predictive mode flag associated with the current block to indicate that the current block is intra-encoded. [Brief explanation of the drawing]

[0013] [Figure 1A] This is a diagram of the intra-prediction mode in HEVC. [Figure 1B] This is a diagram of the intra-prediction mode in VVC. [Figure 2] This is a simplified block diagram of a communication system according to an embodiment. [Figure 3] This diagram shows the arrangement of a video encoder and a video decoder in a streaming environment according to an embodiment. [Figure 4] This is a functional block diagram of a video decoder according to an embodiment. [Figure 5] This is a functional block diagram of a video encoder according to an embodiment. [Figure 6] This is a diagram of the current block and adjacent blocks to the current block according to an embodiment. [Figure 7] This flowchart shows a method for controlling intrainterface prediction for decoding or encoding a video sequence according to an embodiment. [Figure 8]Simplified block diagram of an apparatus for controlling intra-inter prediction for decoding or encoding of a video sequence according to an embodiment. [Figure 9] Drawing of a computer system suitable for implementing an embodiment.

Embodiments of the Invention

[0014] FIG. 2 is a simplified block diagram of a communication system (200) according to an embodiment. The communication system (200) may include at least two terminals (210-220) interconnected via a network (250). In the case of unidirectional data transmission, the first terminal (210) can transmit, via the network (250), video data at a local location to another terminal (220) by encoding the video data. The second terminal (220) can receive the encoded video data of another terminal from the network (250), decode the encoded video data, and display the restored video data. Unidirectional data transmission may be common in media service applications and the like.

[0015] FIG. 2 shows a second pair of terminals (230, 240) provided to support bidirectional transmission of encoded video that may occur, for example, during a video conference. In the case of bidirectional data transmission, each terminal (230, 240) can transmit, via the network (250), video data captured at a local location to another terminal by encoding the video data. Each terminal (230, 240) can also receive the encoded video data transmitted from another terminal, decode the encoded video data, and display the restored video data on a local display device.

[0016] In FIG. 2, the terminals (210 to 240) are shown as servers, personal computers, and smartphones, but the principles of the embodiments are not limited thereto. The embodiments are applicable to laptop computers, tablets, media players, and / or dedicated video conferencing devices. The network (250) represents any number of networks, including, for example, wired and / or wireless communication networks, for transmitting encoded video data between the terminals (210 to 240). The communication network (250) can exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of the examination of this application, the architecture and topology of the network (250) may not be important for the operation of the embodiments, unless otherwise described herein below.

[0017] FIG. 3 is a diagram of the arrangement of a video encoder and a video decoder in a streaming environment according to an embodiment. The disclosed theme can be equivalently applied to other applications for supporting video, including, for example, storage on digital media such as video conferencing, digital TV, CDs, DVDs, memory sticks, etc. that contain compressed video.

[0018] The streaming system may include a capture subsystem (313), which may include a video source (301) (e.g., a digital camera) for creating, for example, an uncompressed video sample stream (302). The sample stream (302) is drawn as a thick line to emphasize its larger data volume compared to the encoded video bitstream, and the sample stream (302) may be processed by an encoder (303) coupled to the imaging device (301). The encoder (303) may include hardware, software, or a combination thereof to realize or implement each aspect of the disclosed theme, which will be described in more detail below. The encoded video bitstream (304) is drawn as a thin line to emphasize its smaller data volume compared to the sample stream, and the encoded video bitstream (304) may be stored in a streaming server (305) for future use. One or more streaming clients (306, 308) can access a streaming server (305) to retrieve replicas (307, 309) of an encoded video bitstream (304). A client (306) may include a video decoder (310) for decoding an incoming replica (307) of encoded video data and creating an outgoing video sample stream (311) to be rendered to a display (312) or other rendering equipment (not shown). In some streaming systems, the video bitstreams (304, 307, 309) can be encoded based on several video encoding / compression standards. Examples of these standards include the ITU-T H.265 proposal. A video encoding standard informally called VVC is under development. The disclosed topics may apply in the context of VVC.

[0019] Figure 4 is a functional block diagram of a video decoder (310) according to an embodiment.

[0020] The receiver (410) can receive one or more codec video sequences to be decoded by the decoder (310), and in the same or other embodiments, it can receive one encoded video sequence at a time, with the decoding of each encoded video sequence being independent of other encoded video sequences. It can receive encoded video sequences from a channel (412), which may be a hardware / software link to a storage device for storing encoded video data. The receiver (410) can receive encoded video data and other data, such as encoded audio data and / or auxiliary data streams that can be transferred to their respective user entities (not shown). The receiver (410) can isolate the encoded video sequences from other data. To address network jitter, a buffer memory (415) can be linked between the receiver (410) and the entropy decoder / parser (hereinafter referred to as the "parser"). When the receiver (410) receives data from a storage / transfer device with sufficient bandwidth and controllability, or from an isochronous real-time network, it may not require a buffer memory (415), or the buffer (615) may be small. To make the most use of packet networks such as the Internet, a buffer memory (415) may be required, and the buffer memory may be relatively large and advantageously adaptive in size.

[0021] The video decoder (310) may include a parser (420) to reconstruct symbols (421) based on the entropy-encoded video sequence. These symbol categories include information for managing the operation of the video decoder (310) and potential information for controlling rendering equipment such as a display (312), which, as shown in Figure 4, is not a component of the decoder but can be linked to it. The control information for the rendering equipment may be in the form of Supplemental Enhancement Information (SEI messages) or Video Usability Information (VUI) parameter set fragments (not shown). The parser (420) performs parsing / entropy decoding on the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, and arithmetic coding with or without context. The parser extracts a set of subgroup parameters for at least one subgroup of pixels in the video decoder from the encoded video sequence, based on at least one parameter corresponding to a group. Subgroups include groups of picture (GOP), picture, tile, slice, macroblock, coding unit (CU), block, transform unit (TU), prediction unit (PU), and others. The entropy decoder / parser may further extract information from the encoded video sequence, such as transform coefficients, quantizer parameter (QP) values, and motion vectors.

[0022] The parser (420) can create symbols (421) by performing an entropy decoding / analysis operation on the video sequence received from the buffer memory (415). The parser may receive encoded data and selectively decode specific symbols (421). The parser can also determine whether a particular symbol (421) is provided to a motion compensation prediction unit (453), a scaler / inverse transform unit (451), an intra prediction unit (452), or a loop filter (454).

[0023] Depending on the encoded video picture or certain types of encoded video pictures (e.g., interpicture and intrapicture, interblock and intrablock) and other factors, the reconstruction of the symbol (421) may involve multiple different units. The units and methods of involvement are controlled by subgroup control information analyzed by the parser (420) from the encoded video sequence. For brevity, the flow of such subgroup control information between the parser (420) and the following multiple units is not described.

[0024] In addition to the functional blocks already mentioned, the decoder (310) can be conceptually subdivided into several functional units, as described below. In actual implementations carried out under commercial constraints, many of these units may interact closely with each other and be at least partially integrated with each other. However, for the purpose of illustrating the disclosed topic, it is appropriate to conceptually subdivide it into the following functional units.

[0025] The first unit is the scaler / inverse unit (451). The scaler / inverse unit (451) receives quantization transformation coefficients and control information as symbols (421) from the parser (420), including the transformation scheme to be used, block size, quantization factor, and quantization scaling matrix. It can output a block containing sample values ​​that can be input to the aggregator (455).

[0026] In some cases, the output samples of the scaler / inverse transform (451) may belong to intracoding blocks, i.e., blocks that do not use prediction information from previously reconstructed pictures, but can use prediction information from previously reconstructed portions of the current picture. Such prediction information may be provided by an intra-prediction unit (452). In some cases, the intra-prediction unit (452) generates a block of the same size and shape as the block being reconstructed, using information from the already reconstructed periphery obtained from the currently (partially reconstructed) picture (456). In some cases, the aggregator (455) adds the prediction information generated by the intra-prediction unit (452) to the output sample information provided by the scaler / inverse transform unit (451), based on each sample.

[0027] In other cases, the output samples of the scaler / inverse unit (451) may belong to an intercoded and potentially motion-compensated block. In such cases, the motion-compensated prediction unit (453) can access the reference picture buffer (457) to obtain samples for prediction. After motion compensation is performed on the obtained samples based on the symbols (421) belonging to the block, these samples can be added by the aggregator (455) to the output of the scaler / inverse unit (in this case, called residual samples or residual signals) to generate output sample information. The address in the reference picture memory from which the motion-compensated prediction unit extracts prediction samples may be controlled by a motion vector, which can be used by the motion-compensated prediction unit in the form of a symbol (421), and which may have, for example, X, Y, and reference picture components. Motion compensation may further include interpolation of sample values ​​obtained from the reference picture memory when the exact motion vector of a subsample is used, a motion vector prediction mechanism, and so on.

[0028] The output samples of the aggregator (455) may be processed in the loop filter unit (454) by various loop filtering techniques. The video compression technique may include an in-loop filtering technique, which is controlled by parameters contained in the encoded video bitstream, which can be applied to the loop filter unit (454) as symbols (421) from the parser (420). However, the video compression technique may further respond to metadata obtained during the period of decoding earlier portions (in the decoding order) of the encoded picture or encoded video sequence, or to previously constructed and loop-filtered sample values.

[0029] The output of the loop filter unit (454) may be a sample stream, which may be output to the rendering device (312) and stored in the reference picture buffer (456) for use in future inter-picture prediction.

[0030] Once an encoded picture has been fully reconstructed, it is used as a reference picture for future predictions. When an encoded picture has been fully reconstructed and is recognized as a reference picture (for example by a parser (420)), the current reference picture (456) becomes part of the reference picture buffer (457), allowing new current picture memory to be reallocated before subsequent reconstructions of encoded pictures begin.

[0031] The video decoder (310) may perform the decoding operation based on a specified video compression technique, for example, as documented in the ITU-T H.265 proposal. The encoded video sequence conforms to the syntax specified by the video compression technique or standard used, in the sense that it conforms to the syntax of the video compression technique or standard explicitly specified in its profile, for example, in the video compression technique documentation or standard. For compliance, it is also required that the complexity of the encoded video sequence is within the limits set by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megabytes of samples per second), maximum reference picture size, etc. In some cases, the limits set by the level may be further limited by the specification of the Hypothetical Reference Decoder (HRD) and the HRD buffer management metadata signaled in the encoded video sequence.

[0032] In one embodiment, the receiver (410) can receive additional (redundant) data along with the encoded video. The additional data is included as part of the encoded video sequence. The additional data is utilized by the video decoder (310) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, a time, space, or signal-to-noise ratio (SNR) extension layer, redundant slices, redundant pictures, or forward error correction codes.

[0033] Figure 5 may be a functional block diagram of the video encoder (303) according to an embodiment.

[0034] The encoder (303) can receive video samples from a video source (301) (which is not part of the encoder), and the video source can capture video images that are encoded by the encoder (303).

[0035] The video source (301) may provide a source video sequence in the form of a digital video sample stream encoded by an encoder (303), the digital video sample stream may include any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 Y CrCB, RGB, etc.), and any suitable sampling configuration (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source (301) may be a storage device for storing previously prepared video. In a video conferencing system, the video source (301) may be a camera for capturing local image information as a video sequence. The video data may be provided as a series of separate pictures that are given motion when viewed sequentially. The pictures themselves may be organized as spatial pixel arrays, and each pixel may contain one or more samples depending on the sampling configuration, color space, etc. used. The relationship between pixels and samples will be easily understood by those skilled in the art. The following description focuses on samples.

[0036] According to the embodiment, the encoder (303) encodes the pictures of the source video sequence in real time or under any other time constraints required by the application, and compresses them as an encoded video sequence (543). Performing this at an appropriate encoding speed is one of the functions of the controller (550). The controller controls and is functionally coupled to other functional units described below. For brevity, the coupling is not shown. Parameters set by the controller may include rate control-related parameters (e.g., picture skip, quantizer, λ value of rate distortion optimization technique), picture size, picture group (GOP) arrangement, maximum motion vector search range, etc. Those skilled in the art will readily recognize that the other functions of the controller (550) relate to the video encoder (303) optimized for a particular system design.

[0037] Some video encoders operate in an "encoding loop" that is readily understandable to those skilled in the art. In a very simplified description, the encoding loop may include an encoding portion of an encoder (530) (hereafter referred to as the "source encoder") (responsible for creating symbols based on the input picture to be encoded and a reference picture), and a (local) decoder (533) embedded in the encoder (303), the decoder (533) reconstructs the symbols to create sample data that would also be created by a (remote) decoder (for any compression between the symbols and the encoded video bitstream is reversible in the video compression techniques considered in the subject of this disclosure). The reconstructed sample stream is input to a reference picture memory (534). Decoding of the symbol stream results in bit-accurate results regardless of the decoder location (local or remote), so that the contents of the reference picture buffer are bit-accurate between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the predictive portion of the encoder are exactly the same as the sample values ​​"seen" by the decoder when it uses the prediction during decoding. This fundamental principle of reference picture synchronization (and the drift that occurs when synchronization cannot be maintained, for example, due to channel errors) is well known to those skilled in the art.

[0038] The operation of the “local” decoder (533) may be the same as the operation of the “remote” decoder (310) as described above with reference to Figure 4. However, if, with reference to Figure 4, the symbols are available and the entropy encoder (545) and parser (420) can losslessly encode / decode the symbols into an encoded video sequence, then the entropy decoding portion of the decoder (310), which includes the channels (412), receiver (410), buffer memory (415), and parser, does not need to be fully implemented in the decoder (533).

[0039] In this case, any decoding techniques other than analysis / entropy decoding present in the decoder can inevitably be observed to exist in the corresponding encoder in essentially the same functional form. Since encoder techniques and fully described decoder techniques are inverses of each other, the explanation of encoder techniques can be simplified. A more detailed explanation is necessary only in a few places and is provided below.

[0040] As part of the operation of the source encoder (530), the source encoder (530) may perform motion-compensated predictive coding, which predictively codes the input frame by referencing one or more previously coded frames from the video sequence designated as “reference frames”. In this way, the coding engine (532) may encode the difference between a pixel block of the input frame and a pixel block that can be selected as the reference frame of the predictive reference of the input frame.

[0041] The decoder (533) decodes previously encoded video data of a frame designated as a reference frame based on the code created by the source encoder (530). The operation of the encoding engine (532) may favorably be lossy. When the encoded video data is decoded by a video decoder (not shown in Figure 4), the reconstructed video sequence may generally be a replica of the source video sequence with some errors. The decoder (533) copies the decoding process that may be performed on the reference frame by the video decoder and can store the reconstructed reference frame in a reference picture cache (534). In this way, the video encoder (303) can locally store replicas of the reconstructed reference frame, and these replicas have common content (no transmission errors) with the reconstructed reference frame obtained by a remote video decoder.

[0042] The predictor (535) performs a predictive search against the encoding engine (532). That is, for a new frame to be encoded, the predictor (535) may search the reference picture memory (534) for sample data (as candidate reference pixel blocks) that can be used as appropriate predictive references for the new picture, or for specific metadata such as the motion vector or block shape of the reference picture. The predictor (535) can find appropriate predictive references by working pixel block by pixel based on the sample blocks. In some cases, the input picture may have predictive references obtained from multiple reference pictures stored in the reference picture memory (534), as determined from the search results obtained by the predictor (535).

[0043] The controller (550) can manage the encoding operations of the source encoder (530), including, for example, setting parameters and subgroup parameters for encoding video data.

[0044] In the entropy encoder (545), entropy coding may be performed on the outputs of all the above functional units. The entropy encoder converts the symbols generated by the various functional units into an encoded video sequence by performing lossless compression on the symbols based on techniques known to those skilled in the art (e.g., Huffman coding, variable-length coding, arithmetic coding, etc.).

[0045] The transmitter (540) may buffer the encoded video sequence created by the entropy encoder (545) in preparation for transmission over the communication channel (560), which may be a hardware / software link to a storage device for storing the encoded video data. The transmitter (540) may merge the encoded video data from the source encoder (530) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0046] The controller (550) can manage the operation of the encoder (303). During encoding, the controller (550) may also assign a specific encoded picture type to each encoded picture, which may affect the encoding technique applied to the corresponding picture. For example, generally, a picture is assigned as one of the following frame types:

[0047] An intra-picture (I-picture) may be a picture that is encoded and decoded without using any other frame in the sequence as a source for prediction. Some video codecs allow different types of intra-pictures, including, for example, "Independent Decoder Refresh" pictures. Those skilled in the art will know of their variations of I-pictures and their corresponding uses and characteristics.

[0048] A prediction picture (P-picture) may be a picture that is encoded and decoded using intra-prediction or inter-prediction to predict the sample value of each block using at most one motion vector and reference index.

[0049] A bidirectional predictive picture (B-picture) may be a picture that encodes and decodes using intra-prediction or inter-prediction to predict the sample values ​​of each block using at most two motion vectors and a reference index. Similarly, multiple predictive pictures can use two or more reference images and associated metadata to reconstruct a single block.

[0050] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each), and each block may be encoded. A block may be predictively encoded by referencing other (already encoded) blocks determined by the encoding assignment applied to the corresponding picture of that block. For example, a block of picture I may be encoded unpredictably or predictively by referencing encoded blocks of the same picture (spatial prediction or intra-prediction). A pixel block of picture P may be encoded unpredictably by referencing one previously encoded reference picture via spatial prediction or temporal prediction. A block of picture B may be encoded unpredictably by referencing one or two previously encoded reference pictures via spatial prediction or temporal prediction.

[0051] The video encoder (303) may perform encoding operations based on, for example, the specified video encoding techniques or standards in the ITU-T H.265 proposal. During operation of the video encoder (303), it may perform various compression operations, including predictive encoding operations based on temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard being used.

[0052] In an embodiment, the transmitter (540) can transmit additional data along with the encoded video. The source encoder (530) may include such data as part of the encoded video sequence. The additional data may include time / space / SNR extension layers, other forms of redundant data such as redundant pictures and slices, supplemental extension information (SEI) messages, video usability information (VUI) parameter set segments, and the like.

[0053] In conventional techniques, to encode the flag pred_mode_flag, which indicates whether a block is intra-encoded or inter-encoded, only one context is used, rather than the value of a flag applied to adjacent blocks. Furthermore, if adjacent blocks are encoded by intra-inter-prediction mode, it may be more efficient to use a combination of intra-prediction mode and inter-prediction mode to predict the adjacent block and therefore consider whether the adjacent block is encoded by intra-inter-prediction mode in the context design for signaling the flag pred_mode_flag.

[0054] The embodiments described herein may be used individually or in any combination of any order. The flag pred_mode_flag indicates whether the current block is intra-encoded or inter-encoded.

[0055] Figure 6 shows the current block and the adjacent blocks to the current block according to the embodiment.

[0056] Referring to Figure 6, the current block (610) and its top adjacent block (620) and left adjacent block (630) are shown. The top adjacent block (620) and left adjacent block (630) each have a width of 4 and a height of 4.

[0057] In one embodiment, information on whether adjacent blocks (e.g., the top adjacent block (620) and the left adjacent block (630)) are encoded by intra-prediction mode, inter-prediction mode, or intra-inter-prediction mode is used to obtain a context value for entropy coding the flag pred_mode_flag of the current block (e.g., the current block (610)). Specifically, if adjacent blocks are encoded by intra-inter-prediction mode, the associated intra-prediction mode is applied to the intra-mode encoding and / or MPM derivation of the current block; however, when deriving a context value for entropy coding the flag pred_mode_flag of the current block, the adjacent block is considered an intercoding block, even though intra-prediction was used for it.

[0058] In one example, the intra-prediction mode associated with an intra-prediction mode is always a planar mode.

[0059] In other examples, the associated intra-prediction mode of an intra-inter prediction mode is always DC mode.

[0060] In further examples, the associated intra-prediction mode aligns with the intra-prediction mode applied in the intra-internal prediction mode.

[0061] In one embodiment, when an intra-interpretation mode encodes adjacent blocks (e.g., the top adjacent block (620) and the left adjacent block (630)), the associated intra-prediction mode is applied to the intra-mode encoding and / or MPM derivation of the current block (e.g., the current block (610)), and when deriving a context value for entropy encoding the flag pred_mode_flag of the current block, the adjacent blocks are also considered to be intra-coding blocks.

[0062] In one example, the intra-prediction mode associated with the intra-prediction mode is always a planar mode.

[0063] In other examples, the associated intra-prediction mode for an intra-inter prediction mode is always DC mode.

[0064] In further examples, the associated intra-prediction mode aligns with the intra-prediction mode applied in the intra-inter-prediction mode.

[0065] In one embodiment, if adjacent blocks are encoded using intra-prediction mode, inter-prediction mode, and inter-intra-prediction mode, respectively, the context index or value is incremented by 2, 0, and 1, respectively.

[0066] In other embodiments, if adjacent blocks are encoded by intra-prediction mode, inter-prediction mode, and inter-intra-prediction mode, respectively, the context index or value is incremented by 1, 0, and 0.5, respectively, and the final context index is rounded to the nearest integer.

[0067] The context index or value can be incremented for all adjacent blocks of the current block to determine the final context index. The average context index can then be determined by dividing the determined final context index by the number of adjacent blocks and rounding to the nearest integer. Based on the determined average context index, the flag pred_mode_flag is set to indicate that the current block is intra-encoded or inter-encoded, and the pred_mode_flag of the current block is encoded by performing arithmetic coding.

[0068] In one embodiment, information on whether the current block (e.g., current block (610)) is encoded in intra-prediction mode, inter-prediction mode, or inter-intra-prediction mode is used to obtain one or more context values ​​for entropy encoding the CBF of the current block.

[0069] In one embodiment, three distinct contexts (e.g., variables) are used to entropy encode the CBF, one context used when the current block is encoded by intra-predictive mode, one context used when the current block is encoded by inter-predictive mode, and one context used when the current block is encoded by intra-inter-predictive mode. These three distinct contexts may be applied only to encode the luminance CBF, only to encode the chromaticity CBF, or only to encode both the luminance CBF and the chromaticity CBF.

[0070] In other embodiments, two separate contexts (e.g., variables) are used to entropy encode the CBF, one context used when the current block is encoded by intra-predictive mode, and the other context used when the current block is encoded by inter-predictive mode or intra-inter-predictive mode. These two separate contexts may be applied only to encode the luminance CBF, only to encode the chromaticity CBF, or only to encode both the luminance CBF and the chromaticity CBF.

[0071] In yet another embodiment, two separate contexts (e.g., variables) are used to entropy encode the CBF, one context used when the current block is encoded by intra-predictive mode or intra-interpredictive mode, and the other context used when the current block is encoded by interpredictive mode. These two separate contexts may be applied only to encode a luminance CBF, only to encode a chromaticity CBF, or only to encode both a luminance and a chromaticity CBF.

[0072] Figure 7 is a flowchart illustrating a method (700) for controlling intrainterpretation for decoding or encoding a video sequence according to an embodiment. In some implementations, one or more processing blocks in Figure 7 may be performed by a decoder (310). In some implementations, one or more processing blocks in Figure 7 may be performed by a decoder (310) and other devices or groups of devices, separate from or including the decoder (310) (e.g., an encoder (303)).

[0073] Referring to Figure 7, in the first block (710), method (700) includes the step of determining whether the adjacent block of the current block is encoded by intrainterpretation mode. Based on the determination that the adjacent block is not encoded by intrainterpretation mode (710—NO), method (700) terminates.

[0074] Based on the determination that the adjacent block is encoded by an intra-interpretation mode (710—YES), in the second block (720), method (700) includes the step of performing intra-mode encoding of the current block using the intraprediction mode associated with the intra-interpretation mode.

[0075] In the third block (730), the method (700) includes the step of setting a prediction mode flag associated with an adjacent block.

[0076] In the fourth block (740), the method (700) includes the step of obtaining a context value based on a set prediction mode flag associated with an adjacent block.

[0077] In block 5 (750), method (700) includes the step of using the acquired context value to perform entropy coding of the prediction mode flag associated with the current block, indicating that the current block is intra-encoded.

[0078] Method (700) further includes the step of performing the derivation of the MPM of the current block using the intra-prediction mode associated with the intra-inter-prediction mode, based on the determination that the adjacent block is encoded by the intra-inter-prediction mode (710—YES).

[0079] The intra-prediction mode associated with the intra-inter prediction mode may be the planar mode, DC mode, or the intra-prediction mode applied in the intra-inter prediction mode.

[0080] Setting a prediction mode flag associated with an adjacent block may include the step of setting a prediction mode flag associated with an adjacent block to indicate that the adjacent block is intra-encoded.

[0081] Setting a prediction mode flag associated with an adjacent block may include the step of setting a prediction mode flag associated with an adjacent block to indicate that the adjacent block is intercoded.

[0082] Method (700) may further include the steps of: determining whether an adjacent block is encoded by intra-prediction mode, inter-prediction mode, or intra-inter-prediction mode; incrementing the context index of the prediction mode flag associated with the current block by 2 based on the determination that the adjacent block is encoded by intra-prediction mode, incrementing the context index by 0 based on the determination that the adjacent block is encoded by inter-prediction mode, and incrementing the context index by 1 based on the determination that the adjacent block is encoded by intra-inter-prediction mode; determining the average context index based on the incremented context index and the number of adjacent blocks of the current block; and setting the prediction mode flag associated with the current block based on the determined average context index.

[0083] The method may further include the steps of: determining whether an adjacent block is encoded by intra-prediction mode, inter-prediction mode, or intra-inter-prediction mode; incrementing the context index of the prediction mode flag associated with the current block by 1 based on the determination that the adjacent block is encoded by intra-prediction mode, incrementing the context index by 0 based on the determination that the adjacent block is encoded by inter-prediction mode, and incrementing the context index by 0.5 based on the determination that the adjacent block is encoded by intra-inter-prediction mode; determining the average context index based on the incremented context index and the number of adjacent blocks of the current block; and setting the prediction mode flag associated with the current block based on the determined average context index.

[0084] Figure 7 shows an example of blocks of method (700), but in some implementations, method (700) may include additional blocks, fewer blocks, different blocks, or blocks in different arrangements than those depicted in Figure 7. In addition or alternatively, two or more blocks of method (700) may be executed in parallel.

[0085] Furthermore, the proposed method may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-temporary computer-readable medium for performing one or more of the proposed methods.

[0086] Figure 8 is a simplified block diagram of an apparatus (800) for controlling intrainterface prediction for decoding or encoding a video sequence, according to an embodiment.

[0087] Referring to Figure 8, the device (800) includes a first decision code (810), an execution code (820), and a setting code (830). The device (800) may further include an increment code (840) and a second decision code (850).

[0088] The first decision code (810) is positioned to cause at least one processor to determine whether the adjacent blocks of the current block are encoded by the intrainterpredictive mode.

[0089] The executable code (820) is configured to cause at least one processor to perform intra-mode coding of the current block using the intra-prediction mode associated with the intra-inter-prediction mode, based on the determination that the adjacent block is coded by the intra-inter-prediction mode.

[0090] The configuration code (830) is configured to cause at least one processor to operate as follows, based on the determination that an adjacent block is encoded by intra-interpredictive mode: set a predictive mode flag associated with the adjacent block; obtain a context value based on the set predictive mode flag associated with the adjacent block; and use the obtained context value to perform entropy coding of the predictive mode flag associated with the current block, so that the predictive mode flag indicates that the current block is intra-encoded.

[0091] The executable code (820) may further configure at least one processor to perform the derivation of the most likely mode (MPM) of the current block, using the intra-prediction mode associated with the intra-inter-prediction mode, based on the determination that the adjacent block is encoded by the intra-inter-prediction mode.

[0092] The intra-prediction mode associated with the intra-inter prediction mode may be the planar mode, DC mode, or any other intra-prediction mode applied in the intra-inter prediction mode.

[0093] The configuration code (830) may further be configured to cause at least one processor to set a prediction mode flag associated with the adjacent block to indicate that the adjacent block is intra-encoded.

[0094] The configuration code (830) may further be configured to cause at least one processor to set a prediction mode flag associated with the adjacent block to indicate that the adjacent block is intercoded.

[0095] The first decision code (810) may further cause at least one processor to determine whether an adjacent block is encoded in intra-prediction mode, inter-prediction mode, or intra-inter-prediction mode. The increment code (840) may further cause at least one processor to perform the following steps: increment the context index of the prediction mode flag associated with the current block by 2 based on the determination that the adjacent block is encoded in intra-prediction mode; increment the context index by 0 based on the determination that the adjacent block is encoded in inter-prediction mode; and increment the context index by 1 based on the determination that the adjacent block is encoded in intra-inter-prediction mode. The second decision code (850) may further cause at least one processor to determine the average context index based on the incremented context index and the number of adjacent blocks of the current block. The setting code (830) may further cause at least one processor to set the prediction mode flag associated with the current block based on the determined average context index.

[0096] The first decision code (810) may further cause at least one processor to determine whether an adjacent block is encoded in intra-prediction mode, inter-prediction mode, or intra-inter-prediction mode. The increment code (840) may cause at least one processor to perform the following steps: increment the context index of the prediction mode flag associated with the current block by 1 based on the determination that the adjacent block is encoded in intra-prediction mode; increment the context index by 0 based on the determination that the adjacent block is encoded in inter-prediction mode; and increment the context index by 0.5 based on the determination that the adjacent block is encoded in intra-inter-prediction mode. The second decision code (850) may further cause at least one processor to determine the average context index based on the incremented context index and the number of adjacent blocks of the current block. The setting code (830) may further cause at least one processor to set the prediction mode flag associated with the current block based on the determined average context index.

[0097] The above technology may be implemented as computer software using computer-readable instructions and may be physically stored on one or more computer-readable media.

[0098] Figure 9 shows a computer system (900) suitable for realizing the embodiment.

[0099] Computer software can be encoded in any suitable machine code or computer language, and by performing mechanisms such as assembly, compilation, and linking on the machine code or computer language, code can be created that contains instructions that are executed directly by a computer's central processing unit (CPU), graphics processing unit (GPU), etc., or by interpretation, microcode, etc.

[0100] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, and Internet of Things devices.

[0101] The components for the computer system (900) shown in Figure 9 are illustrative in nature and are not intended to limit the scope or functionality of the computer software used to implement each embodiment. The arrangement of the components should not be interpreted as having any dependencies or requirements relating to any of the components or combinations thereof shown in the exemplary embodiment of the computer system (900).

[0102] The computer system (900) may include several human interface input devices. Such human interface input devices can respond to input from one or more human users, such as tactile input (e.g., keystrokes, slides, data glove movements), audio input (e.g., voices, clapping sounds), visual input (e.g., posture), and olfactory input (not shown). Human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., voices, music, ambient sounds), images (e.g., scanned images, photographic images acquired from a static imaging device), and video (e.g., 2D video, 3D video including stereo video).

[0103] Human interface input devices may include one or more of the following: keyboard (901), mouse (902), touchpad (903), touch panel (910), data glove (904), joystick (905), microphone (906), scanner (907), and imaging device (908) (only one of each is shown).

[0104] The computer system (900) may further include several human interface output devices. Such human interface output devices can stimulate one or more human user senses, for example, through tactile output, sound, light and smell / taste. Such human interface output devices include tactile output devices (e.g., tactile feedback devices such as touch panels (910), data gloves (904), or joysticks (905), but there are also tactile feedback devices that are not used as input devices), audio output devices (e.g., speakers (909), headphones (not shown)), visual output devices (e.g., screens (910), including cathode ray tube (CRT) screens, liquid crystal display (LCD) screens, plasma screens, organic light-emitting diode (OLED) screens, each of which may or may not have touchscreen input capability and tactile feedback capability, and some of these screens may be able to output two-dimensional visual output or three-dimensional or more output by means such as stereographic graphics output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0105] The computer system (900) may further include human-accessible storage devices and related media, including optical media such as CD / DVD ROM / RW (920) having media such as CD / DVD (921), thumb drives (922), removable hard drives or solid-state drives (923), conventional magnetic media such as magnetic tape and floppy disks (not shown), and devices based on dedicated ROM / ASIC / PLD such as security dongles (not shown).

[0106] Furthermore, those skilled in the art should understand that the term “computer-readable medium” as used in relation to the present disclosed topic does not include transmission media, carrier waves, or other transient signals.

[0107] The computer system (900) may further include interfaces to one or more communication networks. These networks may be, for example, wireless, wired, or optical. The networks may also be local area, wide area, metropolitan, vehicle and industrial, real-time, or latency-tolerant. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks (including Global Mobile Communication Systems (GSM), third-generation (3G), fourth-generation (4G), fifth-generation (5G), and Long-Term Evolution (LTE)), wired or wireless wide-area digital television networks (including wired television, satellite television, and terrestrial television), and vehicle and industrial networks (including CANBus). Some networks generally require external network interface adapters connected to specific general-purpose data ports or peripheral buses (949) (e.g., the Universal Serial Bus (USB) port of the computer system (900)), while others are generally integrated into the core of the computer system (900) by connecting to system buses described below (e.g., the Ethernet interface of a PC computer system, or the cellular network interface of a smartphone computer system). Using any of these networks, the computer system (900) can communicate with other entities. Such communication may be one-way receive only (e.g., broadcast television), one-way transmit only (e.g., CANbus to a CANbus device), or two-way (e.g., reaching other computer systems via a local area or wide area digital network). Specific protocols and protocol stacks may be used for each of these networks and network interfaces as described above.

[0108] The above-mentioned human interface devices, human-accessible memory devices, and network interfaces may be connected to the core (940) of the computer system (900).

[0109] The core (940) includes one or more central processing units (CPUs) (941), graphics processing units (GPUs) (942), specialized programmable processing units in the form of field-programmable gate arrays (FPGAs) (943), and hardware accelerators (944) for several tasks. These devices, along with read-only memory (ROM) (945), random access memory (RAM) (946), and internal mass storage devices (e.g., internal hard disk drives, solid-state drives (SSDs), etc.) (947), are connected via a system bus (948). In some computer systems, expansion with additional CPUs, GPUs, etc., can be enabled by accessing the system bus (948) in the form of one or more physical plugs. Peripheral devices may be connected to the core's system bus (948) directly or via a peripheral bus (949). Peripheral bus architectures include Peripheral Component Interconnect (PCI), USB, etc.

[0110] The CPU (941), GPU (942), FPGA (943), and accelerator (944) can execute several instructions, and these instructions can be combined to form the computer code described above. This computer code may be stored in ROM (945) or RAM (946). Temporary data may also be stored in RAM (946), and permanent data may be stored in, for example, an internal mass storage device (947). Cache memory can enable high-speed storage and retrieval to any of the storage devices, and this cache memory can be closely associated with one or more CPUs (941), GPUs (942), mass storage devices (947), ROMs (945), RAM (946), etc.

[0111] A computer-readable medium may have computer code thereon for performing various operations that a computer can perform. The medium and computer code may be a medium and computer code specifically designed and constructed for the purposes of the embodiment, or they may be of a type that is well known and available to those skilled in the art of computer software.

[0112] To the present, and not to the extent of the limitations, a computer system (900) having an architecture, particularly a core (940), can provide functionality by having a processor (including a CPU, GPU, FPGA, accelerator, etc.) execute software implemented on one or more tangible computer-readable media. Such computer-readable media may be media relating to user-accessible mass storage devices as described above, and some non-transient storage devices of the core (940), such as internal mass storage devices (947) or ROM (945). Software for realizing various embodiments is stored in such devices and executed by the core (940). Depending on the specific needs, the computer-readable media may include one or more memory devices or chips. The software causes the core (940), particularly its processor (including a CPU, GPU, FPGA, etc.), to execute a specific process or a specific part of a specific process, which includes limiting data configurations stored in RAM (946) as described herein, and modifying these data configurations based on processes limited by the software. Furthermore, or alternatively, the computer system may provide functionality through logic implemented in hardwired or otherwise in circuitry (e.g., accelerators (944)), and such Rozoc may, in place of or in conjunction with software, execute specific processes or specific parts of specific processes described herein. Where appropriate, references to software may include logic, and conversely, references to logic may include software. Where appropriate, references to computer-readable media may include circuitry (e.g., integrated circuits (ICs)) in which software for execution is stored, circuitry embodying logic for execution, or both. Embodiments include any suitable combination of hardware and software.

[0113] While several exemplary embodiments have already been described in this disclosure, there are many variations, substitutions, and alternative equivalents that fall within the scope of this disclosure. Therefore, those skilled in the art will understand that numerous systems and methods, not expressly shown or described herein, can be devised that embody the principles of this disclosure and thus fall within its spirit and scope.

Claims

1. A video encoding method performed by an encoder, The steps include setting a first prediction mode flag pred_mode_flag associated with the adjacent block above or to the left of the current block to indicate whether the adjacent block is in inter-prediction mode or intra-prediction mode, The steps include entropy encoding the first prediction mode flag pred_mode_flag into the video bitstream, A step of setting a decision flag, the decision flag being used to make a final determination whether the adjacent block above or to the left of the current block is encoded in inter-prediction mode or intra-inter-prediction mode, wherein the decision flag being true indicates that the adjacent block is encoded in the intra-inter-prediction mode. The steps include entropy encoding the decision flag into the video bitstream, The steps include: deriving a context value for the second prediction mode flag pred_mode_flag associated with the current block based on the first prediction mode flag pred_mode_flag associated with the adjacent block; The steps include: using the derived context value to entropy encode the second prediction mode flag pred_mode_flag associated with the current block into the video bitstream; The steps include transmitting the aforementioned video bitstream, Methods that include...

2. A video encoding method performed by an encoder, The steps include setting a first prediction mode flag pred_mode_flag associated with the adjacent block above or to the left of the current block to indicate whether the adjacent block is in inter-prediction mode or intra-prediction mode, The steps include entropy encoding the first prediction mode flag pred_mode_flag into the video bitstream, A step of setting a decision flag, the decision flag being used to make a final determination whether the adjacent block above or to the left of the current block is encoded in inter-prediction mode or intra-inter-prediction mode, wherein the decision flag being true indicates that the adjacent block is encoded in the intra-inter-prediction mode. The steps include entropy encoding the decision flag into the video bitstream, The steps include: deriving a context value for the second prediction mode flag pred_mode_flag associated with the current block based on the first prediction mode flag pred_mode_flag associated with the adjacent block; The steps include: using the derived context value to entropy encode the second prediction mode flag pred_mode_flag associated with the current block into the video bitstream; The steps include storing the aforementioned video bitstream in a computer-readable medium, Methods that include...

3. The method according to claim 1 or 2, wherein the planar mode is always associated with the intrainter prediction mode.

4. Based on the final determination that the adjacent block is encoded by the intrainterprediction mode, the step of performing intraprediction coding of the current block using the planar mode associated with the intrainterprediction mode, The method according to claim 3, further comprising:

5. The method according to claim 3 or 4, further comprising the step of deriving the most likely mode (MPM) of the current block using the planar mode associated with the intrainter prediction mode, based on the final determination that the adjacent block is encoded by the intrainter prediction mode.

6. The step of deriving the aforementioned context value is: Based on the determination that the adjacent block is encoded by intra-prediction mode, the step of incrementing the context value by 2, Based on the determination that the adjacent block is encoded by interprediction mode, the step of incrementing the context value by 0, Based on the determination that the adjacent block is encoded by the intrainterpretation mode, the step of incrementing the context value by 1, The method further includes, The steps include determining the average context index based on the context value after increment and the number of adjacent blocks to the current block, The steps include setting the second prediction mode flag based on the determined average context index, The method according to any one of claims 1 to 5, further comprising:

7. The step of deriving the context value relating to the second prediction mode flag is: Based on the determination that the adjacent block is encoded by intra-prediction mode, the step of incrementing the context value by 1, Based on the determination that the adjacent block is encoded by interprediction mode, the step of incrementing the context value by 0, Based on the determination that the adjacent block is encoded by the intrainterpretation mode, the step of incrementing the context value by 0.5, The method further includes, The steps include determining the average context index based on the context value after increment and the number of adjacent blocks to the current block, The steps include setting the second prediction mode flag based on the determined average context index, The method according to any one of claims 1 to 5, further comprising:

8. At least one memory location that stores the program, At least one processor connected to the memory, Includes, The program is configured to cause the at least one processor to execute the method described in any one of claims 1 to 7. Encoder.

9. A program for causing a computer to perform the method described in any one of claims 1 to 7.

Images