Video encoding and decoding method, device and storage medium

Abstract

Various aspects of the present disclosure provide methods, devices and non-storage media for video coding and decoding. The device includes a processing circuit. The processing circuit can decode pictures associated with views of each layer in a coded video sequence CVS from a code stream. The processing circuit is configured to determine a one-dimensional position of the view to be displayed based on a first auxiliary enhancement information SEI message and a second SEI message. The first SEI message includes scalability dimension information SDI, and the second SEI message includes view position MVP information of multiple views. The processing circuit is configured to display the decoded picture based on the one-dimensional position of the view.

Description

[0001] Incorporation by Reference

[0002] This application claims priority to U.S. Patent Application No. 17 / 850,581, filed on June 27, 2022, for "View Position Assistance Enhancement Information Messages for Multi-Views," which claims priority to U.S. Provisional Application No. 63 / 215,944, filed on June 28, 2021, for "View Position SEI Message Techniques for Coding Multi-View Video Streams." The disclosure of that prior application is incorporated herein by reference. Technical Field

[0003] This disclosure describes embodiments that generally relate to video coding. Background Art

[0004] The background description provided herein is intended to present the background of the present application as a whole. The extent to which the work of the presently named inventors described in the background section and various aspects of this specification is performed does not indicate that it is prior art at the time of filing, and it is never explicitly or implicitly admitted that it is prior art to the present application.

[0005] Inter-picture prediction with motion compensation can be used for picture and / or video encoding and decoding. Uncompressed digital pictures and / or videos may include a series of pictures, each picture having a spatial dimension of, for example, 1920×1080 luma samples and associated chroma samples. The series of pictures has a fixed or variable picture rate (also informally referred to as a frame rate), for example, 60 pictures per second or 60Hz. Uncompressed pictures and / or videos have specific bit rate requirements. For example, 1920×1080p60 4:2:0 video (1920×1080 luma sample resolution at 60Hz frame rate) with 8 bits per sample requires a bandwidth of nearly 1.5Gbit / s. One hour of such video requires more than 600GB of storage space.

[0006] One purpose of picture and / or video encoding and decoding is to reduce the redundant information of input picture and / or video signal by compression. Video compression can help reduce the requirements for the above-mentioned bandwidth and / or storage space, in some cases by two or more orders of magnitude. Although the description here uses video encoding / decoding as an illustrative example, the same technology can be applied to image encoding / decoding in a similar manner without departing from the spirit of the present disclosure. Lossless compression and lossy compression, as well as a combination of the two, can be adopted. Lossless compression refers to the technology of reconstructing an exact copy of the original signal from the compressed original signal. When lossy compression is used, the reconstructed signal may be different from the original signal, but the distortion between the original signal and the reconstructed signal is small enough to make the reconstructed signal usable for the intended application. For video, lossy compression is widely used. The amount of tolerable distortion depends on the application; for example, users of certain consumer streaming applications can tolerate higher distortion than users of TV distribution applications. The compression ratio can reflect that higher allowable / tolerable distortion can produce higher compression ratios.

[0007] Video encoders and decoders may utilize techniques from several broad categories including, for example, motion compensation, transforms, quantization, and entropy coding.

[0008] Video codec techniques may include known intra-frame coding techniques. In intra-frame coding, sample values ​​are represented without reference to samples or other data of previously reconstructed reference pictures. In some video codecs, pictures are spatially subdivided into sample blocks. When all sample blocks are encoded in intra-frame mode, the picture can be an intra-frame picture. Intra-frame pictures and their derivatives (such as independent decoder refresh pictures) can be used to reset the decoder state, and can therefore be used as the first picture in the encoded video bitstream and video session, or as a still image. Then, the samples of the intra-frame block can be transformed, and the transform coefficients can be quantized before entropy coding. Intra-frame prediction can be a technique to minimize the sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after transformation and the smaller the AC coefficient, the fewer bits are needed to represent the block after entropy coding at a given quantization step size.

[0009] Conventional intra-frame coding, as known from coding techniques such as MPEG-2, does not use intra-frame prediction. However, some newer video compression techniques include techniques that attempt to do so based on, for example, surrounding sample data and / or metadata that is obtained during encoding and / or decoding of spatial neighbors and precedes the data block in decoding order. Such techniques are hereinafter referred to as "intra-frame prediction" techniques. Note that, in at least some cases, intra-frame prediction uses only reference data from the current picture being reconstructed, and not reference data from reference pictures.

[0010] There can be many different forms of intra-frame prediction. When more than one such technique is applicable in a given video coding technique, the techniques used can be coded or decoded in intra-frame prediction mode. In some cases, a mode can have sub-modes and / or parameters, and these can be encoded separately or contained in the mode codeword. For a given mode, sub-mode and / or parameter combination, which codeword is used can have an impact on the coding efficiency gain through intra-frame prediction, and the same is true for the entropy coding technique used to convert the codeword into a codestream.

[0011] Some mode of intra prediction was introduced with H.264, modified in H.265, and further modified in newer coding techniques such as Joint Detection Mode (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). The predictor block can be formed using values ​​of neighboring samples that are already available. For example, sample values ​​of neighboring samples can be copied into the predictor block according to one direction. The reference to the direction of use can be encoded in the bitstream, or it can be predicted itself.

[0012] refer to Figure 1A , depicted at the bottom right is a subset of the 9 known predictor directions out of the 33 possible predictor directions in H.265 (corresponding to the 33 angular modes for the 35 intra modes). The point (101) where the arrows converge represents the sample being predicted. The arrows represent the direction according to which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted according to one or more samples to the upper right at an angle of 45 degrees from horizontal. Similarly, arrow (103) indicates that sample (101) is predicted according to one or more samples to the lower left of sample (101) at an angle of 22.5 degrees from horizontal.

[0013] Still reference Figure 1A , a square block (104) consisting of 4×4 samples is shown in the upper left (indicated by the thick dashed line). The square block (104) consists of 16 samples, each of which is labeled with "S" and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (starting from the top) and the first sample in the X dimension (starting from the left). Similarly, sample S44 is the fourth sample in the block (104) in both the Y dimension and the X dimension. Since the block is a 4×4 size sample, S44 is located in the lower right corner. Reference samples following a similar numbering scheme are also shown. Reference samples are labeled with "R" and their Y position (e.g., row index) and X position (e.g., column index) relative to the block (104). In H.264 and H.265, the prediction samples are adjacent to the block being reconstructed; therefore, there is no need to use negative values.

[0014] Intra-picture prediction can be achieved by copying reference sample values ​​from neighboring samples assigned to the signaled prediction direction. For example, assume that the encoded video bitstream includes signaling that, for this block, indicates a prediction direction consistent with arrow (102) - that is, the sample is predicted based on one or more prediction samples to the upper right at a 45 degree angle to the horizontal direction. In such a case, samples S41, S32, S23, and S14 are predicted based on the same reference sample R05. Sample S44 is then predicted based on reference sample R08.

[0015] In some cases, such as by interpolation, the values ​​of multiple reference samples may be combined in order to compute the reference sample, particularly when the direction is not divisible by 45 degrees.

[0016] As video coding techniques have evolved, the number of possible directions has increased. In H.264 (2003), 9 different directions could be represented. This increased to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions at the time of this disclosure. Experiments have been conducted to identify the most likely directions, and certain techniques in entropy coding can be used to represent those possible directions with a small number of bits, accepting some cost for less likely directions. In addition, the direction itself can sometimes be predicted based on neighboring directions for decoded blocks.

[0017] Figure 1B A schematic diagram (110) depicting 65 intra prediction directions according to JEM is shown to illustrate the increase in the number of prediction directions over time.

[0018] The mapping of the bits representing the intra-frame prediction direction in the encoded video bitstream can vary with different video coding techniques; and can be, for example, a simple direct mapping from prediction direction to intra-frame prediction mode, to codewords, to complex adaptive schemes involving most likely modes, and similar techniques. However, in all cases, there may be some directions in the video content that are statistically less likely to occur than some other directions. Since the goal of video compression is to reduce redundancy, in a well-functioning video coding technique, those less likely directions will be represented by a larger number of bits than the more likely directions.

[0019] Motion compensation may be a lossy compression technique and may involve techniques where a block of sample data from a previously reconstructed picture or part thereof (reference picture) is spatially shifted in the direction indicated by a motion vector (hereinafter referred to as MV) and used to predict a newly reconstructed picture or part of a picture. In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or three dimensions, where the third is an indication of the reference picture in use (the latter, for simplicity, may be the temporal dimension).

[0020] In some video compression techniques, the MV applicable to a certain area of ​​sample data can be predicted from other MVs, such as those MVs that are predicted from another area of ​​sample data that is spatially adjacent to the area being reconstructed and that precede the MV in decoding order. Doing so can significantly reduce the amount of data required to encode the MV, thereby removing redundancy and increasing compression efficiency. MV prediction can be performed efficiently, for example, because when encoding an input video signal derived from a camera (called natural video), there is a statistical possibility that a larger area than the area applicable to a single MV moves in a similar direction. Therefore, in some cases, similar motion vectors derived from MVs of adjacent areas can be used for prediction. This results in the MV found for a given area being similar or identical to the MV predicted based on the surrounding MVs. After entropy coding, the MV can be represented by fewer bits than the number of bits used to directly encode the MV. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from the original signal (i.e., sample stream). In other cases, MV prediction itself may be lossy, for example due to rounding errors when calculating predicted values ​​from several surrounding MVs.

[0021] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Recommendation H.265, "High Efficiency Video Coding", December 2016). Among the various MV prediction mechanisms provided by H.265, a technique referred to as "spatial merging" is described here.

[0022] Referring to FIG. 2 , the current block ( 201 ) includes samples that have been discovered by the encoder during the motion search process, and the samples can be predicted based on the previous block of the same size that has generated a spatial offset. In addition, the MV can be derived from metadata associated with one or more reference pictures instead of encoding the MV directly. For example, the MV is derived from the metadata of the nearest reference picture (in decoding order) using the MV associated with any of the five surrounding samples A0, A1 and B0, B1, B2 (corresponding to 202 to 206, respectively). In H.265, MV prediction can use the prediction value of the same reference picture also used by the neighboring blocks. Summary of the invention

[0023] Various aspects of the present disclosure provide methods and devices for video encoding and decoding. In some examples, the device for video decoding includes a processing circuit. The processing circuit is configured to decode pictures associated with views of each layer in a coded video sequence (CVS) from a code stream. The processing circuit can determine the one-dimensional position of the view to be displayed based on a first auxiliary enhancement information (SEI) message and a second SEI message. The first SEI message includes scalability dimension information (SDI), and the second SEI message includes view position (MVP) information of multiple views. The processing circuit can display the decoded picture based on the one-dimensional position of the view.

[0024] In an embodiment, for each of the views of the respective layers of the CVS, the processing circuit may determine a view identifier of the corresponding view based on the first SEI message. The processing circuit may determine a one-dimensional position of the corresponding view to be displayed based on a view position variable in the second SEI message and the determined view identifier.

[0025] In an embodiment, the processing circuit derives a first value indicating the number of views from the first SEI message. The processing circuit may obtain a second value associated with the number of views from the second SEI message, the second value being different from the first value. The processing circuit may compare 1 plus the second value to the first value in a consistency check.

[0026] In an embodiment, the second SEI message is associated with an intra random access point (IRAP) access unit of the CVS.

[0027] In an embodiment, said position of said view applies to access units in said CVS.

[0028] In an embodiment, the determining and the displaying are performed based on the first SEI message being included in the CVS and the second SEI message being included in the CVS.

[0029] In an embodiment, the CVS includes a third SEI message, the third SEI message including 3D reference display information. The 3D reference display information indicates a view identifier of a left view, a view identifier of a right view, and a reference display width of a reference display. The processing circuit may derive a first value indicating the number of views from the first SEI message. The processing circuit may obtain a third value associated with the number of views from the third SEI message. The third value is different from the first value. The processing circuit may compare 1 plus the third value with the first value in a consistency check.

[0030] In an embodiment, said third SEI message indicates a number of reference displays signaled in said third SEI message.A maximum value of said number of reference displays is based on said third value associated with said number of views in said third SEI message.

[0031] In an embodiment, the first SEI message is an SDI SEI message and the second SEI message is an MVP SEI message.

[0032] Various aspects of the present disclosure also provide a non-transitory computer-readable storage medium storing a program executed by at least one processor to perform a video decoding method and a video encoding method. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Other features, properties, and various advantages of the disclosed subject matter will be more apparent from the following detailed description and accompanying drawings.

[0034] Figure 1A is a schematic diagram of an exemplary subset of intra prediction modes.

[0035] Figure 1B is a diagram of exemplary intra prediction directions.

[0036] FIG. 2 shows a current block and its surrounding samples according to an embodiment.

[0037] Figure 3 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment.

[0038] Figure 4 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment.

[0039] Figure 5 is a schematic diagram of a simplified block diagram of a decoder according to an embodiment.

[0040] Figure 6 is a schematic diagram of a simplified block diagram of an encoder according to an embodiment.

[0041] Figure 7 A block diagram of an encoder according to another example embodiment is shown.

[0042] Figure 8 A block diagram of a decoder according to another example embodiment is shown.

[0043] Fig. 9 A diagram showing an autostereoscopic display according to an embodiment of the present disclosure is shown.

[0044] FIG10A shows an example of reordering pictures based on view positions of multiple views according to an embodiment of the present disclosure.

[0045] Figures 10B to 10C An example of reordering pictures based on view positions of multiple views according to an embodiment of the present disclosure is shown.

[0046] Fig.11 An example of syntax in a Supplementary Enhancement Information (SEI) message for indicating view positions of a multi-view video according to an embodiment of the present disclosure is shown.

[0047] Fig.12 An example of syntax in a SEI message for indicating 3D reference display information according to an embodiment of the present disclosure is shown.

[0048] Fig.13 A flow chart outlining an encoding method according to an embodiment of the present disclosure is shown.

[0049] Fig.14 A flowchart outlining a decoding method according to an embodiment of the present disclosure is shown.

[0050] Fig.15 A schematic diagram of a computer system according to an embodiment is shown. DETAILED DESCRIPTION

[0051] Figure 3 3 is a simplified block diagram of a communication system (300) according to an embodiment disclosed in the present application. The communication system (300) includes a plurality of terminal devices, which can communicate with each other through, for example, a network (350). For example, the communication system (300) includes a first terminal device (310) and a second terminal device (320) interconnected through the network (350). Figure 3 In an embodiment of the present invention, the first terminal device (310) and the second terminal device (320) perform unidirectional data transmission. For example, the first terminal device (310) may encode video data (e.g., a video picture stream collected by the first terminal device (310)) for transmission to the second terminal device (320) via the network (350). The encoded video data is transmitted in the form of one or more encoded video code streams. The second terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to restore the video data, and display the video picture based on the restored video data. Unidirectional data transmission is more common in applications such as media services.

[0052] In another embodiment, the communication system (300) includes a third terminal device (330) and a fourth terminal device (340) for performing bidirectional transmission of encoded video data, which may occur, for example, during a video conference. For bidirectional data transmission, each of the third terminal device (330) and the fourth terminal device (340) may encode video data (e.g., a video picture stream collected by the terminal device) to transmit to the other terminal device of the third terminal device (330) and the fourth terminal device (340) through a network (350). Each of the third terminal device (330) and the fourth terminal device (340) may also receive the encoded video data transmitted by the other terminal device of the third terminal device (330) and the fourth terminal device (340), and may decode the encoded video data to restore the video data, and may display the video picture on an accessible display device based on the restored video data.

[0053] exist Figure 3 In the embodiment of the present invention, the first terminal device (310), the second terminal device (320), the third terminal device (330) and the fourth terminal device (340) can be interpreted as a server, a personal computer and a smart phone, but the principles disclosed in the present application are not limited thereto. The embodiments disclosed in the present application use a laptop computer, a tablet computer, a media player and / or a dedicated video conferencing device. The network (350) represents any number of networks that transmit encoded video data between the first terminal device (310), the second terminal device (320), the third terminal device (330) and the fourth terminal device (340), including, for example, a wired (wired) and / or wireless communication network. The communication network (350) can exchange data in a circuit switching and / or packet switching channel. The network may include a telecommunications network, a local area network, a wide area network and / or the Internet. For the purpose of the present application, unless otherwise explained below, the architecture and topology of the network (350) may be irrelevant to the operation disclosed in the present application.

[0054] As an example, Figure 4 The video encoder and video decoder are shown in a video streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, compressed video storage on digital media including CD, DVD, memory stick, etc.

[0055] The streaming system may include an acquisition subsystem (413), which may include a video source (401) such as a digital camera, creating, for example, an uncompressed video picture stream (402). In an embodiment, the video picture stream (402) includes samples acquired by the digital camera. Compared to the encoded video data (404) (or the encoded video bitstream), the uncompressed video picture stream (402) is depicted as a thick line to emphasize the high data volume of the video picture stream, and the video picture stream (402) can be processed by an electronic device (420), which includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. Compared to the video picture stream (402), the encoded video data (404) (or the encoded video bitstream (404)) is depicted as a thin line to emphasize the lower amount of data of the encoded video data (404) (or the encoded video bitstream), which can be stored on the streaming server (405) for future use. One or more streaming client subsystems, such as Figure 4 The client subsystem (406) and the client subsystem (408) in the video transmission system can access the streaming server (405) to retrieve the copy (407) and the copy (409) of the encoded video data (404). The client subsystem (406) may include, for example, a video decoder (410) in the electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and generates an output video picture stream (411) that can be presented on a display (412) (e.g., a display screen) or another presentation device (not depicted). In some streaming transmission systems, the encoded video data (404), the video data (407), and the video data (409) (e.g., a video bitstream) can be encoded according to certain video encoding / compression standards. Examples of these standards include ITU-T H.265. In an embodiment, the video coding standard under development is informally referred to as Versatile Video Coding (VVC), and the present application can be used in the context of the VVC standard.

[0056] It should be noted that the electronic device (420) and the electronic device (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0057] Figure 5is a block diagram of a video decoder (510) according to an embodiment disclosed in the present application. The video decoder (510) may be provided in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used to replace Figure 4 A video decoder (410) of an embodiment.

[0058] The receiver (531) may receive one or more encoded video sequences to be decoded by the video decoder (510); in the same or another embodiment, one encoded video sequence at a time, wherein the decoding of each encoded video sequence is independent of the other encoded video sequences. The encoded video sequence may be received from a channel (501), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (531) may receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams that may be forwarded to their respective entities (not shown) for use. The receiver (531) may separate the encoded video sequence from the other data. To prevent network jitter, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) may be part of the video decoder (510). In other applications, the buffer memory (515) may be provided external to the video decoder (510) (not shown). In other applications, a buffer memory (not shown) is provided outside the video decoder (510) to, for example, prevent network jitter, and another buffer memory (515) may be provided inside the video decoder (510) to, for example, handle broadcast timing. When the receiver (531) receives data from a storage / forward device with sufficient bandwidth and controllability or from an isochronous network, it may not be necessary to configure the buffer memory (515), or the buffer memory may be made smaller. Of course, for use on a service packet network such as the Internet, a buffer memory (515) may be required, and the buffer memory may be relatively large. Such a buffer memory may advantageously have an adaptive size and may be at least partially implemented in an operating system or a similar element (not shown) external to the video decoder (510).

[0059] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. The types of symbols include information used to manage the operation of the video decoder (510) and potential information used to control a display device such as a display device (512) (e.g., a display screen) that is not part of the electronic device (530) but can be coupled to the electronic device (530), such as Figure 5As shown in . The control information for the display device may be a parameter set fragment (not indicated) of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI). The parser (520) may parse / entropy decode the received coded video sequence. The encoding of the coded video sequence may be performed according to a video coding technique or standard, and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and the like. The parser (520) may extract a subgroup parameter set of at least one subgroup of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), and the like. The parser (520) may also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[0060] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515), thereby creating symbols (521).

[0061] Depending on the type of coded video picture or portion of coded video picture (e.g., inter- and intra-pictures, inter- and intra-blocks) and other factors, the reconstruction of the symbol (521) may involve multiple different units. Which units are involved and how they are involved may be controlled by subgroup control information parsed by the parser (520) from the coded video sequence. For clarity, such subgroup control information flow between the parser (520) and the multiple units below is not described.

[0062] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units as described below. In a practical embodiment operating under commercial constraints, many of these units interact closely with each other and can be integrated with each other. However, for the purpose of describing the disclosed subject matter, it is appropriate to subdivide the concept of functional units below.

[0063] The first unit may be a sealer / inverse transform unit (551). The sealer / inverse transform unit (551) receives quantized transform coefficients as symbols (521) and control information from the parser (520), including information such as which type of inverse transform to use, block size, quantization factor, quantization scaling matrix, etc. The sealer / inverse transform unit (551) may output a block including sample values, which may be input into an aggregator (555).

[0064] In some cases, the output samples of the scaler / inverse transform unit (551) may belong to an intra-coded block; for example, a block that does not use predictive information from a previously reconstructed picture, but may use predictive information from a previously reconstructed portion of the current picture. Such predictive information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) generates surrounding blocks of the same size and shape as the block being reconstructed using surrounding reconstructed information retrieved from a current picture buffer (558). For example, the current picture buffer (558) buffers a partially reconstructed current picture and / or a fully reconstructed current picture. In some implementations, the aggregator (555) adds the prediction information generated by the intra-picture prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a per-sample basis.

[0065] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to an inter-frame coded and potentially motion compensated block. In this case, the motion compensated prediction unit (553) may access the reference picture memory (557) to extract samples for prediction. After the extracted samples are motion compensated according to the symbols (521), these samples may be added to the output of the scaler / inverse transform unit (551) (in this case referred to as residual samples or residual signals) by the aggregator (555) to generate output sample information. The acquisition of the predicted samples by the motion compensated prediction unit (553) from the address in the reference picture memory (557) may be controlled by a motion vector, and the motion vector is provided to the motion compensated prediction unit (553) in the form of the symbols (521), for example, including X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​extracted from the reference picture memory (557) when using sub-sample accurate motion vectors, motion compensation may include motion vector prediction mechanisms, etc.

[0066] The output samples of the aggregator (555) may be used by various loop filtering techniques in a loop filter unit (554). The video compression techniques may include in-loop filter techniques that are controlled by parameters included in the encoded video sequence (also referred to as the encoded video bitstream) and that are available to the loop filter unit (556) as symbols (521) from the parser (520). However, in other embodiments, the video compression techniques may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of an encoded picture or encoded video sequence, as well as to previously reconstructed and loop filtered sample values.

[0067] The output of the loop filter unit (556) may be a sample stream that may be output to a display device (512) and stored in a reference picture memory (557) for subsequent inter-picture prediction.

[0068] Once fully reconstructed, certain coded pictures may be used as reference pictures for future prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed, and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) may become part of the reference picture memory (557), and a new current picture buffer may be reallocated before starting to reconstruct a subsequent coded picture.

[0069] The video decoder (510) may perform decoding operations according to a predetermined video compression technique, such as in the ITU-T H.265 standard. The encoded video sequence may conform to the syntax specified by the video compression technique or standard used in the sense that the encoded video sequence follows the syntax of the video compression technique or standard and the profile recorded in the video compression technique or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technique or standard as the only tools available for use under the profile. For compliance, it is also necessary that the complexity of the encoded video sequence is within the range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum picture size, the maximum frame rate, the maximum reconstruction sampling rate (measured in, for example, mega samples per second), the maximum reference picture size, etc. In some cases, the limits set by the hierarchy may be further limited by the Hypothetical Reference Decoder (HRD) specification and metadata of the HRD buffer management signaled in the encoded video sequence.

[0070] In one embodiment, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be part of the encoded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0071] Figure 6 6 is a block diagram of a video encoder (603) according to an embodiment disclosed in the present application. The video encoder (603) is provided in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used to replace Figure 4 A video encoder (403) in an embodiment.

[0072] The video encoder (603) can be used to obtain the video source (601) (not Figure 6 In another embodiment, the video source (601) may be a part of the electronic device (620) to receive the video samples, and the video source may capture the video images to be encoded by the video encoder (603). In another embodiment, the video source (601) may be a part of the electronic device (620).

[0073] The video source (601) may provide a source video sequence in the form of a digital video sample stream to be encoded by the video encoder (603), wherein the digital video sample stream may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source (601) may be a storage device capable of storing previously prepared videos. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of separate pictures that are given motion when viewed sequentially. The pictures themselves may be constructed as a spatial pixel array, wherein each pixel may include one or more samples depending on the sampling structure, color space, etc. being used. The relationship between pixels and samples may be readily understood by those skilled in the art. The following description focuses on samples.

[0074] According to an embodiment, the video encoder (603) may encode and compress pictures of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed is a function of the controller (650). In some embodiments, the controller (650) controls other functional units as described below and is functionally coupled to these units. For clarity, coupling is not indicated in the figure. The parameters set by the controller (650) may include rate control related parameters (picture skipping, quantizer, lambda value of rate distortion optimization technology, etc.), picture size, group of pictures (GOP) layout, maximum allowed motion vector search range, etc. The controller (650) can be used to have other suitable functions that are related to the video encoder (503) optimized for a certain system design.

[0075] In some embodiments, the video encoder (603) operates in a coding loop. As a simple description, in embodiments, the coding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be encoded and a reference picture) and a (local) decoder (633) embedded in the video encoder (603). In addition, the decoder (633) reconstructs the symbols to create sample data in a manner similar to that of the (remote) decoder to create sample data (because in the video compression technology considered in this application, any compression between the symbols and the encoded video code stream is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream produces bit-accurate results that are independent of the decoder location (local or remote), the contents of the reference picture memory (634) are also bit-accurate between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction part of the encoder are exactly the same as the sample values ​​that the decoder will "see" when using the prediction during decoding. This basic principle of reference picture synchronization (and the drift that occurs when synchronization cannot be maintained, eg due to channel errors) is also used in some related techniques.

[0076] The operation of the "local" decoder (633) can be combined with the above Figure 5 The "remote" decoder described in detail for the video decoder (510) is identical. However, additional brief reference is made to Figure 5 , when symbols are available and the entropy encoder (645) and parser (520) are capable of losslessly encoding / decoding the symbols into an encoded video sequence, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and the parser (520), may not be fully implemented in the encoder's local decoder (633).

[0077] In one embodiment, any decoder technology other than the parsing / entropy decoding present in the decoder is also present in the corresponding encoder in the same or substantially the same functional form. Therefore, the subject matter of the present application focuses on the decoder operation. The description of the encoder technology can be simplified because the encoder technology is mutually inverse to the decoder technology described comprehensively. A more detailed description is given below in some areas.

[0078] During operation, in some embodiments, the source encoder (630) may perform motion compensated predictive coding. The motion compensated predictive coding predictively encodes an input picture with reference to one or more previously encoded pictures from a video sequence designated as "reference pictures." In this manner, the encoding engine (632) encodes the differences between pixel blocks of the input picture and pixel blocks of a reference picture that may be selected as a prediction reference for the input picture.

[0079] The local video decoder (633) may decode the encoded video data of the picture that may be designated as the reference picture based on the symbol created by the source encoder (630). The operation of the encoding engine (632) may be a lossy process. When the encoded video data is available at the video decoder ( Figure 6 When the video sequence is decoded at a remote location (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process that may be performed by the video decoder on the reference picture and may cause the reconstructed reference picture to be stored in the reference picture cache (634). In this way, the video encoder (603) may store a copy of the reconstructed reference picture locally that has common content (absent transmission errors) with the reconstructed reference picture to be obtained by the remote video decoder.

[0080] The predictor (635) may perform a prediction search for the encoding engine (632). That is, for a new picture to be encoded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that may serve as appropriate prediction references for the new picture. The predictor (635) may operate pixel-by-pixel based on sample blocks to find a suitable prediction reference. In some cases, based on the search results obtained by the predictor (635), it may be determined that the input picture may have prediction references taken from a plurality of reference pictures stored in the reference picture memory (634).

[0081] The controller (650) may manage encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.

[0082] The outputs of all the above functional units may be entropy encoded in an entropy encoder (645). The entropy encoder (645) performs lossless compression on the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc., thereby converting the symbols into a coded video sequence.

[0083] The transmitter (640) may buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission over a communication channel (660), which may be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (640) may combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or ancillary data streams (source not shown).

[0084] The controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) may assign a certain coded picture type to each coded picture, but this may affect the coding techniques that can be applied to the corresponding picture. For example, a picture may generally be assigned to any of the following picture types:

[0085] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of the variations of I pictures and their corresponding applications and features.

[0086] A predictive picture (P picture) may be a picture that can be encoded and decoded using intra prediction or inter prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0087] Bidirectional predictive pictures (B pictures), which can be pictures that can be encoded and decoded using intra prediction or inter prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for reconstruction of a single block.

[0088] The source picture may typically be spatially subdivided into blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples), and coded block by block. These blocks may be predictively coded with reference to other (already coded) blocks, which are determined according to the coding allocation applied to the block's corresponding picture. For example, blocks of an I picture may be non-predictively coded, or the blocks may be predictively coded (spatial prediction or intra prediction) with reference to already coded blocks of the same picture. Blocks of pixels of a P picture may be predictively coded by spatial prediction with reference to one previously coded reference picture or by temporal prediction. Blocks of a B picture may be predictively coded by spatial prediction with reference to one or two previously coded reference pictures or by temporal prediction.

[0089] The video encoder (603) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (603) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in an input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.

[0090] In an embodiment, the transmitter (640) may transmit additional data when transmitting the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include other forms of redundant data such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0091] The captured video may be taken as a plurality of source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often simplified to intra-prediction) exploits spatial correlations in a given picture, while inter-picture prediction exploits (temporal or other) correlations between pictures. In an embodiment, a particular picture being encoded / decoded is divided into blocks, and the particular picture being encoded / decoded is referred to as the current picture. When a block in the current picture is similar to a reference block in a reference picture that was previously encoded in the video and is still buffered, the block in the current picture may be encoded by a vector called a motion vector. The motion vector points to a reference block in a reference picture, and in the case where multiple reference pictures are used, the motion vector may have a third dimension that identifies the reference picture.

[0092] In some embodiments, bidirectional prediction techniques may be used for inter-picture prediction. According to the bidirectional prediction technique, two reference pictures are used, for example, a first reference picture and a second reference picture that are both before the current picture in the video in decoding order (but may be in the past and future in display order, respectively). A block in the current picture may be encoded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. Specifically, the block may be jointly predicted by a combination of the first reference block and the second reference block.

[0093] In addition, merge mode technology can be used in inter-picture prediction to improve coding efficiency.

[0094] According to some embodiments disclosed in the present application, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, according to the HEVC standard, pictures in a video picture sequence are divided into coding tree units (CTUs) for compression, and the CTUs in the pictures have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU includes three coding tree blocks (CTBs): a luminance CTB and two chrominance CTBs. Furthermore, each CTU can be split into one or more coding units (CUs) using a quadtree. For example, a 64×64 pixel CTU can be split into a 64×64 pixel CU, or 4 32×32 pixel CUs, or 16 16×16 pixel CUs. In one embodiment, each CU can be analyzed to determine the prediction type for the CU, and various prediction types are, for example, inter-frame prediction types or intra-frame prediction types. In addition, depending on temporal and / or spatial predictability, the CU is split into one or more prediction units (PUs). Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In an embodiment, the prediction operation in encoding (encoding / decoding) is performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a value matrix (e.g., luma values) of pixels (e.g., 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.).

[0095] Figure 7 is a diagram of a video encoder (703) according to another embodiment disclosed in the present application. The video encoder (703) is used to receive a processed block (e.g., a prediction block) of sample values ​​in a current video picture in a video picture sequence, and encode the processed block into an encoded picture that is part of an encoded video sequence. In this embodiment, the video encoder (703) is used to replace Figure 4A video encoder (303) in an embodiment.

[0096] In one HEVC example, a video encoder (703) receives a matrix of sample values ​​for a processing block, such as a prediction block of 8×8 samples. The video encoder (703) uses, for example, rate-distortion optimization to determine whether to use intra mode, inter mode, or bidirectional prediction mode to encode the processing block. When encoding the processing block in intra mode, the video encoder (703) may use intra prediction techniques to encode the processing block into an encoded picture; and when determining to encode the processing block in inter mode or bidirectional prediction mode, the video encoder (703) may encode the processing block into an encoded picture using inter prediction or bidirectional prediction techniques, respectively. In some video coding techniques, merge mode may be a sub-mode of inter-picture prediction, in which a motion vector is derived from one or more motion vector predictors without the aid of an encoded motion vector component external to the predictor. In certain other video coding techniques, there may be a motion vector component applicable to the subject block. In one example, the video encoder (703) includes other components, such as a mode decision module (not shown) for determining a mode for processing a block.

[0097] exist Figure 7 In an embodiment of the present invention, the video encoder (703) includes: Figure 7 Shown are an inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together.

[0098] The inter-frame encoder (730) is used to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference picture (e.g., blocks in a previous picture and a subsequent picture), generate inter-frame prediction information (e.g., redundant information description according to an inter-frame coding technique, motion vectors, merge mode information), and calculate an inter-frame prediction result (e.g., a prediction block) based on the inter-frame prediction information using any suitable technique. In some embodiments, the reference picture is a decoded reference picture decoded based on the encoded video information.

[0099] The intra-frame encoder (722) is used to receive samples of a current block (e.g., a processing block), compare the block with an encoded block in the same picture in some cases, generate quantization coefficients after transformation, and in some cases also generate intra-frame prediction information (e.g., intra-frame prediction direction information according to one or more intra-frame coding techniques). The intra-frame encoder (722) calculates an intra-frame prediction result (e.g., a prediction block) based on the intra-frame prediction information and a reference block in the same picture.

[0100] The general controller (721) is used to determine the general control data and control other components of the video encoder (703) based on the general control data. In an embodiment, the general controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, when the mode is intra mode, the general controller (721) controls the switch (726) to select the intra mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra prediction information and add the intra prediction information to the bitstream; and when the mode is inter mode, the general controller (721) controls the switch (726) to select the inter prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select the inter prediction information and add the inter prediction information to the bitstream.

[0101] The residual calculator (723) is used to calculate the difference (residual data) between the received block and the prediction result selected from the intra-frame encoder (722) or the inter-frame encoder (730). The residual encoder (724) is used to encode the residual data to generate a transform coefficient. In an embodiment, the residual encoder (724) is configured to operate based on the residual data to convert the residual data from the time domain to the frequency domain to generate a transform coefficient. The transform coefficient is then quantized to obtain a quantized transform coefficient. In various exemplary embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is used to perform an inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra-frame encoder (722) and the inter-frame encoder (730). For example, the inter-frame encoder (730) can generate a decoded block based on the decoded residual data and the inter-frame prediction information, and the intra-frame encoder (722) can generate a decoded block based on the decoded residual data and the intra-frame prediction information. The decoded blocks are processed appropriately to generate decoded pictures, and the decoded pictures may be buffered in memory circuits (not shown) and used as reference pictures in some embodiments.

[0102] The entropy encoder (725) is used to format the code stream to produce an encoded block. According to a suitable standard, such as the HEVC standard, the entropy encoder (725) is configured to include various information. In an embodiment, the entropy encoder (725) is used to obtain general control data, selected prediction information (such as intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the code stream. It should be noted that according to the subject matter of the present disclosure, when the block is encoded in the merge sub-mode of the inter-frame mode or the bidirectional prediction mode, there is no residual information.

[0103] Figure 8FIG. 8 is a diagram of a video decoder (810) according to another embodiment disclosed in the present application. The video decoder (810) is used to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed picture. In an embodiment, the video decoder (810) is used to replace Figure 4 A video decoder (410) of an embodiment.

[0104] exist Figure 8 In an embodiment, the video decoder (810) includes Figure 8 Shown are an entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-frame decoder (872) coupled together.

[0105] The entropy decoder (871) may be used to reconstruct certain symbols from the encoded picture, which represent syntax elements constituting the encoded picture. Such symbols may include, for example, a mode for encoding the block (e.g., intra mode, inter mode, bidirectional prediction mode, the latter two in a merge submode or another submode), prediction information (e.g., intra prediction information or inter prediction information) that may identify certain samples or metadata for prediction by the intra decoder (872) or the inter decoder (880), respectively, residual information in the form of, for example, quantized transform coefficients, and the like. In an embodiment, when the prediction mode is an inter or bidirectional prediction mode, the inter prediction information is provided to the inter decoder (880); and when the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information may be inverse quantized and provided to the residual decoder (873).

[0106] The inter-frame decoder (880) is used to receive the inter-frame prediction information and generate an inter-frame prediction result based on the inter-frame prediction information.

[0107] The intra-frame decoder (872) is used to receive intra-frame prediction information and generate a prediction result based on the intra-frame prediction information.

[0108] The residual decoder (873) is used to perform inverse quantization to extract dequantized transform coefficients, and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require some control information (to obtain the quantizer parameter QP), which can be provided by the entropy decoder (871) (the data path is not indicated because this is only low-data-volume control information).

[0109] The reconstruction module (874) is used to combine the residual output by the residual decoder (873) with the prediction result (which can be output by the inter-frame prediction module or the intra-frame prediction module) in the spatial domain to form a reconstructed block, and the reconstructed block forms a part of the reconstructed picture, and the reconstructed picture can be used as a part of the reconstructed video. It should be noted that other suitable operations such as deblocking operations can be performed to improve visual quality.

[0110] It should be noted that the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810) may be implemented using any suitable technology. In an embodiment, the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810) may be implemented using one or more processors that execute software instructions.

[0111] According to an embodiment of the present disclosure, a code stream may include one or more coded video sequences (CVS). A CVS may be encoded independently of other CVSs. Each CVS may include one or more layers, and each layer may be a representation of a video with a specific quality (e.g., spatial resolution), or a representation of a component interpretation characteristic, such as a depth map, a transparency map, or a perspective map. In the time dimension, each CVS may include one or more access units (AUs). Each AU may include one or more pictures corresponding to different layers of the same time instance. A coded layer video sequence (CLVS) is a layered CVS that may include a sequence of picture units in the same layer. If the code stream has multiple layers, the CVS in the code stream may have one or more CLVS for each layer.

[0112] In an embodiment, the CVS includes a sequence of AUs, wherein the sequence of AUs includes, in decoding order, an intra random access point (IRAP) AU followed by zero or more AUs that are non-IRAP AUs. In an example, the zero or more AUs include all subsequent AUs, up to (but not including) any subsequent AU that is an IRAP AU. In an example, the CLVS includes a sequence of pictures and associated non-video codec layer (VCL) network abstraction layer (NAL) units of a base layer of the CVS.

[0113] Aspects of the present disclosure include view position assistance enhancement information (SEI) messages for multiple views of an encoded video stream.

[0114] According to some aspects of the present disclosure, videos can be classified into single-view videos and multi-view videos. For example, single-view videos (e.g., monoscopic videos) are two-dimensional media that provide a single view of a scene to a viewer. Multi-view videos can provide multiple viewpoints of a scene and can provide a sense of realism to the viewer. In an example, a 3D video can provide two views, such as a left view and a right view corresponding to a human viewer. The two views can be displayed (presented) simultaneously or nearly simultaneously using different light polarizations, and the viewer can wear polarized glasses so that each of the viewer's eyes receives a corresponding one of these views.

[0115] In another example, a display device such as an autostereoscopic display device is configured to generate different pictures depending on the position of the viewer's eyes and does not require glasses for viewing. The display device may be referred to as a glass-free 3D display.

[0116] Multi-view video can be created by capturing a scene simultaneously using multiple cameras, where the multiple cameras are appropriately positioned so that each camera captures the scene from a viewpoint. Multiple cameras can capture multiple video sequences corresponding to multiple viewpoints. In order to provide more views, more cameras can be used to generate a multi-view video with a large number of video sequences associated with the views. Multi-view video may require a large storage space to store and / or require high bandwidth to transmit. Multi-view video coding and decoding techniques have been developed in the art to reduce the required storage space or transmission bandwidth.

[0117] In order to improve the efficiency of multi-view video encoding and decoding, the similarity between views is utilized. In some embodiments, one of the views, called the base view, is encoded as a single-view video. For example, during the encoding of the base view, intra-frame (intra-picture) and / or temporal inter-frame (inter-picture) prediction is used. The base view can be decoded using a single-view decoder that performs intra-frame (intra-picture) prediction and inter-frame (inter-picture) prediction. Other views in a multi-view video other than the base view can be called dependent views. In order to encode the dependent view, in addition to intra-frame (intra-picture) and inter-frame (inter-picture) prediction, inter-view prediction with disparity compensation can also be used. In the example, in inter-view prediction, a reference block of a sample of a picture of another view of the same time instance is used to predict the current block in the dependent view. The position of the reference block is indicated by a disparity vector. Inter-view prediction is similar to inter-frame (inter-picture) prediction, but the motion vector is replaced by a disparity vector, and the temporal reference picture is replaced by a reference picture from another view.

[0118] According to some aspects of the present disclosure, multi-view coding can adopt a multi-layer approach. The multi-layer approach can multiplex different encoded (e.g., HEVC encoded) representations (called layers) of video sequences into a code stream. These layers can depend on each other. By exploiting the similarities between different layers, inter-layer prediction can use dependencies to achieve improved compression performance. A layer can represent the texture, depth, or other auxiliary information of a scene related to a specific camera view. In some examples, all layers belonging to the same camera perspective are represented as views; and layers carrying the same type of information (e.g., texture or depth) are referred to as components in the scope of multi-view video.

[0119] According to one aspect of the present disclosure, multi-view video codec may include the addition of a high-level syntax (HLS) to an existing single-layer decoding core (e.g., above the slice level). In some examples, the view codec of multi-view does not change the syntax or decoding process required for the single-layer codec (e.g., HEVC) below the slice level. It may allow existing implementations to be reused without making significant changes to establishing a multi-view video decoder. For example, a multi-view video decoder may be implemented based on a video decoder (510) or a video decoder (810).

[0120] In some examples, all pictures associated with the same capture or display time instance are contained in the AU and have the same picture order count (POC). Multi-view video codecs can allow inter-view prediction, which performs prediction from pictures in the same AU. For example, decoded pictures from other views can be inserted into one or both of the reference picture lists of the current picture. Further, in some examples, the motion vector can be an actual temporal motion vector when associated with a temporal reference picture of the same view, or a disparity vector when associated with an inter-view reference picture. Block-level motion compensation modules (e.g., block-level encoding software or hardware, block-level decoding software or hardware) can be used, which operate in the same manner regardless of whether the motion vector is a temporal motion vector or a disparity vector.

[0121] According to one aspect of the present disclosure, multi-view video coding can encode pictures of different views in a display time instance in an order (eg, decoding order) that is not necessarily associated with a corresponding display position order or a display order.

[0122] Fig. 9 An illustration of an autostereoscopic display (900) in some examples is shown. The autostereoscopic display (900) can display pictures of different views at different positions. In an example, the autostereoscopic display (900) displays pictures of different views in response to detected eye positions of an observer. Fig. 9In the example of , the observer's eye position can be detected in one dimension, such as between the leftmost position and the rightmost position. For example, when the observer's eye position is at E0, the autostereoscopic display (900) displays a picture of a view with a view identifier ViewId[0]; when the observer's eye position is at E1, the autostereoscopic display (900) displays a picture of a view with a view identifier ViewId[1]; when the observer's eye position is at E2, the autostereoscopic display (900) displays a picture of a view with a view identifier ViewId[2]; and when the observer's eye position is at E3, the autostereoscopic display (900) displays a picture of a view with a view identifier ViewId[3]. The order of the observer's eye positions from left to right is E2, E0, E1, and E3.

[0123] Fig. 10A An example of reordering pictures based on the view position of multiple views according to an embodiment of the present disclosure is shown. When multiple views or different views are presented, for the expected user experience, view position information indicating the display order in one dimension (e.g., display from left to right along the horizontal axis) is signaled. The display order may be indicated by the view positions of multiple views or multiple views of the corresponding pictures. In the example, the multiple views include a first view to be displayed at a view position of a first multi-view and a second view to be displayed at a view position of a second multi-view. A first picture of the first view may be displayed at the view position of the first multi-view. A second picture of the second view may be displayed at the view position of the second multi-view.

[0124] In some examples, pictures P0 to P3 correspond to multiple views (e.g., ViewId[0] to ViewId[3]), respectively. Pictures P0 to P3 are associated with the same capture time instance or the same display time instance. Pictures P0 to P3 can be encoded and decoded (encoded or decoded) in a codec order (e.g., encoding order or decoding order), which can be determined based on certain codec requirements such as rate-distortion optimization. The decoding order can be indicated based on pictures P0 to P3 or based on corresponding views ViewId[0] to ViewId[3]. Reference Fig. 10A , an example of the decoding order is P0 of view ViewId[0], P1 of view ViewId[1], P2 of view ViewId[2], and P3 of view ViewId[3], where P0 of view ViewId[0] is decoded first, and then P1 to P3 of ViewId[1] to ViewId[3] are decoded respectively. Alternatively, the decoding order is ViewId[0], ViewId[1], ViewId[2], and ViewId[3].

[0125] In an embodiment, the decoding order is different from the display order of pictures (e.g., P0 to P3) or corresponding views (e.g., ViewId[0] to ViewId[3]). According to an embodiment of the present disclosure, decoded pictures (e.g., P0 to P3) or corresponding views (e.g., ViewId[0] to ViewId[3]) may be reordered according to the display order. For the expected user experience, the display order may be signaled. Reference Fig. 10A , the display order is ViewId[2], ViewId[0], ViewId[1], and ViewId[3], where view ViewId[2] is displayed as the leftmost view and view ViewId[3] is displayed as the rightmost view. The corresponding display order based on pictures is P2, P0, P1, and P3. Reorder the views (and decoded pictures) according to the display order.

[0126] In some examples, an AU includes encoded information for pictures of different views associated with the same capture or display time instance. For example, an AU includes encoded information for picture P0 of view ViewId[0], encoded information for picture P1 of view ViewId[1], encoded information for picture P2 of view ViewId[2], and encoded information for picture P3 of view ViewId[3]. The encoding and decoding order of pictures P0 to P3 does not necessarily follow the order of the viewer's eye position.

[0127] FIG. 10B to FIG. 10C An example of reordering pictures according to view positions of multiple views in the example is shown. Fig. 10B 1 shows the pictures in the AU decoded according to the decoding order. Fig. 10B In the example of , pictures P0 to P3 in the AU are decoded in the order of P0, P1, P2, and P3. Because the order of the observer's eye position from left to right is E2, E0, E1, and E3. The decoded picture order does not correspond to the order of the observer's eye position from left to right.

[0128] In some examples, the decoded pictures may be reordered according to a display order that is signaled, for example, for an expected user experience. In an example, the display order is associated with the order of the viewer's eye positions, such as from left to right.

[0129] Fig. 10C Decoded pictures in an AU in some examples are shown reordered according to the display order. For example, the display order is associated with the order of the viewer's eye positions from left to right.

[0130] According to some aspects of the present disclosure, auxiliary enhancement information (SEI) messages may be included in the encoded code stream, for example, to assist in decoding and / or display of the encoded code stream, or for another purpose. One or more SEI messages may include information that is not necessary for decoding, such as decoded samples of encoded pictures from VCL NAL units. One or more SEI messages are optional for constructing luma or chroma samples through the decoding process. In some examples, SEI messages are not required for reconstructing luma or chroma samples during the decoding process. In addition, decoders that conform to video codec standards that support SEI messages are not required to process SEI messages to be conformed. For some codec standards, some SEI message information may be required to check code stream consistency or for output timing decoder consistency. Optionally, one or more SEI messages may be processed by a conforming decoder to make the output order conform to a certain standard (e.g., HEVC 265 or VVC). In an embodiment, one or more SEI messages are present in the code stream.

[0131] SEI messages can contain various types of data that indicate the timing of video pictures or describe various characteristics of the encoded video or describe how to use or enhance various characteristics. In some examples, SEI messages do not affect the core decoding process, but can indicate suggestions for how to post-process or display the video.

[0132] SEI messages can be used to provide additional information about the coded stream, which can be used to change the presentation of the stream once it is decoded, or to provide information to the decoder. For example, SEI messages have been used to provide frame packing information (e.g., describing the way video data is arranged in video frames), content description (e.g., to indicate that the coded stream is, for example, 360-degree video), and color information (e.g., color saturation and / or color range), etc.

[0133] In some examples, the SEI message can be used to signal a decoder that the encoded bitstream includes 360-degree video. The decoder can use the above information to render video data for 360-degree presentation. Alternatively, if the decoder cannot render 360-degree video, the decoder can use the above information to not render the video data.

[0134] According to an embodiment of the present disclosure, a first SEI message may indicate scalability dimension information (SDI) of a multi-view video. For example, the first SEI message (e.g., an SDI SEI message) may include the number and type of scalability dimensions, such as information indicating the number of views of the multi-view video. According to an embodiment of the present disclosure, a view identifier (ID) (e.g., expressed as ViewID) of the i-th layer in the current CVS may be signaled in the first SEI message (e.g., an SDI SEI message). In an example, the syntax element sdi_view_id_val[i] specifies the value of the view ID (e.g., ViewID) of the i-th layer in the current CVS. The length of the syntax element sdi_view_id_val[i] (e.g., the number of bits used to represent the syntax element sdi_view_id_val[i]) may be indicated by the syntax element sdi_view_id_len_minus1. The unit of the length may be bits. In an example, the value of the length of the syntax element sdi_view_id_val[i] is equal to the value of the syntax element sdi_view_id_len_minus1 plus 1.

[0135] According to one aspect of the present disclosure, a second SEI message (such as an SEI message indicating multiview view position (MVP) information) may be used to signal the display order. In some related examples, the second SEI message may include information indicating a view position in one dimension (e.g., along a horizontal axis). For example, the second SEI message may include a one-dimensional view position corresponding to an observer's eye position. The second SEI message (e.g., an MVP SEI message) may be used to indicate a relative view position along a horizontal axis of a 3D rendered view, such as a glassless 3D display.

[0136] Fig.11 An example of syntax in a second SEI message (e.g., an MVP SEI message) for indicating view positions of a multi-view video is shown (1100). For example, the MVP SEI message specifies the relative view positions of views within a CVS along the horizontal axis of a 3D rendered view. When there are multiple views, for the expected user experience, MVP information in the MVP SEI message may be signaled, indicating a display order from left to right.

[0137] The MVP SEI message may be associated with the IRAP AU of the CVS. The MVP information signaled in the MVP SEI message may apply to the entire CVS. In some examples, the MVP SEI message may signal the number of views and then signal the corresponding positions of each view separately.

[0138] exist Fig.11 In the example of , a parameter denoted by num_views_minus1 may be signaled by the MVP SEI message, as shown in (1110). The parameter num_views_minus1 indicates, for example, the number of views in an access unit (e.g., a parameter denoted as NumViews). For example, the number of views is equal to the sum of 1 and the value of the parameter num_views_minus1.

[0139] The parameter view_positions[i] may be used to signal the order of views (eg, display order) in one dimension in the MVP SEI message, such as the position of the view in the display order. The value i is an integer from 0 to num_views_minus1, such as Fig.11 As shown in (1120) in .

[0140] For display purposes, the parameter view_position[i] may indicate the order (e.g., display order) of a view whose view identifier (ID) ViewId is equal to the value of the syntax element sdi_view_id_val[i] in the first SEI message among views (e.g., all views) from left to right. The value of the order of the leftmost view may be equal to 0, and for the next view from left to right, the value of the order may increase by 1. The value of the parameter view_position[i] may be in the range of 0 to 62, inclusive.

[0141] In the example, to signal Fig. 10AFor the view positions in the example in FIG. 1 , “3” may be signaled as the parameter num_views_minus1 to indicate that the number of views is 4. “1” is signaled as view_position[0] when ViewId is 0, “2” is signaled as view_position[1] when ViewId is 1, “0” is signaled as view_position[2] when ViewId is 2, and “3” is signaled as view_position[3] when ViewId is 3, where “0” is the leftmost view in the position from left to right, and “3” is the rightmost view. Then, when views ViewId[0] to ViewId[3] are decoded from an access unit, view ViewId[0] has view_position[0], view ViewId[1] has view_position[1], view ViewId[2] has view_position[2], and view ViewId[3] has view_position[3]. Views ViewId[0] to ViewId[3] may be reordered according to the corresponding view positions view_position[0] to view_position[3] to obtain Fig. 10A The display order in .

[0142] According to an embodiment of the present disclosure, pictures associated with views of various layers in a CVS can be decoded from a code stream. Based on a first SEI message and a second SEI message, a one-dimensional position (e.g., along a horizontal axis) of a view to be displayed can be determined. A decoded picture can be displayed based on the one-dimensional position (e.g., along a horizontal axis) of a view. For each of the views of various layers of a CVS, a view ID (e.g., ViewId[i]) of the corresponding view can be determined based on the first SEI message. The one-dimensional position (e.g., along a horizontal axis) of the corresponding view to be displayed can be determined based on a view position variable (e.g., view_position[i]) in a second SEI message and a determined view identifier (e.g., ViewId[i]).

[0143] refer to Fig. 10A and Fig.11, in the example, the value of the syntax element num_view_minus1 is 3, and the number of views to be displayed is 3 plus 1, that is, 4. The integer i of the syntax element sdi_view_id_val[i] in the first SEI message (e.g., the SDI SEI message) is 0, 1, 2, or 3, corresponding to the syntax element sdi_view_id_val[0], sdi_view_id_val[1], sdi_view_id_val[2], or sdi_view_id_val[3], respectively. In the example, sdi_view_id_val[0] is 0, sdi_view_id_val[1] is 1, sdi_view_id_val[2] is 2, and sdi_view_id_val[3] is 3. Based on the second SEI message (e.g., the MVP SEI message), the syntax element view_position[i] indicates the display order or view position of the ViewId equal to the syntax element sdi_view_id_val[i] in the first SEI message. The syntax element view_position[0] indicates the view position with ViewId equal to sdi_view_id_val[0] (e.g., 0), so view ViewId[0] is located at view_position[0] (e.g., 1). The syntax element view_position[1] indicates the view position with ViewId equal to sdi_view_id_val[1] (e.g., 1), so view ViewId[1] is located at view_position[1] (e.g., 2). The syntax element view_position[2] indicates the view position with ViewId equal to sdi_view_id_val[2] (e.g., 2), so view ViewId[2] is located at view_position[2] (e.g., 0). The syntax element view_position[3] indicates the view position with ViewId equal to sdi_view_id_val[3] (e.g., 3), so view ViewId[3] is located at view_position[3] (e.g., 3).

[0144] According to one aspect of the present disclosure, the MVP SEI message includes one or more parameters that may have semantic dependencies on information in the SDI SEI message. For example, the syntax element num_views_minus1 in the MVP SEI message has a semantic dependency on the value of the parameter NumViews derived from the SDI SEI message. In the example, a first value indicating the number of views (e.g., the value of the parameter NumViews) is derived from the SDI SEI message. A second value associated with the number of views (e.g., the value of the syntax element num_views_minus1) is obtained from the MVP SEI message. The second value may be different from the first value. In a consistency check, the number "1" plus the second value may be compared to the first value. Reference Fig. 10A , the first value is 4, and the second value is 3. The second value (e.g., 3) plus 1 is equal to the first value (e.g., 4), and the consistency check is satisfied. Therefore, the MVP SEI message may be subject to constraints associated with the SDI SEI message.

[0145] In an example, when the CVS does not include an SDI SEI message, the CVS does not include an MVP SEI message. In some examples, when there is no SDI SEI message, there is no MVP SEI message. In an example, based on the SDI SEI message being included in the CVS, the MVP SEI message is included in the CVS.

[0146] In an embodiment, the third SEI message indicates that 3D reference display information is signaled. Fig.12 A syntax example (1200) in a third SEI message for indicating 3D reference display information according to an embodiment of the present disclosure is shown.

[0147] The third SEI message (eg, Fig.12 The 3D reference display information SEI message shown in the third SEI message may include reference display information about one or more reference display widths, one or more reference viewing distances, and information about one or more corresponding reference stereo pairs (e.g., one or more view pairs to be displayed for the left and right eyes of the viewer on a reference display at a reference viewing distance). The reference display information may enable a view renderer to generate appropriate stereo pairs for a target screen width and viewing distance. The values ​​of one or more reference display widths and one or more reference viewing distances may be signaled in centimeters. The reference stereo pair of the view specified in the third SEI message may be used to extract or infer parameters related to the distance between the camera centers in the reference stereo pair, which may be used to generate a view for a target display. For a multi-view display, a reference stereo pair may correspond to a view pair that may be simultaneously observed by the left and right eyes of a viewer.

[0148] When the third SEI message is present, the third SEI message may be associated with the AU (e.g., IRAP AU or non-IRAP AU) when all AUs following the AU in decoding order follow the AU in output order. The third SEI message may be applied to the current AU and all AUs following the AU in both output order and decoding order, until but not including the next IRAP AU or the next AU containing another 3D reference display information SEI message.

[0149] The third SEI message (e.g., 3D reference display information SEI message) may specify display parameters (for which the 3D video sequence is optimized) and corresponding reference parameters. Each reference display (e.g., reference display width and optionally corresponding viewing distance) may be associated with a reference pair of views by signaling a corresponding view identifier (e.g., denoted as ViewId). The difference between the values ​​of ViewId is referred to as the baseline distance (e.g., the distance between the centers of the cameras used to obtain the 3D video sequence).

[0150] When the ratio between the viewing distance of the receiver and the reference viewing distance and the ratio between the screen width of the receiver and the reference screen width are the same, the following equations can be used to determine the baseline distance and horizontal shift of the display at the receiver.

[0151] baseline[i] = refBaseline[i] × (refDisplayWidth[i] ÷ displayWidth) Equation 1

[0152] shift[i] = refShift[i] *(refDisplayWidth[i] ÷ displayWidth) Equation 2

[0153] Where refBaseline[i] is equal to (right_view_id[i]-left_view_id[i]) signaled in the third SEI message. Other parameters related to view generation can be determined by using similar equations.

[0154] parameter[i] = refParameter[i] * (refDisplayWidth[i] ÷displayWidth) Equation 3

[0155] refParameter[i] is a view generation related parameter corresponding to a reference pair of views signaled by left_view_id[i] and right_view_id[i]. In the above equation, the width of the visible portion of the display used to display the video sequence may be understood as the "display width". The above equation may be used to determine view pairs and horizontal shifts or other view synthesis parameters when the viewing distance is not scaled proportionally to the screen width compared to the reference display parameters, wherein the effect of applying the above equation is to keep the perceived depth in the same ratio with the viewing distance as in the reference setting.

[0156] When the view synthesis related parameters corresponding to the reference stereo pair are changed from one AU to another AU, the view synthesis related parameters may be scaled by the same scaling factor as the parameters in the AU associated with the third SEI message. The above equation may be applied to obtain the parameters of the subsequent AU, where refParameter[i] is the parameter related to the reference stereo pair associated with the subsequent access unit.

[0157] The horizontal shift of the receiver's display may be modified by scaling it by the same factor as used to scale the baseline distance (or other view synthesis parameter).

[0158] When the CVS does not include the first SEI message (eg, the SDI SEI message), the CVS does not include the third SEI message (eg, the 3D reference display information SEI message).

[0159] refer to Fig.12 , the syntax element prec_ref_display_width may specify an exponent of the maximum allowable truncation error of refDisplayWidth[i], as given by (2-prec_ref_display_width). The value of the syntax element prec_ref_display_width may be in the range of 0 to 31, inclusive.

[0160] The syntax element ref_viewing_distance_flag equal to 1 may indicate the presence of a reference viewing distance. The syntax element ref_viewing_distance_flag equal to 0 indicates the absence of a reference viewing distance.

[0161] The syntax element prec_ref_viewing_dist may specify an exponent of the maximum allowable truncation error of refViewingDist[i], as given by (2-prec_ref_viewing_dist). The value of prec_ref_viewing_dist may be in the range of 0 to 31, inclusive.

[0162] The syntax element num_views_minus1 plus 1 may be equal to NumViews derived from a first SEI message of the CVS (e.g., an SDI SEI message). According to one aspect of the present disclosure, a third SEI message includes one or more parameters that have semantic relevance to information in the first SEI message (e.g., an SDI SEI message). In an example, a first value (e.g., NumViews) indicating the number of views is derived from the first SEI message. A third value (e.g., num_views_minus1) associated with the number of views from the third SEI message is obtained. The third value may be different from the first value. In a consistency check, 1 (e.g., a value of "1") plus the third value may be compared to the first value.

[0163] The syntax element num_ref_displays_minus1 plus 1 may specify the number of reference displays signaled in the third SEI message. The maximum value of the number of reference displays may be based on a third value associated with the number of views in the third SEI message (e.g., num_views_minus1). The value of num_ref_displays_minus1 may be in the range of 0 to num_views_minus1, inclusive.

[0164] The syntax element left_view_id[i] may indicate the ViewId of the left view of the stereoscopic pair corresponding to the i-th reference display.

[0165] The syntax element right_view_id[i] may indicate the ViewId of the right view of the stereoscopic pair corresponding to the i-th reference display.

[0166] The syntax element exponent_ref_display_width[i] may specify the exponent part of the reference display width of the i-th reference display. The value of exponent_ref_display_width[i] may be in the range of 0 to 62, inclusive. The value 63 is reserved for future use by ITU-T|ISO / IEC. A decoder may treat the value 63 as indicating an unspecified reference display width.

[0167] The syntax element mantissa_ref_display_width[i] may specify the mantissa portion of the reference display width of the i-th reference display. The variable refDispWidthBits that specifies the number of bits of the mantissa_ref_display_width[i] syntax element may be derived as follows:

[0168] If the syntax element exponent_ref_display_width[i] is equal to 0, then set refDispWidthBits to be equal to Max(0, prec_ref_display_width - 30).

[0169] Otherwise (0 < exponent_ref_display_width[i] < 63), set refDispWidthBits to be equal to Max(0, exponent_ref_display_width[i] + prec_ref_display_width - 31).

[0170] The syntax element exponent_ref_viewing_distance[i] may specify the exponent part of the reference viewing distance for the i-th reference display. The value of exponent_ref_viewing_distance[i] may be in the range of 0 to 62, inclusive of 0 and 62. The value 63 is reserved for future use by ITU-T|ISO / IEC. The decoder may treat the value 63 as indicating an unspecified reference display width.

[0171] The syntax element mantissa_ref_viewing_distance[i] may specify the mantissa part of the reference viewing distance for the i-th reference display. The variable refViewDistBits, which specifies the number of bits of the syntax element mantissa_ref_viewing_distance[i], is derived as follows:

[0172] If the syntax element exponent_ref_viewing_distance[i] is equal to 0, then set refViewDistBits to be equal to Max(0, prec_ref_viewing_distance - 30).

[0173] Otherwise (0 < exponent_ref_viewing_distance[i] < 63), set refViewDistBits to be equal to Max(0, exponent_ref_viewing_distance[i] + prec_ref_viewing_distance - 31).

[0174] Table 1 shows an exemplary association between the camera parameter variables and the syntax elements in the third SEI message.

[0175] Table 1 - Association between camera parameter variables and syntax elements

[0176] x refDisplayWidth[i] refViewingDistance[i] e exponent_ref_display_width[i] exponent_ref_viewing_distance[i] n mantissa_ref_display_width[i] mantissa_ref_viewing_distance[i] v refDispWidthBits refViewDistBits

[0177] The variables in row x of Table 1 are derived from the corresponding variables or values ​​in rows e, n, and v of Table 1 as follows:

[0178] If e is not equal to 0, the following equation applies:

[0179] x=2 (e–31) ×(1+n÷2 v )

[0180] Otherwise (e.g., e equals 0), the following equation applies:

[0181] x=2 -(30+v) ×n

[0182] Return to reference Fig.12 , the syntax element additional_shift_present_flag[i] equal to 1 may indicate that information about additional horizontal shifts of the left and right views of the i-th reference display is present in the third SEI message. The syntax element additional_shift_present_flag[i] equal to 0 may indicate that information about additional horizontal shifts of the left and right views of the i-th reference display is not present in the third SEI message.

[0183] The syntax element num_sample_shift_plus 512[i] may indicate a recommended additional horizontal shift for a stereo pair corresponding to the i-th reference baseline and the i-th reference display.

[0184] If (num_sample_shift_plus512[i]-512) is less than 0, the left view of the stereoscopic pair corresponding to the i-th reference baseline and the i-th reference display may be shifted by (512-num_sample_shift_plus512[i]) samples in the left direction relative to the right view of the stereoscopic pair.

[0185] Otherwise, if the syntax element num_sample_shift_plus512[i] is equal to 512, no shift may be applied.

[0186] Otherwise, (num_sample_shift_plus512[i]-512) is greater than 0, the left view in the stereoscopic pair corresponding to the i-th reference baseline and the i-th reference display may be shifted by (512-num_sample_shift_plus512[i]) samples in the right direction relative to the right view of the stereoscopic pair.

[0187] The value of num_sample_shift_plus512[i] can be in the range of 0 to 1023, inclusive.

[0188] In an embodiment, shifting the left view by x samples in the left (or right) direction relative to the right view may be performed by the following two steps: 1) shifting the left view by x / 2 samples in the left (or right) direction and shifting the right view by x / 2 samples in the right (or left) direction; 2) filling the left image edge and the right image edge with a width of x / 2 samples in the left view and the right view with the background color.

[0189] The following pseudo code shows an example of a shift process where the left view is shifted by x samples in the left direction relative to the right view:

[0190]

[0191] The following pseudo code shows an example of a shift process where the left view is shifted by x samples in the right direction relative to the right view:

[0192]

[0193] The variable backgroundColour can take different values ​​in different systems, such as black or gray.

[0194] The syntax element three_dimensional_reference_displays_extension_flag equal to 0 may indicate that there is no additional data following in the third SEI message (e.g., 3D reference display SEI message). In an example, in a codestream conforming to some standards, the value of three_dimensional_reference_displays_extension_flag is equal to 0. The value of three_dimensional_reference_displays_extension_flag of 1 is reserved for future use by ITU-T|ISO / IEC. The decoder may ignore all data following the value of three_dimensional_reference_displays_extension_flag of 1 in the third SEI message (e.g., 3D reference display SEI message).

[0195] Fig.13A flowchart outlining an encoding method (1300) according to an embodiment of the present disclosure is shown. In various embodiments, the method (1300) is performed by a processing circuit, such as a processing circuit in a terminal device (310), (320) (330) and (340), a processing circuit that performs the functions of a video encoder (e.g., (403), (603), (703)), etc. In some embodiments, the method (1300) is implemented in software instructions, so when the processing circuit executes the software instructions, the processing circuit performs the method (1300). The method starts at (S1301) and proceeds to (S1310).

[0196] At (S1310), the bitstream corresponding to the view (eg, FIG. 10A to FIG. 10C In an example, the pictures associated with the views may be encoded in a coding order. In an example, the pictures are included in an AU of a CVS, and the pictures (and views) are associated with the same capture time instance or display time instance.

[0197] At (S1320), a first auxiliary enhancement information (SEI) message indicating scalability dimension information (SDI) and a second SEI message indicating view position (MVP) information of a multi-view may be formed. In an example, the first SEI message is represented as an SDI SEI message, and the second SEI message is represented as an MVP SEI message, as described in the present disclosure. In one embodiment, the position of the view to be displayed is based on the first SEI information and the second SEI message.

[0198] The first SEI message (e.g., an SDI SEI message) may indicate the number of views of the CVS. The first SEI message may specify the view ID of the i-th layer in the CVS, where the value i is an integer. The value i may be greater than or equal to 0. In an example, the value i is an integer from 0 to num_views_minus1, as shown in FIG. Fig.11 In an example, the syntax element sdi_view_id_val[i] specifies the value of the view ID (eg, ViewID) of the i-th layer in the CVS. The view IDs of the respective layers in the CVS may be referenced by the second SEI message (eg, MVP SEI message).

[0199] As described above, the coding order (e.g., encoding order or decoding order) may be different from the display order for displaying pictures associated with the view. The second SEI message may indicate the display order. In an example, the second SEI message indicates a view position (e.g., view_position[i]) of one dimension (e.g., horizontal axis) of the view, where the view may be specified by ViewId in the first SEI message, as described in the present disclosure.

[0200] In an example, the second SEI message is associated with the IRAP AU of the CVS. In an example, based on the first SEI message being included in the CVS, the second SEI message is included in the CVS.

[0201] At (S1330), the first SEI message and the second SEI message may be included in the code stream. In an example, the first SEI message and the second SEI message are encoded and transmitted. The view IDs of the layers corresponding to the views in the CVS may be signaled. In an example, the layers correspond to the views, respectively. The view position of one dimension of the view may be signaled.

[0202] Method (1300) proceeds to (S1399) and terminates.

[0203] The method (1300) may be appropriately adapted to various scenarios, and the steps in the method (1300) may be adjusted accordingly. One or more of the steps in the method (1300) may be modified, omitted, repeated, and / or combined. The method (1300) may be implemented in any suitable order. One or more additional steps may be added.

[0204] In an example, the CVS includes a third SEI message (eg, Fig.12 The third SEI message includes a 3D reference display information SEI message indicating a view identifier of a left view, a view identifier of a right view, and 3D reference display information indicating a reference display width of a reference display. The third SEI message may indicate the number of views signaled in the third SEI message and the number of reference displays. The maximum value of the number of reference displays is based on the number of views indicated in the third SEI message.

[0205] In an example, a first value (e.g., NumViews) indicating the number of views is derived from a first SEI message. A third value associated with the number of views from a third SEI message is obtained. The third value may be different from the first value. In a consistency check, 1 (e.g., a value of "1") plus the third value may be compared to the first value.

[0206] Fig.14A flowchart outlining a decoding method (1400) according to one embodiment of the present disclosure is shown. In various embodiments, the method (1400) is performed by a processing circuit, such as a processing circuit in a terminal device (310), (320), (330) and (340), a processing circuit that performs the functions of a video encoder (403), a processing circuit that performs the functions of a video decoder (410), a processing circuit that performs the functions of a video decoder (510), a processing circuit that performs the functions of a video encoder (603), etc. In some embodiments, the method (1400) is implemented in software instructions, so when the processing circuit executes the software instructions, the processing circuit performs the method (1400). The method starts at (S1401) and proceeds to (S1410).

[0207] At (S1410), pictures associated with views of various layers in a coded video sequence (CVS) may be decoded from a bitstream. In an example, pictures associated with a view may be decoded in a decoding order. In an example, the pictures are included in an AU of a CVS, and the pictures (and views) are associated with the same capture time instance or display time instance.

[0208] At (S1420), a position of a view to be displayed in one dimension (e.g., along a horizontal axis) may be determined based on a first auxiliary enhancement information (SEI) message and a second SEI message. The first SEI message may include scalability dimension information (SDI). The second SEI message may include view position (MVP) information of multiple views.

[0209] In an example, the first SEI message comprises an SDI SEI message, and the second SEI message comprises an MVP SEI message.

[0210] The first SEI message (e.g., an SDI SEI message) may indicate the number of views of the CVS. The first SEI message may specify the view ID of the i-th layer in the CVS (e.g., ViewID[i]), where the value i is an integer. The value i may be greater than or equal to 0. In an example, the value i is an integer from 0 to num_views_minus1, as shown in FIG. Fig.11 In an example, the syntax element sdi_view_id_val[i] in the first SEI message specifies the value of the view ID (eg, ViewID) of the i-th layer in the CVS. The view IDs of the respective layers in the CVS may be referenced by the second SEI message (eg, MVP SEI message).

[0211] The second SEI message (e.g., MVP SEI message) may indicate the position of the view in one dimension (e.g., view_position[i]), where the view may be specified by ViewId in the first SEI message. In an example, the one dimension is a horizontal axis, and the position of the view to be displayed is along the horizontal axis.

[0212] In an embodiment, for each of the views of each layer of the CVS, a view ID (e.g., ViewId[i]) of the corresponding view is determined based on the first SEI message. For example, the syntax element sdi_view_id_val[i] in the first SEI message specifies the value of the view ID (e.g., expressed as ViewID). The position of the corresponding view to be displayed in one dimension (e.g., along the horizontal axis) may be determined based on the view position variable (e.g., view_position[i]) in the second SEI message and the determined view ID (e.g., ViewId[i]).

[0213] In an embodiment, the second SEI message is associated with an Intra Random Access Point (IRAP) AU of the CVS.In an example, the location of the view applies to the AUs in the CVS.

[0214] At (S1410), the decoded picture may be displayed based on the determined position of the view in one dimension (eg, along the horizontal axis), such as reference FIG. 10A to FIG. 10C described.

[0215] Method (1400) proceeds to (S1499) and terminates.

[0216] The method (1400) may be appropriately adapted to various scenarios, and the steps in the method (1400) may be adjusted accordingly. One or more of the steps in the method (1400) may be modified, omitted, repeated, and / or combined. The method (1400) may be implemented in any suitable order. One or more additional steps may be added.

[0217] In an embodiment, whether the second SEI message is included in the CVS is based on whether the first SEI message is included in the CVS. When the first SEI message exists in the CVS, the second SEI message may exist in the CVS.

[0218] When the CVS does not include the first SEI message (e.g., SDI SEI message), the CVS does not include the second SEI message (e.g., MVP SEI message). When the first SEI message is not in the CVS, (S1420) is omitted. (S1430) can be applicable to displaying decoded pictures based on decoding order, default order, etc.

[0219] In an embodiment, a first value indicating the number of views (e.g., NumViews) is derived from a first SEI message. A second value associated with the number of views (e.g., num_views_minus1) may be obtained from a second SEI message. The second value may be different from the first value. A consistency check may be performed by comparing 1 plus the second value to the first value.

[0220] In an example, the CVS includes a third SEI message (eg, Fig.12 The third SEI message includes 3D reference display information SEI message indicating the view distance and the view identifier of the left view, the view identifier of the right view, and the reference display width of the reference display. The third SEI message may indicate the number of views and the number of reference displays signaled in the third SEI message. In the example, a first value (e.g., NumViews) indicating the number of views is derived from the first SEI message. A third value (e.g., num_views_minus1) associated with the number of views from the third SEI message is obtained. The third value may be different from the first value. In a consistency check, 1 (e.g., a value of "1") plus the third value may be compared to the first value. The maximum value of the number of reference displays is based on the third value associated with the number of views in the third SEI message.

[0221] The embodiments of the present disclosure may be used alone or in any order. In addition, each of the method (or embodiment), encoder, and decoder may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-volatile computer-readable medium.

[0222] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. Fig.15 A computer system (1500) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0223] Computer software may be encoded using any suitable machine code or computer language that may be assembled, compiled, linked, or similar mechanisms to create code comprising instructions that may be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., either directly or through interpretation, microcode execution, etc.

[0224] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, etc.

[0225] Fig.15 The components shown for the computer system (1500) are exemplary in nature and are not intended to suggest any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. Nor should the configuration of components be interpreted as having any dependency or requirement on any one component or combination of components illustrated in the exemplary embodiment of the computer system (1500).

[0226] The computer system (1500) may include certain human-machine interface input devices. Such human-machine interface input devices may be responsive to input from one or more human users through, for example, tactile input (such as keystrokes, swipes, data glove movements), audio input (such as voice, tapping), visual input (such as gestures), olfactory input (not shown). Human-machine interface devices may also be used to capture certain media that are not necessarily directly related to a person's conscious input, such as audio (such as voice, music, ambient sounds), images (such as scanned images, photographic images obtained from a still image camera), and videos (such as two-dimensional video, three-dimensional video including stereoscopic video).

[0227] Input human interface devices may include one or more of the following (only one of each is depicted): keyboard (1501), mouse (1502), touchpad (1503), touch screen (1510), data gloves (not shown), joystick (1505), microphone (1506), scanner (1507), camera (1508).

[0228] The computer system (1500) may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate one or more human user senses through, for example, tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include tactile output devices (e.g., tactile feedback of a touch screen (1510), a data glove (not shown), or a joystick (1505), but there may also be tactile feedback devices that are not used as input devices), audio output devices (such as: speakers (1509), headphones (not depicted)), visual output devices (such as screens (1510), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities - some of which are capable of outputting two-dimensional visual output or more than three-dimensional output through means such as stereo output; virtual reality glasses (not depicted), holographic displays, and smoke canisters (not depicted)), and printers (not depicted).

[0229] The computer system (1500) may also include human-accessible storage devices and their associated media, such as optical media including media (1521) such as CD / DVD ROM / RW (1520) with CD / DVD, thumb drives (1522), removable hard drives or solid-state drives (1523), traditional magnetic media such as tapes and floppy disks (not depicted), dedicated ROM / ASIC / PLD-based devices such as security dongles (not depicted), and the like.

[0230] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other volatile signals.

[0231] The computer system (1500) may also include an interface (1554) to one or more communication networks (1555). The network may be, for example, wireless, wired, optical. The network may also be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, and the like. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial networks including CAN buses, and the like. Some networks typically require an external network interface adapter attached to some common data port or peripheral bus (1549) (such as, for example, a USB port of the computer system (1500)); other networks are typically integrated into the core of the computer system (1500) by attaching to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1500) can communicate with other entities. Such communications may be one-way receive-only (e.g., broadcast TV), one-way send-only (e.g., CANbus to certain CANbus devices), or two-way, such as to other computer systems using a local area digital network or a wide area digital network. Certain protocols and protocol stacks may be used on each of those networks and network interfaces as described above.

[0232] The above-mentioned human interface devices, human-accessible storage devices, and network interfaces may be attached to the core (1540) of the computer system (1500).

[0233] The core (1540) may include one or more central processing units (CPUs) (1541), graphics processing units (GPUs) (1542), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (1543), hardware accelerators (1544) for certain tasks, graphics adapters (1550), etc. These devices, along with read-only memory (ROM) (1545), random access memory (1546), internal mass storage (1547) such as internal non-user accessible hard drives, SSDs, etc., may be connected via a system bus (1548). In some computer systems, the system bus (1548) may be accessible in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices may be attached to the core's system bus (1548) directly or via a peripheral bus (1549). In one example, a screen (1510) may be connected to a graphics adapter (1550). The architecture of the peripheral bus includes PCI, USB, etc.

[0234] The CPU (1541), GPU (1542), FPGA (1543) and accelerator (1544) can execute certain instructions, and the combination of these instructions can constitute the above-mentioned computer code. The computer code can be stored in ROM (1545) or RAM (1546). Transition data can also be stored in RAM (1546), while permanent data can be stored in, for example, internal mass storage (1547). Fast storage and retrieval of any memory device can be enabled by using cache memory, which can be closely associated with one or more CPUs (1541), GPUs (1542), mass storage (1547), ROM (1545), RAM (1546), etc.

[0235] The computer readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of a type well known and available to those skilled in the art of computer software.

[0236] As an example and not by way of limitation, a computer system (1500) having an architecture, and in particular a core (1540), may provide functionality as a result of one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be media associated with a user-accessible mass storage as described above, as well as certain memories of the core (1540) having non-volatile properties, such as a core internal mass storage (1547) or ROM (1545). Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the core (1540). Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software may cause the core (1540) and in particular the processors therein (including CPUs, GPUs, FPGAs, etc.) to perform specific parts of specific methods or specific processes described herein, including defining data structures stored in RAM (1546) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of hard-wiring logic or otherwise embodied in circuitry (e.g., accelerator (1544)) that may replace software or operate in conjunction with software to perform specific portions of a particular method or specific process described herein. Where appropriate, references to software may include logic and vice versa. Where appropriate, references to computer-readable media may include circuitry (such as an integrated circuit (IC)) storing software for execution, circuitry embodying logic for execution, or both. The present disclosure includes any suitable combination of hardware and software.

[0237] Appendix A: Acronyms

[0238] JEM: joint exploration model

[0239] VVC: versatile video coding

[0240] BMS: Benchmark set

[0241] MV: Motion Vector

[0242] HEVC: High Efficiency Video Coding

[0243] SEI: Supplementary Enhancement Information

[0244] VUI: Video Usability Information

[0245] GOPs: Groups of Pictures

[0246] TUs: Transform Units

[0247] PUs: Prediction Units

[0248] CTUs: Coding Tree Units

[0249] CTBs: Coding Tree Blocks

[0250] PBs: Prediction Blocks

[0251] HRD: Hypothetical Reference Decoder

[0252] SNR: Signal Noise Ratio

[0253] CPUs: Central Processing Units

[0254] GPUs: Graphics Processing Units

[0255] CRT: Cathode Ray Tube

[0256] LCD: Liquid-Crystal Display

[0257] OLED: Organic Light-Emitting Diode

[0258] CD: Compact Disc

[0259] DVD: Digital Video Disc

[0260] ROM: Read-Only Memory

[0261] RAM: Random Access Memory

[0262] ASIC: Application-Specific Integrated Circuit

[0263] PLD: Programmable Logic Device

[0264] LAN: Local Area Network

[0265] GSM: Global System for Mobile communications

[0266] LTE: Long-Term Evolution

[0267] CANBus: Controller Area Network Bus

[0268] USB: Universal Serial Bus

[0269] PCI: Peripheral Component Interconnect

[0270] FPGA: Field Programmable Gate Areas

[0271] SSD: solid-state drive

[0272] IC:Integrated Circuit

[0273] CU: Coding Unit

[0274] RD: Rate-Distortion

[0275] Although the present disclosure describes several exemplary embodiments, there are changes, arrangements, and various alternative equivalents that fall within the scope of the present disclosure. Therefore, it will be appreciated that those skilled in the art will be able to design many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope.

Claims

1. A video decoding method, characterized in that: include: Decoding pictures associated with multiple views of each layer in the coded video sequence CVS from the bitstream, wherein each layer corresponds to each view; Based on the first auxiliary enhancement information SEI message being included in the CVS, determining that a second SEI message is also included in the CVS; Based on the first SEI message and the second SEI message, positions of one dimension of the plurality of views are determined, wherein: The first SEI message comprises scalability dimension information SDI, a first value NumViews indicating the number of the plurality of views being derived from the first SEI message; The second SEI message includes view position MVP information of multiple views, and the second SEI message includes a syntax element num_views_minus1 and a syntax element view_position[i], wherein: num_views_minus1 indicates a second value of the number of the plurality of views, and num_views_minus1 plus 1 is equal to NumViews; view_position[i] indicates a display order of an i-th view in the multiple views, a view identifier of the i-th view is equal to a value of a syntax element sdi_view_id_val[i] in the first SEI message, sdi_view_id_val[i] specifies a value of a view identifier of an i-th layer in the CVS, the view identifier being used to indicate a decoding order; determining that the CVS further includes a third SEI message, the third SEI message including three-dimensional 3D reference display information about one or more reference display widths and one or more reference viewing distances, information about one or more corresponding reference stereo pairs, and the syntax element num_views_minus1; and, The decoded picture is displayed on an autostereoscopic display based on the one-dimensional positions of the multiple views and the third SEI message.

2. The method according to claim 1, characterized in that In the time dimension, each CVS includes one or more access units AU, and each AU includes one or more pictures of different layers corresponding to the same time instance.

3. The method according to claim 2, characterized in that The information of the reference stereo pair is used to extract or infer parameters related to the distance between the camera centers in the reference stereo pair; The method further comprises: When view synthesis related parameters corresponding to the reference stereo pair are changed from one AU to another AU, the view synthesis related parameters are scaled with the same scaling factor as the parameters in the AU associated with the third SEI message.

4. The method according to claim 1, characterized in that: The second SEI message is associated with an intra random access point IRAP access unit of the CVS.

5. The method according to claim 1, characterized in that The positions of the plurality of views apply to access units in the CVS.

6. A video encoding method, characterized in that: include: Encode the pictures associated with the multiple views of each layer in the bitstream to obtain a coded video sequence CVS, wherein each layer corresponds to each view; When it is determined that a first auxiliary enhancement information SEI message is included in the CVS, it is determined that a second SEI message is also included in the CVS, wherein the first SEI message and the second SEI message are used to determine a one-dimensional position of the multiple views, and the position is used to display the decoded picture on an autostereoscopic display, wherein, The first SEI message includes scalability dimension information SDI, the first SEI message includes a first value NumViews indicating the number of the plurality of views; The second SEI message includes view position MVP information of multiple views, and the second SEI message includes a syntax element num_views_minus1 and a syntax element view_position[i], wherein: num_views_minus1 indicates a second value of the number of the plurality of views, and num_views_minus1 plus 1 is equal to NumViews; view_position[i] indicates a display order of an i-th view in the multiple views, a view identifier of the i-th view is equal to a value of a syntax element sdi_view_id_val[i] in the first SEI message, sdi_view_id_val[i] specifies a value of a view identifier of an i-th layer in the CVS, the view identifier being used to indicate a decoding order; determining that the CVS further includes a third SEI message, the third SEI message including three-dimensional 3D reference display information about one or more reference display widths and one or more reference viewing distances, information about one or more corresponding reference stereo pairs, and the syntax element num_views_minus1; and, The CVS, the first SEI message, the second SEI message and the third SEI message are transmitted in the code stream.

7. A video decoding device, characterized in that: include: A memory for storing program instructions; A processor, configured to call the program instructions stored in the memory to execute the method according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium, characterized in that: A program executable by at least one processor is stored to perform the method according to any one of claims 1 to 5.

Images