Method, apparatus and medium for video processing

Abstract

Aspects of the disclosure provide a method, an apparatus, and a non-transitory computer-readable storage medium for video processing. The method for video processing includes receiving, from a bitstream, pictures associated with a plurality of views, decoding the pictures associated with the plurality of views, determining, from a supplemental enhancement information (SEI) message in the bitstream, positions of multi-dimensional coordinates in a multi-dimensional space for the plurality of views, the positions including at least both a vertical view position and a horizontal view position, rendering the pictures based on a rendering view corresponding to the vertical view position and the horizontal view position, and reordering the pictures based on the rendered pictures and the positions of the multi-dimensional coordinates in the multi-dimensional space.

Description

Video processing methods, devices and media

[0001] Cross-reference to related applications

[0002] This application claims the benefit of priority to U.S. Patent Application No. 17 / 824,639, filed May 25, 2022, entitled “TECHNIQUES FOR SIGNALING MULTIVIEW VIEW POSITIONS IN SEI MESSAGE,” which claims the benefit of priority to U.S. Provisional Application No. 63 / 250,182, filed September 29, 2021, entitled “TECHNIQUES FOR TWODIMENSIONAL MULTIVIEW VIEW POSITION SEI MESSAGE FOR CODED VIDEO STREAM.” The entire disclosure of the earlier applications is incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to the field of video encoding / decoding technology, and more specifically to methods, apparatuses, and non-transitory computer-readable storage media for video processing. Background Technology

[0004] The background description provided herein is for the purpose of presenting the overall context of this disclosure. To the extent that the work described in this background section is intended, neither the work of the currently identified inventors nor any aspects of the description which at the time of filing may not be otherwise limited to being prior art are expressly or implicitly acknowledged as prior art to this disclosure.

[0005] Uncompressed digital video can comprise a series of images, each with spatial dimensions such as 1920×1080 luminance samples and associated chrominance samples. This series of images can have a fixed or variable frame rate (also informally referred to as the frame rate), such as 60 images per second or 60Hz. Uncompressed video has specific bitrate requirements. For example, 1080p60 4:2:0 video at 8 bits per sample (1920×1080 luminance sample resolution at 60Hz frame rate) requires approximately 1.5 Gbit / s of bandwidth. One hour of such video would require over 600 gigabytes of storage.

[0006] One objective of video encoding and decoding is to reduce redundancy in the input video signal through compression. Compression can help reduce the bandwidth and / or storage requirements mentioned above, in some cases by two orders of magnitude or more. Lossless compression, lossy compression, and combinations thereof can be used. Lossless compression refers to the technique of reconstructing an accurate copy of the original signal from the compressed original signal. When using lossy compression, the reconstructed signal may differ from the original signal, but the distortion between the original and reconstructed signals is small enough that the reconstructed signal is useful for the intended application. In the case of video, lossy compression is widely used. The amount of distortion tolerated depends on the application; for example, users of some consumer streaming applications may tolerate higher distortion than users of television distribution applications. The achievable compression ratio can be reflected in the fact that higher allowable / tolerable distortion can result in a higher compression ratio.

[0007] Video encoders and decoders can utilize techniques from several broad categories, including motion compensation, transform, quantization, and entropy coding.

[0008] Video codec techniques can include techniques known as intra-frame coding. In intra-frame coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference images. In some video codecs, images are spatially subdivided into sample blocks. When all sample blocks are encoded in intra-frame mode, the image can be an intra-frame image. Intra-frame images and their derivatives, such as independent decoder refresh images, can be used to reset the decoder state and thus can be used as the first image in an encoded video bitstream and video session, or as a still image. Samples of intra-frame blocks can be transformed, and the transform coefficients can be quantized before entropy coding. Intra-frame prediction can be a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the transformed DC value and the smaller the AC coefficients, the fewer bits are needed to represent the entropy-coded block at a given quantization step size.

[0009] Traditional intra-frame coding, such as intra-frame coding techniques known from, for example, MPEG (Moving Picture Experts Group)-2, does not use intra-frame prediction. However, some newer video compression techniques include attempts to make predictions based on surrounding sample data and / or metadata obtained during the encoding and / or decoding of, for example, spatially adjacent data blocks that are earlier in the decoding order. Such techniques are referred to below as "intra-frame prediction" techniques. Note that in at least some cases, intra-frame prediction uses reference data only from the current picture being reconstructed, and not reference data from a reference picture.

[0010] Many different forms of intra-prediction can exist. When more than one such technique can be used for a given video coding technique, the technique used can be encoded within an intra-prediction mode. In some cases, a mode can have sub-modes and / or parameters, and these sub-modes and / or parameters can be encoded separately or included in the mode codeword. Which codeword is used for a given combination of modes, sub-modes, and / or parameters can affect the coding efficiency gain through intra-prediction, and therefore also the entropy coding technique used to convert the codeword into a bitstream.

[0011] Some modes of intra-frame prediction were introduced with H.264, refined in H.265, and further refined in newer coding techniques such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Sets (BMS). Prediction blocks can be formed using neighboring sample values ​​belonging to already available samples. The sample values ​​of neighboring samples are copied into the prediction block according to the direction. The reference to the direction in use can be encoded in the bitstream, or the reference to the direction in use can be predicted manually.

[0012] Videos can be categorized into single-view videos and multi-view videos. For example, a single-view video (also known as a single-field-of-view video) is a two-dimensional medium that provides the viewer with a single view of a scene; while a multi-view video can provide multiple viewpoints of a scene and offer the viewer a sense of realism. For instance, 3D video can provide two views, such as a left view and a right view corresponding to a human viewer. These two views can be displayed (presented) simultaneously or nearly simultaneously using light with different polarizations, and the viewer can wear polarized glasses so that each eye of the viewer receives the corresponding view.

[0013] Multiview video is typically created by simultaneously capturing a scene using multiple camera devices, each appropriately positioned to capture the scene from a single viewpoint. Therefore, the multiple camera devices will capture multiple video sequences corresponding to multiple viewpoints. To provide even more views, even more camera devices can be used to generate multiview video with a large number of video sequences associated with each view.

[0014] The supplementary enhancement information (SEI) message used to indicate the view position can be called a multiview view position (MVP) SEI message. Current MVP SEIs cannot represent multidimensional arrays, such as a two-dimensional array of multiple views along vertical and horizontal axes. Since most state-of-the-art 3D display devices based on light field and holographic technologies support two-dimensional representations of multiple views, it is quite important to extend the dimension of the view position in the MVP SEI information from one dimension to multiple dimensions. Summary of the Invention

[0015] This disclosure provides methods and apparatus for video processing in various aspects. In some embodiments, the video processing method includes: receiving images associated with a plurality of views from a bitstream; decoding the images associated with the plurality of views; determining, based on Supplemental Enhancement Information (SEI) messages in the bitstream, the positions of multidimensional coordinates in a multidimensional space for each of the plurality of views, the positions including at least a vertical view position and a horizontal view position; rendering the images based on rendering views corresponding to the vertical and horizontal view positions; and reordering the images based on the rendered images and the positions of the multidimensional coordinates in the multidimensional space.

[0016] In some implementations, determining the position of the multidimensional coordinates in a multidimensional space for multiple views based on the SEI message further includes: determining the position of the two-dimensional coordinates in a two-dimensional space for each view based on the SEI message. In some examples, determining the position of the two-dimensional coordinates in a two-dimensional space for multiple views based on the SEI message further includes: obtaining from the SEI message: a first value indicating the number of views; a second value indicating a first number of positions in the vertical dimension of the two-dimensional space; and a third value indicating a second number of positions in the horizontal dimension of the two-dimensional space. Furthermore, determining the position of the two-dimensional coordinates in a two-dimensional space for each view based on the SEI message further includes: obtaining a first coordinate value in a first dimension and a second coordinate value in a second dimension for each of the multiple views as the view position in the two-dimensional space.

[0017] In some examples, SEI messages are associated with Intra-Frame Random Access Point (IRAP) access units of an encoded video sequence.

[0018] In some examples, the SEI message is not in another SEI message.

[0019] In some examples, the SEI message is carried in the encoded video sequence by the bitstream, and the position of the multidimensional coordinates for the view is applied to the access unit in the encoded video sequence.

[0020] In some examples, the SEI message is represented as a Multi-View Position (MVP) SEI message, and the encoded video sequence includes Scalability Dimension Information (SDI) SEI messages. The method also includes obtaining a first value indicating the number of views from the SDI SEI message, obtaining a second value associated with that number from the MVP SEI message, and comparing the second value with the first value in a consistency check.

[0021] In some embodiments, an apparatus for video processing includes: a memory for storing instructions; and a processor for calling the instructions stored in the memory to implement the video processing method described above.

[0022] In some embodiments, the video processing apparatus includes: a receiving unit configured to receive images associated with multiple views from a bitstream; a decoding unit configured to decode the images associated with the multiple views; a position determination unit configured to determine the positions of multidimensional coordinates in a multidimensional space for each of the multiple views based on supplementary enhancement information (SEI) messages in the bitstream, the positions including at least a vertical view position and a horizontal view position; a rendering unit configured to render images based on rendered views corresponding to the vertical view positions and the horizontal view positions; and a sorting unit configured to reorder the images based on the rendered images and the positions of the multidimensional coordinates in the multidimensional space.

[0023] This disclosure also provides a non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause at least one processor to perform a method of video processing.

[0024] Embodiments of this disclosure provide a method, apparatus, and non-transitory computer-readable medium for video processing. The method includes: receiving images associated with multiple views from a bitstream; decoding the images associated with the multiple views; determining the positions of multidimensional coordinates in a multidimensional space for each of the multiple views based on complementary SEI messages in the bitstream, the positions including at least a vertical view position and a horizontal view position; rendering the images based on rendered views corresponding to the vertical and horizontal view positions; and reordering the images based on the rendered images and the positions of the multidimensional coordinates in the multidimensional space. Through the method, apparatus, and non-transitory computer-readable medium of this disclosure, an MVP SEI can represent a multidimensional array, such as a two-dimensional array of multiple views along vertical and horizontal axes, thereby extending the dimension of the view positions in the MVP SEI information from one dimension to multiple dimensions. Attached Figure Description

[0025] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0026] Figure 1A is a schematic illustration of an exemplary subset of intra-prediction modes.

[0027] Figure 1B is an illustration of an exemplary intra-frame prediction direction.

[0028] Figure 2 is a schematic illustration of the current block and its surrounding space merge candidates in an example.

[0029] Figure 3 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment.

[0030] Figure 4 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment.

[0031] Figure 5 is a schematic diagram of a simplified block diagram of the decoder according to an embodiment.

[0032] Figure 6 is a schematic diagram of a simplified block diagram of an encoder according to an embodiment.

[0033] Figure 7 shows a block diagram of an encoder according to another embodiment.

[0034] Figure 8 shows a block diagram of a decoder according to another embodiment.

[0035] Figure 9 shows a diagram of an automatic stereoscopic display in some examples.

[0036] Figures 10A and 10B show examples of reordering images based on the position of the multi-view view.

[0037] Figure 11 shows a syntax example in the supplemental enhancement information (SEI) message used to indicate the view position of multi-view video.

[0038] Figure 12 shows a diagram of an automatic stereoscopic display in some examples.

[0039] Figure 13 shows an example of associated images based on view location in the example.

[0040] Figure 14 shows a syntax example in an SEI message used to indicate the view position in a two-dimensional matrix for multi-view video.

[0041] Figure 15 shows a flowchart outlining some embodiments of the process according to this disclosure.

[0042] Figure 16 shows a flowchart outlining another process according to some embodiments of this disclosure.

[0043] Figure 17 is a schematic illustration of a computer system according to an embodiment. Detailed Implementation

[0044] Referring to Figure 1A, the lower right corner depicts a subset of nine predictor directions known from the 33 possible predictor directions of H.265 (corresponding to 33 angular modes of the 35 intra-frame modes). The point (101) where the arrows converge represents the sample being predicted. The arrows indicate the direction in which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted based on one or more samples at a 45-degree angle to the horizontal line in the upper right corner. Similarly, arrow (103) indicates that sample (101) is predicted based on one or more samples at a 22.5-degree angle to the horizontal line in the lower left corner.

[0045] Referring again to Figure 1A, the upper left corner depicts a square block (104) of 4×4 samples (indicated by a bold dashed line). This square block (104) comprises 16 samples, each labeled with "S," its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from top) and the first sample in the X dimension (from left). Similarly, sample S44 is the fourth sample in block (104) in both the Y and X dimensions. Since the block size is 4×4 samples, S44 is located in the lower right corner. Also shown are reference samples, which follow a similar numbering scheme. Reference samples are labeled with R, their Y position (e.g., row index) relative to block (104), and their X position (column index). In both H.264 and H.265, the predicted samples are adjacent to the blocks being reconstructed; therefore, negative values ​​are not required.

[0046] Intra-frame image prediction works by appropriately copying reference sample values ​​from neighboring samples according to the prediction direction indicated by a signal. For example, suppose the encoded video bitstream includes the following signaling for this block, which indicates a prediction direction consistent with arrow (102)—that is, predicting samples based on one or more prediction samples at a 45-degree angle to the horizontal from the upper right. In this case, samples S41, S32, S23, and S14 are predicted based on the same reference sample R05. Then, sample S44 is predicted based on reference sample R08.

[0047] In some cases, the values ​​of multiple reference samples can be combined, for example, by interpolation, to calculate the reference sample; especially when the orientation cannot be uniformly divided at 45 degrees.

[0048] As video coding technology has evolved, the number of possible directions has also increased. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013), and JEM / VVC / BMS could support up to 65 directions at the time of publication. Experiments have been conducted to identify the most probable directions, and certain techniques in entropy coding have been used to represent these possible directions with a small number of bits, thus incurring some penalty for less likely directions. Furthermore, the direction itself can sometimes be predicted based on neighboring directions used in nearby decoded blocks.

[0049] Figure 1B shows a schematic diagram (110) depicting 65 intra-frame prediction directions according to JEM to illustrate the increase in the number of prediction directions over time.

[0050] The mapping of intra-predicted direction bits representing direction in an encoded video bitstream can vary depending on the video coding technique; and this mapping can range from, for example, a simple direct mapping of the predicted direction to the intra-predicted mode, to codewords, to complex adaptive schemes involving the most probable mode, and similar techniques. However, in all cases, there may be certain directions that are statistically less likely to appear in the video content compared to certain other directions. Since the goal of video compression is to reduce redundancy, in well-functioning video coding techniques, those less likely directions will be represented by a larger number of bits compared to the more likely directions.

[0051] Video encoding and decoding can be performed using inter-frame image prediction with motion compensation. Motion compensation can be a lossy compression technique and can involve using blocks of sample data from a previously reconstructed image or a portion thereof (the reference image), spatially shifted in a direction indicated by a motion vector (hereinafter referred to as the Motion Vector, MV), to predict a newly reconstructed image or a portion thereof. In some cases, the reference image can be the same as the image currently being reconstructed. The MV can have two dimensions, X and Y, or three dimensions, with the third dimension being an indication of the reference image used (the third dimension can indirectly be a temporal dimension).

[0052] In some video compression techniques, an MV applicable to a specific region of sample data can be predicted based on other MVs, such as an MV that is spatially adjacent to the region being reconstructed and precedes that MV in the decoding order. This significantly reduces the amount of data required to encode the MV, thereby eliminating redundancy and improving compression. MV prediction can work effectively, for example, because when encoding an input video signal (referred to as natural video) from a camera device, there is a statistical probability that a larger region than the region applicable to a single MV moves in similar directions, and therefore, in some cases, the larger region can be predicted using similar motion vectors derived from the MVs of adjacent regions. This makes the MV found for a given region similar to or identical to the MV predicted based on surrounding MVs, and thus can be represented after entropy encoding with fewer bits than would be used if the MV were encoded directly. In some cases, MV prediction can be an example of lossless compression of the signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, MV prediction itself may be lossy, for example, due to rounding errors when calculating the predictor based on several surrounding MVs.

[0053] Various video prediction mechanisms are described in H.265 / HEVC (High Efficiency Video Coding) (ITU-T (International Telecommunication Union-Telecommunication Standardization Sector, ITU-T H.265 Recommendation, "High Efficiency Video Coding"), December 2016. Of the various video prediction mechanisms provided by H.265, the technique described here is referred to as "spatial combining" in the following text.

[0054] Referring to Figure 2, the current block (201) may include samples that the encoder discovered during the motion search process and that can be predicted based on previous blocks of the same size that have been spatially shifted. Instead of directly encoding this MV, the MV can be derived from metadata associated with one or more reference images, for example, from the most recent (in decoding order) reference image, using the MV associated with any of the five surrounding samples denoted as A0, A1 and B0, B1, B2 (corresponding to 202 to 206, respectively). In H.265, MV prediction can use a predictor from the same reference image being used by neighboring blocks.

[0055] Figure 3 shows a simplified block diagram of a communication system (300) according to an embodiment of the present disclosure. The communication system (300) includes a plurality of terminal devices that can communicate with each other via, for example, a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of Figure 3, the first pair of terminal devices (310) and (320) perform unidirectional data transmission. For example, terminal device (310) may encode video data (e.g., a video image stream captured by terminal device (310)) for transmission to another terminal device (320) via the network (350). The encoded video data may be transmitted in the form of one or more encoded video bitstreams. Terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to recover the video images, and display the video images based on the recovered video data. Unidirectional data transmission may be common in media service applications, etc.

[0056] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of encoded video data, which may occur, for example, during a video conference. For the bidirectional transmission of data, in this example, each of the terminal devices (330) and (340) may encode video data (e.g., a stream of video images captured by the terminal device) for transmission via a network (350) to the other terminal device (330) and (340). Each of the terminal devices (330) and (340) may also receive encoded video data transmitted by the other terminal device (330) and (340), and may decode the encoded video data to recover the video images, and may display the video images at an accessible display device based on the recovered video data.

[0057] In the example of Figure 3, the terminal devices (310), (320), (330), and (340) may be shown as servers, personal computers, and smartphones, but the principles of this disclosure are not limited thereto. Embodiments of this disclosure are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network (350) refers to any number of networks that transmit encoded video data between the terminal devices (310), (320), (330), and (340), including, for example, wired (connected) and / or wireless communication networks. Communication networks (350) may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet. For the purposes of this discussion, unless otherwise stated below, the architecture and topology of the network (350) may be irrelevant to the operation of this disclosure.

[0058] As an example of an application of the disclosed subject matter, Figure 4 illustrates the placement of a video encoder and a video decoder in a streaming environment. The disclosed subject matter can also be applied to other video-enabled applications, including, for example, video conferencing, digital TV, and storing compressed video on digital media including CDs (Compact Discs), DVDs (Digital Versatile Discs), memory sticks, etc.

[0059] The streaming system may include a capture subsystem (413) that may include a video source (401), such as a digital camera device, that creates, for example, an uncompressed video picture stream (402). In the example, the video picture stream (402) includes samples captured by the digital camera device. The video picture stream (402) is depicted as a thick line to emphasize the high data volume when compared with encoded video data (404) (or encoded video bitstream), which may be processed by an electronic device (420) including a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to implement or enforce aspects of the disclosed subject matter as described in more detail below. Encoded video data (404) (or encoded video bitstream (404)) is depicted as a thin line to emphasize the lower data volume when compared to a video picture stream (402), which may be stored on a streaming server (405) for future use. One or more streaming client subsystems, such as client subsystems (406) and (408) in Figure 4, may access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). Client subsystem (406) may include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and creates an output video picture stream (411) that can be displayed on a display (412) (e.g., a screen) or another presentation device (not depicted). In some streaming systems, encoded video data (404), (407), and (409) (e.g., video bitstreams) can be encoded according to certain video coding / compression standards. Examples of these standards include ITU-T Recommendation H.265. In this example, the video coding standard under development is informally referred to as Versatile Video Coding (VVC). The topics disclosed can be used in the context of VVC.

[0060] Note that electronic devices (420) and (430) may include other components (not shown). For example, electronic device (420) may include a video decoder (not shown), and electronic device (430) may also include a video encoder (not shown).

[0061] Figure 5 shows a block diagram of a video decoder (510) according to an embodiment of the present disclosure. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuitry system). The video decoder (510) may be used in place of the video decoder (410) in the example of Figure 4.

[0062] The receiver (531) can receive one or more encoded video sequences to be decoded by the video decoder (510); one encoded video sequence at a time in the same or another embodiment, wherein the decoding of each encoded video sequence is independent of the others. The encoded video sequences can be received from a channel (501), which can be a hardware / software link to a storage device storing the encoded video data. The receiver (531) can receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams, which can be forwarded to their respective user entities (not depicted). The receiver (531) can separate the encoded video sequences from other data. To prevent network jitter, a buffer memory (515) can be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) is part of the video decoder (510). In other applications, the buffer memory (515) can be external to the video decoder (510) (not depicted). In some other applications, a buffer memory (not depicted) may exist outside the video decoder (510) to prevent network jitter, for example, and another buffer memory (515) may exist inside the video decoder (510) to handle broadcast timing, for example. The buffer memory (515) may not be necessary, or the buffer memory (515) may be small, when the receiver (531) receives data from a store / forward device with sufficient bandwidth and controllability, or from an isochronous synchronization network. For the purpose of utilizing packet networks such as the Internet, a buffer memory (515) may be required. This buffer memory (515) may be relatively large and advantageously have an adaptive size, and may be implemented at least partially in the operating system or in a similar element (not depicted) outside the video decoder (510).

[0063] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. These symbols may include information for managing the operation of the video decoder (510) and potential information for controlling a presentation device such as a presentation device (512) (e.g., a display screen), which is not part of the electronic device (530) but may be coupled to it, as shown in Figure 5. Control information for the presentation device may be in the form of Supplemental Enhancement Information (SEI) messages or fragments of Video Usability Information (VUI) parameter sets (not depicted). The parser (520) may perform parsing / entropy decoding on the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) can extract a subgroup parameter set from the encoded video sequence for use in the video decoder based on at least one parameter corresponding to a group. Subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser (520) can also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc., from the encoded video sequence.

[0064] The parser (520) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515) to create symbols (521).

[0065] Depending on the type of encoded video frames or a subset of encoded video frames (e.g., inter-frame and intra-frame frames, inter-frame and intra-frame blocks) and other factors, the reconstruction of the symbol (521) may involve multiple different units. Which units are involved and how they are involved can be controlled by the subgroup control information parsed by the parser (520) from the encoded video sequence. For simplicity, the flow of this subgroup control information between the parser (520) and the following multiple units is not depicted.

[0066] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and can be at least partially integrated with each other. However, for the purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into the functional units described below.

[0067] The first unit is the scaler / inverse transform unit (551). The scaler / inverse transform unit (551) receives quantization transform coefficients as symbols (521) from the parser (520) and control information, including which transform to use, block size, quantization factor, quantization scaling matrix, etc. The scaler / inverse transform unit (551) can output a block containing sample values, which can be input into the aggregator (555).

[0068] In some cases, the output samples of the scaler / inverse transform (551) may belong to intra-coded blocks; that is, blocks that do not use predictive information from previously reconstructed images but can use predictive information from previously reconstructed portions of the current image. Such predictive information can be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) uses surrounding reconstructed information obtained from the current picture buffer (558) to generate blocks of the same size and shape as the blocks in the reconstruction. For example, the current picture buffer (558) buffers partially reconstructed and / or fully reconstructed current images. In some cases, the aggregator (555) adds the predictive information already generated by the intra-picture prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) based on each sample.

[0069] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to an inter-frame coded block and potentially to a motion-compensated block. In this case, the motion-compensated prediction unit (553) can access the reference image memory (557) to obtain samples for prediction. After motion compensation of the obtained samples according to the symbols (521) belonging to the block, these samples can be added by the aggregator (555) to the output of the scaler / inverse transform unit (551) (referred to in this case as residual samples or residual signals) to generate output sample information. The address in the reference image memory (557) for obtaining the predicted samples can be controlled by motion vectors, which can be used by the motion-compensated prediction unit (553) in the form of symbols (521), which may have, for example, X components, Y components, and reference image components. Motion compensation may also include interpolation of sample values ​​obtained from the reference image memory (557) when using subsample precise motion vectors, motion vector prediction mechanisms, etc.

[0070] The output samples of the aggregator (555) can undergo various loop filtering techniques in the loop filter unit (556). The video compression technique may include an in-loop filtering technique controlled by parameters included in the encoded video sequence (also referred to as the encoded video bitstream) and available to the loop filter unit (556) as symbols (521) from the parser (520). However, the video compression technique may also respond to metadata obtained during decoding of a previous (in decoding order) portion of the encoded picture or encoded video sequence, as well as sample values ​​in response to previous reconstruction and loop filtering.

[0071] The output of the loop filter unit (556) can be a sample stream, which can be output to the presentation device (512) and stored in the reference image memory (557) for future inter-frame image prediction.

[0072] Once fully reconstructed, some of the encoded images can be used as reference images for future predictions. For example, once the encoded image corresponding to the current image has been fully reconstructed and that encoded image (by, for example, the parser (520)) has been identified as the reference image, the current image buffer (558) can become part of the reference image memory (557), and a new current image buffer can be reallocated before the reconstruction of subsequent encoded images begins.

[0073] The video decoder (510) can perform decoding operations according to a predetermined video compression technique as specified in a standard such as ITU-T Recommendation H.265. An encoded video sequence can conform to the syntax specified by the video compression technique or standard used, in the sense that the encoded video sequence follows the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. Specifically, the profile can select certain tools from all available tools in the video compression technique or standard as tools available only under that profile. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (measured in megasamples per second, for example), maximum reference picture size, etc. In some cases, the limitations set by the hierarchy can be further limited by the Hypothetical Reference Decoder (HRD) specification and metadata managed by the HRD buffer used for signaling in the encoded video sequence.

[0074] In this implementation, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The video decoder (510) may use the additional data to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0075] Figure 6 shows a block diagram of a video encoder (603) according to an embodiment of the present disclosure. The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuitry system). The video encoder (603) can be used in place of the video encoder (403) in the example of Figure 4.

[0076] The video encoder (603) can receive video samples from a video source (601) (not part of the electronic device (620) in the example of Figure 6) that can capture video images to be encoded by the video encoder (603). In another example, the video source (601) is part of the electronic device (620).

[0077] A video source (601) can provide a sequence of source video samples in the form of a digital video sample stream to be encoded by a video encoder (603). This digital video sample stream can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601YCrCb, RGB, ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (601) can be a storage device storing previously prepared video. In a video conferencing system, the video source (601) can be a camera device capturing local image information as a video sequence. Video data can be provided as multiple individual pictures that are given motion when viewed sequentially. The pictures themselves can be organized as spatial pixel arrays, where each pixel can include one or more samples, depending on the sampling structure, color space, etc., used. Those skilled in the art will readily understand the relationship between pixels and samples. The following description focuses on samples.

[0078] According to the implementation, the video encoder (603) can encode and compress images of the source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate is a function of the controller (650). In some implementations, the controller (650) controls and is functionally coupled to other functional units as described below. For simplicity, the coupling is not depicted. Parameters set by the controller (650) may include rate control related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other suitable functions belonging to the video encoder (603) optimized for a specific system design.

[0079] In some implementations, the video encoder (603) is configured to operate within an encoding loop. For simplicity, in this example, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on the input picture to be encoded and a reference picture) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to how a (remote) decoder would also create sample data (since any compression between the symbols and the encoded video bitstream is lossless in the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since decoding of the symbol stream produces bit-accurate results regardless of the decoder's location (local or remote), the contents of the reference picture memory (634) are also bit-accurate between the local and remote encoders. In other words, the reference picture samples “seen” by the encoder's prediction portion are exactly the same sample values ​​that the decoder will “see” when using prediction during decoding. The basic principles of reference image synchronization (and drift that occurs when synchronization cannot be maintained, for example, due to channel errors) are also used in some related technologies.

[0080] The operation of the “local” decoder (633) can be the same as that of the “remote” decoder, such as the video decoder (510) which has been described in detail above in conjunction with FIG5. However, referring briefly to FIG5 as well, since symbols are available and the encoding of symbols into a encoded video sequence by the entropy encoder (645) and the decoding of symbols by the parser (520) can be lossless, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and the parser (520), may not be fully implemented in the local decoder (633).

[0081] It can be observed that any decoder technique other than parsing / entropy decoding, which exists in the decoder, must also necessarily exist in the corresponding encoder in essentially the same functional form. For this reason, the subject matter presented focuses on decoder operation. The description of encoder techniques can be simplified, as encoder techniques are the opposite of those comprehensively described decoder techniques. More detailed descriptions are only provided in certain sections below.

[0082] In some examples, during operation, the source encoder (630) may perform motion-compensated predictive coding, which predictively codes the input image with reference to one or more previously encoded images from the video sequence designated as "reference images." In this way, the encoding engine (632) encodes the differences between pixel blocks of the input image and pixel blocks of the reference image, which may be selected as the predictive reference for the input image.

[0083] The local video decoder (633) can decode encoded video data of a picture that can be designated as a reference picture based on symbols created by the source encoder (630). The operation of the encoding engine (632) can advantageously be lossy. When the encoded video data can be decoded at the video decoder (not shown in FIG. 6), the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding processing that can be performed on the reference picture by the video decoder, and can store the reconstructed reference picture in a reference picture buffer (634). In this way, the video encoder (603) can locally store a copy of the reconstructed reference picture that shares common content with the reconstructed reference picture that will be obtained by the remote video decoder (no transmission errors).

[0084] The predictor (635) can perform a prediction search against the encoding engine (632). That is, for a new image to be encoded, the predictor (635) can search in the reference image memory (634) for sample data (as candidate reference pixel blocks) or specific metadata such as reference image motion vectors, block shapes, etc., that can be used as appropriate prediction references for the new image. The predictor (635) can operate pixel-by-pixel based on the sample blocks to find appropriate prediction references. In some cases, as determined by the search results obtained by the predictor (635), the input image may have prediction references obtained from multiple reference images stored in the reference image memory (634).

[0085] The controller (650) can manage the encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.

[0086] The outputs of all the aforementioned functional units can undergo entropy encoding in the entropy encoder (645). The entropy encoder (645) converts the symbols generated by the various functional units into an encoded video sequence by performing lossless compression on the symbols according to techniques such as Huffman coding, variable-length coding, arithmetic coding, etc.

[0087] The transmitter (640) can buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission via a communication channel (660), which can be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (640) can combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (sources not shown).

[0088] The controller (650) can manage the operation of the video encoder (603). During encoding, the controller (650) can assign a specific encoding image type to each encoded image, which may affect the encoding techniques that can be applied to the corresponding image. For example, images can typically be assigned to one of the following image types:

[0089] An intra-frame picture (I-picture) can be a picture that can be encoded and decoded without using any other pictures in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art will understand those variations of I-pictures and their corresponding applications and characteristics.

[0090] Predictive images (P-images) can be images that are encoded and decoded using inter-frame or intra-frame prediction that uses at most one motion vector and a reference index to predict the sample values ​​of each block.

[0091] A bidirectional predictive picture (B-picture) can be an image that can be encoded and decoded using inter-frame or intra-frame prediction that uses at most two motion vectors and reference indices to predict sample values ​​for each block. Similarly, a multi-predictive picture can use more than two reference pictures and associated metadata to reconstruct a single block.

[0092] The source image can typically be spatially subdivided into multiple sample blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples), and encoded block by block. These blocks can be predictively encoded with reference to other (already encoded) blocks, determined by the encoding assignments of the corresponding images applied to the blocks. For example, blocks of image I can be non-predictively encoded, or blocks of image I can be predictively encoded (spatial prediction or intra-frame prediction) with reference to already encoded blocks of the same image. Pixel blocks of image P can be predictively encoded with reference to a previously encoded reference image via spatial prediction or temporal prediction. Blocks of image B can be predictively encoded with reference to one or two previously encoded reference images via spatial prediction or temporal prediction.

[0093] The video encoder (603) can perform encoding operations according to a predetermined video coding technique or standard, such as ITU-T H.265 Recommendation. In the operation of the video encoder (603), various compression operations can be performed, including predictive coding operations utilizing temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0094] In one implementation, the transmitter (640) may transmit additional data along with the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant images and slices, SEI messages, VUI parameter set fragments, etc.

[0095] Video can be captured as multiple source images (video images) in a time-series manner. Intra-frame image prediction (often simply called intra-prediction) utilizes spatial correlations within a given image, while inter-frame image prediction utilizes (temporal or other) correlations between images. In the example, a specific image during encoding / decoding—referred to as the current image—is segmented into blocks. Where a block in the current image resembles a reference block in a previously encoded and still buffered reference image in the video, the block in the current image can be encoded using a vector called a motion vector. The motion vector points to the reference block in the reference image, and when using multiple reference images, the motion vector can have a third dimension that identifies the reference images.

[0096] In some implementations, bidirectional prediction techniques can be used in inter-frame image prediction. According to bidirectional prediction, two reference images are used, such as a first reference image and a second reference image that both precede the current image in the video in decoding order (but may be past and future in display order). A block in the current image can be encoded using a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. A block can be predicted using a combination of the first and second reference blocks.

[0097] In addition, merging mode techniques can be used in inter-frame image prediction to improve coding efficiency.

[0098] According to some embodiments of this disclosure, predictions such as inter-frame picture prediction and intra-frame picture prediction are performed on a block-by-block basis. For example, according to the HEVC standard, pictures in a video picture sequence are segmented into Coding Tree Units (CTUs) for compression, with CTUs in the pictures having the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Typically, a CTU comprises three Coding Tree Blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU can be recursively divided into one or more Coding Units (CUs) using a quadtree. For example, a 64×64 pixel CTU can be divided into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In the example, each CU is analyzed to determine the prediction type used for that CU, such as inter-frame prediction or intra-frame prediction. Based on temporal and / or spatial predictability, a CU is divided into one or more Prediction Units (PUs). Typically, each PU includes a luminance prediction block (PB) and two chrominance PBs. In implementations, prediction operations during decoding (encoding / decoding) are performed on a per-prediction-block basis. Using a luminance prediction block as an example, this block comprises a matrix of pixel values ​​(e.g., luminance values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0099] Figure 7 illustrates a diagram of a video encoder (703) according to another embodiment of the present disclosure. The video encoder (703) is configured to receive processing blocks (e.g., prediction blocks) of sample values ​​within a current video image in a video image sequence, and to encode the processing blocks into an encoded image that is part of an encoded video sequence. In this example, the video encoder (703) is used instead of the video encoder (403) in the example of Figure 4.

[0100] In the HEVC example, the video encoder (703) receives a sample value matrix for processing blocks, such as a prediction block of 8×8 samples. The video encoder (703) uses, for example, rate-distortion optimization to determine the optimal encoding mode for the processing block using intra-frame mode, inter-frame mode, or bidirectional prediction mode. When encoding the processing block in intra-frame mode, the video encoder (703) can use intra-frame prediction techniques to encode the processing block into the encoded picture; while when encoding the processing block in inter-frame mode or bidirectional prediction mode, the video encoder (703) can use inter-frame prediction or bidirectional prediction techniques respectively to encode the processing block into the encoded picture. In some video coding techniques, the merging mode can be an inter-frame picture prediction sub-mode, where motion vectors are derived from one or more motion vector predictors without the aid of encoded motion vector components outside the predictors. In some other video coding techniques, motion vector components applicable to the subject block may exist. In the example, the video encoder (703) includes other components, such as a mode decision module (not shown) for determining the mode of the processing block.

[0101] In the example of Figure 7, the video encoder (703) includes an inter-frame encoder (730), an intra-frame encoder (722), a residual encoder (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together as shown in Figure 7.

[0102] The inter-frame encoder (730) is configured to: receive samples of the current block (e.g., the processing block); compare the block with one or more reference blocks in a reference image (e.g., blocks in previous and subsequent images); generate inter-frame prediction information (e.g., a description based on redundancy information, motion vectors, and merging mode information of the inter-frame coding technique); and compute inter-frame prediction results (e.g., prediction blocks) based on the inter-frame prediction information using any suitable technique. In some examples, the reference image is a decoded reference image based on encoded video information.

[0103] The intra encoder (722) is configured to: receive samples of the current block (e.g., the processing block); in some cases compare the block with already encoded blocks in the same image; generate quantization coefficients after transformation; and in some cases also generate intra prediction information (e.g., intra prediction direction information based on one or more intra coding techniques). In the example, the intra encoder (722) also computes an intra prediction result (e.g., a prediction block) based on the intra prediction information and a reference block in the same image.

[0104] A general controller (721) is configured to determine general control data and, based on said general control data, control other components of the video encoder (703). In an example, the general controller (721) determines the mode of a block and, based on that mode, provides a control signal to a switch (726). For example, when the mode is intra-frame mode, the general controller (721) controls the switch (726) to select an intra-frame mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra-frame prediction information and include said intra-frame prediction information in the bitstream; and when the mode is inter-frame mode, the general controller (721) controls the switch (726) to select an inter-frame prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select inter-frame prediction information and include said inter-frame prediction information in the bitstream.

[0105] A residual calculator (723) is configured to calculate the difference (residual data) between the received block and the prediction selected from the intra-encoder (722) or inter-encoder (730). A residual encoder (724) is configured to operate based on the residual data to encode the residual data to generate transform coefficients. In the example, the residual encoder (724) is configured to transform the residual data from the spatial domain to the frequency domain and generate transform coefficients. The transform coefficients are then quantized to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be appropriately used by the intra-encoder (722) and the inter-encoder (730). For example, an inter-frame encoder (730) can generate a decoded block based on the decoded residual data and inter-frame prediction information, and an intra-frame encoder (722) can generate a decoded block based on the decoded residual data and intra-frame prediction information. The decoded blocks are processed appropriately to generate a decoded image, and in some examples, the decoded image can be buffered in a memory circuit (not shown) and used as a reference image.

[0106] An entropy encoder (725) is configured to format the bitstream to include encoded blocks. The entropy encoder (725) is configured to include various information according to a suitable standard, such as the HEVC standard. In this example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the bitstream. Note that, according to the disclosed subject matter, residual information is not present when blocks are encoded in a merged sub-mode of inter-frame mode or bidirectional prediction mode.

[0107] Figure 8 illustrates a diagram of a video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed image. In this example, the video decoder (810) is used instead of the video decoder (410) in the example of Figure 4.

[0108] In the example of Figure 8, the video decoder (810) includes an entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-frame decoder (872) coupled together as shown in Figure 8.

[0109] The entropy decoder (871) can be configured to reconstruct certain symbols from the encoded picture, which represent the syntax elements constituting the encoded picture. Such symbols may include, for example, the mode encoding the block (such as, for example, intra-mode, inter-mode, bidirectional prediction mode, a merged sub-mode of inter-mode and bidirectional prediction mode, or another sub-mode), prediction information (such as, for example, intra-prediction information or inter-prediction information) that can respectively identify certain samples or metadata used by the intra-decoder (872) or inter-decoder (880) for prediction, residual information in the form of, for example, quantized transform coefficients, etc. In the example, when the prediction mode is inter-mode or bidirectional prediction mode, inter-prediction information is provided to the inter-decoder (880); while when the prediction type is intra-prediction type, intra-prediction information is provided to the intra-decoder (872). The residual information may be inversely quantized and provided to the residual decoder (873).

[0110] The inter-frame decoder (880) is configured to receive inter-frame prediction information and generate inter-frame prediction results based on the inter-frame prediction information.

[0111] The intra-frame decoder (872) is configured to receive intra-frame prediction information and generate prediction results based on the intra-frame prediction information.

[0112] The residual decoder (873) is configured to perform inverse quantization to extract the dequantized transform coefficients and process the dequantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require some control information (to include quantizer parameters (QP)), and this information may be provided by the entropy decoder (871) (the data path is not depicted because this may only be a small amount of control information).

[0113] The reconstruction module (874) is configured to combine the residual output by the residual decoder (873) with the prediction result (in this case, the output of the inter-frame prediction module or the intra-frame prediction module) in the spatial domain to form a reconstructed block, which can be a part of a reconstructed image, which in turn can be a part of a reconstructed video. Note that other suitable operations, such as deblocking, can be performed to improve visual quality.

[0114] Note that any suitable technology can be used to implement the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810). In one implementation, one or more integrated circuits can be used to implement the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810). In another implementation, one or more processors that execute software instructions can be used to implement the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810).

[0115] Various aspects of this disclosure provide techniques for signaling the multi-view location of a multi-view video using Supplemental Enhancement Information (SEI) messages from an encoded video stream.

[0116] Based on some aspects of this disclosure, videos can be classified as single-view videos and multi-view videos. For example, a single-view video (also known as a single-field-of-view video) is a two-dimensional medium that provides a single view of a scene to the viewer; while a multi-view video can provide multiple viewpoints of a scene and can provide the viewer with a sense of realism. In an example, a 3D video can provide two views, such as a left view and a right view corresponding to a human viewer. These two views can be displayed (presented) simultaneously or nearly simultaneously using light with different polarizations, and the viewer can wear polarized glasses so that each eye of the viewer receives the corresponding view.

[0117] In another example, some display devices, such as autostereoscopic displays, can send different images based on the viewer's eye position and do not require glasses. These display devices are called glasses-free 3D displays.

[0118] Multiview video is typically created by simultaneously capturing a scene using multiple camera devices, each appropriately positioned to capture the scene from a single viewpoint. Thus, the multiple camera devices capture multiple video sequences corresponding to multiple viewpoints. To provide more views, even more camera devices can be used to generate multiview video with a large number of video sequences associated with each view. Therefore, multiview video may require significant storage space for storage and / or high bandwidth for transmission. Consequently, multiview video coding techniques have been developed in this field to reduce the required storage space or transmission bandwidth.

[0119] To improve the efficiency of multi-view video coding, the similarity between views is utilized. In some implementations, a view, referred to as the base view, is encoded as a similar single-view video. For example, during the encoding of the base view, intra-frame prediction and / or temporal inter-frame prediction are used. The base view can be decoded using a single-view decoder that performs intra-frame and inter-frame prediction. Other views in the multi-view video besides the base view can be referred to as dependent views. To encode dependent views, in addition to intra-frame and inter-frame prediction, inter-view prediction with disparity compensation can also be used. In an example, in inter-view prediction, a reference block of samples from frames of another view at the same time is used to predict the current block in the dependent view. The position of the reference block is indicated by a disparity vector. This inter-view prediction is similar to inter-frame prediction, but the motion vector is replaced by the disparity vector, and the temporal reference frame is replaced by a reference frame from another view.

[0120] According to some aspects of this disclosure, multi-view coding can employ a multi-layer approach. A multi-layer approach can multiplex different HEVC coded representations of a video sequence—referred to as layers—into a single bitstream. These layers can depend on each other. Dependencies are used by inter-layer prediction to achieve increased compression performance by leveraging the similarity between different layers. Layers can represent texture, depth, or other auxiliary information of a scene relating to a specific camera device's viewpoint. In some examples, all layers belonging to the same camera device's viewpoint are represented as views; and layers carrying the same type of information (e.g., texture or depth) are often referred to as components within the scope of the multi-view video.

[0121] According to one aspect of this disclosure, multi-view video coding may include a combination of High Level Syntax (HLS) (e.g., above the slice level) and an existing single-layer decoding kernel. Therefore, in some examples, multi-view video coding does not change the syntax or decoding processing required for single-layer coding below the slice level (e.g., HEVC). This allows for the reuse of existing implementations to build multi-view video decoders without significant changes. For example, a multi-view video decoder may be implemented based on a video decoder (510) or a video decoder (810).

[0122] In some examples, all pictures associated with the same capture or display moment are contained in a single Access Unit (AU) and have the same Picture Order Count (POC). Multi-view video coding allows for inter-view prediction based on pictures within the same AU. For example, decoded pictures from other views can be inserted into one or two of the reference picture list for the current picture. Furthermore, in some examples, motion vectors can be actual temporal motion vectors when associated with a temporal reference picture of the same view, or disparity vectors when associated with inter-view reference pictures. Therefore, block-level motion compensation modules (e.g., block-level encoding software or hardware, block-level decoding software or hardware) can be used, operating in the same manner regardless of whether the motion vector is a temporal motion vector or a disparity vector.

[0123] According to one aspect of this disclosure, multi-view video coding encodes images of different views at the time of display in an order that is not necessarily related to the order of display positions.

[0124] Figure 9 shows a diagram of an autostereoscopic display (900) in some examples. The autostereoscopic display (900) can display images of different views in response to the detected eye position of an observer. In the examples of Figure 9, the observer's eye position can be detected in one dimension, for example, between the leftmost and rightmost positions. For example, when the observer's eye position is at E0, the autostereoscopic display (900) displays an image of the view with the view identifier ViewId[0]; when the observer's eye position is at E1, the autostereoscopic display (900) displays an image of the view with the view identifier ViewId[1]; when the observer's eye position is at E2, the autostereoscopic display (900) displays an image of the view with the view identifier ViewId[2]; and when the observer's eye position is at E3, the autostereoscopic display (900) displays an image of the view with the view identifier ViewId[3]. The observer's eye positions from left to right are E2, E0, E1, and E3.

[0125] In some examples, the AU includes encoded information of images of different views associated with the same capture or display moment. For example, the AU includes encoded information of the following: image P0 of view ViewId[0], image P1 of view ViewId[1], image P2 of view ViewId[2], and image P3 of view ViewId[3]. Images P0 through P3 may be decoded (encoded or decoded) in a decoding order that may be determined based on certain decoding requirements such as rate-distortion optimization. This decoding order does not necessarily follow the order of the observer's eye position.

[0126] Figures 10A and 10B show examples of reordering images based on the position of the multi-view view.

[0127] Figure 10A shows the images decoded in the AU according to the decoding order. In the example of Figure 10A, images P0 to P3 in the AU are decoded in the order P0, P1, P2, and P3. Since the observer's eye position from left to right is E2, E0, E1, and E3, the order of the decoded images does not correspond to the order of the observer's eye position from left to right.

[0128] In some examples, decoded images can be reordered based on, for example, the display order signaled to the intended user experience. In these examples, the display order is associated with the order of the observer's eye position, such as from left to right.

[0129] Figure 10B shows decoded images that have been reordered in the AU according to the display order in some examples. For example, the display order is associated with the order of the observer's eye position from left to right.

[0130] According to one aspect of this disclosure, supplementary enhancement information (SEI) messages can be used to signal the display order.

[0131] According to some aspects of this disclosure, Supplemental Enhancement Information (SEI) messages may be included in the encoded bitstream to assist in the decoding and / or display of the encoded bitstream, or for another purpose. In some examples, SEI messages are not required for reconstructing luma or chroma samples during decoding processing. Additionally, decoders conforming to video coding standards that support SEI messages do not need to process SEI messages for compliance. For some coding standards, some SEI message information may be required to check bitstream consistency or for output timing decoder consistency.

[0132] SEI messages can contain various types of data that indicate the timing of video frames or describe various characteristics of the encoded video, or how these characteristics can be used or enhanced. In some examples, SEI messages do not affect core decoding processing, but can indicate suggestions on how to post-process or display the video.

[0133] SEI messages can be used to provide additional information about the encoded bitstream, which can be used to change the presentation of the bitstream once it is decoded, or to provide information to the decoder. For example, SEI messages have been used to provide frame packing information (e.g., describing how video data is arranged in video frames), content descriptions (e.g., indicating that the encoded bitstream is, for example, 360-degree video), and color information (e.g., color gamut and / or color range), etc.

[0134] In some examples, the SEI message can be used to signal to the decoder that the encoded bitstream includes 360-degree video. The decoder can use this information to render the video data for 360-degree rendering. Alternatively, if the decoder cannot render 360-degree video, it can use this information to not render the video data.

[0135] In some relevant examples, an SEI message may include information indicating the view's position in a dimension. For instance, an SEI message may include the view's position in a dimension corresponding to the observer's eye position.

[0136] Figure 11 shows a syntax example (1100) of an SEI message used to indicate the view position in a multiview video. An SEI message used to indicate the view position can be called a Multiview View Position (MVP) SEI message. In some examples, the MVP SEI message can signal the number of views, and then signal the position of each view individually.

[0137] In the example of Figure 11, the parameter represented by num_views_minus1 can be signaled by the MVP SEI message, as shown in (1110). The parameter num_views_minus1 indicates, for example, the number of views in the access unit. For example, the number of views is equal to the sum of 1 and the value of the parameter num_views_minus1. The views in the decoding order have view identifiers ViewId[0] to ViewId[3]. The positions of the views in the display order can then be signaled, for example, view_positions[i], in the MVP SEI message, where i is an integer from 0 to num_views_minus1, as shown in (1120) of Figure 11.

[0138] In the example, to signal the view position for the example in Figure 9, “3” can be signaled as the parameter num_views_minus1, and then “1” can be signaled as view_position[0], “2” as view_position[1], “0” as view_position[2], and “3” as view_position[3], where “0” is the leftmost and “3” is the rightmost in the left-to-right positions. Then, when decoding the views ViewId[0] to ViewId[3] from the access unit, the view ViewId[0] has view_position[0], the view ViewId[1] has view_position[1], the view ViewId[2] has view_position[2], and the view ViewId[3] has view_position[3]. The ViewId[0] to ViewId[3] can be reordered according to the corresponding view positions view_position[0] to view_position[3] to obtain the display order in Figure 10B.

[0139] This disclosure provides techniques for using in MVP SEI messages to represent view positions in multidimensional coordinates within a two-dimensional array, such as along both the vertical and horizontal axes, in a multidimensional space. In some examples, the techniques for two-dimensional representations of multi-view view positions can be used to support appropriate displays in 3D display devices that can support 2D representations of multi-view video, such as light field display devices or holographic display devices.

[0140] In some implementations, the technique may employ a list of vertical view positions (e.g., denoted by view_position_y[i]) and a list of horizontal view positions (e.g., denoted by view_position_x[i]), and use a parameter denoted by num_vertical_view_positions_minus1 to indicate the number of vertical view positions and a parameter denoted by num_horizontal_view_positions_minus1 to indicate the number of horizontal view positions.

[0141] Based on some aspects of this disclosure, MVP SEI messages may be subject to similar constraints as those imposed on other multiview-related SEI messages, such as Scalability Dimension Information (SDI) SEI messages, Multiview Acquisition Information (MAI) SEI messages, etc. In some examples, MVP SEI messages are constrained not to be included within scalable nested SEI messages. A scalable nested SEI message is an SEI message that contains one or more additional SEI messages. SEI messages contained within a scalable nested SEI message may be referred to as scalable nested SEI messages.

[0142] An SDI SEI message is an SEI message that signals information about the scalability dimensions of a multiview video. For example, an SDI SEI message may include the number and type of scalability dimensions, such as information indicating the number of views in the multiview video. In some examples (e.g., versions of the generic SEI, such as JVET (Joint Video Experts Team, JVET)-W2006), SDI SEI messages may be subject to the constraint that SDI SEI messages should not be included in scalable nested SEI messages.

[0143] According to one aspect of this disclosure, an MVP SEI message includes one or more parameters that are semantically dependent on information in an SDI SEI message. For example, the syntax element `num_views_minus1` is semantically dependent on the value of the parameter `NumViews` obtained from an SDI SEI message. Therefore, an MVP SEI message may be subject to constraints associated with an SDI SEI message. In some examples, a CVS should not contain an MVP SEI message if it does not contain an SDI SEI message. In some examples, an MVP SEI message should also not exist if the associated SDI SEI message is absent. In another example, because an SDI SEI message should not be included in a scalable nested SEI message, an MVP SEI message should also not be included in a scalable nested SEI message.

[0144] In some examples, a bitstream may include one or more encoded video sequences (CVSs). CVSs are encoded independently of other CVSs. Each CVS may include one or more layers, each layer being a representation of video with a specific quality or spatial resolution, or a representation of some component-specific characteristics, such as a depth or transparency map or perspective view. In the temporal dimension, each CVS includes one or more access units (AUs). Each AU includes one or more pictures belonging to different layers at the same time. A Coded Layer Video Sequence (CLVS) is a layer-by-layer CVS that includes a sequence of picture units within the same layer. If the bitstream has multiple layers, there are one or more CLVSs for each layer of the bitstream's CVS.

[0145] Figure 12 shows some examples of an autostereoscopic display (1200). The autostereoscopic display (1200) can display images of different views in response to, for example, the detected eye position of the observer, the detected head posture of the observer, etc. In the example of Figure 12, the eye position of the observer can be detected in a two-dimensional plane. For example, when the observer's eye position is at E0, the autostereoscopic display (1200) displays an image of the view with the view identifier ViewId[0]; when the observer's eye position is at E1, the autostereoscopic display (1200) displays an image of the view with the view identifier ViewId[1]; when the observer's eye position is at E2, the autostereoscopic display (1200) displays an image of the view with the view identifier ViewId[2]; and when the observer's eye position is at E3, the autostereoscopic display (1200) displays an image of the view with the view identifier ViewId[3]. When the observer's eye position is at E4, the autostereo display (1200) displays an image of the view with the view identifier ViewId[4]; when the observer's eye position is at E5, the autostereo display (1200) displays an image of the view with the view identifier ViewId[5]; when the observer's eye position is at E6, the autostereo display (1200) displays an image of the view with the view identifier ViewId[6]; and when the observer's eye position is at E7, the autostereo display (1200) displays an image of the view with the view identifier ViewId[7].

[0146] In some examples, the AU includes encoded information of images of different views associated with the same capture or display moment. For example, the images in the AU can be decoded into image P0 of view ViewId[0], image P1 of view ViewId[1], image P2 of view ViewId[2], image P3 of view ViewId[3], image P4 of view ViewId[4], image P5 of view ViewId[5], image P6 of view ViewId[6], and image P7 of view ViewId[7]. Images P0 through P7 can be decoded (encoded or decoded) in an encoding order that can be determined based on certain encoding requirements such as rate-distortion optimization. This decoding order does not necessarily follow the order of the observer's eye position.

[0147] Figure 13 shows an example of associated images based on the location of the multi-view view in the example.

[0148] In the example of Figure 13, the images in the AU are decoded in the order of P0, P1, P2, P3, P4, P5, P6, and P7. In some implementations, the MVP SEI message can provide the view position in a 2D matrix, and the images can then be associated with that view position, thus ordering the images according to the view position. In some examples, the ordered images in the 2D matrix can be provided for display in response to the detected observer's eye position.

[0149] In the example of Figure 13, the view position is provided, for example, in the form of two-dimensional coordinates in a 2D matrix. In the example of Figure 13, the 2D matrix has a horizontal axis represented by arrows from left to right and a vertical axis represented by arrows from top to bottom. Images P0 to P7 are associated with the view position.

[0150] Note that although Figure 13 shows a Cartesian coordinate system, other suitable coordinate systems such as polar coordinates, spherical coordinates, etc. can be used in some examples.

[0151] The view position can be provided by an SEI message, also known as an MVP SEI message. The MVP SEI message specifies the relative view position of the view within the encoded video sequence (CVS) along the horizontal and vertical axes. In some examples, when the MVP SEI message is presented in the CVS, it should be associated with an Intra Random Access Picture (IRAP) access unit. A random access point picture is a picture in which the decoder can begin decoding the encoded video sequence. When the random access point picture is intra-coded, it is called an Intra Random Access Picture (IRAP). Note that in some examples, the information in the MVP SEI message signaled within the CVS applies to the entire CVS.

[0152] According to one aspect of this disclosure, an MVP SEI message includes one or more parameters that are semantically dependent on information in an SDI SEI message. For example, the syntax element `num_views_minus1` is semantically dependent on the value of the parameter `NumViews` obtained from an SDI SEI message. Therefore, an MVP SEI message may be subject to constraints associated with an SDI SEI message. In some examples, a CVS should not contain an MVP SEI message if it does not contain an SDI SEI message. In some examples, an MVP SEI message should also not exist if the associated SDI SEI message is absent. In another example, because an SDI SEI message should not be included in a scalable nested SEI message, an MVP SEI message should also not be included in a scalable nested SEI message.

[0153] Figure 14 shows a syntax example (1400) of an MVP SEI message used to indicate the view positions in a 2D matrix of multi-view video. In some examples, the MVP SEI message may signal a first value indicating the number of views, a second value indicating the number of view positions in the horizontal direction, a third value indicating the number of view positions in the vertical direction, and then signal the relative view positions in the horizontal and vertical directions, respectively.

[0154] In the example of Figure 14, the parameter represented by num_views_minus1 can be signaled by the MVP SEI message, as shown in (1410). The parameter num_views_minus1 indicates, for example, the number of views in the access unit. For example, the number of views is equal to the sum of 1 and the value of the parameter num_views_minus1. In some examples, the number of views should be equal to NumViews obtained from the SDI SEI message for CVS.

[0155] In the example of Figure 14, the parameter represented by num_vertical_view_positions_minus1 can be signaled by the MVPSEI message, as shown in (1420). The parameter num_vertical_view_positions_minus1 indicates, for example, the number of vertical view positions in a 2D matrix. For example, the number of vertical view positions is equal to the sum of 1 and the value of the parameter num_vertical_view_positions_minus1. In some examples, the value of num_vertical_view_positions_minus1 should be in the range of 0 to 62 (inclusive).

[0156] In the example of Figure 14, the parameter represented by num_horizontal_view_positions_minus1 can be signaled by the MVP SEI message, as shown in (1430). The parameter num_horizontal_view_positions_minus1 indicates, for example, the number of horizontal view positions in a 2D matrix. For example, the number of horizontal view positions is equal to the sum of 1 and the value of the parameter num_horizontal_view_positions_minus1. In some examples, the value of num_horizontal_view_positions_minus1 should be in the range of 0 to 62 (inclusive).

[0157] Furthermore, in the example of Figure 14, the vertical view position and the horizontal view position are respectively notified by the MVP SEI message signal, as shown in (1460).

[0158] In some examples, the views can be sorted in decoding order. Then, the vertical and horizontal positions of the views in decoding order can be signaled in the MVP SEI message, for example, view_position_y[i] and view_position_x[i], where i is an integer from 0 to num_views_minus1, as shown in (1440) and (1450) of Figure 14.

[0159] The parameter `view_position_y[i]` indicates the vertical order of all views from top to bottom that have a view identifier equal to `ViewId[i]` for display purposes, where the order of the topmost view is equal to 0 and the value increments by 1 for the order of the next view from top to bottom. The value of `view_position_y[i]` should be in the range of 0 to `num_vertical_view_positions_minus1` (inclusive of endpoints).

[0160] The parameter `view_position_x[i]` indicates the horizontal order of views from left to right that have a view identifier equal to `ViewId[i]` for display purposes, where the order of the leftmost view is equal to 0 and the value increments by 1 for the order of the next view from left to right. The value of `view_position_x[i]` should be in the range of 0 to `num_horizontal_view_positions_minus1` (inclusive of endpoints).

[0161] Using the view positions in Figure 13 as an example, for ViewId[0], 0 is signaled as view_position_y[0] and 0 is signaled as view_position_x[0]; for ViewId[1], 0 is signaled as view_position_y[1] and 1 is signaled as view_position_x[1]; for ViewId[2], 0 is signaled as view_position_y[2] and 2 is signaled as view_position_x[2]; for ViewId[3], 0 is signaled as view_position_y[3] and 3 is signaled as view_position_x[3] The signals are used to notify the viewId[4], 1 is used as view_position_y[4] and 0 is used as view_position_x[4]; the signals are used to notify the viewid[5], 1 is used as view_position_y[5] and 1 is used as view_position_x[5]; the signals are used to notify the viewId[6], 1 is used as view_position_y[6] and 2 is used as view_position_x[6]; the signals are used to notify the viewId[7], 1 is used as view_position_y[7] and 3 is used as view_position_x[7].

[0162] Figure 15 shows a flowchart outlining a process (1500) according to an embodiment of the present disclosure. This process (1500) can be used for a video encoder. In various embodiments, the process (1500) is executed by a processing circuitry system, such as the processing circuitry system in terminal devices (310), (320), (330), and (340), a processing circuitry system performing the functions of a video encoder (403), a processing circuitry system performing the functions of a video encoder (603), a processing circuitry system performing the functions of a video encoder (703), etc. In some embodiments, the process (1500) is implemented as software instructions, so that the processing circuitry system executes the process (1500) when the software instructions are executed. The process begins at (S1501) and proceeds to (S1510).

[0163] At (S1510), the image associated with the view is encoded in the bitstream. In some examples, images are captured simultaneously by multiple cameras of the same scene. In some examples, images are generated for displaying the same scene from different views at the same time. In the examples, the images are encoded in access units within the encoded video sequence carried by the bitstream.

[0164] In some examples, the images are videos captured by multiple camera devices. The images are encoded in an access unit of an encoded video sequence.

[0165] At (S1520), an SEI message is formed to include the position of the multidimensional coordinates in the multidimensional space for the view.

[0166] In some implementations, the multidimensional space is a two-dimensional space, so the position of the two-dimensional coordinates in the two-dimensional space for the view is included in the SEI message.

[0167] In some examples, a first value indicating the number of views (e.g., the value of num_views_minus1) is included in the SEI message; a second value indicating the first number of positions in the vertical dimension of the two-dimensional space (e.g., the value of num_vertical_view_positions_minus1) is included in the SEI message; and a third value indicating the second number of positions in the horizontal dimension of the two-dimensional space (e.g., the value of num_horizontal_view_positions_minus1) is included in the SEI message.

[0168] In some examples, for each of multiple views, the first coordinate value of the first dimension and the second coordinate value of the second dimension are included in the SEI message as the view position in two-dimensional space.

[0169] At (S1530), the SEI message is included in the bitstream.

[0170] In some examples, the SEI message is associated with an encoded video sequence, and all access units throughout the encoded video sequence have the same view as the location defined in the SEI message. In some examples, the SEI message is associated with an Intra-Frame Random Access Point (IRAP) access unit of the encoded video sequence.

[0171] Note that in some examples, the SEI message is a non-nested SEI message, and the non-nested SEI message is not within another SEI message.

[0172] In some examples, the SEI message is represented as a Multiview View Position (MVP) SEI message. When the encoded video sequence includes an MVP SEI message, the encoded video sequence also includes a Scalability Dimension Information (SDI) SEI message. In the example, the first value in the SDI SEI message indicates the number of views from the SDI SEI message, and the second value in the MVP SEI message (e.g., the value of num_views_minus1) is associated with the first value in the SDI SEI message, and a consistency check can be applied to examine the relationship between the first and second values.

[0173] Then, the process proceeds to (S1599) and ends.

[0174] The process (1500) can be adjusted as appropriate. Steps in the process (1500) can be modified and / or omitted. Additional steps can be added. Any suitable implementation order can be used.

[0175] Figure 16 shows a flowchart outlining a process (1600) according to an embodiment of the present disclosure. This process (1600) can be used in a video decoder. In various embodiments, the process (1600) is executed by a processing circuitry system, such as the processing circuitry system in terminal devices (310), (320), (330), and (340), a processing circuitry system performing the functions of a video decoder (410), a processing circuitry system performing the functions of a video decoder (510), etc. In some embodiments, the process (1600) is implemented as software instructions, so that the processing circuitry system executes the process (1600) when the software instructions are executed. The process begins at (S1601) and proceeds to (S1610).

[0176] At (S1610), the image associated with the view is decoded from the bitstream. In this example, the image is decoded from an access unit in the encoded video sequence carried by the bitstream. In another example, the image is decoded from multiple access units of the encoded video sequence.

[0177] At (S1620), the positions of the multidimensional coordinates in the multidimensional space for the view are determined according to the SEI message.

[0178] In some implementations, the multidimensional space is a two-dimensional space, so the position of the two-dimensional coordinates in the two-dimensional space for the view is determined according to the SEI message.

[0179] In some examples, a first value obtained from the SEI message indicates the number of views; a second value obtained from the SEI message indicates a first number of positions in the vertical dimension of the two-dimensional space; and a third value obtained from the SEI message indicates a second number of positions in the horizontal dimension of the two-dimensional space.

[0180] In some examples, for each of multiple views, the first coordinate value of the first dimension and the second coordinate value of the second dimension can be obtained from the SEI message as the view position in two-dimensional space.

[0181] In some examples, SEI messages are associated with Intra-Frame Random Access Point (IRAP) access units of an encoded video sequence.

[0182] Note that in some examples, the SEI message is a non-nested SEI message, and the non-nested SEI message is not within another SEI message.

[0183] In some examples, the SEI message is carried in the encoded video sequence by the bitstream, and the position of the multidimensional coordinates for the view is applied to all access units throughout the encoded video sequence.

[0184] In some examples, the SEI message is represented as a Multi-View Position (MVP) SEI message. When the encoded video sequence includes an MVP SEI message, it also includes a Scalability Dimension Information (SDI) SEI message. In the example, a first value indicating the number of views is obtained from the SDI SEI message, and a second value associated with that number is obtained from the MVP SEI message. The second value is then compared to the first value in a consistency check.

[0185] At (S1630), a rendered image is determined from the images based on the rendered view in multidimensional space. In some examples, the rendered view is one of the views, and the rendered image is one of the images selected based on the rendered view. In some examples, when the rendered view is not any of the views, the rendered image can be obtained from the images. In some examples, the rendered view is determined based on the observer's viewport information, such as the observer's eye position, the observer's head pose, the observer's position, etc. Then, the process proceeds to (S1699) and ends.

[0186] The processing (1600) can be adjusted appropriately. Steps in the processing (1600) can be modified and / or omitted. Additional steps can be added. Any suitable implementation order can be used. In some examples, the processing circuitry can receive images associated with multiple views from a bitstream. The processing circuitry decodes the images associated with the multiple views and determines the positions of multidimensional coordinates in the multidimensional space for each of the multiple views based on Supplemental Enhancement Information (SEI) messages in the bitstream. These positions include at least both vertical view positions and horizontal view positions. The processing circuitry renders the images based on the rendered views corresponding to the vertical and horizontal view positions. The processing circuitry reorders the images based on the rendered images and the positions of the multidimensional coordinates in the multidimensional space.

[0187] In embodiments of this disclosure, a video processing apparatus is provided, including a receiving unit configured to receive images associated with multiple views from a bitstream; a decoding unit configured to decode the images associated with the multiple views; a position determination unit configured to determine the positions of multidimensional coordinates in a multidimensional space for each of the multiple views based on supplementary enhancement information (SEI) messages in the bitstream, the positions including at least a vertical view position and a horizontal view position; a rendering unit configured to render images based on rendered views corresponding to the vertical view positions and the horizontal view positions; and a sorting unit configured to reorder the images based on the rendered images and the positions of the multidimensional coordinates in the multidimensional space.

[0188] In embodiments of this disclosure, an apparatus for video processing is provided, comprising: a memory for storing instructions; and a processor for calling the instructions stored in the memory to implement the aforementioned video processing method.

[0189] In embodiments of this disclosure, a non-transitory computer-readable medium is provided storing instructions that, when executed by at least one processor, cause at least one processor to perform the video processing method described above.

[0190] The techniques described above can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Figure 17 illustrates a computer system (1700) suitable for implementing certain embodiments of the disclosed subject matter.

[0191] Computer software can be coded using any suitable machine code or computer language. Machine code or computer language can be subjected to mechanisms such as assembly, compilation, and linking to create code that includes instructions. These instructions can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or through interpretation, microcode execution, etc.

[0192] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0193] The components for the computer system (1700) shown in Figure 17 are exemplary in nature and are not intended to impose any limitation on the scope or functionality of computer software implementing embodiments of this disclosure. The configuration of the components should also not be construed as having any dependency or requirement relating to any one or a combination of components shown in the exemplary embodiments of the computer system (1700).

[0194] The computer system (1700) may include certain human-machine interface input devices. Such human-machine interface input devices can respond to input from one or more human users via, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., speech, tapping), visual input (e.g., gestures), and olfactory input (not depicted). The human-machine interface device can also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image capturing devices), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0195] The input human-machine interface device may include one or more of the following (only one of each is depicted): keyboard (1701), mouse (1702), touchpad (1703), touch screen (1710), data glove (not shown), joystick (1705), microphone (1706), scanner (1707), and camera device (1708).

[0196] The computer system (1700) may also include certain human-machine interface output devices. Such human-machine interface output devices can stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include: haptic output devices (e.g., haptic feedback via a touchscreen (1710), data gloves (not shown), or joystick (1705), but haptic feedback devices that are not used as input devices may also exist); audio output devices (e.g., speakers (1709), headphones (not depicted)); visual output devices (e.g., screens (1710), including CRT (Cathode Ray Tube) screens, LCD (Liquid Crystal Display) screens, plasma screens, OLED (Organic Light Emitting Diode) screens, each with or without touchscreen input capability, each with or without haptic feedback capability—some of which may be able to output two-dimensional or more than three-dimensional visual output in a manner such as stereoscopic image output; virtual reality glasses (not depicted); holographic displays and ashtrays (not depicted)); and printers (not depicted).

[0197] The computer system (1700) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM (Read Only Memory, ROM) / RW (1720) having media such as CD / DVD (1721), thumb drives (1722), removable hard disk drives or solid-state drives (1723), legacy magnetic media such as magnetic tapes and floppy disks (not depicted), devices based on dedicated ROM / ASIC (Application Specific Integrated Circuit, ASIC) / PLD (Programmable Logic Device, PLD) such as security dongles (not depicted), etc.

[0198] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0199] The computer system (1700) may also include network interfaces (1754) to one or more communication networks (1755). The network may be, for example, wireless, wired, or optical. The network may also be local area, wide area, metropolitan area, vehicular and industrial, real-time, latency-tolerant, etc. Examples of networks include: local area networks such as Ethernet; wireless LANs (Local Area Networks); cellular networks including GSM (Global System for Mobile Communications), 3G (the Third Generation), 4G (the Fourth Generation), 5G (the Fifth Generation), LTE (Long Term Evolution), etc.; wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV; vehicle and industrial networks including CANbus, etc. Some networks typically require external network interface adapters that attach to certain general-purpose data ports or peripheral buses (1749) (such as, for example, the USB port of a computer system (1700)); other networks are typically integrated into the core of the computer system (1700) via system buses that attach to systems described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1700) can communicate with other entities. Such communication can be one-way receiving (e.g., broadcasting TV), one-way transmitting (e.g., a CAN bus to certain CAN bus devices), or bidirectional, for example, to other computer systems using local area digital networks or wide area digital networks. Certain protocols and protocol stacks can be used on each of these networks and network interfaces as described above.

[0200] The human-machine interface devices, human-accessible storage devices, and network interfaces mentioned above can be attached to the core (1740) of the computer system (1700).

[0201] The core (1740) may include one or more central processing units (CPUs) (1741), graphics processing units (GPUs) (1742), dedicated programmable processing units (1743) in the form of field-programmable gate areas (FPGAs), hardware accelerators (1744) for certain tasks, graphics adapters (1750), etc. These devices, along with read-only memory (ROM) (1745), random access memory (1746), and internal mass storage devices (1747) such as internal non-user-accessible hard disk drives, SSDs (Solid State Drives), etc., can be connected via the system bus (1748). In some computer systems, the system bus (1748) may be accessed as one or more physical connectors to allow for expansion via additional CPUs, GPUs, etc. Peripheral devices may be attached directly or via a peripheral bus (1749) to the core's system bus (1748). In the example, a screen (1710) may be connected to the graphics adapter (1750). Peripheral bus architectures include PCI (Peripheral Component Interconnect / Interface) and USB (Universal Serial Bus).

[0202] The CPU (1741), GPU (1742), FPGA (1743), and accelerator (1744) can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM (1745) or RAM (Random Access Memory) (1746). Transient data can also be stored in RAM (1746), while permanent data can be stored, for example, in an internal mass storage device (1747). Fast storage and retrieval of any storage device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (1741), GPUs (1742), mass storage devices (1747), ROMs (1745), RAMs (1746), etc.

[0203] Computer-readable media may have computer code thereon for performing operations of various computer implementations. The media and computer code may be specifically designed and constructed for the purposes of this disclosure, or the media and computer code may be of a type known and available to those skilled in the art of computer software.

[0204] By way of example and not limitation, a computer system (1700) having an architecture, and particularly a core (1740), can be functionalized by a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage devices as described above, and certain storage devices of the non-transitory core (1740), such as internal mass storage (1747) or ROM (1745). Software implementing various embodiments of this disclosure can be stored in such devices and executed by the core (1740). Depending on specific needs, the computer-readable media may include one or more memory devices or chips. The software can cause the core (1740), and particularly the processors therein (including CPUs, GPUs, FPGAs, etc.), to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM (1746) and modifying such data structures according to the processes defined by the software. Alternatively or as an alternative, the computer system may provide functionality resulting from or otherwise embodied in circuitry (e.g., an accelerator (1744)) by means of hard-wired logic, which may replace or operate with software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic, and references to logic may also include software. Where appropriate, references to a computer-readable medium may include circuitry storing software for execution (e.g., an integrated circuit (IC)), circuitry implementing logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

[0205] Appendix A: Acronyms

[0206] JEM: Joint Exploration Model

[0207] VVC: Universal Video Coding

[0208] BMS: Benchmark Set

[0209] MV: Motion Vector

[0210] HEVC: High-Efficiency Video Coding

[0211] SEI: Supplemental Enhancement Information

[0212] VUI: Video Availability Information

[0213] GOP: Image Group

[0214] TU: Transformation Unit

[0215] PU: Prediction Unit

[0216] CTU: Coding Tree Unit

[0217] CTB: Coded Tree Block

[0218] PB: Prediction Block

[0219] HRD: Assuming a reference decoder

[0220] SNR: Signal-to-noise ratio

[0221] CPU: Central Processing Unit

[0222] GPU: Graphics Processing Unit

[0223] CRT: Cathode Ray Tube

[0224] LCD: Liquid Crystal Display

[0225] OLED: Organic Light Emitting Diode

[0226] CD: Compact Disc

[0227] DVD: Digital Video Disc

[0228] ROM: Read-Only Memory

[0229] RAM: Random Access Memory

[0230] ASIC: Application-Specific Integrated Circuit

[0231] PLD: Programmable Logic Device

[0232] LAN: Local Area Network

[0233] GSM: Global System for Mobile Communications

[0234] LTE: Long Term Evolution

[0235] CANBus: Controller Area Network Bus

[0236] USB: Universal Serial Bus

[0237] PCI: Peripheral Component Interconnect

[0238] FPGA: Field Programmable Gate Domain

[0239] SSD: Solid State Drive

[0240] IC: Integrated Circuit

[0241] CU: Encoding Unit

[0242] While this disclosure has described several exemplary embodiments, variations, substitutions, and various alternative equivalents fall within the scope of this disclosure. It will therefore be appreciated that those skilled in the art will be able to conceive of many systems and methods that, while not expressly shown or described herein, embody the principles of this disclosure and are therefore within its spirit and scope.

Claims

1. A video processing method, characterized in that, The method includes: receiving images associated with multiple views from a bitstream; decoding the images associated with the multiple views; determining, based on Supplemental Enhancement Information (SEI) messages in the bitstream, the positions of multidimensional coordinates in a multidimensional space for each of the multiple views, the positions including at least a vertical view position corresponding to a top-to-bottom vertical order and a horizontal view position corresponding to a left-to-right horizontal order, the SEI messages including Multi-View Position (MVP) SEI messages indicating the position of the views, the MVP SEI messages not being included in scalable nested SEI messages; rendering the images based on rendering views corresponding to the vertical view positions and the horizontal view positions; and rendering the images based on the rendered images and the bit positions of the multidimensional coordinates in the multidimensional space. The images are reordered; a rendered image is determined based on the observer's viewport information, including the observer's eye position, head pose, and the observer's location; determining the position of the multidimensional coordinates in the multidimensional space for each of the multiple views according to the SEI message further includes: determining the position of the two-dimensional coordinates in the two-dimensional space for each of the multiple views according to the SEI message; determining the position of the two-dimensional coordinates in the two-dimensional space for each of the multiple views according to the SEI message further includes: obtaining a first value from the SEI message, the first value indicating the number of multiple views; obtaining a second value from the SEI message, the second value indicating a first number of positions in the vertical dimension of the two-dimensional space; and obtaining a third value from the SEI message, the third value indicating a second number of positions in the horizontal dimension of the two-dimensional space.

2. The method according to claim 1, characterized in that, Determining the position of the two-dimensional coordinates in the two-dimensional space for the view according to the SEI message further includes: obtaining a first coordinate value in the vertical dimension and a second coordinate value in the horizontal dimension for each of the plurality of views as the view position in the two-dimensional space.

3. The method according to claim 1, characterized in that, The SEI message is associated with an Intra-Frame Random Access Picture (IRAP) access unit of an encoded video sequence.

4. The method according to claim 1, characterized in that, The SEI message is carried in the encoded video sequence by the bitstream, and the position of the multidimensional coordinates relative to the view is applied to the access unit in the encoded video sequence.

5. The method according to claim 4, characterized in that, The encoded video sequence includes Scalability Dimension Information (SDI) SEI messages.

6. The method according to claim 5, characterized in that, The method further includes: obtaining a first value indicating the number of views from the Multi-View Position (MVP) SEI message; obtaining a second value associated with the number from the Scalability Dimension Information (SDI) SEI message; and comparing the second value with the first value in a consistency check.

7. An apparatus for video processing, characterized in that, include: Memory, used to store instructions; A processor configured to invoke instructions stored in the memory to implement the method according to any one of claims 1-6.

8. A video processing device, characterized in that, The device includes: a receiving unit configured to receive images associated with multiple views from a bitstream; a decoding unit configured to decode the images associated with the multiple views; a position determination unit configured to determine, based on Supplemental Enhancement Information (SEI) messages in the bitstream, the positions of the multiple views in a multidimensional space, the positions including at least a vertical view position corresponding to a top-to-bottom vertical order and a horizontal view position corresponding to a left-to-right horizontal order, the SEI messages including Multi-View Position (MVP) SEI messages indicating the position of the views, the MVP SEI messages not being included in scalable nested SEI messages; and a sorting unit configured to render the images based on rendering views corresponding to the vertical view positions and the horizontal view positions, and based on the rendered images... The rendered image and the position of the multidimensional coordinates in the multidimensional space are reordered; the rendered image is determined based on the observer's viewport information, including the observer's eye position, head pose, and the observer's location; determining the position of the multidimensional coordinates in the multidimensional space for each of the multiple views according to the SEI message further includes: determining the position of the two-dimensional coordinates in a two-dimensional space for each of the multiple views according to the SEI message; determining the position of the two-dimensional coordinates in a two-dimensional space for each of the multiple views according to the SEI message further includes: obtaining a first value from the SEI message, the first value indicating the number of multiple views; obtaining a second value from the SEI message, the second value indicating a first number of positions in the vertical dimension of the two-dimensional space; and obtaining a third value from the SEI message, the third value indicating a second number of positions in the horizontal dimension of the two-dimensional space.

9. A non-transitory computer-readable storage medium storing instructions, characterized in that, When executed by at least one processor, the instructions cause the at least one processor to perform the method of any one of claims 1 to 6.

Images