Method and device for video coding

Abstract

Aspects of the present disclosure include a method, an apparatus for video encoding / decoding, and a non-transient computer-readable storage medium. The apparatus includes a processing circuit that receives metadata associated with a coded video bitstream. The metadata includes labeling information of one or more objects detected in a first picture coded in the coded video bitstream. The processing circuit decodes the labeling information of one or more objects of the first picture coded in the coded video bitstream. The processing circuit applies the labeling information to one or more objects of the first picture.

Description

Technology Field

[0001] The present application claims the benefit of priority to U.S. Provisional Application No. 63 / 135,530 ("Signaling of an Object for Machine Work") filed January 8, 2021, and the benefit of priority to U.S. Patent Application No. 17 / 459,753 ("Method and Apparatus for Video Coding") filed August 27, 2021. The entire disclosure of the prior applications is incorporated herein by reference in its entirety.

[0002] The present disclosure describes embodiments generally related to video coding. Background Technology

[0003] The background description provided in this specification is intended to provide the context of the present disclosure in general. The work of the inventors currently named is not recognized as prior art to the present disclosure, either expressly or implicitly, to the extent described in these background sections, as well as aspects of the description that may not have been recognized as prior art at the time of filing.

[0004] Video coding and decoding can be performed using inter-picture prediction along with motion compensation. Uncompressed digital video may contain a series of pictures, each having, for example, a spatial dimension of 1920 x 1080 luminance samples and associated chrominance samples. The series of pictures may have, for example, 60 pictures per second or a fixed or variable picture rate of 60 Hz (informally also called the frame rate). Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (1920x1080 luminance sample resolution at a 60 Hz frame rate) requires bandwidth close to 1.5 Gbit / s. One hour of such video requires more than 600 GB of storage space.

[0005] One objective of video coding and decoding may be to reduce the redundancy of the input video signal through compression. Compression can help reduce the aforementioned bandwidth or storage requirements by more than twofold in some cases. Both lossless and lossy compression, as well as combinations thereof, can be used. Lossless compression refers to a technique where an exact copy of the original signal can be reconstructed from the compressed original signal. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal useful for the intended application. For video, lossy compression is widely used. The amount of acceptable distortion varies depending on the application; for example, users of certain consumer streaming applications may tolerate higher distortion than users of television distribution applications. The achievable compression ratio may reflect that the higher the acceptable / tolerable distortion, the higher the compression ratio can be.

[0006] Video encoders and decoders can utilize techniques from various broad categories, including, for example, motion compensation, transformation, quantization, and entropy coding.

[0007] Video codec technology may include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples from a previously reconstructed reference picture or other data. In some video codecs, a picture is spatially subdivided into sample blocks. If all blocks of samples are coded in intra mode, that picture may be an intra-picture. Derivatives such as intra-pictures and independent decoder refresh pictures can be used to reset the decoder state, and thus can be used as the first picture or still image of a coded video bitstream and video session. Samples in an intra-block may be exposed to a transform, and the transform coefficients may be quantized before entropy coding. Intra-prediction may be a technique that minimizes sample values ​​in a pre-transform domain. In some cases, the smaller the DC value after the transform and the smaller the AC coefficients, the fewer bits are required at a given quantization step size to represent the block after entropy coding.

[0008] For example, traditional intra-coding, such as that known from MPEG-2 generative coding techniques, does not use intra-prediction. However, some new video compression techniques include methods that attempt to utilize, for example, surrounding sample data and / or metadata acquired during the encoding and / or decoding of spatially neighboring and preceding data blocks in the decoding order. These techniques are hereinafter referred to as "intra-prediction" techniques. Note that, at least in some cases, intra-prediction uses only the reference data of the current picture being reconstructed, rather than the reference picture.

[0009] Intra-prediction can take many different forms. If one or more of these techniques can be used in a given video coding technique, the technique in use may be coded in an intra-prediction mode. In certain cases, a mode may have submodes and / or parameters, which may be coded individually or included in a mode codeword. The codeword used for a given combination of mode, submode, and / or parameter can affect the coding efficiency gain through intra-prediction, as can the entropy coding technique used to convert the codeword into a bitstream.

[0010] Specific modes of intra-prediction were introduced in H.264, improved in H.265, and further enhanced in new coding techniques such as the Joint Explosion Model (JEM), versatile video coding (VVC), and benchmark set (BMS). Predictor blocks can be formed using neighbor sample values ​​belonging to already available samples. Sample values ​​from neighbor samples are copied to the predictor block according to direction. References for the direction of use can be encoded in the bitstream or predicted themselves.

[0011] Referring to FIG. 1a, what is depicted in the lower right is a subset of the nine known predictor directions from the 33 possible predictor directions of H.265 (corresponding to the 33 known angle modes of the 35 intra modes). The point where the arrows converge (101) indicates the sample being predicted. The arrows indicate the direction in which the sample is predicted. For example, arrow (102) indicates that the sample (101) is predicted from the sample or samples in the upper right at an angle of 45 degrees from the horizontal. Likewise, arrow (103) indicates that the sample (101) is predicted from the sample or samples in the lower left of the sample (101) at an angle of 22.5 degrees from the horizontal.

[0012] Referring still to FIG. 1a, a square block (104) of 4 x 4 samples is shown in the upper left (indicated by a bold line in the form of a dash). Each square block (104) is labeled "S" and contains 16 samples of positions in the Y dimension (e.g., row index) and positions in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Similarly, sample S44 is the fourth sample of the block (104) in both the Y and X dimensions. Since the block size is 4 x 4 samples, S44 is located in the lower right. Reference samples following a similar numbering scheme are additionally shown. The reference samples are labeled R for the block (104) and are the Y position (e.g., row index) and X position (column index). In both H.264 and H.265, the predicted samples are adjacent to the block being reconstructed. Therefore, there is no need to use negative values.

[0013] Intra-picture prediction can be performed by copying reference sample values ​​from appropriate neighbor samples by the signaled prediction direction. For example, assume that the coded video bitstream includes signaling indicating a prediction direction corresponding to the arrow (102) for these blocks. That is, the samples are predicted from the prediction samples or samples located in the upper right at a 45-degree angle from the horizontal. In this case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from the reference sample R08.

[0014] In certain cases, the values ​​of multiple reference samples can be combined, for example, through interpolation to calculate the reference samples, especially when the directions are not evenly divided by 45 degrees.

[0015] As video coding technology develops, the number of possible directions is increasing. In H.264 (2003), nine different directions could be represented. In H.265 (2013), this increased to 33, and at the time of its release, JEM / VVC / BMS could support up to 65 directions. Experiments were conducted to identify the most likely directions, and specific techniques of entropy coding are used to represent these possible directions in a small number of bits, allowing for a specific penalty for less likely directions. Additionally, the direction itself can sometimes be predicted from neighboring directions used in neighboring blocks that have already been decoded.

[0016] FIG. 1b illustrates a schematic diagram (105) depicting 65 intra-predicted directions according to JEM to illustrate the increasing number of predicted directions over time.

[0017] The mapping of intra-predicted direction bits in a coded video bitstream representing direction can vary depending on the video coding technique; for example, it can range from a simple direct mapping of the predicted direction to complex adaptive schemes involving intra-predicted modes, codewords, and most likely modes, as well as similar techniques. However, in all cases, there may be a specific direction that is statistically less likely to occur in the video content than other specific directions. Since the goal of video compression is to reduce redundancy, in a well-functioning video coding technique, less likely directions are represented by a greater number of bits than more likely directions.

[0018] Motion compensation can be a lossy compression technique and is used to predict a newly reconstructed picture or part of a picture after a block of sample data from a previously reconstructed picture or part thereof (reference picture) has been spatially transitioned in a direction indicated by a motion vector (hereinafter MV). In some cases, the reference picture may be identical to the picture currently being reconstructed. The MV can have two or three dimensions of X and Y, with the third being an indication of the reference picture in use (the latter can indirectly be the time dimension).

[0019] In some video compression techniques, applicable MVs for a specific region of sample data can be predicted from other MVs, for example, from those related to other regions of sample data that are spatially adjacent to the region being reconstructed and precede the MV in the decoding order. Doing so can significantly reduce the amount of data required to code the MV, thereby eliminating redundancy and increasing compression. For example, when coding an input video signal derived from a camera (called natural video), MV prediction can work effectively because regions larger than the applicable area of ​​a single MV shift in a similar direction, and therefore, in some cases, there is a statistical possibility that similar MVs derived from neighboring regions can be predicted. As a result, the MV found for a given region becomes similar or identical to the MV predicted from surrounding MVs, and can be represented with fewer bits than would be used when directly coding the MV after entropy coding. In some cases, MV prediction can serve as an example of lossless compression of a signal (i.e., MV) derived from the original signal (i.e., sample stream). In other cases, the MV prediction itself may be lost due to rounding errors, for example, when calculating the predictor from several surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms provided by H.265, the technique described here is the technique referred to hereafter as "spatial merge".

[0021] Referring to FIG. 1c, the current block (111) may contain samples discovered by the encoder during the motion search process that are predictable from a previous block of the same size that has been spatially transitioned. Instead of directly coding the corresponding MV, the MV may be derived from metadata associated with one or more reference pictures from the most recent (in decoding order) reference picture using an MV associated with one of five neighboring samples, for example, A0, A1, and B0, B1, B2 (112 to 116, respectively). In H.265, the MV prediction may use the predictor of the same reference picture used by the neighboring block.

[0022] An aspect of the present disclosure provides an apparatus for video encoding / decoding. The apparatus includes a processing circuit that receives metadata associated with a coded video bitstream. This metadata includes labeling information of one or more objects detected in a first picture coded in the coded video bitstream. The processing circuit decodes the labeling information of one or more objects of the first picture coded in the coded video bitstream. The processing circuit applies the labeling information to one or more objects of the first picture.

[0023] In one embodiment, metadata is included in the supplementary enhancement information (SEI) message of the coded video bitstream.

[0024] In one embodiment, metadata is included in a file separate from the coded video bitstream.

[0025] In one embodiment, the labeling information indicates the total number of bounding boxes of the first picture and includes location information and size information of each bounding box, and each bounding box is associated with one or more objects of the first picture.

[0026] In one embodiment, the labeling information includes category information indicating a category for each of one or more objects.

[0027] In one embodiment, the labeling information includes identification information that identifies each of one or more objects in a video sequence.

[0028] In one embodiment, position information of one of the bounding boxes includes a position offset of one of the bounding boxes between the first picture and the second picture coded in the video bitstream.

[0029] In one embodiment, location information of one of the bounding boxes indicates a location outside the first picture for one of the bounding boxes based on an object associated with one of the bounding boxes that does not exist in the first picture.

[0030] In one embodiment, the processing circuit transmits a request to receive metadata associated with a coded video bitstream.

[0031] An aspect of the present disclosure provides a method for video encoding / decoding. The method may perform any one or a combination of processes performed by an apparatus for video encoding / decoding. In the method, metadata associated with a coded video bitstream is received. This metadata includes labeling information of one or more objects detected in a first picture coded in the coded video bitstream. The labeling information of one or more objects of the first picture coded in the coded video bitstream is decoded. The labeling information is applied to one or more objects of the first picture.

[0032] An aspect of the present disclosure also provides a non-transient computer-readable medium storing instructions that, when executed by at least one processor, cause at least one processor to perform any one or a combination of methods for video encoding / decoding. Brief explanation of the drawing

[0033] Additional features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. Figure 1a is a schematic diagram of an exemplary subset of intra-prediction modes. Figure 1b is a diagram of an exemplary intra-predicted direction. Figure 1c is a schematic diagram of the current block and its surrounding space merge candidate in one example. FIG. 2 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment. FIG. 3 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment. FIG. 4 is a schematic diagram of a simplified block diagram of a decoder according to an embodiment. FIG. 5 is a schematic diagram of a simplified block diagram of an encoder according to an embodiment. FIG. 6 illustrates a block diagram of an encoder according to another embodiment. FIG. 7 illustrates a block diagram of a decoder according to another embodiment. FIG. 8 illustrates an exemplary architecture used in machine video coding (VCM) according to an embodiment. FIG. 9 illustrates an exemplary encoder used in multi-purpose video coding (VVC) according to an embodiment. FIG. 10 illustrates an exemplary flowchart according to an embodiment. FIG. 11 is a schematic diagram of a computer system according to an embodiment. Specific details for implementing the invention

[0034] I. Video Decoder and Encoder System

[0035] FIG. 2 illustrates a simplified block diagram of a communication system (200) according to an embodiment of the present disclosure. The communication system (200) includes a plurality of terminal devices capable of communicating with each other, for example, through a network (250). For example, the communication system (200) includes a first pair of terminal devices (210, 220) interconnected through the network (250). In the example of FIG. 3, the first pair of terminal devices (210, 220) perform unidirectional transmission of data. For example, the terminal device (210) may encode video data (for example, a stream of video pictures captured by the terminal device (210)) for transmission to another terminal device (220) through the network (250). The encoded video data may be transmitted in the form of one or more coded video bitstreams. The terminal device (220) receives coded video data from the network (250), decodes the coded video data to restore a video picture, and can display the video picture according to the restored video data. Unidirectional data transmission may be common in media serving applications, etc.

[0036] In another example, the communication system (200) includes a second pair of terminal devices (230, 240) that perform bidirectional transmission of coded video data, for example, which may occur during a video conference. For bidirectional transmission of data, in the example, each terminal device of the terminal devices (230, 240) may code video data (e.g., a stream of video pictures captured by the terminal devices) for transmission to another terminal device of the terminal devices (230, 240) via a network (250). Each terminal device of the terminal devices (230, 240) may also receive coded video data transmitted by another terminal device of the terminal devices (230, 240), decode the coded video data to restore the video picture, and display the video picture on an accessible display device according to the restored video data.

[0037] In FIG. 2, terminal devices (210, 220, 230, 2340) may be exemplified as servers, personal computers, and smartphones, but the principles of the present disclosure are not limited thereto. Embodiments of the present disclosure find applications using laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network (250) represents any number of networks that transmit coded video data between terminal devices (210, 220, 230, 240), for example, including wired and / or wireless communication networks. The communication network (250) may exchange data over circuit-switched and / or packet-switched channels. Representative networks include communication networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of the network (250) may not be important to the operation of the present disclosure unless described below.

[0038] FIG. 3 illustrates the arrangement of a video encoder and a video decoder in a streaming environment as an example of an application to the disclosed subject. The disclosed subject may be equally applied to other video-enabled applications, such as video conferencing, digital TV, and storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0039] A streaming system may include a capture subsystem (313) that may include a digital camera generating a stream of a video source (301), for example, an uncompressed video picture (302). In the example, the stream of video picture (302) includes a sample captured by the digital camera. The stream of video picture (302), indicated by a bold line to emphasize a high data volume when compared to encoded video data (304) (or coded video bitstream), may be processed by an electronic device (320) comprising a video encoder (303) combined with the video source (301). The video encoder (303) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter as described in more detail below. Encoded video data (304) (or encoded video bitstream (304)), depicted as a thin line to emphasize a lower data volume when compared to a stream (302) of a video picture, may be stored in a streaming server (305) for future use. One or more streaming client subsystems, such as the client subsystems (306, 308) of FIG. 3, may access the streaming server (305) to retrieve copies (307, 309) of the encoded video data (304). The client subsystem (306) may include a video decoder (410) in, for example, an electronic device (330). The video decoder (310) decodes an incoming copy (307) of the encoded video data and generates an outgoing stream of a video picture (311) that can be rendered on a display (312) (e.g., a display screen) or another rendering device (not shown). In some streaming systems, encoded video data (304, 307, 309) (e.g., video bitstream) may be encoded according to specific video coding / compression standards. Examples of such standards include ITU-T Recommendation H.265.In one example, a video coding standard under development is informally known as Versatile Video Coding (VVC). The disclosed topic may be used in the context of VVC.

[0040] Note that the electronic device (320, 330) may include other components (not shown). For example, the electronic device (320) may include a video decoder (not shown), and the electronic device (330) may also include a video encoder (not shown).

[0041] FIG. 4 illustrates a block diagram of a video decoder (410) according to an embodiment of the present disclosure. The video decoder (410) may be included in an electronic device (430). The electronic device (430) may include a receiver (431) (e.g., a receiving circuit). The video decoder (410) may be used instead of the video decoder (310) in the example of FIG. 3.

[0042] The receiver (431) may receive one or more coded video sequences to be decoded by the video decoder (410), and in the same or different embodiments, may receive one coded video sequence at a time, wherein the decoding of each coded video sequence is independent of the other coded video sequences. The coded video sequences may be received from a channel (401) which may be a hardware / software link to a storage device that stores the encoded video data. The receiver (431) may receive the encoded video data along with other data, e.g., coded audio data and / or auxiliary data streams, which may be delivered to each using an entity (not shown). The receiver (431) may separate the coded video sequences from the other data. To prevent network jitter, a buffer memory (415) may be connected between the receiver (431) and the entropy decoder / parser (420) (hereinafter "parser (420)"). In certain applications, the buffer memory (415) is part of the video decoder (410). In other cases, it may be located outside the video decoder (410) (not shown). In yet another case, the buffer memory (not shown) may be located outside the video decoder (410), for example, to prevent network jitter, and another buffer memory (415) may be located inside the video decoder (410), for example, to handle playback timing. If the receiver (431) receives data from a storage / transmission device or an isochronous network with sufficient bandwidth and controllability, the buffer memory (415) may not be needed or may be small. For use on a packet network such as the Internet, the buffer memory (415) may be required, may be relatively large and advantageously may have an adaptive size, and may be implemented at least partially in the operating system or a similar element (not shown) outside the video decoder (410).

[0043] The video decoder (410) may include a parser (420) to reconstruct symbols (421) from a coded video sequence. Categories of these symbols include information used to manage the operation of the video decoder (410) and information for controlling a rendering device (412) (e.g., a display screen) which is not an essential part of the electronic device but can be connected to the electronic device (430) as illustrated in FIG. 4. The control information for the rendering device(s) may be in the form of Supplemental Enhancement Information (SEI messages) or fragments of Video Usability Information (VUI) parameter sets (not illustrated). The parser (420) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow video coding techniques or standards and may follow various principles including variable-length coding, Huffman coding, context-sensitive arithmetic coding, etc. The parser (420) can extract a set of subgroup parameters for at least one of the subgroups of pixels of a video decoder from a coded video sequence based on at least one parameter corresponding to a group. The subgroups may include a Group of Picture (GOP), a picture, a tile, a slice, a macroblock, a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), etc. The parser (420) can also extract from coded video sequence information such as transform coefficients, quantizer parameter values, MV, etc.

[0044] The parser (420) can generate symbols (421) by performing entropy decoding / parsing operations on a video sequence received from the buffer memory (45).

[0045] The reconstruction of the symbol (421) may include a number of different units depending on the type of coded video picture or a part thereof (e.g., inter- and intra-picture, inter- and intra-block) and other factors. Which units are included and how they are included may be controlled by subgroup control information parsed from the coded video sequence by the parser (420). The flow of this subgroup control information between the parser (420) and the number of units below is not illustrated for clarity.

[0046] In addition to the functional blocks already mentioned, the video decoder (410) can be conceptually subdivided into a number of functional units as described below. In actual implementations operating under commercial constraints, many of these units may interact closely with one another and be at least partially integrated with one another. However, the conceptual subdivision into functional units below is appropriate for illustrating the subject matter disclosed.

[0047] The first unit may be a scaler / inverse transform unit (451). The scaler / inverse transform unit (451) receives quantized transform coefficients as well as control information including a transform for using a block size, a quantization factor, a quantization scaling matrix, etc., as symbol(s) (421) from the parser (420). The scaler / inverse transform unit (451) may output a block containing sample values ​​that can be input to an aggregator (455).

[0048] In some cases, the output sample of the scaler / inverse transform (451) may belong to an intra-coded block, that is, a block that does not use prediction information from a previously reconstructed picture but can use prediction information from a previously reconstructed part of the current picture. This prediction information may be provided by an intra-picture prediction unit (452). In some cases, the intra-picture prediction unit (452) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information fetched from the current (partially reconstructed) picture from the current picture buffer (558). The current picture buffer (458) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. In some cases, the aggregator (455) adds the prediction information generated by the intra-prediction unit (452) on a sample basis to the output sample information provided by the scaler / inverse transform unit (451).

[0049] In other cases, the output samples of the scaler / inverse unit (451) may belong to an intercoded and potentially motion-compensated block. In such cases, the motion compensation prediction unit (453) may access the reference picture memory (457) to fetch samples used for prediction. After motion-compensating the samples fetched according to the symbols (421) belonging to the block, these samples may be added to the output of the scaler / inverse unit (451) (in this case, residual samples or residual signals) by the aggregator (455) to generate output sample information. The address in the reference picture memory (457) where the motion compensation prediction unit (453) fetches the prediction samples may be controlled by a motion vector available to the motion compensation prediction unit (453), for example, in the form of a symbol (421) that may have X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from reference picture memory (457) when accurate subsample MV is in use, MV prediction mechanism, etc.

[0050] The output samples of the aggregator (455) may be subject to various loop filtering techniques in the loop filter unit (456). The video compression technique may include an in-loop filter technique that is controlled by parameters contained in the coded video sequence (also referred to as the coded video bitstream) and made available to the loop filter unit (456) as a symbol (421) from the parser (420), but may correspond not only to previously reconstructed and loop-filtered sample values ​​but also to meta-information obtained while decoding a previous (in the decoding order) part of the coded picture or coded video sequence.

[0051] The output of the loop filter unit (456) may be a sample stream that can be output to the render device (412) as well as stored in the reference picture memory (457) for use in predicting future pictures.

[0052] A specific coded picture that has been completely reconstructed can be used as a reference picture for future prediction. For example, if a coded picture corresponding to a current picture is completely reconstructed and the coded picture is identified as a reference picture (e.g., by a parser (420)), the current picture buffer (458) can become part of the reference picture memory (457), and the fresh current picture buffer can be reallocated before starting the reconstruction of the next coded picture.

[0053] The video decoder (410) can perform decoding operations according to a video compression technology predetermined in a standard such as ITU-T Rec. H.265. In the sense that the coded video sequence complies with both the syntax of the video compression technology or standard and the profile documented in the video compression technology or standard, the coded video sequence may follow the syntax specified by the video compression technology or standard in use. Specifically, the profile may select a specific tool as the only tool available for use under that profile from all tools available in the video compression technology or standard. Additionally, for compliance, it may be required that the complexity of the coded video sequence be within the range defined by the level of the video compression technology or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may, in some cases, be further restricted by the Hypothetical Reference Decoder (HRD) specifications signaled in the coded video sequence and metadata for HRD buffer management.

[0054] In an embodiment, the receiver (431) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (410) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form, for example, a time, spatial, or signal noise ratio (SNR) enhancement layer, redundant slice, redundant picture, forward error correction code, etc.

[0055] FIG. 5 illustrates a block diagram of a video encoder (503) according to an embodiment of the present disclosure. The video encoder (503) is included in an electronic device (520). The electronic device (520) includes a transmitter (540) (e.g., a transmission circuit). The video encoder (503) may be used instead of the video encoder (303) of FIG. 3 example.

[0056] The video encoder (503) can receive video samples from a video source (501) (not part of the electronic device (520) in the example of FIG. 5) capable of capturing video image(s) to be encoded by the video encoder (503). In another example, the video source (501) is part of the electronic device (520).

[0057] A video source (501) may provide a source video sequence to be coded by a video encoder (503) in the form of a digital video sample stream, which may have any appropriate bit depth (e.g., 8 bits, 10 bits, 12 bits, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any appropriate sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (501) may be a storage device that stores a pre-prepared video. In a picture conferencing system, the video source (501) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures that impart motion when viewed in sequence. The picture itself may consist of a spatial array of pixels, where each pixel may contain one or more samples depending on the sampling structure, color space, etc. being used. Those skilled in the art can easily understand the relationship between pixels and samples. The description below focuses on samples.

[0058] According to an embodiment, the video encoder (503) can code and compress pictures of a source video sequence into a coded video sequence (543) in real time or under any other time constraint required by the application. Enhancing the appropriate coding speed is one of the functions of the controller (550). In some embodiments, the controller (550) controls other function units and is functionally coupled to other function units as described below. The coupling is not shown for clarity. Parameters set by the controller (550) may include rate control-related parameters (picture skip, quantizer, lambda value of rate-distortion optimization technique, ...), picture size, picture group (GOP) layout, maximum motion vector search range, etc. The controller (550) may be configured to have other appropriate functions belonging to the video encoder (503) optimized for a specific system design.

[0059] In some embodiments, the video encoder (503) is configured to operate in a coding loop. For the sake of oversimplification, in the example, the coding loop may include a source coder (530) (which, for example, is responsible for generating symbols such as a symbol stream based on the input picture to be coded, and is a reference picture(s)) and a (local) decoder (533) built into the video encoder (503). The decoder (533) reconstructs the symbols to generate sample data in a manner similar to that generated by the (remote) decoder (since any compression between the symbols and the coded video bitstream in the video compression techniques considered in the disclosed subject is lossless). The reconstructed sample stream (sample data) may be input into the reference picture memory (534). Because the decoding of the symbol stream yields an accurate bit result regardless of the decoder location (local or remote), the content within the reference picture memory (534) is also an accurate bit between the local encoder and the remote encoder. In other words, the encoder's prediction part "sees" the reference picture samples as exactly the same sample values ​​that the decoder "sees" when using the prediction during decoding. This fundamental principle of reference picture synchronization (e.g., the resulting drift when synchronization cannot be maintained due to channel errors) is also used in some related technologies.

[0060] The operation of the “local” decoder (533) may be the same as the operation of the “remote” decoder, such as the video decoder (410), which has already been described in detail above in relation to FIG. 4. However, with brief reference to FIG. 5, the entropy decoding part of the video decoder (410), which includes the buffer memory (415) and the parser (420), may not be fully implemented in the local decoder (533) because the symbols are available and the encoding / decoding of the symbols for the video sequence coded by the entropy coder (545) and the parser (420) may be lossless.

[0061] An observation that can be made at this point is that any decoder technique, excluding the parsing / entropy decoding present in the decoder, may also need to exist in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject focuses on decoder operation. The description of the encoder technique may be abbreviated as it is the opposite of the comprehensively described decoder technique. Further details are required only in specific areas and are provided below.

[0062] During operation, in some examples, the source coder (530) may perform motion-compensated predictive coding that predictively codes the input picture by referencing one or more previously coded pictures from a video sequence designated as a "reference picture." In this way, the coding engine (532) codes the difference between a pixel block of the input picture and a pixel block of the reference picture(s) that can be selected as predictive reference(s) for the input picture.

[0063] The local video decoder (533) can decode the coded video data of a picture that can be designated as a reference picture based on symbols generated by the source coder (530). The operation of the coding engine (532) may advantageously be a lossy process. If the coded video data can be decoded by a video decoder (not shown in FIG. 5), the reconstructed video sequence may generally be a copy of the source video sequence with some errors. The local video decoder (533) can replicate the decoding process that can be performed by the video decoder on the reference picture and allow the reconstructed reference picture to be stored in the reference picture cache (534). In this way, the video encoder (503) can locally store a copy of the reconstructed reference picture having common content as the reconstructed reference picture to be acquired by the original video decoder (without transmission errors).

[0064] The predictor (535) can perform a predictive search for the coding engine (532). That is, for a new picture to be coded, the predictor (535) can search the reference picture memory (534) for specific metadata, such as a reference picture MV, block shape, etc., which can serve as a suitable predictive reference for the new picture or sample data (as a candidate reference pixel block) or a suitable predictive reference for the new picture. The predictor (535) can operate based on the sample block to find a suitable predictive reference. In some cases, as determined by the search results obtained by the predictor (535), the input picture may have a predictive reference derived from a number of reference pictures stored in the reference picture memory (534).

[0065] The controller (550) can manage the coding operation of the source coder (530), including the settings of parameters and subgroup parameters used to encode video data, for example.

[0066] The output of all the aforementioned function units can be entropy-coded in an entropy coder (545). The entropy coder (545) converts the symbols generated by the various function units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable-length coding, and arithmetic coding.

[0067] The transmitter (540) may buffer the coded video sequence(s) generated by the entropy coder (545) to prepare for transmission through a communication channel (560), which may be a hardware / software link to a storage device capable of storing the encoded video data. The transmitter (540) may merge the coded video data from the video coder (603) with other data to be transmitted, e.g., coded audio data and / or an auxiliary data stream (source not shown).

[0068] The controller (550) can manage the operation of the video encoder (503). During coding, the controller (550) can assign a specific coded picture type to each coded picture that can affect the coding technique that can be applied to each picture. For example, a picture can often be assigned as one of the following picture types:

[0069] An intra-picture (I-picture) may be one that can be coded and decoded without using other pictures in the sequence as a prediction source. Some video codecs allow various types of intra-pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are aware of these variations of I-pictures and their respective applications and characteristics.

[0070] The prediction picture (P picture) may be coded and decoded using intra prediction or inter prediction, using up to one MV and reference index to predict the sample value of each block.

[0071] A bidirectional prediction picture (Picture B) may be coded and decoded using intra-prediction or inter-prediction, using up to two MVs and reference indices to predict sample values ​​of each block. Similarly, multiple prediction pictures may use two or more reference pictures and associated metadata for the reconstruction of a single block.

[0072] A source picture can generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively) and coded on a block-based basis. Blocks can be predictively coded by referencing other (already coded) blocks as determined by the coding assignment applied to each picture in the block. For example, blocks of picture I can be non-predictively coded, or they can be predictively coded by referencing already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of picture P can be predictively coded via spatial prediction or temporal prediction by referencing one previously coded reference picture. Blocks of picture B can be predictively coded via spatial prediction or temporal prediction by referencing one or two previously coded reference pictures.

[0073] The video encoder (503) can perform coding operations according to a predetermined video coding technique or standard, such as ITU-T Rec. H265. In the operation, the video encoder (503) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Thus, the coded video data can follow the syntax specified by the video coding technique or standard used.

[0074] In an embodiment, the transmitter (540) may transmit additional data along with the encoded video. The source coder (530) may include such data as part of the encoded video sequence. The additional data may include other forms of redundant data, such as time / space / SNR enhancement layers, redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0075] Video can be captured with multiple source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often abbreviated as intra prediction) uses spatial correlations within a given picture, while inter-picture prediction uses (temporal or other) correlations between pictures. In the example, a specific picture being encoded / decoded, referred to as the current picture, is divided into blocks. If a block of the current picture is similar to a reference block of a previously encoded and still buffered reference picture in the video, the block of the current picture can be encoded by a vector referred to as MV. MV points to a reference block of the reference picture and may have three dimensions identifying the reference picture if multiple reference pictures are in use.

[0076] In some embodiments, a bidirectional prediction technique may be used in picture-to-picture prediction. According to the bidirectional prediction technique, two reference pictures, such as a first reference picture and a second reference picture, are used for the current picture of the video, in which both decoding order precedes (however, they may be past and future in display order, respectively). A block of the current picture may be coded by a first MV pointing to a first reference block of the first reference picture and a second MV pointing to a second reference block of the second reference picture. A block may be predicted by a combination of the first reference block and the second reference block.

[0077] In addition, merge mode technology can be used for inter-picture prediction to improve coding efficiency.

[0078] According to some embodiments of the present disclosure, predictions such as between-picture prediction and within-picture prediction are performed on a block basis. For example, according to the HEVC standard, pictures in a video picture sequence are divided into coding tree units for compression, and the CTUs of a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU includes three coding tree blocks (CTBs), which are one luma CTB and two chroma CTBs. Each CTU can recursively divide the quadtree into one or more coding units (CUs). For example, a 64x64 pixel CTU can be divided into one 64x64 pixel CU, four 32x32 pixel CUs, or sixteen 16x16 pixel CUs. In the example, each CU is analyzed to determine the prediction type for the CU, such as an inter-prediction type or an intra-prediction type. The CU is divided into one or more prediction units (PUs) based on temporal and / or spatial predictability. Typically, each PU includes a luminal prediction block (PB) and two chroma PBs. In the embodiment, prediction operations in coding (encoding / decoding) are performed on a prediction block basis. Using a luminal prediction block as an example of a prediction block, the prediction block contains a matrix of values ​​(e.g., luminal values) for pixels such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0079] FIG. 6 illustrates a video encoder (603) according to another embodiment of the present disclosure. The video encoder (603) is configured to receive a processing block (e.g., a prediction block) of sample values ​​within the current video picture in a sequence of video pictures, and to encode the processing block into a coded picture that is part of a coded video sequence. In the example, the video encoder (603) is used instead of the video encoder (303) of FIG. 3 example.

[0080] In the HEVC example, the video encoder (603) receives a matrix of sample values ​​for a processing block, such as a prediction block of 8x8 samples. The video encoder (603) determines whether the processing block is best coded using an intra-mode, inter-mode, or dual prediction mode, for example, using rate-distortion optimization. If the processing block is coded in intra-mode, the video encoder (603) may use an intra-prediction technique to encode the processing block into a coded picture, and if the processing block must be coded in inter-mode or dual prediction mode, the video encoder (603) may use an inter-prediction or dual-prediction technique, respectively, to encode the processing block into a coded picture. In a specific video coding technique, the merge mode may be an inter-picture prediction submode in which the MV is derived from one or more motion vector predictors without benefiting from the coded MV component outside the predictor. In a specific other video coding technique, there may be an MV component applicable to the target block. In the example, the video encoder (603) includes other components such as a mode determination module (not shown) for determining the mode of the processing block.

[0081] In FIG. 6, the video encoder (603) includes an inter-encoder (630), an intra-encoder (622), a residual calculator (623), a switch (626), a residual encoder (624), a general controller (621), and an entropy encoder (625) combined together as shown in FIG. 6.

[0082] The inter encoder (630) is configured to receive a sample of the current block (e.g., processing block), compare the block with one or more reference blocks of a reference picture (e.g., blocks of the previous and subsequent pictures), generate inter prediction information (e.g., description of redundancy information according to the inter encoding technique, MV, merge mode information), and calculate an inter prediction result (e.g., prediction block) based on the inter prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture that is decoded based on the encoded video information.

[0083] The intra-encoder (622) is configured to receive a sample of the current block (e.g., processing block), compare the block with a block already coded in the same picture in some cases, and generate quantized coefficients after conversion and, in some cases, also intra-prediction information (e.g., intra-prediction direction information according to one or more intra-encoding techniques). In the example, the intra-encoder (622) also calculates an intra-prediction result (e.g., prediction block) based on the intra-prediction information and reference block of the same picture.

[0084] A general controller (621) is configured to determine general control data and to control other components of the video encoder (603) based on the general control data. In an example, the general controller (621) determines the mode of the block and provides a control signal to the switch (626) based on the mode. For example, if the mode is intra mode, the general controller (621) controls the switch (626) to select an intra mode result to be used by the residual calculator (623), and controls the entropy encoder (625) to select intra prediction information and include intra prediction information in the bitstream; and if the mode is inter mode, the general controller (621) controls the switch (626) to select an inter prediction result to be used by the residual calculator (623), and controls the entropy encoder (625) to select inter prediction information and include inter prediction information in the bitstream.

[0085] The residual calculator (623) is configured to calculate the difference (residual data) between the received block and the prediction result selected from the intra-encoder (622) or the inter-encoder (630). The residual encoder (624) is configured to operate based on the residual data to encode the residual data to generate transformation coefficients. In the example, the residual encoder (624) is configured to transform the residual data from the spatial domain to the frequency domain and generate transformation coefficients. Then, the transformation coefficients are quantized to obtain quantized transformation coefficients. In various embodiments, the video encoder (603) also includes a residual decoder (628). The residual decoder (628) is configured to perform an inverse transformation and generate decoded residual data. The decoded residual data can be appropriately used by the intra-encoder (622) and the inter-encoder (630). For example, the inter encoder (630) can generate a decoded block based on decoded residual data and inter prediction information, and the intra encoder (622) can generate a decoded block based on decoded residual data and intra prediction information. The decoded block is appropriately processed to generate a decoded picture, and the decoded picture can be buffered in a memory circuit (not shown) and, in some examples, used as a reference picture.

[0086] The entropy encoder (625) is configured to format the bitstream to include the encoded block. The entropy encoder (625) is configured to include various information according to an appropriate standard, such as HEVC. In the example, the entropy encoder (625) is configured to include general control data, selected prediction information (e.g., intra prediction information or inter prediction information), residual information, and other appropriate information in the bitstream. According to the subject matter disclosed, note that when coding a block in an inter mode or a merged submode of a dual prediction mode, there is no residual information.

[0087] FIG. 7 illustrates a drawing of a video decoder (710) according to another embodiment of the present disclosure. The video decoder (710) is configured to receive a coded picture that is part of a coded video sequence and to decode the coded picture to generate a reconstructed picture. In the example, the video decoder (710) is used instead of the video decoder (310) of FIG. 3 example.

[0088] In FIG. 7, the video decoder (710) includes an entropy decoder (771), an inter decoder (780), a residual decoder (773), a reconstruction module (774), and an intra decoder (772) combined together as shown in FIG. 7.

[0089] The entropy decoder (771) may be configured to reconstruct from the coded picture a specific symbol representing a syntax element constituting the coded picture. Such a symbol may include, for example, the mode in which the block is coded (e.g., intra mode, inter mode, dual prediction mode, merged submode, or the latter two in other submodes), prediction information (e.g., intra prediction information or inter prediction information) capable of identifying a specific sample or metadata used for prediction by each of the intra decoder (772) or inter decoder (780), for example, residual information in the form of quantized transformation coefficients. In the example, if the prediction mode is an inter or dual prediction mode, the inter prediction information is provided to the inter decoder (780), and if the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder (772). The residual information may be inversely quantized and provided to the residual decoder (773).

[0090] The inter decoder (780) is configured to receive inter prediction information and generate an inter prediction result based on the inter prediction information.

[0091] The intra decoder (772) is configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0092] The residual decoder (773) is configured to perform inverse quantization to extract inverse quantized transform coefficients and to process the inverse quantized transform coefficients to transform the residual from the frequency domain to the space domain. The residual decoder (773) may also require specific control information (to include quantizer parameters (QP)), and such information may be provided by the entropy decoder (771) (the data path is not shown because it may be low-volume control information).

[0093] The reconstruction module (774) is configured to combine the residuals output by the char decoder (773) in the spatial domain and the prediction results (output by the inter or intra prediction module in some cases) to form a reconstruction block that may be part of a reconstruction picture that may be part of a reconstruction video. Note that other appropriate operations, such as deblocking operations, may be performed to improve visual quality.

[0094] It should be noted that the video encoder (303, 503, 603) and video decoder (310, 410, 710) can be implemented using any suitable technology. In an embodiment, the video encoder (303, 503, 603) and video decoder (310, 410, 710) can be implemented using one or more integrated circuits. In another embodiment, the video encoder (303, 503, 603) and video decoder (310, 410, 710) can be implemented using one or more processors that execute software instructions.

[0095] II. Video Coding for Machines

[0096] Videos and images can be consumed by humans for various purposes, such as entertainment and education. Therefore, video or image coding often utilizes the characteristics of the human visual system to achieve better compression efficiency while maintaining excellent subjective quality.

[0097] In recent years, with the increase in machine learning applications and the proliferation of sensors, many intelligent platforms have utilized video or images for machine vision tasks such as object detection, segmentation, or tracking. The method of encoding video or images for machine vision tasks can be an interesting and challenging problem. This led to the introduction of Video Coding for Machines (VCM). To achieve this goal, the international standards group MPEG created an ad-hoc group called "Video Coding for Machines (VCM)" to standardize related technologies for better interoperability between different devices.

[0098] FIG. 8 illustrates an exemplary VCM architecture according to an embodiment of the present disclosure. The output of the video decoding module of FIG. 8 is primarily for machine consumption, i.e., machine vision. However, in some cases, the output of the video decoding module may also be used for human vision, as shown in FIG. 8. Additionally, the exemplary VCM architecture may include an interface for a neural network (NN) module.

[0099] On the client (or decoder) side, video coding for a machine system can generally first perform video decoding to acquire a video sequence in the sample domain. Then, one or more machine operations may be performed to understand the video content of the video sequence. In some cases, the output of one or more machine operations may be labeling information of target objects, for example, in the form of a decoded image or a rectangular bounding box within the video sequence.

[0100] III. Multipurpose Video Coding

[0101] VVC was recently jointly developed by two international standards organizations, the ITU and ISO / IEC. One version of VVC was completed in July 2020 and is one of the most advanced video coding standards.

[0102] FIG. 9 illustrates an exemplary VVC encoder according to an embodiment of the present disclosure. The VVC encoder includes a transform and quantization (TR+Q) module, an entropy coding module, a motion estimation (ME) module, and a decoder. The decoder includes an inverse transform and inverse quantization (iTR+iQ) module, an intra prediction module, an inter prediction module, a combined prediction module, a deblocking filter (Deblk) module, a sample adaptive offset filter (SAO) module, an adaptive loop filter (ALF) module, and a buffer. The encoder receives a coding block (CB) of an input picture and outputs a coded bitstream and a reconstructed picture. Both the transform and quantization module and the inverse transform and inverse quantization module process a transform block (TB). Both the intra prediction module and the intra prediction module process a PB. The combined prediction module combines the inter prediction and the intra prediction.

[0103] For intra prediction, various coding tools may be included, for example, cross-component linear mode (CCLM), multiple reference line prediction (MRLP), intra sub-partitioning (ISP), and matrix-based intra prediction (MIP).

[0104] For inter prediction, for example, an affine motion model (AFFINE), a subblock-based temporal merging candidate (SbTMC), adaptive motion vector resolution (AMVR), geometric partition mode (GPM), combined intra / inter prediction (CIIP), merge mode with motion vector difference (MMVD), bi-predictive with CU weights (BCW), decoder motion vector refinement (DMVR), bi-directional optical flow (BDOF), and prediction refinement using optical flow (PROF) may be included in the set of coding tools.

[0105] For example, dependent quantization (DQ), multiple transform set (MTS), low frequency non-separable transform (LFNST), luma mapping and chroma scaling (LMCS), cross-component adaptive loop filter (CCALF), adaptive loop filter (ALF), sample adaptive offset filter (SAO), deblocking filter (Deblk), etc., may be included in other tools such as transforms, quantizations, and in-loop filters.

[0106] In a video coding system, some useful information that is not necessary for accurately decoding the video bitstream may be conveyed by one or more supplementary enhancement information (SEI) messages. For example, the decoder may ignore such information. Alternatively, the decoder may decode the SEI messages for use after decoding the video sequence. As an example, in virtual reality (VR) 360-degree video streaming, omnidirectional video can be packed into traditional 2D video for compression and transmission. After decoding the 2-D video, the system may need to process the packed 2-D video in several processing steps before the video can be viewed in a 3-D format. For example, the packed 2-D video is converted back into a spherical domain according to the packing format, such as equi-rectangular projection (ERP) or cube-map projection (CMP), to display the spherical signal in a central view containing the most relevant information of the video content. These steps are not related to accurately decoding the video itself, but are useful for displaying video content. Information such as packing methods (ERP, CMP, etc.) or information about the proposed view center of the 360-degree video can be transmitted via SEI messages. By receiving this information, the VR client system can process the decoded video content more efficiently.

[0107] IV. Signaling of Object Labeling Information

[0108] In some related video coding standards, such as H.264, HEVC, or the recently completed VVC standard, input video signals (or sequences) are processed as waveforms without understanding the semantics of the video content, such as how many people or objects are in the video and how they move around. On the other hand, machine vision tasks such as object detection, segmentation, or tracking can be designed to understand this type of information from video sequences. After decoding the video sequence, the client (or decoder) side can perform machine operations to obtain labeling information for target objects(s) in the image or video sequence.

[0109] Performing machine operations on the client (or decoder) side can result in the consumption of not only computation but also time to obtain the results of the machine operations. In some systems, these costs (e.g., computation and time consumption) are undesirable.

[0110] The present disclosure includes a method for transmitting detection or recognition information, such as labeling results, to a client (or decoder) side. The method included in the present disclosure is not limited to the VVC standard. The principles of the method may also be applied to other video coding standards developed by the Alliance for Open Media, such as H.264, HEVC, or AV1.

[0111] According to aspects of the present disclosure, labeling information, such as one or more bounding boxes, classes, and / or indices of objects for machine vision targets, may be determined at the encoder side rather than after decoding the video bitstream coded at the decoder side. The determined labeling information may be coded and transmitted to the decoder and / or client side in various ways. After decoding the labeling information, the labeling result may be applied directly to the decoded image or video sequence without further performing machine work to understand the video content. In an example, each of one or more bounding boxes may be applied to different target objects of the decoded image. In another example, the class and / or index of each target object may be displayed on the decoded image. In this way, machine work may be performed regardless of the resolution of the picture(s).

[0112] In an embodiment, the encoder system may apply a machine vision task module similar to the module illustrated in FIG. 9 to acquire relevant labeling information or to use other labeling information. For example, the output of the machine vision task may be in the form of a series of bounding boxes for each picture. Each bounding box may be represented by position and size information of each bounding box.

[0113] According to aspects of the present disclosure, labeling information, such as bounding box information for objects in each picture, may be conveyed in some additional information messages in the coded video bitstream. For example, the labeling information may be transmitted via SEI messages in the coded video bitstream. In other embodiments, the labeling information may also be provided separately, such as in a separate file or a plugin file associated with the coded video bitstream.

[0114] In some embodiments, the labeling information includes location information of one or more bounding boxes in each picture. Various reference points and / or size information may be used to indicate the location of the bounding boxes. In an embodiment, the location information of the bounding boxes may include the top-left location of the bounding box and the size of the bounding box.

[0115] [Table 1] shows an exemplary syntax table that can be used to signal labeling information including location information of a bounding box in the current picture.

[0116]

[0117] In [Table 1], the syntax element num_boxes indicates the total number of bounding boxes in the current picture. The syntax element box_loc_x[i] indicates the horizontal position of the top-left corner of the i-th bounding box relative to the top-left corner of the current picture in the luma sample. The syntax element box_loc_y[i] indicates the vertical position of the top-left corner of the i-th bounding box relative to the top-left corner of the current picture in the luma sample. The syntax element box_width[i] indicates the width of the i-th bounding box in the luma sample. The syntax element box_height[i] indicates the height of the i-th bounding box in the luma sample.

[0118] The representation of a bounding box may not be limited to the above format (top-left position + size). For example, top-left position + bottom-right position can also describe a bounding box.

[0119] Additionally, the position of the bounding box can be represented by a fixed location within the box. The top-left corner is one example, and other positions may also be utilized. For instance, the center position, width, and height of the bounding box can be used to represent the bounding box.

[0120] In one embodiment, the labeling information may include category information for each object in the current picture. The category information indicates the category to which the object within the bounding box belongs. For example, the category may be a person, a car, an airplane, etc. [Table 2] shows an exemplary syntax table that can be used to signal labeling information including location information of the bounding box in the current picture and category information of the object in the current picture.

[0121]

[0122] In [Table 2], the syntax element num_boxes indicates the total number of bounding boxes within the current picture. The syntax element category_id indicates the category to which the object of the i-th bounding box belongs. The syntax element box_loc_x[i] indicates the horizontal position of the top-left corner of the i-th bounding box relative to the top-left corner of the current picture in the luma sample. The syntax element box_loc_y[i] indicates the vertical position of the top-left corner of the i-th bounding box relative to the top-left corner of the current picture in the luma sample. The syntax element box_width[i] indicates the width of the i-th bounding box in the luma sample. The syntax element box_height[i] indicates the height of the i-th bounding box in the luma sample.

[0123] In one embodiment, in addition to category information, the labeling information may include identification information of one or more objects in the current picture. In one example, the labeling information may include identification information for each object in the current picture. The identification information may be used to identify objects in a video sequence. For example, in object tracking, the identification information may be used to represent the same object in a video sequence. [Table 3] shows a syntax table used to signal the location information of the bounding box of the current picture and the labeling information including category information and identification information of the objects in the current picture.

[0124]

[0125] In [Table 3], the syntax element num_boxes indicates the total number of bounding boxes within the current picture. The syntax element category_id indicates the category to which the object of the i-th bounding box belongs. The syntax element instance_id indicates the identification number for the object of the i-th bounding box. The syntax element box_loc_x[i] indicates the horizontal position of the top-left corner of the i-th bounding box relative to the top-left corner of the current picture in the luma sample. The syntax element box_loc_y[i] indicates the vertical position of the top-left corner of the i-th bounding box relative to the top-left corner of the current picture in the luma sample. The syntax element box_width[i] indicates the width of the i-th bounding box in the luma sample. The syntax element box_height[i] indicates the height of the i-th bounding box in the luma sample.

[0126] In some machine vision tasks, such as object tracking tasks, each object may appear in different pictures. To facilitate signaling of the same bounding box in different pictures, the same bounding box ID may be used to represent the same object. In subsequent picture(s), the absolute position and size of the bounding box may be signaled. In other embodiments, a relative change to the previous value of the previous picture may be used instead. [Table 4] shows an exemplary syntax table that can be used to signal labeling information including position offset information of the bounding box in the current picture. In [Table 4], the bounding box of the current picture is described by signaling the position of the top-left corner and the position of the bottom-right corner of the bounding box.

[0127]

[0128] [In Table 4], the syntax element num_boxes indicates the total number of bounding boxes in the current picture. The syntax element sign_tl_x[i] indicates the sign of the difference in the position of the i-th bounding box's top-left corner relative to the same bounding box in the previous picture in the luminance sample. The syntax element sign_tl_y[i] indicates the sign of the difference in the position of the i-th bounding box's top-left corner relative to the same bounding box in the previous picture in the luminance sample. The syntax element sign_br_x[i] indicates the sign of the difference in the position of the i-th bounding box's bottom-right corner relative to the same bounding box in the previous picture in the luminance sample. The syntax element sign_br_y[i] indicates the sign of the difference in the position of the i-th bounding box's bottom-right corner relative to the same bounding box in the previous picture in the luminance sample. The syntax element delta_box_tl_loc_x[i] indicates the absolute value of the difference in position between the horizontal top-left corner of the i-th bounding box and the identical bounding box of the previous picture in the luma sample. The syntax element delta_box_tl_loc_y[i] indicates the absolute value of the difference in position between the vertical top-left corner of the i-th bounding box and the identical bounding box of the previous picture in the luma sample. The syntax element delta_box_br_loc_x[i] indicates the absolute value of the difference in position between the horizontal bottom-right corner of the i-th bounding box and the identical bounding box of the previous picture in the luma sample. The syntax element delta_box_br_loc_y[i] indicates the absolute value of the difference in position between the vertical bottom-right corner of the i-th bounding box and the identical bounding box of the previous picture in the luma sample.

[0129] The syntax elements sign_tl_x[i], sign_tl_y[i], sign_br_x[i], and sign_br_y[i] can be set to 0 when there is no previous picture before the current picture. Alternatively, these syntax elements can be conditionally signaled only if there is a previous picture for the current picture in the decoding order or display order. If not signaled, these syntax elements can be inferred to be 0.

[0130] The variable PrevTopLeftBoxX[i] indicates the position of the horizontal top-left corner of the i-th bounding box in the previous picture prior to the current picture in the decoding or display order. The variable TopLeftBoxX[i] indicates the position of the horizontal top-left corner of the i-th bounding box in the current picture.

[0131] TopLeftBoxX[i] = PrevTopLeftBoxX[i] + sign_tl_x[i] * delta_box_tl_loc_x[i].

[0132] The variable PrevTopLeftBoxY[i] indicates the position of the vertical top-left corner of the i-th bounding box in the picture prior to the current picture in the decoding or display order. The variable TopLeftBoxY[i] indicates the position of the vertical top-left corner of the i-th bounding box in the current picture.

[0133] TopLeftBoxY[i] = PrevTopLeftBoxY[i] + sign_tl_y[i] * delta_box_tl_loc_y[i].

[0134] The variable PrevBotRightBoxX[i] indicates the position of the horizontal bottom-right corner of the i-th bounding box in the previous picture prior to the current picture in the decoding or display order. The variable BotRightBoxX[i] indicates the position of the horizontal bottom-right corner of the i-th bounding box in the current picture.

[0135] BotRightBoxX[i] = PrevBotRightBoxX[i] + sign_br_x[i] * delta_box_br_loc_x[i].

[0136] The variable PrevBotRightBoxY[i] indicates the vertical bottom-right corner position of the i-th bounding box in the previous picture prior to the current picture in the decoding or display order. The variable BotRightBoxY[i] indicates the vertical bottom-right corner position of the i-th bounding box in the current picture.

[0137] BotRightBoxY[i] = PrevBotRightBoxY[i] + sign_br_y[i] * delta_box_br_loc_y[i].

[0138] Variables PrevTopLeftBoxX[i], PrevTopLeftBoxY[i], PrevBotRightBoxX[i], and PrevBotRightBoxY[i] can be initialized to 0 when there is no previous picture before the current picture.

[0139] The variables PrevTopLeftBoxX[i], PrevTopLeftBoxY[i], PrevBotRightBoxX[i], and PrevBotRightBoxY[i] can be set to TopLeftBoxX[i], TopLeftBoxY[i], BotRightBoxX[i], and BotRightBoxY[i], respectively, after the decoding or display of the current picture is complete.

[0140] In the embodiment, the signaled syntax structure is similar to the syntax table above, but the bounding box of the current picture can be described by signaling its top-left corner position plus its width and height. In this case, the signaled delta value may be the delta value of the top-left corner position of the i-th bounding box, the delta values ​​of the width and height of the i-th bounding box, and their respective sign values.

[0141] In the example, the object's category information (e.g., category_id) and / or identification information (e.g., instance_id) may be included in [Table 4].

[0142] According to aspects of the present disclosure, the top-left and bottom-right positions of a bounding box may be placed within the current picture. However, in some cases, if an object no longer appears in the current picture, the bounding box should no longer be displayed in the current picture. In one embodiment, for the i-th bounding box that does not appear in the current picture, a predetermined position value, such as a position value outside the picture boundary, may be used. For example, if the picture size is 1920x1080, the top-left and bottom-right positions of the bounding box may be set to 2000 (greater than the picture width and height) to indicate that the object does not exist in the current picture and that the object's bounding box does not need to be displayed.

[0143] According to an aspect of the present disclosure, in addition to being included in metadata information (e.g., SEI messages) of a coded video bitstream, labeling information may be included in a metadata file (e.g., plugin files) separated from the coded video. Labeling information may be considered as metadata information and may be transmitted to a decoder and / or client side via system layer approaches such as the Real-Time Transport Protocol (RTP), ISO-based media format files, and dynamic adaptive streaming over hypertext transport protocol (DASH).

[0144] The client (or decoder) side may request this information from the system layer to enable labeling information on the decoded picture(s) when necessary.

[0145] In an embodiment, the client (or decoder) side may send a request message to the encoder side to request labeling information from the system layer.

[0146] In an embodiment, the client (or decoder) side can receive labeling information from the encoder side without sending a request message.

[0147] In an embodiment, the client can send a request message to the decoder to enable labeling information in the decoded picture(s).

[0148] In an embodiment, the decoder can activate labeling information in the decoder picture(s) without receiving a request message from the client.

[0149] V. Flowchart

[0150] FIG. 10 illustrates a schematic flowchart of an exemplary process (1000) according to an embodiment of the present disclosure. In various embodiments, the process (1000) is executed by a processing circuit such as a processing circuit of a terminal device (210, 220, 230, 240), a processing circuit that performs the function of a video encoder (303), a processing circuit that performs the function of a video decoder (310), a processing circuit that performs the function of a video decoder (410), a processing circuit that performs the function of an intra prediction module (452), a processing circuit that performs the function of a video encoder (503), a processing circuit (535) that performs the function of a predictor (535), a processing circuit that performs the function of an intra encoder (622), a processing circuit that performs the function of an intra decoder (772), etc. In some embodiments, the process (1000) is implemented as a software instruction, so when the processing circuit executes the software instruction, the processing circuit performs the process (1000).

[0151] The process (1000) can generally start at step (S1010), where the process (1000) receives metadata associated with a coded video bitstream. The metadata includes labeling information of one or more objects detected in a first picture coded in the coded video bitstream. Then, the process (1000) proceeds to step (S1020).

[0152] In step (S1020), the process (1000) decodes labeling information of one or more objects within the first picture coded in the coded video bitstream. Then, the process (1000) proceeds to step (S1030).

[0153] In step (S1030), the process (1000) applies labeling information to one or more objects of the first picture. After that, the process (1000) terminates.

[0154] In one example, a bounding box may be applied to an object detected in the first picture. In another example, the category to which the detected object belongs may be displayed in the first picture. In another example, the identification number of the detected object may be displayed in the first picture.

[0155] In one embodiment, metadata is included in the SEI message of the coded video bitstream.

[0156] In one embodiment, metadata is included in a file separate from the coded video bitstream.

[0157] In one embodiment, the labeling information indicates the total number of bounding boxes in the first picture and includes location information and size information for each bounding box, and each bounding box is associated with one or more objects in the first picture.

[0158] In one embodiment, the labeling information includes category information indicating a category for each of one or more objects.

[0159] In one embodiment, the labeling information includes identification information that identifies each of one or more objects in a video sequence.

[0160] In one embodiment, position information of one of the bounding boxes includes a position offset of one of the bounding boxes between the first picture and the second picture coded in the video bitstream.

[0161] In one embodiment, location information of one of the bounding boxes indicates a location outside the first picture for one of the bounding boxes based on an object associated with one of the bounding boxes that does not exist in the first picture.

[0162] In one embodiment, the process (1000) transmits a request to receive metadata associated with a coded video bitstream.

[0163] VI. Computer Systems

[0164] The technology described above may be implemented as computer software using computer-readable instructions and may be physically stored on one or more computer-readable media. For example, FIG. 11 illustrates a computer system (1100) suitable for implementing a specific embodiment of the disclosed subject matter.

[0165] Computer software may be coded using any suitable machine code or computer language capable of generating code containing instructions that can be directly executed through assembly, compilation, linking, or similar mechanisms by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0166] The instruction can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0167] The components illustrated in FIG. 11 for the computer system (1100) are essentially exemplary and are not intended to suggest any limitations on the scope of use or function of the computer software implementing the embodiments of the present disclosure. The configuration of the components should not be interpreted as having any dependencies or requirements related to any one or combination of the components illustrated in the exemplary embodiments of the computer system (1100).

[0168] The computer system (1100) may include a specific human interface input device. Such a human interface input device may respond to input by one or more human users through, for example, tactile input (e.g., key input, swipe, data glove motion), audio input (e.g., voice, clap), visual input (e.g., gesture), and olfactory input (not shown). The human interface device may also be used to capture specific media that are not directly related to conscious input by humans, such as audio (e.g., voice, music, ambient sound), images (e.g., scanned images, picture images obtained from a still image camera), and video (e.g., 2D video, 3D video including stereoscopic video).

[0169] The input human interface device may include one or more of a keyboard (1101), a mouse (1102), a trackpad (1103), a touch screen (1110), a data globe (not shown), a joystick (1105), a microphone (1106), a scanner (1107), and a camera (1108) (each only one of which is shown).

[0170] The computer system (1100) may also include a specific human interface output device. Such human interface output device may stimulate the senses of one or more human users, for example, through tactile output, sound, light and smell / taste. These human interface output devices may include a tactile output device (e.g., a touch screen (1110), a data globe (not shown), or a tactile feedback device that includes tactile feedback via a joystick (1105) but does not function as an input device), an audio output device (e.g., a speaker (1109), headphones (not shown)), a visual output device (e.g., a CRT screen, an LCD screen, a plasma screen, an OLED screen, each having or not having touch screen input capability, each having or not having tactile feedback capability—some of which may output two-dimensional visual output or three-dimensional or higher output through means such as stereoscopic output, virtual reality glasses (not shown), a holographic display and a smoke tank (not shown)) — and a printer (not shown). Such a visual output device (e.g., screen (1110)) can be connected to the system bus (1148) via a graphics adapter (1150).

[0171] The computer system (1100) may also include optical media or similar media (1121) including a CD / DVD ROM / RW (1120) having a human-accessible storage device and a CD / DVD, a thumb drive (1122), a removable hard drive or solid-state drive (1123), legacy magnetic media (not shown) such as tape and floppy disks, specialized ROM / ASIC / PLD-based devices (not shown) such as security dongles, etc.

[0172] A person skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not include transmission media, carrier waves, or other transient signals.

[0173] The computer system (1100) may also include a network interface (1154) for one or more communication networks (1155). One or more communication networks (1155) may be, for example, wireless, wired, optical. One or more communication networks (1155) may also be local, wide area, metropolitan, automotive and industrial, real-time, latency-tolerant, etc. Examples of one or more communication networks (1155) include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, automotive and industrial including CANBus, etc. A specific network generally requires an external network interface adapter attached to a specific general-purpose data port or peripheral device bus (1149) (e.g., the USB port of the computer system (1100)). Others are generally integrated into the core of the computer system (1100) by attachment to the system bus as described below (e.g., an Ethernet interface for a PC computer system or a cellular network interface for a smartphone computer system). Using one of these networks, the computer system (1100) can communicate with other entities. This communication may be unidirectional, receive-only (e.g., broadcast TV), unidirectional transmit-only (e.g., a CANbus for a specific CANbus device), or bidirectional, with other computer systems using, for example, local or wide-area digital networks. Specific protocols and protocol stacks may be used in each network and network interface as described above.

[0174] The aforementioned human interface device, human-accessible storage device, and network interface can be attached to the core (1140) of the computer system (1100).

[0175] The core (1140) may include one or more central processing units (CPUs) (1141), graphics processing units (GPUs) (1142), specialized programmable processing units in the form of field programmable gate areas (FPGAs) (1143), hardware accelerators (1144) for specific tasks, graphics adapters (1150), etc. These devices, along with read-only memory (ROM) (1145), random-access memory (1146), and internal mass storage (1147) such as internal non-user accessible hard drives, SSDs, etc., may be connected via a system bus (1148). In some computer systems, the system bus (1148) may be accessed in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (1148) or via a peripheral device bus (1149). In the example, the screen (1110) can be connected to a graphics adapter (1150). The architecture of the peripheral bus includes PCI, USB, etc.

[0176] The CPU (1141), GPU (1142), FPGA (1143), and accelerator (1144) can be combined to execute specific instructions that can construct the aforementioned computer code. This computer code may be stored in ROM (1145) or RAM (1146). Transitional data may be stored in RAM (1146), but permanent data may be stored, for example, in internal mass storage (1147). Fast storage and retrieval of all memory devices may be made possible through the use of cache memory that may be closely associated with one or more CPUs (1141), GPUs (1142), mass storage (1147), ROM (1145), RAM (1146), etc.

[0177] A computer-readable medium may have computer code for performing various computer-implemented operations. The medium and the computer code may be specifically designed and configured for the purposes of this disclosure, or they may be of a type well known and available to those skilled in the art of computer software.

[0178] For example and without limitation, a computer system (1100) having an architecture, in particular a core (1140), may provide function as a result of processor(s) (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software implemented on one or more types of computer-readable media. Such computer-readable media may be media associated with user-accessible mass storage as described above, or may be specific storage of the core (1140) having non-transient characteristics, such as internal mass storage (1147) or ROM (1145). Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the core (1140). The computer-readable media may include one or more memory devices or chips as needed. Software may enable the core (1140) and, in particular, the processor within it (including a CPU, GPU, FPGA, etc.) to execute a specific process or a specific part of a specific process described herein, including defining data structures stored in RAM (1146) and modifying such data structures according to a process defined in the software. Additionally or otherwise, the computer system may provide function as a result of logic fixed to or otherwise implemented in a circuit (e.g., an accelerator (1144)) that can operate instead of or with the software to execute a specific process or a specific part of a specific process described herein. References to software may include logic, and where appropriate, vice versa. References to computer-readable media may include circuits that store software for execution (such as an integrated circuit, IC), circuits that implement logic for execution, or both, where appropriate. The present disclosure includes any suitable combination of hardware and software.

[0179] Although the present disclosure describes some exemplary embodiments, there are modifications, permutations, and various alternative equivalents that fall within the scope of the disclosure. Accordingly, those skilled in the art will understand that numerous systems and methods can be devised that implement the principles of the present disclosure and thus fall within the spirit and scope of the invention, even though they are not explicitly shown or described in this specification. Explanation of the symbols

[0180] ALF (Adaptive Loop Filter): Adaptive Loop Filter AMVP (Advanced Motion Vector Prediction): Advanced Motion Vector Prediction APS (Adaptation Parameter Set): Adaptation Parameter Set ASIC (Application-Specific Integrated Circuit): Application-Specific Integrated Circuit ATMVP (Alternative / Advanced Temporal Motion Vector Prediction): Alternative / Advanced Temporal Motion Vector Prediction AV1(AOMedia Video 1): AOMedia Video 1 AV2(AOMedia Video 2): AOMedia Video 2 BMS (Benchmark Set): Benchmark Set BV(Block Vector): Block Vector CANBus (Controller Area Network Bus): Controller Area Network Bus CB (Coding Block): Coding Block CC-ALF (Cross-Component Adaptive Loop Filter): Cross-Component Adaptive Loop Filter CD (Compact Disc): Compact Disc CDEF (Constrained Directional Enhancement Filter): Constrained Directional Enhancement Filter CPR (Current Picture Referencing): Referencing the current picture CPU (Central Processing Unit): Central Processing Unit CRT (Cathode Ray Tube): Cathode Ray Tube CTB (Coding Tree Block): Coding Tree Block CTU (Coding Tree Unit): Coding Tree Unit CU (Coding Unit): Coding Unit DPB (Decoder Picture Buffer): Decoder Picture Buffer DPCM (Differential Pulse-Code Modulation): Differential Pulse-Code Modulation DPS (Decoding Parameter Set): Decoding parameter set DVD (Digital Video Disc): Digital Video Disc FPGA (Field Programmable Gate Area): Field Programmable Gate Area JCCR (Joint CbCr Residual Coding): Joint CbCr Residual Coding JVET (Joint Video Exploration Team): Joint Video Exploration Team GOP (Groups of Pictures): Picture Group GPU (Graphics Processing Unit): Graphics Processing Unit GSM (Global System for Mobile Communications): Global System for Mobile Communications HDR (High Dynamic Range): High Dynamic Range HEVC (High Efficiency Video Coding): High-efficiency video coding HRD (Hypothetical Reference Decoder): Virtual Reference Decoder IBC (Intra Block Copy): Intra Block Copy IC (Integrated Circuit): Integrated Circuit ISP (Intra Sub-Partitions): Intra sub-partitions JEM (Joint Exploration Model): Joint Exploration Model LAN (Local Area Network): Local Area Network LCD (Liquid-Crystal Display): Liquid Crystal Display LR (Loop Restoration Filter): Loop Restoration Filter LRU (Loop Restoration Unit): Loop Restoration Unit LTE (Long-Term Evolution): Long-Term Evolution MPM (Most Probable Mode): Most Probable Mode MV (Motion Vector): Motion Vector OLED (Organic Light-Emitting Diode): Organic Light-Emitting Diode PB(Prediction Block): Prediction Block PCI (Peripheral Component Interconnect): Peripheral Component Interconnect PDPC (Position Dependent Prediction Combination): Position-dependent prediction combination PLD (Programmable Logic Device): Programmable logic device PPS (Picture Parameter Set): Picture Parameter Set PU (Prediction Unit): Prediction Unit RAM (Random Access Memory): Random Access Memory ROM (Read-Only Memory): Read-only memory SAO (Sample Adaptive Offset): Sample Adaptive Offset SCC (Screen Content Coding): Screen Content Coding SDR (Standard Dynamic Range): Standard dynamic range SEI (Supplementary Enhancement Information): Supplementary Enhancement Information SNR (Signal Noise Ratio): Signal-noise ratio SPS (Sequence Parameter Set): Sequence parameter set SSD (Solid-state Drive): Solid-state drive TU (Transform Unit): Transform Unit USB (Universal Serial Bus): Universal Serial Bus VPS (Video Parameter Set): Video Parameter Set VUI (Video Usability Information): Video usability information VVC (Versatile Video Coding): Multipurpose Video Coding WAIP (Wide-Angle Intra Prediction): Wide-Angle Intra Prediction

Claims

Claim 1 A video coding method in a decoder comprising: receiving metadata associated with a coded video bitstream — said metadata includes labeling information of one or more objects detected in a first picture coded in said coded video bitstream —; decoding the labeling information of one or more objects of a first picture coded in said coded video bitstream; and applying said labeling information to one or more objects of said first picture, wherein the labeling information indicates the total number of bounding boxes of said first picture and includes position information and size information of each bounding box, said bounding box is associated with one of one or more objects of said first picture, and the position information of one of said bounding boxes includes a position offset of one of said bounding boxes between the first picture and the second picture coded in said video bitstream. Claim 2 A video coding method according to claim 1, wherein the metadata is included in a supplementary enhancement information (SEI) message of the coded video bitstream. Claim 3 A video coding method according to claim 1, wherein the metadata is included in a file separate from the coded video bitstream. Claim 4 A video coding method according to claim 1, further comprising the step of transmitting a request to receive metadata associated with the coded video bitstream. Claim 5 A device comprising a processing circuit configured to perform the method of any one of claims 1 to 4. Claim 6 A non-transient computer-readable storage medium, the non-transient computer-readable storage medium, which stores instructions, and when the instructions are executed by at least one processor, causes the at least one processor to perform the method of any one of claims 1 to 4. Claim 7 delete Claim 8 delete Claim 9 delete Claim 10 delete Claim 11 delete Claim 12 delete Claim 13 delete Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 delete

Images