Method and apparatus for video coding

Abstract

Aspects of the application include methods and apparatus for video coding. An apparatus includes processing circuitry that receives metadata associated with a coded video bitstream. The metadata includes labeling information for one or more objects detected in a first picture encoded into the coded video bitstream. The processing circuitry decodes the labeling information for the one or more objects in the first picture encoded into the coded video bitstream. The processing circuitry applies the labeling information to the one or more objects in the first picture.

Description

[0001] References merged

[0002] This application claims priority to U.S. Patent Application No. 17 / 459,753, filed August 27, 2021, entitled "Method and Apparatus for Video Coding," and U.S. Patent Application No. 63 / 135,530, filed January 8, 2021, entitled "Signaling of Objects for Machine Tasks." The entire disclosure of the earlier applications is incorporated herein by reference. Technical Field

[0003] The embodiments described in this application generally relate to video encoding and decoding. Background Technology

[0004] The background description provided herein is intended to present the background of this application. The work of the named inventors, the work described in this background section, and the content within the scope of the embodiments of this specification may not be considered prior art at the time of submission and are not expressly or impliedly acknowledged as prior art that is detrimental to this application.

[0005] Video encoding and decoding can be performed using inter-image prediction with motion compensation. Uncompressed digital video can comprise a series of images, each with a certain spatial dimension, such as 1920x1080 luminance samples and associated chrominance samples. The image sequence can have a fixed or variable image rate (commonly known as the frame rate), for example, 60 images per second or 60Hz. Uncompressed video has a large bit rate requirement. For example, a 1080p60 4:2:0 video (1920x1080 luminance sample resolution at 60Hz frame rate) with 8 bits per sample requires nearly 1.5 Gbit / s of bandwidth. Such a video, one hour long, would require more than 600 Gbytes of storage space.

[0006] One goal of video encoding and decoding is to reduce redundancy in the input video signal through compression. In some cases, compression can reduce bandwidth or memory requirements by at least two orders of magnitude. Lossless compression, lossy compression, or a combination thereof can be used. Lossless compression refers to techniques that reconstruct an accurate copy of the original signal from the compressed original signal. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough that the reconstructed signal can achieve its intended use. Lossy compression is widely used in the video industry. The amount of distortion that is permissible depends on the application; for example, some users of consumer live streaming applications can tolerate more distortion than users of television program applications. The achievable compression ratio reflects the allowable / tolerable distortion: the higher the compression ratio that can be produced.

[0007] Video encoders and decoders can utilize several major categories of techniques, including motion compensation, transform, quantization, and entropy coding.

[0008] Video encoding and decoding techniques may include those known as intra-frame coding. In intra-frame coding, the representation of sample values ​​does not require reference to samples or other data in a previously reconstructed reference image. In some video codecs, the image is spatially subdivided into sample blocks. When all sample blocks are encoded in intra-frame mode, the image can be an intra-frame image. Intra-frame images and their derivatives (such as independent decoder refresh images) can be used to reset the decoder state and thus can be used as the first image in the encoded video stream and video session, or as a still image. Samples of an intra-frame block can be transformed, and the transformation coefficients can be quantized before entropy coding. Intra-frame prediction can be a technique that minimizes the sample values ​​in the pre-transform domain. In some cases, the smaller the transformed DC value, the smaller the AC coefficient, and the fewer bits required after entropy coding to represent the block with a given quantization step size.

[0009] Traditional intra-frame coding (as known from, for example, MPEG-2 generation coding techniques) does not use intra-frame prediction. However, some newer video compression techniques include attempts, for example, to obtain surrounding sample data and / or metadata during the encoding and / or decoding of spatially adjacent blocks of data that are decoded first. This technique is hereby referred to as intra-frame prediction. Note that in at least some cases, intra-frame prediction uses only reference data from the current image being reconstructed (rather than the reference image).

[0010] There are many different forms of intra-prediction. When more than one such technique can be used in a given video coding technique, the techniques used can be encoded in the intra-prediction mode. In some cases, the mode can have sub-modes and / or parameters, which can be encoded separately or included in the mode codeword. Which codeword is used for a given combination of mode, sub-mode, and / or parameters can affect the coding efficiency gain through intra-prediction, and the entropy coding technique used to convert the codeword into a bitstream can also be considered.

[0011] A certain intra-frame prediction mode was introduced with H.264, refined in H.265, and further improved in new coding techniques such as Joint Exploration Model (JEM), Universal Video Coding (VVC), and Baseline Set (BMS). Prediction blocks can be formed using neighboring sample values, which belong to already available samples. The sample values ​​of neighboring samples are copied into the prediction block according to the direction. The direction information used can be encoded in the bitstream or predicted automatically.

[0012] See Figure 1A The lower right corner depicts a subset of nine known prediction directions out of the 33 possible prediction directions for H.265 (corresponding to 33 angular modes of 35 intra-frame modes). The point (101) where the arrows converge represents the sample being predicted. The arrows indicate the direction of the sample being predicted. For example, arrow (102) indicates that the prediction direction of sample (101) is from one or more samples to the upper right corner, at a 45-degree angle to the horizontal. Similarly, arrow (103) indicates that the prediction direction of sample (101) is from one or more samples to the lower left of sample (101), at a 22.5-degree angle to the horizontal.

[0013] Still referencing Figure 1A A 4×4 sampled square block (104) is shown in the upper left (indicated by bold dashed lines). The square block (104) comprises 16 samples, each labeled with "S" indicating its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from top to bottom) and the first sample in the X dimension (from left to right). Similarly, sample S44 in block (104) is the fourth sample in both the Y and X dimensions. Since the block size is 4×4 samples, S44 is located in the lower right corner. Reference samples following a similar numbering scheme are also shown. The reference samples are labeled with R, and their Y position (e.g., row index) and X position (column index) relative to block (104). In H.264 and H.265, the predicted samples are adjacent to the block being reconstructed; therefore, negative values ​​are not required.

[0014] Intra-frame image prediction works by copying reference sample values ​​from neighboring samples covered by the prediction direction indicated by the signal. For example, suppose the signaling included in the encoded video bitstream indicates that the prediction direction for the block is aligned with arrow (102)—that is, from one or more prediction samples to the upper right corner, at a 45-degree angle to the horizontal plane—to predict the samples. In this case, samples S41, S32, S23, and S14 are predicted using the same reference sample R05. Sample S44 is then predicted using reference sample R08.

[0015] In some cases, to calculate a reference sample, the values ​​of multiple reference samples can be combined, for example, through interpolation; especially when the direction is not divisible by 45 degrees.

[0016] With the development of video coding technology, the number of possible directions has been increasing. In H.264 (2003), nine different directions could be represented. This number increased to 33 in H.265 (2013), and JEM / VVC / BMS could support up to 65 directions at the time of its publication. Some experiments have been conducted to identify the most likely directions and to use certain techniques from entropy coding to represent those possible directions with a small number of bits, while accepting the unfavorable results of less likely directions. Furthermore, these directions can sometimes be predicted from the adjacent directions used by adjacent decoded blocks.

[0017] Figure 1B A schematic diagram (105) of the 65 intra-frame prediction directions of JEM is shown to illustrate the number of prediction directions increasing over time.

[0018] The mapping method for intra-prediction direction bits representing direction in an encoded video bitstream can vary across different video coding techniques. It can range from simple direct mappings from prediction direction to intra-prediction mode or codeword, to complex adaptive schemes involving most possible modes, and similar techniques. However, in all cases, there may be some directions that are statistically less likely to occur in the video content compared to others. Since the goal of video compression is to reduce redundancy, in well-functioning video coding techniques, those less likely directions will be represented by more bits than the more likely directions.

[0019] Motion compensation can be a lossy compression technique and can involve techniques where sample data blocks of a previously reconstructed image or a portion thereof (the reference image) are spatially shifted along the direction indicated by the motion vector (hereinafter referred to as MV) and used to predict a newly reconstructed image or image portion. In some cases, the reference image can be the same as the image currently being reconstructed. The MV can have two dimensions, X and Y, or three dimensions, with the third dimension being an indication of the reference image in use (the latter can indirectly be a temporal dimension).

[0020] In some video compression techniques, a motion vector (MV) applicable to a region of sample data can be predicted using other MVs, such as those related to sample data from another region spatially adjacent to the region currently being reconstructed and decoded before that MV. This substantially reduces the amount of data required to encode the MV, thereby eliminating redundancy and increasing compression strength. MV prediction can work effectively, for example, because when encoding the input video signal derived from the camera (called natural video), there is a statistical likelihood that a larger region than the region to which a single MV can be applied moves in a similar direction, and therefore, in some cases, similar motion vectors derived from MVs in neighboring regions can be used for prediction. This results in an MV found for a given region being similar or identical to the MV predicted from surrounding MVs, and after entropy encoding / decoding, it can be represented with fewer bits than would be needed if the MV were encoded directly. In some cases, MV prediction can be an example of lossless compression of the signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, MV prediction itself may be lossy, for example, due to rounding errors that arise when calculating predictions from several surrounding MVs.

[0021] H.265 / HEVC (ITU-T H.265 Recommendation, “High Efficiency Video Coding”, December 2016) describes various MV prediction mechanisms. Among the various MV prediction mechanisms provided by H.265, this application describes the technique referred to below as “spatial combining”.

[0022] Please refer to Figure 1CThe current block (111) may include samples discovered by the encoder during the motion search process, which can be predicted based on previous blocks of the same size that have generated spatial offsets. Alternatively, the MV can be derived from metadata associated with one or more reference images, rather than being directly encoded. For example, using the MV associated with any of the five surrounding samples A0, A1 and B0, B1, B2 (corresponding to 112 to 116 respectively), the MV is derived from the metadata of the nearest reference image (in decoding order). In H.265, MV prediction can use predictions from the same reference image that is also being used in adjacent blocks. Summary of the Invention

[0023] Various aspects of this application provide apparatus for video encoding / decoding. One apparatus includes processing circuitry that receives metadata associated with an encoded video stream. The metadata includes tagging information of one or more objects detected in a first image encoded in the encoded video stream. The processing circuitry decodes the tagging information of the one or more objects in the first image encoded in the encoded video stream. The processing circuitry applies the tagging information to the one or more objects in the first image.

[0024] In one embodiment, metadata is included in a Supplemental Enhancement Information (SEI) message within the encoded video stream.

[0025] In one embodiment, metadata is included in a file separate from the encoded video stream.

[0026] In one embodiment, the tagging information indicates the total number of bounding boxes in the first image and includes the location and size information of each bounding box, each bounding box being associated with one of one or more objects in the first image.

[0027] In one embodiment, the tagging information includes category information indicating the category of each of one or more objects.

[0028] In one embodiment, the tagging information includes identification information for each of one or more objects in the video sequence.

[0029] In one embodiment, the position information of a bounding box within a bounding box includes the positional offset of a bounding box within a bounding box between the first image and the second image encoded in the video stream.

[0030] In one embodiment, the location information of a bounding box within a bounding box indicates the location of a bounding box outside the first image based on the fact that the object associated with a bounding box within a bounding box does not exist in the first image.

[0031] In one embodiment, the processing circuitry sends a request to receive metadata associated with the encoded video stream.

[0032] This application provides methods for video encoding / decoding. The methods can perform any one or a combination of processes executed by means of a video encoding / decoding apparatus. In this method, metadata associated with an encoded video stream is received. The metadata includes tagging information of one or more objects detected in a first image encoded in the encoded video stream. The tagging information of the one or more objects in the first image encoded in the encoded video stream is decoded. The tagging information is then applied to the one or more objects in the first image.

[0033] This application also provides a non-volatile computer-readable medium storing instructions that, when executed by at least one processor, cause at least one processor to perform any one or a combination of methods for video encoding / decoding. Attached Figure Description

[0034] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0035] Figure 1A This is a schematic diagram of an exemplary subset of intra-frame prediction modes.

[0036] Figure 1B This is an illustration of an exemplary intra-frame prediction direction.

[0037] Figure 1C This is a schematic diagram of the current block and its surrounding space merge candidates in an example.

[0038] Figure 2 This is a simplified block diagram of a communication system according to an embodiment;

[0039] Figure 3 This is a simplified block diagram of a communication system according to an embodiment;

[0040] Figure 4 This is a simplified block diagram of the decoder according to an embodiment;

[0041] Figure 5 This is a simplified block diagram of the encoder according to an embodiment;

[0042] Figure 6 A block diagram of an encoder according to another embodiment is shown;

[0043] Figure 7 A block diagram of a decoder according to another embodiment is shown;

[0044] Figure 8An exemplary architecture for use in video codec (VCM) for machines is shown according to an embodiment;

[0045] Figure 9 An exemplary encoder used in a universal video codec (VVC) according to an embodiment is shown;

[0046] Figure 10 An exemplary flowchart according to an embodiment is shown; and

[0047] Figure 11 This is a schematic diagram of a computer system according to an embodiment. Detailed Implementation

[0048] I. Video Decoder and Encoder Systems

[0049] Figure 2 This is a simplified block diagram of a communication system (200) according to an embodiment disclosed in this application. The communication system (200) includes a plurality of terminal devices that can communicate with each other via, for example, a network (250). For example, the communication system (200) includes a first terminal device (210) and a second terminal device (220) interconnected via a network (250). Figure 2 In this embodiment, a first terminal device (210) and a second terminal device (220) perform unidirectional data transmission. For example, the first terminal device (210) may encode video data (e.g., a video image stream captured by the terminal device (210)) for transmission over a network (250) to another terminal device (220). The encoded video data is transmitted in the form of one or more encoded video streams. The second terminal device (220) may receive the encoded video data from the network (250), decode the encoded video data to recover the video data, and display video images based on the recovered video data. Unidirectional data transmission is common in applications such as media services.

[0050] In another embodiment, the communication system (200) includes a third terminal device (230) and a fourth terminal device (240) that perform bidirectional transmission of encoded video data, which may occur, for example, during a video conference. For bidirectional data transmission, each of the third terminal device (230) and the fourth terminal device (240) may encode video data (e.g., a video image stream captured by the terminal device) for transmission over a network (250) to the other terminal device. Each of the third terminal device (230) and the fourth terminal device (240) may also receive encoded video data transmitted by the other terminal device and may decode the encoded video data to recover the video data, and may display the video image on an accessible display device based on the recovered video data.

[0051] exist Figure 2 In the embodiments disclosed herein, the first terminal device (210), the second terminal device (220), the third terminal device (230), and the fourth terminal device (240) may be servers, personal computers, and smartphones, but the principles disclosed herein are not limited thereto. The embodiments disclosed herein are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network (250) refers to any number of networks that transmit encoded video data between the first terminal device (210), the second terminal device (220), the third terminal device (230), and the fourth terminal device (240), including, for example, wired (connected) and / or wireless communication networks. The communication network (250) may exchange data in circuit-switched and / or packet-switched channels. The network may include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this application, unless explained below, the architecture and topology of the network (250) may be irrelevant to the operation of this application.

[0052] As an example, Figure 3 The diagram illustrates the placement of a video encoder and a video decoder in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0053] The streaming system may include an acquisition subsystem (313) that may include a video source (301) such as a digital camera, which creates an uncompressed video image stream (302). In one embodiment, the video image stream (302) includes samples captured by a digital camera. The video image stream (302) is depicted as a thick line to emphasize the high data volume of the video image stream compared to encoded video data (304) (or encoded video bitstream). The video image stream (302) may be processed by an electronic device (320) that includes a video encoder (303) coupled to the video source (301). The video encoder (303) may include hardware, software, or a combination of hardware and software to implement or carry out aspects of the disclosed subject matter as described in more detail below. Compared to the video image stream (302), the encoded video data (304) (or the encoded video bitstream (304)) is depicted as a thin line to emphasize the lower data volume of the encoded video data (304) (or the encoded video bitstream (304)), which can be stored on a streaming server (305) for future use. One or more streaming client subsystems, such as Figure 3 Client subsystems (306) and (308) can access a streaming server (305) to retrieve copies (307) and (309) of encoded video data (304). Client subsystem (306) may include, for example, a video decoder (310) in an electronic device (330). The video decoder (310) decodes the incoming copy (307) of the encoded video data and produces an output video image stream (311) that can be displayed on a display (312) (e.g., a screen) or another presentation device (not depicted). In some streaming systems, the encoded video data (304), video data (307), and video data (309) (e.g., video stream) may be encoded according to certain video coding / compression standards. Examples of such standards include ITU-T H.265. In one embodiment, the video coding standard under development is informally referred to as Versatile Video Coding (VVC), and this application can be used in the context of the VVC standard.

[0054] It should be noted that the electronic devices (320) and (330) may include other components (not shown). For example, the electronic device (320) may include a video decoder (not shown), and the electronic device (330) may also include a video encoder (not shown).

[0055] Figure 4This is a block diagram of a video decoder (410) according to an embodiment disclosed in this application. The video decoder (410) may be disposed in an electronic device (430). The electronic device (430) may include a receiver (431) (e.g., receiving circuitry). The video decoder (410) may be used in place of Figure 3 The video decoder (310) in the embodiment.

[0056] The receiver (431) may receive one or more encoded video sequences to be decoded by the video decoder (410); in the same embodiment or another embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of the other encoded video sequences. The encoded video sequences may be received from a channel (401), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (431) may receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams that may be forwarded to their respective user entities (not indicated). The receiver (431) may separate the encoded video sequences from other data. To prevent network jitter, a buffer memory (415) may be coupled between the receiver (431) and the entropy decoder / parser (420) (hereinafter referred to as "parser (420)"). In some applications, the buffer memory (415) is part of the video decoder (410). In other cases, the buffer memory (415) may be located external to the video decoder (410) (not indicated). In other cases, an external buffer memory (not shown) may be provided for the video decoder (410) to prevent network jitter, for example, and another buffer memory (415) may be configured internally for, for example, handling broadcast timing. When the receiver (431) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous synchronization network, the buffer memory (415) may not be necessary, or it may be made smaller. Of course, for use on packet networks such as the Internet, a buffer memory (415) may also be required; this buffer memory may be relatively large and have an adaptive size, and may be at least partially implemented in the operating system or a similar component (not shown) external to the video decoder (410).

[0057] The video decoder (410) may include a parser (420) to reconstruct symbols (421) from the encoded video sequence. These symbols may include information for managing the operation of the video decoder (410) and potential information for controlling display devices (412), such as displays that are not part of the electronic device (430) but may be coupled to it. Figure 4As shown in the diagram. The control information for the display device may be a parameter set fragment (not indicated) of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI). The parser (420) may parse / decode the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (420) may extract a subgroup parameter set of at least one subgroup of pixels in the subgroup of pixels for use in the video decoder based on at least one parameter corresponding to a group. The subgroup may include a Group of Pictures (GOP), image, tile, slice, macroblock, coding unit (CU), block, transform unit (TU), prediction unit (PU), etc. The parser (320) can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0058] The parser (420) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (415) to create symbols (421).

[0059] Depending on the type of encoded video image or a portion thereof (e.g., inter-frame and intra-frame images, inter-frame and intra-frame blocks) and other factors, the reconstruction of the symbol (421) may involve multiple different units. Which units are involved and how they are involved can be controlled by the subgroup control information parsed from the encoded video sequence by the parser (420). For the sake of brevity, the flow of such subgroup control information between the parser (420) and the various units described below is not described.

[0060] In addition to the functional blocks already mentioned, the video decoder (410) can be conceptually subdivided into several functional units as described below. In practical embodiments operating under commercial constraints, many of these units interact closely with each other and can be integrated with one another. However, for the purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into the functional units described below.

[0061] The first unit is the scaler / inverse transform module (451). The scaler / inverse transform module (451) receives quantization transform coefficients as symbols (421) and control information from the parser (420), including the transform method used, block size, quantization factor, quantization scaling matrix, etc. The scaler / inverse transform module (451) can output a block containing sample values, which can be input into the aggregator (455).

[0062] In some cases, the output samples of the scaler / inverse transform module (451) may belong to intra-coded blocks; that is, blocks that do not use predictive information from previously reconstructed images but can use predictive information from previously reconstructed portions of the current image. Such predictive information may be provided by the intra-prediction module (452). In some cases, the intra-prediction module (452) uses reconstructed information extracted from the current image buffer (458) to generate surrounding blocks of the same size and shape as the block being reconstructed. For example, the current image buffer (458) buffers partially reconstructed and / or fully reconstructed current images. In some cases, the aggregator (455) adds the predictive information generated by the intra-prediction module (452) to the output sample information provided by the scaler / inverse transform module (451) based on each sample.

[0063] In other cases, the output samples of the scaler / inverse transform module (451) may belong to the inter-frame coding and latent motion compensation blocks. In this case, the motion compensation prediction module (453) can access the reference image memory (457) to extract samples for prediction. After motion compensation is performed on the extracted samples according to the symbols (421), these samples can be added by the aggregator (455) to the output of the scaler / inverse transform module (451) (referred to as residual samples or residual signals in this case) to generate output sample information. The motion compensation prediction module (453) can obtain the predicted samples from the address in the reference image memory (457) under motion vector control, and the motion vector is available to the motion compensation prediction module (453) in the form of the symbols (421), which, for example, include X, Y and reference image components. Motion compensation may also include interpolation of sample values ​​extracted from the reference image memory (457) when using subsample precise motion vectors, motion vector prediction mechanisms, etc.

[0064] The output samples of the aggregator (455) can be employed by various loop filtering techniques in the loop filter (456). Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video sequence (also referred to as the encoded video stream), and these parameters can be used as symbols (421) from the parser (420) in the loop filter (456). However, in other embodiments, the video compression techniques may also respond to metadata obtained during decoding of a previous (in decoding order) portion of the encoded image or encoded video sequence, and to previously reconstructed and loop-filtered sample values.

[0065] The output of the loop filter (456) can be a sample stream, which can be output to a display device (412) and stored in a reference image memory (457) for subsequent inter-frame image prediction.

[0066] Once fully reconstructed, some of the encoded images can be used as reference images for future predictions. For example, once the encoded image corresponding to the current image has been fully reconstructed and the encoded image (by, for example, the parser (420)) is identified as the reference image, the current image buffer (458) can become part of the reference image memory (457), and a new current image buffer can be reallocated before the reconstruction of subsequent encoded images begins.

[0067] The video decoder (410) can perform decoding operations according to a predetermined video compression technique, such as that specified in the ITU-T H.265 standard. The encoded video sequence may conform to the syntax specified by the video compression technique or standard in the sense that the encoded video sequence follows the syntax of the video compression technique or standard and the configuration file recorded in the video compression technique or standard. Specifically, the configuration file may select certain tools from all available tools in the video compression technique or standard as the only tools available under said configuration file. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum image size, maximum frame rate, maximum reconstruction sampling rate (measured in megasamples per second, for example), maximum reference image size, etc. In some cases, the limitations set by the hierarchy can be further limited by the Hypothetical Reference Decoder (HRD) specification and the metadata managed by the HRD buffer, which is represented by signals in the encoded video sequence.

[0068] In one embodiment, the receiver (431) may receive additional (redundant) data along with the encoded video. The additional data may be a portion of the encoded video sequence. The additional data may be used by the video decoder (410) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0069] Figure 5 This is a block diagram of a video encoder (503) according to an embodiment disclosed in this application. The video encoder (503) is disposed in an electronic device (520). The electronic device (520) includes a transmitter (540) (e.g., a transmission circuit). The video encoder (503) can be used to replace... Figure 3 The video encoder (303) in the embodiment.

[0070] The video encoder (503) can obtain data from the video source (501) (not...). Figure 5 In one embodiment, a portion of the electronic device (520) receives video samples, the video source being capable of capturing video images to be encoded by a video encoder (503). In another embodiment, the video source (501) is a portion of the electronic device (520).

[0071] A video source (501) can provide a sequence of source video samples encoded by a video encoder (503) in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCb, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (501) can be a storage device storing previously prepared video. In a video conferencing system, the video source (501) can be a camera that captures local image information as a video sequence. Video data can be provided as multiple individual images, which are given motion when viewed sequentially. The images themselves can be constructed as spatial pixel arrays, where each pixel can include one or more samples depending on the sampling structure, color space, etc., used. Those skilled in the art can easily understand the relationship between pixels and samples. The following focuses on describing samples.

[0072] According to an embodiment, the video encoder (503) can encode and compress images of a source video sequence into an encoded video sequence (543) in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate is a function of the controller (550). In some embodiments, the controller (550) controls and is functionally coupled to other functional units described below. For simplicity, coupling is not shown in the figures. Parameters set by the controller (550) may include rate control related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of pictures (GOP) layout, maximum allowable motion vector reference range, etc. The controller (550) may be used with other suitable functions related to the video encoder (503) optimized for a particular system design.

[0073] In some embodiments, the video encoder (503) operates within an encoding loop. As a simplified description, in one embodiment, the encoding loop may include a source encoder (530) (e.g., responsible for creating symbols, such as a symbol stream, based on the input image to be encoded and a reference image) and a (local) decoder (533) embedded within the video encoder (503). The decoder (533) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data (because in the video compression techniques considered in this application, any compression between the symbols and the encoded video stream is lossless). The reconstructed sample stream (sample data) is input to a reference image memory (534). Since decoding of the symbol stream produces bit-accurate results regardless of the decoder's location (local or remote), the contents of the reference image memory (534) also correspond bit-accurately between the local encoder and the remote encoder. In other words, the reference image samples "seen" by the encoder's prediction portion are exactly the same sample values ​​that the decoder will "see" when using the prediction during decoding. This fundamental principle of reference image synchronization (and the drift that occurs when synchronization cannot be maintained, for example, due to channel errors) is also used in some related techniques.

[0074] The operation of the “local” decoder (533) can be combined with, for example, the above-mentioned... Figure 4 The video decoder (410) is described in detail as the same as the "remote" decoder. However, a further brief reference is provided. Figure 4 When symbols are available and the entropy encoder (545) and parser (420) are able to encode / decode the symbols into an encoded video sequence without loss, the entropy decoding portion of the video decoder (410), including the buffer (415) and parser (420), may not be fully implemented in the local decoder (533).

[0075] It can be observed that any decoder technique other than parsing / entropy decoding present in the decoder must also exist in the corresponding encoder in essentially the same functional form. For this reason, this application focuses on decoder operation. The description of encoder techniques can be simplified because encoder techniques are inverses of the fully described decoder techniques. More detailed descriptions are only required in certain areas, and are provided below.

[0076] During operation, in some embodiments, the source encoder (530) may perform motion-compensated predictive coding. The motion-compensated predictive coding predictively encodes the input image with reference to one or more previously encoded images designated as "reference images" from the video sequence. In this manner, the encoding engine (532) encodes the differences between pixel blocks of the input image and pixel blocks of the reference image, which may be selected as a predictive reference for the input image.

[0077] The local video decoder (533) can decode encoded video data of an image that can be designated as a reference image, based on symbols created by the source encoder (530). The operation of the encoding engine (532) can be a lossy process. When the encoded video data can be decoded by the video decoder (533), Figure 5 When decoded at (not shown), the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (533) replicates the decoding process, which can be performed by the video decoder on the reference image, and allows the reconstructed reference image to be stored in a reference image cache (534). In this way, the video encoder (503) can locally store a copy of the reconstructed reference image that shares common content (no transmission errors) with the reconstructed reference image to be obtained by the remote video decoder.

[0078] The predictor (535) can perform a prediction search against the encoding engine (532). That is, for a new image to be encoded, the predictor (535) can search in the reference image memory (534) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can serve as appropriate prediction references for the new image. The predictor (535) can operate pixel-by-pixel based on the sample blocks to find suitable prediction references. In some cases, based on the search results obtained by the predictor (535), it can be determined that the input image may have prediction references obtained from multiple reference images stored in the reference image memory (534).

[0079] The controller (550) can manage the encoding operations of the source encoder (530), including, for example, setting parameters and subgroup parameters for encoding video data.

[0080] The outputs of all the above-mentioned functional units can be entropy encoded in the entropy encoder (545). The entropy encoder (545) performs lossless compression on the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc., thereby converting the symbols into an encoded video sequence.

[0081] The transmitter (540) may buffer one or more encoded video sequences created by the entropy encoder (545) in preparation for transmission via a communication channel (560), which may be a hardware / software link to a storage device that will store the encoded video data. The transmitter (540) may combine the encoded video data from the video encoder (503) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0082] The controller (550) manages the operation of the video encoder (503). During encoding, the controller (550) can assign a specific encoded image type to each encoded image, but this may affect the encoding techniques applicable to the corresponding images. For example, images can typically be assigned to any of the following image types:

[0083] Intraframe images (I-images) are images that can be encoded and decoded without using any other images in the sequence as prediction sources. Some video codecs allow different types of intraframe images, including, for example, Independent Decoder Refresh (IDR) images. Those skilled in the art are familiar with variations of I-images and their corresponding applications and characteristics.

[0084] A predictive image (P-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most one motion vector and reference index to predict sample values ​​for each block.

[0085] A bidirectional predictive image (B-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses up to two motion vectors and a reference index to predict sample values ​​for each block. Similarly, multiple predictive images can use more than two reference images and associated metadata to reconstruct a single block.

[0086] The source image is typically spatially subdivided into multiple sample blocks (e.g., 4×4, 8×8, 4×8, or 16×16 sample blocks), and each block is encoded sequentially. These blocks can be predictively coded with reference to other (already coded) blocks, determined based on the coding assignment of the corresponding images applied to the blocks. For example, blocks of an I-image can be non-predictively coded, or the blocks can be predictively coded (spatial or intra-frame prediction) with reference to already coded blocks of the same image. Pixel blocks of a P-image can be predictively coded with reference to a previously coded reference image via spatial prediction or temporal prediction. Blocks of a B-image can be predictively coded with reference to one or two previously coded reference images via spatial prediction or temporal prediction.

[0087] The video encoder (503) can perform encoding operations according to a predetermined video coding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (503) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0088] In one embodiment, the transmitter (540) may transmit additional data along with the encoded video while transmitting the encoded video. The source encoder (530) may transmit such data as part of the encoded video sequence. The additional data may include other forms of redundant data such as temporal / spatial / SNR enhancement layers, redundant images and slices, SEI messages, VUI parameter set fragments, etc.

[0089] The acquired video can be presented as multiple source images (video images) in a time series. Intra-frame image prediction (often simplified to intra-frame prediction) utilizes spatial correlations within a given image, while inter-frame image prediction utilizes (temporal or other) correlations between images. In one embodiment, a specific image being encoded / decoded is segmented into blocks, referred to as the current image. When a block in the current image resembles a reference block in a previously encoded and still buffered reference image in the video, the block in the current image can be encoded using a vector called a motion vector. This motion vector points to the reference block in the reference image, and when using multiple reference images, the motion vector may have a third dimension that identifies the reference image.

[0090] In some embodiments, bidirectional prediction techniques can be used in inter-frame image prediction. According to bidirectional prediction, two reference images are used, such as a first reference image and a second reference image, both preceding the current image in the video in decoding order (but possibly past and future in display order). A block in the current image can be encoded using a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. Specifically, the block can be predicted using a combination of the first and second reference blocks.

[0091] In addition, merging mode techniques can be used in inter-frame image prediction to improve coding efficiency.

[0092] According to some embodiments disclosed in this application, predictions such as inter-frame image prediction and intra-frame image prediction are performed on a block-by-block basis. For example, according to the HEVC standard, images in a video image sequence are segmented into coding tree units (CTUs) for compression. The CTUs in the images have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU includes three coding tree blocks (CTBs): one luma CTB and two chroma CTBs. Furthermore, each CTU can be further subdivided into one or more coding units (CUs) using a quadtree. For example, a 64×64 pixel CTU can be subdivided into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In one embodiment, each CU is analyzed to determine the prediction type used for the CU, such as inter-frame prediction or intra-frame prediction. Furthermore, depending on temporal and / or spatial predictability, the CU is divided into one or more prediction units (PUs). Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in encoding (encoding / decoding) are performed on a per-prediction-block basis. Taking a luma prediction block as an example, a prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0093] Figure 6 This is a diagram of a video encoder (603) according to another embodiment disclosed in this application. The video encoder (603) is used to receive processing blocks (e.g., prediction blocks) of sample values ​​within a current video image in a video image sequence, and to encode said processing blocks into an encoded image that is part of an encoded video sequence. In this embodiment, the video encoder (603) is used instead of Figure 3The video encoder (303) in the embodiment.

[0094] In the HEVC embodiment, the video encoder (603) receives a matrix of sample values ​​for a processing block, such as an 8×8 sample prediction block. The video encoder (603) uses, for example, rate-distortion (RD) optimization to determine whether to use an intra-frame mode, an inter-frame mode, or a bidirectional prediction mode to encode the processing block. When encoding the processing block in intra-frame mode, the video encoder (603) can use intra-frame prediction techniques to encode the processing block into the encoded image; and when encoding the processing block in inter-frame mode or bidirectional prediction mode, the video encoder (603) can use inter-frame prediction or bidirectional prediction techniques to encode the processing block into the encoded image, respectively. In some video coding techniques, the merging mode can be an inter-frame image prediction sub-mode, in which motion vectors are derived from one or more motion vector prediction values ​​without relying on encoded motion vector components outside the prediction values. In some other video coding techniques, motion vector components applicable to the subject block may exist. In one embodiment, the video encoder (603) includes other components, such as a mode decision module (not shown) for determining the processing block mode.

[0095] exist Figure 6 In one embodiment, the video encoder (603) includes, as shown below: Figure 6 The inter-frame encoder (630), intra-frame encoder (622), residual calculator (623), switch (626), residual encoder (624), general controller (621) and entropy encoder (625) are shown coupled together.

[0096] An inter-frame encoder (630) is configured to receive samples of the current block (e.g., the processing block), compare the block with one or more reference blocks in a reference image (e.g., blocks in previous and later images), generate inter-frame prediction information (e.g., a description of redundancy information, motion vectors, and merging mode information based on the inter-frame prediction information), and calculate inter-frame prediction results (e.g., predicted blocks) using any suitable technique based on the inter-frame prediction information. In some embodiments, the reference image is a decoded reference image based on encoded video information.

[0097] The intra encoder (622) is used to receive samples of the current block (e.g., the processed block), in some cases compare the block with an encoded block in the same image, generate quantization coefficients after transformation, and in some cases also generate intra prediction information (e.g., based on intra prediction direction information of one or more intra coding techniques). In one embodiment, the intra encoder (622) also calculates intra prediction results (e.g., predicted blocks) based on the intra prediction information and a reference block in the same image.

[0098] A general-purpose controller (621) determines general-purpose control data and controls other components of the video encoder (603) based on the general-purpose control data. In one embodiment, the general-purpose controller (621) determines the mode of a block and provides control signals to a switch (626) based on the mode. For example, when the mode is an intra-frame mode, the general-purpose controller (621) controls the switch (626) to select an intra-frame mode result for use by the residual calculator (623) and controls the entropy encoder (625) to select intra-frame prediction information and add the intra-frame prediction information to the bitstream; and when the mode is an inter-frame mode, the general-purpose controller (621) controls the switch (626) to select an inter-frame prediction result for use by the residual calculator (623) and controls the entropy encoder (625) to select inter-frame prediction information and add the inter-frame prediction information to the bitstream.

[0099] A residual calculator (623) is used to calculate the difference (residual data) between the received block and the prediction result selected from the intra encoder (622) or the inter encoder (630). A residual encoder (624) is used to operate on the residual data to encode the residual data to generate transform coefficients. In one embodiment, the residual encoder (624) is used to transform the residual data from the time domain to the frequency domain and generate transform coefficients. The transform coefficients are then quantized to obtain quantized transform coefficients. In various embodiments, the video encoder (603) also includes a residual decoder (628). The residual decoder (628) is used to perform an inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra encoder (622) and the inter encoder (630). For example, the inter encoder (630) can generate a decoded block based on the decoded residual data and inter-frame prediction information, and the intra encoder (622) can generate a decoded block based on the decoded residual data and intra-frame prediction information. The decoded blocks are processed appropriately to generate a decoded image, and in some embodiments, the decoded image may be buffered in a memory circuit (not shown) and used as a reference image.

[0100] An entropy encoder (625) is used to format the bitstream to produce encoded blocks. The entropy encoder (625) generates various information according to a suitable standard such as HEVC. In one embodiment, the entropy encoder (625) is used to obtain general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the bitstream. It should be noted that, according to the disclosed subject matter, residual information is not present when blocks are encoded in a merged sub-mode of inter-frame mode or bidirectional prediction mode.

[0101] Figure 7This is a diagram of a video decoder (710) according to another embodiment disclosed in this application. The video decoder (710) is used to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed image. In one embodiment, the video decoder (710) is used instead of Figure 3 The video decoder (310) in the embodiment.

[0102] exist Figure 7 In this embodiment, the video decoder (710) includes, as follows: Figure 7 The entropy decoder (771), inter-frame decoder (780), residual decoder (773), reconstruction module (774) and intra-frame decoder (772) are shown coupled together.

[0103] An entropy decoder (771) can be used to reconstruct certain symbols from an encoded image, these symbols representing the syntax elements constituting the encoded image. Such symbols may include, for example, a mode for encoding the block (e.g., intra-frame mode, inter-frame mode, bidirectional prediction mode, a merged sub-mode of the latter two, or another sub-mode), prediction information (e.g., intra-frame prediction information or inter-frame prediction information) that can respectively identify certain samples or metadata used by the intra-frame decoder (772) or the inter-frame decoder (780) for prediction, residual information in the form of, for example, quantized transform coefficients, and so on. In one embodiment, when the prediction mode is inter-frame or bidirectional prediction mode, inter-frame prediction information is provided to the inter-frame decoder (780); and when the prediction type is intra-frame prediction type, intra-frame prediction information is provided to the intra-frame decoder (772). Residual information may be provided to the residual decoder (773) via inverse quantization.

[0104] The inter-frame decoder (780) is used to receive inter-frame prediction information and generate inter-frame prediction results based on the inter-frame prediction information.

[0105] The intra-frame decoder (772) is used to receive intra-frame prediction information and generate prediction results based on the intra-frame prediction information.

[0106] The residual decoder (773) performs inverse quantization to extract the dequantized transform coefficients and processes the dequantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (773) may also require some control information (to obtain the quantizer parameter QP), and this information may be provided by the entropy decoder (771) (the data path is not indicated because this is only a small amount of control information).

[0107] The reconstruction module (774) is used to merge the residual output by the residual decoder (773) with the prediction result (which may be output by the inter-frame prediction module or the intra-frame prediction module) in the spatial domain to form a reconstructed block, which may be a part of a reconstructed image, which in turn may be a part of a reconstructed video. It should be noted that other suitable operations, such as deblocking, may be performed to improve visual quality.

[0108] It should be noted that any suitable technology can be used to implement the video encoder (303), video encoder (503), and video encoder (603), as well as the video decoder (310), video decoder (410), and video decoder (710). In one embodiment, one or more integrated circuits can be used to implement the video encoder (303), video encoder (503), and video encoder (603), as well as the video decoder (310), video decoder (410), and video decoder (710). In another embodiment, one or more processors executing software instructions can be used to implement the video encoder (303), video encoder (503), and video encoder (603), as well as the video decoder (310), video decoder (410), and video decoder (710). Various aspects of this application provide filtering techniques for video encoding / decoding.

[0109] II. Video encoding and decoding for machines

[0110] Videos or images can be consumed by humans for various purposes, such as entertainment and education. Therefore, video coding or image coding often leverages the characteristics of the human visual system to achieve better compression efficiency while maintaining good subjective quality.

[0111] In recent years, with the rise of machine learning applications and the deployment of numerous sensors, many intelligent platforms have utilized video or images for machine vision tasks, such as object detection, segmentation, or tracking. How to encode video or images for use in machine vision tasks is an interesting and challenging problem. This has led to research into video coding for machines (VCM). To achieve this goal, the international standards group MPEG created the ad hoc group "Video Coding for Machines (VCM)" to standardize related technologies, thereby enabling better interoperability between different devices.

[0112] Figure 8 An exemplary VCM architecture according to an embodiment of this application is shown. It should be noted that... Figure 8 The output of the video decoding module is primarily used by machines, i.e., machine vision. However, in some cases, the output of the video decoding module can also be used for human vision, such as... Figure 8 As indicated in the document. Additionally, the interface for the neural network (NN) module can be included in the exemplary VCM architecture.

[0113] A machine-defined video codec system from the client (or decoder) typically performs video decoding to first obtain a video sequence in a sample domain. Then, one or more machine tasks can be performed to understand the video content of the video sequence. In some cases, the output of one or more machine tasks may be tagging information for one or more target objects, for example, in the form of rectangular bounding boxes within the decoded image or video sequence.

[0114] III. Versatile Video Coding (VVC)

[0115] VVC is a video coding standard developed recently by two international standards organizations (i.e., ITU and ISO / IEC). A version of VVC was finalized in July 2020 and is one of the most advanced video coding standards.

[0116] Figure 9 An exemplary VVC encoder according to an embodiment of this application is illustrated. The VVC encoder includes a transform and quantization (TR+Q) module, an entropy coding module, a motion estimation (ME) module, and a decoder. The decoder includes an inverse transform and inverse quantization (iTR+iQ) module, an intra-frame prediction module, an inter-frame prediction module, a combined prediction module, a deblocking filter (Deblk) module, a sample adaptive offset filter (SAO) module, an adaptive loop filter (ALF) module, and a buffer. The encoder receives the coding block (CB) of the input image and outputs the encoded bitstream and the reconstructed image. The transform and quantization module, as well as the inverse transform and inverse quantization module, both process the transform block (TB). The intra-frame prediction module and the intra-frame prediction module both process the PB. The combined prediction module combines the inter-frame prediction and the intra-frame prediction.

[0117] For intra-frame prediction, various encoding and decoding tools can be included, such as cross-component linear mode (CCLM), multiple reference line prediction (MRLP), intra sub-partitioning (ISP), matrix-based intra-frame prediction (MIP), and / or similar tools.

[0118] For inter-frame prediction, a set of encoding and decoding tools can be included, such as affine motion model (AFFINE), subblock-based temporal merging candidates (SbTMC), adaptive motion vector resolution (AMVR), geometric partition mode (GPM), combined intra / inter prediction (CIIP), merge mode with motion vector difference (MMVD), bi-predictive with CU weights (BCW), decoder motion vector refinement (DMVR), bi-directional optical flow (BDOF), and / or prediction refinement using optical flow (PROF), etc.

[0119] It may also include other tools, such as transforms, quantizations, and in-loop filters. These tools may include, for example, dependent quantization (DQ), multiple transform set (MTS), low frequency non-separable transform (LFNST), luma mapping and chroma scaling (LMCS), cross-component adaptive loop filter (CCALF), adaptive loop filter (ALF), sample adaptive offset filter (SAO), and / or deblocking filter (Deblk).

[0120] In video codec systems, useful information that is not essential for correctly decoding the video stream can be transmitted, for example, through one or more supplementary enhancement information (SEI) messages. The decoder can, for example, ignore such information. Alternatively, the decoder can decode the SEI messages for use after decoding the video sequence. In one example, in a 360-degree virtual reality (VR) video stream, omnidirectional video can be packaged into 2-D conventional video for compression and transmission. After decoding the 2-D video, the system may need to process the packaged 2-D video in several processing steps before it can be viewed in 3-D format. For example, the packaged 2-D video may be converted back to a spherical domain according to the packaging format (e.g., equi-rectangular projection (ERP), cube-map projection (CMP), etc.), and the spherical signal may be displayed using a hypothetical central view containing the most relevant information of the video content. These steps are not related to correctly decoding the video itself, but are useful for displaying the video content. Information such as packaging methods (ERP, CMP, etc.) or suggested viewing centers for 360-degree videos can be sent via SEI messages. By receiving this information, the VR client system can process the decoded video content more efficiently.

[0121] IV. Signaling of object tagging information

[0122] In some relevant video codec standards (such as H.264, HEVC, or the recently completed VVC standard), the input video signal (or sequence) is treated as a waveform, without needing to understand the meaning of the video content, such as how many people or objects are in the video and how they move back and forth. On the other hand, machine vision tasks (such as object detection, segmentation, or tracking) can be designed to understand these types of information in video sequences. After decoding the video sequence, the client (or decoder) can perform machine tasks to obtain tagging information of one or more target objects in the image or video sequence.

[0123] Performing machine tasks on the client (or decoder) side not only consumes computing power but also incurs time costs in obtaining the results. In some systems, these costs (e.g., the consumption of computing power and time) are unnecessary.

[0124] This application includes methods for sending detection information or identification information (e.g., tagging results) to a client (or decoder). It should be noted that the methods included in this application are not limited to the VVC standard. The principles of these methods can also be applied to other video codec standards, such as H.264, HEVC, or AV1 developed by the Alliance for Open Media.

[0125] According to various aspects of this application, the labeling information of machine vision targets (e.g., one or more bounding boxes, categories, and / or indices of objects) can be determined at the encoder end, rather than after decoding the encoded video stream at the decoder end. The determined labeling information can be encoded in various ways and sent to the decoder and / or client. After decoding the labeling information, the labeling results can be directly applied to the decoded image or video sequence without further machine tasks to understand the video content. In one example, each bounding box in one or more bounding boxes can be applied to different target objects in the decoded image. In another example, the category and / or index of each target object can be displayed in the decoded image. Thus, machine tasks can be performed independently of the resolution of one or more images.

[0126] In one embodiment, the encoder system may employ similar machine vision task modules (e.g. Figure 9 The module shown can be used (or different machine vision task modules can be used to acquire relevant labeled information). For example, the output of a machine vision task can be in the form of a series of bounding boxes for each image. Each bounding box can be represented by the position and size information of the corresponding bounding box.

[0127] According to various aspects of this application, tagging information, such as bounding box information of objects in each image, can be conveyed through some side information messages in the encoded video stream. For example, tagging information can be sent via SEI messages in the encoded video stream. In another embodiment, tagging information can also be provided separately, for example, in a separate file or plugin file associated with the encoded video stream.

[0128] In some embodiments, the marking information includes positional information of one or more bounding boxes in each image. Various reference points and / or size information can be used to indicate the position of the bounding boxes. In one embodiment, the positional information of the bounding boxes may include the top-left position of the bounding box and the size of the bounding box.

[0129] Table 1 shows an exemplary syntax table that can be used to label information that uses signals to represent the location information of bounding boxes in the current image.

[0130] Table 1

[0131] In Table 1, the syntax element `num_boxes` indicates the total number of bounding boxes within the current image. The syntax element `box_loc_x[i]` indicates the horizontal position of the top-left corner of the i-th bounding box relative to the top-left corner of the current image, in units of brightness samples. The syntax element `box_loc_y[i]` indicates the vertical position of the top-left corner of the i-th bounding box relative to the top-left corner of the current image, in units of brightness samples. The syntax element `box_width[i]` indicates the width of the i-th bounding box, in units of brightness samples. The syntax element `box_height[i]` indicates the height of the i-th bounding box, in units of brightness samples.

[0132] Note that the representation of a bounding box is not limited to the above format (top left position + size). For example, top left position + bottom right position can also describe a bounding box.

[0133] Also note that the position of the bounding box can be represented by any fixed position of the box. The top left corner is an example, but other positions can also be used. For example, the center position of the bounding box, as well as its width and height, can be used to represent the bounding box.

[0134] In one embodiment, the tagging information may include category information for each object in the current image. The category information indicates the category to which the object within the bounding box belongs. For example, the category could be people, cars, and / or airplanes. Table 2 shows an exemplary syntax table that can be used to represent tagging information using signals, which includes the location information of the bounding boxes in the current image and the category information of the objects in the current image.

[0135] Table 2

[0136] In Table 2, the syntax element `num_boxes` indicates the total number of bounding boxes in the current image. The syntax element `category_id` indicates the category to which the object in the i-th bounding box belongs. The syntax element `box_loc_x[i]` indicates the horizontal position of the top-left corner of the i-th bounding box relative to the top-left corner of the current image, in units of brightness samples. The syntax element `box_loc_y[i]` indicates the vertical position of the top-left corner of the i-th bounding box relative to the top-left corner of the current image, in units of brightness samples. The syntax element `box_width[i]` indicates the width of the i-th bounding box, in units of brightness samples. The syntax element `box_height[i]` indicates the height of the i-th bounding box, in units of brightness samples.

[0137] In one embodiment, in addition to category information, the tagging information may also include identification information for one or more objects in the current image. In one example, the tagging information may include identification information for each object in the current image. The identification information can be used to identify objects in a video sequence. For example, in object tracking, the identification information can be used to represent the same object in a video sequence. Table 3 shows a syntax table for representing tagging information with signals, which includes the location information of bounding boxes in the current image and the category and identification information of objects in the current image.

[0138] Table 3

[0139] In Table 3, the syntax element `num_boxes` indicates the total number of bounding boxes in the current image. The syntax element `category_id` indicates the category to which the object in the i-th bounding box belongs. The syntax element `instance_id` represents the identifier number of the object in the i-th bounding box. The syntax element `box_loc_x[i]` indicates the horizontal position of the top-left corner of the i-th bounding box relative to the top-left corner of the current image, in units of brightness samples. The syntax element `box_loc_y[i]` indicates the vertical position of the top-left corner of the i-th bounding box relative to the top-left corner of the current image, in units of brightness samples. The syntax element `box_width[i]` indicates the width of the i-th bounding box, in units of brightness samples. The syntax element `box_height[i]` indicates the height of the i-th bounding box, in units of brightness samples.

[0140] In some machine vision tasks (such as object tracking), each object may appear in different images. To facilitate the representation of the same bounding box in different images using signals, the same bounding box ID can be used to represent the same object. In one or more subsequent images, the absolute position and size of the bounding box can be represented using signals. In another embodiment, a relative transformation relative to a previous value in the preceding image can be used instead. Table 4 shows an exemplary syntax table that can be used to represent labeling information using signals, including the positional offset information of the bounding box in the current image. In Table 4, the bounding box in the current image is described by representing the top-left and bottom-right positions of the bounding box using signals.

[0141] Table 4

[0142] In Table 4, the syntax element `num_boxes` indicates the total number of bounding boxes in the current image. The syntax element `sign_tl_x[i]` indicates the sign of the difference in the top-left horizontal position of the i-th bounding box relative to the same bounding box in the previous image, measured in brightness samples. The syntax element `sign_tl_y[i]` indicates the sign of the difference in the top-left vertical position of the i-th bounding box relative to the same bounding box in the previous image, measured in brightness samples. The syntax element `sign_br_x[i]` indicates the sign of the difference in the bottom-right horizontal position of the i-th bounding box relative to the same bounding box in the previous image, measured in brightness samples. The syntax element `sign_br_y[i]` indicates the sign of the difference in the bottom-right vertical position of the i-th bounding box relative to the same bounding box in the previous image, measured in brightness samples. The syntax element `delta_box_tl_loc_x[i]` indicates the absolute value of the difference in the top-left horizontal position of the i-th bounding box relative to the same bounding box in the previous image, measured in brightness samples. The syntax element `delta_box_tl_loc_y[i]` indicates the absolute value of the vertical position difference of the top-left corner of the i-th bounding box relative to the same bounding box in the previous image, in units of luminance samples. The syntax element `delta_box_br_loc_x[i]` indicates the absolute value of the horizontal position difference of the bottom-right corner of the i-th bounding box relative to the same bounding box in the previous image, in units of luminance samples. The syntax element `delta_box_br_loc_y[i]` indicates the absolute value of the vertical position difference of the bottom-right corner of the i-th bounding box relative to the same bounding box in the previous image, in units of luminance samples.

[0143] If no previous image exists before the current image, the syntax elements sign_tl_x[i], sign_tl_y[i], sign_br_x[i], and sign_br_y[i] can be set to 0. Alternatively, these syntax elements can be conditionally signaled only if the current image has a previous image in the decoding or display order. When not signaled, these syntax elements can be inferred to be 0.

[0144] The variable PrevTopLeftBoxX[i] indicates the horizontal position of the top-left corner of the i-th bounding box in the previous image, according to the decoding or display order. The variable TopLeftBoxX[i] indicates the horizontal position of the top-left corner of the i-th bounding box in the current image.

[0145] TopLeftBoxX[i] = PrevTopLeftBoxX[i] + sign_tl_x[i] delta_box_tl_loc_x[i].

[0146] The variable PrevTopLeftBoxY[i] indicates the top-left vertical position of the i-th bounding box in the previous image, according to the decoding or display order. The variable TopLeftBoxY[i] indicates the top-left vertical position of the i-th bounding box in the current image.

[0147] TopLeftBoxY[i] = PrevTopLeftBoxY[i] + sign_tl_y[i] delta_box_tl_loc_y[i].

[0148] The variable PrevBotRightBoxX[i] indicates the bottom-right horizontal position of the i-th bounding box in the previous image, according to the decoding or display order. The variable BotRightBoxX[i] indicates the bottom-right horizontal position of the i-th bounding box in the current image.

[0149] BotRightBoxX[i] = PrevBotRightBoxX[i] + sign_br_x[i] delta_box_br_loc_x[i].

[0150] The variable PrevBotRightBoxY[i] indicates the vertical position of the bottom right corner of the i-th bounding box in the previous image, according to the decoding or display order. The variable BotRightBoxY[i] indicates the vertical position of the bottom right corner of the i-th bounding box in the current image.

[0151] BotRightBoxY[i] = PrevBotRightBoxY[i] + sign_br_y[i] delta_box_br_loc_y[i].

[0152] When there is no previous image before the current image, the variables PrevTopLeftBoxX[i], PrevTopLeftBoxY[i], PrevBotRightBoxX[i], and PrevBotRightBoxY[i] can be initialized to 0.

[0153] After decoding or displaying the current image, the variables PrevTopLeftBoxX[i], PrevTopLeftBoxY[i], PrevBotRightBoxX[i], and PrevBotRightBoxY[i] can be set to equal TopLeftBoxX[i], TopLeftBoxY[i], BotRightBoxX[i], and BotRightBoxY[i], respectively.

[0154] In one embodiment, the syntax structure represented by signals is similar to the syntax table above, but the bounding boxes in the current image can be described by representing their top-left corner position plus their width and height using signals. In this case, the incremental value to be represented by signals can be the incremental value of the top-left corner position of the i-th bounding box, the incremental values ​​of the width and height of the i-th bounding box, and the corresponding positive or negative sign value.

[0155] In one embodiment, the object’s category information (e.g., category_id) and / or identification information (e.g., instance_id) may be included in Table 4.

[0156] According to various aspects of this application, the upper-left and lower-right positions of the bounding box can be placed within the current image. However, in some cases, when an object no longer appears in the current image, the bounding box should no longer be displayed in the current image. In one embodiment, for the i-th bounding box that is not displayed in the current image, a predetermined position value can be used, such as a position value outside the image boundary. For example, if the image size is 1920×1080, the upper-left and lower-right positions of the bounding box can be set to 2000 (greater than the image width and height) to indicate that the object does not exist in the current image and that the object's bounding box does not need to be displayed.

[0157] According to various aspects of this application, in addition to being included in the metadata information (e.g., SEI messages) of the encoded video stream, tagging information can be included in a metadata file (e.g., a plugin file) separate from the encoded video stream. Tagging information can be considered metadata information and is transmitted to the decoder and / or client via system-level means (e.g., real-time transport protocol (RTP), ISO basic media format files, and dynamic adaptive streaming over hypertext transport protocol (DASH)).

[0158] When needed, the client (or decoder) can request such information from the system layer to enable tagging information in one or more decoded images.

[0159] In one embodiment, the client (or decoder) may send a request message to the encoder to request tagging information from the system layer.

[0160] In one embodiment, the client (or decoder) can receive tagging information from the encoder without sending a request message.

[0161] In one embodiment, the client may send a request message to the decoder to enable tagging information in one or more decoded images.

[0162] In one embodiment, the decoder can enable tagging information in one or more decoder images without receiving a request message from the client.

[0163] V. Flowchart

[0164] Figure 10 An overview flowchart of an exemplary process (1000) according to an embodiment of this application is shown. In various embodiments, the process (1000) is executed by processing circuitry, such as processing circuitry in terminal devices (210), (220), (230), and (240), processing circuitry performing the functions of a video encoder (303), processing circuitry performing the functions of a video decoder (310), processing circuitry performing the functions of a video decoder (410), processing circuitry performing the functions of an intra-frame prediction module (452), processing circuitry performing the functions of a video encoder (503), processing circuitry performing the functions of a predictor (535), processing circuitry performing the functions of an intra-frame encoder (622), processing circuitry performing the functions of an intra-frame decoder (772), and so on. In some embodiments, the process (1000) is implemented as software instructions, so that the processing circuitry executes the process (1000) when the software instructions are executed.

[0165] The process (1000) can typically begin at step (S1010), where the process (1000) receives metadata associated with the encoded video stream. This metadata includes tagging information of one or more objects detected in the first image encoded into the encoded video stream. The process (1000) then proceeds to step (S1020).

[0166] At step (S1020), process (1000) decodes the tagging information of one or more objects in the first image encoded into the encoded video stream. Then, process (1000) proceeds to step (S1030).

[0167] In step (S1030), process (1000) applies the marking information to one or more objects in the first image. Then, process (1000) ends.

[0168] In one example, a bounding box can be applied to the detected objects in the first image. In another example, the category to which the detected objects belong can be displayed in the first image. In yet another example, the identification number of the detected objects can be displayed in the first image.

[0169] In one embodiment, the metadata is included in the SEI message within the encoded video stream.

[0170] In one embodiment, the metadata is included in a file separate from the encoded video stream.

[0171] In one embodiment, the marking information indicates the total number of bounding boxes in the first image and includes positional and size information for each bounding box, each bounding box being associated with one of one or more objects in the first image.

[0172] In one embodiment, the tagging information includes category information, which indicates the category of each of one or more objects.

[0173] In one embodiment, the tagging information includes identification information used to identify each of one or more objects in the video sequence.

[0174] In one embodiment, the position information of a bounding box within a bounding box includes the position offset between the position of a bounding box within a bounding box in the first image and its position in the second image encoded into the video stream.

[0175] In one embodiment, for a bounding box among the bounding boxes, the location information of the bounding box indicates a location outside the first image, based on the case that the object associated with the bounding box does not exist in the first image.

[0176] In one embodiment, the process (1000) sends a request to receive metadata associated with the encoded video stream.

[0177] VI. Computer System

[0178] The techniques described above can be implemented as computer software that uses computer-readable instructions and is physically stored on one or more computer-readable media. For example, Figure 11 A computer system (1100) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0179] Computer software can be encoded using any suitable machine code or computer language, which can be assembled, compiled, linked, or similarly used to create code that includes instructions that can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), or through interpretation, microcode execution, or other means.

[0180] These instructions can be executed on various types of computers or their components, including, for example, personal computers, tablets, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0181] Figure 11 The components shown for the computer system (1100) are exemplary in nature and are not intended to impose any limitation on the scope or functionality of the computer software used to implement embodiments of this application. The configuration of the components should also not be construed as having any dependency or requirement on any component or combination thereof illustrated in the exemplary embodiments of the computer system (1100).

[0182] The computer system (1100) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, tapping), visual input (e.g., gestures), and olfactory input (not depicted). Human-machine interface devices may also be used to acquire certain media that are not necessarily directly related to conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0183] The input human-machine interface device may include one or more of the following (only one of each is depicted): keyboard (1101), mouse (1102), touchpad (1103), touch screen (1110), data glove (not shown), joystick (1105), microphone (1106), scanner (1107), camera (1108).

[0184] The computer system (1100) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback via a touchscreen (1110), data gloves (not shown), or joystick (1105), but tactile feedback devices that are not used as input devices may also exist), audio output devices (e.g., speakers (1109), headphones (not depicted)), visual output devices (e.g., for screens (1110) including CRT screens, LCD screens, plasma screens, OLED screens, each screen may or may not have touchscreen input capability, each screen may or may not have tactile feedback capability, some of which are capable of outputting two-dimensional or more than three-dimensional visual output by means of, for example, stereoscopic output; virtual reality glasses (not depicted), holographic displays and smoke canisters (not depicted), and printers (not depicted). These visual output devices (e.g., screens (1110)) may be connected to the system bus (1148) via a graphics adapter (1150).

[0185] The computer system (1100) may also include human-accessible storage devices and their associated media, such as optical media including media (1121) such as CD / DVD ROM / RW (1120) with CD / DVD, thumb drives (1122), removable hard disk drives or solid-state drives (1123), conventional magnetic media such as magnetic tapes and floppy disks (not depicted), devices based on dedicated ROM / ASIC / PLD such as security dongles (not depicted), etc.

[0186] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other volatile signals.

[0187] The computer system (1100) may also include an interface (1154) to one or more communication networks (1155). The network may be, for example, wireless, wired, or optical. The one or more communication networks (1155) may further be local, wide area, metropolitan area, vehicular, and industrial, real-time, latency-tolerant, etc. Examples of one or more communication networks (1155) include local area networks (e.g., Ethernet), wireless LANs, cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), cable or wireless wide area digital television networks (including cable television, satellite television, and terrestrial broadcast television), vehicle and industrial networks (including CANBus), etc. Some networks typically require external network interface adapters attached to certain general-purpose data ports or peripheral buses (1149) (e.g., USB ports of the computer system (1100)); other networks are typically integrated into the core of the computer system (1100) by attaching to system buses as described below (e.g., Ethernet interfaces integrated into PC computer systems or cellular network interfaces integrated into smartphone computer systems). Using any of these networks, the computer system (1100) can communicate with other entities. This communication can be unidirectional, receive-only (e.g., broadcasting TV), send-only (e.g., CANbus to certain CANbus devices), or bidirectional (e.g., to other computer systems using a local area digital network or a wide area digital network). Certain protocols and protocol stacks can be used on each of those networks and network interfaces as described above.

[0188] The aforementioned human-machine interface device, human-accessible storage device, and network interface can be attached to the kernel (1140) of the computer system (1100).

[0189] The kernel (1140) may include one or more central processing units (CPU) (1141), graphics processing units (GPUs) (1142), dedicated programmable processing units (1143) in the form of field-programmable gate areas (FPGAs), hardware accelerators (1144) for certain tasks, graphics adapters (1150), etc. These devices, along with read-only memory (ROM) (1145), random access memory (1146), internal mass storage devices such as internal non-user-accessible hard disk drives, SSDs, etc. (1147), can be connected via the system bus (1148). In some computer systems, the system bus (1148) may be accessed as one or more physical plugs to allow for expansion by additional CPUs, GPUs, etc. Peripheral devices may be directly attached to the kernel's system bus (1148) or attached to the system bus (1148) via a peripheral bus (1149). In one example, a screen (1110) may be connected to a graphics adapter (1150). Peripheral bus architectures include PCI, USB, etc.

[0190] The CPU (1141), GPU (1142), FPGA (1143), and accelerator (1144) can execute certain instructions, and combinations of these instructions can constitute the aforementioned computer code. This computer code can be stored in ROM (1145) or RAM (1146). Transient data can also be stored in RAM (1146), while permanent data can be stored, for example, in an internal mass storage device (1147). Fast storage and retrieval of any memory device can be enabled by using a cache memory, which can be closely associated with one or more CPUs (1141), GPUs (1142), mass storage devices (1147), ROMs (1145), RAMs (1146), etc.

[0191] Computer-readable media may contain computer code for performing operations of various computer implementations. The media and computer code may be those specifically designed and constructed for the purposes of this application, or they may be of types known and available to those skilled in the art of computer software.

[0192] By way of example and not limitation, a computer system having an architecture (1100) and, in particular, a kernel (1140) can provide functionality as a result of one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software implemented in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage devices as described above, as well as certain storage devices of the kernel (1140) having non-volatile properties (e.g., internal kernel mass storage (1147) or ROM (1145)). Software implementing various embodiments of this application can be stored in such devices and executed by the kernel (1140). Depending on specific needs, the computer-readable media may include one or more memory devices or chips. The software can cause the kernel (1140) and, in particular, the processors therein (including CPUs, GPUs, FPGAs, etc.) to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM (1146) and modifying such data structures according to software-defined processes. Alternatively or as an alternative, the computer system may provide functionality (e.g., an accelerator (1144)) as a result of hard-wired logic or otherwise implemented in circuitry, which may replace or operate with software to perform a particular process or a particular portion of a particular process described herein. References to software may, where appropriate, include logic, and vice versa. References to computer-readable media may, where appropriate, include circuitry storing software for execution (e.g., an integrated circuit (IC)), circuitry containing logic for execution, or both. This application includes any suitable combination of hardware and software.

[0193] While this application has described several exemplary embodiments, various modifications, substitutions, and alternatives to the embodiments are within the scope of this application. Therefore, it should be understood that those skilled in the art can design various systems and methods that, while not explicitly shown or described herein, embody the principles of this application and are thus within its spirit and scope.

[0194] Appendix A: Acronyms ALF: Adaptive Loop Filter AMVP: Advanced Motion Vector Prediction APS: Adaptive parameter set ASIC: Application-Specific Integrated Circuit ATMVP: Alternative Temporal Motion Vector Prediction AV1: AOMedia Video 1 AV2: AOMedia Video 2 BMS: benchmark set BV: block vector CANBus: Controller Area Network Bus CB: coding block CC-ALF: Cross-Component-ALF Adaptive Loop Filter CD: Compact Disc CDEF: Constrained Directional Enhanced Filter CPR: Current picture reference CPU: Central Processing Unit CRT: Cathode Ray Tube CTB: Coding Tree Block CTU: Coding Tree Unit CU: Coding Unit DPB: decoded picture buffer DPCM: Differential Pulse Code Modulation DPS: decoding parameter set DVD: Digital Video Disc FPGA: Field Programmable Gate Array JCCR: Coding of Chroma Residual (Joint CbCr Residual Encoding and Decoding) JVET: Joint Video Explore Team GOP: Groups of Pictures GPU: Graphics Processing Unit GSM: Global System for Mobile Communication HDR: High-Dynamic Range HEVC: High Efficiency Video Coding HRD: Hypothetical Reference Decoder IBC: Intra Block Copy IC: Integrated Circuit ISP: intra sub-partition JEM: Joint Exploration Model LAN: Local Area Network LCD: Liquid Crystal Display LR: Loop recovering filter LRU: Loop Recovering Unit LTE: Long-Term Evolution MPM: Most Probable Mode MV: Motion Vector OLED: Organic Light-Emitting Diode PB: Prediction Block PCI: Peripheral Component Interconnect PDPC: Position Dependent Intra Prediction Combination PLD: Programmable Logic Device PPS: Picture Parameter Set PU: Prediction Unit RAM: Random Access Memory ROM: Read-Only Memory SAO: Sample Adaptive Offset SCC: Screen Content Coding SDR: Standard Dynamic Range SEI: Supplementary Enhancement Information SNR: Signal Noise Ratio SPS: Sequence Parameter Set SSD: solid-state drive TU: Transform Unit USB: Universal Serial Bus VPS: video parameter set VUI: Video Usability Information VVC: Universal Video Coding WAIP: Wide-angle intra prediction

Claims

1. A video encoding / decoding method at a decoder, characterized in that, include: Receive metadata associated with an encoded video stream, the metadata including tagging information of one or more objects detected in a first image encoded into the encoded video stream; wherein the tagging information includes the total number of bounding boxes in the first image, and includes positional and size information for each bounding box, each bounding box being associated with one of the one or more objects in the first image; the positional information of a bounding box within a bounding box includes a positional offset between the position of the bounding box within the bounding box in the first image and its position in a second image encoded into the video stream; Decoding the tagging information of one or more objects in the first image encoded into the encoded video stream; and The marking information is applied to the one or more objects in the first image by displaying the marking information of the one or more objects in the first image.

2. The method according to claim 1, characterized in that, The metadata is included in a supplemental enhancement information message in the encoded video stream.

3. The method according to claim 1, characterized in that, The metadata is included in a file separate from the encoded video stream.

4. The method according to claim 1, characterized in that, The tagging information includes category information, which indicates the category of each of the one or more objects.

5. The method according to claim 1, characterized in that, The tagging information includes identification information, which identifies each of the one or more objects in the video sequence.

6. The method according to claim 1, characterized in that, If an object associated with one of the bounding boxes in the bounding box does not exist in the first image, the position information of the bounding box in the bounding box indicates the position of the bounding box outside the first image.

7. The method according to claim 1, characterized in that, Further includes: Send a request to receive the metadata associated with the encoded video stream.

8. A video encoding / decoding apparatus, characterized in that, include: A receiving module is configured to receive metadata associated with an encoded video stream, the metadata including tagging information of one or more objects detected in a first image encoded into the encoded video stream; wherein the tagging information includes the total number of bounding boxes in the first image, and includes positional and size information for each bounding box, each bounding box being associated with one of the one or more objects in the first image; the positional information of a bounding box within a bounding box includes a positional offset between the position of the bounding box within the bounding box in the first image and its position in a second image encoded into the video stream; A decoding module is configured to decode the tagging information of one or more objects in the first image encoded into the encoded video stream; and An application module is configured to apply the marking information to the one or more objects in the first image by displaying the marking information of the one or more objects in the first image.

9. The apparatus according to claim 8, characterized in that, The metadata is included in a supplemental enhancement information message in the encoded video stream.

10. The apparatus according to claim 8, characterized in that, The metadata is included in a file separate from the encoded video stream.

11. The apparatus according to claim 8, characterized in that, The tagging information includes category information, which indicates the category of each of the one or more objects.

12. The apparatus according to claim 8, characterized in that, The tagging information includes identification information, which identifies each of the one or more objects in the video sequence.

13. The apparatus according to claim 8, characterized in that, If an object associated with one of the bounding boxes in the bounding box does not exist in the first image, the position information of the bounding box in the bounding box indicates the position of the bounding box outside the first image.

14. The apparatus according to claim 8, characterized in that, Further includes: Sending module: Used to send a request to receive the metadata associated with the encoded video stream.

15. A non-volatile computer-readable storage medium for storing instructions, characterized in that, When executed by at least one processor, the instructions cause the at least one processor to perform the method according to any one of claims 1-9.

Images