Video encoding and decoding method and device, method for processing machine vision media data, method for processing video code stream, and computer readable storage medium

Abstract

The invention discloses a video encoding and decoding method and device, a method for processing machine vision media data, a method for processing a video code stream and a computer readable storage medium. The video decoding method includes: receiving a code stream, the code stream including encoded information of a video for a machine task, the encoded information including a first portion and a second portion interleaved with each other, the first portion being encoded according to a video encoding standard, and the second portion including recovery data for the machine task; reordering the encoded information to generate a substream of the code stream, the substream having the first part; generating a first reconstructed picture of the video from the sub-stream according to the video coding standard; determining the recovery data from the second part; and generating a second reconstructed picture for the machine task based on the first reconstructed picture and the recovery data.

Description

Cross-referencing

[0001] This application claims priority to U.S. Patent Application No. 19 / 188,460, filed April 24, 2025, and U.S. Provisional Application No. 63 / 651,666, filed May 24, 2024. The entire disclosure of these earlier applications is incorporated herein by reference. Technical Field

[0002] This disclosure describes aspects generally related to video encoding and decoding, and in particular to the bitstream structure in machine video encoding. Background Technology

[0003] The background description provided herein is for the purpose of presenting the general context of this disclosure. The extent of the work of the currently named inventors described in this background section and in various aspects of this specification does not indicate that it was prior art at the time of filing of this application, nor is it expressly or implied that it was acknowledged as prior art to this disclosure.

[0004] Image / video compression helps transmit image / video data with minimal quality degradation across different devices, storage, and networks. In some examples, video codec techniques can compress video based on spatial and temporal redundancy. For instance, a video codec can use a technique called intra-frame prediction, which compresses images based on spatial redundancy. For example, intra-frame prediction can use reference data from the current image being reconstructed to make sample predictions. In another example, a video codec can use a technique called inter-frame prediction, which compresses images based on temporal redundancy. For example, inter-frame prediction can predict samples in the current image based on a previously reconstructed image with motion compensation. This motion compensation can be indicated by a motion vector (MV).

[0005] Existing encoding tools and standards, which are mainly designed for human visual perception, cannot efficiently meet the needs of machine tasks. There is an urgent need to optimize the encoding and decoding framework to reduce redundancy and complexity and improve the performance of machine tasks. Summary of the Invention

[0006] Various aspects of this disclosure include methods and apparatus for video encoding / decoding, bitstreams, methods for processing visual media data, methods for processing video bitstreams, and computer-readable storage media. In some examples, the apparatus for video encoding / decoding includes processing circuitry.

[0007] One aspect of this disclosure provides a video decoding method. For example, receiving a bitstream. The bitstream includes encoded information of video for a machine task, the encoded information comprising an interleaved first portion and a second portion, the first portion being encoded according to a video coding standard, and the second portion including recovery data for the machine task. The encoded information is reordered to generate a substream of the bitstream, the substream having the first portion. A first reconstructed image of the video is generated from the substream according to the video coding standard. The recovery data is determined from the second portion. A second reconstructed image for the machine task is generated based on the first reconstructed image and the recovery data.

[0008] Another aspect of this disclosure provides a video encoding method. For example, encoded information of a video is generated according to a video encoding standard. Based on the encoded information of the video, recovery data is determined, which is used to generate reconstructed images for machine tasks. The encoded information of the video is interleaved with the recovery data to form a bitstream.

[0009] Another aspect of this disclosure provides a method for processing machine vision media data. In this method, a conversion between a visual media file and a bitstream of visual media data is performed according to format rules. In an example, the bitstream carries encoded information of video for a machine task, the encoded information comprising an interleaved first portion and a second portion, the first portion being encoded according to a video coding standard, and the second portion comprising reconstructed data for the machine task. The format rules specify: reordering the encoded information to generate a substream of the bitstream having the first portion; generating a first reconstructed image from the substream according to the video coding standard; determining the reconstructed data from the second portion; and generating a second reconstructed image for the machine task based on the first reconstructed image and the reconstructed data.

[0010] This disclosure also provides a video encoding apparatus. The video encoding apparatus includes processing circuitry configured to implement any of the described video encoding methods.

[0011] This disclosure also provides a video decoding apparatus. The video decoding apparatus includes processing circuitry configured to implement any of the described video decoding methods.

[0012] Various aspects of this disclosure also provide a method for processing a video stream, which is decoded based on the described video decoding method or generated based on the described video encoding method.

[0013] Various aspects of this disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform any of the described video decoding / encoding methods.

[0014] According to various aspects of the present invention, a technique for a bitstream structure for VCM is provided, which can be matched with machine task requirements, reduce resource waste, reduce the computational complexity of encoding and decoding, and improve encoding and decoding efficiency. Attached Figure Description

[0015] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0016] Figure 1 This is a schematic diagram of an example block diagram of a communication system.

[0017] Figure 2 This is a schematic diagram of an example block diagram of the decoder.

[0018] Figure 3 This is a schematic diagram of an example block diagram of an encoder.

[0019] Figure 4 Block diagrams of video processing systems in machine scenarios are shown in some examples.

[0020] Figure 5 Syntax tables for Sequence Parameter Sets (SPS) in some examples of VVC are shown.

[0021] Figure 6 The syntax table for image headers in some examples of VVC is shown.

[0022] Figure 7 The syntax table for the End of Bitstream (EOB) in some examples of VVC is shown.

[0023] Figure 8 A diagram showing the bitstream structure according to some aspects of this disclosure is provided.

[0024] Figure 9 A diagram showing the bitstream structure according to some aspects of this disclosure is provided.

[0025] Figure 10 The diagram shows a sequence-level syntax table in a Network Abstraction Layer (NAL) unit in some examples.

[0026] Figure 11 A diagram of the frame-level syntax table in a NAL unit is shown in some examples.

[0027] Figure 12 A diagram of the syntax table at EOB in NAL units is shown in some examples.

[0028] Figure 13 Syntax tables for the Video Coding for Machine (VCM) unit are shown in some examples.

[0029] Figure 14 The syntax table for the header container in some example VCM units is shown.

[0030] Figure 15 The syntax table for the load container in some example VCM cells is shown.

[0031] Figure 16 A syntax table for restoring data containers is shown in some examples.

[0032] Figure 17 The syntax table for substream headers is shown in some examples.

[0033] Figure 18 The syntax table for substream NAL units is shown in some examples.

[0034] Figure 19 The syntax table for VCM NAL units is shown in some examples.

[0035] Figure 20 A syntax table for raw byte sequence payload (RBSP) for sequence recovery data is shown in some examples.

[0036] Figure 21 The syntax table for RBSP (Image Recovery Data Service) is shown in some examples.

[0037] Figure 22 A flowchart outlining a process according to one aspect of this disclosure is shown.

[0038] Figure 23 A flowchart outlining a process according to one aspect of this disclosure is shown.

[0039] Figure 24 It is a schematic diagram of a computer system based on one aspect. Detailed Implementation

[0040] Figure 1Block diagrams of some examples of video processing systems (100) are shown. These video processing systems (100) are examples of video encoders and decoders applied to applications of the disclosed subject matter, in streaming environments. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0041] The video processing system (100) includes an acquisition subsystem (113) that may include a video source (101), such as a digital camera, which creates, for example, an uncompressed video image stream (102). In an embodiment, the video image stream (102) includes samples captured by a digital camera. The video image stream (102) is depicted as a thick line to emphasize the high data volume of the video image stream compared to encoded video data (104) (or encoded video bitstream). The video image stream (102) may be processed by an electronic device (120) that includes a video encoder (103) coupled to the video source (101). The video encoder (103) may include hardware, software, or a combination of hardware and software to implement or carry out aspects of the disclosed subject matter as described in more detail below. Compared to the video image stream (102), the encoded video data (104) (or the encoded video bitstream (104)) is depicted as a thin line to emphasize the lower data volume of the encoded video data (104) (or the encoded video bitstream (104)), which can be stored on a streaming server (105) for future use. At least one streaming client subsystem, such as Figure 1 Client subsystems (106) and (108) can access a streaming server (105) to retrieve copies (107) and (109) of encoded video data (104). Client subsystem (106) may include, for example, a video decoder (110) in an electronic device (130). The video decoder (110) decodes the incoming copy (107) of the encoded video data and produces an output video picture stream (111) that can be displayed on a display (112) (e.g., a screen) or another presentation device (not depicted). In some streaming systems, the encoded video data (104), video data (107), and video data (109) (e.g., video streams) may be encoded according to certain video coding / compression standards. Examples of such standards include ITU-T H.265. In embodiments, the video coding standard under development is informally referred to as Versatile Video Coding (VVC), and this application can be used in the context of the VVC standard.

[0042] It should be noted that the electronic devices (120) and (130) may include other components (not shown). For example, the electronic device (120) may include a video decoder (not shown), and the electronic device (130) may also include a video encoder (not shown).

[0043] Figure 2 An example block diagram of a video decoder (210) is shown. The video decoder (210) may be disposed in an electronic device (230). The electronic device (230) may include a receiver (231) (e.g., receiving circuitry). The video decoder (210) may be used in place of... Figure 1 The video decoder (110) in the embodiment.

[0044] The receiver (231) can receive at least one encoded video sequence, for example, included in the bitstream, to be decoded by the video decoder (210). In one aspect, one encoded video sequence is received at a time, with the decoding of each encoded video sequence independent of the decoding of other encoded video sequences. The encoded video sequences can be received from a channel (201), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (231) can receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams that can be forwarded to their respective user entities (not indicated). The receiver (231) can separate the encoded video sequences from other data. To prevent network jitter, a buffer (215) may be coupled between the receiver (231) and the entropy decoder / parser (220) (hereinafter referred to as "parser (220)"). In some applications, the buffer (215) is part of the video decoder (210). In other cases, the buffer memory (215) may be located externally to the video decoder (210) (not shown). In other cases, a buffer memory (not shown) may be located externally to the video decoder (210) to prevent network jitter, for example, and another buffer memory (215) may be configured internally to handle broadcast timing, for example. When the receiver (231) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous synchronization network, a buffer memory (215) may not be necessary, or it may be made smaller. Of course, a buffer memory (215) may also be required for use on packet networks such as the Internet. This buffer memory may be relatively large and adaptive in size, and may be at least partially implemented in the operating system or a similar component (not shown) external to the video decoder (210).

[0045] The video decoder (210) may include a parser (220) to reconstruct symbols (221) from the encoded video sequence. These symbols may include information for managing the operation of the video decoder (210) and potential information for controlling a display device (212) (e.g., a display screen), which is not part of the electronic device (230) but may be coupled to it, such as... Figure 2 As shown in the figure. The control information for the display device may be a parameter set fragment (not shown) of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI). The parser (220) may parse / entropy decode the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (220) may extract a subgroup parameter set of at least one subgroup of pixels in the subgroup of pixels in the encoded video sequence for use in the video decoder based on at least one parameter corresponding to a group. The subgroup may include Group of Pictures (GOP), picture, tile, slice, macroblock, coding unit (CU), block, transform unit (TU), prediction unit (PU), etc. The parser (220) can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0046] The parser (220) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (215) to create symbols (221).

[0047] Depending on the type of encoded video frames or a subset of encoded video frames (e.g., inter-frame and intra-frame frames, inter-frame and intra-frame blocks) and other factors, the reconstruction of the symbol (221) may involve multiple different units. Which units are involved and how they are involved can be controlled by the subgroup control information parsed from the encoded video sequence by the parser (220). For brevity, the flow of such subgroup control information between the parser (220) and the various units described below is not described.

[0048] In addition to the functional blocks already mentioned, the video decoder (210) can be conceptually subdivided into several functional units as described below. In practical embodiments operating under commercial constraints, many of these units interact closely with each other and can be at least partially integrated with each other. However, for the purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into the functional units described below.

[0049] The first unit is the scaler / inverse transform unit (251). The scaler / inverse transform unit (251) receives quantization transform coefficients as symbols (221) and control information from the parser (220), including the transform method used, block size, quantization factor, quantization scaling matrix, etc. The scaler / inverse transform unit (251) can output a block containing sample values, which can be input into the aggregator (255).

[0050] In some cases, the output samples of the scaler / inverse transform unit (251) may belong to an intra-coded block. An intra-coded block is a block that does not use predictive information from a previously reconstructed image, but can use predictive information from a previously reconstructed portion of the current image. Such predictive information may be provided by the intra-picture prediction unit (252). In some cases, the intra-picture prediction unit (252) uses reconstructed information extracted from the current picture buffer (258) to generate a surrounding block of the same size and shape as the block being reconstructed. For example, the current picture buffer (258) buffers a partially reconstructed current image and / or a fully reconstructed current image. In some cases, the aggregator (255) adds the predictive information generated by the intra-picture prediction unit (252) to the output sample information provided by the scaler / inverse transform unit (251) based on each sample.

[0051] In other cases, the output samples of the scaler / inverse transform unit (251) may belong to inter-frame coding and latent motion compensation blocks. In this case, the motion compensation prediction unit (253) may access the reference image memory (257) to extract samples for prediction. After motion compensation of the extracted samples according to the symbols (221) belonging to the block, these samples may be added by the aggregator (255) to the output of the scaler / inverse transform unit (251) (referred to in this case as residual samples or residual signals) to generate output sample information. The motion compensation prediction unit (253) may obtain the predicted samples from the address in the reference image memory (257) under motion vector control, and the motion vector is available to the motion compensation prediction unit (253) in the form of the symbols (221), which may include, for example, X, Y and reference image components. Motion compensation may also include interpolation of sample values ​​extracted from the reference image memory (257) when using subsample precise motion vectors, motion vector prediction mechanisms, etc.

[0052] The output samples of the aggregator (255) can be employed by various loop filtering techniques in the loop filter unit (256). Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video sequence (also referred to as the encoded video stream), and these parameters can be used as symbols (221) from the parser (220) in the loop filter unit (256). However, in other embodiments, the video compression techniques may also respond to metadata obtained during decoding of a previous (in decoding order) portion of the encoded picture or encoded video sequence, and to previously reconstructed and loop-filtered sample values.

[0053] The output of the loop filter unit (256) can be a sample stream, which can be output to a display device (212) and stored in a reference image memory (257) for subsequent inter-frame image prediction.

[0054] Once fully reconstructed, some of the encoded images can be used as reference images for future predictions. For example, once the encoded images corresponding to the current image have been fully reconstructed and the encoded images (by, for example, the parser (220)) are identified as reference images, the current image buffer (258) can become part of the reference image memory (257), and a new current image buffer can be reallocated before the reconstruction of subsequent encoded images begins.

[0055] The video decoder (210) can perform decoding operations according to, for example, a standard or predetermined video compression technology such as ITU-T H.265. The encoded video sequence may conform to the syntax specified by the video compression technology or standard in the sense that the encoded video sequence follows the syntax of the video compression technology or standard and the configuration file recorded in the video compression technology or standard. Specifically, the configuration file may select certain tools from all available tools in the video compression technology or standard as the only tools available under said configuration file. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technology or standard. In some cases, the hierarchy limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (measured in, for example, megasamples per second), maximum reference picture size, etc. In some cases, the limitations set by the hierarchy can be further limited by the Hypothetical Reference Decoder (HRD) specification and the metadata managed by the HRD buffer, which is represented by signals in the encoded video sequence.

[0056] On one hand, the receiver (231) may receive additional (redundant) data along with the encoded video. The additional data may be a portion of the encoded video sequence. The additional data may be used by the video decoder (210) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0057] Figure 3 An example block diagram of a video encoder (303) is shown. The video encoder (303) is disposed in an electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., transmission circuitry). The video encoder (303) can be used in place of... Figure 1 The video encoder (103) in the embodiment.

[0058] The video encoder (303) can obtain data from the video source (301) (not) Figure 3 In one embodiment, a portion of the electronic device (320) receives video samples, the video source being capable of capturing video images to be encoded by a video encoder (303). In another embodiment, the video source (301) is a portion of the electronic device (320).

[0059] A video source (301) can provide a sequence of source videos encoded by a video encoder (303) in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601YCrCb, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (301) can be a storage device storing previously prepared video. In a video conferencing system, the video source (301) can be a camera that captures local image information as a video sequence. Video data can be provided as multiple individual pictures, which are given motion when viewed in sequence. The pictures themselves can be constructed as spatial pixel arrays, where each pixel can include at least one sample, depending on the sampling structure, color space, etc. used. The following focuses on describing the samples.

[0060] According to one aspect, the video encoder (303) can encode and compress images of a source video sequence into an encoded video sequence (343) in real time or under any other required time constraints. Implementing an appropriate encoding rate is a function of the controller (350). In some aspects, the controller (350) controls and is functionally coupled to other functional units described below. For simplicity, coupling is not indicated in the figures. Parameters set by the controller (350) may include rate control related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (350) may be used with other suitable functions related to the video encoder (303) optimized for a particular system design.

[0061] In some aspects, the video encoder (303) operates within an encoding loop. As a simplified description, in an embodiment, the encoding loop may include a source encoder (330) (e.g., responsible for creating symbols, such as a symbol stream, based on the input image to be encoded and a reference image) and a (local) decoder (333) embedded within the video encoder (303). The decoder (333) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data. The reconstructed sample stream (sample data) is input to a reference image memory (334). Since decoding of the symbol stream produces bit-accurate results regardless of the decoder's location (local or remote), the contents of the reference image memory (334) also correspond bit-accurately between the local and remote encoders. In other words, the reference image samples "seen" by the encoder's prediction section are exactly the same sample values ​​that the decoder will "see" when using the prediction during decoding. This fundamental principle of reference image synchronization (and the drift that occurs, for example, when synchronization cannot be maintained due to channel errors) is also used in some related techniques.

[0062] The operation of the “local” decoder (333) can be combined with, for example, the above-mentioned... Figure 2 The video decoder (210) is described in detail as the same as the "remote" decoder. However, a further brief reference is provided. Figure 2 When symbols are available and the entropy encoder (345) and parser (220) are able to encode / decode the symbols into an encoded video sequence without loss, the entropy decoding portion of the video decoder (210), including the buffer (215) and parser (220), may not be fully implemented in the local decoder (333).

[0063] On the one hand, any decoder technique other than parsing / entropy decoding present in the decoder also exists in the corresponding encoder in the same or substantially the same functional form. Therefore, this application focuses on decoder operation. The description of encoder techniques can be simplified because encoder techniques are inverses of the fully described decoder techniques. More detailed descriptions of certain areas are provided below.

[0064] During operation, in some embodiments, the source encoder (330) may perform motion-compensated predictive coding. The motion-compensated predictive coding predictively encodes the input image with reference to at least one previously encoded image from the video sequence designated as a "reference image." In this manner, the encoding engine (332) encodes the differences between pixel blocks of the input image and pixel blocks of the reference image, which may be selected as a predictive reference for the input image.

[0065] The local video decoder (333) can decode encoded video data of a picture that can be designated as a reference picture, based on symbols created by the source encoder (330). The operation of the encoding engine (332) can be a lossy process. When the encoded video data can be decoded by the video decoder (333), Figure 3 When the source video sequence (not shown) is decoded, the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (333) replicates the decoding process, which can be performed by the video decoder on the reference image, and allows the reconstructed reference image to be stored in a high-speed reference image memory (334). In this way, the video encoder (303) can locally store a copy of the reconstructed reference image that shares the same content (no transmission errors) as the reconstructed reference image to be obtained by the remote video decoder.

[0066] The predictor (335) can perform a prediction search against the encoding engine (332). That is, for a new image to be encoded, the predictor (335) can search in the reference image memory (334) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can serve as appropriate prediction references for the new image. The predictor (335) can operate pixel-by-pixel based on the sample blocks to find suitable prediction references. In some cases, based on the search results obtained by the predictor (335), it can be determined that the input image may have prediction references obtained from multiple reference images stored in the reference image memory (334).

[0067] The controller (350) can manage the encoding operations of the source encoder (330), including, for example, setting parameters and subgroup parameters for encoding video data.

[0068] The outputs of all the above-mentioned functional units can be entropy encoded in the entropy encoder (345). The entropy encoder (345) applies lossless compression to the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, and arithmetic coding, thereby converting the symbols into an encoded video sequence.

[0069] The transmitter (340) can buffer the encoded video sequence created by the entropy encoder (345) in preparation for transmission via a communication channel (360), which may be a hardware / software link to a storage device that will store the encoded video data. The transmitter (340) can combine the encoded video data from the video encoder (303) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0070] The controller (350) manages the operation of the video encoder (303). During encoding, the controller (350) can assign a specific encoded image type to each encoded image, but this may affect the encoding techniques applicable to the corresponding images. For example, images can typically be assigned to any of the following image types:

[0071] Intra-frame pictures (I-pictures) are pictures that can be encoded and decoded without using any other pictures in the sequence as prediction sources. Some video codecs allow different types of intra-frame pictures, including, for example, Independent Decoder Refresh (IDR) pictures.

[0072] A predictive image (P-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses motion vectors and reference indices to predict sample values ​​for each block.

[0073] A bidirectional predictive image (B-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses two motion vectors and a reference index to predict sample values ​​for each block. Similarly, multiple predictive images can use more than two reference images and associated metadata to reconstruct a single block.

[0074] Source images are typically spatially subdivided into multiple sample blocks (e.g., 4×4, 8×8, 4×8, or 16×16 sample blocks), and encoded block by block. These blocks can be predictively coded with reference to other (already coded) blocks, determined based on the coding assignments of the corresponding images applied to the blocks. For example, blocks of an I-image can be non-predictively coded, or the blocks can be predictively coded (spatial or intra-frame prediction) with reference to already coded blocks of the same image. Pixel blocks of a P-image can be predictively coded with reference to a previously coded reference image via spatial or temporal prediction. Blocks of a B-image can be predictively coded with reference to one or two previously coded reference images via spatial or temporal prediction.

[0075] The video encoder (303) can perform encoding operations according to a predetermined video coding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (303) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0076] On one hand, the transmitter (340) can transmit additional data while transmitting encoded video. The source encoder (330) can include such data as part of the encoded video sequence. Additional data may include temporal / spatial / SNR enhancement layers, redundant images and slices, other forms of redundant data, SEI messages, VUI parameter set fragments, etc.

[0077] The captured video can be presented as multiple source images (video images) in a time-series format. Intra-frame image prediction (often simplified to intra-frame prediction) utilizes spatial correlations within a given image, while inter-frame image prediction utilizes (temporal or other) correlations between images. In the example, a specific image being encoded / decoded is segmented into blocks, and this specific image being encoded / decoded is called the current image. When a block in the current image resembles a reference block in a previously encoded and still buffered reference image in the video, the block in the current image can be encoded using a vector called a motion vector. This motion vector points to the reference block in the reference image, and when using multiple reference images, the motion vector may have a third dimension that identifies the reference image.

[0078] In some respects, bidirectional prediction techniques can be used in inter-frame image prediction. According to bidirectional prediction, two reference images are used, such as a first reference image and a second reference image that precede the current image in the video in decoding order (but may be past and future in display order, respectively). A block in the current image can be encoded using a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. Specifically, the block can be predicted using a combination of the first and second reference blocks.

[0079] In addition, merging mode techniques can be used in inter-frame image prediction to improve coding efficiency.

[0080] According to some aspects disclosed in this application, predictions such as inter-frame image prediction and intra-frame image prediction are performed on a block-by-block basis. For example, according to the HEVC standard, images in a video image sequence are segmented into Coding Tree Units (CTUs) for compression. The CTUs in the images have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU comprises three Coding Tree Blocks (CTBs), one luma CTB and two chroma CTBs. Furthermore, each CTU can be further subdivided into at least one Coding Unit (CU) using a quadtree. For example, a 64×64 pixel CTU can be subdivided into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In one aspect, each CU is analyzed to determine the prediction type used for the CU, such as inter-frame prediction or intra-frame prediction. Furthermore, depending on temporal and / or spatial predictability, the CU is divided into at least one Prediction Unit (PU). Typically, each PU includes a Luminance Prediction Block (PB) and two Chroma PBs. In embodiments, prediction operations during encoding (encoding / decoding) are performed on a block-by-block basis. Taking a Luminance Prediction Block as an example, a prediction block includes a matrix of pixel values ​​(e.g., luminance values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0081] It should be noted that any suitable technology can be used to implement the video encoder (103) and video encoder (303), as well as the video decoder (110) and video decoder (210). On one hand, at least one integrated circuit can be used to implement the video encoder (103) and video encoder (303), as well as the video decoder (110) and video decoder (210). On the other hand, at least one processor executing software instructions can be used to implement the video encoder (103) and video encoder (303), as well as the video decoder (110) and video decoder (210).

[0082] According to some aspects of this disclosure, video coding can be used in machine scenarios.

[0083] Figure 4 Block diagrams of a video processing system (400) in some example machine scenarios are shown. Figure 4 In the example, the video processing system (400) includes multiple video sources for machine consumption. For example, the video processing system (400) includes a camera (401) coupled to an encoder device (402).

[0084] In some examples, the encoder can be optimized for machine consumption. For example, the encoder device (402) can be optimized for encoding a bitstream for machine consumption. For example, the encoder device (402) can be coupled to a decoder device (404) via a network (403), and the decoder device (404) can be coupled to a machine (405) that consumes the decoded video (e.g., for further analysis, detection, etc.). Thus, the encoded bitstream can be provided from the encoder device (402) to the decoder device (404) via the network (403), the encoded bitstream can be decoded by the decoder device (404), and the decoded video data can be further processed by the machine (405).

[0085] Video compression can be used for both human and machine consumption. Machine video coding (VCM) can be standardized as part of standards such as ISO / IEC JTC 1SC 29WG 4. In some examples, a reference model for VCM can use a relevant video coding standard (e.g., a relevant video coding standard for human consumption) as the core codec. In certain environments, such as in video coding for machine consumption (rather than human consumption), it may not be necessary to make the bitstream pass a quality threshold based on human perception. Instead, even if the quality is insufficient for human consumption, it may be sufficient for machine consumption. However, relevant video coding standards are primarily standardized for video applications serving human vision. In some examples, it has been identified that several coding tools in relevant video coding standards are not beneficial for machine tasks, but these tools can potentially increase complexity, such as encoding / decoding time.

[0086] This disclosure provides techniques for bitstream structures used in VCMs, including bitstream structures for information signaling, encoding, and parsing in video coding systems. It should be noted that in some examples, the techniques used for VCMs can be used in conventional video coding systems. In some examples, an encoder can generate a bitstream and send it to a decoder. The bitstream includes encoded information of video for machine tasks, said encoded information comprising interleaved first and second portions, the first portion being encoded according to a video coding standard, and the second portion including recovery data for machine tasks. A substream having the first portion can be generated by reordering. A first reconstructed image of the video can be generated from said substream according to a video coding standard. Recovery data can be determined from the second portion. A second reconstructed image for machine tasks can be generated based on the first reconstructed image and the recovery data.

[0087] It should be noted that when using relevant video coding standards in this disclosure, such relevant video coding standards may include, but are not limited to: Audio Video Standard (AVS) 1, AVS 2, AVS 3, AVS 4, Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), Alliance for Open Media (AOMedia) Video 1 (AV1), AV2, etc.

[0088] In some aspects, VCM-related information can be sequence-level information, which is notified, transmitted, and parsed using signals at the sequence level. In other aspects, VCM-related information is only frame-level information, which is notified, transmitted, and parsed using signals at the frame level. In still other aspects, VCM-related information can include both sequence-level and frame-level information, which is notified, transmitted, and parsed using signals at both the sequence and frame levels.

[0089] In some respects, VCM-related information may include sequence-level and / or frame-level information, which is signaled, transmitted, and parsed via SEI messages based on relevant video coding standards such as VVC.

[0090] In some respects, VCM-related information is embedded in the structure of the relevant video coding standard, which may include, but is not limited to, the Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Picture Header (PH), and Slice Header (SH).

[0091] Figure 5 A syntax table for SPS in VVC is shown in some examples. VCM-related information can be included as an extension at the SPS level; this extension is a sequence-level container to include VCM-related information, such as temporal resampling, bit depth truncation, etc. In some examples, the extension of VCM-related information may include modules such as Region of Interest (RoI), bit depth truncation, etc.

[0092] exist Figure 5 In the example, a VCM extension flag (e.g., sps_vcm_extension_flag) is signaled in the bitstream to indicate whether the VCM-related information is included in the extension at the SPS level, as shown in (501). Figure 5 As shown in (502), when the VCM extension flag is true, an extension (e.g., sps_vcm_extention()) is included at the SPS level, which includes the VCM-related information. The VCM-related information at the SPS level can be decoded from the VCM extension in the bitstream.

[0093] Figure 6 The syntax table for the image header in some examples of VVC is shown. At the frame level, VCM-related information can be included as an extension in the image header. This extension is a frame-level container to include the VCM-related information, such as temporal resampling, bit depth truncation, etc., which can be used for recovery.

[0094] exist Figure 6 In the example, if the VCM extension flag (e.g., `sps_vcm_extension_flag`) is true, then it includes an extension at the frame header level (e.g., `picture_header_vcm_extention()`), which includes the VCM-related information, such as... Figure 6 As shown in (601) in the code stream, VCM-related information at the frame header level can be decoded from the VCM extension in the bitstream.

[0095] Figure 7 The syntax table for the End-of-Batch (EOB) in a VVC is shown in some examples. At the sequence level, VCM-related information can be included as an extension in this EOB, which includes VCM-related information such as temporal resampling, bit depth truncation, etc., which can be used for recovery.

[0096] exist Figure 7 In the example, at the sequence level, a container (e.g., eob_vcm()) is included at the end of the bitstream to include the VCM-related information, as provided by... Figure 7 As shown in (701). In the example, when a bitstream is generated for use in a video stream, the recovery information at the end of the video may differ from the recovery information within the video, and the relevant VCM information for recovery at the end of the bitstream may be included in a container in the EOB.

[0097] Figure 8 A diagram illustrating a bitstream structure (800) according to some aspects of this disclosure is shown. The bitstream structure (800) is formed according to relevant video coding standards, wherein, by extension (such as using...) Figures 5 to 7 The syntax table in the code embeds VCM-related information into the bitstream structure (800). Figure 8 In the example, the relevant video coding standard is the VVC TestModel (VTM).

[0098] For example, the bitstream structure (800) includes sequence-level parameters, such as VTM sequence parameters (810) and VTM bitstream end (EOB) (840). The bitstream structure (800) also includes multiple frame data, such as a first frame (f1) (820) and a second frame (f2) (830).

[0099] exist Figure 8In the example, the bitstream structure (800) includes sequence-level VCM-related information, such as VCM sequence parameters (811) embedded in the VTM sequence parameters (810) and VCM EOBs (841) embedded in the VTM EOBs (840). The bitstream structure (800) also includes frame-level VCM-related information, such as VCM frame 1 (f1) parameters (821) embedded in the VTM frame 1 (f1) parameters (825) and VCM frame 2 (f2) parameters (831) embedded in the VTM frame 2 (f2) parameters (835).

[0100] According to some aspects of this disclosure, VCM-related structures (e.g., sequence-level VTM-related structures and / or frame-level VTM-related structures) are intertwined with the structures of relevant video coding standards.

[0101] In some aspects, a bitstream (e.g., referred to as a VCM bitstream) comprises multiple structures (also called VCM units in some examples), some structures (e.g., referred to as a first structure) containing VCM-related information, and some structures (e.g., referred to as a second structure) containing video information related to the video coding standard. The first structure and the second structure can be interleaved in the bitstream. In some examples, each of the multiple structures may include information indicating the type of that structure. Therefore, the structures in the bitstream can be appropriately parsed, organized, and decoded.

[0102] In some examples, the bitstream is transmitted in Network Abstraction Layer (NAL) units. For example, on the encoder side, multiple NAL units are generated at the NAL layer. Each NAL unit includes a header containing information identifying that NAL unit. Multiple NAL units can be transmitted as a whole, for example, separately over the network. On the decoder side, NAL units are received (e.g., from the network) and passed to the decoder. In some examples, some NAL units (also called first NAL units) include VCM-related information, and some NAL units (also called second NAL units) include information related to the relevant video coding standard, such as video data according to VVC. The first NAL units and the second NAL units can be interleaved in the bitstream.

[0103] In some examples, on the decoder side, when parsing the bitstream, the NAL units can be reordered and separated into substreams. For example, VCM-related information and information related to the relevant video coding standard can be separated into a first substream and a second substream, respectively. The first and second substreams can then be parsed separately. In some examples, the first substream containing VCM-related information can be parsed by the VCM decoder, and the second substream containing information related to the relevant video coding standard (e.g., VVC) can be parsed according to the relevant video coding standard.

[0104] Figure 9 A diagram showing the bitstream structure (900) according to some aspects of this disclosure is provided. Figure 9 In the example, VCM-related information is intertwined with information from relevant video coding standards (e.g., the VVC Test Model (VTM) in the bitstream structure (900)).

[0105] exist Figure 9 In the example, VCM-related information and VTM-related information are intertwined. For example, the bitstream structure (900) includes parts (901)-(910), some parts such as (901), (903), (906), and (910) are VCM-related; and some parts such as (902), (904), (905), (907), (908), and (909) are VTM-related. It should be noted that... Figure 9 The portion of the example can be included in at least one NAL unit.

[0106] exist Figure 9 In the example, NAL units can be reordered into substreams. For example, the NAL units corresponding to parts (902), (904), (905), (907), (908), and (909) can be reordered into substream (950). The substream (950) can be parsed and decoded according to VVC. Figure 9 As not shown in the diagram, the VCM-related parts can be reordered and parsed using a VCM decoder.

[0107] Figure 10 The diagram shows a sequence-level syntax table in a NAL unit in some examples. Figure 10 The syntax table in the table is configured to include VCM sequence parameters. For example, the container (e.g., sequence_level_vcm_data())(1001) is a sequence-level container that includes VCM-related information such as time resampling, bit depth truncation, etc.

[0108] Figure 11 A diagram of the frame-level syntax table in a NAL unit is shown in some examples. Figure 11 The syntax table in the table is configured to include VCM frame parameters. For example, a container (e.g., frame_level_vcm_data())(1101) is a frame-level container that includes VCM-related information such as temporal resampling, bit depth truncation, etc.

[0109] Figure 12 A diagram of the syntax table at EOB in NAL units is shown in some examples. Figure 12The syntax table in the code is configured to include end-of-stream information for the VCM bitstream. For example, the container (e.g., eob_vcm_extention())(1201) is a sequence-level container that includes VCM-related information such as temporal resampling, bit depth truncation, etc.

[0110] In some examples, multiple types of VCM units are used. For example, these multiple types include: a first type called a VCM parameter set (e.g., denoted as VCM_VPS), a second type called restored data (e.g., denoted as VCM_RSD), and a third type called encoded video data (e.g., denoted as VCM_CVD). The first and second type VCM units include VCM-related information. The third type VCM unit includes information about the relevant video coding standard. In some examples, the bitstream may include: VCM units of type VCM_VPS, at least one VCM unit of type VCM_RSD, and at least one VCM unit of type VCM_CVD.

[0111] In some examples, the VCM unit includes a header and a payload. Figure 13 The syntax table for a VCM unit in some examples is shown. The VCM unit includes a header container (1310) and a payload container (1320).

[0112] In some examples, the header container includes syntax elements that identify the VCM unit type. Figure 14 Syntax tables for header containers in VCM units are shown in some examples. The header container includes syntax elements that identify the type of VCM unit, as shown in (1411).

[0113] In some examples, the load container includes different containers for different types of VCM units. Figure 15 Syntax tables for payload containers in VCM units are shown in some examples. When the VCM unit is of type VCM_VPS, the payload container includes a VCM parameter set container (1521). When the VCM unit is of type VCM_RSD, the payload container includes a recovery data container (1522). When the VCM unit is of type VCM_CVD, the payload container includes an encoded video data container (1523).

[0114] The VCM parameter set container can include high-level parameters such as control flags. The recovery data container can include information that enables VCM recovery. According to relevant standards such as VVC and HEVC, the encoded video data container can include (sub)streams of video data.

[0115] In some examples, the recovery data container may include a substream of information for recovery. Figure 16Syntax tables for a recovery data container are shown in some examples. The recovery data container includes a substream header as shown in (1610) and multiple substream NAL units as shown in (1620). In the examples, the substream header includes substream hierarchy information, such as precision information as shown in the examples. The multiple substream NAL units each include a VCM NAL unit.

[0116] Figure 17 The syntax table for the substream header is shown in some examples. The substream header includes precision information (1710).

[0117] Figure 18 The syntax table of a substream NAL cell is shown in some examples. The substream NAL cell includes a VCM NAL cell (1820) and the size information of the NAL cell (1810).

[0118] In some examples, the VCM NAL unit may include a header and a raw bytesequence payload (RBSP). Figure 19 The syntax table of a VCM NAL unit is shown in some examples. The VCM NAL unit includes a header (1910) and a section for RBSP as shown in (1920).

[0119] It should be noted that the RBSP in the VCM NAL unit can be a sequence recovery data RBSP, an image recovery data RBSP, etc.

[0120] Figure 20 Syntax tables for RBSP (Retrievable Series-Based Programming) are shown in some examples. Figure 20 In the example, when the spatial resampling enabled flag (e.g., srd_spatial_resampling_enabled_flag) at the sequence recovery data is true, spatial resampling information can be extracted from the sequence recovery data RBSP, as shown in (2010).

[0121] Still as Figure 20 As shown in the example, when the redirection enable flag (e.g., srd_retargeting_enabled_flag) at the sequence recovery data is true, redirection parameters can be extracted from the sequence recovery data RBSP, as shown in (2020).

[0122] Still as Figure 20 As shown in the example, time recovery information can be extracted from the sequence recovery data RBSP when the time recovery enabled flag (e.g., srd_temporal_restoration_enabled_flag) at the sequence recovery data is true, as shown in (2030).

[0123] Still as Figure 20 As shown in the example, when the bit depth shift enable flag (e.g., srd_bit_depth_shift_enabled_flag) at the sequence recovery data is true, bit depth shift information can be extracted from the sequence recovery data RBSP, as shown in (2040).

[0124] Figure 21 The syntax table for RBSP (Image Recovery Data Service) is shown in some examples.

[0125] exist Figure 21 In the example, when the spatial resampling update flag (e.g., prd_spatial_resampling_update_flag) at the image restoration data is true, spatial resampling information can be extracted from the image restoration data RBSP, as shown in (2110).

[0126] Also in Figure 21 In the example, when the redirection update flag (e.g., prd_retargeting_update_flag) at the image recovery data is true, the redirection parameters can be extracted from the image recovery data RBSP, as shown in (2120).

[0127] Also in Figure 21 In the example, time recovery information can be extracted from the image recovery data RBSP, as shown in (2130).

[0128] Also in Figure 21 In the example, when the bit depth shift update flag (e.g., prd_bit_depth_shift_update_flag) at the image recovery data is true, bit depth shift information can be extracted from the image recovery data RBSP, as shown in (2140).

[0129] It is important to note that in some examples, the Image Recovery Data RBSP may include the Picture Order Count (POC) of the images for use in applying image recovery data within the RBSP. For example, the POC indicates the bitstream from the relevant video coding standard (e.g., Figure 9 The image decoded from the substream (950) in the middle.

[0130] It should also be noted that recovery data can be included in the bitstream using other suitable techniques. For example, recovery data can be included in the SEI message. In another example, the end of the recovery data RBSP container is used to include recovery data, which is useful for recovering the final portion of the VCM video.

[0131] Figure 22 A flowchart of an overview process (2200) according to one aspect of this disclosure is shown. Process (2200) can be used in a video decoder. In various aspects, process (2200) is executed by processing circuitry, such as processing circuitry that performs the functions of video decoder (110), processing circuitry that performs the functions of video decoder (210), etc. In some aspects, process (2200) is implemented as software instructions, so that when the processing circuitry executes the software instructions, the processing circuitry executes process (2200). The process begins at step (S2201) and proceeds to step (S2210).

[0132] At step (S2210), a bitstream is received. The bitstream includes encoded information of video for a machine task, the encoded information including an interleaved first part and a second part, the first part being encoded according to a video coding standard, and the second part including recovery data for the machine task.

[0133] At step (S2220), the encoded information is reordered to generate a substream having the first part of the bitstream.

[0134] At step (S2230), a first reconstructed image of the video is generated from the substream according to the video coding standard.

[0135] At step (S2240), the recovered data is determined from the second part.

[0136] At step (S2250), a second reconstructed image for the machine task is generated based on the first reconstructed image and the recovery data.

[0137] In some respects, the video coding standard includes at least one of the following: Audio Video Standard (AVS) 1, AVS 2, AVS 3, AVS 4, Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), Universal Video Coding (VVC), Alliance for Open Media (AOMedia) Video 1 (AV1), and AV 2.

[0138] In some examples, to determine the recovered data, at least one of the following is parsed from the second part: sequence-level recovered data and / or image-level recovered data. In some examples, at least one Supplemental Enhancement Information (SEI) message is parsed to obtain the recovered data.

[0139] In some examples, extensions to the structure of the video coding standard are detected. The recovered data is obtained from these extensions. The structure of the video coding standard includes at least one of the following: Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Picture Header (PH), and Slice Header (SH).

[0140] In some examples, the recovered data includes at least one of the following from the second part: spatial resampling parameters, redirection parameters, temporal recovery parameters, and bit depth shift parameters.

[0141] In some examples, image-level recovery data associated with the image sequence number (POC) can be parsed from the second part.

[0142] In some examples, recovery data for the end of the bitstream is parsed from the second part.

[0143] In some examples, the first portion includes multiple network abstraction layer (NAL) units that are encapsulated as machine video coding (VCM) units of the encoded video data (CVD) type.

[0144] In some examples, the second portion includes a plurality of Network Abstraction Layer (NAL) units, which are encapsulated as Machine Video Coding (VCM) units of the Recovered Data (RSD) type. In one example, the plurality of NAL units includes at least: NAL units comprising sequence-level recovered data. In another example, the plurality of NAL units includes at least: NAL units comprising picture-level recovered data. In yet another example, the plurality of NAL units includes at least: NAL units corresponding to Supplemental Enhancement Information (SEI) comprising the recovered data. In yet another example, the plurality of NAL units includes at least: NAL units comprising recovered data for the end of a substream.

[0145] Then, the process proceeds to (S2299) and terminates.

[0146] The procedure (2200) can be adjusted as appropriate. Steps in the procedure (2200) can be modified and / or omitted. Additional steps can be added. Any suitable implementation order can be used.

[0147] Figure 23A flowchart outlining a process (2300) according to one aspect of this disclosure is shown. The process (2300) can be used in a video encoder. In various aspects, the process (2300) is executed by processing circuitry, such as processing circuitry that performs the functions of a video encoder (103), processing circuitry that performs the functions of a video encoder (303), etc. In some embodiments, the process (2300) is implemented as software instructions, so that when the processing circuitry executes the software instructions, the processing circuitry executes the process (2300). The process begins at step (S2301) and proceeds to step (S2310).

[0148] In step (S2310), the encoded information of the video is generated according to the video encoding standard.

[0149] At step (S2320), recovery data is determined based on the encoded information of the video, and the recovery data is used to generate reconstructed images for machine tasks.

[0150] In step (S2330), the encoded information of the video is interleaved with the recovered data to form a bitstream.

[0151] In some respects, the video coding standard includes at least one of the following: Audio Video Standard (AVS) 1, AVS 2, AVS 3, AVS 4, Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), Universal Video Coding (VVC), Alliance for Open Media (AOMedia) Video 1 (AV 1), and AV 2.

[0152] In some examples, the recovered data includes at least one of the following: sequence-level recovered data and / or image-level recovered data.

[0153] In some examples, the bitstream includes multiple machine video coding (VCM) units, which are one of the following: encoded video data (CVD) type, restored data (RSD) type, and VCM parameter set (VPS) type.

[0154] In some examples, the recovery data is included in a Supplemental Enhancement Information (SEI) message.

[0155] In some respects, the recovered data is included in an extension of the structure of the video coding standard. In some examples, the structure of the video coding standard includes at least one of the following: Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Picture Header (PH), and Slice Header (SH).

[0156] In some examples, the recovered data includes at least one of the following from the second part: spatial resampling parameters, redirection parameters, temporal recovery parameters, and bit depth shift parameters.

[0157] In some examples, image-level recovery data is associated with image sequence numbers (POCs).

[0158] In some examples, the recovery data includes recovery data from the second part for the end of the bitstream.

[0159] In some examples, the encoded information of the video includes multiple Network Abstraction Layer (NAL) units, which are encapsulated as Machine Video Coding (VCM) units of the Encoded Video Data (CVD) type.

[0160] In some examples, the recovered data includes multiple Network Abstraction Layer (NAL) units, which are encapsulated as Machine Video Coding (VCM) units of type Recovered Data (RSD). In one example, the multiple NAL units include at least: NAL units comprising sequence-level recovered data. In another example, the multiple NAL units include at least: NAL units comprising picture-level recovered data. In yet another example, the multiple NAL units include at least: NAL units corresponding to Supplemental Enhancement Information (SEI) corresponding to the recovered data. In yet another example, the multiple NAL units include at least: NAL units comprising recovered data for the end of the bitstream.

[0161] Then, the process proceeds to (S2399) and terminates.

[0162] The process (2300) can be adjusted as appropriate. Steps in the process (2300) can be modified and / or omitted. Additional steps can be added. Any suitable implementation order can be used.

[0163] According to one aspect of this disclosure, a method for processing visual media data is provided. In this method, a conversion between a visual media file and a bitstream of visual media data is performed according to format rules. For example, the bitstream may be a bitstream decoded / encoded in any decoding and / or encoding method described herein. The format rules may specify at least one constraint on the bitstream, and / or at least one process performed by a decoder and / or encoder.

[0164] In the example, the bitstream carries encoded information of video for a machine task. This encoded information includes an interleaved first portion and a second portion. The first portion is encoded according to a video coding standard, and the second portion includes recovery data for the machine task. The format rules specify: reordering the encoded information to generate a substream of the bitstream, the substream having the first portion; generating a first reconstructed image from the substream according to the video coding standard; determining the recovery data from the second portion; and generating a second reconstructed image for the machine task based on the first reconstructed image and the recovery data.

[0165] The above-described techniques can be implemented as computer software that uses computer-readable instructions and is physically stored on at least one computer-readable medium. For example, Figure 24 A computer system (2400) suitable for implementing certain aspects of the disclosed subject matter is shown.

[0166] Computer software can be coded using any suitable machine code or computer language. Any suitable machine code or computer language can be assembled, compiled, linked, or similarly used to create code containing instructions that can be executed directly by at least one computer central processing unit (CPU), graphics processing unit (GPU), or through decoding, microcode, etc.

[0167] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0168] Figure 24 The components of the computer system (2400) shown are exemplary and are not intended to impose any limitation on the scope or functionality of computer software implementing aspects of this disclosure. Nor should the configuration of the components be construed as having any dependency or requirement relating to any component or combination of components shown in the exemplary aspects of the computer system (2400).

[0169] The computer system (2400) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from at least one human user, for example, through input such as: tactile input (e.g., keystrokes, swipes, data glove movement), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not depicted). The human-machine interface device may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., voice, music, ambient sound), images (e.g., scanned images, photographic images obtained from still image cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video), etc.

[0170] The input human-machine interface device may include one or more of the following (only one of each is shown): keyboard (2401), mouse (2402), touchpad (2403), touch screen (2410), data glove (not shown), joystick (2405), microphone (2406), scanner (2407), camera (2408).

[0171] The computer system (2400) may also include certain human-machine interface output devices. Such human-machine interface output devices may, for example, stimulate the senses of at least one human user through tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (2410), a data glove (not shown), or a joystick (2405), but may also be haptic feedback devices that are not input devices), audio output devices (e.g., speakers (2409), headphones (not shown)), visual output devices (e.g., screens including cathode ray tube (CRT) screens, liquid-crystal display (LCD) screens, plasma screens, organic light-emitting diode (OLED) screens (2410), each screen having or not having touchscreen input functionality, each screen having or not having haptic feedback functionality, some of which are capable of outputting two-dimensional or more than three-dimensional visual output via devices such as stereoscopic image output, virtual reality glasses (not shown), holographic displays, and smoke boxes (not shown), and printers (not shown).

[0172] The computer system (2400) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (2420) having CD / DVD or similar media (2421), finger drives (2422), removable hard disk drives or solid-state drives (2423), conventional magnetic media such as magnetic tapes and floppy disks (not shown), devices based on dedicated ROM / ASIC / PLD such as security dongles (not shown), etc.

[0173] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0174] The computer system (2400) may also include an interface (2454) to at least one communication network (2455). The network may be, for example, a wireless network, a wired network, or an optical network. The network may also be a local area network, a wide area network, a metropolitan area network, a vehicle and industrial network, a real-time network, a latency-tolerant network, and so on. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including Global System for Mobile Communications (GSM), 3G, 4G, 5G, Long-Term Evolution (LTE), etc., cable or wireless wide area digital television networks including cable television, satellite television, and terrestrial broadcast television, vehicle and industrial networks including CAN buses, and so on. Some networks typically require an external network interface adapter (e.g., a USB port on the computer system (2400)) to connect to some general-purpose data port or peripheral bus (2449). Other network interfaces are typically integrated into the core of the computer system (2400) by connecting to a system bus (e.g., an Ethernet interface connected to a PC computer system or a cellular network interface connected to a smartphone computer system). The computer system (2400) can use any of these networks to communicate with other entities. Such communication can be one-way receiving (e.g., broadcast television), one-way transmitting (e.g., to a CAN bus of some CAN bus device), or bidirectional, for example, using a local area network or wide area network to connect to other computer systems. As mentioned above, certain protocols and protocol stacks can be used on each of those networks and network interfaces.

[0175] The aforementioned human-machine interface devices, human-accessible storage devices, and network interfaces can be attached to the kernel (2440) of the computer system (2400).

[0176] The core (2440) may include at least one central processing unit (CPU) (2441), a graphics processing unit (GPU) (2442), a dedicated programmable processing unit (2443) in the form of a field-programmable gate array (FPGA), a hardware accelerator (2444) for certain tasks, a graphics adapter (2450), and so on. These devices, along with read-only memory (ROM) (2445), random access memory (2446), and internal mass storage (2447) such as internal non-user-accessible hard disk drives, SSDs, etc., can be connected via a system bus (2448). In some computer systems, the system bus (2448) may be accessed in the form of at least one physical plug to allow for expansion with additional CPUs, GPUs, etc. Peripheral devices may be directly connected to the core's system bus (2448) or connected via a peripheral bus (2449) to the core's system bus (2448). In the example, a screen (2410) may be connected to a graphics adapter (2450). Peripheral bus architectures include PCI, USB, etc.

[0177] The CPU (2441), GPU (2442), FPGA (2443), and accelerator (2444) can execute certain instructions, which can be combined to form the aforementioned computer code. This computer code can be stored in ROM (2445) or RAM (2446). Transient data can also be stored in RAM (2446), while permanent data can be stored, for example, in internal mass storage (2447). Fast storage and retrieval to any storage device can be achieved using a cache, which can be closely associated with at least one CPU (2441), GPU (2442), mass storage (2447), ROM (2445), RAM (2446), etc.

[0178] Computer-readable media may have computer code thereon for performing various computer-implemented operations. The media and computer code may be media and computer code specifically designed and constructed for the purposes of this disclosure, or the media and computer code may be of a type known and available to those skilled in the art of computer software.

[0179] As a non-limiting example, a computer system having an architecture (2400), particularly a kernel (2440), can provide functionality because at least one processor (including a CPU, GPU, FPGA, accelerator, etc.) executes software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above, and some non-transitory memory of the kernel (2440), such as internal kernel mass storage (2447) or ROM (2445). Software implementing various embodiments of this disclosure can be stored in such devices and executed by the kernel (2440). Depending on specific needs, the computer-readable media may include at least one storage device or chip. The software can cause the kernel (2440), particularly the processors therein (including CPUs, GPUs, FPGAs, etc.), to execute specific processes or specific portions of specific processes described in this disclosure, including defining data structures stored in RAM (2446) and modifying such data structures according to software-defined processes. Additionally or alternatively, logic, either hardwired or otherwise embodied in the circuitry (e.g., the accelerator (2444)), enables the computer system to provide functionality that can replace or operate with the software to perform a particular process or a particular portion of a particular process described in this disclosure. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuitry storing software for execution (e.g., an integrated circuit (IC)), circuitry embodying logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

[0180] The use of "at least one" or "one" in this disclosure is intended to include any one or a combination of the elements. For example, referring to at least one of A, B, or C, at least one of A, B, and C, at least one of A, B, and / or C, and at least one of A through C is intended to include only A, only B, only C, or any combination thereof. Referring to one of A or B and one of A and B is intended to include either A or B or (A and B). Where appropriate, the use of "one" does not exclude any combination of the elements, for example, when the elements are not mutually exclusive.

[0181] Although several exemplary aspects have been described in this disclosure, modifications, substitutions, and various equivalent alternatives fall within the scope of this disclosure. Therefore, it should be understood that those skilled in the art will be able to design numerous systems and methods that, while not expressly shown or described in this disclosure, embody the principles of this disclosure and thus fall within its spirit and scope.

[0182] The above disclosure also covers the features mentioned below. These features can be combined in various ways, and are not limited to the combinations mentioned below.

[0183] (1) A video decoding method, comprising: receiving a bitstream, the bitstream including encoded information of video for a machine task, the encoded information including an interleaved first part and a second part, the first part being encoded according to a video coding standard, and the second part including recovery data for the machine task; reordering the encoded information to generate a substream of the bitstream, the substream having the first part; generating a first reconstructed image of the video from the substream according to the video coding standard; determining the recovery data from the second part; and generating a second reconstructed image for the machine task based on the first reconstructed image and the recovery data.

[0184] (2) The method according to feature (1), wherein the video coding standard includes at least one of the following: Audio Video Standard (AVS) 1, AVS 2, AVS 3, AVS 4, Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), Universal Video Coding (VVC), Open Media Consortium (AOMedia) Video 1 (AV 1) and AV 2.

[0185] (3) The method according to any one of features (1) to (2), wherein determining the recovered data includes at least one of: parsing sequence-level recovered data from the second part; and parsing image-level recovered data from the second part.

[0186] (4) The method according to any one of features (1) to (3), wherein determining the recovery data comprises: parsing at least one supplemental enhancement information (SEI) message to obtain the recovery data.

[0187] (5) The method according to any one of features (1) to (4), wherein determining the recovered data comprises: detecting an extension of the structure of the video coding standard; and obtaining the recovered data from the extension of the structure.

[0188] (6) The method according to any one of features (1) to (5), wherein the structure of the video coding standard includes at least one of the following: Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Picture Header (PH), and Slice Header (SH).

[0189] (7) The method according to any one of features (1) to (6), wherein determining the recovered data comprises: determining at least one of the following from the second part: spatial resampling parameters, redirection parameters, temporal recovery parameters and bit depth shift parameters.

[0190] (8) The method according to any one of features (1) to (7), wherein determining the recovery data comprises: parsing image-level recovery data associated with the image sequence number (POC) from the second portion.

[0191] (9) The method according to any one of features (1) to (8), wherein determining the recovery data comprises: parsing recovery data for the end of the bitstream from the second portion.

[0192] (10) The method according to any one of features (1) to (9), wherein the first part comprises a plurality of network abstraction layer (NAL) units, the plurality of network abstraction layer (NAL) units being encapsulated as machine video coding (VCM) units of type encoded video data (CVD).

[0193] (11) The method according to any one of features (1) to (10), wherein the second part comprises a plurality of network abstraction layer (NAL) units, the plurality of network abstraction layer (NAL) units being encapsulated as machine video coding (VCM) units of recovery data (RSD) type.

[0194] (12) The method according to any one of features (1) to (11), wherein the plurality of NAL units include at least: NAL units including sequence-level recovery data.

[0195] (13) The method according to any one of features (1) to (12), wherein the plurality of NAL units include at least: NAL units including image-level recovery data.

[0196] (14) The method according to any one of features (1) to (13), wherein the plurality of NAL units include at least: NAL units corresponding to supplementary enhancement information (SEI) including recovery data.

[0197] (15) The method according to any one of features (1) to (14), wherein the plurality of NAL units include at least: a NAL unit including recovery data, the recovery data being used at the end of the substream of the first portion.

[0198] (16) A video encoding method, comprising: generating encoded information of a video according to a video encoding standard; determining recovery data according to the encoded information of the video, the recovery data being used to generate a reconstructed image for a machine task; and interleaving the encoded information of the video with the recovery data to form a bitstream.

[0199] (17) The method according to feature (16), wherein the video coding standard includes at least one of the following: Audio Video Standard (AVS) 1, AVS 2, AVS 3, AVS 4, Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), Universal Video Coding (VVC), Open Media Consortium (AOMedia) Video 1 (AV1) and AV 2.

[0200] (18) The method according to any one of features (16) to (17), wherein the recovered data includes: sequence-level recovered data and / or image-level recovered data.

[0201] (19) The method according to any one of features (16) to (18), wherein the bitstream comprises a plurality of machine video coding (VCM) units, the plurality of VCM units being of one of the following types: encoded video data (CVD) type, restored data (RSD) type, and VCM parameter set (VPS) type.

[0202] (20) The method according to any one of features (16) to (19), wherein the recovery data is included in a supplemental enhancement information (SEI) message.

[0203] (21) The method according to any one of features (16) to (20), wherein the recovered data is included in an extension of the structure of the video coding standard.

[0204] (22) The method according to any one of features (16) to (21), wherein the structure of the video coding standard includes at least one of the following: Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Picture Header (PH), and Slice Header (SH).

[0205] (23) The method according to any one of features (16) to (22), wherein the recovered data includes at least one of the following: spatial resampling parameters, redirection parameters, temporal recovery parameters and bit depth shift parameters.

[0206] (24) The method according to any one of features (16) to (23), wherein the image-level recovery data is associated with an image sequence number (POC).

[0207] (25) The method according to any one of features (16) to (24), wherein the recovery data includes recovery data for the end of the bitstream.

[0208] (26) The method according to any one of features (16) to (25), wherein the encoded information of the video includes a plurality of network abstraction layer (NAL) units, the plurality of network abstraction layer (NAL) units being encapsulated as machine video coding (VCM) units of type encoded video data (CVD).

[0209] (27) The method according to any one of features (16) to (26), wherein the recovery data comprises a plurality of network abstraction layer (NAL) units, the plurality of network abstraction layer (NAL) units being encapsulated as machine video coding (VCM) units of recovery data (RSD) type.

[0210] (28) The method according to any one of features (16) to (27), wherein the plurality of NAL units include at least: NAL units including sequence-level recovery data.

[0211] (29) The method according to any one of features (16) to (28), wherein the plurality of NAL units include at least: NAL units including image-level recovery data.

[0212] (30) The method according to any one of features (16) to (29), wherein the plurality of NAL units include at least: NAL units corresponding to supplementary enhancement information (SEI) including recovery data.

[0213] (31) The method according to any one of features (16) to (30), wherein the plurality of NAL units include at least: a NAL unit including recovery data, the recovery data being used at the end of the bitstream.

[0214] (32) A method for processing visual media data, the method comprising: processing a bitstream including the visual media data according to a format rule, wherein the bitstream carries encoded information of video for a machine task, the encoded information including an interleaved first part and a second part, the first part being encoded according to a video coding standard, and the second part including recovery data for the machine task; and the format rule specifying: reordering the encoded information to generate a substream of the bitstream, the substream having the first part; generating a first reconstructed image from the substream according to the video coding standard; determining the recovery data from the second part; and generating a second reconstructed image for the machine task based on the first reconstructed image and the recovery data.

[0215] (33) A video decoding apparatus, including a processing circuit configured to perform a method of any one of features (1) to (15).

[0216] (34) A video encoding apparatus, including processing circuitry configured to perform a method of any one of features (16) to (31).

[0217] (35) A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform the method of any one of features (1) to (32).

Claims

1. A video decoding method, characterized in that, The method includes: Receive a bitstream, the bitstream including encoded information of video for a machine task, the encoded information including an interleaved first part and a second part, the first part being encoded according to a video coding standard, and the second part including recovery data for the machine task; The encoded information is reordered to generate a sub-stream of the bitstream, the sub-stream of the bitstream having the first part; According to the video coding standard, a first reconstructed image of the video is generated from the substream; Determine the recovered data from the second part; and A second reconstructed image is generated based on the first reconstructed image and the recovery data for the machine task.

2. The method according to claim 1, characterized in that, The video coding standards include at least one of the following: Audio Video Standard (AVS) 1, AVS 2, AVS 3, AVS 4, Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), Universal Video Coding (VVC), Open Media Consortium (AOMedia) Video 1 (AV 1), and AV 2.

3. The method according to claim 1, characterized in that, The determination of the recovered data includes at least one of the following: Parse the sequence-level recovery data from the second part; and Image-level recovery data is extracted from the second part.

4. The method according to claim 1, characterized in that, The determination of the recovered data includes: At least one Supplemental Enhancement Information (SEI) message is parsed to obtain the recovery data.

5. The method according to claim 1, wherein, The determination of the recovered data includes: Detecting extensions to the structure of the video coding standard; and The recovered data is obtained from the extension of the structure.

6. The method according to claim 5, characterized in that, The structure of the video coding standard includes at least one of the following: Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Picture Header (PH), and Slice Header (SH).

7. The method according to claim 1, characterized in that, The determination of the recovered data includes: Determine at least one of the following from the second part: spatial resampling parameters, redirection parameters, time recovery parameters, and bit depth shift parameters.

8. The method according to claim 1, characterized in that, The determination of the recovered data includes: Extract the image-level recovery data associated with the image sequence number (POC) from the second part; Alternatively, determining the recovered data includes: Extract the recovery data for the end of the bitstream from the second part.

9. The method according to claim 1, characterized in that, The first part includes a plurality of network abstraction layer (NAL) units, which are encapsulated as machine video coding (VCM) units of type encoded video data (CVD).

10. The method according to claim 1, characterized in that, The second part includes multiple Network Abstraction Layer (NAL) units, which are encapsulated as Machine Video Coding (VCM) units of the Recovery Data (RSD) type.

11. The method according to claim 10, characterized in that, The plurality of NAL units include at least: NAL units that include sequence-level recovery data; Alternatively, the plurality of NAL units may include at least: NAL units that include image-level recovery data.

12. The method according to claim 10, characterized in that, The plurality of NAL units include at least: NAL units corresponding to supplementary enhancement information (SEI) including the recovered data; Alternatively, the plurality of NAL units may include at least: a NAL unit comprising recovery data used at the end of the substream of the first portion.

13. A video coding method, comprising: The encoded information of the video is generated according to the video coding standard; Based on the encoded information of the video, recovery data is determined, which is used to generate reconstructed images for machine tasks; as well as The encoded information of the video is interleaved with the recovered data to form a bitstream.

14. The method according to claim 13, wherein, The recovered data includes: sequence-level recovered data and / or image-level recovered data.

15. The method according to claim 13, wherein, The bitstream includes multiple machine video coding (VCM) units, each of which is of one of the following types: encoded video data (CVD), restored data (RSD), or VCM parameter set (VPS).

16. A method for processing visual media data, the method comprising: According to the format rules, the bitstream including the visual media data is processed, wherein... The bitstream carries encoded information of video for machine tasks, the encoded information comprising an interleaved first part and a second part, the first part being encoded according to a video coding standard, and the second part including recovery data for the machine tasks; and The format rules stipulate that: The encoded information is reordered to generate a sub-stream of the bitstream, the sub-stream of the bitstream having the first part; A first reconstructed image is generated from the substream according to the video coding standard; Determine the recovered data from the second part; and A second reconstructed image for machine tasks is generated based on the first reconstructed image and the recovery data.

17. A video decoding device, characterized in that, The video decoding device is configured to perform the video decoding method according to any one of claims 1 to 12.

18. A video encoding apparatus, characterized in that, The video encoding apparatus is configured to perform the video encoding method according to any one of claims 13 to 15.

19. A method for processing video bitstreams, characterized in that, The video stream is decoded using the video decoding method according to any one of claims 1 to 12, or generated using the video encoding method according to any one of claims 13 to 15.

20. A computer-readable storage medium storing instructions, characterized in that, When executed by a computer, the instructions cause the computer to perform the method of any one of claims 1 to 16.

Images