Techniques for signal flag optimization in video bitstream

Abstract

A method includes receiving an encoded video bitstream including encoded information for one or more pictures and determining a signal flag value for a syntax element from the encoded video bitstream. The syntax element is associated with a usage state of an optimization technique for encoding one or more pictures into an encoded video bitstream by an encoder, the syntax element being configured to have at least three possible values corresponding to three usage states of the optimization technique, the first value of the syntax element indicates an unknown state of an optimization technique for encoding the one or more pictures, and the second value of the syntax element indicates a used state of the optimization technique for encoding the one or more pictures, and the third value of the syntax element indicates an unused state of an optimization technique for encoding the one or more pictures.

Description

Related Applications

[0001] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 460,564, filed on April 19, 2023, “SIGNALING OPTIMIZATION TECHNIQUES IN VIDEO BITSTREAMS,” which is incorporated herein by reference in its entirety. Technical Field

[0002] The present disclosure generally describes implementations related to video coding, including optimization of a video bitstream using signaling flags as applied by an encoder, for example, in video coding for machine scenarios. Background Art

[0003] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. To the extent the work is described in this background section, the work of the presently named inventors and aspects of the description that may not otherwise be considered prior art at the time of filing are neither explicitly nor implicitly admitted to be prior art against the present disclosure.

[0004] Image / video compression can help to transmit image / video data across different devices, storage devices, and networks with minimal quality degradation. In some examples, video codec technology can compress video based on spatial redundancy and temporal redundancy. In an example, a video codec can use a technique called intra-frame prediction, which can compress an image based on spatial redundancy. For example, intra-frame prediction can use reference data from the current picture being reconstructed for sample prediction. In another example, a video codec can use a technique called inter-frame prediction, which can compress an image based on temporal redundancy. For example, inter-frame prediction can use motion compensation to predict samples in the current picture based on a previously reconstructed picture. Motion compensation can be indicated by a motion vector (MV). Summary of the invention

[0005] Aspects of the present disclosure include methods and apparatus for video encoding / decoding.In some examples, an apparatus for video decoding includes processing circuitry.

[0006] Some aspects of the present disclosure provide a method for processing visual media data. The method includes performing conversion between a visual media file and a bitstream of the visual media data according to a format rule. The bitstream includes a first signal flag value of a first syntax element and a second signal flag value of a second syntax element, the first signal flag value and the second signal flag value are individually signaled, the first syntax element is associated with a first use state of a foreground preprocessing technique, and the second syntax element is associated with a second use state of a background preprocessing technique. The format rule specifies that: the first syntax element is constructed to have at least three possible values ​​corresponding to the three use states of the foreground preprocessing technique, the first value of the first syntax element indicates an unknown state of the foreground preprocessing technique, the second value of the first syntax element indicates a used state of the foreground preprocessing technique, and the third value of the first syntax element indicates an unused state of the foreground preprocessing technique. The second syntax element is constructed to have at least three possible values ​​corresponding to the three use states of the background preprocessing technique, the first value of the second syntax element indicates an unknown state of the background preprocessing technique, the second value of the second syntax element indicates a used state of the background preprocessing technique, and the third value of the second syntax element indicates an unused state of the background preprocessing technique. In an example, the format rule specifies that the first syntax element and the second syntax element are reset to an unknown state at the end of the persistence interval.

[0007] Some aspects of the present disclosure provide a method for video decoding. The method includes: receiving an encoded video bitstream, the encoded video bitstream including encoded information of one or more pictures; and determining a signal flag value of a syntax element according to the encoded video bitstream. The syntax element is associated with a usage state of an optimization technique used by an encoder to encode one or more pictures into the encoded video bitstream, and the syntax element is constructed to have at least three possible values ​​corresponding to three usage states of the optimization technique, the first value of the syntax element indicates an unknown state of the optimization technique used to encode one or more pictures, the second value of the syntax element indicates a used state of the optimization technique used to encode one or more pictures, and the third value of the syntax element indicates an unused state of the optimization technique used to encode one or more pictures. The method also includes processing the encoded information of one or more pictures according to the signal flag value of the syntax element associated with the optimization technique.

[0008] In some examples, the optimization techniques include at least one of the following: region of interest (ROI) preprocessing technology, foreground preprocessing technology, background preprocessing technology, temporal subsampling technology, spatial subsampling technology, quantization parameter adaptation technology for ROI, and quantization step size change technology for temporal layer.

[0009] In some examples, the syntax elements are parsed from a Supplemental Enhancement Information (SEI) message. In some examples, the syntax elements are parsed from at least one of the following: a sequence parameter set, a video parameter set, a picture parameter set, an adaptation parameter set, a picture header, a slice header, a group of blocks (GOB) header, and a group of pictures (GOP) header.

[0010] In some examples, the method includes parsing the encoded information to obtain a presence flag indicating whether a set of syntax elements for a set of optimization techniques are present in the encoded information. When the presence flag indicates that the set of syntax elements are present, the method includes parsing the encoded information to obtain values ​​of the set of syntax elements.

[0011] In some examples, the method includes setting an initial state of the optimization technique to an unknown state.

[0012] In some examples, the method includes determining a persistence interval for the optimization technique based on the persistence signaled in the encoded information, and setting a usage state of the optimization technique to an unknown state at the end of the persistence interval.

[0013] In some examples, the method includes determining a first signal flag value for a first syntax element and a second signal flag value for a second syntax element based on encoded information of one or more pictures, the first syntax element being associated with a first usage state of a foreground preprocessing technique for the one or more pictures, and the second syntax element being associated with a second usage state of a background preprocessing technique for the one or more pictures.

[0014] In some examples, a signal flag value of a syntax element indicates an unknown state, the signal flag value of the syntax element being inserted by a device configured to switch from a first source to a second source to provide an encoded video bitstream.

[0015] In some examples, the method includes parsing an update flag from a coded video bitstream, the update flag indicating whether an update to one or more optimization techniques is encoded into the coded information. When the update flag indicates that the update is encoded into the coded information, the method includes determining the specific optimization technique based on an identification of the specific optimization technique in the coded video bitstream, and determining one or more properties of the specific optimization technique based on the coded video bitstream.

[0016] Some aspects of the present disclosure provide another method for video decoding. The method includes: receiving an encoded video bitstream, the encoded video bitstream including encoded information of one or more pictures; and determining a first signal flag value of a first syntax element and a second signal flag value of a second syntax element based on the encoded video bitstream. The first signal flag value and the second signal flag value are separately signaled, the first syntax element is associated with a first usage state of a foreground preprocessing technique by an encoder that encodes one or more pictures into the encoded video bitstream, and the second syntax element is associated with a second usage state of a background preprocessing technique by the encoder. The method also includes processing the encoded information of one or more pictures based on the first signal flag value of the first syntax element and the second signal flag value of the second syntax element.

[0017] In some examples, the first syntax element is constructed to have at least two possible values ​​corresponding to two usage states of the foreground preprocessing technique, the first value of the first syntax element indicates a used state of the foreground preprocessing technique for encoding one or more pictures, and the second value of the first syntax element indicates an unused state of the foreground preprocessing technique for encoding the one or more pictures.

[0018] In some examples, the second syntax element is constructed to have at least two possible values ​​corresponding to two usage states of the background preprocessing technique, a first value of the second syntax element indicates a used state of the background preprocessing technique for encoding one or more pictures, and a second value of the second syntax element indicates a non-used state of the background preprocessing technique for encoding the one or more pictures.

[0019] In some examples, the first syntax element is constructed to have at least three possible values ​​corresponding to three usage states of the foreground preprocessing technique, the first value of the first syntax element indicates an unknown state of the foreground preprocessing technique used to encode one or more pictures, the second value of the first syntax element indicates a used state of the foreground preprocessing technique used to encode the one or more pictures, and the third value of the first syntax element indicates an unused state of the foreground preprocessing technique used to encode the one or more pictures.

[0020] In some examples, the second syntax element is constructed to have at least three possible values ​​corresponding to three usage states of the background preprocessing technique, a first value of the second syntax element indicates an unknown state of the background preprocessing technique used for encoding one or more pictures, a second value of the second syntax element indicates a used state of the background preprocessing technique used for encoding the one or more pictures, and a third value of the second syntax element indicates an unused state of the background preprocessing technique used for encoding the one or more pictures.

[0021] In some examples, the first syntax element and the second syntax element are parsed from a supplemental enhancement information (SEI) message.

[0022] In some examples, the first syntax element and the second syntax element are parsed from at least one of: a sequence parameter set, a video parameter set, a picture parameter set, an adaptation parameter set, a picture header, a slice header, a group of blocks (GOB) header, and a group of pictures (GOP) header.

[0023] In some examples, the method includes: parsing the encoded information to obtain a presence flag indicating whether a group of at least a first syntax element and a second syntax element exists in the encoded information; and when the presence flag indicates that a group of at least a first syntax element and a second syntax element exists, the method includes parsing the encoded information to obtain a first signal flag value for the first syntax element and a second signal flag value for the second syntax element.

[0024] According to another aspect of the present disclosure, an apparatus is provided. The apparatus includes a processing circuit system. The processing circuit system can be configured to perform any of the methods described for video decoding / encoding.

[0025] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions which, when executed by a computer, cause the computer to perform any of the described methods for video decoding / encoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Additional features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0027] Figure 1 is a schematic illustration of an exemplary block diagram of a video processing system (100).

[0028] Figure 2 is a schematic illustration of an exemplary block diagram of a decoder.

[0029] Figure 3 is a schematic illustration of an exemplary block diagram of an encoder.

[0030] Figure 4 A block diagram of a video processing system in a machine scenario in some examples is shown.

[0031] Figure 5 A table showing syntax and semantics that may be part of a SEI message in some examples.

[0032] Figure 6 Another table shows the syntax and semantics that may be part of a SEI message in some examples.

[0033] Figure 7 A flowchart outlining a decoding process according to some embodiments of the present disclosure is shown.

[0034] Figure 8 A flow chart outlining another decoding process according to some embodiments of the present disclosure is shown.

[0035] Fig. 9 is a schematic illustration of a computer system according to an embodiment. DETAILED DESCRIPTION

[0036] Some aspects of the present disclosure provide machine video coding techniques. In some examples, a video encoder that complies with a video coding standard such as ITU-T (International Telecommunication Union-TelecommunicationStandardization Sector, ITU-T) H.266 Recommendation (VVC (Versatile Video Coding, VVC)) is used to generate a compatible bitstream that is customized for consumption by a machine rather than a human viewer after being decoded by a compatible decoder. According to one aspect of the present disclosure, such a machine may be more concerned with certain aspects of the content than a human viewer. For example, a machine analyzing traffic conditions (e.g., in an autonomous driving scenario) cares about the distance, speed, and acceleration of a convertible being captured, but does not care about the clothes the convertible's driver is wearing, or the color of the convertible, or whether the convertible is dirty or clean.

[0037] In some examples, in order to guide the decoder and receiving system what the encoder considers relevant and what is not relevant for a particular video encoding for a machine scene, it can be helpful to express in the bitstream the form of optimization that the encoder has applied. In some related examples, the encoder uses SEI messages to indicate what optimizations were applied on the encoder side to generate the bitstream. The information of the SEI message can be used on the decoder side to determine whether there is an optimization that does not match the intended use on the decoder side (or at the receiving system).

[0038] Some aspects of the present disclosure provide signal flag optimization techniques that can address shortcomings and errors in related examples and can help implement, for example, a better video system for a machine. In some examples, signal flag optimization techniques can be applied to a video codec bitstream to indicate at least three usage states of one or more optimizations adopted by the encoder, such as a used state, an unused state, and an unknown (used / unused) state. In some examples, information about one or more optimized uses, unused or unknown can be transmitted persistently, for example, in a single picture, multiple pictures, or an encoded video sequence. In some examples, syntax elements, such as syntax elements in a supplemental enhancement information (SEI) message, etc., are constructed for signal flag optimization techniques.

[0039] Figure 1 A block diagram of a video processing system (100) in some examples is shown. The video processing system (100) is an example of an application of the disclosed subject matter, video encoders, and video decoders in a streaming environment. The disclosed subject matter can be equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, streaming services, storing compressed video on digital media including CDs (Compact Discs, CDs), DVDs (Digital Versatile Discs, DVDs), memory sticks, etc.

[0040] The video processing system (100) includes a capture subsystem (113) that may include a video source (101), such as a digital camera, that creates, for example, an uncompressed video picture stream (102). In an example, the video picture stream (102) includes samples captured by the digital camera. The video picture stream (102) is depicted as a thick line to emphasize the high amount of data when compared to the encoded video data (104) (or encoded video bitstream), which may be processed by an electronic device (120) coupled to the video source (101) including a video encoder (103). The video encoder (103) may include hardware, software, or a combination thereof to implement or embody aspects of the disclosed subject matter as described in more detail below. The encoded video data (104) (or encoded video bitstream) is depicted as a thin line to emphasize the lower data volume when compared to the video picture stream (102), which can be stored on the streaming server (105) for future use. One or more streaming client subsystems such as Figure 1The client subsystems (106) and (108) in the streaming server (105) can access the streaming server (105) to retrieve copies (107) and (109) of the encoded video data (104). The client subsystem (106) may include, for example, a video decoder (110) in an electronic device (130). The video decoder (110) decodes the incoming copy (107) of the encoded video data and creates an outgoing video picture stream (111) that can be presented on a display (112) (e.g., a display screen) or other presentation device (not depicted). In some streaming systems, the encoded video data (104), (107) and (109) (e.g., a video bitstream) can be encoded according to certain video encoding / compression standards. Examples of these standards include ITU-T (International Telecommunication Union-Telecommunication Standardization Sector, ITU-T) H.265 Recommendation. In the example, the video coding standard under development is informally referred to as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC.

[0041] Note that the electronic devices (120) and (130) may include other components (not shown). For example, the electronic device (120) may include a video decoder (not shown), and the electronic device (130) may also include a video encoder (not shown).

[0042] Figure 2 An exemplary block diagram of a video decoder (210) is shown. The video decoder (210) may be included in an electronic device (230). The electronic device (230) may include a receiver (231) (e.g., receiving circuitry). The video decoder (210) may be used to replace Figure 1 A video decoder (110) is shown in an example.

[0043] The receiver (231) may receive one or more encoded video sequences, for example, included in a bitstream, to be decoded by the video decoder (210). In an embodiment, the encoded video sequences are received one at a time, wherein the decoding of each encoded video sequence is independent of the decoding of other encoded video sequences. The encoded video sequence may be received from a channel (201), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (231) may receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams, which may be forwarded to their respective consuming entities (not depicted). The receiver (231) may separate the encoded video sequence from the other data. To prevent network jitter, a buffer memory (215) may be coupled between the receiver (231) and an entropy decoder / parser (220) (hereinafter referred to as "parser (220)"). In some applications, the buffer memory (215) is part of the video decoder (210). In other applications, the buffer memory (215) may be external to the video decoder (210) (not depicted). In still other applications, there may be a buffer memory (not depicted) external to the video decoder (210) to, for example, prevent network jitter, and there may be another buffer memory (215) internal to the video decoder (210) to, for example, handle playout timing. When the receiver (231) is receiving data from a store / forward device with sufficient bandwidth and controllability or from an isochronous network, the buffer memory (215) may not be needed, or the buffer memory (215) may be smaller. In order to use a packet network such as the Internet as much as possible, a buffer memory (215) may be needed, which may be relatively large and may advantageously have an adaptive size, and may be implemented at least in part in an operating system or similar element (not depicted) external to the video decoder (210).

[0044] The video decoder (210) may include a parser (220) to reconstruct symbols (221) from the encoded video sequence. The categories of symbols include information for managing the operation of the video decoder (210) and potential information for controlling a presentation device such as a presentation device (212) (e.g., a display screen) that is not part of the electronic device (230) but can be coupled to the electronic device (230), such as Figure 2As shown. The control information for the rendering device may be in the form of a supplemental enhancement information (SEI) message or a video usability information (VUI) parameter set fragment (not depicted). The parser (220) may parse / entropy decode the received coded video sequence. The encoding of the coded video sequence may be performed according to a video coding technique or standard, and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (220) may extract a subgroup parameter set of at least one subgroup of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a group of pictures (Group Of Picture, GOP), a picture, a tile, a slice, a macroblock, a coding unit (Coding Unit, CU), a block, a transform unit (Transform Unit, TU), a prediction unit (Prediction Unit, PU), etc. The parser (220) may also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the coded video sequence.

[0045] The parser (220) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (215) to create symbols (221).

[0046] Depending on the type of the coded video picture or a portion of the coded video picture (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors, the reconstruction of the symbol (221) may involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the coded video sequence by the parser (220). For clarity, this subgroup control information flow between the parser (220) and the following multiple units is not depicted.

[0047] In addition to the functional blocks already mentioned, the video decoder (210) can be conceptually subdivided into a number of functional units as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and may be at least partially integrated with each other. However, for the purposes of describing the disclosed subject matter, the conceptual subdivision into the following functional units is appropriate.

[0048] The first unit is a sealer / inverse transform unit (251). The sealer / inverse transform unit (251) receives quantized transform coefficients as symbols (221) from the parser (220) and control information including which transform to use, block size, quantization factor, quantization scaling matrix, etc. The sealer / inverse transform unit (251) can output blocks including sample values, which can be input into an aggregator (255).

[0049] In some cases, the output samples of the scaler / inverse transform unit (251) may belong to an intra-coded block. An intra-coded block is a block that does not use predictive information from a previously reconstructed picture, but may use predictive information from a previously reconstructed portion of the current picture. Such predictive information may be provided by an intra-picture prediction unit (252). In some cases, the intra-picture prediction unit (252) uses surrounding reconstructed information obtained from a current picture buffer (258) to generate a block of the same size and shape as the block being reconstructed. For example, the current picture buffer (258) caches partially reconstructed current pictures and / or fully reconstructed current pictures. In some cases, the aggregator (255) adds the prediction information already generated by the intra-prediction unit (252) to the output sample information as provided by the scaler / inverse transform unit (251) on a per-sample basis.

[0050] In other cases, the output samples of the scaler / inverse transform unit (251) may belong to an inter-coded and potentially motion compensated block. In such a case, the motion compensated prediction unit (253) may access the reference picture memory (257) to obtain samples for prediction. After motion compensation of the obtained samples according to the symbols (221) belonging to the block, these samples may be added to the output of the scaler / inverse transform unit (251) (in this case, referred to as residual samples or residual signals) by the aggregator (255) to generate output sample information. The address within the reference picture memory (257) from which the motion compensated prediction unit (253) obtains the predicted samples may be controlled by a motion vector, which may be obtained by the motion compensated prediction unit (253) in the form of a symbol (221), which may have, for example, an X component, a Y component, and a reference picture component. Motion compensation may also include interpolation of sample values ​​obtained from the reference picture memory (257) when using sub-sample accurate motion vectors, motion vector prediction mechanisms, etc.

[0051] The output samples of the aggregator (255) may be subjected to various loop filtering techniques in a loop filter unit (256). The video compression techniques may include in-loop filter techniques controlled by parameters included in the coded video sequence (also referred to as the coded video bitstream) and available to the loop filter unit (256) as symbols (221) from the parser (220). The video compression may also be responsive to meta-information obtained during decoding of a coded picture or a previous (in decoding order) portion of the coded video sequence, and to previously reconstructed and loop filtered sample values.

[0052] The output of the loop filter unit (256) may be a sample stream that may be output to a rendering device (212) and stored in a reference picture memory (257) for use in future inter-picture prediction.

[0053] Once fully reconstructed, certain coded pictures may be used as reference pictures for future prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture has been identified as a reference picture (e.g., by the parser (220)), the current picture buffer (258) may become part of the reference picture memory (257) and a new current picture buffer may be reallocated before starting to reconstruct a subsequent coded picture.

[0054] The video decoder (210) may perform decoding operations according to a predetermined video compression technology or standard, such as ITU-T H.265 Recommendation. The encoded video sequence may conform to the syntax specified by the video compression technology or standard used, in the sense that the encoded video sequence follows both the syntax of the video compression technology or standard and the profile as recorded in the video compression technology or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technology or standard as tools that can only be used under the profile. For compliance, the complexity of the encoded video sequence is also required to be within a range as defined by the hierarchy of the video compression technology or standard. In some cases, the hierarchy limits the maximum picture size, the maximum frame rate, the maximum reconstruction sampling rate (measured in, for example, mega samples per second), the maximum reference picture size, etc. In some cases, the limits set by the hierarchy may be further limited by the Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management of signals in the encoded video sequence.

[0055] In an embodiment, the receiver (231) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The video decoder (210) may use the additional data to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0056] Figure 3 An exemplary block diagram of a video encoder (303) is shown. The video encoder (303) is included in an electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., a transmission circuit system). The video encoder (303) can be used to replace Figure 1 A video encoder (103) in an example.

[0057] The video encoder (303) can receive video from a video source (301) (which is Figure 3 In one example, the video source (301) is not part of the electronic device (320) to receive the video samples, and the video source (301) can capture the video images to be encoded by the video encoder (303). In another example, the video source (301) is part of the electronic device (320).

[0058] The video source (301) may provide a source video sequence in the form of a digital video sample stream to be encoded by a video encoder (303), the digital video sample stream may have any suitable bit depth (e.g., 8 bits, 10 bits, 12 bits, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source (301) may be a storage device storing previously prepared videos. In a video conferencing system, the video source (301) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of separate pictures that are given motion when viewed sequentially. The pictures themselves may be organized as a spatial pixel array, wherein each pixel may include one or more samples, depending on the sampling structure, color space, etc. used. The following description focuses on samples.

[0059] According to an embodiment, the video encoder (303) can encode and compress the pictures of the source video sequence into an encoded video sequence (343) in real time or under any other time constraints as required. Implementing the appropriate encoding speed is a function of the controller (350). In some embodiments, the controller (350) controls other functional units as described below and is functionally coupled to the other functional units. For the sake of brevity, the coupling is not depicted. The parameters set by the controller (350) may include rate control related parameters (picture skipping, quantizer, lambda value of rate distortion optimization technology, ...), picture size, picture group (GOP) layout, maximum motion vector search range, etc. The controller (350) can be configured to have other suitable functions belonging to the video encoder (303) optimized for a specific system design.

[0060] In some embodiments, the video encoder (303) is configured to operate in a codec loop. As an oversimplified description, in an example, the codec loop may include a source encoder (330) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be encoded and a reference picture) and a (local) decoder (333) embedded in the video encoder (303). The decoder (333) reconstructs the symbols to create sample data in a manner similar to that which a (remote) decoder would also create. The reconstructed sample stream (sample data) is input to a reference picture memory (334). Since the decoding of the symbol stream produces a bit-accurate result that is independent of the decoder location (local or remote), the contents of the reference picture memory (334) are also bit-accurate between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction part of the encoder are exactly the same as the sample values ​​that the decoder will "see" when using prediction during decoding. This basic principle of reference picture synchronization (and the drift that occurs when synchronization cannot be maintained, such as due to channel errors) is also used in some related technologies.

[0061] The operation of the "local" decoder (333) can be combined with the "remote" decoder such as has been described above. Figure 2 The operation of the video decoder (210) described in detail is the same. However, reference is also briefly made to Figure 2 , since the symbols are available and the encoding of the symbols into an encoded video sequence by the entropy encoder (345) and the decoding of the symbols by the parser (220) can be lossless, the entropy decoding portion of the video decoder (210) including the buffer memory (215) and the parser (220) may not be fully implemented in the local decoder (333).

[0062] In an embodiment, decoder techniques other than parsing / entropy decoding present in a decoder are present in a corresponding encoder in the same or substantially the same functional form. Therefore, the disclosed subject matter focuses on decoder operation. Since encoder techniques are inverse to the fully described decoder techniques, the description of encoder techniques can be simplified. A more detailed description is provided below in certain parts.

[0063] In some examples, during operation, the source encoder (330) may perform motion compensated predictive encoding that predictively encodes an input picture with reference to one or more previously encoded pictures from a video sequence designated as "reference pictures." In this manner, the encoding engine (332) encodes the differences between pixel blocks of the input picture and pixel blocks of a reference picture, which may be selected as a prediction reference for the input picture.

[0064] The local video decoder (333) can decode the encoded video data of the picture that can be designated as the reference picture based on the symbol created by the source encoder (330). The operation of the encoding engine (332) can advantageously be a lossy process. When the encoded video data can be decoded at the video decoder ( Figure 3 When the video encoder (303) is decoded at a remote location (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (333) replicates the decoding process that may be performed by the video decoder on the reference picture and may cause the reconstructed reference picture to be stored in the reference picture memory (334). In this way, the video encoder (303) may locally store a copy of the reconstructed reference picture that has common content (absent transmission errors) with the reconstructed reference picture that will be obtained by the remote video decoder.

[0065] The predictor (335) may perform a prediction search for the encoding engine (332). That is, for a new picture to be encoded, the predictor (335) may search the reference picture memory (334) for sample data (as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc. that may be used as appropriate prediction references for the new picture. The predictor (335) may operate on a pixel block by pixel block basis to find an appropriate prediction reference. In some cases, as determined by the search results obtained by the predictor (335), the input picture may have prediction references taken from multiple reference pictures stored in the reference picture memory (334).

[0066] The controller (350) can manage encoding operations of the source encoder (330), including, for example, setting parameters and subgroup parameters for encoding video data.

[0067] The outputs of all the above mentioned functional units may be subjected to entropy coding in the entropy encoder (345). The entropy encoder (345) converts the symbols generated by the various functional units into a coded video sequence by applying lossless compression according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc. to the symbols.

[0068] The transmitter (340) may buffer the encoded video sequence as created by the entropy encoder (345) in preparation for transmission via a communication channel (360), which may be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (340) may combine the encoded video data from the video encoder (303) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0069] The controller (350) may manage the operation of the video encoder (303). During encoding, the controller (350) may assign a specific coded picture type to each coded picture, which may affect the coding techniques that may be applied to the corresponding picture. For example, a picture may generally be assigned one of the following picture types:

[0070] Intra pictures (I pictures), which can be encoded and decoded without using any other pictures in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures.

[0071] Predictive pictures (P pictures) that can be encoded and decoded using intra prediction or inter prediction that predicts sample values ​​of each block using a motion vector and a reference index.

[0072] Bidirectional predictive pictures (B pictures), which can be encoded and decoded using intra-prediction or inter-prediction that predicts the sample values ​​of each block using two motion vectors and reference indices. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for reconstruction of a single block.

[0073] The source picture may typically be spatially subdivided into a number of blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples, respectively), and encoded on a block-by-block basis. The blocks may be predictively encoded with reference to other (already encoded) blocks, which are determined by the coding allocation applied to the corresponding picture of the block. For example, blocks of an I picture may be non-predictively encoded, or may be predictively encoded (spatial prediction or intra-frame prediction) with reference to already encoded blocks of the same picture. Pixel blocks of a P picture may be predictively encoded via spatial prediction or via temporal prediction with reference to one previously encoded reference picture. Blocks of a B picture may be predictively encoded via spatial prediction or via temporal prediction with reference to one or two previously encoded reference pictures.

[0074] The video encoder (303) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T H.265 Recommendation. In the operation of the video encoder (303), the video encoder (303) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard being used.

[0075] In an embodiment, the transmitter (340) may transmit additional data along with the encoded video. The source encoder (330) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0076] Video can be captured as multiple source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often referred to simply as intra-picture prediction) exploits spatial correlations in a given picture, while inter-picture prediction exploits (temporal or other) correlations between pictures. In an example, a particular picture in encoding / decoding, which is referred to as the current picture, is divided into blocks. In the case where a block in the current picture is similar to a reference block in a reference picture that has been previously encoded in the video and is still cached, the block in the current picture can be encoded by a vector called a motion vector. The motion vector points to a reference block in a reference picture, and in the case where multiple reference pictures are used, the motion vector may have a third dimension that identifies the reference picture.

[0077] In some embodiments, a bidirectional prediction technique may be used in inter-picture prediction. According to the bidirectional prediction technique, two reference pictures are used, such as a first reference picture and a second reference picture that are both before the current picture in the video in decoding order (but may be in the past and in the future in display order, respectively). A block in the current picture may be encoded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. The block may be predicted by a combination of the first reference block and the second reference block.

[0078] In addition, merge mode techniques can be used in inter-picture prediction to improve coding efficiency.

[0079] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, according to the HEVC (High-Efficiency Video Coding, HEVC) standard, a picture in a video picture sequence is divided into coding tree units (Coding Tree Unit, CTU) for compression, and the CTUs in the picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Typically, a CTU includes three coding tree blocks (CodingTree Block, CTB), each of which is a luminance CTB and two chrominance CTBs. Each CTU can be recursively split into one or more coding units (CUs) in a quadtree. For example, a 64×64 pixel CTU can be split into a 64×64 pixel CU, or 4 32×32 pixel CUs, or 16 16×16 pixel CUs. In the example, each CU is analyzed to determine the prediction type for the CU, such as an inter-frame prediction type or an intra-frame prediction type. The CU is split into one or more prediction units (PUs) according to temporal and / or spatial predictability. Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In an embodiment, the prediction operation in the codec (encoding / decoding) is performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0080] Note that the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using any suitable technology. In an embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more processors executing software instructions.

[0081] According to some aspects of the present disclosure, video encoding may be used in a machine scenario.

[0082] Figure 4 A block diagram of a video processing system (400) in a machine context is shown in some examples. Figure 4 In the example, a video processing system (400) includes a plurality of video sources for machine consumption. For example, the video processing system (400) includes a camera (401) coupled to an encoder (402), a camera (406) coupled to an encoder (408), and a camera (407) coupled to an encoder (409).

[0083] In some examples, the encoder can be optimized for machine consumption. For example, the encoder (402) is optimized for encoding a bitstream for machine consumption. For example, the encoder (402) is coupled to a decoder (404) via a network (403), and the decoder (404) is coupled to a machine (405) that consumes the decoded video (e.g., performs further analysis, detection, etc. on the video). Thus, the encoded bitstream can be provided from the encoder (402) to the decoder (404) via the network (403), and the encoded bitstream can be decoded by the decoder (404), and the decoded video data can be further processed by the machine (405).

[0084] Note that in some examples, the machine may need to process video from different video sources. Figure 4 In the example, the video processing system (400) includes a machine (412) coupled to a decoder (411) to process videos from different video sources via a switch (410) in a network (403). By controlling the switch (410), the machine (412) coupled to the decoder (411) can process a coded bit stream from an encoder (408) or can process a coded bit stream from an encoder (409). In some examples, the machine and the decoder coupled to the machine can be referred to as a receiving system. For example, the machine (412) and the decoder (411) form a receiving system.

[0085] According to one aspect of the present disclosure, it may be advantageous for a receiving system to know what optimization techniques an encoder uses when encoding a certain amount of video. Note that various optimization techniques may be applied on the encoder side, such as region-of-interest (ROI) preprocessing, foreground preprocessing, background preprocessing, temporal subsampling, spatial subsampling, quantization parameter adaptation for ROI, quantization step size variation for temporal layers, etc.

[0086] According to one aspect of the present disclosure, background and foreground are two components that contribute significantly to the overall effect of an image. Typically, background refers to the area or space behind a subject or element in an image. For example, background serves as a foundation and acts to set a stage and create a background for a subject or element. Typically, foreground refers to the area or space that appears right in front of a subject or element. For example, foreground acts as a frame and provides a background for a subject, allowing the subject to stand out and attract the viewer's attention.

[0087] In some examples, foreground and / or background segmentation may be performed, and suitable foreground preprocessing techniques and / or background preprocessing techniques may be applied to appropriate portions. In an example, a foreground segmentation algorithm (e.g., one of the foreground preprocessing techniques) that segments a moving object from a relatively static background may be performed to extract a foreground portion of the moving object, and other suitable foreground preprocessing techniques may be applied to the foreground portion. In another example, a background subtraction algorithm (e.g., one of the background preprocessing techniques) based on training data may be performed on a mobile camera video to perform background segmentation and foreground segmentation, and additional foreground preprocessing techniques may be applied to the foreground portion and a background preprocessing technique may be applied to the background portion.

[0088] In some examples, background pre-processing techniques may be applied to change visual properties of background elements, such as color, texture, pattern, depth, perspective, etc.

[0089] In some examples, foreground preprocessing techniques may be applied for emphasis. For example, foreground preprocessing techniques may adjust various visual attributes of foreground elements, such as focus, depth of field, angle, perspective, etc.

[0090] The present disclosure uses some background preprocessing techniques and foreground preprocessing techniques as examples to illustrate the method for optimizing techniques with signal flags, and the method for optimizing techniques with signal flags can be applied to any suitable foreground preprocessing techniques and background preprocessing techniques. In some examples, the optimization technique can also be signaled. The relevant example combines the signal flag for the foreground preprocessing technique and the signal flag for the background preprocessing technique as a single optimization technique. According to the relevant example, a single syntax indicates that both the foreground preprocessing technique and the background preprocessing technique are applied or none of the foreground preprocessing technique and the background preprocessing technique is applied. According to some aspects of the present disclosure, in some examples, the optimization technique for foreground processing and the optimization technique for background processing are separately signaled. Separately signaling foreground preprocessing and background preprocessing can provide detailed information on the application of the optimization technique and can allow more optimization features. In an example, foreground preprocessing can be applied without background preprocessing. In another example, background preprocessing can be applied without foreground preprocessing. Other suitable optimization techniques can be applied separately with signal flags. In some examples, a decoder can receive an encoded video bitstream, which includes encoded information of one or more pictures. The decoder may determine a first signal flag value of a first syntax element and a second signal flag value of a second syntax element based on an encoded video bitstream, the first signal flag value and the second signal flag value being separately signaled, the first syntax element being associated with a first usage state of a foreground preprocessing technique by an encoder encoding one or more pictures into the encoded video bitstream, and the second syntax element being associated with a second usage state of a background preprocessing technique by the encoder. The decoder and / or the machine may process the encoded information of the one or more pictures based on the first signal flag value of the first syntax element and the second signal flag value of the second syntax element.

[0091] According to some aspects of the present disclosure, for an optimization technique, it may be advantageous to signal one of at least three usage states of the optimization technique in a bitstream. The three usage states of the optimization technique may be referred to as a used state, an unused state, and an unknown state. For example, a bitstream may include one of the three usage states, and a decoder may decode the usage state signaled in the bitstream, and then operate accordingly. For example, a decoder may determine a signal flag value of a syntax element based on an encoded video bitstream, the syntax element being associated with a usage state of an optimization technique used by an encoder to encode one or more pictures into the encoded video bitstream, the syntax element being constructed to have at least three possible values ​​corresponding to the three usage states of the optimization technique, the first value of the syntax element indicating an unknown state of the optimization technique used to encode one or more pictures, the second value of the syntax element indicating a used state of the optimization technique used to encode one or more pictures, and the third value of the syntax element indicating an unused state of the optimization technique used to encode one or more pictures.

[0092] In some examples, the first state of the optimization technique (also referred to as the used state) can indicate that the optimization technique has been used on the encoder side. In an example, when such information is received and decoded by a receiving system, the decoder of the receiving system can rely on the knowledge that the optimization technique has been used and react appropriately thereto.

[0093] exist Figure 4 In the example, based on the signal flag of the first state, the decoder (404) has established knowledge that the quantization parameters have been adjusted by the encoder (402) based on the region of interest, and the decoder (404) can use the established knowledge as a heuristic method to guide the priority of the image analysis tool running in the machine (405) to the region of interest. For example, in a traffic supervision scenario, in the video encoding for the machine system, the encoder (402) encodes the identified license plate as a region of interest (ROI) and encodes the ROI with particularly fine quantization to preserve details. At the receiving system, the decoder (404) can use the information of the ROI to guide the optical character recognition (OCR) module in the machine (405) to the reconstructed picture area encoded with particularly fine quantization. Therefore, the machine (405) can recognize the characters of the license plate.

[0094] In some examples, the second state (also referred to as the unused state) may indicate that no optimization technique has been used on the encoder side. In an example, based on the signal flag of the second state, the decoder (404) may determine not to rely on the knowledge of the used optimization technique. According to an aspect of the present disclosure, it may be disadvantageous for the decoder (404) to draw special attention of a license plate OCR reader in a machine (405) to a picture area that has been encoded with a fine QP (Quantizer Parameter, QP) because such a fine QP is not indicative of the identified region of interest. Therefore, using the signal flag of the second state, such disadvantage of the decoder (404) may be avoided.

[0095] In some examples, a third state (also referred to as an unknown state) may indicate that the used state or unused state of the optimization technique is unknown. In some examples, the optimization technique is transmitted in the bitstream by an optional syntax mechanism, such as using an SEI message, and the third state may be a default in the decoder of the receiving system until more explicit information according to the first state or the second state is received. According to one aspect of the present disclosure, for example, in a machine scenario, it may be beneficial to specifically signal an unknown state. For example, a camera (406) and an encoder (408) coupled to the camera (406) are in a traditional traffic camera system, and a camera (407) and an encoder (409) coupled to the camera (407) are in a modern traffic camera system. In the example, the traditional traffic camera system does not support signaling of the optimization technique, while the modern traffic camera system supports signaling of the optimization technique. The switch (410) is a video switch in the compression domain and can be controlled to switch between the traditional traffic camera system and the modern traffic camera system. In an example, when the switch (410) switches from a modern traffic camera system to a traditional traffic camera system, the knowledge established at the decoder (411) may be a state left behind by the modern traffic camera system. The traditional traffic camera system does not support signaling of optimization techniques, and the bitstream generated by the traditional traffic camera system does not include inserted information about the optimization techniques. In some examples, the switch (410) may be configured to insert something into the bitstream. For example, since the switch (410) is a compression domain device and does not have knowledge of which optimization techniques have been used by the traditional traffic camera system, nor does it have a way to derive such knowledge, the switch (410) may insert information indicating that the optimization technique is unknown (e.g., an unknown state). Note that in an example, inserting other information such as not using an optimization technique may mislead the receiving system. The switch (410) may be any suitable device having the ability to insert appropriate information into the bitstream. The switch (410) may be a network switch, a server, and may be a device having a processing circuit system and software running on the processing circuit system.

[0096] In some embodiments, at least three states described above may be included in the bitstream with each optimization technique scenario. For example, in a traffic camera scenario, multiple traffic camera systems may be used, such as a conventional traffic camera system, a first generation traffic camera system, and a second generation traffic camera system. The second generation traffic camera system may provide information about more optimization techniques than the first generation traffic camera system, while the conventional traffic camera system does not provide such information.

[0097] According to one aspect of the present disclosure, when the video bitstream syntax includes only two states (e.g., a used state and an unused state) for indicating that optimization is on or off, the receiving system may receive misleading information and perform in a significantly disadvantageous manner as described above. In some examples, the switch (410) is a video switch and syntax regarding unknown states of the optimization technique may be inserted when the switch (410) switches between video sources. Therefore, on the receiving system side, the receiving system may look for other information to make the right decision regarding the optimization technique. In some examples, the three-state model is unacceptable for compression or other reasons, and a partial solution may be used. In the partial solution, a single bit with two values ​​is used, one of the two values, such as 1, indicates that the optimization technique has been used, and the other value, such as 0, indicates that the use of the optimization technique is unknown. Note that "Unknown" is semantically different from "off".

[0098] According to some video coding standards such as VVC, the on / off information can be encoded by a single flag occupying one bit in the bitstream, which is usually denoted as u(1). Information with three values ​​(0..2) (e.g., three states) can be encoded, for example, as a two-bit fixed-length codeword (u(2)), or can be encoded using a variable-length codeword with, for example, an extended Golomb (ext-golomb) code that can be denoted as ue(v).

[0099] In some examples, the used state, unused state, and unknown state of the optimization technique may be described by optimization technique. For example, for each optimization technique, a value of "0" indicates an unknown state, a value of "1" indicates a used state, and a value of "2" indicates an unused state. In an example, these values ​​may be represented by a codeword encoded by ue(v). In another example, these values ​​may be represented by a codeword encoded by u(2).

[0100] According to one aspect of the present disclosure, the syntax of a supplemental enhancement information (SEI) message can be used to signal optimization techniques. In an example, the syntax uses a ue(v) encoded codeword for a three-state optimization technique signal flag. In another example, the syntax uses a u(1) single-bit signal flag to distinguish between "on" and "unknown". Note that the SEI message is described as an example of information including optimization techniques, which can be included at other locations in the bitstream. In the example, the optimization technique information is part of the visual availability information syntax (part of a sequence parameter set), which is located in any one of a slice header, a picture header, a GOB (Group of Block, GOB) header, a GOP header, or in any parameter set including a picture parameter set, an adaptation parameter set, a sequence parameter set, and a video parameter set.

[0101] In some embodiments, to save bits in the syntax, signal flags for semantically similar optimization techniques can be grouped in a group, where each group is gated by the presence flag. In the example, a single group is used to group optimization techniques related to a region of interest. Note that other suitable groupings can be derived.

[0102] In some embodiments, the presence or absence of an optimization technique signal flag is indicated by a flag.

[0103] In some embodiments, before receiving the first SEI message (e.g., at the beginning of the encoded video sequence), the initial state of the optimization information at the decoder can be set to an unknown state for each of the optimization techniques. Such a mechanism is most likely to achieve cross-vendor interoperability.

[0104] In some embodiments, before receiving the first SEI message, the initial state of the optimization information at the decoder can be "on" or "off" for all optimization techniques or a subset of optimization techniques. In some examples, predefining such information can save bitrate because the SEI message does not necessarily need to be sent. However, doing so requires pre-established knowledge between the encoder and the decoder about what optimization technique the encoder is to use. The pre-established knowledge can be established through a signal flag outside the video bitstream (e.g., during capability exchange), through a cross-vendor agreement, etc.

[0105] In some embodiments, the SEI message includes persistence information indicating that the information included in the SEI message persists across multiple pictures, access units, time, blocks / CTUs, coded video sequences (CVS), or other specified units of the bitstream. In some examples, a single bit is used to indicate that the SEI message belongs to a picture (of a single-layer bitstream) / access unit (of a multi-layer bitstream) (as the case may be), or to the remainder of the coded video sequence.

[0106] In some examples, when the signaled persistence is less than the encoded video sequence, at the end of the persistence interval, the state of the optimization technique at the decoder is designated as an unknown state.

[0107] Figure 5 A table showing syntax and semantics that may be part of a SEI message in some examples.

[0108] exist Figure 5In the example, the flag optimization_information_present_flag equal to 1 specifies that optimization information is present and will be applied as indicated by the persistence information. The flag optimization_information_present_flag equal to 0 specifies that optimization information is not present and the decoder / receiver needs to be assumed to be in an unknown state.

[0109] exist Figure 5 In the example, the flag roi_optimization_present_flag equal to 1 specifies that roi_pre_processing and roi_quanty_adaptaion are present. The flag roi_optimization_present_flag equal to 0 specifies that roi_pre_processing and roi_quanty_adaptaion are not present.

[0110] exist Figure 5 In the example, the syntaxes roi_pre_processing, roi-quant_adapation, foreground_preprocessing, background_preprocessing, temporal_subsampling, spatial_subsampling, and temp_layer_qp_change are used to indicate the status of the optimization technique, respectively. The syntax roi_pre_processing is used to indicate the status of ROI preprocessing, the syntax roi-quant_adapation is used to indicate the status of ROI-based quantization adaptation, the syntax foreground_preprocessing is used to indicate the status of foreground preprocessing, the syntax background_preprocessing is used to indicate the status of background preprocessing, the syntax temporal_subsampling is used to indicate the status of temporal subsampling, the syntax spatial_subsampling is used to indicate the status of spatial subsampling, and the syntax temp_layer_qp_change is used to indicate the temporal layer quantization parameter change. In the example, a value of 0 for the syntax for the optimization technique specifies that the use of the optimization technique is unknown; a value of 1 for the syntax for the optimization technique specifies that the optimization technique is in use; and a value of 2 for the syntax for the optimization technique specifies that the optimization technique is not in use. Note that other values ​​of the syntax may be reserved.

[0111] exist Figure 5In the example, the persistence flag optimization_persistence_flag specifies the persistence of the optimization information indicated by the syntax element, which is gated by the presence flag optimization_information_present_flag. The persistence flag optimization_persistence_flag equal to 0 specifies that the optimization information applies only to the current picture. The optimization information can be reset to an unknown state until an optimization SEI message (e.g., encoder_optimization_info SEI message) is received with optimization_information_present_flag equal to 1. The persistence flag optimization_persistence_flag equal to 1 specifies that the optimization method identified by optimization_method applies to the current picture and all subsequent pictures.

[0112] Figure 6 Another table showing the syntax and semantics that may be part of an SEI message in some examples. Figure 6 In the example, the flag optimization_information_present_flag equal to 1 specifies that the optimization information is updated for the current picture. The flag optimization_information_present_flag equal to 0 specifies that the optimization information is not updated, and the optimization method and the properties associated with the optimization method applied to the previously encoded picture are used for the current picture. When this flag is used for the first time, the value can be set to 1.

[0113] exist Figure 6 In the example, the syntax "optimization_method_id" specifies the index of the optimization method to be applied to the current image. When the value of this syntax is equal to 0, no optimization is applied. The syntax attributes can be used to specify the attribute values ​​of a certain optimization method, such as resampling rate, bounding box and ROI coordinates.

[0114] Figure 7A flow chart summarizing a process (700) according to an embodiment of the present disclosure is shown. The process (700) may be used in a video decoder. In various embodiments, the process (700) is performed by a processing circuit system, such as a processing circuit system that performs the functions of a video decoder (110), a processing circuit system that performs the functions of a video decoder (210), a processing circuit system that performs the functions of a video decoder (404), a processing circuit system that performs the functions of a video decoder (411), etc. In some embodiments, the process (700) is implemented in software instructions, so when the processing circuit system executes the software instructions, the processing circuit system performs the process (700). The process starts at (S701) and proceeds to (S710).

[0115] At (S710), an encoded video bitstream is received, the encoded video bitstream including encoded information of one or more pictures.

[0116] At (S720), a signal flag value of a syntax element is determined according to the encoded video bitstream. The syntax element is associated with a usage state of an optimization technique used by an encoder to encode one or more pictures into the encoded video bitstream. The syntax element is constructed to have at least three possible values ​​corresponding to three usage states of the optimization technique, a first value of the syntax element indicating an unknown state of the optimization technique used to encode the one or more pictures, a second value of the syntax element indicating a used state of the optimization technique used to encode the one or more pictures, and a third value of the syntax element indicating an unused state of the optimization technique used to encode the one or more pictures.

[0117] At (S730), the encoded information of one or more pictures is processed according to the signal flag value of the syntax element associated with the optimization technique. In an example, the usage status of the optimization technique is provided to the machine to perform further video processing accordingly.

[0118] In some examples, the optimization techniques include at least one of the following: region of interest (ROI) preprocessing technology, foreground preprocessing technology, background preprocessing technology, temporal subsampling technology, spatial subsampling technology, quantization parameter adaptation technology for ROI, and quantization step size change technology for temporal layer.

[0119] In some examples, syntax elements are parsed from a supplemental enhancement information (SEI) message.

[0120] In some examples, the syntax element is parsed from at least one of: a sequence parameter set, a video parameter set, a picture parameter set, an adaptation parameter set, a picture header, a slice header, a group of blocks (GOB) header, and a group of pictures (GOP) header.

[0121] In some examples, to determine the signal flag value of the syntax element, the encoded information is parsed to obtain a presence flag indicating whether a set of syntax elements for a set of optimization techniques are present in the encoded information. When the presence flag indicates that the set of syntax elements are present, the encoded information is parsed to obtain the value of the set of syntax elements.

[0122] In some examples, an initial state of the optimization technique is set to an unknown state.

[0123] In some examples, a persistence interval of the optimization technique is determined based on a persistence signaled in the encoded information. Then, the use state of the optimization technique is set to an unknown state at the end of the persistence interval.

[0124] In some examples, a first signal flag value for a first syntax element and a second signal flag value for a second syntax element are determined based on encoded information of one or more pictures, the first syntax element is associated with a first usage state of a foreground preprocessing technique for the one or more pictures, and the second syntax element is associated with a second usage state of a background preprocessing technique for the one or more pictures.

[0125] In some examples, a signal flag value of a syntax element indicates an unknown state, the signal flag value of the syntax element being inserted by a device configured to switch from a first source to a second source to provide an encoded video bitstream.

[0126] In some examples, an update flag is parsed from a coded video bitstream, the update flag indicating whether updates to one or more optimization techniques are encoded into the coded information. When the update flag indicates that the updates are encoded into the coded information, the specific optimization technique is determined based on an identification of the specific optimization technique in the coded video bitstream, and one or more properties of the specific optimization technique are determined based on the coded video bitstream.

[0127] Then, the process proceeds to (S799) and terminates.

[0128] The process (700) may be adjusted appropriately. Steps in the process (700) may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0129] Figure 8A flow chart outlining a process (800) according to an embodiment of the present disclosure is shown. The process (800) may be used for a video decoder. In various embodiments, the process (800) is performed by a processing circuit system, such as a processing circuit system that performs the functions of a video decoder (110), a processing circuit system that performs the functions of a video decoder (210), etc. In some embodiments, the process (800) is implemented in software instructions, so that when the processing circuit system executes the software instructions, the processing circuit system performs the process (800). The process starts at (S801) and proceeds to (S810).

[0130] At (S810), an encoded video bitstream is received, the encoded video bitstream including encoded information of one or more pictures.

[0131] At (S820), a first signal flag value for a first syntax element and a second signal flag value for a second syntax element are determined based on the encoded video bitstream, the first signal flag value and the second signal flag value are individually signaled, the first syntax element is associated with a first usage state of a foreground preprocessing technique by an encoder that encodes one or more pictures into the encoded video bitstream, and the second syntax element is associated with a second usage state of a background preprocessing technique by the encoder.

[0132] At (S830), the encoded information of one or more pictures is processed according to the first signal flag value of the first syntax element and the second signal flag value of the second syntax element. In some examples of machine scenes, the first use state of foreground processing and the second use state of background preprocessing are provided to the machine for further video processing.

[0133] In some examples, a first syntax element is constructed to have at least two possible values ​​corresponding to two usage states of a foreground preprocessing technique, a first value of the first syntax element indicating a used state of the foreground preprocessing technique for encoding one or more pictures, and a second value of the first syntax element indicating an unused state of the foreground preprocessing technique for encoding one or more pictures.

[0134] In some examples, the second syntax element is constructed to have at least two possible values ​​corresponding to two usage states of the background preprocessing technique, a first value of the second syntax element indicates a used state of the background preprocessing technique for encoding one or more pictures, and a second value of the second syntax element indicates a non-used state of the background preprocessing technique for encoding the one or more pictures.

[0135] In some examples, the first syntax element is constructed to have at least three possible values ​​corresponding to three usage states of the foreground preprocessing technique, the first value of the first syntax element indicates an unknown state of the foreground preprocessing technique used to encode one or more pictures, the second value of the first syntax element indicates a used state of the foreground preprocessing technique used to encode the one or more pictures, and the third value of the first syntax element indicates an unused state of the foreground preprocessing technique used to encode the one or more pictures.

[0136] In some examples, the second syntax element is constructed to have at least three possible values ​​corresponding to three usage states of the background preprocessing technique, a first value of the second syntax element indicates an unknown state of the background preprocessing technique used for encoding one or more pictures, a second value of the second syntax element indicates a used state of the background preprocessing technique used for encoding the one or more pictures, and a third value of the second syntax element indicates an unused state of the background preprocessing technique used for encoding the one or more pictures.

[0137] In some examples, the first syntax element and the second syntax element are parsed from a supplemental enhancement information (SEI) message.

[0138] In some examples, the first syntax element and the second syntax element are parsed from at least one of: a sequence parameter set, a video parameter set, a picture parameter set, an adaptation parameter set, a picture header, a slice header, a group of blocks (GOB) header, and a group of pictures (GOP) header.

[0139] In some examples, the encoded information is parsed to obtain a presence flag indicating whether a group of at least a first syntax element and a second syntax element is present in the encoded information. When the presence flag indicates that the group of at least the first syntax element and the second syntax element is present, the encoded information is parsed to obtain a first signal flag value for the first syntax element and a second signal flag value for the second syntax element.

[0140] Then, the process proceeds to (S899) and terminates.

[0141] The process (800) may be adjusted appropriately. Steps in the process (800) may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0142] Note that the above technique can be used to perform conversion between a visual media file and a bitstream of visual media data according to a format rule. For example, the bitstream includes a first signal flag value of a first syntax element and a second signal flag value of a second syntax element, the first signal flag value and the second signal flag value are individually signaled, the first syntax element is associated with a first usage state of a foreground preprocessing technique, and the second syntax element is associated with a second usage state of a background preprocessing technique. The format rule specifies that the first syntax element is constructed to have at least three possible values ​​corresponding to the three usage states of the foreground preprocessing technique, the first value of the first syntax element indicates an unknown state of the foreground preprocessing technique, the second value of the first syntax element indicates a used state of the foreground preprocessing technique, and the third value of the first syntax element indicates an unused state of the foreground preprocessing technique. In addition, the format rule specifies that the second syntax element is constructed to have at least three possible values ​​corresponding to the three usage states of the background preprocessing technique, the first value of the second syntax element indicates an unknown state of the background preprocessing technique, the second value of the second syntax element indicates a used state of the background preprocessing technique, and the third value of the second syntax element indicates an unused state of the background preprocessing technique.

[0143] In an example, the format rule specifies that the first syntax element and the second syntax element are reset to an unknown state at the end of the persistence interval.

[0144] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. Fig. 9 A computer system (900) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0145] Computer software may be encoded using any suitable machine code or computer language, which may be subjected to mechanisms such as assembly, compilation, linking, etc. to create code comprising instructions that may be executed directly by one or more computer central processing units (CPU), graphics processing units (GPU), etc., or by interpretation, microcode execution, etc.

[0146] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, IoT devices, and the like.

[0147] Fig. 9The components for the computer system (900) shown in the example are exemplary in nature and are not intended to suggest any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. Nor should the configuration of components be interpreted as having any dependency or requirement related to any one or combination of components shown in the exemplary embodiment of the computer system (900).

[0148] Computer system 900 may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, tapping), visual input (e.g., gestures), olfactory input (not depicted). Human interface devices may also be used to capture certain media that are not necessarily directly related to conscious input by humans, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and videos (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0149] Input human-machine interface devices may include one or more of the following (only one of each is depicted): keyboard (901), mouse (902), touchpad (903), touch screen (910), data gloves (not shown), joystick (905), microphone (906), scanner (907), camera (908).

[0150] The computer system (900) may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate one or more senses of a human user through, for example, tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include: tactile output devices (e.g., tactile feedback through a touch screen (910), a data glove (not shown), or a joystick (905), but there may also be tactile feedback devices that are not used as input devices); audio output devices (e.g., speakers (909), headphones (not depicted)); visual output devices (e.g., screens (910), including CRT (Cathode Ray Tube) screens, LCD (Liquid Crystal Display, LCD) screens, plasma screens, OLED (Organic Light-Emitting Diode, OLED) screens, each screen with or without touch screen input capability, each screen with or without tactile feedback capability - some of the screens may be able to output two-dimensional visual output or more than three-dimensional output through means such as stereo output; virtual reality glasses (not depicted); holographic displays and smoke generators (not depicted); and printers (not depicted).

[0151] The computer system (900) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM (Read Only Memory, ROM) / RW (920) with CD / DVD and other media (921), thumb drives (922), removable hard drives or solid-state drives (923), traditional magnetic media such as magnetic tapes and floppy disks (not depicted), dedicated ROM / ASIC (Application Specific Integrated Circuit, ASIC) / PLD based (Programable Logic Device, PLD) devices such as security dongles (not shown).

[0152] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0153] The computer system (900) may also include an interface (954) to one or more communication networks (955). The network may be, for example, wireless, wired, optical. The network may also be local area, wide area, metropolitan area, vehicle-mounted and industrial, real-time, delay-tolerant, etc. Examples of networks include: local area networks such as Ethernet; wireless LAN (Local Area Network, LAN); cellular networks including GSM (Global System for Mobile Communications, GSM), 3G (the Third Generation, 3G), 4G (the Fourth Generation, 4G), 5G (the Fifth Generation, 5G), LTE (Long Term Evolution, LTE), etc.; TV wired or wireless wide area digital networks including cable TV, satellite TV and terrestrial broadcast TV; including CANBus (Controller Area Network-BUS, CANBus) vehicle and industrial networks, etc. Some networks typically require an external network interface adapter attached to some universal data port or peripheral bus (949) (such as, for example, the USB (Universal Serial Bus, USB) port of the computer system (900)); other networks are typically integrated into the core of the computer system (900) by attaching to a system bus as described below (for example, to an Ethernet interface in a PC (Personal Computer, PC) computer system or to a cellular network interface in a smart phone computer system). The computer system (900) can use any of these networks to communicate with other entities. Such communication can be one-way reception only (for example, broadcast TV), one-way transmission only (for example, a CAN bus to certain CAN bus devices), or bidirectional, for example, to other computer systems using a local area digital network or a wide area digital network. As described above, certain protocols and protocol stacks can be used on each of these networks and network interfaces.

[0154] The aforementioned human-machine interface devices, human-accessible storage devices, and network interfaces may be attached to the core ( 940 ) of the computer system ( 900 ).

[0155] The core (940) may include one or more central processing units (CPUs) (941), graphics processing units (GPUs) (942), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (943), hardware accelerators for certain tasks (944), graphics adapters (950), etc. These devices, along with read-only memory (ROM) (945), random access memory (946), internal mass storage devices (947) such as internal non-user accessible hard drives, SSDs (Solid-State Drives, SSDs), etc., may be connected via a system bus (948). In some computer systems, the system bus (948) may be accessible in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices may be directly attached to the core's system bus (948) or may be attached to the core's system bus (948) via a peripheral bus (949). In an example, a screen (910) may be connected to a graphics adapter (950). The architecture of the peripheral bus includes PCI (Peripheral Component Interconnect, PCI), USB, etc.

[0156] The CPU (941), GPU (942), FPGA (943) and accelerator (944) can execute certain instructions, which in combination can constitute the computer code mentioned above. The computer code can be stored in ROM (945) or RAM (946). Transient data can also be stored in RAM (946), while permanent data can be stored in, for example, an internal mass storage device (947). Fast storage and retrieval of any of the memory devices can be achieved by using a cache memory, which can be used with one or more CPUs (941), GPUs, or other memory devices. (942), large-capacity storage device (947), ROM (945), RAM (946), etc. are closely related.

[0157] The computer readable medium may have computer codes thereon for performing various computer-implemented operations. The media and computer codes may be those specially designed and constructed for the purposes of the present disclosure, or they may be of a type well known and available to those skilled in the art of computer software.

[0158] As an example and not a limitation, a computer system (900) having an architecture and in particular a core (940) can provide functionality due to a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software contained in one or more tangible computer-readable media. Such a computer-readable medium can be a medium associated with a user-accessible mass storage device as described above, and certain storage devices of the core (940) with non-transient properties, such as a mass storage device (947) or ROM (945) inside the core. Software implementing various embodiments of the present disclosure can be stored in such a device and executed by the core (940). Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software can enable the core (940) and in particular the processor therein (including a CPU, GPU, FPGA, etc.) to perform a specific process or a specific part of a specific process described herein, including defining a data structure stored in a RAM (Random Access Memory, RAM) (946) and modifying such a data structure according to a process defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (944)) that may replace or operate in conjunction with software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic, and references to logic may include software. Where appropriate, references to computer-readable media may include circuitry (e.g., an integrated circuit (IC)) storing software for execution, circuitry implementing logic for execution, or both. The present disclosure includes any suitable combination of hardware and software.

[0159] The use of "at least one of" or "one of" in the present disclosure is intended to include any one or combination of the elements. For example, references to at least one of A, B, or C; at least one of A, B, and C; at least one of A, B, and / or C; and at least one of A through C are intended to include only A, only B, only C, or any combination thereof. References to one of A or B and one of A and B are intended to include A or B or (A and B). The use of "one of" does not exclude any combination of the recited elements where applicable, such as when the elements are not mutually exclusive.

[0160] Although the present disclosure has described several exemplary embodiments, there are changes, permutations, and various alternative equivalents that fall within the scope of the present disclosure. It will therefore be appreciated that those skilled in the art will be able to conceive of many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.

Claims

1. A method for processing visual media data, the method comprising: Conversion between the visual media file and the bitstream of visual media data is performed according to format rules, wherein: The bitstream includes a first signal flag value for a first syntax element and a second signal flag value for a second syntax element, the first signal flag value and the second signal flag value being individually signaled, the first syntax element being associated with a first usage state of a foreground pre-processing technique, the second syntax element being associated with a second usage state of a background pre-processing technique; and The format rules specify: The first syntax element is constructed to have at least three possible values ​​corresponding to three usage states of the foreground pre-processing technique, a first value of the first syntax element indicating an unknown state of the foreground pre-processing technique, a second value of the first syntax element indicating a used state of the foreground pre-processing technique, and a third value of the first syntax element indicating an unused state of the foreground pre-processing technique; and The second syntax element is constructed to have at least three possible values ​​corresponding to three usage states of the background pre-processing technique, a first value of the second syntax element indicates an unknown state of the background pre-processing technique, a second value of the second syntax element indicates a used state of the background pre-processing technique, and a third value of the second syntax element indicates an unused state of the background pre-processing technique.

2. The method according to claim 1, wherein: The format rule specifies that the first syntax element and the second syntax element are reset to the unknown state at the end of a persistence interval.

3. A video decoding method, comprising: Receiving a coded video bitstream, the coded video bitstream comprising coded information of one or more pictures; determining, according to the coded video bitstream, a signal flag value of a syntax element, the syntax element being associated with a use state of an optimization technique used by an encoder to encode the one or more pictures into the coded video bitstream, the syntax element being constructed to have at least three possible values ​​corresponding to three use states of the optimization technique, a first value of the syntax element indicating an unknown state of the optimization technique used to encode the one or more pictures, a second value of the syntax element indicating a used state of the optimization technique used to encode the one or more pictures, and a third value of the syntax element indicating an unused state of the optimization technique used to encode the one or more pictures; as well as The encoded information of the one or more pictures is processed according to the signal flag value of the syntax element associated with the optimization technique.

4. The method according to claim 3, wherein: The optimization techniques include at least one of the following: Region of interest (ROI) preprocessing techniques; Foreground preprocessing techniques; Background preprocessing technology; Temporal subsampling techniques; Spatial subsampling technology; ROI-specific quantization parameter adaptive technology; as well as Quantization step size variation technique for temporal layers.

5. The method according to any one of claims 3 to 4, wherein: The syntax element is parsed from a Supplemental Enhancement Information (SEI) message.

6. The method according to any one of claims 3 to 5, wherein: The syntax element is parsed from at least one of: a sequence parameter set, a video parameter set, a picture parameter set, an adaptation parameter set, a picture header, a slice header, a group of blocks (GOB) header, and a group of pictures (GOP) header.

7. The method according to any one of claims 3 to 6, wherein: Determining the signal flag value of the syntax element further includes: parsing the encoded information to obtain a presence flag indicating whether a set of syntax elements for a set of optimization techniques exists in the encoded information; and When the presence flag indicates that the set of syntax elements exists, the encoded information is parsed to obtain values ​​of the set of syntax elements.

8. The method according to any one of claims 3 to 7, further comprising: An initial state of the optimization technique is set to the unknown state.

9. The method according to any one of claims 3 to 8, further comprising: determining a persistence interval for the optimization technique based on the persistence signaled in the encoded information; as well as The usage state of the optimization technique is set to the unknown state at the end of the persistence interval.

10. The method according to any one of claims 3 to 9, further comprising: A first signal flag value of a first syntax element and a second signal flag value of a second syntax element are determined according to encoded information of the one or more pictures, the first syntax element being associated with a first usage state of a foreground pre-processing technique for the one or more pictures, and the second syntax element being associated with a second usage state of a background pre-processing technique for the one or more pictures.

11. The method according to any one of claims 3 to 10, wherein: The signal flag value of the syntax element indicates the unknown state, the signal flag value of the syntax element being inserted by a device configured to switch from a first source to a second source to provide the coded video bitstream.

12. The method according to any one of claims 3 to 11, further comprising: parsing an update flag from the encoded video bitstream, the update flag indicating whether updates to one or more optimization techniques are encoded into the encoded information; When the update flag indicates that the update is encoded into the encoded information, determining the specific optimization technique based on an identification of the specific optimization technique in the encoded video bitstream; as well as One or more properties of the particular optimization technique are determined based on the encoded video bitstream.

13. The method according to claim 10, wherein: The foreground preprocessing technique adjusts visual properties of foreground elements, and the background preprocessing technique adjusts visual properties of background elements.

14. The method according to claim 3, wherein: The unknown state of the optimization technique indicates that it is unknown whether the optimization technique is used to encode the one or more pictures, the used state of the optimization technique indicates that the optimization technique is used to encode the one or more pictures, and the unused state of the optimization technique indicates that the optimization technique is not used to encode the one or more pictures.

15. An apparatus for video decoding, comprising processing circuitry, the processing circuitry being configured to: Receiving a coded video bitstream, the coded video bitstream comprising coded information of one or more pictures; determining, based on the coded video bitstream, a first signal flag value for a first syntax element and a second signal flag value for a second syntax element, the first signal flag value and the second signal flag value being individually signaled, the first syntax element being associated with a first usage state of a foreground pre-processing technique by an encoder that encodes the one or more pictures into the coded video bitstream, and the second syntax element being associated with a second usage state of a background pre-processing technique by the encoder; and The encoded information of the one or more pictures is processed according to the first signal flag value of the first syntax element and the second signal flag value of the second syntax element.

16. The device according to claim 15, wherein: The first syntax element is constructed to have at least two possible values ​​corresponding to two usage states of the foreground preprocessing technique, a first value of the first syntax element indicating a used state of the foreground preprocessing technique for encoding the one or more pictures, and a second value of the first syntax element indicating a non-used state of the foreground preprocessing technique for encoding the one or more pictures.

17. The device according to any one of claims 15 to 16, wherein: The second syntax element is constructed to have at least two possible values ​​corresponding to two usage states of the background pre-processing technique, a first value of the second syntax element indicates a used state of the background pre-processing technique for encoding the one or more pictures, and a second value of the second syntax element indicates a non-used state of the background pre-processing technique for encoding the one or more pictures.

18. The device according to any one of claims 15 to 16, wherein: The first syntax element is constructed to have at least three possible values ​​corresponding to three usage states of the foreground preprocessing technique, a first value of the first syntax element indicating an unknown state of the foreground preprocessing technique used to encode the one or more pictures, a second value of the first syntax element indicating a used state of the foreground preprocessing technique used to encode the one or more pictures, and a third value of the first syntax element indicating an unused state of the foreground preprocessing technique used to encode the one or more pictures.

19. The device according to any one of claims 15 to 18, wherein: The second syntax element is constructed to have at least three possible values ​​corresponding to three usage states of the background preprocessing technique, a first value of the second syntax element indicates an unknown state of the background preprocessing technique used for encoding the one or more pictures, a second value of the second syntax element indicates a used state of the background preprocessing technique used for encoding the one or more pictures, and a third value of the second syntax element indicates an unused state of the background preprocessing technique used for encoding the one or more pictures.

20. The device according to any one of claims 15 to 19, wherein: The first syntax element and the second syntax element are parsed from a supplemental enhancement information (SEI) message.

21. The device according to any one of claims 15 to 20, wherein: The first syntax element and the second syntax element are parsed from at least one of: a sequence parameter set, a video parameter set, a picture parameter set, an adaptation parameter set, a picture header, a slice header, a group of blocks (GOB) header, and a group of pictures (GOP) header.

22. The device according to any one of claims 15 to 21, wherein The processing circuit system is configured to: parsing the encoded information to obtain an existence flag, the existence flag indicating whether a group of at least the first syntax element and the second syntax element exists in the encoded information; as well as When the presence flag indicates that at least the group of the first syntax element and the second syntax element exists, the encoded information is parsed to obtain the first signal flag value of the first syntax element and the second signal flag value of the second syntax element.

Images