Method and apparatus for signaling use of ai-driven processes on video in supplemental enhancement information messages

Abstract

Embodiments of the present application provide a method for video encoding and video decoding based on a video, an image and an SEI message, the SEI message being associated with the image of the video and including a syntax element, the syntax element indicating whether the image is allowed to be used as an input by an AI engine.

Description

Cross-references to related applications

[0001] This application claims priority to provisional application US 63 / 635,960, filed April 18, 2024, and application US 19 / 180,618, filed April 16, 2025, the entire contents of which are hereby expressly incorporated by reference. Technical Field

[0002] The disclosed topics relate to video encoding and / or decoding, and in particular to signaling the use of video by AI-driven processes in the form of Supplementary Enhancement Information (SEI) messages. Background Technology

[0003] Techniques for video coding and / or decoding using motion-compensated inter-frame prediction have existed for decades. Uncompressed digital video can consist of a series of images, each with a spatial dimension of, for example, 1920×1080 luma samples and associated chroma samples. This series of images can have a fixed or variable image rate (commonly known as the frame rate), such as 60 images per second, or 60 Hz. Uncompressed video has extremely high bitrate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (1920×1080 luma sample resolution, 60 Hz frame rate) requires close to 1.5 Gbit / s of bandwidth. One hour of such video would require more than 600 gigabytes of storage space.

[0004] One objective of video encoding and decoding is to reduce redundancy in the input video signal through compression. Compression helps reduce bandwidth or storage requirements, in some cases by two orders of magnitude or more. Lossless compression, lossy compression, and combinations of both can be used. Lossless compression refers to techniques that can reconstruct an exact copy of the original signal from the compressed original signal. With lossy compression, the reconstructed signal may not be exactly the same as the original signal, but the distortion between the original and reconstructed signals is small enough that the reconstructed signal can be used for the intended application. Lossy compression is widely used for video. The tolerable amount of distortion depends on the specific application; for example, users of some consumer streaming applications may tolerate higher distortion than users of television program delivery applications. The achievable compression ratio reflects the higher the allowable or tolerable distortion; the higher the achievable compression ratio.

[0005] Video encoders and decoders can employ a variety of broad categories of techniques, including motion compensation, transform, quantization, and entropy coding, some of which will be discussed below.

[0006] Generative artificial intelligence (AI) can be used to create or modify images or image sequences (videos). For example, at the time of writing, the World Wide Web has various applications and web pages that can generate images—i.e., pixel images—based on strings input into the application. Images can be downloaded and processed using image compression tools, including, for example, encoders conforming to an H.266 still image level. At the time of writing, similar techniques are available for image sequences that can be input into, for example, an H.266 video encoder. Similarly, image or video tampering is also possible. For example, so-called deepfake techniques are known to exist, which utilize source images or videos and tamper with them in ways not intended by the original content creator. A common example is audio-visual sequences of celebrities, where the audio stream is modified to make the celebrity make statements they would not otherwise make, and the video is manipulated to synchronize with the modified audio (lip movements, gestures, etc.).

[0007] Copyright serves as a tool for defining the laws and regulations governing the use and distribution of content. However, with the advancement of AI technology, new challenges and considerations have emerged regarding the diversification of content use. The ways in which content is used with the help of AI may become highly diverse and complex, making it difficult for traditional copyright or licensing terms to clearly define and manage all possible use cases. Therefore, relying solely on copyright or licensing for protection may be insufficient to address the complexities brought about by AI-driven content use. Summary of the Invention

[0008] This application includes a method and apparatus comprising: a memory configured to store computer program code; and at least one processor configured to access the computer program code and operate according to the instructions of the computer program code. The computer program is configured to cause the processor to perform a conversion between a visual media file and a visual media data stream according to format rules, the format rules instructing the determination of at least one SEI message, the SEI message including syntax elements indicating whether an image is allowed to be used as input by an AI engine.

[0009] This application also provides a video decoding method executed by at least one processor, comprising: receiving an encoded video stream, the encoded video stream including an image to be used by an AI engine; and parsing at least one SEI message from the received image, the SEI message including syntax elements indicating whether the image is allowed to be used as input by the AI ​​engine.

[0010] This application also provides a video encoding method executed by at least one processor, comprising: receiving a video, the video including at least one image; determining at least one SEI message, the SEI message including a syntax element indicating whether the image is allowed to be used as input by an AI engine; and encoding a video stream, the video stream including the image and at least one SEI message.

[0011] Syntax elements can indicate whether images are allowed to be used as input for the AI ​​engine to create derivative works.

[0012] Syntax elements can indicate whether images are allowed to be used as input for training the AI ​​engine.

[0013] Syntax elements can indicate whether an image is allowed to be used as input for any purpose by the AI ​​engine.

[0014] Syntax elements can indicate whether an image requires authorization from the content owner before it can be used as input by the AI ​​engine for any purpose.

[0015] Syntax elements may include any of the following: ai_usage_restrictions, aur_cancel_flag, aur_persistence_flag, aur_num_restrictions_minus1, aur_restriction[i], aur_context_present_flag[i], and aur_context[i].

[0016] aur_restriction[i] can represent any of four values, which include:

[0017] The first value indicating "not for any AI application"

[0018] The second value indicates "not for AI training".

[0019] The third value indicating "not for use in generative (modified or created) AI," and

[0020] The fourth value indicates "not for AI inference," and

[0021] aur_context[i] can represent any of the following:

[0022] The first bitmask indicating "commercial use"

[0023] The second bitmask indicates "non-commercial use".

[0024] The third bitmask indicating "official government use", and

[0025] The fourth bit of the mask indicates "scientific and academic use". Attached Figure Description

[0026] Other features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, wherein:

[0027] Figure 1 It is a simplified block diagram of a communication system according to at least one embodiment;

[0028] Figure 2 It is a simplified block diagram of a communication system according to at least one embodiment;

[0029] Figure 3 This is a simplified block diagram of a decoder according to at least one embodiment;

[0030] Figure 4 It is a schematic diagram of a simplified block diagram of an encoder according to at least one embodiment;

[0031] Figure 5 This is a schematic diagram of a Network Abstraction Layer (NAL) unit and an SEI header according to at least one embodiment;

[0032] Figure 6 It is a schematic diagram of a system according to at least one embodiment; and

[0033] Figure 7 It is a schematic diagram of a computer system according to at least one embodiment. Detailed Implementation

[0034] The features discussed below can be used individually or in any combination in any order. Furthermore, embodiments can be implemented using processing circuitry (e.g., at least one processor or at least one integrated circuit). In one example, at least one processor executes a program stored in a non-transitory computer-readable storage medium.

[0035] According to embodiments of this application, an SEI message is provided for marking content created and / or modified by a generative AI engine, thereby meeting anticipated regulatory requirements. The disclosed subject matter relates to video encoding and decoding, and more particularly to including markers in SEI messages to mark content that uses generative AI to generate video clips.

[0036] Figure 1A simplified block diagram of a communication system 100 according to an embodiment of the present invention is shown. The communication system 100 may include at least two terminals 102 and 103 interconnected via a network 105. For unidirectional data transmission, the first terminal 103 may encode video data locally for transmission to the other terminal 102 via the network 105. The second terminal 102 may receive the encoded video data from the other terminal via the network 105, decode the encoded data, and display the recovered video data. Unidirectional data transmission is common in scenarios such as media service applications.

[0037] Figure 1 A second pair of terminals 101 and 104 is shown, which are used to support bidirectional transmission of encoded video, for example, during video conferencing. For bidirectional data transmission, each terminal 101 and 104 can encode locally acquired video data for transmission to the other terminal via network 105. Each terminal 101 and 104 can also receive encoded video data transmitted by the other terminal, decode the encoded data, and display the recovered video data on a local display device.

[0038] exist Figure 1 In this invention, terminals 101, 102, 103, and 104 may be represented as servers, personal computers, and smartphones, but the principles of the invention are not limited thereto. Embodiments of the invention are applicable to laptops, tablets, media players, and / or dedicated video conferencing equipment. Network 105 represents any number of networks, including, for example, wired and / or wireless communication networks, capable of transmitting encoded video data between terminals 101, 102, 103, and 104. Communication network 105 may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of network 105 may be irrelevant to the implementation of the invention unless otherwise stated below. Network (150) may include Media Aware Network Elements (MANEs, 160), which may be included in, for example, the transmission path between terminals (130) and (140). The purpose of MANEs can be to selectively forward portions of media data in response to network congestion, media switching, media mixing, archiving, and similar tasks typically performed by service providers rather than end users. Such MANEs are capable of parsing and responding to limited portions of media transmitted over a network, such as syntax elements related to video coding technologies or standard NAL.

[0039] Figure 2As an application example of the disclosed subject matter, the arrangement of a video encoder and decoder in a streaming media environment is shown. The disclosed subject matter is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, and storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0040] The streaming media system may include an acquisition subsystem 203, which may include a video source 201, such as a digital camera, for creating, for example, an uncompressed sample video stream 213. This sample stream 213 may exhibit a higher data volume characteristic compared to an encoded video stream and may be processed by an encoder 202 coupled to the camera 201. The encoder 202 may include hardware, software, or a combination of hardware and software to enable or implement aspects of the disclosed subject matter, which are described in more detail below. The encoded video stream 204 may exhibit a lower data volume characteristic compared to the sample stream and may be stored on a streaming media server 205 for future use. At least one streaming media client 212 and 207 may access the streaming media server 205 to retrieve copies 208 and 206 of the encoded video stream 204. Client 212 may include video decoder 211, which decodes a copy 208 of the input encoded video stream and creates an output video sample stream 210, which may be displayed on display 209 or other display device (not shown). In some streaming media systems, video streams 204, 206, and 208 may be encoded according to certain video encoding / compression standards. Examples of these standards have been mentioned above and are further described in this application. Examples of these standards also include ITU-T Recommendations H.265 and H.266. For example, the disclosed subject matter can be used in the context of VVC.

[0041] Figure 3 This can be a functional block diagram of the video decoder 300 according to an embodiment of the present invention.

[0042] Receiver 302 can receive at least one coded video sequence (CVS) to be decoded by decoder 300; in this embodiment or another embodiment, one CVS is received at a time, wherein the decoding of each CVS is independent of the others. CVSs can be received from channel 301, which can be a hardware / software link connected to a storage device storing the coded video data. Receiver 302 can receive coded video data and other data, such as coded audio data and / or auxiliary data streams that can be forwarded to their respective user entities (not shown). Receiver 302 can separate CVSs from other data. To combat / mitigate network jitter, a buffer memory 303 can be coupled between receiver 302 and entropy decoder / parser 304 (hereinafter referred to as the "parser"). Buffer 303 may not be necessary when receiver 302 receives data from a store / forward device with sufficient bandwidth and controllability or from an isosynchronous network, or the buffer can be made smaller. When used in best-effort packet networks such as the Internet, buffer 303 may be necessary, can be relatively large, and can advantageously have an adaptive size.

[0043] Video decoder 300 may include parser 304 for reconstructing symbols 313 from an entropy-encoded video sequence. Categories of those symbols include information for managing the operation of decoder 300 and may include information for controlling display devices such as display 312, which is not part of the decoder but may be coupled to it. Control information for one or more display devices may be in the form of SEI messages or fragments of Video Usability Information (VUI) parameter sets (not shown). Parser 304 may perform parsing / entropy decoding on the received CVS. The encoding of the CVS may be based on video coding techniques or standards and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. Parser 304 may extract a subgroup parameter set from the CVS for at least one pixel subgroup in the video decoder based on at least one parameter corresponding to a group. Subgroups can include Group of Pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), prediction units (PU), etc. The entropy decoder / parser can also extract information from the CVS such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0044] The parser 304 can perform entropy decoding / parsing operations on the video sequence received from the buffer 303 to create symbols 313. The parser 304 can receive encoded data and selectively decode specific symbols 313. Furthermore, the parser 304 can determine whether to provide specific symbols 313 to the motion compensation prediction unit 306, the scaler / inverse transform unit 305, the intra-frame prediction unit 307, or the loop filter 311.

[0045] Depending on the type of the encoded video image or its portions (e.g., inter-frame and intra-frame images, inter-frame and intra-frame blocks) and other factors, the reconstruction of symbol 313 may involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the CVS by the parser 304. For clarity, the flow of such subgroup control information between the parser 304 and the various units described below is not shown.

[0046] In addition to the functional blocks already mentioned, the decoder 300 can be conceptually subdivided into several functional units as described below. In practical implementations under commercial constraints, many of these units interact closely with each other and can be at least partially integrated with one another. However, for the purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into the functional units described below.

[0047] The first unit is the scaler / inverse transform unit 305. The scaler / inverse transform unit 305 receives from the parser 304 quantized transform coefficients as one or more symbols 313, along with control information including the transform to be used, block size, quantization factor, quantization scaling matrix, etc. The scaler / inverse transform unit can output blocks containing sample values, which can be input into the aggregator 310.

[0048] In some cases, the output samples of the scaler / inverse transform 305 may involve intra-coded blocks; that is, blocks that do not use predictive information from previously reconstructed images, but instead use predictive information from previously reconstructed portions of the current image. This predictive information can be provided by the intra-prediction unit 307. In some cases, the intra-prediction unit 307 uses surrounding reconstructed information obtained from the current (partially reconstructed) image 309 to generate blocks of the same size and shape as the blocks in the reconstruction. In some cases, the aggregator 310 adds the predictive information already generated by the intra-prediction unit 307 to the output sample information provided by the scaler / inverse transform unit 305 on a per-sample basis.

[0049] In other cases, the output samples of the scaler / inverse transform unit 305 may involve inter-frame coded and potentially motion-compensated blocks. In this case, the motion-compensated prediction unit 306 can access the reference image memory 308 to obtain samples for prediction. After motion compensation of the obtained samples according to the block-related symbol 313, these samples can be added to the output of the scaler / inverse transform unit (referred to in this case as residual samples or residual signals) by the aggregator 310 to generate output sample information. The address at which the motion compensation unit obtains the predicted samples in the reference image memory can be controlled by motion vectors, which are provided to the motion compensation unit in the form of symbol 313, which may have, for example, X components, Y components, and reference image components. Motion compensation may also include interpolation of sample values ​​obtained from the reference image memory when using subsample precise motion vectors, motion vector prediction mechanisms, etc.

[0050] The output samples of aggregator 310 can be processed in loop filter unit 311 using various loop filtering techniques. Video compression techniques may include loop filtering techniques, which are controlled by parameters included in the encoded video bitstream. These parameters are provided to loop filter unit 311 by parser 304 in the form of symbol 313. Loop filtering techniques may also respond to metadata obtained during decoding of previous (in decoding order) portions of the encoded image or CVS, and may respond to previously reconstructed and loop-filtered sample values.

[0051] The output of the loop filter unit 311 can be a sample stream, which can be output to the display 312, which can be a display device. The sample stream can also be stored in the reference image memory 557 for future inter-frame prediction.

[0052] Some encoded images can be used as reference images for future predictions after full reconstruction. Once an encoded image has been fully reconstructed and has been identified as a reference image (e.g., by parser 304), the current reference image 309 can become part of the reference image buffer 308, and new current image memory can be reallocated before the reconstruction of subsequent encoded images begins.

[0053] The video decoder 300 can perform decoding operations according to a predetermined video compression technology, which may be described in standards such as ITU-T H.265 and / or H.266 Recommendations. The CVS may conform to the syntax specified by the video compression technology or standard used; that is, the CVS follows the syntax of the video compression technology or standard, such as the syntax specified in the video compression technology documentation or standard, particularly in its configuration file documentation. For compliance, the complexity of the CVS must also be within the limits defined by the hierarchy of the video compression technology or standard. In some cases, the hierarchy limits the maximum image size, maximum frame rate, maximum reconstruction sampling rate (measured in megasamples per second, for example), maximum reference image size, etc. In some cases, the limitations set by the hierarchy can be further limited by the Hypothetical Reference Decoder (HRD) specification and the HRD buffer management metadata indicated by signaling in the CVS.

[0054] In one embodiment, receiver 302 may receive supplemental (redundant) data along with the encoded video. The supplemental data may be part of one or more CVSs. The supplemental data can be used by video decoder 300 to properly decode the data and / or more accurately reconstruct the original video data. The supplemental data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0055] Figure 4 This can be a functional block diagram of a video encoder 400 according to an embodiment of the present invention.

[0056] Encoder 400 can receive video samples from video source 401 (not part of the encoder), which can capture one or more video images to be encoded by encoder 400.

[0057] Video source 401 can provide a source video sequence in the form of a digital video sample stream to be encoded by encoder (303). This digital video sample stream can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCb, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, video source 401 can be a storage device storing previously prepared video. In a video conferencing system, video source 401 can be a camera capturing local image information as a video sequence. Video data can be provided as multiple individual images that, when viewed sequentially, produce an animated effect. The images themselves can be organized as spatial pixel arrays, where each pixel can include at least one sample, depending on the sampling structure, color space, etc., used. Those skilled in the art can readily understand the relationship between pixels and samples. Samples are described in detail below.

[0058] According to one embodiment, encoder 400 can encode and compress images of a source video sequence into CVS 410 in real time or as needed by the application, according to any other time constraints. Enforcing an appropriate encoding rate is a function of controller 402. The controller controls and is functionally coupled to other functional units described below. For clarity, the coupling is not shown. Parameters set by the controller may include rate control-related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, GOP layout, maximum motion vector search range, etc. Those skilled in the art will readily identify other functions of controller 402 that may relate to the video encoder 400 optimized for a particular system design.

[0059] Some video encoders operate under what those skilled in the art call an "encoding loop." As a highly simplified description, an encoding loop can consist of the encoding portion of the encoder (e.g., source encoder 403) (responsible for creating symbols based on the input image to be encoded and one or more reference images) and a (local) decoder 406 embedded in encoder 400, which reconstructs the symbols to create sample data, and a (remote) decoder similarly creates this sample data (because in the video compression techniques considered in the disclosed subject, any compression between the symbols and the encoded video stream is lossless compression). This reconstructed sample stream is input to a reference image memory 405. Since decoding the symbol stream produces a bit-by-bit accurate result unaffected by the decoder's location (local or remote), the contents of the reference image buffer are also bit-by-bit accurate between the local and remote encoders. In other words, the reference image samples "seen" by the encoder's prediction portion are exactly the same as the sample values ​​"seen" by the decoder during prediction. Those skilled in the art are familiar with the basic principle of reference image synchronicity (and the drift that occurs, for example, due to channel errors, when synchronicity cannot be maintained).

[0060] The operation of the "local" decoder 406 is the same as that of the "remote" decoder 300, which has already been described above. Figure 3 A detailed description has been provided. However, a brief additional reference is also available. Figure 4 When symbols are available and the entropy encoder 408 and the parser 304 are able to encode / decode symbols to CVS without loss, the entropy decoding portion of the decoder 300, including the channel 301, receiver 302, buffer 303 and parser 304, may not be fully implemented in the local decoder 406.

[0061] At this point, we can conclude that, in addition to parsing / entropy decoding present in the decoder, any decoder technique must also exist in its corresponding encoder in essentially the same functional form. The description of encoder techniques can be simplified, as encoder techniques are the reverse operation of a fully described decoder technique. More detailed descriptions are only needed in certain areas, and these descriptions are provided below.

[0062] As part of the operation of the source encoder 403, the source encoder 403 may perform motion-compensated predictive coding, that is, predictively coding the input frame with at least one previously encoded frame designated as a "reference frame" in the reference video sequence. In this way, the encoding engine 407 encodes the differences between pixel blocks of the input frame and pixel blocks of one or more reference frames, which may be selected as one or more predictive references for the input frame.

[0063] The local video decoder 406 can decode encoded video data of frames that can be designated as reference frames, based on symbols created by the source encoder 403. Advantageously, the operation of the encoding engine 407 can be a lossy process. When the encoded video data can be decoded by the video decoder (… Figure 4 During decoding (not shown), the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder 406 replicates the decoding process, which can be performed on reference frames by the video decoder, and the reconstructed reference frames can be stored in a reference image memory 405, which can be, for example, a cache. In this way, the encoder 400 can store copies of the reconstructed reference frames locally, these copies having the same content (no transmission errors) as the reconstructed reference frames that will be obtained by the remote video decoder.

[0064] Predictor 404 can perform a prediction search against encoding engine 407. That is, for a new frame to be encoded, predictor 404 can search the reference image memory 405 for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., which can be used as appropriate prediction references for the new image. Predictor 404 can operate pixel-by-pixel based on sample blocks to find appropriate prediction references. In some cases, as determined by the search results obtained by predictor 404, the input image may have prediction references obtained from multiple reference images stored in reference image memory 405.

[0065] The controller 402 can manage the encoding operations of the video encoder 403, including, for example, setting parameters and subgroup parameters for video data encoding.

[0066] The outputs of all the aforementioned functional units can be entropy encoded in the entropy encoder 408. By performing lossless compression on the symbols generated by the various functional units according to techniques known to those skilled in the art (e.g., Huffman coding, variable-length coding, arithmetic coding, etc.), the entropy encoder converts these symbols into CVS.

[0067] Transmitter 409 may buffer one or more CVSs created by entropy encoder 408 in preparation for transmission via communication channel 411, which may be a hardware / software link connected to a storage device that will store the encoded video data. Transmitter 409 may combine encoded video data from video encoder 403 with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0068] Controller 402 can manage the operation of encoder 400. During encoding, controller 405 can assign a specific encoded image type to each encoded image, which may affect the encoding techniques applicable to the corresponding image. For example, one of the following frame types can typically be assigned to an image:

[0069] Intraframe images (I-images) are images that can be encoded and decoded without using any other frames in the sequence as a prediction source. Some video codecs support different types of intraframe images, including, for example, images refreshed by a standalone decoder. Those skilled in the art are familiar with those variations of I-images and their respective applications and characteristics.

[0070] A predictive image (P-image) is an image that can be encoded and decoded by predicting sample values ​​for each block via intra-frame prediction or via inter-frame prediction using at most one motion vector and a reference index.

[0071] A bidirectional predictive image (B-image) can be an image encoded and decoded by predicting sample values ​​for each block via intra-frame prediction or via inter-frame prediction using up to two motion vectors and reference indices. Similarly, a multi-predictive image can reconstruct a single block using more than two reference images and associated metadata.

[0072] The source image can typically be spatially subdivided into multiple sample blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples each) and encoded block by block. These blocks can be predictively coded with reference to other (already coded) blocks, which are determined by the coding assignments of the corresponding images applied to the blocks. For example, blocks of an I-image can be unpredictably coded, or these blocks can be predictively coded (spatial prediction or intra-frame prediction) with reference to already coded blocks of the same image. Pixel blocks of a P-image can be unpredictably coded, coded via spatial prediction, or coded via temporal prediction with reference to a previously coded reference image. Blocks of a B-image can be unpredictably coded, coded via spatial prediction, or coded via temporal prediction with reference to one or two previously coded reference images.

[0073] The video encoder 400 can perform encoding operations according to predetermined video coding technologies or standards, such as ITU-T H.265 and / or H.266 recommendations. In operation, the video encoder 400 can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technology or standard used.

[0074] In one embodiment, transmitter 409 may transmit additional data along with the encoded video. Source encoder 403 may include such data as part of the CVS. Additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data (e.g., redundant images and slices), SEI messages, VUI parameter set fragments, etc.

[0075] Video compression can be enhanced within the video stream using SEI (Search Engine Information), for example, through SEI messages or VUI (Video User Interface). Video coding standards can include specification sections for SEI and VUI. SEI and VUI information can also be specified in separate specifications that can be referenced by the video coding specification.

[0076] According to embodiments of this application, both the encoding and decoding processes can be considered as visual media data processing, which performs conversion between visual media files and visual media data streams according to format rules.

[0077] refer to Figure 5 Example 500 illustrates an exemplary layout of a CVS according to H.266 and embodiments of this application. The CVS is subdivided into NAL units. An exemplary NAL unit 501 may include a NAL unit header 502, which in turn includes the following 16 bits: forbidden_zero_bit 503 and nuh_reserved_zero_bit 504 may not be used in H.266 and may be zero in NAL units conforming to H.266. The three bits of nuh_layer_id 505 may indicate the layer (spatial, SNR, or multi-view enhancement) to which the NAL unit belongs. The five bits of nuh_nal_unit_type 506 define the type of the NAL unit. According to embodiments of this application, in H.266, 22 NAL unit type values ​​are defined for NAL unit types, 6 NAL unit types are reserved, and 4 NAL unit type values ​​are unspecified and can be used by specifications other than H.266. Finally, the three bits of the NAL unit header 502 indicate the time layer to which the NAL unit belongs, nuh_temporal_id_plus1 506.

[0078] An encoded image may contain at least one Video Coding Layer (VCL) NAL unit and zero or more non-VCL NAL units. VCL NAL units may contain encoded data that conceptually belongs to the VCL as described above. Non-VCL NAL units may contain data that conceptually does not belong to the VCL. Taking H.266 as an example, these units can be categorized as parameter sets, image headers, NAL units, prefix and suffix SEI NAL unit types, padding data NAL unit type FD_NUT, and reserved and unspecified NAL unit types.

[0079] (1) Parameter sets, which include information that may be necessary for the decoding process and can be applied to more than one encoded image. Parameter sets and conceptually similar NAL units may belong to the following NAL unit types: DCI_NUT (Decoding Capability Information, DCI), VPS_NUT (Video Parameter Set, VPS, used to establish layer relationships, etc.), SPS_NUT (Sequence Parameter Set, SPS, used to establish parameters used and unchanged throughout the CVS, etc.), PPS_NUT (Picture Parameter Set, PPS, used to establish parameters used and unchanged within the encoded image, etc.), and PREFIX_APS_NUT and SUFFIX_APS_NUT (prefix and suffix adaptive parameter sets). Parameter sets may include information required by the decoder to decode VCL NAL units, and are therefore referred to here as "canonical" NAL units.

[0080] (2) Image Header (PH_NUT), which is also a “normal” NAL unit.

[0081] (3) Mark NAL units at specific locations in the NAL unit stream. These NAL units include those of type AUD_NUT (Access Unit Separator), EOS_NUT (End of Sequence), and EOB_NUT (End of Stream). These NAL units are non-canonical, also known as informative, meaning that a standard-compliant decoder does not need these NAL units during its decoding process, but needs to be able to receive them in the NAL unit stream.

[0082] (4) Prefix and suffix SEI NAL unit types (PREFIX_SEI_NUT and SUFFIX_SEI_NUT), which indicate NAL units containing prefix and suffix SEI. In H.266 (04 / 2022), those NAL units are informational because they are not needed in the decoding process.

[0083] (5) Fill data NAL cell type FD_NUT, which indicates the fill data; the data can be random and can be used to “waste” bits in the NAL cell stream or bit stream, which may be necessary for transmission in some isochronous transmission environments.

[0084] (6) Reserved and unspecified NAL cell types.

[0085] Still referencing Figure 5 The diagram illustrates the layout of a NAL unit stream with a decoding order 510, which contains encoded images 511, each containing NAL units of a previously described type. Somewhere along the NAL unit stream, DCI 512, VPS 513, and SPS 514 can jointly establish parameters that the decoder can use to decode the encoded images of the CVS, including the encoded images 511 of the NAL unit stream.

[0086] The encoded image 511 may include, in the order shown or in any other order conforming to the video coding technology or standard used (here: H.266), the following prefixes: APS 516, Picture header (PH, 517), SEI 518, at least one VCL NAL unit 519, and SEI suffix 520.

[0087] Prefix SEI NAL unit 518 and suffix SEI NAL unit 520 were introduced during the standardization process because for some SEI messages, the message content is known before the encoding of a given image begins, while for others it is only known after the image encoding is complete. Having certain SEI messages appear earlier or later in the NAL unit stream of the encoded image as prefix and suffix SEI messages avoids buffering. As an example, in an encoder, the sampling time of the image to be encoded is known before image encoding, so the image timing SEI message could be a prefix SEI message 516. On the other hand, the decoded image hash SEI message contains the hash value of the sample values ​​of the decoded image and can be used, for example, for debugging encoder implementations; this decoded image hash SEI message is a suffix SEI message 518 because the encoder cannot calculate the hash value based on the reconstructed samples before the image is encoded. The location of the prefix and suffix SEI NAL units is not limited to their position in the NAL unit stream. The terms “prefix” and “suffix” may refer to the encoded image or NAL unit to which the prefix / suffix SEI message may be applied, and the details of this applicability may be specified, for example, in the semantic description of a given SEI message.

[0088] Still referencing Figure 5 A simplified syntax diagram of a NAL unit containing a prefixed or suffixed SEI message 520 is shown. This syntax can be a container format for carrying multiple SEI messages within a single NAL unit. For clarity, details of the extension mechanism for payload size and payload type numbering ranges as specified in H.266 are omitted here. Like other NAL units, an SEI NAL unit begins with a NAL unit header 521. This header is followed by at least one SEI message; two SEI messages 530 and 531 are shown and described below. Each SEI message within an SEI NAL unit may include: an 8-bit payload_type_byte 522 specifying one of 256 different SEI types (or extension indicators); an 8-bit payload_size_byte 523 specifying the number of bytes of the SEI payload (or the presence of an extension block); and a payload 524 consisting of payload_size_byte minus one byte. The syntax for payload 524 depends on the SEI message and can be any length between 0 and 254 bytes unless an extension mechanism (not shown) is used. When the extension mechanism is used, the syntax will support unlimited payload size.

[0089] From a societal perspective, generative AI has become a challenging new technology. Modifying or "tampering" with media such as audio, images, and videos, which required highly skilled professionals and was typically time-consuming and costly, can now be done by individuals using a computer and internet access, either for free or at a relatively low cost. Content creation is increasingly reliant on AI-based tools.

[0090] Today, many people view using AI to create entirely new original content as simply using another tool, like a paintbrush or a violin. However, some AI-generated content is clearly malicious—for example, fabricating statements from celebrities or other media figures, or creating fake documentaries (which the general public often takes for granted as original footage). Copyright is a tool for defining the laws and regulations governing the use and distribution of content. However, with the advancement of AI technology, new challenges and considerations have emerged regarding the diversification of content use. The ways in which content is used with AI may become highly diverse and complex, making it difficult for traditional copyright or licensing terms to clearly define and manage all possible use cases. Therefore, relying solely on copyright or licensing for protection may be insufficient to address the complexities brought about by AI-driven content use.

[0091] Current copyright regulations and licensing terms in most jurisdictions may not explicitly cover many aspects of AI-driven content use. Specifically, existing regulations are ill-equipped to handle massive and diverse data, fair use guidelines lack clarity, and they fail to keep pace with rapid technological advancements.

[0092] First, current laws and regulations are inadequate to handle massive and diverse data, while AI requires vast amounts of data for effective training. The sheer volume and variety of video content used may make it difficult to verify the AI's usage rights for each piece of content, potentially leading to unintentional copyright infringement.

[0093] Secondly, current laws and regulations lack clarity regarding the fair use principle. The concept of fair use often requires legal interpretation, and the criteria for such interpretation can vary from jurisdiction to jurisdiction. Users may find it difficult to determine whether their use of copyrighted video content to train AI models falls under the category of fair use, especially when the content is used in a transformative manner or in a way that the copyright holder did not initially intend.

[0094] Third, existing laws and regulations are unable to keep pace with rapid technological advancements. The rapid development of AI technology often outpaces the development of corresponding legal frameworks. This can leave users in a gray area, uncertain about the legal consequences of using specific video content for AI training purposes. This uncertainty is further exacerbated by the lack of restrictions on AI use within current copyright frameworks, making the legal environment for AI applications in the video content field even more complex.

[0095] Where guidelines and regulations regarding the use of AI in video media content are unclear or not yet established, copyright holders (such as filmmakers, studios, and digital content creators) should develop and publish specific restrictions and recommendations regarding the use of AI in video. This allows content owners to clearly indicate how their content should be used by AI, providing clear guidance for AI developers and similar users. Such initiatives not only promote ethical and legal compliance in the use of AI in video but also strengthen cooperation between content owners and AI developers, thereby preventing unnecessary disputes.

[0096] Embodiments of this application provide mechanisms for signaling AI usage and / or constraint information regarding video media content. Such AI usage and / or constraint information may be in the form of a string expressing guidelines, permissions, and constraints regarding how reconstructed images or videos can be permitted for use in AI-driven processes. The following list provides some examples of AI usage and / or constraint information regarding video media content that can be signaled according to this application.

[0097] "This image or video data is permitted to be used as input for any AI process."

[0098] "This image or video data must be authorized by the owner before it can be used as input for any AI process."

[0099] "This image or video data must not be used as input for any AI process."

[0100] "This image or video data is permitted to be used as input for any AI training."

[0101] "This image or video data must be authorized by the owner before it can be used as input for any AI training."

[0102] "This image or video data must not be used as input for any AI training."

[0103] - "This image or video data is permitted to be used as input for generative AI training."

[0104] "This image or video data may be used as input for generative AI training only with the owner's authorization."

[0105] "This image or video data must not be used as input for generative AI training."

[0106] - "This image or video data is permitted to be used as input for generative AI processes to create derivative content."

[0107] "This image or video data must be authorized by the owner before it can be used as input for the creation of derivative content in a generative AI process."

[0108] "This image or video data may not be used as input for generative AI processes to create derivative content."

[0109] - "Personal information contained in this image or video data is prohibited."

[0110] "The use of personal information contained in this image or video data requires the owner's authorization."

[0111] See Figure 6 The display 621 in Example 600 shows an output display message 622 that, according to an exemplary embodiment, displays a message such as a warning to the user 623.

[0112] The list above is merely an example. These examples are not exhaustive and are not restrictive. The common aim behind these examples is to provide machine-readable information that gives explicit guidance to AI on whether and to what extent it is capable of using images or videos as input to generate AI-driven outputs in a video stream for its own purposes.

[0113] SEI messages can indicate such AI usage and / or constraint information at the image or sequence level. SEI messages can be indicated by signaling in various ways, such as within a copyright SEI message or in a predefined SEI message.

[0114] In some embodiments, usage and / or constraint information for AI-driven processes is defined in the Copyright SEI and indicated by signaling. The Copyright SEI defines `ci_ai_usage_information_present_flag` to indicate the presence of usage and / or constraint information for AI-driven processes. `ci_ai_usage_information_present_flag` equal to 1 indicates the presence of usage and / or constraint information for AI-driven processes. `ci_ai_usage_information_present_flag` equal to 0 indicates the absence of usage and / or constraint information for AI-driven processes. When usage and / or constraint information for AI-driven processes exists, a second syntax element `ci_ai_usage_information` can be further indicated by signaling to define specific usage and / or constraint information for the AI-driven process or AI-related recommendations.

[0115] Table 1 below shows an example syntax for a copyright SEI message.

[0116] Table 1: Exemplary Syntax of Copyright SEI Messages

[0117]

[0118] The following are exemplary semantics of the new flag in the Copyright SEI message:

[0119] … …

[0120] A value of 1 for `ci_ai_usage_information_present_flag` indicates that the flag `aiua_ai_usage_information` exists. A value of 0 for `aiua_ai_usage_information_present_flag` indicates that the flag `aiua_ai_usage_information` does not exist.

[0121] ci_ai_usage_information represents information on the use and constraints of AI-driven processes or recommendations related to using AI for images, within the persistence scope defined by aiua_cancel_flag and aiua_persistence_flag.

[0122] … …

[0123] In some embodiments, usage and / or constraint information for AI-driven processes is defined in predefined SEI messages and indicated by signaling. The predefined SEI message defines `aiua_ai_cancel_flag` to indicate whether the SEI message cancels the persistence of any previous image or video content usage and constraint information. Furthermore, the predefined SEI defines `aiua_persistence_flag` to indicate the persistence of image or video content usage and constraints. Additionally, the predefined SEI defines `aiua_ai_information` to indicate constraints and recommendations related to the AI ​​process using the image, within the persistence scope defined by `aiua_cancel_flag` and `aiua_persistence_flag`.

[0124] Table 2 below shows an example syntax for predefined SEI messages.

[0125] Table 2: Exemplary Syntax of Predefined SEI Messages

[0126]

[0127] The following are examples of semantics for new flags in predefined SEI messages.

[0128] … …

[0129] `aiua_cancel_flag` indicates whether the SEI message cancels the persistence of any previous image or video content usage and constraint information in the current layer's output order. `aiua_cancel_flag` equal to 1 indicates that the SEI message cancels the persistence of any previous image or video content usage and constraint information in the current layer's output order. `aiua_cancel_flag` equal to 0 indicates that the image or video content usage and constraint information remains valid.

[0130] `aiua_persistence_flag` indicates the persistence of usage and constraint information for previous image or video content in the current layer. A `aiua_persistence_flag` value of 1 indicates that the usage and constraint information for previous image or video content applies to the currently decoded image and continues to apply to all subsequent images in the current layer's output order until at least one of the following conditions is met:

[0131] – A new CLVS begins for the current layer.

[0132] – End of stream.

[0133] - Outputs the image of the current layer in the AU associated with the Content Usage Information (SEI) message, which is output after the current image in the output order.

[0134] A value of 0 for aiua_persistence_flag indicates that the persistence of previous image or video content usage and constraint information applies only to the currently decoded image.

[0135] aiua_ai_information contains constraints and recommendations related to AI processes that use images, within the persistence scope defined by aiua_cancel_flag and aiua_persistence_flag.

[0136] In some embodiments, the use and / or constraint information of the AI-driven process can be defined in a predefined SEI message and indicated by signaling. For example, a predefined SEI message defines `aiua_deviation_not_allowed_flag` to indicate whether an image is permitted for use by AI to create derivative works within the persistence range defined by `aiua_cancel_flag` and `aiua_persistence_flag`. `aiua_deviation_not_allowed_flag` equal to 1 indicates to the generative AI engine aware of the SEI message that the image is not permitted for creating derivative works. `aiua_deviation_not_allowed_flag` equal to 0 indicates that it is not defined whether the generative AI engine is permitted to use the image for creating derivative works. Note that it cannot be inferred that when an image is marked with `aiua_deviation_not_allowed_flag` equal to 0, the image is permitted for use by the generative AI engine for creating derivative works, as circumstances unknown to the encoder may legally prohibit the image from being used for generative AI processing. When aiua_deviation_not_allowed_flag equals 0, image or video data may require explicit authorization from the owner before it can be used as input for generative AI processes to create derivative content.

[0137] In some embodiments, usage and / or constraint information for AI-driven processes can be defined in predefined SEI messages and indicated by signaling. For example, a predefined SEI message defines `aiua_training_not_allowed_flag` to indicate that images are permitted for AI training within the persistence range defined by `aiua_cancel_flag` and `aiua_persistence_flag`. `aiua_training_not_allowed_flag` equal to 1 indicates to the AI ​​engine aware of the SEI message that image training is not permitted. `aiua_training_not_allowed_flag` equal to 0 indicates that whether the AI ​​engine is permitted to use images for training is undefined. Note that it cannot be inferred that an image can be used for AI training when `aiua_training_not_allowed` is marked as 0, as circumstances unknown to the encoder may legally prohibit the image from being used for training. When `aiua_training_not_allowed_flag` equals 0, explicit authorization from the owner may be required for image or video data to be used as input for generative AI processes to create derivative content.

[0138] In some embodiments, usage and / or constraint information for AI-driven processes can be defined in predefined SEI messages and indicated by signaling. For example, a predefined SEI message defines `aiua_input_not_allowed_flag` to indicate whether an image can be used as input for any AI process, regardless of its purpose, such as derivative work creation, training, etc., within the persistence range defined by `aiua_cancel_flag` and `aiua_persistence_flag`. `aiua_input_not_allowed_flag` equal to 1 indicates to the AI ​​engine aware of the SEI message that the image is not allowed to be used as input for any purpose. `aiua_input_not_allowed_flag` equal to 0 indicates that it is not defined whether the AI ​​engine is allowed to use the image for any purpose. Note that it cannot be inferred that an image can be used by the AI ​​engine when it is marked with `aiua_input_not_allowed` equal to 0, as circumstances unknown to the encoder may legally prohibit the image from being used for training. When aiua_input_not_allowed_flag equals 0, image or video data may require explicit authorization from the owner before it can be used as input for generative AI processes to create derivative content.

[0139] In some embodiments, usage and / or constraint information for AI-driven processes can be defined in predefined SEI messages and indicated by signaling. For example, a predefined SEI message defines `aiua_owner_auth_required_flag` to indicate whether, within the persistence range defined by `aiua_cancel_flag` and `aiua_persistence_flag`, an image requires explicit authorization from the owner to be used as input for any AI process. `aiua_owner_auth_required_flag` equal to 1 indicates to the AI ​​engine aware of the SEI message that explicit authorization from the content owner is required to use the image for any purpose. `aiua_training_not_allowed_flag` equal to 0 indicates that it is not defined whether the AI ​​engine requires explicit authorization from the content owner to use the image for any purpose.

[0140] Table 3 below shows another exemplary syntax for predefined SEI messages.

[0141] Table 3: Exemplary Syntax of Predefined SEI Messages

[0142]

[0143] Table 4 below shows another exemplary syntax for predefined SEI messages.

[0144] Table 4: Exemplary Syntax of Predefined SEI Messages

[0145]

[0146] As shown in Tables 3 and 4, the advantage of setting a 1-bit flag in the predefined SEI message is that machines can more easily parse well-defined bits compared to free text information. In other words, the guidance provided by the syntax defined in Tables 3 and / or 4 can be processed by the machine without additional human interpretation, while the syntax defined in Table 2 provides more flexibility and detail in using and constraining the information.

[0147] The disclosed subject matter relates to signaling the use of video by AI-driven processes in the form of SEI messages. SEI messages have been used to insert metadata into compressed video streams since at least 1997. As previously stated, at least one SEI message can be carried within a NAL unit. In this regard, the concept of SEI messages can also be applied to other NAL unit-based encoding techniques, even if the term "SEI message" may not exist in the specifications of these techniques. Furthermore, the disclosed subject matter can also be applied to areas beyond NAL unit-based codecs, as long as the bitstream format of the codec or stream formatter in question allows the bitstream format, which can be associated with metadata, to be inserted into the encoded media bits. Such codecs or stream formatters can include audio or system-level multiplexing standards. For clarity, the following description uses the above-described SEI message syntax or closely related syntax, but those skilled in the art can adjust or modify the subject matter of the invention to suit application needs beyond SEI messages. Furthermore, the following focuses on a video stream consisting of at least one image encoded in a format supporting SEI messages such as H.266.

[0148] Some jurisdictions are implementing regulatory activities that mandate the use of AI tagging. Other information regulators may consider includes the identification of the content before modification (if any) and the instructions received by the AI ​​for creating / modifying the content. Furthermore, regulators in different countries may not only set different requirements, but may even set conflicting ones, for example, regarding the inclusion of personally identifiable information, and the different priorities of privacy over tagging requirements. For instance, one regulator might require identification (such as a Social Security number) of the person who instructed the AI ​​to generate the image or video content, while another regulator might not require such information because it prioritizes personal privacy over including it in the SEI (Self-Identification Information) used by the AI.

[0149] Figure 6A scenario involving generative AI, according to an exemplary embodiment, is illustrated. The user 601 may have created instructions 602 for the generative AI engine 604. These instructions can be as simple as a few words of free-form text (e.g., “Generate a 10-second video showing a panda riding a bicycle through the city”) or more complex. For example, the instructions could be the video stream itself, provided in symbolic language. In the AI ​​creation scenario described above, these instructions could be the only information provided by the user in response to the AI. In the AI ​​modification scenario, the AI ​​may also receive user-specified content 603 to be modified, shown here as film reel. A hybrid form of these two techniques is also possible; for example, the content to be modified 603 could include metadata containing instructions instructing the AI ​​engine 604 how to modify the content.

[0150] Details regarding the mechanism for providing the AI ​​engine 604 with user-supplied instructions 602 and (only in the case of content modification) content to be modified 603 are omitted here. In some cases, the AI ​​engine may be physically located in a data center operated by a third-party provider, in which case it is likely that instructions 602 and content 603 will be provided to it via the Internet, for example, using an interactive webpage. However, according to embodiments of this application, other forms of transmitting instructions 602 and content 603 from user 601 to AI engine 604 are also envisioned and provided.

[0151] AI engine 604 can receive instructions 602 and (in a modified use case) content 603 to create output, possibly uncompressed media, in this example, a series of uncompressed images (and timing information), which together form an uncompressed video stream 605. The AI ​​engine can also generate certain metadata 606, which includes, for example, data. The AI ​​engine can further determine whether the images are permitted as input for derivative works creation, model training, or both, or for any other purpose.

[0152] Both the uncompressed video stream 605 and metadata 606 can be input to the video encoder 607. The video encoder 607 can compress the uncompressed video stream 605 into a compressed video stream 608, which may include an SEI message 609 associated with a specific encoded image in the encoded video stream 608. The example here shows three encoded images 610, 611, and 612, where only the first encoded image 610 has an associated SEI message 609.

[0153] The encoded video stream 608 can be distributed to consumer users, including producer user 601, in a manner known to those skilled in the art. The encoded video stream 609 shown here is stored in file 613 and then streamed by streaming server 614 over network 615 to the consumer user's endpoint. This endpoint may include video decoder 619, which reconstructs the streamed encoded video stream 609 into a series of decoded images 620. Decoder 619 can also extract AI usage and constraint information, as well as other metadata, from the SEI message 609 included in the encoded video stream 608. Therefore, according to an exemplary embodiment, the AI ​​usage and constraint information is now available to downstream AI engines.

[0154] Information regarding the use and constraints of AI can be used in accordance with the regulations of the regulatory environment in which the decoder and display are located, which may differ from the regulatory environment that specifies what AI should include.

[0155] Based on the modifications, the AI ​​can also use the restricted SEI message syntax as shown in Table 5 below:

[0156] Table 5 - AI Usage Restrictions SEI Message Syntax

[0157]

[0158] According to an exemplary embodiment, the AI ​​uses restriction SEI messages (such as the messages in Table 5 and other messages mentioned above) to signal restrictions and optional context information available for use by the AI ​​application.

[0159] AI usage restriction SEI message types may be included in SEI processing order (SPO) SEI messages to indicate that the usage restrictions indicated by signaling apply to the post-processing output indicated by the SPO SEI message. Furthermore, when the usage restrictions indicated by signaling apply to the post-processed video but not to the cropped decoded image, the AI ​​usage restriction SEI message may be included in processing order nesting (PON) SEI messages. Additionally, non-nested AI usage restriction SEI messages may be used to indicate a first set of usage restrictions applicable to the cropped decoded image, and AI usage restriction SEI messages included in PON SEI messages may be used to indicate a second set of usage restrictions applicable to the post-processed video. According to an exemplary embodiment, the second set of usage restrictions may include additional usage restrictions compared to the first set.

[0160] When an SPO SEI message exists, the processing chain defined by the SPO SEI message does not include the AI ​​usage restriction SEI message type, and an AI usage restriction SEI message not included in the PON SEI message exists, the indicated AI usage restriction applies to both the cropped decoded image and the output of the processing chain.

[0161] When an SPO SEI message exists and an AI usage restriction SEI message not included in the PON SEI message exists, the SEI message associated with the processing chain defined by the SPO SEI message must not contradict the indicated AI usage restriction.

[0162] For example, you cannot simultaneously indicate that aur_restriction[i] is equal to 2 ("not for generative AI") and treat the generative neural network as an NNPF in the processing chain.

[0163] `aur_cancel_flag` equal to 1 indicates that the SEI message cancels the persistence of any previous AI usage restriction SEI messages in the output order. `aur_cancel_flag` equal to 0 indicates that the AI ​​usage restriction information remains valid.

[0164] aur_persistence_flag indicates the persistence of the current layer's AI using restricted SEI messages.

[0165] A value of 0 for aur_persistence_flag indicates that AI uses the restriction that SEI messages are only applicable to currently decoded images.

[0166] A `aur_persistence_flag` value of 1 indicates that the AI ​​uses the restricted SEI message, which applies to the currently decoded image and continues to apply to all subsequent images of the current layer in the output order until at least one of the following conditions is met:

[0167] – A new CLVS begins for the current layer.

[0168] – End of stream.

[0169] – Outputs the image of the current layer in the AU associated with the AI ​​usage restriction SEI message, which is output after the current image in the output order.

[0170] The increment of 1 in aur_num_restrictions_minus1 indicates the number of subsequent restriction entries.

[0171] aur_restriction[i] represents the i-th restriction specified in Table 6. The value of aur_restriction[i] should be in the range of 0 to 3, inclusive. For aur_restriction[i], values ​​greater than 3 are reserved for future use by ITU-T | ISO / IEC and should be ignored in the bitstream conforming to this version of the specification according to embodiments of this application.

[0172] Table 6 – Definition of aur_restriction[i]

[0173]

[0174] `aur_context_present_flag[i]` equal to 1 indicates that the context information for `aur_restriction[i]` exists. `aur_context_present_flag[i]` equal to 0 indicates that the context information for `aur_restriction[i]` does not exist. When `aur_context_present_flag[i]` equals 0, `aur_restriction[i]` applies regardless of the context.

[0175] If `aur_context[i]` exists, it indicates the context of the i-th `aur_restriction[i]`, as specified in Table 7, where `(aur_context[i] & bitMask)` not equal to 0 indicates that the i-th restriction applies to the context associated with the `bitMask` value in table xy. When `aur_context[i]` is greater than 0 and `(aur_context[i] & bitMask)` equals 0, the applicability of the i-th restriction in the context associated with the `bitMask` value in table xy is undefined. In a bitstream conforming to this version of the specification, the value of `aur_context[i]` should be in the range of 1 to 15 (inclusive). Values ​​greater than 15 are reserved for future use by ITU-T | ISO / IEC and should be ignored in a bitstream conforming to this version of the specification according to embodiments of this application. When the value of `aur_context[i]` is greater than 15, a decoder conforming to this version of the specification should ignore `aur_context[i]`. aur_context[i] should not be equal to 0.

[0176] Table 7 – Definition of aur_contex[i]

[0177]

[0178] Table 8 below shows the persistence range of SEI messages (informative) including SEI message AI usage restrictions according to embodiments of this application.

[0179]

[0180]

[0181] When multiple NNPFs specified by NNPFC SEI messages with different nnpfc_purpose values ​​are activated for the same image, conflicts may occur in the corresponding NNPF procedures. Similarly, when an SEI message implying a non-NNPF post-processing procedure is associated with an image, and an NNPF specified by an NNPFC SEI message with a related purpose is activated for the same image, conflicts may occur in the corresponding procedures. Extra caution is required when designing encoders that generate such multiple SEI messages. For example, conflicting implicit SEI messages can be included in different processing chains defined by SPO SEI messages. Alternatively, multiple implicit SEI messages that perform related post-processing operations can be used in situations where conflicts are inconsequential, unlikely, or already managed—e.g., defined or managed in control applications or transport specifications, or managed through the control bitstream distribution environment.

[0182] The aforementioned techniques for signaling the use of video by an AI-driven process in SEI messages can be implemented as computer software using computer-readable instructions and physically stored on at least one computer-readable medium. For example, Figure 7 A computer system 700 suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0183] Computer software can be coded using any suitable machine code or computer language. This machine code or computer language may be processed by mechanisms such as assembly, compilation, and linking to generate code containing instructions. These instructions can be executed directly or by the computer's central processing unit (CPU), graphics processing unit (GPU), etc., through interpretation, microcode execution, or other means.

[0184] These instructions can be executed on various types of computers or their components, including, for example, personal computers, tablets, servers, smartphones, gaming devices, and Internet of Things (IoT) devices.

[0185] Figure 7The components of the computer system 700 shown are exemplary in nature and are not intended to imply any limitation on the scope or functionality of the computer software implementing embodiments of the present invention. The configuration of the components should also not be construed as any dependency or requirement on any component or combination of components shown in the exemplary embodiments of the computer system 700.

[0186] Computer system 700 may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from at least one human user via, for example, tactile input (e.g., button presses, swipes, data glove movements), audio input (e.g., voice, clapping sounds), visual input (e.g., gestures), or olfactory input (not shown). Human-machine interface devices may also be used to acquire media that are not necessarily directly related to conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0187] The human-machine interface input device may include at least one of the following (only one of each is shown): keyboard 701, mouse 702, touchpad 703, touch screen 710, joystick 705, microphone 706, scanner 708, and camera 707.

[0188] Computer system 700 may also include certain human-machine interface (HMI) output devices. Such HMI output devices can stimulate the senses of at least one human user through, for example, tactile output, sound, light, and smell / taste. These HMI output devices may include tactile output devices (e.g., tactile feedback provided by a touchscreen 710 or joystick 705, but there may also be tactile feedback devices that do not act as input devices), audio output devices (e.g., speakers 709, headphones (not shown)), visual output devices (e.g., screens 710, including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touchscreen input functionality, each with or without tactile feedback functionality—some of which are capable of outputting two-dimensional or more than three-dimensional visual output via stereoscopic output, etc.; virtual reality glasses (not shown), holographic displays, and smoke boxes (not shown)), and printers (not shown).

[0189] The computer system 700 may also include user-accessible storage devices and related media, such as optical media, including CD / DVD ROM / RW 720 and CD / DVD 711 or similar media, flash memory drives 722, removable hard disk drives or solid-state drives 723, conventional magnetic media such as magnetic tapes and floppy disks (not shown), and ROM / ASIC / PLD-based special-purpose devices such as security dongles (not shown).

[0190] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not cover transmission media, carrier waves, or other transient signals.

[0191] Computer system 700 may also include an interface 799 connected to at least one communication network 798. Network 798 may be, for example, a wireless, wired, or optical network. Network 798 may also be a local area network (LAN), wide area network (WAN), metropolitan area network (MAN), vehicular and industrial network, real-time network, latency-tolerant network, etc. Examples of network 798 include, for example, LANs such as Ethernet and wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial networks including CAN buses, etc. Some networks 798 typically require an external network interface adapter (e.g., a USB port of computer system 700) attached to certain general-purpose data ports or peripheral buses (750 and 751); other networks are typically integrated into the core of computer system 700 via a system bus as described below (e.g., integrated into a PC computer system via an Ethernet interface, or integrated into a smartphone computer system via a cellular network interface). Using any of these networks 798, computer system 700 can communicate with other entities. This communication can be one-way receiving (e.g., broadcasting TV), one-way transmitting (e.g., CAN bus to certain CAN bus devices), or bidirectional, such as connecting to other computer systems using a local area or wide area digital network. Certain protocols and protocol stacks can be used on each of the networks and network interfaces described above.

[0192] The aforementioned human-machine interface device, user-accessible storage device, and network interface can be attached to the core 740 of the computer system 700.

[0193] Core 740 may include at least one CPU 741, GPU 742, graphics adapter 717, dedicated programmable processing unit in the form of a Field Programmable Gate Array (FPGA) 743, hardware accelerators 744 for certain tasks, etc. These devices, along with read-only memory (ROM) 745, random-access memory (RAM) 746, and internal high-capacity storage devices 747 such as internal non-user-accessible hard disk drives (HDDs) and SSDs, can be connected via system bus 748. In some computer systems, system bus 748 may be accessed via at least one physical connector to allow for expansion through additional CPUs, GPUs, etc. Peripheral devices can be directly attached to the core's system bus 748 or connected via peripheral bus 751. Peripheral bus architectures include PCI, USB, etc.

[0194] CPUs 741, GPUs 742, FPGAs 743, and accelerators 744 can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM 745 or RAM 746. Temporary data can also be stored in RAM 746, while permanent data can be stored, for example, in an internal mass storage device 747. Fast storage and retrieval of any of the storage devices can be achieved by using a cache memory, which can be closely associated with at least one CPU 741, GPU 742, mass storage device 747, ROM 745, RAM 746, etc.

[0195] The computer-readable medium may contain computer code for performing various computer-implemented operations. The medium and computer code may be designed and constructed specifically for the purposes of this invention, or may belong to a class well-known and available to those skilled in the art of computer software.

[0196] For example, but not as a limitation, the architecture corresponding to computer system 700, particularly core 740, can provide functionality generated when software embodied in at least one tangible computer-readable medium is executed by one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.). Such computer-readable medium can be a medium associated with certain storage devices (e.g., internal core mass storage 747 or ROM 745) that are user-accessible mass storage devices as described above and with the non-volatile nature of core 740. Software implementing various embodiments of the invention can be stored in such devices and executed by core 740. Depending on specific needs, the computer-readable medium may include at least one storage device or chip. The software can cause core 740, particularly its processors (including CPUs, GPUs, FPGAs, etc.), to execute specific processes or specific portions of specific processes described in this application, including defining data structures stored in RAM 746 and modifying such data structures according to the processes defined by the software. Alternatively or as an alternative, the computer system may provide functionality generated by hard-wired logic or otherwise embodied in circuitry (e.g., accelerator 744), which may operate in place of or in conjunction with software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may cover logic, and vice versa. Where appropriate, references to computer-readable media may cover circuitry (e.g., integrated circuits) storing software for execution, circuitry embodying logic for execution, or both. This invention covers any suitable combination of hardware and software.

[0197] While several exemplary embodiments have been described in this invention, modifications, substitutions, and various alternative equivalents exist that fall within the scope of this invention. Therefore, it should be understood that those skilled in the art will be able to design numerous systems and methods that, while not explicitly stated or described herein, embody the principles of the invention and are therefore within its spirit and scope.

Claims

1. A video decoding method executed by at least one processor, characterized in that, include: Receive an encoded video stream, the encoded video stream including images to be used by an artificial intelligence (AI) engine; as well as At least one Supplemental Enhancement Information (SEI) message is parsed from the received image. The SEI message includes syntax elements that indicate whether the image is allowed to be used as input by the AI ​​engine.

2. The method according to claim 1, characterized in that, The syntax element indicates whether the image is allowed to be used by the AI ​​engine as input for the creation of derivative works.

3. The method according to claim 1, characterized in that, The syntax element indicates whether the image is allowed to be used as input for training by the AI ​​engine.

4. The method according to claim 1, characterized in that, The syntax element indicates whether the image is allowed to be used as input by the AI ​​engine for any purpose.

5. The method according to claim 1, characterized in that, The syntax element indicates whether the image requires authorization from the content owner to be used as input by the AI ​​engine for any purpose.

6. The method according to claim 1, characterized in that, The syntax elements include the syntax of any of the following: ai_usage_restrictions, aur_cancel_flag, aur_persistence_flag, aur_num_restrictions_minus1, aur_restriction[i], aur_context_present_flag[i], and aur_context[i].

7. The method according to claim 1, characterized in that, aur_restriction[i] represents any one of four values, which include: The first value indicating "not for any AI application" The second value indicates "not for AI training". The third value indicating "not for use in generative (modified or created) AI", and The fourth value indicates "not for AI inference," and aur_context[i] represents any of the following: The first bitmask indicating "commercial use" The second bitmask indicates "non-commercial use". The third bitmask indicating "official government use", and The fourth bit of the mask indicates "scientific and academic use".

8. A video encoding method executed by at least one processor, characterized in that, include: Receive video, the video including at least one image; Determine at least one Supplemental Enhancement Information (SEI) message, the SEI message including a syntax element indicating whether the image is allowed to be used as input by the AI ​​engine; and Encode a video stream, the video stream including the image and the at least one SEI message.

9. The method according to claim 8, characterized in that, The syntax element indicates whether the image is allowed to be used by the AI ​​engine as input for the creation of derivative works.

10. The method according to claim 8, characterized in that, The syntax element indicates whether the image is allowed to be used as input for training by the AI ​​engine.

11. The method according to claim 8, characterized in that, The syntax element indicates whether the image is allowed to be used as input by the AI ​​engine for any purpose.

12. The method according to claim 8, characterized in that, The syntax element indicates whether the image requires authorization from the content owner to be used as input by the AI ​​engine for any purpose.

13. The method according to claim 8, characterized in that, The syntax elements include any of the following syntaxes: ai_usage_restrictions, aur_cancel_flag, aur_persistence_flag, aur_num_restrictions_minus1, aur_restriction[i], aur_context_present_flag[i], and aur_context[i].

14. The method according to claim 13, characterized in that, aur_restriction[i] represents any one of four values, which include: The first value indicating "not for any AI application" The second value indicates "not for AI training". The third value indicating "not for use in generative (modified or created) AI", and The fourth value indicates "not for AI inference," and aur_context[i] represents any of the following: The first bitmask indicating "commercial use" The second bitmask indicates "non-commercial use". The third bitmask indicating "official government use", and The fourth bit of the mask indicates "scientific and academic use".

15. A method for processing visual media data, characterized in that, The method includes: Obtain at least one of the visual media file and the visual media data stream; and A conversion is performed between a visual media file and a visual media data stream according to format rules, which indicate the determination of at least one SEI message, the SEI message including syntax elements indicating whether the image is allowed to be used as input by the AI ​​engine.

16. The method according to claim 15, characterized in that, The syntax element indicates whether the image is allowed to be used by the AI ​​engine as input for the creation of derivative works.

17. The method according to claim 15, characterized in that, The syntax element indicates whether the image is allowed to be used as input for training by the AI ​​engine.

18. The method according to claim 15, characterized in that, The syntax element indicates whether the image is allowed to be used as input by the AI ​​engine for any purpose.

19. The method according to claim 15, characterized in that, The syntax element indicates whether the image requires authorization from the content owner to be used as input by the AI ​​engine for any purpose.

20. The method according to claim 15, characterized in that, The syntax elements include any of the following syntaxes: ai_usage_restrictions, aur_cancel_flag, aur_persistence_flag, aur_num_restrictions_minus1, aur_restriction[i], aur_context_present_flag[i], and aur_context[i]. aur_restriction[i] represents any one of four values, which include: The first value indicating "not for any AI application" The second value indicates "not for AI training". The third value indicating "not for use in generative (modified or created) AI", and The fourth value indicates "not for AI inference," and aur_context[i] represents any of the following: The first bitmask indicating "commercial use" The second bitmask indicates "non-commercial use". The third bitmask indicating "official government use", and The fourth bit of the mask indicates "scientific and academic use".

Images