Image encoding / decoding methods and apparatus and recording media for storing bit streams
By generating and encoding usage restriction information into a network abstraction layer unit in the image encoding method and device, the problem of ineffective management of usage licenses in the prior art is solved, and the management of usage conditions and the amount of information transmitted are reduced.
Patent Information
- Application Number
- CN202580025019.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-03-29
- Filing Date
- 2025-03-28
- Publication Date
- 2026-11-17
AI Technical Summary
There is a lack of effective methods and devices in the existing technology to configure and notify information related to usage restrictions, especially in the process of image encoding and decoding, and to effectively manage and extend usage license conditions.
By generating information related to usage restrictions in image encoding methods and devices, and encoding it into the network abstraction layer unit of the bitstream, including usage restriction information such as CI cancellation flag, UR cancellation flag and usage license flag, defined as free-form strings or bitmasks, multiple usage license types are specified, supporting discrimination, generation, commercial, non-commercial, academic, modification, redistribution and AI usage licenses.
It enables data users to check the copyright holder's terms of use, reduces the amount of usage restriction information transmitted, and is easy to extend with new restrictions.
Smart Images

Figure CN123125871A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to an image encoding / decoding method and apparatus, as well as a recording medium for storing bit streams. Background Technology
[0002] Recently, the demand for high-resolution and high-quality images, such as HD (high-definition) and UHD (ultra-high-definition) images, has been increasing in various application areas, and therefore, efficient image compression technologies are being discussed.
[0003] Various techniques exist, such as inter-frame prediction techniques that use video compression technology to predict pixel values included in the current frame from frames before or after the current frame, intra-frame prediction techniques that use pixel information in the current frame to predict pixel values included in the current frame, and entropy coding techniques that assign short codes to values that occur frequently and long codes to values that occur infrequently. These image compression techniques can be used to effectively compress image data and send or store it. Summary of the Invention
[0004] Technical issues
[0005] This disclosure provides methods and apparatus for configuring information related to usage restrictions.
[0006] This disclosure provides methods and apparatus for signaling information related to usage restrictions.
[0007] Technical solution
[0008] The image encoding method and apparatus according to this disclosure can receive a video frame to be encoded, encode the received video frame to generate a compressed video frame, generate information related to usage restrictions, and generate a bitstream including the compressed video frame and the information related to usage restrictions. Here, the information related to usage restrictions can be encoded into the network abstraction layer (NAL) unit of the bitstream. The information related to usage restrictions may include usage restriction information related to usage restrictions on the frame.
[0009] In the image encoding method and apparatus according to this disclosure, usage restriction information can be configured in the copyright information of the bitstream.
[0010] In the image encoding method and apparatus according to this disclosure, the use restriction information may be generated based on a CI cancellation mark associated with the persistence of prior copyright information.
[0011] In the image encoding method and apparatus according to this disclosure, the information related to usage restrictions may also include a usage restriction flag related to the presence or absence of usage restriction information.
[0012] In the image encoding method and apparatus according to this disclosure, usage restriction information may be generated based on usage restriction marks.
[0013] In the image encoding method and apparatus according to this disclosure, the usage restriction information may be generated based on the UR cancellation flag, which is associated with the persistence of information related to previous usage restrictions.
[0014] In the image encoding method and apparatus according to this disclosure, the restriction information can be defined as a free-form string.
[0015] In the image encoding method and apparatus according to this disclosure, the restriction information can be defined as a bit mask of u(x).
[0016] In the image encoding method and apparatus according to the present disclosure, the use restriction information can specify at least one of a predefined plurality of use licenses to be applied to the bitstream.
[0017] In the image encoding method and apparatus according to this disclosure, a plurality of licenses may include at least one of the following: a license for use in a discriminative model, a license for use in a generative model, a license for use for commercial purposes, a license for use for non-commercial purposes, a license for use for academic purposes, a license for use for modification, a license for use for redistribution, a license for use for personally identifiable information, a license for use in artificial intelligence (AI), a license for use in AI training, or a license for use in AI inference.
[0018] In the image encoding method and apparatus according to this disclosure, the restriction information may include at least one of the following: a discrimination model license mark, a generation model license mark, a commercial use license mark, a non-commercial use license mark, a redistribution license mark, a modification license mark, a personal identification information license mark, an AI license mark, a training license mark, or an inference license mark.
[0019] The image decoding method and apparatus according to this disclosure can receive a bitstream including encoded video frames and reconstruct the encoded video frames included in the bitstream. Here, the bitstream may include information related to usage restrictions, and this information related to usage restrictions may be obtained from the network abstraction layer (NAL) unit of the bitstream. The information related to usage restrictions may include usage restriction information related to usage restrictions for the frames.
[0020] A computer-readable digital storage medium is provided for storing encoded video / image information, which enables a decoding device according to the present disclosure to perform an image decoding method.
[0021] A computer-readable digital storage medium is provided for storing video / image information generated based on the image encoding method of this disclosure.
[0022] A method and apparatus are provided for transmitting video / image information generated according to the image encoding method of this disclosure.
[0023] Beneficial effects
[0024] According to this disclosure, by configuring usage restriction information related to usage restrictions, data users can check information on usage conditions for copyright holders.
[0025] According to this disclosure, the amount of information transmitted related to usage restrictions can be reduced, and new restrictions can be easily extended. Attached Figure Description
[0026] Figure 1 A video / image encoding system according to this disclosure is shown.
[0027] Figure 2 A schematic block diagram of an encoding apparatus to which embodiments of the present disclosure are applicable and which performs encoding of video / image signals is shown.
[0028] Figure 3 A schematic block diagram of a decoding apparatus to which embodiments of the present disclosure are applicable and which performs decoding of video / image signals is shown.
[0029] Figure 4 A method for generating a bit stream, performed by an encoding device 200 according to the present disclosure, is illustrated.
[0030] Figure 5 An illustrative configuration of an encoding device 200 for performing a method for generating a bitstream according to the present disclosure is shown.
[0031] Figure 6 A method for reconstructing a video frame, performed by a decoding device 300 according to the present disclosure, is illustrated.
[0032] Figure 7 An illustrative configuration of a decoding device 300 for performing a method for reconstructing a video frame according to the present disclosure is shown.
[0033] Figures 8 to 11 Examples of applications subject to restrictions defined by the use of restriction information according to this disclosure are provided.
[0034] Figure 12 Examples of content streaming systems to which embodiments of this disclosure can be applied are shown. Detailed Implementation
[0035] Because this disclosure can be modified in various ways and has multiple embodiments, specific embodiments will be shown in the accompanying drawings and described in detail in the specific embodiments. However, this disclosure is not intended to be limited to the specific embodiments, but should be understood to include all changes, equivalents, and substitutions included within the spirit and scope of this disclosure. Similar reference numerals are used for similar components in the description of the various figures.
[0036] Terms such as "first," "second," etc., may be used to describe various components, but components should not be limited by these terms. Terms are used only to distinguish one component from others. For example, without departing from the scope of this disclosure, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component. Terms include any combination of one or more of the associated terms.
[0037] When a component is described as "connected" or "linked" to another component, it should be understood that it can be directly connected or linked to the other component, but the other component can exist in between. On the other hand, when a component is described as "directly connected" or "directly linked" to another component, it should be understood that there is no other component in between.
[0038] The terminology used in this application is for describing particular embodiments only and is not intended to limit this disclosure. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, it should be understood that terms such as “comprising” or “having” are intended to specify the presence of the features, quantities, steps, operations, components, portions, or combinations thereof described in this specification, but do not preclude the possibility of the presence or addition of one or more other features, quantities, steps, operations, components, portions, or combinations thereof.
[0039] This disclosure relates to video / image coding. For example, the methods / implementations disclosed herein can be applied to methods disclosed in the Multifunctional Video Coding (VVC) standard. Additionally, the methods / implementations disclosed herein can be applied to methods disclosed in the Basic Video Coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the Audio Video Coding 2 (AVS2) standard, or next-generation video / image coding standards (e.g., H.267 or H.268).
[0040] This specification sets forth various implementations of video / image encoding, and unless otherwise specified, these implementations may be combined with each other.
[0041] In this article, video can refer to a collection of images over time. A frame typically refers to a unit representing an image within a specific time period, and a slice / tile is a unit that forms part of a frame during encoding. A slice / tile can include at least one Code Tree Unit (CTU). A frame can consist of at least one slice / tile. A tile is a rectangular area composed of multiple CTUs within a specific tile column and a specific tile row of a frame. A tile column is a rectangular area of CTUs with a height equal to the height of the frame and a width specified by the syntax requirements of the frame parameter set. A tile row is a rectangular area of CTUs with a height specified by the frame parameter set and a width equal to the width of the frame. CTUs within a tile can be arranged continuously according to CTU raster scans, and tiles within a frame can be arranged continuously according to tile raster scans. A slice can include an integer number of complete tiles of a frame that can be exclusively included in a single NAL unit, or an integer number of consecutive complete CTU rows within a tile. Furthermore, a frame can be divided into at least two sub-frames. A sub-frame can be a rectangular area of at least one slice within a frame.
[0042] A pixel, or pelin, can represent the smallest unit that makes up a frame (or image). Additionally, "sample" can be used as the term corresponding to a pixel. A sample can typically represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luminance component or only the pixel / pixel value of the chrominance component.
[0043] A unit can represent the basic unit of image processing. A unit may include a specific region of the image and at least one of the information associated with that region. A unit may include a luminance block and two chrominance (e.g., cb, cr) blocks. In some cases, the term "unit" may be used interchangeably with terms such as "block" or "region". In general, an M×N block may include a set (or array) of transform coefficients or samples (or sample arrays) consisting of M columns and N rows.
[0044] In this document, “A or B” can mean “A only”, “B only”, or “both A and B”. In other words, “A or B” can be interpreted as “A and / or B”. For example, “A, B or C” can mean “A only”, “B only”, “C only”, or “any combination of A, B and C”.
[0045] The forward slash ( / ) or comma used in this article can indicate "and / or". For example, "A / B" can mean "A and / or B". Therefore, "A / B" can mean "A only", "B only", or "both A and B". For example, "A, B, C" can mean "A, B, or C".
[0046] In this document, "at least one of A and B" can mean "only A", "only B" or "both A and B". Furthermore, in this document, expressions such as "at least one of A or B" or "at least one of A and / or B" can be interpreted in the same way as "at least one of A and B".
[0047] Additionally, in this document, "at least one of A, B, and C" can mean "A only", "B only", "C only" or "any combination of A, B, and C". Furthermore, "at least one of A, B, or C" or "at least one of A, B, and / or C" can mean "at least one of A, B, and C".
[0048] Additionally, the parentheses used in this document can indicate "for example". Specifically, when the indication is "prediction (intra-frame prediction)", "intra-frame prediction" can be cited as an example of "prediction". In other words, "prediction" in this document is not limited to "intra-frame prediction", and "intra-frame prediction" can be cited as an example of "prediction". Furthermore, even when the indication is "prediction (i.e., intra-frame prediction)", "intra-frame prediction" can be cited as an example of "prediction".
[0049] In this article, the technical features described individually in a single diagram can be implemented individually or simultaneously.
[0050] Figure 1 A video / image encoding system according to this disclosure is shown.
[0051] Reference Figure 1 A video / image encoding system may include a first device (source device) and a second device (receiving device).
[0052] A source device can transmit encoded video / image information or data to a receiving device in the form of a file or stream via a digital storage medium or network. The source device may include a video source, an encoding device, and a transmitting unit. The receiving device may include a receiving unit, a decoding device, and a renderer. The encoding device may be referred to as a video / image encoding device, and the decoding device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display unit, and the display unit may consist of a separate device or external components.
[0053] A video source can acquire video / images through processes that capture, synthesize, or generate video / images. A video source may include means for capturing video / images and means for generating video / images. Means for capturing video / images may include at least one camera, a video / image archive containing previously captured video / images, etc. Means for generating video / images may include a computer, tablet computer, smartphone, etc., and can generate video / images (electronically). For example, virtual video / images can be generated by a computer, etc., and in this case, the process of capturing video / images can be replaced by a process of generating related data.
[0054] Encoding devices can encode input video / images. They can perform a series of processes such as prediction, transformation, and quantization for compression and encoding efficiency. The encoded data (encoded video / image information) can be output as a bitstream.
[0055] The transmitting unit can send encoded video / image information or data, output in bitstream form, to the receiving unit of the receiving device in the form of a file or stream via a digital storage medium or network. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitting unit can include elements for generating media files according to a predetermined file format, and may include elements for transmission via a broadcast / communication network. The receiving unit can receive / extract the bitstream and send it to a decoding device.
[0056] Decoding devices can decode video / images by performing a series of processes, such as dequantization, inverse transform, and prediction, that correspond to the operations of encoding devices.
[0057] The renderer can render decoded video / images. The rendered video / images can be displayed through a display unit.
[0058] Figure 2 A schematic block diagram of an encoding apparatus that can be applied to embodiments of the present disclosure and perform encoding of video / image signals is shown.
[0059] Reference Figure 2The encoding device 200 may consist of an image segmenter 210, a predictor 220, a residual processor 230, an entropy encoder 240, an adder 250, a filter 260, and a memory 270. The predictor 220 may include an inter-frame predictor 221 and an intra-frame predictor 222. The residual processor 230 may include a transformer 232, a quantizer 233, a dequantizer 234, and an inverse transformer 235. The residual processor 230 may also include a subtractor 231. The adder 250 may be referred to as a reconstructor or a reconstruction block generator. According to embodiments, the image segmenter 210, predictor 220, residual processor 230, entropy encoder 240, adder 250, and filter 260 may be configured by at least one hardware component (e.g., an encoder chipset or processor). Additionally, the memory 270 may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may also include the memory 270 as an internal / external component.
[0060] Image segmenter 210 can segment an input image (or picture or frame) input to encoding device 200 into at least one processing unit. As an example, a processing unit can be referred to as a coding unit (CU). In this case, the coding unit can be recursively segmented from coding tree unit (CTU) or maximum coding unit (LCU) according to a quadtree-binary-tritree (QTBTTT) structure.
[0061] For example, a coding unit can be segmented into multiple deeper coding units based on a quadtree, binary tree, and / or ternary tree structure. In this case, for example, a quadtree structure can be applied first, followed by a binary tree and / or ternary tree structure. Alternatively, a binary tree structure can be applied before the quadtree structure. The coding process according to this specification can be performed based on the final coding unit that is no longer segmented. In this case, based on image characteristics, coding efficiency, etc., the largest coding unit can be directly used as the final coding unit, or, if necessary, the coding unit can be recursively segmented into deeper coding units, and the coding unit with the optimal size can be used as the final coding unit. Here, the coding process can include processes such as prediction, transformation, and reconstruction, as described later.
[0062] As another example, the processing unit may also include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit can be divided or segmented from the aforementioned final encoding unit, respectively. The prediction unit may be a unit for predicting samples, and the transform unit may be a unit for deriving transform coefficients and / or a unit for deriving residual signals from transform coefficients.
[0063] In some cases, a unit can be used interchangeably with terms such as block or region. Generally, an M×N block can represent a set of transform coefficients or samples consisting of M columns and N rows. Samples can typically represent pixels or pixel values, and can represent only the pixel / pixel value of the luminance component, or only the pixel / pixel value of the chrominance component. Samples can be used as a term to form a frame (or image) corresponding to a pixel or cell.
[0064] Encoding device 200 can subtract the prediction signal (prediction block, prediction sample array) output from inter-frame predictor 221 or intra-frame predictor 222 from the input image signal (original block, original sample array) to generate a residual signal (residual signal, residual sample array), and the generated residual signal is sent to converter 232. In this case, the unit in encoding device 200 that subtracts the prediction signal (prediction block, prediction sample array) from the input image signal (original block, original sample array) can be called subtractor 231.
[0065] Predictor 220 can perform prediction on the block to be processed (hereinafter referred to as the current block) and generate a prediction block that includes prediction samples of the current block. Predictor 220 can determine whether to apply intra-frame prediction or inter-frame prediction on a per-block or per-unit basis. Predictor 220 can generate various information about the prediction (e.g., prediction mode information) and send it to entropy encoder 240, as described later in the description of the various prediction modes. The information about the prediction can be encoded in entropy encoder 240 and output as a bitstream.
[0066] Intra-frame predictor 222 can predict the current block by referencing samples within the current frame. Depending on the prediction mode, the referenced samples can be located near the current block or positioned at a specific distance away from the current block. In intra-frame prediction, the prediction mode can include at least one non-directional mode and multiple directional modes. The non-directional mode can include at least one of a DC mode or a planar mode. Depending on the granularity of the prediction direction, the directional modes can include 33 or 65 directional modes. However, this is just an example; more or fewer directional modes can be used depending on the configuration. Intra-frame predictor 222 can determine the prediction mode applied to the current block by using prediction modes applied to neighboring blocks.
[0067] Inter-frame predictor 221 can deduce the predicted block of the current block based on a reference block (reference sample array) specified by motion vectors on a reference frame. In this case, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted on a block, sub-block, or sample basis based on the correlation between motion information between neighboring blocks and the current block. Motion information may include motion vectors and reference frame indices. Motion information may also include inter-frame prediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). For inter-frame prediction, neighboring blocks may include spatially neighboring blocks existing in the current frame and temporally neighboring blocks existing in the reference frame. The reference frame including the reference block and the reference frame including the temporally neighboring block may be the same or different. The temporally neighboring block may be referred to as a co-located reference block, a co-located CU (colCU), etc., and the reference frame including the temporally neighboring block may be referred to as a co-located frame (colPic). For example, inter-frame predictor 221 can configure a motion information candidate list based on neighboring blocks and generate information indicating which candidate to use to deduce the motion vector and / or reference frame index of the current block. Inter-frame prediction can be performed based on various prediction modes. For example, in skip mode and merge mode, the inter-frame predictor 221 can use the motion information of neighboring blocks as the motion information of the current block. In skip mode, unlike merge mode, residual signals may not be sent. In motion vector prediction (MVP) mode, the motion vectors of neighboring blocks are used as motion vector predictors, and the motion vector difference is signaled to indicate the motion vector of the current block.
[0068] Predictor 220 can generate a prediction signal based on various prediction methods described later. For example, the predictor can not only apply intra-frame prediction or inter-frame prediction to predict a block, but can also apply both intra-frame and inter-frame prediction simultaneously. This can be referred to as the Inter-intra-frame Combined Prediction (CIIP) mode. Alternatively, the predictor can predict blocks based on the Intra-Block Copy (IBC) prediction mode, or it can predict blocks based on a palette mode. The IBC prediction mode or palette mode can be used for content image / video coding such as Screen Content Coding (SCC) in games, etc. IBC essentially performs prediction within the current frame, but it can be performed similarly to inter-frame prediction because it derives a reference block within the current frame. In other words, IBC can use at least one of the inter-frame prediction techniques described herein. The palette mode can be considered an example of intra-frame coding or intra-frame prediction. When applying a palette mode, sample values within the frame can be signaled based on information about the palette table and palette index. The prediction signal generated by predictor 220 can be used to generate a reconstructed signal or a residual signal.
[0069] Transformer 232 can generate transform coefficients by applying transform techniques to the residual signal. For example, the transform techniques may include at least one of Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), Karhunen–Loève Transform (KLT), Graph-Based Transform (GBT), or Conditional Nonlinear Transform (CNT). Here, GBT represents the transform obtained from a graph when the relationship information between pixels is represented as a graph. CNT represents the transform obtained based on generating a prediction signal using all previously reconstructed pixels. Furthermore, the transform processing can be applied to square pixel blocks of the same size, or it can be applied to non-square blocks of variable size.
[0070] Quantizer 233 can quantize the transform coefficients and send them to entropy encoder 240, which can encode the quantized signal (information about the quantized transform coefficients) and output it as a bitstream. This information about the quantized transform coefficients can be called residual information. Quantizer 233 can reorder the block-form quantized transform coefficients into a 1D vector form based on the coefficient scan order, and can generate information about the quantized transform coefficients based on this 1D vector form.
[0071] The entropy encoder 240 can perform various encoding methods such as Golomb, context-adaptive variable-length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). The entropy encoder 240 can encode information required for video / image reconstruction other than quantization transform coefficients (e.g., values of syntax elements, etc.) together or separately.
[0072] Encoded information (e.g., encoded video / image information) can be transmitted or stored in bitstream form at the Network Abstraction Layer (NAL) unit level. The video / image information may also include information about various parameter sets such as Adaptive Parameter Set (APS), Picture Parameter Set (PPS), Sequence Parameter Set (SPS), or Video Parameter Set (VPS). Additionally, the video / image information may include general constraint information. Information and / or syntax elements transmitted from the encoding device to / signaled to the decoding device may be included in the video / image information. The video / image information can be encoded and included in the bitstream through the encoding process described above. The bitstream can be transmitted over a network or stored in a digital storage medium. Here, the network may include broadcast networks and / or communication networks, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmitting unit (not shown) for transmitting the signal output from the entropy encoder 240 and / or a storage unit (not shown) for storing the signal may be configured as internal / external components of the encoding device 200, or the transmitting unit may also be included in the entropy encoder 240.
[0073] The quantized transform coefficients output from quantizer 233 can be used to generate a prediction signal. For example, the residual signal (residual block or residual sample) can be reconstructed by applying dequantization and inverse transform to the quantized transform coefficients via dequantizer 234 and inverse transformer 235. Adder 250 can add the reconstructed residual signal to the prediction signal output from inter-frame predictor 221 or intra-frame predictor 222 to generate a reconstructed signal (reconstructed frame, reconstructed block, reconstructed sample array). When there is no residual for the block to be processed (similar to when a skip mode is applied), the prediction block can be used as a reconstructed block. Adder 250 can be referred to as a reconstructor or reconstructed block generator. The generated reconstructed signal can be used for intra-frame prediction of the next block to be processed in the current frame, and can also be used for inter-frame prediction of the next frame by filtering, as described later. Furthermore, luminance mapping and chroma scaling (LMCS) can be applied in frame encoding and / or reconstruction processing.
[0074] Filter 260 can improve subjective / objective image quality by applying filtering to the reconstructed signal. For example, filter 260 can generate a modified reconstructed image by applying various filtering methods to the reconstructed image, and the modified reconstructed image can be stored in memory 270 (specifically, the DPB of memory 270). Various filtering methods can include deblocking filtering, sample adaptive offsetting, adaptive loop filtering, bilateral filtering, etc. Filter 260 can generate various information about the filtering and send it to entropy encoder 240. The information about the filtering can be encoded in entropy encoder 240 and output as a bitstream.
[0075] The modified reconstructed frame sent to memory 270 can be used as a reference frame in inter-frame predictor 221. When inter-frame prediction is applied through it, the encoding device can avoid prediction mismatch between the encoding device 200 and the decoding device, and can also improve encoding efficiency.
[0076] The DPB of memory 270 can store modified reconstructed frames for use as reference frames in inter-frame predictor 221. Memory 270 can store motion information of blocks from which motion information in the current frame is derived (or encoded) and / or motion information of blocks in previously reconstructed frames. The stored motion information can be sent to inter-frame predictor 221 as motion information for spatially or temporally neighboring blocks. Memory 270 can store reconstructed samples of reconstructed blocks in the current frame and send them to intra-frame predictor 222.
[0077] Figure 3 A schematic block diagram of a decoding device that can be implemented using the embodiments of this disclosure and perform decoding of video / image signals is shown.
[0078] Reference Figure 3 The decoding device 300 can be configured to include an entropy decoder 310, a residual processor 320, a predictor 330, an adder 340, a filter 350, and a memory 360. The predictor 330 may include an inter-frame predictor 332 and an intra-frame predictor 331. The residual processor 320 may include a dequantizer 321 and an inverse transformer 322.
[0079] According to the implementation, the entropy decoder 310, residual processor 320, predictor 330, adder 340, and filter 350 described above can be configured by a single hardware component (e.g., a decoder chipset or processor). Additionally, the memory 360 may include a decoded screen buffer (DPB) and can be configured by a digital storage medium. The hardware component may also include the memory 360 as an internal / external component.
[0080] When the input includes a bitstream containing video / image information, the decoding device 300 can respond to... Figure 2 The encoding device processes video / image information to reconstruct the image. For example, the decoding device 300 can deduce units / blocks based on block segmentation information obtained from the bitstream. The decoding device 300 can perform decoding using processing units applied in the encoding device. Therefore, the decoding processing unit can be an encoding unit, and the encoding unit can be segmented from the encoding tree unit or a larger encoding unit according to a quadtree structure, binary tree structure, and / or ternary tree structure. At least one transform unit can be derived from the encoding unit. Furthermore, the reconstructed image signal decoded and output by the decoding device 300 can be played back by a playback device.
[0081] Decoding device 300 can receive data in bitstream form from... Figure 2The signal output by the encoding device can be decoded by the entropy decoder 310. For example, the entropy decoder 310 can parse the bitstream to derive the information (e.g., video / image information) required for image reconstruction (or picture reconstruction). The video / image information may also include information about various parameter sets such as Adaptive Parameter Set (APS), Picture Parameter Set (PPS), Sequence Parameter Set (SPS), or Video Parameter Set (VPS). In addition, the video / image information may also include general constraint information. The decoding device can also decode the picture based on the information about the parameter sets and / or general constraint information. The information and / or syntax elements that are signaled / received, as described later herein, can be decoded and obtained from the bitstream through the decoding process. For example, the entropy decoder 310 can decode the information in the bitstream based on encoding methods such as Exponential Golomb coding, CAVLC, CABAC, etc., and output the values of the syntax elements required for image reconstruction and the quantized values of the transform coefficients of the residuals. More specifically, the CABAC entropy decoding method can receive bins corresponding to each syntax element from the bitstream, determine a context model using information about the syntax element to be decoded, decoding information of adjacent blocks and the block to be decoded, or information about symbols / bins decoded in previous steps, perform arithmetic decoding of bins by predicting the occurrence probability of bins based on the determined context model, and generate symbols corresponding to the values of each syntax element. In this case, after determining the context model, the CABAC entropy decoding method can update the context model by using information about the decoded symbols / bins for the context model of the next symbol / bin. Among the information decoded in the entropy decoder 310, information about prediction is provided to the predictors (inter-frame predictor 332 and intra-frame predictor 331), and the residual values (i.e., quantization transform coefficients and related parameter information) from which entropy decoding has been performed in the entropy decoder 310 can be input to the residual processor 320. The residual processor 320 can derive residual signals (residual blocks, residual samples, residual sample arrays). In addition, information about filtering from the information decoded in the entropy decoder 310 can be provided to the filter 350. Furthermore, the receiving unit (not shown) that receives the signal output from the encoding device can be further configured as an internal / external component of the decoding device 300, or the receiving unit can be a component of the entropy decoder 310.
[0082] Furthermore, the decoding device according to this specification may be referred to as a video / image / screen decoding device, and the decoding device may be divided into an information decoder (video / image / screen information decoder) and a sample decoder (video / image / screen sample decoder). The information decoder may include an entropy decoder 310, and the sample decoder may include at least one of a dequantizer 321, an inverse transformer 322, an adder 340, a filter 350, a memory 360, an inter-frame predictor 332, and an intra-frame predictor 331.
[0083] Dequantizer 321 can dequantize the quantized transform coefficients and output the transform coefficients. Dequantizer 321 can reorder the quantized transform coefficients into two-dimensional blocks. In this case, the reordering can be performed based on the coefficient scan order performed in the encoding device. Dequantizer 321 can perform dequantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain the transform coefficients.
[0084] The inverse transformer 322 performs an inverse transformation on the transformation coefficients to obtain the residual signal (residual block, residual sample array).
[0085] Predictor 320 can perform prediction on the current block and generate a prediction block that includes prediction samples of the current block. Predictor 320 can determine whether to apply intra-frame prediction or inter-frame prediction to the current block based on the prediction information output from entropy decoder 310, and determine the specific intra-frame / inter-frame prediction mode.
[0086] Predictor 320 can generate prediction signals based on various prediction methods described later. For example, predictor 320 can not only apply intra-frame prediction or inter-frame prediction to predict a block, but can also apply intra-frame prediction and inter-frame prediction simultaneously. This can be referred to as the Inter-Frame Intra-Frame Combined Prediction (CIIP) mode. Alternatively, the predictor can predict blocks based on the Intra-Frame Block Copy (IBC) prediction mode, or it can predict blocks based on a palette mode. The IBC prediction mode or palette mode can be used for content image / video coding such as Screen Content Coding (SCC) in games, etc. IBC essentially performs prediction within the current frame, but it can be performed similarly to inter-frame prediction because it derives a reference block within the current frame. In other words, IBC can use at least one of the inter-frame prediction techniques described herein. The palette mode can be considered an example of intra-frame coding or intra-frame prediction. When the palette mode is applied, information about the palette table and palette index can be included in the video / image information and signaled.
[0087] Intra-predictor 331 can predict the current block by referencing samples within the current frame. Depending on the prediction mode, the referenced samples can be located near the current block or at a specific distance away. In intra-prediction, the prediction mode can include at least one non-directional mode and multiple directional modes. Intra-predictor 331 can determine the prediction mode applied to the current block by using prediction modes applied to neighboring blocks.
[0088] Inter-frame predictor 332 can deduce the predicted block of the current block based on a reference block (reference sample array) specified by motion vectors on a reference frame. In this case, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted on a block, sub-block, or sample basis based on the correlation of motion information between neighboring blocks and the current block. Motion information may include motion vectors and reference frame indices. Motion information may also include inter-frame prediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). For inter-frame prediction, neighboring blocks may include spatially neighboring blocks existing in the current frame and temporally neighboring blocks existing in the reference frame. For example, inter-frame predictor 332 can configure a motion information candidate list based on neighboring blocks and deduce the motion vector and / or reference frame index of the current block based on the received candidate selection information. Inter-frame prediction can be performed based on various prediction modes, and the information about the prediction may include information indicating the inter-frame prediction mode of the current block.
[0089] Adder 340 can add the obtained residual signal to the prediction signal (prediction block, prediction sample array) output from the predictor (including inter-frame predictor 332 and / or intra-frame predictor 331) to generate a reconstruction signal (reconstructed frame, reconstruction block, reconstruction sample array). When there is no residual for the block to be processed (similar to when a skip mode is applied), the prediction block can be used as a reconstruction block.
[0090] Adder 340 can be referred to as a reconstructor or reconstruction block generator. The generated reconstructed signal can be used for intra-frame prediction of the next block to be processed in the current frame, can be output through filtering as described later, or can be used for inter-frame prediction of the next frame. In addition, luminance mapping and chroma scaling (LMCS) can be applied in the frame decoding process.
[0091] Filter 350 can improve subjective / objective image quality by applying filtering to the reconstructed signal. For example, filter 350 can generate a modified reconstructed image by applying various filtering methods to the reconstructed image and send the modified reconstructed image to memory 360 (specifically, the DPB of memory 360). Various filtering methods may include deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.
[0092] The (modified) reconstructed frame stored in the DPB of memory 360 can be used as a reference frame in inter-frame predictor 332. Memory 360 can store motion information of blocks in the current frame from which motion information is derived (or decoded) and / or motion information of blocks in previously reconstructed frames. The stored motion information can be sent to inter-frame predictor 332 as motion information of spatially or temporally neighboring blocks. Memory 360 can store reconstructed samples of reconstructed blocks in the current frame and send them to intra-frame predictor 331.
[0093] The embodiments described in this document in the filter 260, inter-frame predictor 221 and intra-frame predictor 222 of the encoding device 200 can also be applied equivalently or correspondingly to the filter 350, inter-frame predictor 332 and intra-frame predictor 331 of the decoding device 300, respectively.
[0094] Figure 4 A method for generating a bit stream, performed by an encoding device 200 according to the present disclosure, is illustrated.
[0095] It can receive video frames to be encoded using the S400.
[0096] The received video frame can be encoded to generate a compressed video frame S410.
[0097] Information related to usage restrictions can be generated S420.
[0098] This disclosure relates to methods for expressing at least one of copyright, usage conditions, or restrictions on bitstreams.
[0099] Before the advent of generative AI, there was little concern about copyright infringement related to AI training and inference. This was because there was a social consensus that data should not be hindered from being used scientifically and that AI should not be hindered from advancing. Furthermore, since the AI being developed at the time was primarily discriminative models, there was no belief that AI output would pose a copyright threat.
[0100] However, media generated by generative AI has progressed to the point where it is impossible to distinguish them from human-generated content, leading to growing concerns about copyright infringement arising from AI-generated media. The necessity of regulations mandating the public disclosure of AI-generated media is being discussed. To mitigate these concerns, conditions and restrictions on copyright holders' use of AI data should be defined, and data users should be able to inspect it.
[0101] However, in achieving this, the following issues may arise for data users and copyright holders. For data users, there are legal risks due to infringement, difficulty in determining whether each individual piece of data from large-scale training data is protected by rights, and practical difficulty in obtaining permission from rights holders. For copyright holders, it is difficult to determine whether AI infringes their copyrights. Specifically, it is difficult to determine whether a work has been used for AI training, and it is difficult to determine whether the results generated by AI infringe on copyrights. Additionally, there are concerns about infringement arising from TDM (Text / Data Mining) disclaimers.
[0102] To address these issues, data users should be able to access information regarding copyright holders and the terms of use of their individual data. To this end, copyright holders should be able to deliver information to data users regarding the use and licensing conditions of their works. In other words, to protect rights when it is difficult to determine whether copyright has been infringed, copyright holders should have methods to notify users of permission, conditions of use, limitations, etc. This disclosure may include methods for specifying the rights of copyright holders in encoded bitstreams.
[0103] Implementation Method 1
[0104] Information relating to usage restrictions (UR) as disclosed herein may be included in copyright information (CI). Copyright information may be intended to display copyright-related phrases (such as copyright notices) in the bitstream.
[0105] Copyright information may include a CI cancellation flag (ci_cancel_flag). ci_cancel_flag can be associated with the persistence of previous copyright information in the current application. For example, a value of 1 for ci_cancel_flag indicates that the persistence of previous copyright information has been cancelled. A value of 0 for ci_cancel_flag indicates that the persistence of copyright information is maintained. Alternatively, a value of 0 for ci_cancel_flag may indicate that copyright-related information follows.
[0106] Copyright-related information may include at least one of the following: a CI persistence flag (ci_persistence_flag), copyright identification information (ci_information), or information related to usage restrictions. Based on a ci_cancel_flag value of 0, copyright-related information can be generated and encoded into a bitstream.
[0107] `ci_persistence_flag` can be related to the persistence of copyright information. When `ci_persistence_flag` is 0, it means the copyright information applies only to the current frame. When `ci_persistence_flag` is 1, it means the copyright information applies to the current frame and all subsequent frames. Here, subsequent frames are those that are output after the current frame in the output order and belong to the same layer as the current frame.
[0108] `ci_information` can represent copyright information related to the image within the persistence scope defined by `ci_cancel_flag` and `ci_persistence_flag`. For example, `ci_information` can include the string "Copyright" or the symbol " Information identifying the copyright owner (such as an individual or company name), the year the work was created, and a use / license phrase (such as "all rights reserved").
[0109] Information related to usage restrictions can include usage restriction information. `usage_restriction` can be defined as an example of usage restriction information. `usage_restriction` can be information related to restrictions such as the conditions under which a screen is used and usage limitations.
[0110] Specifically, usage_restriction can be defined as a usage restriction that applies only to artificial intelligence (AI), or it can be defined as a usage restriction that applies to all users, including both AI and humans.
[0111] Unlike `ci_information`, when defined as a usage restriction specific to AI, `usage_restriction` can define copyright and licensing information for a specific purpose and objective. `usage_restriction` can represent AI-related usage restrictions associated with a screen within the persistence scope defined by `ci_cancel_flag` and `ci_persistence_flag`. Here, usage restrictions can include permitted and prohibited portions. `usage_restriction` can represent restrictions on a user's use of the corresponding screen, and can represent usage restrictions for a specific purpose, usage restrictions for a specific organization, etc.
[0112] For example, usage_restriction can represent phrases such as "do not want it to be used in generative AI", "do not want it to be used by a specific company A", "allow use in discriminative AI", "allow copying, transmission and processing", or "do not allow copying, transmission and processing".
[0113] Alternatively, when defined as a usage restriction for all users, including AI and humans, `usage_restriction` can represent a usage restriction related to a screen within the persistence range defined by `ci_cancel_flag` and `ci_persistence_flag`. `usage_restriction` can represent restrictions on a user's use of the corresponding screen, and can represent usage restrictions for a specific purpose, usage restrictions for a specific organization, etc. Here, usage restrictions can include permitted and prohibited portions.
[0114] For example, usage_restriction can represent phrases such as "do not want to modify and allocate" or "do not want to use for AI training".
[0115] The copyright information can be defined as shown in Table 1.
[0116] [Table 1]
[0117] According to Table 1, when the value of ci_cancel_flag is 0, a usage_restriction can be generated and encoded into the copyright information of the bitstream. When the value of ci_cancel_flag is 1, a usage_restriction can be omitted and not encoded into the copyright information of the bitstream.
[0118] Here, `usage_restriction` can be defined as a descriptor of `st(v)`. In other words, `usage_restriction` can be defined as a free-form string. This is because the high degree of freedom and extensibility of the restriction makes it difficult to define as structured information. When defined as a free-form string, the content can be modified or extended as needed and expressed as standardized information. For example, constraints such as bot exclusion expressions (e.g., `Root.txt`) can be configured, which are general forms of bot exclusion criteria or bot exclusion protocols used as a way to prevent bots from accessing a website.
[0119] Table 2 shows examples of configuring usage_restriction, which can be defined as a free-form string. However, this is merely an example and can be defined as a part of the information defined in Table 2, or as a combination of multiple components.
[0120] [Table 2]
[0121] Information related to usage restrictions may also include a usage restriction flag (usage_restriction_present_flag). The usage restriction flag indicates whether usage restriction information exists. For example, a value of 1 for the usage restriction flag indicates the existence of usage restriction information for the screen (e.g., usage_restriction). A value of 0 for the usage restriction flag indicates the absence of usage restriction information for the screen.
[0122] The usage_restriction can be adaptively generated based on the usage restriction flag and encoded into the bitstream. For example, if the usage restriction flag is 1, a usage_restriction can be generated and encoded into the bitstream. If the usage restriction flag is 0, a usage_restriction can be omitted and not encoded into the bitstream.
[0123] In this case, copyright information can be defined as shown in Table 3.
[0124] [Table 3]
[0125] According to Table 3, if the value of `ci_cancel_flag` is 0, `usage_restriction_present_flag` can be generated and encoded into the bitstream's copyright information. If the value of `ci_cancel_flag` is 1, `usage_restriction_present_flag` can be omitted and not encoded into the bitstream's copyright information. If the value of `usage_restriction_present_flag` is encoded as 1, `usage_restriction` can be generated and encoded into the bitstream's copyright information. Here, `usage_restriction` can be defined as a free-form string, as described in Table 1.
[0126] In this way, when sending usage_restriction with the value of usage_restriction_present_flag set to 1, for images that do not always require usage_restriction, the minimum amount of information required can be used to indicate whether usage_restriction is needed, thus relatively reducing the amount of information to be sent.
[0127] Information relating to usage restrictions is not included in the copyright information and can be defined independently of the copyright information.
[0128] For example, `ci_information` and `usage_restriction` can be defined independently in different SEI messages. `ci_information` and `usage_restriction` can have different persistence levels. This is because the persistence of `usage_restriction` can be defined based on the type of information present in the screen. In the case of `usage_restriction`, the constraint's persistence is maintained only if the screen contains information whose use is restricted (e.g., information that can identify a specific person). On the other hand, in the case of `ci_information`, persistence can be maintained separately from the information present in the screen, within the copyright holder's content.
[0129] As mentioned above, when usage_restriction is defined independently of ci_information, copyright information and information related to usage restrictions can be defined as shown in Tables 4 and 5, respectively.
[0130] [Table 4]
[0131] According to Table 4, the copyright information includes ci_cancel_flag, and based on the value of ci_cancel_flag, ci_persistence_flag and ci_information can be adaptively generated and encoded into the copyright information of the bitstream. This is as described above, and repeated descriptions are omitted here.
[0132] [Table 5]
[0133] According to Table 5, information related to usage restrictions (UR) may include a UR cancellation flag (ur_cancel_flag). ur_cancel_flag can be associated with the persistence of currently applied and previously used restriction-related information. For example, a value of 1 for ur_cancel_flag may indicate that the persistence of previously used restriction-related information has been cancelled. A value of 0 for ur_cancel_flag may indicate that the persistence of used restriction-related information has been maintained. Alternatively, a value of 0 for ur_cancel_flag may indicate that used restriction-related information follows.
[0134] Usage restriction information can include at least one of the UR persistence flag (ur_persistence_flag) or usage restriction information (usage_restriction). Usage restriction information can be generated based on a ur_cancel_flag value of 0, and can be encoded into a bitstream.
[0135] `ur_persistence_flag` relates to the persistence of usage restriction information. When `ur_persistence_flag` is 0, it indicates that the usage restriction information applies only to the current frame. When `ur_persistence_flag` is 1, it indicates that the usage restriction information applies to the current frame and all subsequent frames. Here, subsequent frames are those that are output after the current frame in the output order and belong to the same layer as the current frame.
[0136] `usage_restriction` can be information related to usage restrictions, such as the conditions and limitations of using the screen, and can be defined as a free-form string. This is as described above.
[0137] Implementation Method 2
[0138] The usage_restriction can be defined as a bitmask of the form u(x), rather than a free-form string. For ease of description, it is assumed below that usage_restriction is defined as a bitmask of u(16), but this is merely an example and u(x) of different sizes can be used. Here, x can be an integer greater than or equal to 1.
[0139] For example, a value of 0 for `usage_restriction` can indicate that no individually defined restriction exists. Alternatively, a value of 0 for `usage_restriction` can indicate that use is prohibited. A value of non-0 for `usage_restriction` can indicate that an individually defined restriction exists. Alternatively, a value of non-0 for `usage_restriction` can indicate permission for use for a specific purpose or objective.
[0140] To indicate that use is prohibited, usage_restriction can be generated as a value of 0 and encoded into the bitstream. On the other hand, to indicate that use is permitted, usage_restriction can be generated as a non-zero value and encoded into the bitstream.
[0141] Multiple usage licenses can be defined in the same manner in both encoding and decoding devices. These multiple usage licenses can be associated with restrictions that can be applied to the bitstream. Predefined multiple usage licenses can include at least one of the following: a usage license in a discriminative model, a usage license in a generative model, a usage license for commercial use, a usage license for non-commercial use, a usage license for academic use, a usage license for modification, a usage license for redistribution, a usage license for personally identifiable information (PII), a usage license in AI, a usage license in AI training, or a usage license in AI inference.
[0142] Among multiple usage licenses that can be applied to a bitstream, at least one usage license to be applied to the corresponding bitstream can be determined, and a usage_restriction can be generated to specify the determined usage license and encoded into the bitstream.
[0143] For example, usage_restriction can be defined as shown in Table 6. However, this is merely an example. usage_restriction can also define some of the usage licenses defined in Table 6. Alternatively, usage_restriction can define additional usage licenses for purposes (or targets) other than those defined in Table 6. Furthermore, the bitmasks in Table 6 merely indicate that different bitmasks are assigned to multiple usage licenses, and do not restrict the values of the bitmasks to mapping to corresponding usage licenses.
[0144] [Table 6]
[0145] The allowableForDiscriminativeModel parameter, which indicates whether a variable is allowed to be used in the discriminative model, can be derived based on usage_restriction. The allowedForDiscriminativeModel parameter can be derived as shown in Equation 1 below.
[0146] [Formula 1]
[0147] allowedForDiscriminativeModel = ((usage_restriction & 0x01) > 0 ) ? 1: 0
[0148] According to Equation 1, when the value of (usage_restriction & 0x01) is not 0, the value of allowedForDiscriminativeModel can be deduced to be 1. When the value of allowedForDiscriminativeModel is 1, this indicates that usage in the discriminative model is allowed. When the value of (usage_restriction & 0x01) is 0, the value of allowedForDiscriminativeModel can be deduced to be 0. When the value of allowedForDiscriminativeModel is 0, this indicates that usage in the discriminative model is not allowed.
[0149] The allowableForGenerativeModel can be derived based on usage_restriction. The allowedForGenerativeModel can be derived as shown in Equation 2 below.
[0150] [Equation 2]
[0151] allowedForGenerativeModel = ((usage_restriction & 0x02) > 0 ) ? 1 : 0
[0152] According to Equation 2, when the value of (usage_restriction & 0x02) is not 0, the value of allowedForGenerativeModel can be deduced to be 1. When the value of allowedForGenerativeModel is 1, this indicates that usage in the generative model is allowed. When the value of (usage_restriction & 0x02) is 0, the value of allowedForGenerativeModel can be deduced to be 0. When the value of allowedForGenerativeModel is 0, this indicates that usage in the generative model is not allowed.
[0153] The variable allowingForCommercialUse, indicating whether use for commercial purposes is permitted, can be derived based on usage_restriction. This can be derived as shown in Equation 3 below.
[0154] [Formula 3]
[0155] allowedForCommercialUse = ((usage_restriction & 0x04) > 0 ) ? 1 : 0
[0156] According to Equation 3, when the value of (usage_restriction & 0x04) is not 0, the value of allowedForCommercialUse can be deduced to be 1. When the value of allowedForCommercialUse is 1, this indicates that use for commercial purposes is permitted. When the value of (usage_restriction & 0x04) is 0, the value of allowedForCommercialUse can be deduced to be 0. When the value of allowedForCommercialUse is 0, this indicates that use for commercial purposes is not permitted.
[0157] The variable allowedForAcademicUse, representing whether use for academic purposes is permitted, can be derived based on usage_restriction. allowedForAcademicUse can be derived as shown in Equation 4 below.
[0158] [Formula 4]
[0159] allowedForAcademicUse = ((usage_restriction & 0x08) > 0 ) ? 1 : 0
[0160] According to Equation 4, when the value of (usage_restriction & 0x08) is not 0, the value of allowedForAcademicUse can be deduced to be 1. When the value of allowedForAcademicUse is 1, this indicates that use for academic purposes is permitted. When the value of (usage_restriction & 0x08) is 0, the value of allowedForAcademicUse can be deduced to be 0. When the value of allowedForAcademicUse is 0, this indicates that use for academic purposes is not permitted.
[0161] The variable `allowedForModification`, indicating whether modification is allowed, can be derived based on `usage_restriction`. Here, modification can refer to actions such as extraction, copying, and transfer. `allowedForModification` can be derived as shown in Equation 5 below.
[0162] [Formula 5]
[0163] allowedForModification = ((usage_restriction & 0x10) > 0 ) ? 1 : 0
[0164] According to Equation 5, when the value of (usage_restriction & 0x10) is not 0, the value of allowedForModification can be deduced to be 1. When the value of allowedForModification is 1, this indicates that use for modification is allowed. When the value of (usage_restriction & 0x10) is 0, the value of allowedForModification can be deduced to be 0. When the value of allowedForModification is 0, this indicates that use for modification is not allowed.
[0165] The allowedForRedistribution variable, indicating whether its use is permitted for redistribution, can be derived based on usage_restriction. allowedForRedistribution can be derived as shown in Equation 6 below.
[0166] [Formula 6]
[0167] allowedForRedistribution = ((usage_restriction & 0x20) > 0 ) ? 1 : 0
[0168] According to Equation 6, when the value of (usage_restriction & 0x20) is not 0, the value of allowedForRedistribution can be deduced to be 1. When the value of allowedForRedistribution is 1, this indicates that usage for reassignment is allowed. When the value of (usage_restriction & 0x20) is 0, the value of allowedForRedistribution can be deduced to be 0. When the value of allowedForRedistribution is 0, this indicates that usage for reassignment is not allowed.
[0169] A variable (allowedToUsePII) representing whether the use of personally identifiable information is permitted can be derived based on usage_restriction. Here, personally identifiable information can be facial features, vehicle registration numbers, dates of birth, etc. allowedToUsePII can be derived as shown in Equation 7 below.
[0170] [Formula 7]
[0171] allowedToUsePII = ((usage_restriction & 0x40) > 0 ) ? 1 : 0
[0172] According to Equation 7, when the value of (usage_restriction & 0x40) is not 0, the value of allowedToUsePII can be deduced to be 1. When the value of allowedToUsePII is 1, this indicates that the use of personally identifiable information is permitted. When the value of (usage_restriction & 0x40) is 0, the value of allowedToUsePII can be deduced to be 0. When the value of allowedToUsePII is 0, this indicates that the use of personally identifiable information is not permitted.
[0173] `usage_restriction` can also be defined with the opposite meaning to the examples above. For instance, a value of 0 for `usage_restriction` can indicate that no separately defined restriction exists. Alternatively, a value of 0 for `usage_restriction` can indicate that usage is permitted. A value of non-0 for `usage_restriction` can indicate that a separately defined restriction exists. Alternatively, a value of non-0 for `usage_restriction` can indicate that usage is prohibited for a specific purpose or objective.
[0174] To indicate permission to use, usage_restriction can be generated as a value of 0 and encoded into the bitstream. On the other hand, to indicate prohibition to use, usage_restriction can be generated as a non-zero value and encoded into the bitstream.
[0175] Multiple usage prohibitions can be defined in the same manner in both encoding and decoding devices. These prohibitions can be related to restrictions that can be applied to the bitstream. A predefined set of usage prohibitions may include at least one of the following: usage prohibitions in discriminative models, usage prohibitions in generative models, usage prohibitions for commercial use, usage prohibitions for non-commercial use, usage prohibitions for academic use, usage prohibitions for modification, usage prohibitions for redistribution, usage prohibitions for personally identifiable information (PII), usage prohibitions in AI, usage prohibitions in AI training, or usage prohibitions in AI inference.
[0176] Among multiple usage restrictions that can be applied to a bitstream, at least one usage restriction to be applied to the corresponding bitstream can be determined, and a usage_restriction can be generated to specify the determined usage restriction, which can then be encoded into the bitstream.
[0177] For example, usage_restriction can be defined as shown in Table 7. However, this is merely an example. usage_restriction can also define some of the usage restrictions defined in Table 7. Alternatively, usage_restriction can define usage restrictions for purposes (or targets) other than those defined in Table 7. Furthermore, the bitmasks in Table 7 merely indicate that different bitmasks are assigned to multiple usage restrictions, and do not restrict the values of the bitmasks to mapping to corresponding usage restrictions.
[0178] [Table 7]
[0179] The expression indicating whether variables are not allowed to be used in the discriminative model can be derived based on usage_restriction (disallowedForDiscriminativeModel). This can be derived as shown in Equation 8 below.
[0180] [Formula 8]
[0181] disallowedForDiscriminativeModel = ((usage_restriction & 0x01) > 0 )? 1 : 0
[0182] According to Equation 8, when the value of (usage_restriction & 0x01) is not 0, the value of disallowedForDiscriminativeModel can be deduced to be 1. When the value of disallowedForDiscriminativeModel is 1, this indicates that usage in the discriminative model is not allowed. When the value of (usage_restriction & 0x01) is 0, the value of disallowedForDiscriminativeModel can be deduced to be 0. When the value of disallowedForDiscriminativeModel is 0, this indicates that usage in the discriminative model is allowed.
[0183] The expression indicating whether variables are not allowed to be used in the generative model can be derived based on usage_restriction (disallowedForGenerativeModel). DisallowedForGenerativeModel can be derived as shown in Equation 9 below.
[0184] [Formula 9]
[0185] disallowedForGenerativeModel = ((usage_restriction & 0x02) > 0 ) ? 1: 0
[0186] According to Equation 9, when the value of (usage_restriction & 0x02) is not 0, the value of disallowedForGenerativeModel can be deduced to be 1. When the value of disallowedForGenerativeModel is 1, this indicates that usage is not allowed in the generative model. When the value of (usage_restriction & 0x02) is 0, the value of disallowedForGenerativeModel can be deduced to be 0. When the value of disallowedForGenerativeModel is 0, this indicates that usage is allowed in the generative model.
[0187] The variable `disallowedForCommercialUse`, indicating whether commercial use is prohibited, can be derived based on `usage_restriction`. `disallowedForCommercialUse` can be derived as shown in Equation 10 below.
[0188] [Formula 10]
[0189] disallowedForCommercialUse = ((usage_restriction & 0x04) > 0 ) ? 1 :0
[0190] According to Equation 10, when the value of (usage_restriction & 0x04) is not 0, the value of disallowedForCommercialUse can be deduced to be 1. When the value of disallowedForCommercialUse is 1, this indicates that use for commercial purposes is not allowed. When the value of (usage_restriction & 0x04) is 0, the value of disallowedForCommercialUse can be deduced to be 0. When the value of disallowedForCommercialUse is 0, this indicates that use for commercial purposes is allowed.
[0191] The variable `disallowedForAcademicUse`, indicating whether academic use is prohibited, can be derived based on `usage_restriction`. It can be derived as shown in Equation 11 below.
[0192] [Equation 11]
[0193] disallowedForAcademicUse = ((usage_restriction & 0x08) > 0 ) ? 1 : 0
[0194] According to Equation 11, when the value of (usage_restriction & 0x08) is not 0, the value of disallowedForAcademicUse can be deduced to be 1. When the value of disallowedForAcademicUse is 1, this indicates that use for academic purposes is not allowed. When the value of (usage_restriction & 0x08) is 0, the value of disallowedForAcademicUse can be deduced to be 0. When the value of disallowedForAcademicUse is 0, this indicates that use for academic purposes is allowed.
[0195] The variable `disallowedForModification`, indicating whether its use is disallowed for modification, can be derived based on `usage_restriction`. Here, modification can refer to actions such as extraction, copying, and transfer used for processing. `disallowedForModification` can be derived as shown in Equation 12 below.
[0196] [Equation 12]
[0197] disallowedForModification = ((usage_restriction & 0x10) > 0 ) ? 1 : 0
[0198] According to Equation 12, when the value of (usage_restriction & 0x10) is not 0, the value of disallowedForModification can be deduced to be 1. When the value of disallowedForModification is 1, this indicates that use for modification is not allowed. When the value of (usage_restriction & 0x10) is 0, the value of disallowedForModification can be deduced to be 0. When the value of disallowedForModification is 0, this indicates that use for modification is allowed.
[0199] Whether a variable is disallowed for redistribution can be deduced based on usage_restriction. DisallowedForRedistribution can be derived as shown in Equation 6 below.
[0200] [Equation 13]
[0201] disallowedForRedistribution = ((usage_restriction & 0x20) > 0 ) ? 1 :0
[0202] According to Equation 13, when the value of (usage_restriction & 0x20) is not 0, the value of disallowedForRedistribution can be deduced to be 1. When the value of disallowedForRedistribution is 1, this indicates that usage for reassignment is not allowed. When the value of (usage_restriction & 0x20) is 0, the value of disallowedForRedistribution can be deduced to be 0. When the value of disallowedForRedistribution is 0, this indicates that usage for reassignment is allowed.
[0203] A variable (disallowedToUsePII) representing whether the use of personally identifiable information is prohibited can be derived based on usage_restriction. Here, personally identifiable information can be facial features, vehicle registration numbers, dates of birth, etc. disallowedToUsePII can be derived as shown in Equation 14 below.
[0204] [Formula 14]
[0205] disallowedToUsePII = ((usage_restriction & 0x40) > 0 ) ? 1 : 0
[0206] According to Equation 14, when the value of (usage_restriction & 0x40) is not 0, the value of disallowedToUsePII can be deduced to be 1. When the value of disallowedToUsePII is 1, this indicates that the use of personally identifiable information is not permitted. When the value of (usage_restriction & 0x40) is 0, the value of disallowedToUsePII can be deduced to be 0. When the value of disallowedToUsePII is 0, this indicates that the use of personally identifiable information is permitted.
[0207] As shown in Table 8, usage_restriction can be defined in the copyright information.
[0208] [Table 8]
[0209] The copyright information in Table 8 is as described with reference to Table 1. However, the usage_restriction in Table 8 can be defined as a descriptor of u(16) instead of a free-form string. In other words, usage_restriction can be encoded as an unsigned integer with 16 bits.
[0210] The usage_restriction can be adaptively generated and encoded into the bitstream based on the usage_restriction_present_flag described above. In this case, the usage_restriction can be defined in the copyright information as shown in Table 9.
[0211] [Table 9]
[0212] According to Table 9, when the value of usage_restriction_present_flag is 1, usage_restriction can be generated and encoded into the copyright information of the bitstream. When the value of usage_restriction_present_flag is 0, usage_restriction can be omitted and not encoded into the copyright information of the bitstream.
[0213] Information relating to usage restrictions is not included in the copyright information and can be defined independently of the copyright information, as described above.
[0214] For example, when usage_restriction is defined independently of ci_information, copyright information and information related to usage restrictions can be defined as shown in Tables 10 and 11, respectively.
[0215] [Table 10]
[0216] Table 10 will be described with reference to Table 4, and repeated descriptions will be omitted here.
[0217] [Table 11]
[0218] Table 11 is described with reference to Table 5, and repeated descriptions will be omitted here. However, the usage_restriction in Table 11 can be defined as a descriptor of u(16) instead of a free-form string. In other words, usage_restriction can be encoded as an unsigned integer with 16 bits.
[0219] Implementation Method 3
[0220] This disclosure relates to a method for defining the limitations defined in Embodiments 1 and 2 as flags.
[0221] Regarding the methods presented in Implementation 1 and Implementation 2 that utilize free-form strings and bitmasks, the specific syntax includes all information related to usage restrictions. Therefore, when the usage restriction information is subdivided / diversified, a single syntax may need to include too much information.
[0222] Therefore, this disclosure proposes a method for defining individual restrictions as independent marks. This clarifies the meaning of the restriction for each grammar point, and furthermore, the new definition is sufficient for the relevant grammar of the new restriction, thus allowing for easy extension of the new restriction.
[0223] The usage restriction information disclosed herein may include at least one of the following: allowed_for_discriminative_ai_flag, allowed_for_generative_ai_flag, allowed_for_commercial_use_flag, allowed_for_non_commercial_use_flag, allowed_for_redistribution_flag, allowed_for_modification_flag, allowed_to_use_pii_flag, allowed_for_ai_flag, allowed_for_training_flag, allowed_for_inferencing_flag, or usage_restriction.
[0224] The `allowed_for_discriminative_ai_flag` flag indicates whether its use is permitted in the discriminative model. For example, a value of 1 indicates that it can be used in the discriminative model, while a value of 0 indicates that it cannot be used in the discriminative model.
[0225] The `allowed_for_generative_ai_flag` flag indicates whether its use is permitted in generative models. For example, a value of 1 indicates that it can be used in generative models, while a value of 0 indicates that it cannot be used in generative models.
[0226] The `allowed_for_commercial_use_flag` flag indicates whether commercial use is permitted. For example, a value of 1 indicates that commercial use is allowed, while a value of 0 indicates that commercial use is not permitted.
[0227] The `allowed_for_non_commercial_use_flag` flag indicates whether use for non-commercial purposes is permitted. For example, a value of 1 for `allowed_for_non_commercial_use_flag` indicates that it can be used for non-commercial purposes (e.g., research / academic purposes). A value of 0 for `allowed_for_non_commercial_use_flag` indicates that it cannot be used for non-commercial purposes.
[0228] The `allowed_for_modification_flag` flag indicates whether modification is allowed. For example, a value of 1 indicates that modification is allowed, while a value of 0 indicates that modification is not allowed.
[0229] The `allowed_for_redistribution_flag` flag indicates whether redistribution is allowed. For example, a value of 1 for `allowed_for_redistribution_flag` indicates that redistribution is allowed. A value of 0 for `allowed_for_redistribution_flag` indicates that redistribution is not allowed.
[0230] The `allowed_to_use_pii_flag` flag indicates whether the use of personally identifiable information is permitted. For example, a value of 1 for `allowed_to_use_pii_flag` indicates that personally identifiable information can be used. A value of 0 for `allowed_to_use_pii_flag` indicates that personally identifiable information cannot be used.
[0231] The `allowed_for_ai_flag` flag indicates whether AI is allowed to be used in data / images. For example, when the value of `allowed_for_ai_flag` is 1, it means that AI is allowed to be used in the corresponding data / image. When the value of `allowed_for_ai_flag` is 0, it means that AI is not allowed to be used in the corresponding data / image. In the case of `allowed_for_ai_flag`, this can be an example where no detailed restrictions related to usage limitations are defined.
[0232] The `allowed_for_training_flag` indicates whether data / images are allowed for AI training. For example, a value of 1 indicates that data / images are allowed for AI training, while a value of 0 indicates that data / images are not allowed for AI training.
[0233] The `allowed_for_inferencing_flag` indicates whether data / images are allowed to be used for AI inference. For example, a value of 1 indicates that data / images are allowed to be used for AI inference, while a value of 0 indicates that data / images are not allowed to be used for AI inference.
[0234] The `allowed_for_training_flag` and `allowed_for_inferencing_flag` mentioned above can be examples of defining AI usage permissions in a more granular way than `allowed_for_ai_flag`. Therefore, at least one of `allowed_for_training_flag` or `allowed_for_inferencing_flag` can be adaptively generated based on `allowed_for_ai_flag` and encoded into a bitstream. For example, based on a value of 1 for `allowed_for_ai_flag`, at least one of `allowed_for_training_flag` or `allowed_for_inferencing_flag` can be generated and encoded into a bitstream. On the other hand, based on a value of 0 for `allowed_for_ai_flag`, at least one of `allowed_for_training_flag` or `allowed_for_inferencing_flag` may not be generated and may not be encoded into a bitstream.
[0235] The usage_restriction is as described in Implementation 1 or Implementation 2, and repeated descriptions are omitted here. In other words, usage_restriction is usage restriction information for the screen and can be defined as a free-form string or a bit mask in the form of u(x).
[0236] The usage restriction can be adaptively generated based on the additional usage restriction flag (additional_restriction_present_flag) and encoded into a bitstream. The additional_restriction_present_flag can indicate whether an additional restriction exists. For example, a value of 1 for additional_restriction_present_flag indicates the presence of an additional restriction, while a value of 0 indicates the absence of an additional restriction. Here, additional restriction can refer to restrictions other than those defined by the flags mentioned above.
[0237] Usage restriction information can be adaptively generated based on the usage restriction flag (usage_restriction_present_flag) and encoded into the bitstream. `usage_restriction_present_flag` indicates whether usage restriction information exists for a given screen, as described above. For example, if the value of `usage_restriction_present_flag` is 1, usage restriction information can be generated and encoded into the bitstream. Conversely, if the value of `usage_restriction_present_flag` is 0, usage restriction information can be omitted and not encoded into the bitstream.
[0238] The aforementioned usage restriction information can be configured within the copyright information. Alternatively, the usage restriction information can be configured within information related to usage restrictions, defining the usage restriction information independently of the copyright information.
[0239] Table 12 shows examples of the restrictions defined as flags for free-form strings and bit masks of the form u(x) in Implementation 1 and Implementation 2.
[0240] [Table 12]
[0241] The usage restriction information in Table 12 can be configured in the usage restriction-related information (usage_restriction(payloadSize)). Here, the usage restriction information can include allowed_for_discriminative_ai_flag, allowed_for_generative_ai_flag, allowed_for_commercial_use_flag, allowed_for_non_commercial_use_flag, allowed_for_redistribution_flag, allowed_for_modification_flag, allowed_to_use_pii_flag, and usage_restriction. Additionally, based on a ur_cancel_flag value of 0, usage restriction information can be generated and encoded into the bitstream. Based on an additional_restriction_present_flag value of 1, usage_restriction can be generated and encoded into the bitstream. Refer to Table 5 to describe ur_cancel_flag and ur_persistence_flag in Table 12.
[0242] Table 13 shows examples of flags that define whether the use of AI is permitted.
[0243] [Table 13]
[0244] The usage restriction information in Table 13 can be configured in the copyright information (copyright_information(payloadSize)). Here, the usage restriction information can include allowed_for_ai_flag. Based on a ci_cancel_flag value of 0, allowed_for_ai_flag can be generated and encoded into the bitstream. Refer to Table 4 to describe ci_persistence_flag and ci_information in Table 13.
[0245] Table 14 shows examples of defining whether the use of AI is permitted in a more granular way.
[0246] [Table 14]
[0247] The usage restriction information according to Table 14 can be configured in the copyright information (copyright_information(payloadSize)). Here, the usage restriction information can include allowed_for_training_flag and allowed_for_inferencing_flag. Based on a ci_cancel_flag value of 0, usage restriction information can be generated and encoded into a bitstream. Refer to Table 4 to describe ci_cancel_flag, ci_persistence_flag, and ci_information in Table 14.
[0248] Table 15 shows examples of usage restriction information defined independently of copyright information.
[0249] [Table 15]
[0250] According to Table 15, usage restriction information can be configured in the usage restriction-related information (usage_restriction(payloadSize)). Here, usage restriction information can include allowed_for_training_flag, allowed_for_inferencing_flag, and usage_restriction. Additionally, based on a ur_cancel_flag value of 0, usage restriction information can be generated and encoded into the bitstream. Based on an additional_restriction_present_flag value of 1, usage_restriction can be generated and encoded into the bitstream. Refer to Table 5 to describe ur_cancel_flag and ur_persistence_flag in Table 15.
[0251] Table 16 shows an example of defining usage restriction information in copyright information.
[0252] [Table 16]
[0253] According to Table 16, usage restriction information can be configured in the copyright information (copyright_information(payloadSize)). Here, usage restriction information can include allowed_for_discriminative_ai_flag, allowed_for_generative_ai_flag, allowed_for_commercial_use_flag, allowed_for_non_commercial_use_flag, allowed_for_redistribution_flag, allowed_for_modification_flag, allowed_to_use_pii_flag, and usage_restriction. Additionally, based on a value of ur_cancel_flag of 0, usage restriction information can be generated and encoded into the bitstream. Similarly, based on a value of usage_restriction_present_flag of 1, usage restriction information can be generated and encoded into the bitstream. Furthermore, based on a value of additional_restriction_present_flag of 1, usage_restriction can be generated and encoded into the bitstream. The ci_cancel_flag, ci_persistence_flag, and ci_information in Table 16 are described with reference to Table 4.
[0254] The flags belonging to the restriction information can be defined with the opposite meaning to the example above.
[0255] For example, usage restriction information may include at least one of the following: allowed_for_discriminative_ai_flag, allowed_for_generative_ai_flag, allowed_for_commercial_use_flag, allowed_for_non_commercial_use_flag, allowed_for_redistribution_flag, allowed_for_modification_flag, allowed_to_use_pii_flag, allowed_for_ai_flag, allowed_for_training_flag, allowed_for_inferencing_flag, or usage_restriction.
[0256] The `disallowed_for_discriminative_ai_flag` flag indicates whether use in the discriminative model is prohibited. For example, a value of 1 for `disallowed_for_discriminative_ai_flag` indicates that use in the discriminative model is prohibited. A value of 0 for `disallowed_for_discriminative_ai_flag` also indicates that use in the discriminative model is prohibited.
[0257] The `disallowed_for_generative_ai_flag` flag indicates whether its use in the generative model is prohibited. For example, a value of 1 for `disallowed_for_generative_ai_flag` indicates that its use in the generative model is prohibited. A value of 0 for `disallowed_for_generative_ai_flag` indicates that its use in the generative model is prohibited.
[0258] The `disallowed_for_commercial_use_flag` flag indicates whether commercial use is prohibited. For example, a value of 1 indicates that commercial use is prohibited, while a value of 0 indicates that commercial use is prohibited.
[0259] The `disallowed_for_non_commercial_use_flag` flag indicates whether non-commercial use is prohibited. For example, a value of 1 for `disallowed_for_non_commercial_use_flag` indicates that non-commercial use (e.g., research / academic purposes) is prohibited. A value of 0 for `disallowed_for_non_commercial_use_flag` indicates that non-commercial use is prohibited.
[0260] The `disallowed_for_modification_flag` option indicates whether modification is disallowed. For example, a value of 1 for `disallowed_for_modification_flag` means modification is not allowed. A value of 0 for `disallowed_for_modification_flag` means modification is not allowed.
[0261] The `disallowed_for_redistribution_flag` indicates whether redistribution is disallowed. For example, a value of 1 for `disallowed_for_redistribution_flag` means redistribution is not allowed. A value of 0 for `disallowed_for_redistribution_flag` means redistribution is not allowed.
[0262] The `disallowed_to_use_pii_flag` option indicates whether the use of personally identifiable information is prohibited. For example, a value of 1 for `disallowed_to_use_pii_flag` means that personally identifiable information cannot be used. A value of 0 for `disallowed_to_use_pii_flag` means that personally identifiable information can be used.
[0263] The `disallowed_for_ai_flag` flag indicates whether AI is not allowed to be used in data / video. For example, when the value of `disallowed_for_ai_flag` is 1, it means that AI is not allowed to be used in the corresponding data / video. When the value of `disallowed_for_ai_flag` is 0, it means that AI is allowed to be used in the corresponding data / video.
[0264] The `disallowed_for_training_flag` indicates whether data / images are not allowed to be used for AI training. For example, a value of 1 indicates that data / images are not allowed to be used for AI training, while a value of 0 indicates that data / images are allowed to be used for AI training.
[0265] The `disallowed_for_inferencing_flag` indicates whether data / images are not allowed to be used for AI inference. For example, a value of 1 indicates that data / images are not allowed to be used for AI inference, while a value of 0 indicates that data / images are allowed to be used for AI inference.
[0266] disallowed_for_training_flag and disallowed_for_inferencing_flag can be examples of defining AI usage permissions in a more granular way than disallowed_for_ai_flag.
[0267] At least one of `disallowed_for_training_flag` or `disallowed_for_inferencing_flag` can be adaptively generated based on `disallowed_for_ai_flag`, and it can be encoded into a bitstream. For example, if the value of `disallowed_for_ai_flag` is 1, at least one of `disallowed_for_training_flag` or `disallowed_for_inferencing_flag` can be generated, and it can be encoded into a bitstream. On the other hand, if the value of `disallowed_for_ai_flag` is 0, at least one of `disallowed_for_training_flag` or `disallowed_for_inferencing_flag` may not be generated, and it may not be encoded into a bitstream.
[0268] The usage_restriction is as described in Implementation 1 or Implementation 2, and repeated descriptions are omitted here. In other words, usage_restriction is usage restriction information for the screen and can be defined as a free-form string or a bit mask in the form of u(x).
[0269] The usage restriction can be adaptively generated based on the additional usage restriction present flag, as described above.
[0270] Usage restriction information can be adaptively generated based on the usage restriction flag (usage_restriction_present_flag), as described above.
[0271] The aforementioned usage restriction information can be configured within the copyright information. Alternatively, the usage restriction information can be configured within information related to usage restrictions, defining the usage restriction information independently of the copyright information.
[0272] Table 17 shows examples of the restrictions defined as flags for free-form strings and bit masks of the form u(x) in Implementation 1 and Implementation 2.
[0273] [Table 17]
[0274] The usage restriction information in Table 17 can be configured in the usage restriction-related information (usage_restriction(payloadSize)). Here, the usage restriction information can include disallowed_for_discriminative_ai_flag, disallowed_for_generative_ai_flag, disallowed_for_commercial_use_flag, disallowed_for_non_commercial_use_flag, disallowed_for_redistribution_flag, disallowed_for_modification_flag, disallowed_to_use_pii_flag, and usage_restriction. Additionally, based on a ur_cancel_flag value of 0, usage restriction information can be generated and encoded into the bitstream. Based on an additional_restriction_present_flag value of 1, usage_restriction can be generated and encoded into the bitstream. Refer to Table 5 to describe ur_cancel_flag and ur_persistence_flag in Table 17.
[0275] Table 18 shows examples of flags that define whether the use of AI is permitted.
[0276] [Table 18]
[0277] The usage restriction information in Table 18 can be configured in the copyright information (copyright_information(payloadSize)). Here, the usage restriction information can include the disallowed_for_ai_flag. Based on a ci_cancel_flag value of 0, the disallowed_for_ai_flag can be generated and encoded into the bitstream. Refer to Table 4 to describe ci_cancel_flag, ci_persistence_flag, and ci_information in Table 18.
[0278] Table 19 shows examples of defining whether the use of AI is permitted in a more granular way.
[0279] [Table 19]
[0280] The usage restriction information according to Table 19 can be configured in the copyright information (copyright_information(payloadSize)). Here, the usage restriction information can include disallowed_for_training_flag and diaallowed_for_inferencing_flag. Based on a ci_cancel_flag value of 0, usage restriction information can be generated and encoded into a bitstream. Refer to Table 4 to describe ci_cancel_flag, ci_persistence_flag, and ci_information in Table 14.
[0281] Table 20 shows examples of usage restriction information defined independently of copyright information.
[0282] [Table 20]
[0283] The usage restriction information in Table 20 can be configured in the usage restriction-related information (usage_restriction(payloadSize)). Here, the usage restriction information can include disallowed_for_training_flag, disallowed_for_inferencing_flag, and usage_restriction. Additionally, based on a ur_cancel_flag value of 0, usage restriction information can be generated and encoded into the bitstream. Based on an additional_restriction_present_flag value of 1, usage_restriction can be generated and encoded into the bitstream. Refer to Table 5 to describe ur_cancel_flag and ur_persistence_flag in Table 20.
[0284] Table 21 shows an example of defining usage restriction information in copyright information.
[0285] [Table 21]
[0286] According to Table 21, usage restriction information can be configured in the copyright information (copyright_information(payloadSize)). Here, usage restriction information can include disallowed_for_discriminative_ai_flag, disallowed_for_generative_ai_flag, disallowed_for_commercial_use_flag, disallowed_for_non_commercial_use_flag, disallowed_for_redistribution_flag, disallowed_for_modification_flag, disallowed_to_use_pii_flag, and usage_restriction. Additionally, based on a value of ur_cancel_flag of 0, usage restriction information can be generated and encoded into a bitstream. Furthermore, based on a value of usage_restriction_present_flag of 1, usage restriction information can be generated and encoded into a bitstream. Additionally, based on a value of additional_restriction_present_flag of 1, usage_restriction can be generated and encoded into a bitstream. The ci_cancel_flag, ci_persistence_flag, and ci_information in Table 21 are described with reference to Table 4.
[0287] It can generate a bitstream of S430 including compressed video footage and information related to usage restrictions.
[0288] Information related to usage restrictions can be configured in the SEI message of the bitstream. The SEI message can be included in the Network Abstraction Layer (NAL) unit of the bitstream. Alternatively, information related to usage restrictions can be configured in the copyright information of the SEI message, or it can be configured as a separate SEI message from the copyright information.
[0289] Alternatively, copyright information and / or information relating to usage restrictions as disclosed herein may be configured in the high-level syntax of the bitstream. Here, the high-level syntax may be at least one of the Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Picture Header (PH), or Slice Header (SH).
[0290] Alternatively, the copyright information and / or information relating to usage restrictions disclosed herein may also be defined as an independent NAL unit type within the bitstream.
[0291] Figure 5 An illustrative configuration of an encoding device 200 for performing a method for generating a bitstream according to the present disclosure is shown.
[0292] Reference Figure 5 The encoding device 200 may include a receiver 500, a video compressor 510, a UR generator 520, and a bitstream generator 530.
[0293] Receiver 500 can receive one or more encoded video frames.
[0294] The video compressor 510 can encode one or more received video frames to generate compressed video frames.
[0295] The UR generator 520 can generate information related to usage restrictions that define the limits to be applied to a bitstream.
[0296] Bitstream generator 530 can generate bitstreams that include compressed video footage and information related to usage restrictions.
[0297] Figure 6 A method for reconstructing a video frame, performed by a decoding device 300 according to the present disclosure, is illustrated.
[0298] It can receive bitstreams of encoded video footage from the S600.
[0299] It can reconstruct encoded video footage from the S610 bitstream.
[0300] Video information related to encoded video frames can be extracted from the bitstream. The extracted video information can then be used to reconstruct the encoded video frames.
[0301] Additionally, information related to usage restrictions can be extracted from the bitstream.
[0302] Information related to usage restrictions can be included in the copyright information. This information may include usage restriction details, such as `usage_restriction` as mentioned above. Here, `usage_restriction` can be defined as a free-form string or as a descriptor of `u(x)`. If `ci_cancel_flag` is 0, usage restriction information can be extracted from the bitstream. If `ci_cancel_flag` is 1, usage restriction information can not be extracted from the bitstream. If `usage_restriction_present_flag` is 1, usage restriction information can be extracted from the bitstream. If `usage_restriction_present_flag` is 0, usage restriction information can not be extracted from the bitstream.
[0303] Alternatively, information related to usage restrictions may not be included in the copyright information and may be defined independently of the copyright information. In this case, information related to usage restrictions may include at least one of ur_persistence_flag or usage_restriction. If the value of ur_cancel_flag is 0, ur_persistence_flag and / or usage_restriction can be extracted from the bitstream. If the value of ur_cancel_flag is 1, ur_persistence_flag and / or usage_restriction may not be extracted from the bitstream.
[0304] The usage_restriction extracted from the bitstream can specify at least one of a predefined set of usage licenses to be applied to the corresponding bitstream. Alternatively, the usage_restriction extracted from the bitstream can specify at least one of a predefined set of usage prohibitions to be applied to the corresponding bitstream.
[0305] Alternatively, as described in Implementation 3, at least one usage permission (or usage prohibition) to be applied to the bitstream can be specified by defining usage restriction information in the form of a flag. In this case, the usage_restriction belonging to the usage restriction information can be adaptively extracted from the bitstream based on the additional_restriction_present_flag.
[0306] Information related to usage restrictions can be included in the SEI message of the bitstream. The SEI message can be included in the Network Abstraction Layer (NAL) unit of the bitstream. Alternatively, information related to usage restrictions can be included in the copyright information of the SEI message, or it can be configured as an SEI message independent of the copyright information.
[0307] Alternatively, copyright information and / or information relating to usage restrictions as disclosed herein may be included in the high-level syntax of the bitstream. Here, the high-level syntax may be at least one of the Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Picture Header (PH), or Slice Header (SH).
[0308] Alternatively, the copyright information and / or information relating to usage restrictions disclosed herein may also be defined as an independent NAL unit type in the bitstream.
[0309] Figure 7 An illustrative configuration of a decoding device 300 for performing a method for reconstructing a video frame according to the present disclosure is shown.
[0310] Reference Figure 7 The decoding device 300 may include a receiver 700, a video information extractor 710, and a video reconstructor 720.
[0311] Receiver 700 can receive bitstreams including encoded video images.
[0312] The video information extractor 710 can extract video information related to the encoded video frames from the bitstream. Additionally, the video information extractor 710 can extract information related to usage restrictions from the bitstream.
[0313] The Video Reconstructor 720 can reconstruct encoded video footage based on the extracted video information.
[0314] In the following text, refer to Figures 8 to 11 This section will describe examples of applying the limitations defined by the limitation information according to this disclosure. The limitations and expressions of this disclosure can be applied to all data, including text, video, and audio.
[0315] Figure 8 Examples of processes for generating and distributing data for applications in accordance with the limitations of this disclosure.
[0316] The data to be encoded, along with the restrictions imposed on that data, can be input into the encoder. These restrictions can be defined in the bitstream in the form of copyright information or the aforementioned information related to usage limitations. This information can reduce storage and network resources by restricting requests for or transmission / reception of unavailable data during the bitstream transmission process. Furthermore, users (e.g., AI) can determine whether to utilize the data based on usage conditions. After utilizing the data, the generation and distribution of the AI's output can be defined according to the usage license for redistribution and modification.
[0317] Figure 9 This represents an example of utilizing restricted information during the training process.
[0318] Whether AI can be utilized can be determined by decoding the usage restriction information of individual data in the configuration dataset. To determine this, a comparative analysis of the restrictions defined in the usage restriction information and the user's intended use (e.g., generative AI training, inference) may be necessary. Depending on regulations and the copyright holder's request, it may be permitted only, or it may be recommended that it be used only when permitted. Figure 9 Examples of exploitation during the training process are illustrated, but the use of restricted information can also be applied to purposes other than training (such as inference).
[0319] Figure 10 This represents an example of how restrictive information is used to manage datasets.
[0320] You can configure which datasets can be used each time by checking the usage restrictions information for individual data within the configuration dataset.
[0321] Figure 11 This illustrates an example of improving data utilization by linking the use of limiting information with data preprocessing and optimization.
[0322] Data that is unavailable due to usage restrictions can be optimized to become available. For example, when modification and redistribution are allowed but the use of specific information (e.g., people, signs, etc.) is not permitted, the corresponding specific information can be optimized to be unrecognizable. The information associated with the optimization can then be included in the bitstream and stored.
[0323] Users can examine the usage restrictions and encoder optimization information to determine whether they can be used for AI. Here, encoder optimization information can be non-normalized or structured information (e.g., EOI_SEI of VSEI, etc.).
[0324] In the above embodiments, the method is described based on a flowchart as a series of steps or blocks. However, the corresponding embodiments are not limited to this order of steps. Some steps may occur simultaneously with other steps or in a different order, as described above. In addition, those skilled in the art will understand that the steps shown in the flowchart are not exclusive. Other steps may be included, or one or more steps in the flowchart may be deleted, without affecting the scope of the embodiments of this disclosure.
[0325] The methods described above according to embodiments of the present disclosure can be implemented in software, and the encoding and / or decoding devices according to the present disclosure can be included in an apparatus for performing image processing, such as a TV, computer, smartphone, set-top box, display device, etc.
[0326] In this disclosure, when the implementation is implemented as software, the above-described method can be implemented as a module (process, function, etc.) performing the above-described functions. The module can be stored in memory and can be executed by a processor. The memory can be internal or external to the processor and can be connected to the processor by various well-known means. The processor may include an application-specific integrated circuit (ASIC), another chipset, logic circuitry, and / or data processing devices. The memory may include read-only memory (ROM), random access memory (RAM), flash memory, memory cards, storage media, and / or other storage devices. In other words, the implementations described herein can be executed by implementation on a processor, microprocessor, controller, or chip. For example, the functional units shown in the various figures can be executed by implementation on a computer, processor, microprocessor, controller, or chip. In this case, information for implementation (e.g., information about instructions) or algorithms can be stored in a digital storage medium.
[0327] Furthermore, decoding and encoding devices employing embodiments of this disclosure can be included in multimedia broadcasting transmitting and receiving devices, mobile communication terminals, home theater video devices, digital cinema video devices, surveillance cameras, video conferencing devices, real-time communication devices similar to video communication, mobile streaming devices, storage media, cameras, devices for providing video-on-demand (VOD) services, OTT (over-the-top) video devices, devices for providing internet streaming services, three-dimensional (3D) video devices, virtual reality (VR) devices, augmented reality (AR) devices, video telephony devices, transportation terminals (e.g., vehicle (including autonomous vehicle) terminals, aircraft terminals, ship terminals, etc.), and medical video devices, and can be used to process video signals or data signals. For example, OTT (over-the-top) video devices can include game consoles, Blu-ray players, internet-connected televisions, home theater systems, smartphones, tablet PCs, digital video recorders (DVRs), etc.
[0328] Furthermore, the processing methods applying the embodiments of this disclosure can be generated in the form of a computer-executable program and can be stored in a computer-readable recording medium. Multimedia data with data structures according to the embodiments of this disclosure can also be stored in a computer-readable recording medium. Computer-readable recording media include all types of storage devices and distributed storage devices that store computer-readable data. Computer-readable recording media can include, for example, Blu-ray discs (BD), Universal Serial Bus (USB), ROM, PROM, EPROM, EEPROM, RAM, CD-ROM, magnetic tape, floppy disks, and optical media storage devices. Additionally, computer-readable recording media include media implemented in the form of carrier waves (e.g., transmission via the Internet). Furthermore, bitstreams generated by encoding methods can be stored in computer-readable recording media or transmitted via wired / wireless communication networks.
[0329] Furthermore, the embodiments of this disclosure can be implemented by a computer program product using program code, and the program code can be executed on a computer using the embodiments of this disclosure. The program code can be stored on a computer-readable medium.
[0330] Figure 12 Examples of content streaming systems to which embodiments of this disclosure can be applied are shown.
[0331] Reference Figure 12 A content streaming system that applies embodiments of the present disclosure may generally include an encoding server, a streaming server, a network server, a media storage device, a user device, and a multimedia input device.
[0332] An encoding server generates a bitstream by compressing content input from multimedia input devices such as smartphones, cameras, and camcorders into digital data and then sends it to a streaming server. As another example, when multimedia input devices such as smartphones, cameras, and camcorders directly generate bitstreams, the encoding server can be omitted.
[0333] A bitstream can be generated by an encoding method or bitstream generation method that applies the embodiments of this disclosure, and the streaming server can temporarily store the bitstream during the sending or receiving of the bitstream.
[0334] A streaming server sends multimedia data to a user device via a web server based on a user request, and the web server acts as a medium to inform the user what services are available. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, and the streaming server sends the multimedia data to the user. In this scenario, the content streaming system may include a separate control server, which controls the commands / responses between the various devices in the content streaming system.
[0335] A streaming server can receive content from media storage and / or encoding servers. For example, when receiving content from an encoding server, the content can be received in real time. In this case, to provide a smooth streaming service, the streaming server can store the bitstream for a specific time period.
[0336] Examples of user devices may include mobile phones, smartphones, laptops, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigators, touchscreen PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, head-mounted displays (HMDs)), digital TVs, desktop computers, digital signage, etc.
[0337] In a content streaming system, each server can operate as a distributed server, and in this case, the data received from each server can be distributed and processed.
[0338] The claims set forth herein can be combined in various ways. For example, the technical features of the method claims of this disclosure can be combined and implemented as an apparatus, and the technical features of the apparatus claims of this disclosure can be combined and implemented as a method. Furthermore, the technical features of the method claims and the apparatus claims of this disclosure can be combined and implemented as an apparatus, and the technical features of the method claims and the apparatus claims of this disclosure can be combined and implemented as a method.
Claims
1. A method comprising the following steps: Receive the video frame to be encoded; The received video footage is encoded to generate compressed video footage; Generate information related to the usage restrictions of the UR; as well as Generate a bitstream that includes the compressed video footage and the information related to the usage restrictions. The information related to the usage restrictions is encoded into the Network Abstraction Layer (NAL) unit of the bitstream, and The information related to the usage restrictions includes usage restriction information related to usage restrictions on the screen.
2. The method according to claim 1, wherein, The usage restriction information is configured in the copyright information of the bitstream.
3. The method according to claim 2, wherein, The usage restriction information is generated based on the CI cancellation flag, which is associated with the persistence of previous copyright information.
4. The method according to claim 1, wherein, The information related to the usage restriction also includes a usage restriction flag related to the presence or absence of the usage restriction information, and The usage restriction information is generated based on the usage restriction flag.
5. The method according to claim 1, wherein, The usage restriction information is generated based on the UR cancellation flag, which is associated with the persistence of information related to previous usage restrictions.
6. The method according to claim 1, wherein, The usage restriction information is defined as a free-form string.
7. The method according to claim 1, wherein, The usage restriction information is defined as a bit mask of u(x).
8. The method according to claim 7, wherein, The usage restriction information specifies at least one of a predefined plurality of usage licenses to be applied to the bitstream.
9. The method according to claim 8, wherein, The plurality of licenses includes at least one of the following: a license for use in a discriminative model, a license for use in a generative model, a license for use for commercial purposes, a license for use for non-commercial purposes, a license for use for academic purposes, a license for use for modification, a license for use for redistribution, a license for use for personally identifiable information, a license for use in artificial intelligence (AI), a license for use in AI training, or a license for use in AI inference.
10. The method according to claim 9, wherein, The usage restriction information includes at least one of the following: discrimination model license mark, generation model license mark, commercial use license mark, non-commercial use license mark, reallocation license mark, modification license mark, personal identification information license mark, AI license mark, training license mark, or inference license mark.
11. A method comprising the following steps: Receive bitstreams including encoded video frames; as well as The reconstructed encoded video frames included in the bitstream The bitstream includes information related to the usage restriction UR. The information related to the usage restrictions is obtained from the Network Abstraction Layer (NAL) unit of the bitstream, and The information related to the usage restrictions includes usage restriction information related to usage restrictions on the screen.
12. A computer-readable storage medium for storing a bit stream generated by the method according to claim 1.
13. A method comprising the following steps: A bitstream is generated based on the following steps: receiving a video frame to be encoded; encoding the received video frame to generate a compressed video frame; generating information related to the usage restriction UR; and generating a bitstream including the compressed video frame and the information related to the usage restriction; and Send data including the bit stream. The information related to the usage restrictions is encoded into the Network Abstraction Layer (NAL) unit of the bitstream, and The information related to the usage restrictions includes usage restriction information related to usage restrictions on the screen.
