Coding method and apparatus

Abstract

The application provides a coding method and device, relates to the technical field of media, and comprises the following steps: performing a coding operation on a plurality of display images and a plurality of CRR images to obtain a code stream. First information is used to authenticate to-be-authenticated data to obtain first tree top abstract data. Second information is used to perform signature calculation on the first tree top abstract data to obtain a digital signature. The digital signature is encapsulated into an authentication data set. The code stream comprises a security parameter set and the authentication data set, the security parameter set comprises the first information and the second information, the first information is used to represent an algorithm adopted for generating tree top abstract data, and the second information is used to represent an algorithm adopted for performing digital signature. The first tree top abstract data is tree top abstract data of an encoding end, and the first tree top abstract data comprises tree top abstract data of N main bit streams and / or M CRR slice bit streams, wherein N and M are positive integers.

Description

[0001] This application is a divisional application. The original application has the application number 202311138721.4 and the original application date is September 1, 2023. The entire contents of the original application are incorporated herein by reference. Technical Field

[0002] This application relates to the field of media technology, and in particular to an encoding / decoding method and apparatus. Background Technology

[0003] The Audio Video Coding Standard 3 (AVS3) proposes a Cross Random Access Point Reference (CRR) encoding / decoding scheme based on a library picture. This scheme obtains the library picture from existing video. A long-term buffer is used at the system layer to store the reconstructed library picture. The reconstructed library picture at the encoding end is obtained by encoding the library picture, and at the decoding end, it is obtained by decoding the library picture bitstream (i.e., the CRR slice bitstream). When encoding a video sequence, the reconstructed library picture corresponding to the current frame is obtained from the long-term buffer, and inter-frame predictive coding is performed on the current frame using the reconstructed library picture as a reference picture, thus obtaining the video bitstream and the library picture bitstream corresponding to each reconstructed library picture used in encoding. The video bitstream includes a reference picture marker for each frame, indicating whether a reconstructed library picture was used as a reference picture when encoding that frame. The decoder's system layer decodes each knowledge base image bitstream and stores the resulting reconstructed knowledge base image in a long-term buffer. When decoding the current image bitstream, the decoder retrieves the corresponding reconstructed knowledge base image from the long-term buffer and uses the reconstructed knowledge base as a reference image to perform inter-frame prediction decoding on the current video bitstream.

[0004] With the rapid development of network video surveillance technology, data security issues have become increasingly prominent. Therefore, how to improve the security of CRR media content transmission is one of the problems that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] This application provides encoding / decoding methods and apparatus to improve the security of CRR media content transmission. To achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, this application provides an encoding method, comprising: encoding multiple display images and the aforementioned multiple CRR images to obtain a bitstream; authenticating data to be authenticated based on first information to obtain first treetop digest data; performing a signature calculation on the first treetop digest data based on second information to obtain a digital signature; and encapsulating the digital signature into an authentication dataset. The bitstream includes a security parameter set and an authentication dataset. The security parameter set includes first information and second information, wherein the first information represents the algorithm used to generate the treetop digest, and the second information represents the algorithm used to perform the digital signature. The first treetop digest data is the treetop digest data at the encoding end, and includes treetop digest data of N main bitstreams and / or M CRR slice bitstreams, where N and M are positive integers.

[0007] As can be seen, the method provided in this application embodiment can perform signature authentication on the main bitstream obtained by encoding the display image and / or the slice bitstream obtained by encoding the CRR image, so that the receiving end can determine the integrity and authenticity of the transmitted bitstream based on the signature authentication data, thereby improving the security of the CRR media content transmission process.

[0008] Furthermore, in certain online / offline storage scenarios, the location of CRR slices may change flexibly, while the authentication operation and the location and order of the network abstraction layer are strongly correlated. In such cases, if the CRR slices are not authenticated independently, it is highly likely that the signature verification at the encoding and decoding ends will fail. However, using the embodiments of this application, the CRR bitstream can be authenticated separately, thereby avoiding the problem of signature verification failure at the encoding and decoding ends due to the inability to authenticate the CRR bitstream separately.

[0009] In one possible implementation, the aforementioned authentication dataset also includes third information, which is used to indicate whether the authentication data belongs to a CRR slice.

[0010] For example, the third piece of information can be is_crr_flag. is_crr_flag indicates whether the current authentication dataset belongs to the CRR image. is_crr_flag = 0 indicates that the current authentication data belongs to the main bitstream; is_crr_flag = 1 indicates that the current authentication data belongs to the CRR slice, constraining all CRR slices to be authenticated.

[0011] In one possible implementation, the authentication dataset also includes a fourth piece of information, which indicates the number of authenticated image streams.

[0012] For example, the fourth piece of information can be authentication_data_length_minus1+1, which represents the length of the signature data in bytes, and the value should be 0 to 255.

[0013] In one possible implementation, the aforementioned authentication dataset also includes fifth information, which is used to indicate the CRR image corresponding to the authentication scope of the specified current authentication dataset.

[0014] For example, the fifth piece of information can be `crr_picture_index`. `crr_picture_index` indicates the CRR image corresponding to the authentication range of the current authentication dataset, that is, which CRR image the authentication range of the current authentication dataset belongs to. This parameter can only be parsed when `is_crr_flag` is 1.

[0015] In one possible implementation, the data signature can be encoded, and the encoded data signature can be encapsulated into the authentication dataset.

[0016] For example, the data signature can be Base64 encoded, and the Base64 encoded data signature can be encapsulated into the authentication dataset.

[0017] In one possible implementation, the CRR image can be segmented to obtain multiple CRR slices; the multiple CRR slices are interleaved and arranged between the multiple display images to obtain an image sequence to be encoded; the CRR slices and the display images in the image sequence to be encoded are encoded sequentially to obtain the bitstream.

[0018] In one possible implementation, the bitstream includes multiple CRR slice bitstreams and multiple main bitstreams that are interleaved and ordered.

[0019] It should be noted that encoding the display image and the CRR image into the same bitstream can reduce the design and deployment costs of the overall encoding and decoding system.

[0020] Secondly, this application provides a decoding method, comprising: acquiring a bitstream, wherein the bitstream is obtained by encoding multiple display images and multiple CRR images, the bitstream including a security parameter set and an authentication dataset, the security parameter set including first information and second information, the first information representing the algorithm used to generate the treetop digest, the second information representing the algorithm used to perform the digital signature, and the authentication dataset including a digital signature; decoding the digital signature according to the second information to obtain first treetop digest data, the first treetop digest data being the treetop digest data of the encoding end, the first treetop digest data including treetop digest data of N main bitstreams and / or M CRR slice bitstreams, where N and M are positive integers; authenticating the authentication data corresponding to the authentication dataset according to the first information to obtain second treetop digest data, the second treetop digest data being the treetop digest data of the decoding end, the first treetop digest data including treetop digest data of N main bitstreams and / or M CRR slice bitstreams; and verifying the authentication data according to the first treetop digest and the second treetop digest data.

[0021] In one possible implementation, the digital signature can be decoded; the decoded digital signature can be further decoded based on the second information to obtain the first treetop digest data.

[0022] In one possible implementation, the aforementioned authentication dataset also includes third information, which is used to indicate whether the authentication data belongs to a CRR slice.

[0023] In one possible implementation, the authentication dataset also includes a fourth piece of information, which indicates the number of authenticated image streams.

[0024] Thirdly, this application provides an encoding method, which includes: encoding a plurality of display images and the plurality of CRR images to obtain a bitstream, wherein the bitstream includes a security parameter set, and the security parameter set includes a video encryption key; and encrypting the bitstream according to the video encryption key.

[0025] In one possible implementation, the payload data of the bitstream can be encrypted according to the aforementioned video encryption key, and the payload data includes at least one of the sequence parameter set SPS, image parameter set PPS, main bitstream, and CRR slice bitstream.

[0026] Fourthly, this application provides a decoding method, which includes: acquiring a bitstream, wherein the bitstream is obtained by encoding multiple display images and the multiple CRR images, the bitstream including a security parameter set, the security parameter set including a video encryption key; decrypting the bitstream according to the video encryption key; and determining a reconstructed image based on the decoded bitstream.

[0027] In one possible implementation, the payload data of the bitstream can be decrypted based on the aforementioned video encryption key, and the payload data includes at least one of SPS, PPS, main bitstream, and CRR slice bitstream.

[0028] Fifthly, this application provides an encoding apparatus, comprising: an encoding unit and an authentication unit. The encoding unit is configured to encode multiple display images and the aforementioned multiple CRR images to obtain a bitstream. The bitstream includes a security parameter set and an authentication dataset. The security parameter set includes first information and second information. The first information represents the algorithm used to generate the treetop digest, and the second information represents the algorithm used to perform the digital signature. The authentication unit is configured to authenticate data to be authenticated based on the first information to obtain first treetop digest data. The first treetop digest data is the treetop digest data of the encoding end, and includes treetop digest data of N main bitstreams and / or M CRR slice bitstreams, where N and M are positive integers. The authentication unit is further configured to perform a signature calculation on the first treetop digest data based on the second information to obtain a digital signature. The authentication unit is further configured to encapsulate the digital signature into the authentication dataset.

[0029] In one possible implementation, the aforementioned authentication dataset also includes third information, which is used to indicate whether the authentication data belongs to a CRR slice.

[0030] For example, the third piece of information can be is_crr_flag. is_crr_flag indicates whether the current authentication dataset belongs to the CRR image. is_crr_flag = 0 indicates that the current authentication data belongs to the main bitstream; is_crr_flag = 1 indicates that the current authentication data belongs to the CRR slice, constraining all CRR slices to be authenticated.

[0031] In one possible implementation, the authentication dataset also includes a fourth piece of information, which indicates the number of authenticated image streams.

[0032] For example, the fourth piece of information can be authentication_data_length_minus1+1, which represents the length of the signature data in bytes, and the value should be 0 to 255.

[0033] In one possible implementation, the aforementioned authentication dataset also includes fifth information, which is used to indicate the CRR image corresponding to the authentication scope of the specified current authentication dataset.

[0034] For example, the fifth piece of information can be `crr_picture_index`. `crr_picture_index` indicates the CRR image corresponding to the authentication range of the current authentication dataset, that is, which CRR image the authentication range of the current authentication dataset belongs to. This parameter can only be parsed when `is_crr_flag` is 1.

[0035] In one possible implementation, the authentication unit is specifically used to: encode the aforementioned data signature; and encapsulate the encoded data signature into the aforementioned authentication dataset.

[0036] In one possible implementation, the encoding unit is specifically used to: segment the CRR image to obtain multiple CRR slices; interleave the multiple CRR slices between the multiple display images to obtain a sequence of images to be encoded; and sequentially encode the CRR slices and the display images in the sequence of images to be encoded to obtain the bitstream.

[0037] In one possible implementation, the bitstream includes multiple CRR slice bitstreams and multiple main bitstreams that are interleaved and ordered.

[0038] Sixthly, this application provides a decoding apparatus, comprising: a transceiver unit and an authentication unit. The transceiver unit is configured to acquire a bitstream, the bitstream including a security parameter set and an authentication dataset. The security parameter set includes first information and second information. The first information represents the algorithm used to generate the treetop digest, and the second information represents the algorithm used to perform the digital signature. The authentication dataset includes a digital signature. The authentication unit is configured to decode the digital signature according to the second information to obtain first treetop digest data. The first treetop digest data is the treetop digest data of the encoding end, and the first treetop digest data includes treetop digest data of N main bitstreams and / or M CRR slice bitstreams, where N and M are positive integers. The authentication unit is further configured to authenticate the authentication data corresponding to the authentication dataset according to the first information to obtain second treetop digest data. The second treetop digest data is the treetop digest data of the decoding end, and the first treetop digest data includes treetop digest data of N main bitstreams and / or M CRR slice bitstreams. The authentication unit is also used to verify the authentication data based on the first treetop summary and the second treetop summary data.

[0039] In one possible implementation, the authentication unit is specifically used to: decode the digital signature; and decode the decoded digital signature according to the second information to obtain the first treetop digest data.

[0040] In one possible implementation, the aforementioned authentication dataset also includes third information, which is used to indicate whether the authentication data belongs to a CRR slice.

[0041] In one possible implementation, the authentication dataset also includes a fourth piece of information, which indicates the number of authenticated image streams.

[0042] Seventhly, this application provides an encoding apparatus, comprising: an encoding unit and an encryption unit. The encoding unit is used to encode a plurality of display images and the plurality of CRR images to obtain a bitstream, the bitstream including a security parameter set, the security parameter set including a video encryption key. The encryption unit is used to encrypt the bitstream according to the video encryption key.

[0043] In one possible implementation, the encryption unit is specifically used to: encrypt the payload data of the bitstream according to the aforementioned video encryption key, wherein the payload data includes at least one of the sequence parameter set SPS, image parameter set PPS, main bitstream, and CRR slice bitstream.

[0044] Eighthly, this application provides a decoding apparatus, comprising: a transceiver unit, a decryption unit, and a reconstruction unit. The transceiver unit is configured to acquire a bitstream, wherein the bitstream includes a security parameter set, and the security parameter set includes a video encryption key. The decryption unit is configured to decrypt the bitstream according to the video encryption key. The reconstruction unit is configured to determine a reconstructed image based on the decoded bitstream.

[0045] In one possible implementation, the decryption unit is specifically used to: decrypt the payload data of the bitstream according to the aforementioned video encryption key, wherein the payload data includes at least one of SPS, PPS, main bitstream, and CRR slice bitstream.

[0046] Ninthly, this application provides an encoding apparatus, comprising: one or more processors; a memory for storing one or more computer programs or instructions; wherein when the one or more computer programs or instructions are executed by the one or more processors, the one or more processors implement the method as described in any one of the first aspects.

[0047] In a tenth aspect, this application provides a decoding apparatus, comprising: one or more processors; a memory for storing one or more computer programs or instructions; wherein when the one or more computer programs or instructions are executed by the one or more processors, the one or more processors implement the method as described in any one of the second aspects.

[0048] Eleventhly, embodiments of this application also provide a chip, including: an input interface, an output interface, and at least one processor. Optionally, the chip further includes a memory. The at least one processor is used to execute code in the memory, and when the at least one processor executes the code, the chip implements the method described in the first aspect or any possible implementation thereof.

[0049] Alternatively, the chip described above can also be an integrated circuit.

[0050] In a twelfth aspect, embodiments of this application also provide a computer-readable storage medium for storing a computer program that includes methods for implementing the first aspect or any possible implementation thereof.

[0051] In a thirteenth aspect, embodiments of this application also provide a computer program product containing instructions that, when run on a computer, cause the computer to implement the method described in the first aspect or any possible implementation thereof.

[0052] In a fourteenth aspect, this application provides an electronic device, comprising: a processing circuit and an interface circuit; wherein the interface circuit is configured to couple with a memory external to an encoding / decoding device and provide a communication interface for the processing circuit to access the memory; the processing circuit is configured to execute program instructions in the memory to implement the method as described in the first aspect or any possible implementation thereof.

[0053] In practical implementation, the electronic device can be a chip, the input circuit can be an input pin, the output circuit can be an output pin, and the processing circuit can be a transistor, gate circuit, flip-flop, and various logic circuits. The input signal received by the input circuit can be received and input by, for example, but not limited to, a receiver, and the signal output by the output circuit can be output to, for example, but not limited to, a transmitter and transmitted by the transmitter. Furthermore, the input circuit and the output circuit can be the same circuit, which is used as the input circuit and the output circuit at different times. This application does not limit the specific implementation of the processor and various circuits.

[0054] In one implementation, the electronic device can be a wireless communication device, i.e., a computer device that supports wireless communication functionality. Specifically, the wireless communication device can be a terminal such as a smartphone, or a wireless access network device such as a base station. A system-on-a-chip (SoC) is also called a system-on-chip (SoC). Communication chips may include baseband processing chips and radio frequency (RF) processing chips. The baseband processing chip is sometimes referred to as a modem or baseband chip. The RF processing chip is sometimes referred to as an RF transceiver or RF chip. In physical implementation, some or all of the communication chips can be integrated within the SoC chip. For example, the baseband processing chip is integrated into the SoC chip, while the RF processing chip is not integrated with the SoC chip. The interface circuit can be the RF processing chip in the wireless communication device, and the processing circuit can be the baseband processing chip in the wireless communication device.

[0055] In another implementation, the electronic device can be a component of a wireless communication device, such as an integrated circuit product like a system-on-a-chip (SoC) or communication chip. The interface circuit can be an input / output interface, interface circuit, output circuit, input circuit, pins, or related circuits on the chip or chip system. The processor can also be represented as a processing circuit or logic circuit.

[0056] The encoding / decoding device, computer storage medium, computer program product, and chip provided in this embodiment are all used to execute the encoding / decoding method provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the encoding / decoding method provided above, and will not be repeated here. Attached Figure Description

[0057] Figure 1a An exemplary block diagram of a decoding system provided in an embodiment of this application;

[0058] Figure 1b An exemplary block diagram of a video decoding system provided in an embodiment of this application;

[0059] Figure 2 An exemplary block diagram of a video encoder provided in an embodiment of this application;

[0060] Figure 3 An exemplary block diagram of a video decoder provided in an embodiment of this application;

[0061] Figure 4 An exemplary schematic diagram of a candidate image block provided in an embodiment of this application;

[0062] Figure 5 An exemplary block diagram of a video decoding device provided in an embodiment of this application;

[0063] Figure 6 An exemplary block diagram of the apparatus provided in the embodiments of this application;

[0064] Figure 7 A schematic diagram of a video sequence provided in an embodiment of this application;

[0065] Figure 8 A flowchart illustrating an encoding method provided in an embodiment of this application;

[0066] Figure 9 This application provides a schematic diagram of an encoding / decoding process.

[0067] Figure 10 A flowchart illustrating a decoding method provided in an embodiment of this application;

[0068] Figure 11 A flowchart illustrating an encoding method provided in an embodiment of this application;

[0069] Figure 12 A flowchart illustrating a decoding method provided in an embodiment of this application;

[0070] Figure 13 This is a schematic diagram of the structure of an encoding device provided in an embodiment of this application;

[0071] Figure 14 This is a schematic diagram of the structure of a decoding device provided in an embodiment of this application;

[0072] Figure 15 This is a schematic diagram of another encoding device provided in an embodiment of this application;

[0073] Figure 16 This is a schematic diagram of another decoding device provided in an embodiment of this application;

[0074] Figure 17 This is a schematic diagram of the structure of a chip provided in an embodiment of this application;

[0075] Figure 18 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0077] The terms "first," "second," etc., used in the specification, embodiments, claims, and drawings of this application are for distinguishing purposes only and should not be construed as indicating or implying relative importance or order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0078] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0079] Data encoding and decoding comprises two parts: data encoding and data decoding. Data encoding is performed on the source side (or commonly referred to as the encoder side) and typically involves processing (e.g., compressing) the raw data to reduce the amount of data required to represent it (thus enabling more efficient storage and / or transmission). Data decoding is performed on the destination side (or commonly referred to as the decoder side) and typically involves performing the inverse processing relative to the encoder side to reconstruct the original data. The "encoding and decoding" of data in the embodiments of this application should be understood as either "encoding" or "decoding" of the data. The encoding and decoding parts are also collectively referred to as encoding and decoding (CODEC).

[0080] In lossless data encoding, the original data can be reconstructed, meaning the reconstructed original data has the same quality as the original data (assuming no transmission loss or other data loss during storage or transmission). In lossy data encoding, further compression is performed through quantization to reduce the amount of data required to represent the original data, but the decoder cannot completely reconstruct the original data, meaning the quality of the reconstructed original data is lower or worse than the quality of the original data.

[0081] This application's embodiments can be applied to video data and other data requiring compression / decompression. The following description uses video data encoding (hereinafter referred to as video encoding) as an example to illustrate this application's embodiments. Other types of data (such as image data, audio data, integer data, and other data requiring compression / decompression) can be referred to the following description, and will not be repeated in this application's embodiments. It should be noted that, compared to video encoding, the encoding of audio data and integer data does not require dividing the data into blocks; instead, the data can be directly encoded.

[0082] Video coding generally refers to the processing of image sequences that form a video or video sequence. In the field of video coding, the terms "picture," "frame," or "image" can be used synonymously.

[0083] Several video coding standards fall under the category of "lossy hybrid video coding and decoding" (i.e., combining spatial and temporal prediction in the pixel domain with 2D transform coding in the transform domain for applying quantization). Each image in a video sequence is typically segmented into a set of non-overlapping blocks, which are usually encoded at the block level. In other words, the encoder typically processes the video at the block (video block) level, for example, generating prediction blocks through spatial (intra-frame) prediction and temporal (inter-frame) prediction; subtracting the prediction blocks from the current block (the block currently being processed / to be processed) to obtain residual blocks; transforming and quantizing the residual blocks in the transform domain to reduce the amount of data to be transmitted (compressed), while the decoder applies the inverse processing relative to the encoder to the encoded or compressed blocks to reconstruct the current block for representation. Additionally, the encoder needs to repeat the decoder's processing steps so that the encoder and decoder generate the same predictions (e.g., intra-frame and inter-frame predictions) and / or reconstruct pixels for processing, i.e., encoding subsequent blocks.

[0084] In the following embodiment of the decoding system 10, the encoder 20 and decoder 30 are based on Figures 1a to 3 Describe it.

[0085] Figure 1a This is an exemplary block diagram of a decoding system 10 provided for an embodiment of this application, such as a video decoding system 10 (or simply decoding system 10) that can utilize the technology of the embodiments of this application. The video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) in the video decoding system 10 represent devices, etc., that can be used to perform various technologies according to the various examples described in the embodiments of this application.

[0086] like Figure 1a As shown, the decoding system 10 includes a source device 12, which provides encoded image data 21, such as encoded images, to a destination device 14 for decoding the encoded image data 21.

[0087] The source device 12 includes an encoder 20, and optionally may include an image source 16, a preprocessor (or preprocessing unit) 18 such as an image preprocessor, and a communication interface (or communication unit) 22.

[0088] Image source 16 may include or may be any type of image capture device for capturing real-world images, and / or any type of image generation device, such as a computer graphics processor for generating computer animation images or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage device storing any of the images mentioned above.

[0089] To distinguish the processing performed by the preprocessor (or preprocessing unit) 18, the image (or image data) 17 may also be referred to as the raw image (or raw image data) 17.

[0090] The preprocessor 18 receives the raw image data 17 and preprocesses it to obtain a preprocessed image (or preprocessed image data) 19. For example, the preprocessing performed by the preprocessor 18 may include cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction. It is understood that the preprocessing unit 18 may be an optional component.

[0091] Video encoder (or encoder) 20 is used to receive preprocessed image data 19 and provide encoded image data 21 (hereinafter referred to as...) Figure 2 (and so on, for further description).

[0092] The communication interface 22 in the source device 12 can be used to: receive encoded image data 21 and send encoded image data 21 (or other arbitrarily processed version) to another device such as the destination device 14 or any other device via the communication channel 13 for storage or direct reconstruction.

[0093] The target device 14 includes a decoder 30, and optionally may include a communication interface (or communication unit) 28, a post-processor (or post-processing unit) 32 and a display device 34.

[0094] The communication interface 28 in the destination device 14 is used to receive encoded image data 21 (or other processed versions) directly from the source device 12 or from any other source device such as a storage device, for example, the storage device is an encoded image data storage device, and to provide the encoded image data 21 to the decoder 30.

[0095] Communication interfaces 22 and 28 can be used to send or receive encoded image data (or encoded data 21) through a direct communication link between source device 12 and destination device 14, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any combination thereof.

[0096] For example, the communication interface 22 can be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission encoding or processing, so as to transmit it on a communication link or communication network.

[0097] Communication interface 28 corresponds to communication interface 22. For example, it can be used to receive transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain encoded image data 21.

[0098] Both communication interface 22 and communication interface 28 can be configured as follows: Figure 1a The arrow pointing from the source device 12 to the corresponding communication channel 13 of the destination device 14 indicates a one-way communication interface or a two-way communication interface, and can be used to send and receive messages, etc., to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission, such as encoded image data transmission, etc.

[0099] Video decoder (or decoder) 30 is used to receive encoded image data 21 and provide decoded image data (or decoded image data) 31 (hereinafter referred to as...). Figure 3 (and so on, for further description).

[0100] The post-processor 32 is used to post-process the decoded image data 31 (also known as the reconstructed image data) to obtain post-processed image data 33. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color adjustment, trimming or resampling, or any other processing to generate the decoded image data 31 for display by the display device 34, etc.

[0101] Display device 34 is used to receive post-processed image data 33 to display the image to a user or viewer. Display device 34 can be or includes any type of display for representing the reconstructed image, such as an integrated or external display screen or monitor. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display screen.

[0102] The decoding system 10 also includes a training engine 25, which is used to train the encoder 20 (especially the entropy coding unit 270 in the encoder 20) or the decoder 30 (especially the entropy decoding unit 304 in the decoder 30) to entropy encode the image block to be encoded according to the estimated probability distribution obtained. For a detailed description of the training engine 25, please refer to the following method test example.

[0103] although Figure 1a The source device 12 and destination device 14 are shown as independent devices, but device embodiments may also include both source device 12 and destination device 14, or the functions of both source device 12 and destination device 14, that is, simultaneously including source device 12 or its corresponding functions and destination device 14 or its corresponding functions. In these embodiments, source device 12 or its corresponding functions and destination device 14 or its corresponding functions may be implemented using the same hardware and / or software, or by separate hardware and / or software, or any combination thereof.

[0104] According to the description, Figure 1a The presence and (accurate) division of different units or functions in the source device 12 and / or destination device 14 shown may vary depending on the actual device and application, which is obvious to those skilled in the art.

[0105] Please refer to Figure 1b , Figure 1b This is an exemplary block diagram of a video decoding system 40 provided in an embodiment of this application. The encoder 20 (e.g., video encoder 20) or decoder 30 (e.g., video decoder 30), or both, can be transmitted via... Figure 1bThe processing circuitry in the video decoding system 40 shown is implemented using, for example, one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video encoding processors, or any combination thereof. Please refer to... Figure 2 and Figure 3 , Figure 2 This is an exemplary block diagram of a video encoder provided in an embodiment of this application. Figure 3 This is an exemplary block diagram of a video decoder provided in an embodiment of this application. The encoder 20 can be implemented using processing circuitry 46 to include reference... Figure 2 Encoder 20 refers to various modules discussed herein and / or any other encoder system or subsystem described herein. Decoder 30 may be implemented via processing circuitry 46 to include references. Figure 3 Decoder 30 refers to the various modules discussed herein and / or any other decoder system or subsystem described herein. The processing circuitry 46 described above can be used to perform the various operations discussed below. For example... Figure 5 As shown, if some of the technology is implemented in software, the device can store the software instructions in a suitable non-transitory computer-readable storage medium and execute the instructions in hardware using one or more processors, thereby performing the technology of the embodiments of this application. One of the video encoder 20 and video decoder 30 can be integrated into a single device as part of a combined codec (encoder / decoder, CODEC), such as... Figure 1b As shown.

[0106] Source device 12 and destination device 14 can include any of a variety of devices, including any type of handheld or fixed device, such as laptops or tablets, mobile phones, smartphones, tablets or tablets, cameras, desktop computers, set-top boxes, televisions, display devices, digital media players, video game consoles, video streaming devices (e.g., content service servers or content distribution servers), broadcast receiving devices, broadcast transmitting devices, and monitoring devices, etc., and may or may not use any type of operating system. Source device 12 and destination device 14 can also be devices in a cloud computing scenario, such as virtual machines in a cloud computing scenario. In some cases, source device 12 and destination device 14 may be equipped with components for wireless communication. Therefore, source device 12 and destination device 14 can be wireless communication devices.

[0107] Source device 12 and destination device 14 can install virtual reality (VR), augmented reality (AR), or mixed reality (MR) applications, and can run VR, AR, or MR applications based on user actions (such as clicking, touching, swiping, shaking, voice control, etc.). Source device 12 and destination device 14 can acquire images / videos of any object in the environment through cameras and / or sensors, and then display virtual objects on the display device based on the acquired images / videos. These virtual objects can be virtual objects in VR, AR, or MR scenes (i.e., objects in the virtual environment).

[0108] It should be noted that, in this embodiment of the application, the virtual scene application in the source device 12 and the destination device 14 can be an application built into the source device 12 and the destination device 14 themselves, or it can be an application provided by a third-party service provider that is installed by the user. No specific limitation is made in this regard.

[0109] Furthermore, source device 12 and destination device 14 can be equipped with real-time video transmission applications, such as live streaming applications. Source device 12 and destination device 14 can capture images / videos via cameras and then display the captured images / videos on a display device.

[0110] In some cases, Figure 1a The video decoding system 10 shown is merely exemplary. The technology provided in this application embodiment can be applied to video encoding settings (e.g., video encoding or video decoding), which do not necessarily include any data communication between the encoding device and the decoding device. In other examples, data is retrieved from local memory, sent over a network, etc. The video encoding device may encode data and store it in memory, and / or the video decoding device may retrieve data from memory and decode it. In some examples, encoding and decoding are performed by devices that do not communicate with each other but only encode data to memory and / or retrieve and decode data from memory.

[0111] Please refer to Figure 1b , Figure 1b An exemplary block diagram of the video decoding system 40 provided in this application embodiment is shown below. Figure 1b As shown, the video decoding system 40 may include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video encoder / decoder implemented by processing circuitry 46), an antenna 42, one or more processors 43, one or more memory storage devices 44, and / or a display device 45.

[0112] like Figure 1b As shown, the imaging device 41, antenna 42, processing circuitry 46, video encoder 20, video decoder 30, processor 43, memory storage 44, and / or display device 45 are capable of communicating with each other. In different instances, the video decoding system 40 may contain only the video encoder 20 or only the video decoder 30.

[0113] In some instances, antenna 42 can be used to transmit or receive encoded bitstreams of video data. Additionally, in some instances, display device 45 can be used to present video data. Processing circuitry 46 can include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, etc. Video decoding system 40 can also include an optional processor 43, which can similarly include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, etc. Furthermore, memory storage 44 can be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.). In a non-limiting example, memory storage 44 can be implemented using high-speed cache memory. In other instances, processing circuitry 46 can include memory (e.g., cache, etc.) for implementing image buffers, etc.

[0114] In some instances, the video encoder 20 implemented via logic circuitry may include (e.g., implemented via processing circuitry 46 or memory storage 44) an image buffer and (e.g., implemented via processing circuitry 46) a graphics processing unit. The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the video encoder 20 implemented via processing circuitry 46 to implement reference... Figure 2 And / or any other encoder system or subsystem described herein, including the various modules discussed. Logic circuits can be used to perform the various operations discussed herein.

[0115] In some instances, the video decoder 30 can be implemented in a similar manner via the processing circuitry 46 to implement the reference. Figure 3The video decoder 30 and / or any other decoder system or subsystem described herein are various modules discussed. In some instances, the logic circuit-implemented video decoder 30 may include (implemented via processing circuitry 46 or memory storage 44) an image buffer and (e.g., implemented via processing circuitry 46) a graphics processing unit. The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the video decoder 30 implemented via processing circuitry 46 to implement reference... Figure 3 And / or any other decoder system or subsystem described herein, including the various modules discussed.

[0116] In some instances, antenna 42 can be used to receive encoded bitstreams of video data. As discussed herein, the encoded bitstream may contain data related to encoded video frames, indicators, index values, mode selection data, etc., such as data related to code segmentation (e.g., transform coefficients or quantized transform coefficients, optional indicators, and / or data defining code segmentation). Video decoding system 40 may also include a video decoder 30 coupled to antenna 42 for decoding the encoded bitstream. Display device 45 is used to display the video frames.

[0117] It should be understood that, for the examples described with reference to video encoder 20 in this application embodiment, video decoder 30 can be used to perform the reverse process. Regarding signaling syntax elements, video decoder 30 can be used to receive and parse such syntax elements, and accordingly decode the associated video data. In some examples, video encoder 20 can entropy-encode syntax elements into an encoded video bitstream. In such instances, video decoder 30 can parse such syntax elements and accordingly decode the associated video data.

[0118] For ease of description, embodiments of this application are described with reference to the Universal Video Coding (VVC) reference software or the High-Efficiency Video Coding (HEVC) developed by the ITU-T Video Coding Experts Group (VCEG) and the Joint Collaboration Team on Video Coding (JCT-VC) of the ISO / IEC Moving Picture Experts Group (MPEG). Those skilled in the art will understand that embodiments of this application are not limited to HEVC or VVC.

[0119] Encoders and Encoding Methods

[0120] like Figure 2As shown, the video encoder 20 includes an input terminal (or input interface) 201, a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output terminal (or output interface) 272. The mode selection unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254, and a segmentation unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). Figure 2 The video encoder 20 shown can also be called a hybrid video encoder or a video encoder based on a hybrid video codec.

[0121] See Figure 2 Inter-frame prediction units are trained target models (also known as neural networks) that process input images, image regions, or image patches to generate predicted values ​​for those patches. For example, a neural network for inter-frame prediction receives an input image, image region, or image patch and generates predicted values ​​for that image, image region, or image patch.

[0122] The residual calculation unit 204, transform processing unit 206, quantization unit 208, and mode selection unit 260 constitute the forward signal path of encoder 20, while the inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, buffer 216, loop filter 220, decoded picture buffer (DPB) 230, inter-frame prediction unit 244, and intra-frame prediction unit 254 constitute the backward signal path of encoder 20. The backward signal path of encoder 20 corresponds to the signal path of decoder (see [link to decoder]). Figure 3 The decoder 30 in the video encoder 20 consists of an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter 220, a decoded image buffer 230, an inter-frame prediction unit 244, and an intra-frame prediction unit 254.

[0123] Image and image segmentation (images and patches)

[0124] Encoder 20 can be used to receive images (or image data) 17 via input terminal 201, for example, images in an image sequence forming a video or video sequence. The received images or image data can also be pre-processed images (or pre-processed image data) 19. For simplicity, the following description uses image 17. Image 17 can also be referred to as the current image or the image to be encoded (especially in video encoding when the current image is distinguished from other images, such as those in the same video sequence, i.e., the video sequence that also includes the current image, previously encoded images, and / or decoded images).

[0125] A digital image is, or can be viewed as, a two-dimensional array or matrix of pixels with intensity values. Pixels in an array are also called pixels (short for image element). The number of pixels in the array or image along the horizontal and vertical directions (or axes) determines the image size and / or resolution. To represent color, three color components are typically used, meaning an image can be represented as or comprise an array of three pixels. In RBG format or color space, an image includes corresponding arrays of red, green, and blue pixels. However, in video coding, each pixel is typically represented in a luma / chroma format or color space, such as YCbCr, which includes the luma component indicated by Y (sometimes also represented by L) and two chroma components represented by Cb and Cr. The luma component Y represents the brightness or grayscale level intensity (e.g., both are the same in grayscale images), while the two chroma components Cb and Cr represent the chroma or color information components. Accordingly, a YCbCr format image consists of a luminance pixel array for the luminance pixel value (Y) and two chrominance pixel arrays for the chrominance values ​​(Cb and Cr). An RGB format image can be converted or transformed to YCbCr format, and vice versa; this process is also known as color conversion or transformation. If the image is black and white, it may only include the luminance pixel array. Accordingly, the image can be, for example, a monochrome format luminance pixel array or a 4:2:0, 4:2:2, and 4:4:4 color format luminance pixel array and two corresponding chrominance pixel arrays.

[0126] In one embodiment, the video encoder 20 may include an image segmentation unit ( Figure 2(Not shown in the image) is used to segment image 17 into multiple (typically non-overlapping) image blocks 203. These blocks may also be referred to as root blocks, macroblocks (H.264 / AVC), coding tree blocks (CTBs), or coding tree units (CTUs) in the H.265 / HEVC and VVC standards. Segmentation units can be used to apply the same block size and a corresponding grid with defined block sizes to all images in a video sequence, or to vary the block size between images, subsets of images, or groups of images, segmenting each image into corresponding blocks.

[0127] In other embodiments, the video encoder may be used to directly receive blocks 203 of image 17, for example, one, several, or all of the blocks that make up image 17. Image block 203 may also be referred to as the current image block or the image block to be encoded.

[0128] Similar to image 17, image block 203 is also a two-dimensional array or matrix composed of pixels with intensity values ​​(pixel values), but image block 203 is smaller than that of image 17. In other words, block 203 may include a pixel array (e.g., a luminance array in the case of monochrome image 17 or a luminance or chrominance array in the case of a color image) or a three-pixel array (e.g., a luminance array and two chrominance arrays in the case of color image 17) or any other number and / or type of array depending on the color format used. The number of pixels in the horizontal and vertical directions (or axes) of block 203 defines the size of block 203. Accordingly, the block may be an M×N (M columns × N rows) pixel array, or an M×N transform coefficient array, etc.

[0129] In one embodiment, Figure 2 The video encoder 20 shown is used to encode the image 17 block by block, for example, to perform encoding and prediction for each block 203.

[0130] In one embodiment, Figure 2 The video encoder 20 shown can also be used to segment and / or encode images using slices (also called video slices), where images can be segmented or encoded using one or more slices (typically non-overlapping). Each slice may include one or more blocks (e.g., coding tree units, CTUs) or one or more groups of blocks (e.g., coded tiles in the H.265 / HEVC / VVC standard and bricks in the VVC standard).

[0131] In one embodiment, Figure 2The video encoder 20 shown can also be used to segment and / or encode an image using slice / encoding block groups (also known as video encoding block groups) and / or encoding blocks (also known as video encoding blocks), wherein the image can be segmented or encoded using one or more slice / encoding block groups (typically non-overlapping), each slice / encoding block group may include one or more blocks (e.g., CTUs) or one or more encoding blocks, wherein each encoding block may be rectangular or the like, and may include one or more complete or partial blocks (e.g., CTUs).

[0132] Residual calculation

[0133] The residual calculation unit 204 is used to calculate the residual block 205 based on the image block (or original block) 203 and the prediction block 265 in the following manner (the prediction block 265 is described in detail later): for example, the residual block 205 in the pixel domain is obtained by subtracting the pixel value of the prediction block 265 from the pixel value of the image block 203 pixel by pixel.

[0134] Transformation

[0135] The transformation processing unit 206 performs discrete cosine transform (DCT) or discrete sine transform (DST) on the pixel values ​​of the residual block 205 to obtain the transformation coefficients 207 in the transform domain. The transformation coefficients 207 can also be called transformation residual coefficients, representing the residual block 205 in the transform domain.

[0136] Transform processing unit 206 can be used to apply an integer approximation of DCT / DST, such as the transform specified for H.265 / HEVC. This integer approximation is typically scaled by a certain factor compared to the orthogonal DCT transform. To maintain the norm of the residual block after both the forward and inverse transforms, other scaling factors are used as part of the transform process. These scaling factors are typically selected based on certain constraints, such as powers of 2 used for shift operations, the bit depth of the transform coefficients, and a trade-off between accuracy and implementation cost. For example, a specific scaling factor can be specified on the encoder 20 side via inverse transform processing unit 212 (and on the decoder 30 side via, for example, inverse transform processing unit 312) for the inverse transform, and correspondingly, a corresponding scaling factor can be specified on the encoder 20 side via transform processing unit 206 for the forward transform.

[0137] In one embodiment, the video encoder 20 (correspondingly, the transform processing unit 206) can be used to output transform parameters such as the type of one or more transforms, for example, directly outputting them or outputting them after being encoded or compressed by the entropy encoding unit 270, for example, so that the video decoder 30 can receive and use the transform parameters for decoding.

[0138] Quantification

[0139] Quantization unit 208 is used to quantize the transform coefficients 207 by, for example, scalar quantization or vector quantization, to obtain quantized transform coefficients 209. Quantized transform coefficients 209 can also be called quantized residual coefficients 209.

[0140] The quantization process can reduce the bit depth associated with some or all of the transform coefficients 207. For example, n-bit transform coefficients can be rounded down to m-bit transform coefficients during quantization, where n is greater than m. The degree of quantization can be modified by adjusting the quantization parameter (QP). For example, for scalar quantization, different scales can be applied to achieve finer or coarser quantization. Smaller quantization steps correspond to finer quantization, while larger quantization steps correspond to coarser quantization. The appropriate quantization step size can be indicated by the quantization parameter (QP). For example, the quantization parameter can be an index to a predefined set of appropriate quantization steps. For example, a smaller quantization parameter can correspond to fine quantization (smaller quantization step size), a larger quantization parameter can correspond to coarse quantization (larger quantization step size), and vice versa. Quantization may include division by the quantization step size, while corresponding or inverse dequantization performed by the dequantization unit 210, etc., may include multiplication by the quantization step size. Embodiments of some HEVC standards, for example, can be used to determine the quantization step size using the quantization parameter. In general, the quantization step size can be calculated using a fixed-point approximation of an equation involving division based on the quantization parameter. Additional scaling factors can be introduced for quantization and dequantization to recover the norm of the residual block, which may have been modified by the scaling used in the fixed-point approximation of the equations used for the quantization step size and quantization parameters. In one exemplary implementation, the scaling of the inverse transform and dequantization can be combined. Alternatively, a custom quantization table can be used and indicated from the encoder to the decoder in the bitstream, etc. Quantization is a lossy operation, where the loss increases with the quantization step size.

[0141] In one embodiment, the video encoder 20 (correspondingly, the quantization unit 208) can be used to output the quantization parameter (QP), for example, directly outputting it or outputting it after being encoded or compressed by the entropy encoding unit 270, for example, so that the video decoder 30 can receive it and use it for decoding.

[0142] Inverse Quantization

[0143] The dequantization unit 210 is used to perform dequantization on the quantization coefficients by the quantization unit 208 to obtain the dequantization coefficients 211. For example, it performs a dequantization scheme based on or using the same quantization step size as the quantization unit 208 to perform the quantization scheme performed by the quantization unit 208. The dequantization coefficients 211 can also be called dequantization residual coefficients 211, corresponding to the transform coefficients 207. However, due to the loss caused by quantization, the dequantization coefficients 211 are usually not exactly the same as the transform coefficients.

[0144] Inverse Transformation

[0145] The inverse transform processing unit 212 is used to perform the inverse transform of the transform performed by the transform processing unit 206, such as the inverse discrete cosine transform (DCT) or the inverse discrete sine transform (DST), to obtain the reconstructed residual block 213 (or the corresponding dequantization coefficients 213) in the pixel domain. The reconstructed residual block 213 may also be referred to as the transform block 213.

[0146] reconstruction

[0147] The reconstruction unit 214 (e.g., summer 214) is used to add the transform block 213 (i.e., the reconstruction residual block 213) to the prediction block 265 to obtain the reconstruction block 215 in the pixel domain, for example, by adding the pixel values ​​of the reconstruction residual block 213 and the pixel values ​​of the prediction block 265.

[0148] Filtering

[0149] Loop filter unit 220 (or simply "loop filter" 220) is used to filter the reconstructed block 215 to obtain the filtered block 221, or typically to filter the reconstructed pixels to obtain filtered pixel values. For example, the loop filter unit is used to smoothly perform pixel transformations or improve video quality. Loop filter unit 220 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. For example, loop filter unit 220 may include a deblocking filter, a SAO filter, and an ALF filter. The filtering process may be performed in the order of deblocking filter, SAO filter, and ALF filter. As another example, a process called luma mapping with chromascaling (LMCS) (i.e., an adaptive in-loop shaper) may be added. This process is performed before deblocking. For example, the deblocking filtering process can also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Although the loop filter unit 220 in... Figure 2 The loop filter is shown in the diagram, but in other configurations, the loop filter unit 220 can be implemented as a post-loop filter. The filter block 221 can also be called the filter reconstruction block 221.

[0150] In one embodiment, the video encoder 20 (correspondingly, the loop filter unit 220) can be used to output loop filter parameters (e.g., SAO filter parameters, ALF filter parameters, or LMCS parameters), for example, directly outputting or outputting after entropy encoding by the entropy encoding unit 270, for example, enabling the decoder 30 to receive and decode using the same or different loop filter parameters.

[0151] Decoding image buffer

[0152] The decoded picture buffer (DPB) 230 can be a reference picture memory that stores reference picture data for use by the video encoder 20 when encoding video data. The DPB 230 can be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The decoded picture buffer 230 can be used to store one or more filter blocks 221. The decoded picture buffer 230 can also be used to store other previous filter blocks of the same current picture or different pictures, such as previously reconstructed pictures, such as previously reconstructed and filtered blocks 221, and can provide complete previously reconstructed i.e., decoded pictures (and corresponding reference blocks and pixels) and / or partially reconstructed current pictures (and corresponding reference blocks and pixels), for example, for inter-frame prediction. The decoded image buffer 230 can also be used to store one or more unfiltered reconstruction blocks 215, or generally store unfiltered reconstruction pixels, such as reconstruction blocks 215 that have not been filtered by the loop filter unit 220, or reconstruction blocks or reconstruction pixels that have not undergone any other processing.

[0153] Pattern selection (segmentation and prediction)

[0154] The mode selection unit 260 includes a segmentation unit 262, an inter-frame prediction unit 244, and an intra-frame prediction unit 254, for selecting data from the decoded image buffer 230 or other buffers (e.g., column buffers). Figure 2 (Not shown) receives or obtains raw image data such as raw block 203 (current block 203 of the current image 17) and reconstructed image data, for example, filtered and / or unfiltered reconstructed pixels or reconstructed blocks of the same (current) image and / or one or more previously decoded images. The reconstructed image data is used as reference image data for prediction such as inter-frame prediction or intra-frame prediction to obtain prediction block 265 or prediction value 265.

[0155] The mode selection unit 260 can be used to determine or select a segmentation for the current block (including no segmentation) and the prediction mode (e.g., intra-frame or inter-frame prediction mode) to generate the corresponding prediction block 265 for calculating the residual block 205 and reconstructing the reconstructed block 215.

[0156] In one embodiment, the mode selection unit 260 can be used to select a segmentation and prediction mode (e.g., from prediction modes supported or available by the mode selection unit 260), which provides the best match or minimum residual (minimum residual means better compression in transmission or storage), or provides minimum signaling overhead (minimum signaling overhead means better compression in transmission or storage), or considers or balances both. The mode selection unit 260 can be used to determine the segmentation and prediction mode based on rate distortion optimization (RDO), i.e., selecting the prediction mode that provides minimum RDO optimization. The terms "best," "lowest," and "optimal" in this document do not necessarily refer to "best," "lowest," or "optimal" overall, but can also refer to situations that meet termination or selection criteria. For example, values ​​exceeding or falling below a threshold or other limitations may lead to a "suboptimal choice," but reduce complexity and processing time.

[0157] In other words, segmentation unit 262 can be used to segment images in a video sequence into a sequence of coding tree units (CTUs), CTUs 203 can be further segmented into smaller block portions or sub-blocks (forming blocks again), for example, by iteratively using quad-tree partitioning (QT), binary-tree partitioning (BT), or triple-tree partitioning (TT), or any combination thereof, and is used to perform prediction, for example, on each of the block portions or sub-blocks, wherein mode selection includes selecting the tree structure of the segmented block 203 and selecting the prediction mode applied to each of the block portions or sub-blocks.

[0158] The segmentation (e.g., performed by segmentation unit 262) and prediction processing (e.g., performed by inter-frame prediction unit 244 and intra-frame prediction unit 254) performed by video encoder 20 will be described in detail below.

[0159] segmentation

[0160] Segmentation unit 262 can divide (or divide) an image block (or CTU) 203 into smaller parts, such as small blocks of square or rectangular shape. For an image with a three-pixel array, a CTU consists of N×N luma pixel blocks and two corresponding chroma pixel blocks. The maximum allowed size of the luma blocks in a CTU is specified as 128×128 in the developing Versatile Video Coding (VVC) standard, but may be specified as a value different from 128×128 in the future, such as 256×256. Image CTUs can be grouped / set into slices / coded block groups, coded blocks, or bricks. A coded block covers a rectangular area of ​​an image, and a coded block can be divided into one or more bricks. A brick consists of multiple rows of CTUs within a coded block. A coded block that is not divided into multiple bricks can be called a brick. However, a brick is a true subset of a coded block and therefore is not called a coded block. VVC supports two coded block group modes: raster scan slice / coded block group mode and rectangular slice mode. In raster scan coded block group mode, a slice / coded block group contains a sequence of coded blocks in a raster scan of an image. In rectangular slice mode, a slice contains multiple bricks of an image, which together form a rectangular region of the image. The bricks within a rectangular slice are arranged in the raster scan order of the slice's bricks. These smaller blocks (also called sub-blocks) can be further subdivided into even smaller parts. This is also called tree segmentation or hierarchical tree segmentation, where the root block at root tree level 0 (level 0, depth 0), etc., can be recursively segmented into two or more blocks at the next lower tree level, such as nodes at tree level 1 (level 1, depth 1). These blocks can then be further segmented into two or more blocks at the next lower level, such as tree level 2 (level 2, depth 2), etc., until the segmentation ends (because the termination criteria are met, such as reaching the maximum tree depth or minimum block size). Blocks that are not further segmented are also called leaf blocks or leaf nodes of the tree. A tree divided into two parts is called a binary tree (BT), a tree divided into three parts is called a ternary tree (TT), and a tree divided into four parts is called a quad tree (QT).

[0161] For example, a coding unit (CTU) can be or include a CTB of luma pixels, two corresponding CTBs of chroma pixels in an image with a three-pixel array, a CTB of pixels in a monochrome image, or a CTB of pixels in an image encoded using three independent color planes and a syntax structure (for encoding pixels). Correspondingly, a coding block (CTB) can be an N×N pixel block, where N can be set to a value such that the components are divided into CTBs; this is called segmentation. Similarly, a coding unit (CU) can be or include a coding block of luma pixels, two corresponding coding blocks of chroma pixels in an image with a three-pixel array, a coding block of pixels in a monochrome image, or a coding block of pixels in an image encoded using three independent color planes and a syntax structure (for encoding pixels). Correspondingly, a coding block (CB) can be an M×N pixel block, where M and N can be set to a value such that the CTB is divided into coding blocks; this is called segmentation.

[0162] For example, in one embodiment, HEVC can divide a Coding Tree Unit (CTU) into multiple Coding Units (CUs) using a quadtree structure represented as a coding tree. At the leaf CU level, a decision is made as to whether to use inter-frame (temporal) prediction or intra-frame (spatial) prediction to encode image regions. Each leaf CU can be further divided into one, two, or four PUs based on the PU partitioning type. The same prediction process is used within a PU, and relevant information is transmitted to the decoder on a PU-by-PU basis. After obtaining residual blocks by applying the prediction process according to the PU partitioning type, the leaf CU can be segmented into Transform Units (TUs) according to another quadtree structure similar to the coding tree used for CUs.

[0163] For example, in one embodiment, according to the latest video coding standard currently under development (called Universal Video Coding (VVC), a combined quadtree of nested multi-type trees (e.g., binary and ternary trees) is used to partition the segmented structure for dividing the coding tree unit. Within the coding tree structure of a coding tree unit, the CU can be a square or a rectangle. For example, a coding tree unit (CTU) is first partitioned by a quadtree structure. The leaf nodes of the quadtree are further partitioned by multi-type tree structures. There are four partitioning types for the multi-type tree structure: vertical binary tree partitioning (SPLIT_BT_VER), horizontal binary tree partitioning (SPLIT_BT_HOR), vertical ternary tree partitioning (SPLIT_BT_HOR), and vertical ternary tree partitioning (SPLIT_BT_VER). SPLIT_TT_VER and horizontal ternary tree partitioning (SPLIT_TT_HOR). Multi-type leaf nodes are called coding units (CUs). Unless the CU is too large for the maximum transform length, such segmentation is used for prediction and transform processing without any further partitioning. In most cases, this means that the CU, PU, ​​and TU have the same block size in a quadtree-nested multi-type tree coding block structure. This anomaly occurs when the maximum supported transform length is less than the width or height of the color component of the CU. VVC specifies a unique signaling mechanism for partitioning information in a quadtree-nested multi-type tree coding structure. In the signaling mechanism, coding tree units ( The CTU (Centralized Unit) is first used as the root of the quadtree and then divided by the quadtree structure. Each quadtree leaf node (when large enough) is then further divided into a multi-type tree structure. In the multi-type tree structure, a first flag (mtt_split_cu_flag) indicates whether a node should be further divided. When a node is further divided, a second flag (mtt_split_cu_vertical_flag) indicates the direction of the division, and a third flag (mtt_split_cu_binary_flag) indicates whether the division is a binary or ternary tree. The division is based on mtt_split_cu_vertical_flag. The values ​​of `ical_flag` and `mtt_split_cu_binary_flag` allow the decoder to deduce the multi-type tree partitioning mode (MttSplitMode) of the CU based on predefined rules or tables. It's important to note that for certain designs, such as a 64×64 luma block and a 32×32 chroma pipeline design in a VVC hardware decoder, TT partitioning is not allowed when the width or height of the luma block exceeds 64. Similarly, TT partitioning is not allowed when the width or height of the chroma block exceeds 32. The pipeline design divides the image into multiple virtual pipeline data units (VPDUs), each defined as a non-overlapping unit in the image. In the hardware decoder, consecutive VPDUs are processed simultaneously in multiple pipeline stages. In most pipeline stages, the VPDU size is roughly proportional to the buffer size, thus requiring small VPDU sizes.In most hardware decoders, the VPDU size can be set to the maximum transform block (TB) size. However, in VVC, the partitioning of the ternary tree (TT) and binary tree (BT) can increase the VPDU size.

[0164] Additionally, it should be noted that when a portion of a tree node block extends beyond the bottom or right edge of the image, the tree node block is forcibly divided until all pixels of each encoded CU are within the image boundary.

[0165] For example, the intra-sub-partitions (ISP) tool described above can divide the luminance intra-prediction block vertically or horizontally into two or four sub-parts based on the block size.

[0166] In one example, the mode selection unit 260 of the video encoder 20 can be used to perform any combination of the segmentation techniques described above.

[0167] As described above, the video encoder 20 is used to determine or select the best or optimal prediction mode from a (predetermined) set of prediction modes. The set of prediction modes may include, for example, intra-frame prediction modes and / or inter-frame prediction modes.

[0168] Intra-frame prediction

[0169] The intra-prediction mode set can include 35 different intra-prediction modes, such as non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in HEVC, or it can include 67 different intra-prediction modes, such as non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in VVC. For example, several conventional angular intra-prediction modes are adaptively replaced with wide-angle intra-prediction modes for non-square blocks as defined in VVC. As another example, to avoid division operations in DC prediction, only the longer side is used to calculate the average value of non-square blocks. Furthermore, the intra-prediction results of planar mode can be modified using the position-dependent intraprediction combination (PDPC) method.

[0170] Intra-prediction unit 254 is used to generate intra-prediction block 265 by reconstructing pixels from adjacent blocks of the same current image according to the intra-prediction modes in the intra-prediction mode set.

[0171] Intra-prediction unit 254 (or typically mode selection unit 260) is also used to output intra-prediction parameters (or typically information indicating the selected intra-prediction mode of the block) to entropy coding unit 270 in the form of syntax element 266 to be included in encoded image data 21, so that video decoder 30 can perform operations such as receiving and using the prediction parameters for decoding.

[0172] HEVC's intra-frame prediction modes include DC prediction mode, planar prediction mode, and 33 angle prediction modes, totaling 35 candidate prediction modes. The current block can use pixels from the reconstructed image blocks to its left and above as references for intra-frame prediction. The image blocks in the surrounding region used for intra-frame prediction of the current block are called reference blocks, and the pixels in the reference blocks are called reference pixels. Among the 35 candidate prediction modes, DC prediction mode is suitable for regions with flat textures in the current block, where all pixels use the average value of the reference pixels in the reference block as the prediction; planar prediction mode is suitable for image blocks with smoothly varying textures, where the current block uses bilinear interpolation of the reference pixels in the reference block to predict all pixels in the current block; angle prediction mode utilizes the high correlation between the texture of the current block and the textures of adjacent reconstructed image blocks, copying the values ​​of the reference pixels in the corresponding reference block along a certain angle as the prediction for all pixels in the current block.

[0173] The HEVC encoder selects an optimal intra-prediction mode from 35 candidate prediction modes for the current block and writes this optimal intra-prediction mode into the video stream. To improve the coding efficiency of intra-prediction, the encoder / decoder derives three most probable modes from the optimal intra-prediction modes of the reconstructed image blocks in the surrounding regions that use intra-prediction. If the optimal intra-prediction mode selected for the current block is one of these three most probable modes, a first index is encoded to indicate that the selected optimal intra-prediction mode is one of these three most probable modes; if the selected optimal intra-prediction mode is not one of these three most probable modes, a second index is encoded to indicate that the selected optimal intra-prediction mode is one of the other 32 modes (other modes from the 35 candidate prediction modes besides the aforementioned three most probable modes). The HEVC standard uses a 5-bit fixed-length code as the aforementioned second index.

[0174] The HEVC encoder derives the three most likely modes by: selecting the best intra-prediction modes of the left and top adjacent image blocks of the current block and adding them to the set; if these two best intra-prediction modes are the same, only one needs to be kept in the set. If these two best intra-prediction modes are the same and both are angle prediction modes, then two angle prediction modes adjacent to that angle direction are selected and added to the set; otherwise, the planar prediction mode, DC mode, and vertical prediction mode are selected and added to the set in sequence until the number of modes in the set reaches three.

[0175] After performing entropy decoding on the bitstream, the HEVC decoder obtains the mode information of the current block. This mode information includes an indicator that indicates whether the optimal intra-prediction mode of the current block is among the three most likely modes, as well as the index of the optimal intra-prediction mode of the current block among the three most likely modes or the index of the optimal intra-prediction mode of the current block among the other 32 modes.

[0176] Inter-frame prediction

[0177] In a possible implementation, the set of inter-frame prediction modes depends on the available reference image (i.e., at least part of the previously decoded image stored in the DBP 230 as described above) and other inter-frame prediction parameters, such as whether to use the entire reference image or only a part of the reference image, such as a search window region near the current block, to search for the best matching reference block, and / or, for example, whether to perform pixel interpolation of half-pixel, quarter-pixel, and / or 1 / 16th interpolation.

[0178] In addition to the prediction modes mentioned above, skip mode and / or direct mode can also be used.

[0179] For example, in extended merge prediction, the merge candidate list for this mode consists of five candidate types in sequence: spatial MVP from spatially adjacent CUs, temporal MVP from co-located CUs, history-based MVP from the FIFO table, pairwise average MVP, and zero MV. Decoder-side motion vector refinement (DMVR) based on bilateral matching can be used to increase the accuracy of the merge mode's MV. Mergemode with MVD (MMVD) comes from merge modes with motion vector differences. The MMVD flag is sent immediately after the skip flag and merge flag to specify whether the CU uses the MMVD mode. The CU-level adaptive motion vector resolution (AMVR) scheme can be used. AMVR supports encoding the CU's MVD with different precisions. The MVD of the current CU is adaptively selected based on the current CU's prediction mode. When the CU is encoding in merge mode, the combined inter / intraprediction (CIIP) mode can be applied to the current CU. CIIP prediction is obtained by weighted averaging of inter-frame and intra-frame prediction signals. For affine motion compensation prediction, the affine motion field of the block is described by motion information from motion vectors of 2 control points (4 parameters) or 3 control points (6 parameters). Subblock-based temporal motion vector prediction (SbTMVP) is similar to temporal motion vector prediction (TMVP) in HEVC, but predicts the motion vectors of sub-CUs within the current CU. Bidirectional optical flow (BDOF), formerly known as BIO, is a simplified version that reduces computation, particularly in terms of the number of multiplications and the size of the multipliers. In the triangular partitioning mode, the CU is uniformly divided into two triangular parts using both diagonal and anti-diagonal partitioning. Furthermore, the bidirectional prediction mode extends the simple averaging to support weighted averaging of the two prediction signals.

[0180] Inter-frame prediction unit 244 may include a motion estimation (ME) unit and a motion compensation (MC) unit (both in... Figure 2(Not shown in the image). The motion estimation unit can be used to receive or acquire image block 203 (current image block 203 of current image 17) and decoded image 231, or at least one or more previously reconstructed blocks, such as one or more other / different previously decoded image blocks 231, to perform motion estimation. For example, the video sequence may include the current image and the previously decoded image 231, or in other words, the current image and the previously decoded image 231 may be part of or form the image sequence that forms the video sequence.

[0181] For example, encoder 20 can be used to select a reference block from multiple reference blocks of the same or different images in multiple other images, and provide the offset (spatial offset) between the position (x, y coordinates) of the reference image (or reference image index) and / or the reference block and the position of the current block as an inter-frame prediction parameter to the motion estimation unit. This offset is also called a motion vector (MV).

[0182] The motion compensation unit is used to acquire, for example, receive, inter-frame prediction parameters, and perform inter-frame prediction based on or using these parameters to obtain inter-frame prediction blocks 246. Motion compensation performed by the motion compensation unit may include extracting or generating prediction blocks based on motion / block vectors determined by motion estimation, and may also include performing interpolation with sub-pixel precision. Interpolation filtering can generate pixels of other pixels from pixels of known pixels, thereby potentially increasing the number of candidate prediction blocks available for encoding image blocks. Once the motion vector corresponding to the PU of the current image block is received, the motion compensation unit can locate the prediction block pointed to by the motion vector in one of the reference image lists.

[0183] The motion compensation unit can also generate syntax elements associated with blocks and video slices for use by the video decoder 30 when decoding image blocks of the video slices. Alternatively, or as an alternative to slices and corresponding syntax elements, coded block groups and / or coded blocks and their corresponding syntax elements can be generated or used.

[0184] In the process of obtaining the candidate motion vector list in the advanced motion vector prediction (AMVP) mode, the motion vectors (MVs) that can be added to the candidate motion vector list as candidates include the MVs of spatially and temporally adjacent image blocks of the current block. The MVs of spatially adjacent image blocks can further include the MVs of the candidate image blocks to the left of the current block and the MVs of the candidate image blocks above the current block. For an example, please refer to... Figure 4 , Figure 4 An exemplary schematic diagram of a candidate image block provided in an embodiment of this application, such as... Figure 4As shown, the set of candidate image blocks to the left includes {A0, A1}, the set of candidate image blocks above includes {B0, B1, B2}, and the set of temporally adjacent candidate image blocks includes {C, T}. All three sets can be added to the candidate motion vector list as candidates. However, according to existing coding standards, the maximum length of the candidate motion vector list for AMVP is 2. Therefore, it is necessary to determine the MV (Motion Vector Model) to be added to the candidate motion vector list from the three sets according to a prescribed order. This order can be: first, consider the set of candidate image blocks to the left of the current block {A0, A1} (consider A0 first, then A1 if A0 is unavailable); second, consider the set of candidate image blocks above the current block {B0, B1, B2} (consider B0 first, then B1 if B0 is unavailable, then B2 if B1 is unavailable); finally, consider the set of temporally adjacent candidate image blocks {C, T} of the current block (consider T first, then C if T is unavailable).

[0185] After obtaining the above list of candidate motion vectors, the optimal motion vector prediction (MV) is determined from the list using the rate distortion cost (RDcost). The candidate motion vector with the lowest RDcost is then used as the motion vector predictor (MVP) for the current block. The rate distortion cost is calculated using the following formula:

[0186] J = SAD + λR

[0187] Where J represents the RD cost, SAD is the sum of absolute differences (SAD) between the pixel values ​​of the predicted block obtained after motion estimation using candidate motion vectors and the pixel values ​​of the current block, R represents the bit rate, and λ represents the Lagrange multiplier.

[0188] The encoder passes the index of the determined MVP in the candidate motion vector list to the decoder. Further, a motion search can be performed within the neighborhood centered on the MVP to obtain the actual motion vector of the current block. The encoder calculates the motion vector difference (MVD) between the MVP and the actual motion vector and also passes the MVD to the decoder. The decoder parses the index, finds the corresponding MVP in the candidate motion vector list based on the index, parses the MVD, and adds the MVD to the MVP to obtain the actual motion vector of the current block.

[0189] During the process of obtaining the candidate motion information list in the merging mode, the motion information that can be added to the candidate motion information list includes the motion information of spatially adjacent or temporally adjacent image blocks of the current block. The spatially adjacent and temporally adjacent image blocks can be referenced... Figure 4 The candidate motion information list corresponds to spatially adjacent blocks (A0, A1, B0, B1, and B2) in the spatial domain. If a spatially adjacent block is unavailable or is intra-frame prediction, its motion information is not added to the candidate motion information list. The temporal candidate motion information for the current block is obtained by scaling the MV of the corresponding block in the reference frame based on the picture order count (POC) of the reference frame and the current frame. First, it is determined whether the block at position T in the reference frame is available; if not, the block at position C is selected. After obtaining the above candidate motion information list, the optimal motion information is determined from the candidate motion information list using the RD cost as the motion information for the current block. The encoder transmits the index value (denoted as mergeindex) of the optimal motion information in the candidate motion information list to the decoder.

[0190] Entropy coding

[0191] Entropy coding unit 270 is used to apply entropy coding algorithms or schemes (e.g., variable length coding (VLC), context adaptive VLC (CALVC), arithmetic coding schemes, binarization algorithms, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques) to quantization residual coefficients 209, inter-frame prediction parameters, intra-frame prediction parameters, loop filter parameters, and / or other syntax elements, to obtain encoded image data 21 that can be output as an encoded bitstream 21 through output terminal 272, so that video decoder 30 and the like can receive and use the parameters for decoding. The encoded bitstream 21 can be transmitted to video decoder 30, or stored in memory for later transmission or retrieval by video decoder 30.

[0192] Other architectural variations of the video encoder 20 can be used to encode the video stream. For example, a non-transform-based encoder 20 can directly quantize the residual signal in certain blocks or frames without the transform processing unit 206. In another implementation, the encoder 20 may have a quantization unit 208 and an inverse quantization unit 210 combined into a single unit.

[0193] Decoder and Decoding Method

[0194] like Figure 3 As shown, the video decoder 30 is used to receive, for example, encoded image data 21 (e.g., encoded bitstream 21) encoded by the encoder 20, to obtain a decoded image 331. The encoded image data or bitstream includes information for decoding the aforementioned encoded image data, such as data representing image blocks (and / or groups or blocks of encoded video segments) and associated syntax elements.

[0195] exist Figure 3 In the example, decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., a summer 314), a loop filter 320, a decoded image buffer (DBP) 330, a mode application unit 360, an inter-frame prediction unit 344, and an intra-frame prediction unit 354. The inter-frame prediction unit 344 may be or include a motion compensation unit. In some examples, video decoder 30 may perform substantially the same functions as the referenced unit. Figure 2 The video encoder 100 describes the encoding process as the opposite of the decoding process.

[0196] As described above with encoder 20, the inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded image buffer DPB 230, inter-frame prediction unit 344, and intra-frame prediction unit 354 also constitute the "built-in decoder" of video encoder 20. Correspondingly, inverse quantization unit 310 can be functionally identical to inverse quantization unit 110, inverse transform processing unit 312 can be functionally identical to inverse transform processing unit 122, reconstruction unit 314 can be functionally identical to reconstruction unit 214, loop filter 320 can be functionally identical to loop filter 220, and decoded image buffer 330 can be functionally identical to decoded image buffer 230. Therefore, the explanation of the corresponding units and functions of video encoder 20 is correspondingly applicable to the corresponding units and functions of video decoder 30.

[0197] Entropy Decoding

[0198] Entropy decoding unit 304 is used to parse bitstream 21 (or generally encoded image data 21) and perform entropy decoding on encoded image data 21 to obtain quantization coefficients 309 and / or decoded encoded parameters. Figure 3(Not shown in the image) Examples of parameters include inter-frame prediction parameters (e.g., reference image index and motion vector), intra-frame prediction parameters (e.g., intra-frame prediction mode or index), transform parameters, quantization parameters, loop filter parameters, and / or other syntax elements. The entropy decoding unit 304 can be used to apply the decoding algorithm or scheme corresponding to the encoding scheme of the entropy coding unit 270 of the encoder 20. The entropy decoding unit 304 can also be used to provide inter-frame prediction parameters, intra-frame prediction parameters, and / or other syntax elements to the mode application unit 360, and to provide other parameters to other units of the decoder 30. The video decoder 30 can receive syntax elements at the video slice and / or video block level. Furthermore, or as an alternative to slices and corresponding syntax elements, it can receive or use coded block groups and / or coded blocks and corresponding syntax elements.

[0199] Inverse Quantization

[0200] The dequantization unit 310 can be used to receive quantization parameters (QP) (or generally information related to dequantization) and quantization coefficients from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304), and dequantize the decoded quantization coefficients 309 based on the aforementioned quantization parameters to obtain dequantization coefficients 311, which may also be referred to as transform coefficients 311. The dequantization process may include using the quantization parameters calculated by the video encoder 20 for each video block in the video slice to determine the degree of quantization, and also to determine the degree of dequantization to be performed.

[0201] Inverse Transformation

[0202] The inverse transform processing unit 312 can be used to receive the dequantized coefficients 311, also known as transform coefficients 311, and apply a transform to the dequantized coefficients 311 to obtain the reconstructed residual block 213 in the pixel domain. The reconstructed residual block 213 can also be called transform block 313. The transform can be an inverse transform, such as inverse DCT, inverse DST, inverse integer transform, or a conceptually similar inverse transform process. The inverse transform processing unit 312 can also be used to receive transform parameters or corresponding information from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304) to determine the transform applied to the dequantized coefficients 311.

[0203] reconstruction

[0204] The reconstruction unit 314 (e.g., summer 314) is used to add the reconstruction residual block 313 to the prediction block 365 to obtain the reconstruction block 315 in the pixel domain, for example, by adding the pixel values ​​of the reconstruction residual block 313 and the pixel values ​​of the prediction block 365.

[0205] Filtering

[0206] Loop filter unit 320 (in or after the encoding loop) is used to filter the reconstructed block 315 to obtain filtered block 321, thereby facilitating pixel transformation or improving video quality. Loop filter unit 320 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. For example, loop filter unit 320 may include a deblocking filter, a SAO filter, and an ALF filter. The filtering process may be performed in the order of deblocking filter, SAO filter, and ALF filter. As another example, a process called luma mapping with chromascaling (LMCS) (i.e., an adaptive in-loop shaper) may be added. This process is performed before deblocking. For example, the deblocking filtering process can also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Although the loop filter unit 320 in... Figure 3 The loop filter is shown in the diagram, but in other configurations, the loop filter unit 320 can be implemented as a post-loop filter.

[0207] Decoding image buffer

[0208] The decoded video block 321 in one image is then stored in the decoded image buffer 330, which stores the decoded image 331 as a reference image. The reference image is used for subsequent motion compensation for other images and / or output displays respectively.

[0209] The decoder 30 is used to output the decoded image 311 through the output terminal 312, etc., for display to the user or for the user to view.

[0210] predict

[0211] Inter-frame prediction unit 344 is functionally identical to inter-frame prediction unit 244 (especially motion compensation unit), and intra-frame prediction unit 354 is functionally identical to inter-frame prediction unit 254. It determines segmentation or partitioning and performs prediction based on segmentation and / or prediction parameters or corresponding information received from coded image data 21 (e.g., parsed and / or decoded by entropy decoding unit 304). Pattern application unit 360 can be used to perform prediction (intra-frame or inter-frame prediction) for each block based on the reconstructed image, block, or corresponding pixel (filtered or unfiltered), resulting in prediction block 365.

[0212] When a video slice is encoded as an intra-coded (I) slice, the intra-prediction unit 354 in the mode application unit 360 generates a prediction block 365 for the current video slice based on the indicated intra-prediction mode and data from the previous decoded block of the current image. When a video image is encoded as an inter-coded (i.e., B or P) slice, the inter-prediction unit 344 (e.g., a motion compensation unit) in the mode application unit 360 generates a prediction block 365 for the current video slice based on motion vectors and other syntax elements received from the entropy decoding unit 304. For inter-prediction, these prediction blocks can be generated from one of the reference images in one of the reference image lists. The video decoder 30 can construct reference frame lists 0 and 1 using a default construction technique based on the reference images stored in the DPB 330. In addition to slices (e.g., video slices) or as a substitute for slices, the same or similar processes can be applied to embodiments of coding block groups (e.g., video coding block groups) and / or coding blocks (e.g., video coding blocks), such as video can be encoded using I, P, or B coding block groups and / or coding blocks.

[0213] The pattern application unit 360 is used to determine prediction information for video blocks in the current video slice by parsing motion vectors and other syntax elements, and to generate prediction blocks for the current video slice being decoded using the prediction information. For example, the pattern application unit 360 uses some received syntax elements to determine the prediction mode (e.g., intra-frame prediction or inter-frame prediction), inter-frame prediction slice type (e.g., B-slice, P-slice, or GPB-slice), construction information for one or more reference image lists for the slice, motion vectors for each inter-frame coded video block in the slice, inter-frame prediction state for each inter-frame coded video block in the slice, and other information to decode video blocks within the current video slice. In addition to slices (e.g., video slices) or as alternatives to slices, the same or similar process can be applied to embodiments of coding block groups (e.g., video coding block groups) and / or coding blocks (e.g., video coding blocks), for example, videos can be encoded using I, P, or B coding block groups and / or coding blocks.

[0214] In one embodiment, Figure 3The video encoder 30 can also be used to segment and / or decode images using slices (also called video slices), where images can be segmented or decoded using one or more slices (typically non-overlapping). Each slice may include one or more blocks (e.g., CTUs) or one or more groups of blocks (e.g., coded blocks in the H.265 / HEVC / VVC standard and bricks in the VVC standard).

[0215] In one embodiment, Figure 3 The video decoder 30 shown can also be used to segment and / or decode an image using slice / coded block groups (also known as video coded block groups) and / or coded blocks (also known as video coded blocks), wherein the image can be segmented or decoded using one or more slice / coded block groups (typically non-overlapping), each slice / coded block group may include one or more blocks (e.g., CTUs) or one or more coded blocks, wherein each coded block may be rectangular or the like, and may include one or more complete or partial blocks (e.g., CTUs).

[0216] Other variations of the video decoder 30 can be used to decode the encoded image data 21. For example, the decoder 30 can generate an output video stream without the loop filter unit 320. For example, the non-transform-based decoder 30 can directly dequantize the residual signal in certain blocks or frames without the inverse transform processing unit 312. In another implementation, the video decoder 30 may have a dequantization unit 310 and an inverse transform processing unit 312 combined into a single unit.

[0217] It should be understood that in encoder 20 and decoder 30, the processing result of the current step can be further processed and then output to the next step. For example, after interpolation filtering, motion vector derivation, or loop filtering, further operations can be performed on the processing result of interpolation filtering, motion vector derivation, or loop filtering, such as clipping or shifting operations.

[0218] It should be noted that further calculations can be performed on the derived motion vector of the current block (including but not limited to control point motion vectors in affine mode, affine, planar, sub-block motion vectors in ATMVP mode, time motion vectors, etc.). For example, the value of the motion vector can be restricted to a predefined range based on the representation bits of the motion vector. If the representation bit of the motion vector is bitDepth, the range is -2^(bitDepth-1) to 2^(bitDepth-1)-1, where "^" represents exponentiation. For example, if bitDepth is set to 16, the range is -32768 to 32767; if bitDepth is set to 18, the range is -131072 to 131071. For example, the value of the derived motion vector (e.g., the MV of four 4×4 sub-blocks in an 8×8 block) is restricted such that the maximum difference between the integer parts of the MV of the four 4×4 sub-blocks does not exceed N pixels, for example, not more than 1 pixel. Two methods for restricting the motion vector based on bitDepth are provided here.

[0219] Although the above embodiments primarily describe video encoding and decoding, it should be noted that embodiments of the decoding system 10, encoder 20, and decoder 30, as well as other embodiments described herein, can also be used for still image processing or encoding and decoding, i.e., the processing or encoding and decoding of a single image independent of any previous or consecutive images in video encoding and decoding. Generally, if image processing is limited to a single image 17, the inter-frame prediction unit 244 (encoder) and inter-frame prediction unit 344 (decoder) may not be available. All other functions (also referred to as tools or techniques) of the video encoder 20 and video decoder 30 can also be used for still image processing, such as residual calculation 204 / 304, transform 206, quantization 208, inverse quantization 210 / 310, (inverse) transform 212 / 312, segmentation 262 / 362, intra-frame prediction 254 / 354 and / or loop filtering 220 / 320, entropy coding 270, and entropy decoding 304.

[0220] Please refer to Figure 5 , Figure 5 This is an exemplary block diagram of a video decoding device 500 provided for embodiments of this application. The video decoding device 500 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 500 may be a decoder, such as... Figure 1a The video decoder 30 in the text can also be an encoder, for example... Figure 1a The video encoder 20 in the middle.

[0221] The video decoding device 500 includes: an input port 510 (or input port 510) and a receiver unit (Rx) 520 for receiving data; a processor, logic unit, or central processing unit (CPU) 530 for processing data; for example, the processor 530 may be a neural network processor 530; a transmitter unit (Tx) 540 and an output port 550 (or output port 550) for transmitting data; and a memory 560 for storing data. The video decoding device 500 may also include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the input port 510, receiver unit 520, transmitter unit 540, and output port 550 for the entry or exit of optical or electrical signals.

[0222] Processor 530 is implemented in both hardware and software. Processor 530 may be implemented as one or more processor chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. Processor 530 communicates with ingress port 510, receiver unit 520, transmitter unit 540, egress port 550, and memory 560. Processor 530 includes a decoding module 570 (e.g., a neural network-based decoding module 570). Decoding module 570 implements the embodiments disclosed above. For example, decoding module 570 performs, processes, prepares, or provides various encoding operations. Therefore, decoding module 570 provides a substantial improvement to the functionality of video decoding device 500 and affects the switching of video decoding device 500 to different states. Alternatively, decoding module 570 may be implemented with instructions stored in memory 560 and executed by processor 530.

[0223] Memory 560 includes one or more disks, tape drives, and solid-state drives, which can be used as overflow data storage devices to store such programs when an executable program is selected, and to store instructions and data read during program execution. Memory 560 can be volatile and / or non-volatile, and can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0224] Please refer to Figure 6 , Figure 6This is an exemplary block diagram of the device 600 provided in the embodiments of this application. The device 600 can be used as... Figure 1a The source device 12 and the destination device 14 are either or both of them.

[0225] The processor 602 in device 600 can be a central processing unit. Alternatively, processor 602 can be any other type of device or multiple devices, existing or to be developed in the future, capable of manipulating or processing information. While the disclosed implementation can be implemented using a single processor, such as processor 602 as shown, using more than one processor is faster and more efficient.

[0226] In one implementation, the memory 604 in device 600 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as memory 604. Memory 604 may include code and data 606 accessed by processor 602 via bus 612. Memory 604 may also include an operating system 608 and an application program 610, which includes at least one program that allows processor 602 to perform the methods described herein. For example, application program 610 may include applications 1 to N, and also includes a video decoding application that performs the methods described herein.

[0227] The device 600 may also include one or more output devices, such as a display 618. In one example, the display 618 may be a touch-sensitive display that combines a display with a touch-sensitive element that can be used to sense touch input. The display 618 may be coupled to the processor 602 via a bus 612.

[0228] Although bus 612 in device 600 is described herein as a single bus, bus 612 may include multiple buses. Furthermore, auxiliary storage may be directly coupled to other components of device 600 or accessed via a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Therefore, device 600 can have a wide variety of configurations.

[0229] Before describing the embodiments of this application, the terms and concepts used in the embodiments of this application will be explained first.

[0230] Reconstructed image: This refers to the lossy image obtained after decoding an encoded image bitstream, which can be displayed and played. It exists simultaneously at both the encoding and decoding ends, and the reconstructed images at both ends are identical. At the encoding end, the reconstructed image can be used as a reference image for inter-frame predictive coding of the image to be encoded. At the decoding end, the reconstructed image can be used as a reference image for inter-frame predictive decoding of the image bitstream to be decoded.

[0231] RAP: Random Access Point in video coding. A video bitstream encoded with at least one RAP is included. When random access occurs, the video bitstream before the RAP can be skipped, and decoding of the subsequent image bitstream can begin from the RAP, playing the reconstructed video image from the decoded output. From a video playback perspective, random access can be understood as dragging the player's progress bar to randomly jump between video streams (or local playback); or, when packet loss and a black screen occur during live streaming, the client will access the video stream at the next RAP to continue playback. RAPs are typically Intra-coded Picture (I) frames. I-frames only use information within the current frame for predictive coding to eliminate spatial information redundancy, without relying on other frames.

[0232] RAS: A RAS consists of all images from one RAP to the next. Images in a RAS are either forward predictive (P) frames or bi-directional interpolated prediction (B) frames. P or B frames require temporal prediction using the already encoded images to eliminate temporal redundancy.

[0233] For example, please refer to Figure 7 , Figure 7 This is a schematic diagram of a video sequence provided in an embodiment of this application. Figure 7 Eleven video frames, a1 to a11, are shown. A2, a6, and a10 are I-frames and form the RAP; all video frames between a2 and a6 are P-frames and form RAP S1; and all video frames between a6 and a10 are P-frames and form RAP S2. Figure 7 As can be seen, each P-frame needs to be encoded by referring to the previous m video frames, where m is an integer greater than 1. For example, a3 can be encoded by referring to a2, and a4 can be encoded by referring to a2 and / or a3.

[0234] CRR encoding: When encoding a video frame in a RAS, video frames in other RASs can be referenced for encoding.

[0235] In most videos (such as movies or TV shows), many picture scenes appear alternately, and the picture scenes displayed at different times may be similar. The interval duration between two adjacent similar picture scenes is usually long. Therefore, two adjacent similar picture scenes are in the same RAP but with a long interval, or the temporal correlation spans the RAP. For example, in video standards, the length of the RAS is generally specified as 1 s or 2 s, and the interval duration between two adjacent similar picture scenes is usually close to the length of the RAS (e.g., 0.7 s) or greater than the length of the RAS (e.g., 3 s). In addition, in video scenes such as security monitoring, similar picture scenes usually appear for a long time. Therefore, there are also cases where two similar picture scenes are in the same RAP but with a long interval, or the temporal correlation spans the RAP.

[0236] Keystream: In cryptography, a keystream is a stream of characters containing random or pseudo-random characters, which is used to combine with the plaintext data stream to generate encrypted messages (ciphertext).

[0237] Network Abstraction Layer Unit (NALU): In video coding, all encoded data is encapsulated into individual NALUs for transmission. They all have a unified structure. Each NALU contains two parts: the NALU header and the NALU payload (RBSP). The length of the NALU Header is fixed at two bytes, reflecting the content payload of the NALU; the length of the RBSP is an integer number of bytes, carrying the bitstream segment after video coding. Whether it is the SPS, PPS, or video coding data, it is encapsulated in the form of NALUs.

[0238] Picture Parameter Set and Sequence Parameter Set (PPS and SPS): They contain all the parameters required for encoding the entire sequence or the entire picture. In principle, in addition to the key parameters describing the characteristics of the encoded sequence, the SPS should also carry the control switches of most encoding tools and the corresponding encoding limit parameters. The scope of the parameters in the SPS is the entire encoded sequence, which means that during the entire video coding process, there is only one SPS and the parameters therein are not allowed to change; for the PPS, in principle, some tool parameters are allowed to change at the frame level but not frequently, so the picture-level tool control parameters can be further repeated and sent at a lower PPS, which means that the parameters in the PPS are allowed to change according to the changes in the picture characteristics. Usually, the scope of the PPS is n pictures (0 < n < all pictures).

[0239] The SPS includes a set of global parameters for the video coding sequence, such as the color space, the dimensions (width and height) of each frame, and the coding level. The PPS includes coding parameters for at least one frame, such as the quantization matrix and filtering parameters. The image header includes coding parameters for a single frame, such as the frame number.

[0240] For example, for SPS, there can be one SPS before every N CRR slice bitstreams, or one SPS before the entire bitstream. For PPS, there can be one PPS before every multiple bitstreams. For image headers, there can be an image header before the first CRR slice bitstream, and a slice header or segment header before each of the other non-first CRR slice bitstreams.

[0241] This application provides an encoding / decoding method that can improve the security of CRR media content transmission.

[0242] Please refer to Figure 8 , Figure 8 This is a flowchart illustrating an encoding method provided in an embodiment of this application. The method can be applied to an encoder and may include the following processes:

[0243] S801, Encode multiple display images and multiple CRR images to obtain a bitstream.

[0244] The aforementioned bitstream includes a security parameter set and an authentication dataset. The security parameter set includes first information and second information. The first information is used to represent the algorithm used to generate the tree top digest, and the second information is used to represent the algorithm used to perform the digital signature.

[0245] In one possible implementation, the aforementioned authentication dataset also includes third information, which is used to indicate whether the authentication data belongs to a CRR slice.

[0246] For example, the third piece of information can be is_crr_flag. is_crr_flag indicates whether the current authentication dataset belongs to the CRR image. is_crr_flag = 0 indicates that the current authentication data belongs to the main bitstream; is_crr_flag = 1 indicates that the current authentication data belongs to the CRR slice, constraining all CRR slices to be authenticated.

[0247] In one possible implementation, the authentication dataset also includes a fourth piece of information, which indicates the number of authenticated image streams.

[0248] For example, the fourth piece of information can be authentication_data_length_minus1+1, which represents the length of the signature data in bytes, and the value should be 0 to 255.

[0249] In one possible implementation, the aforementioned authentication dataset also includes fifth information, which is used to indicate the CRR image corresponding to the authentication scope of the specified current authentication dataset.

[0250] For example, the fifth piece of information can be `crr_picture_index`. `crr_picture_index` indicates the CRR image corresponding to the authentication range of the current authentication dataset, that is, which CRR image the authentication range of the current authentication dataset belongs to. This parameter can only be parsed when `is_crr_flag` is 1.

[0251] For example, as shown in Table 1, the security parameter set provided in this application embodiment may include the following information.

[0252] Table 1

[0253]

[0254]

[0255] Here, encryption_flag equal to 1 indicates that encryption is supported for image coded slices, sequence parameter sets, image parameter sets, or extended data units, meaning that RBSPs in NAL units may be encrypted. encryption_flag equal to 0 indicates that encryption of RBSPs in NAL units is not supported.

[0256] An `authentication_flag` of 1 indicates support for authentication of the entire frame's image data. The NAL units used for authentication include coded slices, as well as the sequence parameter set, image parameter set, security parameter set, and extended data units transmitted in that frame. When authentication of the above data content is supported, the encoded bitstream must carry absolute time extension information, and the authentication data carried in the encoded bitstream should be Base64 encoded. Authentication data is transmitted through NAL units with `nal_unit_type` equal to 10. If the image contains coded slices, sequence parameter sets, image parameter sets, security parameter sets, extended data units, etc., with `authentication_idc` equal to 1, the NAL units with `authentication_idc` equal to 1 in an image are arranged in decoding order and then authenticated to generate the image's digest data. An `authentication_flag` of 0 indicates that image authentication is not supported, and the encoded bitstream should not contain NAL units with `nal_unit_type` equal to 10.

[0257] If spatial_svc_flag equals 1, the base layer image and enhancement layer image of the same frame are authenticated separately. The image parameter set and coded slice of the enhancement layer are authenticated according to the enhancement layer. The sequence parameter set, security parameter set, and extended data unit are authenticated together with the image parameter set and coded slice of the base layer according to the base layer.

[0258] A value of 1 for vek_flag indicates that the video encryption key (VEK) is carried; a value of 0 indicates that the video encryption key (VEK) is not carried.

[0259] An iv_flag value of 1 indicates that an initialization vector (IV) is carried; an iv_flag value of 0 indicates that an initialization vector (IV) is not carried.

[0260] camera_id is a 20-byte string representing the camera ID from which the image originated.

[0261] vek_encryption_type indicates the algorithm used for key encryption, and the specific correspondence is the same as encryption_type.

[0262] evek_length_minus1 is the length of the encrypted key minus 1, in bytes.

[0263] evek is the encrypted key used for encryption calculations, and its length is evek_length_minus1 plus 1 byte.

[0264] vkek_version length_minus1 is the length of the encryption key version number minus 1, in bytes.

[0265] vkek_version is the encryption key version number, with a length of vkek_version_length_minus1 plus 1 byte.

[0266] iv_length_minus1 is the initial vector length minus 1, in bytes.

[0267] iv is the initialization vector used for block encryption, with a length of iv_length_minus1 plus 1 byte.

[0268] The encryption_type indicates the encryption algorithm used.

[0269] For example, as shown in Table 2, when encryption_type is 0, it indicates that the encryption algorithm used is SM1. When encryption_type is 1, it indicates that the encryption algorithm used is SM4.

[0270] Table 2

[0271] encryption_type Encryption Algorithm 0 SM1 1 SM4 2～15 reserve

[0272] hash_type indicates the algorithm used to generate the treetop digest.

[0273] For example, as shown in Table 3, when hash_type is 0, it indicates that the algorithm used to generate the tree top digest is the SM3 algorithm.

[0274] Table 3

[0275]

[0276] The signature_type indicates the algorithm used to digitally sign the image's summary data.

[0277] For example, as shown in Table 4, when the signature_type value is 0, it indicates that the algorithm used to digitally sign the image summary data is the SM2 algorithm.

[0278] Table 4

[0279] signature_type Signature Algorithm 0 SM2 1～3 reserve

[0280] `hash_discard_p_pictures` equal to 1 indicates that no authentication is performed on non-IDR images; equal to 0 indicates that authentication is performed on non-IDR images. If `hash_discard_p_pictures` is not in the bitstream, its default value is 1. The authentication_idc value for each NAL unit in an unauthenticated image should be equal to 0.

[0281] Increasing `successive_hash_pictures_minus1` by 1 indicates the number of consecutive images digitally signed in decoding order, and these consecutive images are limited to a single IDR image interval. The value of `successive_hash_pictures_minus1` should be between 0 and 255.

[0282] If `successive_hash_pictures_minus1` is greater than 0, first, tree-structured digest data is generated from the digest data of the consecutive `SuccessiveHashPictures` images in decoding order, and then a digital signature is applied to the top-of-tree digest data. For example, the top-of-tree digest data of `n` images is generated by arranging the top-of-tree digest data of the first `n-1` images and the digest data of the `n`th image, according to the method shown by `hash_type`.

[0283] camera_idc is a 19-byte string used to represent the certificate identifier of the camera from which the image originates.

[0284] For example, as shown in Table 5, the authentication dataset provided in this application embodiment may include the following information.

[0285] Table 5

[0286]

[0287]

[0288] Here, is_crr_flag indicates whether the current authentication dataset belongs to a CRR image. is_crr_flag = 0 indicates that the current authentication data belongs to the main bitstream; is_crr_flag = 1 indicates that the current authentication data belongs to a CRR slice, constraining all CRR slices to be authenticated.

[0289] `crr_picture_index` specifies which CRR image the authentication range of the current authentication dataset belongs to. This parameter can only be resolved when `is_crr_flag = 1`.

[0290] `frame_num` represents the image that should contain the authentication data. This image is the nearest `frame_num` before the authentication data NAL unit that has the same `frame_num` as the authentication data. When `successive_hash_pictures_minus1` is equal to 0, `frame_num` indicates the image corresponding to the authentication data; when it is greater than 0, `frame_num` indicates the last of `SuccessiveHashPictures` consecutive images.

[0291] A spatial_el_flag value of 1 indicates that the authentication data is an enhancement layer signature, while a spatial_el_flag value of 0 indicates that the authentication data is a base layer signature. If spatial_el_flag does not exist in the bitstream, its default value is 0.

[0292] The value of authentication_data_length_minus1 plus 1 indicates the length of the signature data in bytes, and should be between 0 and 255.

[0293] authentication_data[i] is the i-th byte of a signature data.

[0294] For example, please refer to Figure 9 , Figure 9This is a schematic diagram of an encoding and decoding process provided in an embodiment of this application. The encoder encodes multiple display images and multiple CRR images to obtain a bitstream and sends the bitstream to the decoder. Figure 9 The following example illustrates the process using three CRR slice bitstreams b1 to b3 (n=3) and five main bitstreams c1 to c5. The decoder decodes each bitstream in the encoded image bitstream sequentially, resulting in five reconstructed display images c1′ to c5′ and a reconstructed CRR image. The five reconstructed display images are used for display, while the reconstructed CRR image serves only as a reference image when decoding the display image bitstream and is not used for display.

[0295] S802. Authenticate the data to be authenticated based on the first information to obtain the first treetop summary data.

[0296] The first treetop digest data mentioned above refers to the treetop digest data at the encoding end. The first treetop digest data includes the treetop digest data of N main bit streams and / or M CRR slice bit streams, where N and M are positive integers.

[0297] For example, if the first information, i.e., hash_type, is 0, the SM3 algorithm can be used to authenticate the data to be authenticated (NALU data of N main bitstreams and / or M CRR slice bitstreams) to obtain the first treetop digest data. The NALU data includes the NALU header and the NALU's RBSP.

[0298] S803. Perform signature calculation on the first treetop summary data based on the second information to obtain a digital signature.

[0299] For example, if the second information, signature_type, is 0, the SM2 algorithm can be used to perform signature calculation on the first treetop digest data to obtain a digital signature (i.e., video authentication data).

[0300] S804. Encapsulate the digital signature into the authentication dataset.

[0301] In one possible implementation, the data signature described above can be encoded. The encoded data signature is then encapsulated into the authentication dataset described above.

[0302] For example, the data signature can be Base64 encoded, and the Base64 encoded data signature can be encapsulated into the authentication dataset as an independent NALU type. The scope of this authentication dataset is constrained to all relevant data of the most recent CRR image (SPS, PPS, and CRR slice stream of the CRR image), or all relevant data of the N preceding display images (SPS, PPS, and corresponding display image streams of the display images).

[0303] Please refer to Figure 10 , Figure 10 The following is a flowchart illustrating a decoding method provided in an embodiment of this application. This method can be applied to a decoder and may include the following processes:

[0304] S1001, Obtain the bitstream.

[0305] The aforementioned bitstream is obtained by encoding multiple display images and multiple CRR images. The bitstream includes a security parameter set and an authentication dataset. The security parameter set includes first information and second information. The first information is used to represent the algorithm used to generate the treetop digest, and the second information is used to represent the algorithm used to perform the digital signature. The authentication dataset includes digital signatures.

[0306] In one possible implementation, the aforementioned authentication dataset also includes third information, which is used to indicate whether the authentication data belongs to a CRR slice.

[0307] In one possible implementation, the authentication dataset also includes a fourth piece of information, which indicates the number of authenticated image streams.

[0308] S1002. Decode the digital signature based on the second information to obtain the first treetop digest data.

[0309] The first treetop digest data mentioned above refers to the treetop digest data at the encoding end. The first treetop digest data includes the treetop digest data of N main bit streams and / or M CRR slice bit streams, where N and M are positive integers.

[0310] In one possible implementation, the digital signature can be decoded; the decoded digital signature can be further decoded based on the second information to obtain the first treetop digest data.

[0311] S1003. Authenticate the authentication data corresponding to the authentication dataset according to the first information to obtain the second treetop summary data.

[0312] The second treetop digest data is the treetop digest data of the decoding end, and the first treetop digest data includes the treetop digest data of N main bit streams and / or M CRR slice bit streams.

[0313] S1004. Verify the authentication data based on the first treetop extract and the second treetop extract data.

[0314] For example, if the first treetop summary and the second treetop summary data are consistent, it can be determined that the authentication data is complete and authentic (i.e., the authentication data has not been tampered with).

[0315] For example, if the first treetop summary and the second treetop summary data are inconsistent, it can be determined that the authentication data is incomplete or untrue (i.e., the authentication data may have been tampered with).

[0316] Please refer to Figure 11 , Figure 11 This is a flowchart illustrating an encoding method provided in an embodiment of this application. The method can be applied to an encoder and may include the following processes:

[0317] S1101. Encode multiple display images and multiple CRR images to obtain a bitstream.

[0318] The aforementioned bitstream includes a set of security parameters, which in turn includes a video encryption key.

[0319] S1102. Encrypt the bitstream according to the video encryption key.

[0320] In one possible implementation, the payload data of the bitstream can be encrypted based on the video encryption key.

[0321] The payload data mentioned above includes at least one of the following: sequence parameter set SPS, image parameter set PPS, main bitstream, and CRR slice bitstream.

[0322] For example, the encoder cycle can randomly generate a 16-byte video encryption key (VEK). Then, the payload data to be encrypted is read from the bitstream. The encoder randomly generates a 16-byte initialization vector (IV). The currently used video encryption key (VEK) and initialization vector (IV) are encrypted using the source-end video key encryption key (VKEK), and the corresponding ciphertext is encapsulated in a NALU of the security parameter set. Using the agreed-upon output-feedback (OFB) mode of a block cipher algorithm, a keystream is generated using the video encryption key (VEK) and initialization vector (IV). The stream key is bit-aligned with the RBSP data to be encrypted, and then XORed to obtain the encrypted RBSP data. The encrypted RBSP data is encapsulated in a VCL_NALU, and the encryption_idc of this NALU is set to 1. If a new video encryption key (VEK) is activated or a new initialization vector (IV) is used, the security extension information (NAL) should be re-encoded and encapsulated before the currently encrypted output VCL_NAL.

[0323] Optionally, if this RBSP is the first coded slice data at the beginning of a group of pictures (GOP) and a new video encryption key (VEK) is available, the current video encryption key (VEK) can be invalidated and a new video encryption key (VEK) can be activated; otherwise, the current video encryption key (VEK) can continue to be used.

[0324] Please refer to Figure 12 , Figure 12 The following is a flowchart illustrating a decoding method provided in an embodiment of this application. This method can be applied to a decoder and may include the following processes:

[0325] S1201, Obtain the bitstream.

[0326] The aforementioned bitstream is obtained by encoding multiple display images and multiple CRR images. The bitstream includes a security parameter set, which includes a video encryption key.

[0327] S1202. Decrypt the bitstream according to the video encryption key.

[0328] In one possible implementation, the payload data of the aforementioned bitstream can be decrypted using the aforementioned video encryption key.

[0329] The aforementioned payload data includes at least one of SPS, PPS, main bit stream, and CRR slice bit stream.

[0330] For example, the ciphertext of the initialization vector (IV) and the video encryption key (VEK) can be obtained from the security parameter set NAL, denoted as E(VEK). E(VEK) is then decrypted using the video key encryption key (VKEK) to obtain the video encryption key (VEK). The RBSP data to be decrypted is then read. Using the agreed-upon OFB block cipher algorithm, a keystream is generated using the video encryption key (VEK) and the initialization vector (IV). The stream key is bit-aligned with the RBSP data to be decrypted and XORed to obtain the decrypted RBSP data.

[0331] Optionally, if this RBSP is the first coded slice data at the beginning of a group of pictures (GOP) and a new video encryption key (VEK) is available, the current video encryption key (VEK) can be invalidated and a new video encryption key (VEK) can be activated; otherwise, the current video encryption key (VEK) can continue to be used.

[0332] S1203. Determine the reconstructed image based on the decoded bitstream.

[0333] The following will combine Figures 13-16 An encoding / decoding device for performing the above encoding / decoding method is introduced.

[0334] It is understood that, in order to achieve the above-mentioned functions, the encoding device includes hardware and / or software modules corresponding to the execution of each function. Based on the algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in a hardware-driven or software-driven manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0335] This application embodiment can divide the encoding device into functional modules according to the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0336] When dividing each function into modules according to its corresponding function. Figure 13 A schematic diagram of a possible composition of the encoding device involved in the above embodiments is shown, such as... Figure 13 As shown, the encoding device 1300 may include an encoding unit 1301 and an authentication unit 1302.

[0337] Encoding unit 1301 is used to encode multiple display images and the aforementioned multiple CRR images to obtain a bitstream. The bitstream includes a security parameter set and an authentication dataset. The security parameter set includes first information and second information. The first information is used to represent the algorithm used to generate the tree top digest, and the second information is used to represent the algorithm used to perform the digital signature.

[0338] The authentication unit 1302 is used to authenticate the data to be authenticated according to the first information to obtain the first treetop digest data. The first treetop digest data is the treetop digest data of the encoding end. The first treetop digest data includes the treetop digest data of N main bit streams and / or M CRR slice bit streams, where N and M are positive integers.

[0339] The authentication unit 1302 is further configured to perform a signature calculation on the first treetop digest data based on the second information to obtain a digital signature. The authentication unit is also configured to encapsulate the digital signature into the authentication dataset.

[0340] In one possible implementation, the aforementioned authentication dataset also includes third information, which is used to indicate whether the authentication data belongs to a CRR slice.

[0341] For example, the third piece of information can be is_crr_flag. is_crr_flag indicates whether the current authentication dataset belongs to the CRR image. is_crr_flag = 0 indicates that the current authentication data belongs to the main bitstream; is_crr_flag = 1 indicates that the current authentication data belongs to the CRR slice, constraining all CRR slices to be authenticated.

[0342] In one possible implementation, the authentication dataset also includes a fourth piece of information, which indicates the number of authenticated image streams.

[0343] For example, the fourth piece of information can be authentication_data_length_minus1+1, which represents the length of the signature data in bytes, and the value should be 0 to 255.

[0344] In one possible implementation, the aforementioned authentication dataset also includes fifth information, which is used to indicate the CRR image corresponding to the authentication scope of the specified current authentication dataset.

[0345] For example, the fifth piece of information can be `crr_picture_index`. `crr_picture_index` indicates the CRR image corresponding to the authentication range of the current authentication dataset, that is, which CRR image the authentication range of the current authentication dataset belongs to. This parameter can only be parsed when `is_crr_flag` is 1.

[0346] In one possible implementation, the authentication unit 1302 is specifically used to: encode the above-mentioned data signature; and encapsulate the encoded data signature into the above-mentioned authentication dataset.

[0347] In one possible implementation, the encoding unit 1301 is specifically used to: segment the CRR image to obtain multiple CRR slices; interleave the multiple CRR slices between the multiple display images to obtain a sequence of images to be encoded; and sequentially encode the CRR slices and the display images in the sequence of images to be encoded to obtain the bitstream.

[0348] In one possible implementation, the bitstream includes multiple CRR slice bitstreams and multiple main bitstreams that are interleaved and ordered.

[0349] This application embodiment can divide the decoding device into functional modules according to the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0350] When dividing each function into modules according to its corresponding function. Figure 14 A schematic diagram of a possible composition of the decoding device involved in the above embodiments is shown, such as... Figure 14 As shown, the decoding device 1400 may include a transceiver unit 1401 and an authentication unit 1402.

[0351] The transceiver unit 1401 is used to acquire a bitstream, which includes a security parameter set and an authentication dataset. The security parameter set includes first information and second information. The first information is used to represent the algorithm used to generate the tree top digest, and the second information is used to represent the algorithm used to perform the digital signature. The authentication dataset includes digital signatures.

[0352] The authentication unit 1402 is used to decode the digital signature according to the second information to obtain the first treetop digest data. The first treetop digest data is the treetop digest data of the encoding end. The first treetop digest data includes the treetop digest data of N main bit streams and / or M CRR slice bit streams, where N and M are positive integers.

[0353] The authentication unit 1402 is further configured to authenticate the authentication data corresponding to the authentication dataset according to the first information to obtain the second treetop digest data. The second treetop digest data is the treetop digest data of the decoding end. The first treetop digest data includes the treetop digest data of N main bit streams and / or M CRR slice bit streams.

[0354] The authentication unit 1402 is also used to verify the authentication data based on the first treetop summary and the second treetop summary data.

[0355] In one possible implementation, the authentication unit 1402 is specifically used to: decode the digital signature; and decode the decoded digital signature according to the second information to obtain the first treetop digest data.

[0356] In one possible implementation, the aforementioned authentication dataset also includes third information, which is used to indicate whether the authentication data belongs to a CRR slice.

[0357] In one possible implementation, the authentication dataset also includes a fourth piece of information, which indicates the number of authenticated image streams.

[0358] When dividing each function into modules according to its corresponding function. Figure 15 A schematic diagram of another possible composition of the encoding device involved in the above embodiments is shown, such as... Figure 15 As shown, the encoding device 1500 may include an encoding unit 1501 and an encryption unit 1502.

[0359] The encoding unit 1501 is used to encode multiple display images and the aforementioned multiple CRR images to obtain a bitstream, wherein the bitstream includes a security parameter set, and the security parameter set includes a video encryption key.

[0360] The encryption unit 1502 is used to encrypt the bitstream according to the video encryption key.

[0361] In one possible implementation, the encryption unit 1502 is specifically used to: encrypt the payload data of the bitstream according to the video encryption key, wherein the payload data includes at least one of the sequence parameter set SPS, the image parameter set PPS, the main bitstream, and the CRR slice bitstream.

[0362] When dividing each function into modules according to its corresponding function. Figure 16 A schematic diagram of another possible composition of the decoding device involved in the above embodiments is shown, such as... Figure 16 As shown, the decoding device 1600 may include a transceiver unit 1601, a decryption unit 1602, and a reconstruction unit 1603.

[0363] The transceiver unit 1601 is used to acquire a bitstream, wherein the bitstream includes a security parameter set, and the security parameter set includes a video encryption key.

[0364] The decryption unit 1602 is used to decrypt the bitstream according to the video encryption key.

[0365] The reconstruction unit 1603 is used to determine the reconstructed image based on the decoded bitstream.

[0366] In one possible implementation, the decryption unit 1602 is specifically used to: decrypt the payload data of the bitstream according to the video encryption key, wherein the payload data includes at least one of SPS, PPS, main bitstream and CRR slice bitstream.

[0367] This application also provides an encoding device, which includes at least one processor. When the at least one processor executes program code or instructions, it implements the above-described related method steps to achieve the encoding method in the above embodiments.

[0368] Optionally, the device may further include at least one memory for storing the program code or instructions.

[0369] This application also provides a decoding device, which includes at least one processor. When the at least one processor executes program code or instructions, it implements the above-described related method steps to achieve the decoding method in the above embodiments.

[0370] Optionally, the device may further include at least one memory for storing the program code or instructions.

[0371] This application also provides a computer storage medium storing computer instructions. When the computer instructions are executed on an encoding device, the encoding device performs the aforementioned related method steps to implement the encoding and decoding methods in the above embodiments.

[0372] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the encoding / decoding method described in the above embodiments.

[0373] This application also provides an encoding / decoding apparatus, which may specifically be a chip, integrated circuit, component, or module. Specifically, the apparatus may include a connected processor and a memory for storing instructions, or the apparatus may include at least one processor for fetching instructions from external memory. When the apparatus is running, the processor can execute instructions to cause the chip to perform the encoding / decoding methods in the above-described method embodiments.

[0374] Figure 17 A schematic diagram of a chip 1700 is shown. Chip 1700 includes one or more processors 1701 and interface circuitry 1702. Optionally, the chip 1700 may also include a bus 1703.

[0375] The processor 1701 is likely an integrated circuit chip with signal processing capabilities. In its implementation, the steps of the aforementioned encoding / decoding method can be completed through integrated logic circuits in the processor 1701 or through software instructions.

[0376] Optionally, the processor 1701 described above may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods and steps disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor.

[0377] The interface circuit 1702 can be used to send or receive data, instructions or information. The processor 1701 can use the data, instructions or other information received by the interface circuit 1702 to process the data, instructions or other information, and can send the processed information out through the interface circuit 1702.

[0378] Optionally, the chip may also include memory, which may include read-only memory and random access memory, providing operation instructions and data to the processor. A portion of the memory may also include non-volatile random access memory (NVRAM).

[0379] Optionally, the memory stores executable software modules or data structures, and the processor can execute corresponding operations by calling the operation instructions stored in the memory (which may be stored in the operating system).

[0380] Optionally, the chip can be used in the encoding device or DOP involved in the embodiments of this application. Optionally, the interface circuit 1702 can be used to output the execution result of the processor 1701. For the encoding and decoding methods provided in one or more embodiments of this application, please refer to the foregoing embodiments, which will not be repeated here.

[0381] It should be noted that the functions of processor 1701 and interface circuit 1702 can be implemented through hardware design, software design, or a combination of hardware and software; no restrictions are imposed here.

[0382] Figure 18 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 1800 can be a device or a chip or functional module within a device. Figure 18 As shown, the electronic device 1800 includes a processor 1801, a transceiver 1802, and a communication line 1803.

[0383] The processor 1801 is used to execute any step of the method provided in the embodiments of this application, and when performing processes such as receiving encoded image bit streams, it can selectively call the transceiver 1802 and the communication line 1803 to complete the corresponding operations.

[0384] Furthermore, the electronic device 1800 may also include a memory 1804. The processor 1801, the memory 1804, and the transceiver 1802 can be connected via a communication line 1803.

[0385] Transceiver 1802 is used to communicate with other devices or other communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. Transceiver 1802 can be a module, circuit, transceiver, or any device capable of enabling communication.

[0386] The transceiver 1802 is mainly used for transmitting and receiving bit streams, etc., and may include a transmitter and a receiver to send and receive bit streams, etc., respectively; operations other than transmitting and receiving bit streams, etc., are implemented by the processor, such as enabling at least one target channel in the channel group of the chip.

[0387] Communication line 1803 is used to transmit information between the various components included in electronic device 1800.

[0388] In one design, the processor can be viewed as a logic circuit, and the transceiver as an interface circuit.

[0389] Memory 1804 is used to store instructions. These instructions can be computer programs.

[0390] It should be noted that the memory 1804 can exist independently of the processor 1801, or it can be integrated with the processor 1801. The memory 1804 can be used to store instructions, program code, or some data, etc. The memory 1804 can be located inside or outside the electronic device 1800, without limitation. The processor 1801 is used to execute the instructions stored in the memory 1804 to implement the methods provided in the above embodiments of this application.

[0391] In one example, processor 1801 may include one or more processors, for example Figure 18 Processor 0 and processor 1 in the system.

[0392] As an optional implementation, the electronic device 1800 includes multiple processors, for example, besides Figure 18In addition to processor 1801, it may also include processor 1807.

[0393] As an optional implementation, the electronic device 1800 also includes an output device 1805 and an input device 1806. For example, the input device 1806 is a device such as a keyboard, mouse, microphone, or joystick, and the output device 1805 is a device such as a display screen or speaker.

[0394] It should be noted that the electronic device 1800 can be a chip system or... Figure 18 Devices with similar structures. The chip system can be composed of chips or include chips and other discrete components. Actions, terminology, etc., involved in the various embodiments of this application can be referenced interchangeably without limitation. The message names or parameter names in the messages used for interaction between devices in the embodiments of this application are merely examples; other names can be used in specific implementations without limitation. Furthermore, Figure 18 The structural composition shown does not constitute a limitation on the electronic device 1800, except... Figure 18 In addition to the components shown, the electronic device 1800 may include more than Figure 18 This may indicate more or fewer components, or combinations of certain components, or different component arrangements.

[0395] The processor and transceiver described in this application can be implemented on integrated circuits (ICs), analog ICs, radio frequency integrated circuits, mixed-signal ICs, application-specific integrated circuits (ASICs), printed circuit boards (PCBs), electronic devices, etc. The processor and transceiver can also be manufactured using various IC process technologies, such as complementary metal oxide semiconductors (CMOS), n-type metal-oxide-semiconductor (NMOS), p-type metal oxide semiconductors (PMOS), bipolar junction transistors (BJTs), bipolar CMOS (BiCMOS), silicon germanium (SiGe), gallium arsenide (GaAs), etc.

[0396] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to execute any of the methods described in the embodiments of this application.

[0397] This application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments can be executed by a computer or a device with encoding / decoding capabilities, using computer programs or instructions to control related hardware. The computer program or set of instructions can be stored in the aforementioned computer-readable storage medium. When executed, the computer program or set of instructions can include the processes described in the above method embodiments. The computer-readable storage medium can be an internal storage unit of the control terminal in any of the foregoing embodiments, such as a hard disk or memory of the control terminal. The aforementioned computer-readable storage medium can also be an external storage device of the control terminal, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the control terminal. Further, the aforementioned computer-readable storage medium can include both the internal storage unit of the control terminal and external storage devices. The aforementioned computer-readable storage medium is used to store the aforementioned computer program or instructions, as well as other programs and data required by the control terminal. The aforementioned computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0398] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0399] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0400] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0401] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0402] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0403] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0404] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An encoding method, characterized in that, The method includes: Encoding operations are performed on multiple display images and multiple reference images to obtain a bitstream. The bitstream includes a security parameter set and an authentication dataset. The security parameter set includes first information and second information. The first information is used to represent the algorithm used for authentication, and the second information is used to represent the algorithm used for digital signature. The data to be authenticated is authenticated based on the first information to obtain the first treetop digest data. The first treetop digest data is the treetop digest data of the encoding end. The first treetop digest data includes the treetop digest data of N main bit streams and / or M CRR slice bit streams, where N and M are positive integers. A digital signature is obtained by performing a signature calculation on the first treetop digest data based on the second information; The digital signature is encapsulated into the authentication dataset.

2. The method according to claim 1, characterized in that, The authentication dataset also includes third information, which indicates whether the authentication data belongs to a CRR slice.

3. The method according to claim 1 or 2, characterized in that, The authentication dataset also includes a fourth piece of information, which indicates the length of the signature data.

4. The method according to claim 1 or 2, characterized in that, The authentication dataset also includes fifth information, which is used to indicate the reference image corresponding to the authentication scope of the specified current authentication dataset.

5. The method according to claim 1 or 2, characterized in that, The bitstream includes multiple CRR slice bitstreams and multiple main bitstreams that are interleaved and sorted.

6. The method according to claim 1 or 2, characterized in that, The reference image includes a CRR image.

7. A decoding method, characterized in that, The method includes: A bitstream is obtained by encoding multiple display images and multiple reference images. The bitstream includes a security parameter set, authentication data, and a digital signature. The security parameter set includes first information and second information. The first information is used to indicate the algorithm used for authentication, and the second information is used to indicate the algorithm used for digital signature. The digital signature is decoded according to the second information to obtain first digest data, the first digest data including digest data of N display images and / or digest data of M knowledge images, where N and M are positive integers; The authentication data is authenticated based on the first information to obtain second digest data. The second digest data includes digest data of N display images and / or digest data of M knowledge images. The authentication data is verified based on the first digest data and the second digest data.

8. The method according to claim 7, characterized in that, The bitstream also includes third information, which is used to indicate whether the authentication data belongs to a knowledge image.

9. The method according to claim 7 or 8, characterized in that, The bitstream also includes fourth information, which is used to indicate the length of the signature data.

10. The method according to claim 7 or 8, characterized in that, The code stream also includes fifth information, which is used to indicate the knowledge image corresponding to the authentication scope of the authentication data.

11. The method according to claim 7 or 8, characterized in that, The bitstream includes multiple CRR slice bitstreams and multiple main bitstreams that are interleaved and sorted.

12. The method according to claim 7 or 8, characterized in that, The reference image includes a CRR image.

13. An encoding method, characterized in that, The method includes: Encoding operations are performed on multiple display images and multiple reference images to obtain a bitstream. The bitstream includes a security parameter set and an authentication dataset. The security parameter set includes first information and second information. The first information is used to represent the algorithm used for authentication, and the second information is used to represent the algorithm used for digital signature. The data to be authenticated is authenticated based on the first information to obtain the first treetop digest data. The first treetop digest data is the treetop digest data of the encoding end. The first treetop digest data includes the treetop digest data of N main bit streams and / or M CRR slice bit streams, where N and M are positive integers. A digital signature is obtained by performing a signature calculation on the first treetop digest data based on the second information; The digital signature is encapsulated into the authentication dataset; The security parameter set also includes a video encryption key; The bitstream is encrypted using the video encryption key.

14. A decoding method, characterized in that, The method includes: A bitstream is obtained by encoding multiple display images and multiple reference images. The bitstream includes a security parameter set, authentication data, and a digital signature. The security parameter set includes first information and second information. The first information is used to indicate the algorithm used for authentication, and the second information is used to indicate the algorithm used for digital signature. The digital signature is decoded according to the second information to obtain first digest data, the first digest data including digest data of N display images and / or digest data of M knowledge images, where N and M are positive integers; The authentication data is authenticated based on the first information to obtain second digest data, the second digest data including digest data of N display images and / or digest data of M knowledge images; The authentication data is verified based on the first digest data and the second digest data; The security parameter set also includes a video encryption key; The bitstream is decrypted using the video encryption key; The reconstructed image is determined based on the decoded bitstream.

15. An encoding device, characterized in that, Includes a module for performing the method as described in any one of claims 1 to 6.

16. A decoding device, characterized in that, Includes modules for performing the method as described in any one of claims 7 to 12.

17. An encoding device, characterized in that, The device includes a processor coupled to a memory for storing instructions that, when executed by the processor, cause the device to perform the method as described in any one of claims 1 to 6 or 13.

18. A decoding device, characterized in that, The device includes a processor coupled to a memory for storing instructions that, when executed by the processor, cause the device to perform the method as described in any one of claims 7 to 12 or 14.

19. A computer-readable storage medium for storing a computer program, characterized in that, When the computer program is run, the method of any one of claims 1 to 14 is performed.

20. A computer program product, the computer program product comprising instructions, characterized in that, When the instructions are executed, the method of any one of claims 1 to 14 is performed.

21. A chip comprising at least one processor and a memory, characterized in that, The at least one processor executes a program or instructions stored in a memory to cause the at least one processor to implement the method of any one of claims 1 to 14.

Images