Authentication method, encryption method and related device for TSVC technology

Abstract

The invention provides an authentication method, an encryption method and a related device for a TSVC technology. The authentication method for the TSVC technology comprises the steps that a first abstract corresponding to a first hierarchy is obtained, and the first hierarchy is any hierarchy in a plurality of hierarchy obtained through time stratification; the first abstract is obtained according to coding code streams of n1 images, and n1 is greater than or equal to 1; when the first level is a base layer, the n1 images come from the first level; when the first level is an enhancement layer, the n1 images come from the first level and at least one level below the first level; signing the first abstract to obtain a digital signature of the first level; and packaging the digital signature of the first level into an authentication data set of the first level. According to the invention, various complex and changeable network bandwidth changes can be effectively handled, and it is ensured that coding code streams of all time domain levels can be correctly decrypted and authenticated so as to ensure the transmission efficiency of images.

Description

Technical Field

[0001] The present application relates to video coding and decoding technology, and in particular to an authentication method, an encryption method and related devices for temporal scalable video coding (TSVC) technology. Background Art

[0002] With the rapid development of network video surveillance technology, data security issues have become increasingly prominent, threatening the security of the country, society, enterprises and families. How to ensure that video data will not be stolen or tampered with by others, and how to ensure the security of video transmission have become urgent issues to be solved. Summary of the invention

[0003] The present application provides an authentication method, encryption method and related devices for TSVC technology to effectively cope with various complex and changeable changes in network bandwidth, ensuring that the encoded code streams at each time domain level can be correctly decrypted and authenticated to ensure the transmission efficiency of the image.

[0004] In a first aspect, the present application provides an authentication method for TSVC technology, comprising: obtaining a first summary corresponding to a first level, the first level being any one of a plurality of levels obtained through time stratification; the first summary is obtained based on an encoded bitstream of n1 images, n1≥1; when the first level is a base layer, the n1 images are from the first level; when the first level is an enhancement layer, the n1 images are from the first level and at least one level below the first level; signing the first summary to obtain a digital signature of the first level; and encapsulating the digital signature of the first level into an authentication data set of the first level.

[0005] The embodiment of the present application integrates the TSVC technology and the GB35114 specification, and performs signature authentication on the TSVC time domain levels respectively, which can effectively cope with various complex and changeable network bandwidth changes to ensure that the encoded code streams at each time domain level can be correctly decrypted and authenticated.

[0006] The first level is any one of the multiple levels obtained through time stratification.

[0007] The first summary is obtained based on the encoded bitstreams of n1 images, where n1≥1; when the first layer is a base layer, the n1 images come from the first layer; when the first layer is an enhancement layer, the n1 images come from the first layer and at least one layer below the first layer.

[0008] The encoder may obtain a digest acquisition algorithm according to the security parameter set; and then process the encoded code streams of the n1 images according to the digest acquisition algorithm to obtain a first digest.

[0009] The encoder may obtain a signature algorithm according to the security parameter set; and then sign the first summary according to the signature algorithm to obtain a first-level digital signature.

[0010] The encoder can encode the first-level digital signature using 64 printable characters to represent binary data (Base64), and then encapsulate the Base64-encoded digital signature into the authentication data set, which can be encapsulated in the form of an independent NALU. Optionally, the NALU of the first-level authentication data set can be placed after the encrypted NALU to form a video stream for transmission.

[0011] The above process is the process of the encoder authenticating the first level. The digital signature obtained in the embodiment of the present application corresponds to the level, that is, if there are N levels, N authentications can be performed, and then N authentication data sets containing digital signatures can be obtained. Based on this, the identification of the first level can be carried in the authentication data set of the first level to distinguish multiple authentication data sets.

[0012] Exemplarily, additional syntax elements are added to the authentication data set in SVAC3 above, as shown below:

[0013]

[0014]

[0015] temporal_id is a 3-bit unsigned integer indicating the TSVC temporal level to which the authentication data set belongs. The value range of temporal_id can be 0-7.

[0016] In a second aspect, the present application provides an authentication method for TSVC technology, comprising: obtaining a first authentication data set, the first authentication data set including a first digital signature and a layer identifier, the first authentication data set coming from a received code stream; decrypting the first digital signature to obtain a first summary; obtaining a second summary based on the encoded code stream of n1 images, n1≥1; when the first layer corresponding to the layer identifier is a basic layer, the n1 images are from the first layer; when the first layer is an enhanced layer, the n1 images are from the first layer and at least one layer below the first layer; when the first summary and the second summary are the same, determining that the n1 images have passed authentication.

[0017] The embodiment of the present application integrates the TSVC technology and the GB35114 specification, and performs signature authentication on the TSVC time domain levels respectively, which can effectively cope with various complex and changeable network bandwidth changes to ensure that the encoded code streams at each time domain level can be correctly decrypted and authenticated.

[0018] Corresponding to the encoding end, the first authentication data set may include a first digital signature and a layer identifier, and the first authentication data set comes from the received code stream.

[0019] The decoder can obtain the signature algorithm according to the security parameter set; and then decrypt the first digital signature according to the signature algorithm to obtain the first summary.

[0020] The decoder can obtain the source device information from the NALU of the security parameter set (can be identified by camera_idc and camera_id), find the public key of the source device (this public key is different from the private key used for encryption of the source device mentioned above). According to the signature algorithm specified by signature_type in the NALU of the security parameter set (currently only SM2 can be used in SVAC2, which is an asymmetric encryption algorithm), the first digital signature in the NALU of the authentication data set is decrypted with the public key of the source device to obtain the first summary, which is the summary generated by the encoding end.

[0021] The method for the decoder to obtain the second summary can refer to Figure 7 In step 701 of the illustrated embodiment, the layer identifier in the first authentication data set indicates the layer currently being processed, and n1 images correspond to the aforementioned layers. When the first layer corresponding to the layer identifier is a basic layer, the n1 images come from the first layer; when the first layer is an enhanced layer, the n1 images come from the first layer and at least one layer below the first layer.

[0022] The decoder may obtain a digest acquisition algorithm according to the security parameter set; and then process the encoded code streams of the n1 images according to the digest acquisition algorithm to obtain a second digest.

[0023] Compare the first digest generated by the encoder and the second digest generated by the decoder using the same method. If the two are the same, it means that n1 images have passed authentication.

[0024] In a third aspect, the present application provides an authentication method for TSVC technology, comprising: obtaining a third summary of multiple levels, wherein the multiple levels are obtained through time stratification; the third summary is obtained based on the encoded bitstream of n2 images, n2≥1; the n2 images are from the multiple levels; signing the third summary to obtain digital signatures of the multiple levels; and encapsulating the digital signatures of the multiple levels into an authentication data set.

[0025] The embodiment of the present application integrates the TSVC technology and the GB35114 specification, and does not need to change the existing SVAC3 security parameter set, so that all TSVC time domain code streams can be more conveniently encrypted and authenticated uniformly.

[0026] Multiple levels are obtained through time stratification.

[0027] The third summary is obtained based on the encoded bitstreams of n2 images, n2≥1; the n2 images come from multiple levels.

[0028] The n2 images are all images from the three levels.

[0029] The encoder may obtain a digest acquisition algorithm according to the security parameter set; and then process the encoded code streams of the n2 images according to the digest acquisition algorithm to obtain a third digest.

[0030] The encoder may obtain a signature algorithm according to the security parameter set; and then sign the third digest according to the signature algorithm to obtain a multi-level digital signature.

[0031] The encoder can encode multiple levels of digital signatures using 64 printable characters to represent binary data (Base64), and then encapsulate the Base64-encoded digital signature into the authentication data set, which can be encapsulated in the form of an independent NALU. Optionally, the NALUs of multiple levels of authentication data sets can be placed after the encrypted NALU to form a video stream for transmission.

[0032] The embodiment of the present application does not distinguish between levels when obtaining the third summary. All levels correspond to only one summary, thereby obtaining one authentication data set. Therefore, there is no need for a level identifier to distinguish the authentication data set.

[0033] In a fourth aspect, the present application provides an authentication method for TSVC technology, comprising: obtaining an authentication data set, the authentication data set including a digital signature, the authentication data set coming from a received code stream; decrypting the digital signature to obtain a third summary; obtaining a fourth summary based on the encoded code stream of n2 images, n2 ≥ 1; the n2 images coming from multiple levels obtained through time stratification; when the third summary and the fourth summary are the same, determining that the n2 images have passed authentication.

[0034] The embodiment of the present application integrates the TSVC technology and the GB35114 specification, and does not need to change the existing SVAC3 security parameter set, so that all TSVC time domain code streams can be more conveniently encrypted and authenticated uniformly.

[0035] Corresponding to the encoding end, the authentication data set may include a digital signature, and the authentication data set comes from the received code stream.

[0036] The decoder can obtain the signature algorithm according to the security parameter set; and then decrypt the digital signature according to the signature algorithm to obtain the third summary.

[0037] The decoder can obtain the source device information from the NALU of the security parameter set (can be identified by camera_idc and camera_id), find the public key of the source device (this public key is different from the private key used for encryption of the source device mentioned above). According to the signature algorithm specified by signature_type in the NALU of the security parameter set (currently only SM2 can be used in SVAC2, which is an asymmetric encryption algorithm), the public key of the source device is used to decrypt the digital signature in the NALU of the authentication data set to obtain the third summary, which is the summary generated by the encoding end.

[0038] The decoder may obtain a digest acquisition algorithm according to the security parameter set; and then process the encoded code streams of the n2 images according to the digest acquisition algorithm to obtain a fourth digest.

[0039] Compare the third digest generated by the encoder and the fourth digest generated by the decoder using the same method. If the two are the same, it means that n2 images have passed authentication.

[0040] In a fifth aspect, the present application provides an encryption method for TSVC technology, including: obtaining a current network abstraction layer unit NALU, the original byte sequence payload RBSP of the current NALU including a sequence parameter set SPS, an image parameter set PPS, or multiple levels of video encoding data obtained through time stratification; obtaining a video encryption key VEK according to a security parameter set; encrypting the RBSP according to the VEK to obtain a ciphertext of the current NALU; and transmitting the ciphertext of the NALU.

[0041] The embodiment of the present application integrates the TSVC technology and the GB35114 specification to implement encryption processing of the time-domain layered RBSP of TSVC.

[0042] The object to be encrypted in the embodiment of the present application may be the RBSP of the NALU including the SPS, PPS or multiple levels of video coding data obtained through time stratification. The Header of the NALU does not need to be encrypted.

[0043] The encoder can periodically randomly generate a 16-byte VEK and a 16-byte IV.

[0044] The encoder uses the output feedback OFB mode of the agreed block encryption algorithm to generate a key stream using VEK and IV. The key stream is bit-aligned with the RBSP of the current NALU and an XOR operation is performed to obtain the encrypted RBSP.

[0045] Optionally, the encoder encrypts the above VEK and IV with the VKEK of the source end, and encapsulates the corresponding ciphertext into the NALU of the security parameter set.

[0046] In the sixth aspect, the present application provides a decryption method for TSVC technology, including: obtaining the ciphertext of the current network abstraction layer unit NALU; obtaining the video encryption key VEK; decrypting the ciphertext of the current NALU according to the VEK to obtain the original byte sequence payload RBSP of the current NALU; obtaining a reconstructed image according to the RBSP of the current NALU.

[0047] The embodiment of the present application integrates the TSVC technology and the GB35114 specification to implement the decryption processing of the time-domain layered RBSP of TSVC.

[0048] Corresponding to the encoding end, the RBSP of the NALU carried in the code stream received by the decoder is encrypted, so what is obtained is the ciphertext of the current NALU.

[0049] The decoder obtains the ciphertext of IV and VEK from the NALU of the security parameter set, denoted as EVEK, and decrypts EVEK with VKEK to obtain VEK and IV.

[0050] The decoder uses the output feedback OFB mode of the agreed block encryption algorithm and generates a key stream using VEK and IV. The key stream is XORed with the ciphertext of the current NALU bit by bit to obtain the decrypted RBSP.

[0051] The decompression process according to the RBSP may refer to the above embodiment, which will not be described in detail.

[0052] In the seventh aspect, the present application provides an authentication device for TSVC technology, including: an acquisition module, used to obtain a first summary corresponding to a first level, the first level is any one of multiple levels obtained through time stratification; the first summary is obtained based on the encoded code stream of n1 images, n1≥1; when the first level is a basic layer, the n1 images come from the first level; when the first level is an enhanced layer, the n1 images come from the first level and at least one level below the first level; a signature module, used to sign the first summary to obtain a digital signature of the first level; an encapsulation module, used to encapsulate the digital signature of the first level into an authentication data set of the first level.

[0053] In a possible implementation manner, the acquisition module is specifically configured to acquire a digest acquisition algorithm according to a security parameter set; and process the encoded code streams of the n1 images according to the digest acquisition algorithm to obtain the first digest.

[0054] In a possible implementation, the signature module is specifically configured to obtain a signature algorithm according to a security parameter set; and sign the first summary according to the signature algorithm to obtain a digital signature of the first level.

[0055] In a possible implementation manner, the first-level authentication data set also includes an identifier of the first level.

[0056] In an eighth aspect, the present application provides an authentication device for TSVC technology, comprising: an acquisition module, used to acquire a first authentication data set, the first authentication data set comprising a first digital signature and a layer identifier, the first authentication data set coming from a received code stream; a decryption module, used to decrypt the first digital signature to obtain a first summary; the acquisition module is also used to acquire a second summary based on the encoded code stream of n1 images, n1≥1; when the first layer corresponding to the layer identifier is a basic layer, the n1 images come from the first layer; when the first layer is an enhanced layer, the n1 images come from the first layer and at least one layer below the first layer; a verification module, used to determine that the n1 images have passed the authentication when the first summary and the second summary are the same.

[0057] In a possible implementation, the decryption module is specifically configured to obtain a signature algorithm according to a security parameter set; and decrypt the first digital signature according to the signature algorithm to obtain the first summary.

[0058] In a possible implementation manner, the acquisition module is specifically configured to acquire a digest acquisition algorithm according to a security parameter set; and process the encoded code streams of the n1 images according to the digest acquisition algorithm to obtain the second digest.

[0059] In a ninth aspect, the present application provides an authentication device for TSVC technology, comprising: an acquisition module, used to acquire a third digest of multiple levels, wherein the multiple levels are obtained through time stratification; the third digest is obtained based on the encoded code stream of n2 images, n2≥1; the n2 images are from the multiple levels; a signature module, used to sign the third digest to obtain digital signatures of the multiple levels; and an encapsulation module, used to encapsulate the digital signatures of the multiple levels into an authentication data set.

[0060] In a possible implementation, the acquisition module is specifically configured to acquire a digest acquisition algorithm according to a security parameter set; and process the encoded code streams of the n2 images according to the digest acquisition algorithm to obtain the third digest.

[0061] In a possible implementation, the signature module is specifically configured to obtain a signature algorithm according to a security parameter set; and sign the third digest according to the signature algorithm to obtain the digital signatures of the multiple levels.

[0062] In the tenth aspect, the present application provides an authentication device for TSVC technology, including: an acquisition module, used to acquire an authentication data set, the authentication data set including a digital signature, the authentication data set comes from a received code stream; a decryption module, used to decrypt the digital signature to obtain a third summary; the acquisition module is also used to acquire a fourth summary based on the encoded code stream of n2 images, n2≥1; the n2 images come from multiple levels obtained by time stratification; a verification module, used to determine that the n2 images have passed authentication when the third summary and the fourth summary are the same.

[0063] In a possible implementation, the decryption module is specifically configured to obtain a signature algorithm according to a security parameter set; and decrypt the digital signature according to the signature algorithm to obtain the third summary.

[0064] In a possible implementation manner, the acquisition module is specifically configured to acquire a digest acquisition algorithm according to a security parameter set; and process the encoded code streams of the n2 images according to the digest acquisition algorithm to obtain the fourth digest.

[0065] On the eleventh aspect, the present application provides an encryption device for TSVC technology, including: an acquisition module, used to obtain the current network abstraction layer unit NALU, the original byte sequence payload RBSP of the current NALU includes a sequence parameter set SPS, an image parameter set PPS or multiple levels of video encoding data obtained through time stratification; obtaining a video encryption key VEK according to a security parameter set; an encryption module, used to encrypt the RBSP according to the VEK to obtain the ciphertext of the current NALU; a transmission module, used to transmit the ciphertext of the NALU.

[0066] In a possible implementation, the encryption module is specifically configured to generate a key stream according to the VEK; and perform an XOR operation on the key stream and the RBSP to obtain the ciphertext.

[0067] In a possible implementation, the encryption module is further used to encrypt the VEK to obtain an encrypted VEK.

[0068] In the twelfth aspect, the present application provides a decryption device for TSVC technology, including: an acquisition module, used to obtain the ciphertext of the current network abstraction layer unit NALU; obtain the video encryption key VEK; a decryption module, used to decrypt the ciphertext of the current NALU according to the VEK to obtain the original byte sequence payload RBSP of the current NALU; a reconstruction module, used to obtain a reconstructed image according to the RBSP of the current NALU.

[0069] In a possible implementation, the decryption module is specifically configured to generate a key stream according to the VEK; and perform an XOR operation on the key stream and the ciphertext to obtain the RBSP.

[0070] In the thirteenth aspect, the present application provides an encoder, comprising: one or more processors; a non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein when the program is executed by the processor, the encoder performs a method according to any one of the first, third, and fifth aspects above.

[0071] In the fourteenth aspect, the present application provides a decoder comprising: one or more processors; a non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein when the program is executed by the processor, the decoder performs a method according to any one of the second, fourth, and sixth aspects above.

[0072] In a fifteenth aspect, the present application provides a non-transitory computer-readable storage medium, comprising a program code, which, when executed by a computer device, is used to execute a method according to any one of the first to sixth aspects above. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 is an exemplary block diagram of a decoding system 10 according to an embodiment of the present application;

[0074] Figure 2 is an exemplary block diagram of a video encoder 20 according to an embodiment of the present application;

[0075] Figure 3 is an exemplary block diagram of a video decoder 30 according to an embodiment of the present application;

[0076] Figure 4 is an exemplary block diagram of a video decoding device 400 according to an embodiment of the present application;

[0077] Figure 5a A schematic diagram of the SHVC encoding end architecture of an embodiment of the present application;

[0078] Figure 5b Schematic diagram of the SHVC decoding end architecture of an embodiment of the present application

[0079] Figure 5c Schematic diagram of the SHVC time layered architecture of an embodiment of the present application

[0080] Figure 5d Schematic diagram of the SHVC spatial hierarchical architecture of an embodiment of the present application

[0081] Figure 5eSchematic diagram of the SHVC quality hierarchical architecture of the embodiment of the present application

[0082] Figure 5f Schematic diagram of the code stream organization structure of SHVC encoding in an embodiment of the present application

[0083] Figure 6a A schematic diagram of a NALU structure according to an embodiment of the present application;

[0084] Figure 6b GB35114 encryption flow chart of the embodiment of this application

[0085] Figure 6c GB35114 decryption flow chart of the embodiment of this application

[0086] Figure 6d GB35114 signature flow chart for this application embodiment

[0087] Figure 6e A schematic diagram of a tree summary of an embodiment of the present application

[0088] Figure 7 A flowchart of a process 700 of an authentication method for TSVC technology according to an embodiment of the present application;

[0089] Figure 8 This is a TSVC authentication flow chart of an embodiment of the present application;

[0090] Fig. 9 A flowchart of a process 900 of an authentication method for TSVC technology according to an embodiment of the present application;

[0091] Fig.10 A flowchart of a process 1000 of an authentication method for TSVC technology according to an embodiment of the present application;

[0092] Fig.11 This is a TSVC authentication flow chart of an embodiment of the present application;

[0093] Fig.12 A flowchart of a process 1200 of an authentication method for TSVC technology according to an embodiment of the present application;

[0094] Fig.13 A flowchart of a process 1300 of an encryption method for TSVC technology according to an embodiment of the present application;

[0095] Fig.14 A flowchart of a process 1400 of a decryption method for TSVC technology according to an embodiment of the present application;

[0096] Fig.15 This is an encryption / decryption flow chart of an embodiment of the present application;

[0097] Fig.16 1600 is a schematic diagram of the structure of an encoding device according to an embodiment of the present application;

[0098] Fig.17 17 is a schematic diagram of the structure of a decoding device 1700 according to an embodiment of the present application. DETAILED DESCRIPTION

[0099] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0100] The terms "first", "second", etc. in the specification embodiments, claims, and drawings of the present application are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, including a series of steps or units. The method, system, product, or device is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0101] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0102] Video compression coding technology is widely used in multimedia services, broadcasting, video communication and storage. In recent years, the two major standard organizations, the International Telecommunication Union Telecommunication Standardization Sector (ITU-T) and the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC), jointly formulated and released three video coding standards in 2003, 2013 and 2020: Highly Compression Digital Video Codec Standard (H.264) / Advanced Video Coding (AVC), H.265 / High Efficiency Video Coding (HEVC) and H.266 / Versatile Video Coding (VVC). At the same time, the Audio Video Coding Standard (AVS) also formulated and output a series of video image coding standards such as AVS1, AVS2, and AVS3. In addition, the Alliance for Open Media (AOM) also released the AV1 video codec solution in 2018. The above video codec technology solutions all adopt a hybrid architecture based on block division and transform quantization, and continuously iterate the technology in specific modules such as block division, prediction, transformation, entropy coding, and loop filtering, so as to continuously improve the compression efficiency of video images.

[0103] Video coding generally refers to processing a sequence of images that form a video or video sequence. In the field of video coding, the terms "picture", "frame" or "image" can be used as synonyms. Video coding (or commonly referred to as coding) includes two parts: video encoding and video decoding. Video coding is performed on the source side and generally includes processing (e.g., compressing) the original video image to reduce the amount of data required to represent the video image (thereby more efficiently storing and / or transmitting). Video decoding is performed on the destination side and generally includes inverse processing relative to the encoder to reconstruct the video image. The "encoding" of the video image (or commonly referred to as the image) involved in the embodiment should be understood as the "encoding" or "decoding" of the video image or video sequence. The encoding part and the decoding part are also collectively referred to as codec (encoding and decoding, CODEC).

[0104] In the case of lossless video coding, the original video image can be reconstructed, that is, the reconstructed video image has the same quality as the original video image (assuming no transmission loss or other data loss during storage or transmission). In the case of lossy video coding, further compression is performed by quantization, etc. to reduce the amount of data required to represent the video image, but the decoder side cannot fully reconstruct the video image, that is, the quality of the reconstructed video image is lower or worse than the quality of the original video image.

[0105] Several video coding standards belong to the category of "lossy hybrid video codecs" (i.e., combining spatial and temporal prediction in the pixel domain with 2D transform coding in the transform domain for applying quantization). Each image in a video sequence is usually divided into a set of non-overlapping blocks, which are usually encoded at the block level. In other words, the encoder usually processes, i.e., encodes the video at the block (video block) level, for example, by generating a prediction block through spatial (intra-frame) prediction and temporal (inter-frame) prediction; subtracting the prediction block from the current block (currently processed / to be processed block) to obtain a residual block; transforming the residual block in the transform domain and quantizing the residual block to reduce the amount of data to be transmitted (compressed), while the decoder side applies the inverse processing part relative to the encoder to the encoded or compressed block to reconstruct the current block for representation. In addition, the encoder needs to repeat the processing steps of the decoder so that the encoder and the decoder generate the same predictions (e.g., intra-frame predictions and inter-frame predictions) and / or reconstructed pixels for processing, i.e., encoding subsequent blocks.

[0106] In the following embodiment of the decoding system 10, the encoder 20 and the decoder 30 are based on Figure 1-Figure 3 Give a description.

[0107] Figure 1 1 is an exemplary block diagram of a decoding system 10 of an embodiment of the present application, for example, a video decoding system 10 (or simply a decoding system 10) that can utilize the techniques of the present application. The video encoder 20 (or simply an encoder 20) and the video decoder 30 (or simply a decoder 30) in the video decoding system 10 represent devices that can be used to perform various techniques according to various examples described in the present application.

[0108] like Figure 1 As shown, the decoding system 10 includes a source device 12, which is used to provide encoded image data 21 such as encoded images to a destination device 14 for decoding the encoded image data 21.

[0109] The source device 12 includes an encoder 20 , and may additionally or optionally include an image source 16 , a preprocessor (or a preprocessing unit) 18 such as an image preprocessor, and a communication interface (or a communication unit) 22 .

[0110] The image source 16 may include or may be any type of image capture device for capturing real-world images, etc., and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage that stores any of the above images.

[0111] In order to distinguish the processing performed by the pre-processor (or pre-processing unit) 18 , the image (or image data) 17 may also be referred to as a raw image (or raw image data) 17 .

[0112] The preprocessor 18 is used to receive the original image data 17 and preprocess the original image data 17 to obtain a preprocessed image (or preprocessed image data) 19. For example, the preprocessing performed by the preprocessor 18 may include trimming, color format conversion (e.g., from RGB to YCbCr), color adjustment, or denoising. It is understood that the preprocessing unit 18 may be an optional component.

[0113] The video encoder (or encoder) 20 is used to receive the pre-processed image data 19 and provide the encoded image data 21 (hereinafter referred to as Figure 2 etc. for further description).

[0114] The communication interface 22 in the source device 12 can be used to receive the encoded image data 21 and send the encoded image data 21 (or any other processed version) to another device such as the destination device 14 or any other device through the communication channel 13 for storage or direct reconstruction.

[0115] The destination device 14 includes a decoder 30 and, in addition or optionally, may include a communication interface (or communication unit) 28 , a post-processor (or post-processing unit) 32 , and a display device 34 .

[0116] The communication interface 28 in the destination device 14 is used to receive the encoded image data 21 (or any other processed version) directly from the source device 12 or from any other source device such as a storage device, for example, the storage device is a encoded image data storage device, and provide the encoded image data 21 to the decoder 30.

[0117] The communication interface 22 and the communication interface 28 can be used to send or receive encoded image data (or encoded data) 21 through a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any type of combination thereof.

[0118] For example, the communication interface 22 may be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission coding or processing for transmission over a communication link or network.

[0119] The communication interface 28 corresponds to the communication interface 22 , for example, and can be used to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain the encoded image data 21 .

[0120] Both the communication interface 22 and the communication interface 28 can be configured as follows Figure 1 A unidirectional communication interface, or a bidirectional communication interface, indicated by an arrow pointing from the source device 12 to the corresponding communication channel 13 of the destination device 14, and can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission such as encoded image data transmission, etc.

[0121] The video decoder (or decoder) 30 is used to receive the encoded image data 21 and provide decoded image data (or decoded image data) 31 (hereinafter referred to as Figure 3 etc. for further description).

[0122] The post-processor 32 is used to post-process the decoded image data 31 (also called reconstructed image data) such as the decoded image to obtain the post-processed image data 33 such as the post-processed image. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color adjustment, cropping or resampling, or any other processing for generating the decoded image data 31 for display by the display device 34 or the like.

[0123] The display device 34 is used to receive the post-processed image data 33 to display the image to a user or viewer, etc. The display device 34 can be or include any type of display for displaying the reconstructed image, such as an integrated or external display screen or display. For example, the display screen can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display screen.

[0124] although Figure 1 The source device 12 and the destination device 14 are shown as independent devices, but the device embodiment may also include the source device 12 and the destination device 14 or the functions of the source device 12 and the destination device 14 at the same time, that is, the source device 12 or the corresponding function and the destination device 14 or the corresponding function at the same time. In these embodiments, the source device 12 or the corresponding function and the destination device 14 or the corresponding function can be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0125] According to the description, Figure 1 The presence and (exact) division of different units or functions in the source device 12 and / or the destination device 14 shown may vary depending on the actual device and application, which will be obvious to the skilled person.

[0126] The encoder 20 (e.g., video encoder 20) or the decoder 30 (e.g., video decoder 30) or both may be implemented by processing circuitry, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, video encoding-specific processors, or any combination thereof. The encoder 20 may be implemented by processing circuitry to include reference Figure 2 The various modules discussed in encoder 20 and / or any other encoder system or subsystem described herein. Decoder 30 may be implemented by processing circuitry to include reference Figure 3The various modules discussed in the decoder 30 and / or any other decoder systems or subsystems described herein. The processing circuitry may be used to perform the various operations discussed below. If portions of the technology are implemented in software, the device may store the instructions of the software in a suitable computer-readable storage medium and use one or more processors to execute the instructions in hardware to perform the technology of the present invention. One of the video encoder 20 and the video decoder 30 may be integrated into a single device as part of a combined codec (encoder / decoder, CODEC).

[0127] Source device 12 and destination device 14 may include any of a variety of devices, including any type of handheld or fixed device, such as a notebook or laptop computer, a mobile phone, a smart phone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (e.g., a content service server or a content distribution server), a broadcast receiving device, a broadcast transmitting device, etc., and may not use or use any type of operating system. In some cases, source device 12 and destination device 14 may be equipped with components for wireless communication. Therefore, source device 12 and destination device 14 may be wireless communication devices.

[0128] In some cases, Figure 1 The video decoding system 10 shown is merely exemplary, and the techniques provided herein may be applicable to video encoding settings (e.g., video encoding or video decoding) that do not necessarily include any data communication between an encoding device and a decoding device. In other examples, data is retrieved from a local memory, sent over a network, and so on. A video encoding device may encode data and store the data in a memory, and / or a video decoding device may retrieve data from a memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with each other but only encode data to a memory and / or retrieve and decode data from a memory.

[0129] It should be understood that for the examples described with reference to the video encoder 20 in the embodiments of the present application, the video decoder 30 can be used to perform the reverse process. With respect to the signaling syntax elements, the video decoder 30 can be used to receive and parse such syntax elements and decode the related video data accordingly. In some examples, the video encoder 20 can entropy encode the syntax elements into an encoded video bitstream. In such examples, the video decoder 30 can parse such syntax elements and decode the related video data accordingly.

[0130] For ease of description, the following embodiments are described with reference to the universal video coding (VVC) reference software or the high-efficiency video coding (HEVC) developed by the video coding joint working group (joint collaboration team on video coding, JCT-VC) of the ITU-T video coding experts group (VCEG) and the ISO / IEC motion picture experts group (MPEG). Those skilled in the art will appreciate that the embodiments of the present invention are not limited to HEVC or VVC.

[0131] Encoders and encoding methods

[0132] Figure 2 FIG. 2 is an exemplary block diagram of a video encoder 20 according to an embodiment of the present application. Figure 2 As shown, the video encoder 20 includes an input terminal (or input interface) 201, a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270 and an output terminal (or output interface) 272. The mode selection unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254 and a segmentation unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). Figure 2 The illustrated video encoder 20 may also be referred to as a hybrid video encoder or a hybrid video codec-based video encoder.

[0133] The residual calculation unit 204, the transform processing unit 206, the quantization unit 208 and the mode selection unit 260 constitute the forward signal path of the encoder 20, while the inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the buffer 216, the loop filter 220, the decoded picture buffer (DPB) 230, the inter-frame prediction unit 244 and the intra-frame prediction unit 254 constitute the backward signal path of the encoder, wherein the backward signal path of the encoder 20 corresponds to the signal path of the decoder (see Figure 3The inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the loop filter 220, the decoded image buffer 230, the inter-frame prediction unit 244 and the intra-frame prediction unit 254 also constitute a “built-in decoder” of the video encoder 20.

[0134] Images and image segmentation (images and blocks)

[0135] The encoder 20 may be configured to receive, via an input 201 or the like, an image (or image data) 17, e.g., an image in a sequence of images forming a video or a video sequence. The received image or image data may also be a pre-processed image (or pre-processed image data) 19. For simplicity, the following description uses image 17. Image 17 may also be referred to as a current image or an image to be encoded (particularly when distinguishing the current image from other images in video encoding, such as previously encoded and / or decoded images in the same video sequence, i.e., a video sequence also including the current image).

[0136] A (digital) image is or can be considered as a two-dimensional array or matrix of pixels with intensity values. The pixels in the array can also be called pixels (pixel or pel) (short for picture elements). The number of pixels in the array or image in the horizontal and vertical directions (or axes) determines the size and / or resolution of the image. In order to represent color, three color components are usually used, that is, the image can be represented as or include three pixel arrays. In the RBG format or color space, the image includes corresponding red, green and blue pixel arrays. However, in video coding, each pixel is usually represented in a brightness / chrominance format or color space, such as YCbCr, including a brightness component indicated by Y (sometimes also represented by L) and two chrominance components represented by Cb and Cr. The brightness (luma) component Y represents the brightness or grayscale level intensity (for example, the two are the same in grayscale images), while the two chrominance (abbreviated as chroma) components Cb and Cr represent the chrominance or color information components. Accordingly, an image in YCbCr format includes a luminance pixel array of luminance pixel values ​​(Y) and two chrominance pixel arrays of chrominance values ​​(Cb and Cr). An image in RGB format can be converted or transformed into YCbCr format and vice versa, a process also referred to as color conversion or transformation. If the image is black and white, the image may include only a luminance pixel array. Accordingly, an image may be, for example, a luminance pixel array in a monochrome format or a luminance pixel array and two corresponding chrominance pixel arrays in 4:2:0, 4:2:2 and 4:4:4 color formats.

[0137] In one embodiment, an embodiment of the video encoder 20 may include an image segmentation unit ( Figure 2), for dividing the image 17 into a plurality of (usually non-overlapping) image blocks 203. These blocks may also be referred to as root blocks, macroblocks (H.264 / AVC) or coding tree blocks (CTB), or coding tree units (CTU) in the H.265 / HEVC and VVC standards. The partitioning unit may be used to use the same block size for all images in a video sequence and a corresponding grid of defined block sizes, or to change the block size between images or subsets of images or groups of images, and to partition each image into corresponding blocks.

[0138] In other embodiments, the video encoder may be configured to directly receive a block 203 of the image 17, for example, one, several or all blocks constituting the image 17. The image block 203 may also be referred to as a current image block or an image block to be encoded.

[0139] Like image 17, image block 203 is also or can be considered as a two-dimensional array or matrix composed of pixels with intensity values ​​(pixel values), but image block 203 is smaller than image 17. In other words, block 203 may include one pixel array (e.g., a brightness array in the case of monochrome image 17 or a brightness array or chrominance array in the case of a color image) or three pixel arrays (e.g., one brightness array and two chrominance arrays in the case of a color image 17) or any other number and / or type of arrays according to the color format adopted. The number of pixels in the horizontal and vertical directions (or axes) of block 203 defines the size of block 203. Accordingly, the block can be an M×N (M columns×N rows) pixel array, or an M×N transform coefficient array, etc.

[0140] In one embodiment, Figure 2 The video encoder 20 shown is used to encode the image 17 block by block, for example, encoding and prediction are performed on each block 203.

[0141] In one embodiment, Figure 2 The illustrated video encoder 20 may also be used to partition and / or encode an image using slices (also referred to as video slices), wherein an image may be partitioned or encoded using one or more slices (usually non-overlapping). Each slice may include one or more blocks (e.g., coding tree units CTUs) or one or more groups of blocks (e.g., coding tiles in the H.265 / HEVC / VVC standard and bricks in the VVC standard).

[0142] In one embodiment, Figure 2The video encoder 20 shown can also be used to partition and / or encode an image using slices / coding block groups (also called video coding block groups) and / or coding blocks (also called video coding blocks), wherein an image can be partitioned or encoded using one or more slices / coding block groups (usually non-overlapping), each slice / coding block group may include one or more blocks (e.g., CTU) or one or more coding blocks, etc., wherein each coding block may be in a rectangular shape, etc., and may include one or more complete or partial blocks (e.g., CTU).

[0143] Residual calculation

[0144] The residual calculation unit 204 is used to calculate the residual block 205 (the prediction block 265 is described in detail later) based on the image block 203 and the prediction block 265 in the following manner: for example, the pixel value of the prediction block 265 is subtracted from the pixel value of the image block 203 pixel by pixel (pixel by pixel) to obtain the residual block 205 in the pixel domain.

[0145] Transform

[0146] The transform processing unit 206 is used to perform discrete cosine transform (DCT) or discrete sine transform (DST) on the pixel values ​​of the residual block 205 to obtain transform coefficients 207 in the transform domain. The transform coefficients 207 may also be called transform residual coefficients, representing the residual block 205 in the transform domain.

[0147] The transform processing unit 206 may be used to apply an integerized approximation of the DCT / DST, such as the transform specified for H.265 / HEVC. This integerized approximation is typically scaled by a factor compared to an orthogonal DCT transform. In order to maintain the norm of the residual block processed by the forward transform and the inverse transform, other scaling factors are used as part of the transform process. The scaling factor is typically selected based on certain constraints, such as whether the scaling factor is a power of 2 for the shift operation, the bit depth of the transform coefficients, a trade-off between accuracy and implementation cost, etc. For example, a specific scaling factor is specified for the inverse transform on the encoder 20 side by the inverse transform processing unit 212 (and for the corresponding inverse transform on the decoder 30 side by, for example, the inverse transform processing unit 312), and correspondingly, a corresponding scaling factor may be specified for the forward transform on the encoder 20 side by the transform processing unit 206.

[0148] In one embodiment, the video encoder 20 (correspondingly, the transform processing unit 206) may be used to output transform parameters such as one or more transform types, for example, directly output or output after encoding or compression by the entropy coding unit 270, for example, so that the video decoder 30 may receive and use the transform parameters for decoding.

[0149] Quantification

[0150] The quantization unit 208 is used to quantize the transform coefficient 207 by, for example, scalar quantization or vector quantization to obtain a quantized transform coefficient 209 . The quantized transform coefficient 209 may also be referred to as a quantized residual coefficient 209 .

[0151] The quantization process may reduce the bit depth associated with some or all of the transform coefficients 207. For example, an n-bit transform coefficient may be rounded down to an m-bit transform coefficient during quantization, where n is greater than m. The degree of quantization may be modified by adjusting a quantization parameter (QP). For example, for scalar quantization, different degrees of scaling may be applied to achieve finer or coarser quantization. A smaller quantization step size corresponds to a finer quantization, while a larger quantization step size corresponds to a coarser quantization. A suitable quantization step size may be indicated by a quantization parameter (QP). For example, a quantization parameter may be an index of a predefined set of suitable quantization step sizes. For example, a smaller quantization parameter may correspond to fine quantization (a smaller quantization step size), a larger quantization parameter may correspond to coarse quantization (a larger quantization step size), and vice versa. Quantization may include dividing by a quantization step size, while a corresponding or inverse dequantization performed by an inverse quantization unit 210 or the like may include multiplying by a quantization step size. Embodiments according to some standards, such as HEVC, may be used to determine a quantization step size using a quantization parameter. In general, the quantization step size may be calculated using a fixed-point approximation of an equation containing division according to the quantization parameter. Other scaling factors may be introduced for quantization and dequantization to recover the norm of the residual block that may be modified due to the scaling used in the fixed-point approximation of the equations for the quantization step size and the quantization parameter. In one exemplary implementation, the scaling of the inverse transform and dequantization may be combined. Alternatively, a custom quantization table may be used and indicated from the encoder to the decoder in the bitstream, etc. Quantization is a lossy operation, where the larger the quantization step size, the greater the loss.

[0152] In one embodiment, the video encoder 20 (correspondingly, the quantization unit 208) may be configured to output a quantization parameter (QP), for example, directly output or output after being encoded or compressed by the entropy coding unit 270, for example, so that the video decoder 30 may receive and use the quantization parameter for decoding.

[0153] Dequantization

[0154] The inverse quantization unit 210 is used to perform inverse quantization of the quantization unit 208 on the quantized coefficients to obtain dequantized coefficients 211, for example, performing an inverse quantization scheme of the quantization scheme performed by the quantization unit 208 according to or using the same quantization step size as the quantization unit 208. The dequantized coefficients 211 may also be referred to as dequantized residual coefficients 211, corresponding to the transform coefficients 207, but due to the loss caused by quantization, the dequantized coefficients 211 are usually not completely the same as the transform coefficients.

[0155] Inverse Transform

[0156] The inverse transform processing unit 212 is used to perform an inverse transform of the transform performed by the transform processing unit 206, such as an inverse discrete cosine transform (DCT) or an inverse discrete sine transform (DST), to obtain a reconstructed residual block 213 (or a corresponding dequantized coefficient 213) in the pixel domain. The reconstructed residual block 213 may also be referred to as a transform block 213.

[0157] reconstruction

[0158] The reconstruction unit 214 (e.g., the summer 214) is used to add the transform block 213 (i.e., the reconstructed residual block 213) to the prediction block 265 to obtain the reconstructed block 215 in the pixel domain, for example, by adding the pixel point values ​​of the reconstructed residual block 213 and the pixel point values ​​of the prediction block 265.

[0159] Filtering

[0160] The loop filter unit 220 (or simply "loop filter" 220) is used to filter the reconstruction block 215 to obtain the filter block 221, or is generally used to filter the reconstructed pixel points to obtain the filtered pixel point value. For example, the loop filter unit is used to smoothly perform pixel conversion or improve video quality. The loop filter unit 220 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination. For example, the loop filter unit 220 may include a deblocking filter, an SAO filter, and an ALF filter. The order of the filtering process may be a deblocking filter, an SAO filter, and an ALF filter. For another example, a process called luma mapping with chromascaling (LMCS) (i.e., an adaptive in-loop shaper) is added. This process is performed before deblocking. For another example, the deblocking filtering process may also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Figure 2 2. Although shown as a loop filter in FIG. 2, in other configurations, the loop filter unit 220 may be implemented as a post-loop filter. The filter block 221 may also be referred to as a filter reconstruction block 221.

[0161] In one embodiment, the video encoder 20 (correspondingly, the loop filter unit 220) may be used to output loop filter parameters (e.g., SAO filter parameters, ALF filter parameters, or LMCS parameters), for example, directly or after being entropy encoded by the entropy encoding unit 270, such that the decoder 30 may receive and use the same or different loop filter parameters for decoding.

[0162] Decoded Image Buffer

[0163] The decoded picture buffer (DPB) 230 may be a reference picture memory that stores reference picture data for use by the video encoder 20 when encoding video data. The DPB 230 may be formed by any of a variety of memory devices, such as a dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of storage devices. The decoded picture buffer 230 may be used to store one or more filter blocks 221. The decoded picture buffer 230 may also be used to store other previous filter blocks of the same current picture or a different picture such as a previous reconstructed block, such as a previously reconstructed and filtered block 221, and may provide a complete previously reconstructed, i.e., decoded picture (and corresponding reference blocks and pixels) and / or a partially reconstructed current picture (and corresponding reference blocks and pixels), such as for inter-frame prediction. The decoded image buffer 230 may also be used to store one or more unfiltered reconstructed blocks 215, or generally to store unfiltered reconstructed pixels, for example, reconstructed blocks 215 that have not been filtered by the loop filter unit 220, or reconstructed blocks or reconstructed pixels that have not undergone any other processing.

[0164] Mode selection (segmentation and prediction)

[0165] The mode selection unit 260 includes a segmentation unit 262, an inter-frame prediction unit 244, and an intra-frame prediction unit 254, which are used to receive or obtain original image data such as the original block 203 (the current block 203 of the current image 17) and reconstructed block data from the decoded image buffer 230 or other buffers (e.g., column buffers, not shown in the figure), such as filtered and / or unfiltered reconstructed pixels or reconstructed blocks of the same (current) image and / or one or more previously decoded images. The reconstructed block data is used as reference image data required for prediction such as inter-frame prediction or intra-frame prediction to obtain a prediction block 265 or a prediction value 265.

[0166] The mode selection unit 260 may be used to determine or select a partitioning mode (eg, intra-frame or inter-frame prediction mode) for the current block (including no partitioning) and generate a corresponding prediction block 265 to calculate the residual block 205 and reconstruct the reconstruction block 215 .

[0167] In one embodiment, the mode selection unit 260 may be used to select a segmentation and prediction mode (e.g., from prediction modes supported or available by the mode selection unit 260) that provides the best match or minimum residual (minimum residual means better compression in transmission or storage), or provides minimum signaling overhead (minimum signaling overhead means better compression in transmission or storage), or considers or balances both of the above. The mode selection unit 260 may be used to determine the segmentation and prediction mode based on rate distortion optimization (RDO), i.e., select the prediction mode that provides minimum rate distortion optimization. The terms "best", "lowest", "optimal", etc. herein do not necessarily mean "best", "lowest", "optimal" in general, but may also refer to situations where termination or selection criteria are met, for example, values ​​exceeding or falling below a threshold or other restrictions may result in a "suboptimal selection" but reduce complexity and processing time.

[0168] In other words, the partitioning unit 262 may be used to partition an image in a video sequence into a sequence of coding tree units (CTUs), and the CTU 203 may be further partitioned into smaller block portions or sub-blocks (again forming blocks), for example, by iteratively using quad-tree partitioning (QT), binary-tree partitioning (BT) or triple-tree partitioning (TT) or any combination thereof, and for performing prediction on each of the block portions or sub-blocks, for example, wherein the mode selection comprises selecting a tree structure of the partition block 203 and selecting a prediction mode to be applied to each of the block portions or sub-blocks.

[0169] The segmentation (eg, performed by segmentation unit 262) and prediction processes (eg, performed by inter-prediction unit 244 and intra-prediction unit 254) performed by video encoder 20 are described in detail below.

[0170] segmentation

[0171] The partitioning unit 262 may partition (or divide) a coding tree unit 203 into smaller parts, such as small blocks of square or rectangular shapes. For an image with three pixel arrays, a CTU consists of N×N luminance pixel blocks and two corresponding chrominance pixel blocks.

[0172] The H.265 / HEVC video coding standard divides a frame into non-overlapping CTUs, and the size of the CTU can be set to 64×64 (the size of the CTU can also be set to other values, such as the CTU size in the JVET reference software JEM is increased to 128×128 or 256×256). The 64×64 CTU contains a rectangular pixel matrix with 64 columns and 64 pixels in each column, and each pixel contains a luminance component and / or a chrominance component.

[0173] H.265 uses a QT-based CTU partitioning method, taking CTU as the root node of QT. According to the QT partitioning method, CTU is recursively divided into several leaf nodes. A node corresponds to an image area. If the node is not divided, the node is called a leaf node, and the image area corresponding to it is a CU. If the node continues to be divided, the image area corresponding to the node can be divided into four areas of the same size (the length and width are half of the divided area), each area corresponds to a node, and it is necessary to determine whether these nodes will be divided. Whether a node is divided is indicated by the split flag split_cu_flag corresponding to the node in the bitstream. A node A is divided once to obtain 4 nodes Bi, i=0~3, Bi is called the child node of A, and A is called the parent node of Bi. The QT level (qtDepth) of the root node is 0, and the QT level of the node is the four QT levels of the parent node of the node plus 1.

[0174] In the H.265 / HEVC standard, for images in YUV4:2:0 format, a CTU contains a luma block and two chroma blocks. The luma block and the chroma block can be divided in the same way, which is called a luma-chroma joint coding tree. In VVC, if the current frame is an I frame, when a CTU is a node of a preset size (such as 64×64) in an intra-frame coded frame (I frame), the luma block contained in the node is divided into a group of coding units containing only luma blocks through the luma coding tree, and the chroma block contained in the node is divided into a group of coding units containing only chroma blocks through the chroma coding tree; the division of the luma coding tree and the chroma coding tree are independent of each other. This kind of luma block and chroma block use independent coding trees, which are called separate trees. In H.265, CU contains luma pixels and chroma pixels; in H.266, AVS3 and other standards, in addition to CUs containing both luma pixels and chroma pixels, there are also luma CUs containing only luma pixels and chroma CUs containing only chroma pixels.

[0175] As described above, the video encoder 20 is operable to determine or select the best or optimal prediction mode from a (predetermined) set of prediction modes. The set of prediction modes may include, for example, an intra prediction mode and / or an inter prediction mode.

[0176] Intra prediction

[0177] The intra prediction mode set may include 35 different intra prediction modes, for example, non-directional modes like DC (or mean) mode and plane mode, or directional modes as defined in HEVC, or may include 67 different intra prediction modes, for example, non-directional modes like DC (or mean) mode and plane mode, or directional modes as defined in VVC. For example, several traditional angle intra prediction modes are adaptively replaced with wide-angle intra prediction modes for non-square blocks defined in VVC. For another example, in order to avoid the division operation of DC prediction, only the longer side is used to calculate the average value of non-square blocks. In addition, the intra prediction result of the plane mode can also be modified using the position-dependent intra prediction combination (PDPC) method.

[0178] The intra prediction unit 254 is configured to generate an intra prediction block 265 using reconstructed pixels of adjacent blocks of the same current image according to an intra prediction mode in the intra prediction mode set.

[0179] The intra-frame prediction unit 254 (or generally the mode selection unit 260) is also used to output intra-frame prediction parameters (or generally information indicating the selected intra-frame prediction mode of the block) in the form of syntax elements 266 to the entropy coding unit 270 for inclusion in the encoded image data 21, so that the video decoder 30 can perform operations, such as receiving and using the prediction parameters for decoding.

[0180] Inter prediction

[0181] In a possible implementation, the set of inter-prediction modes depends on the available reference picture (i.e., at least part of the previously decoded picture stored in the DBP 230 as mentioned above) and other inter-prediction parameters, for example, on whether to use the entire reference picture or only a part of the reference picture, such as a search window area around the area of ​​the current block, to search for the best matching reference block, and / or on whether to perform pixel interpolation such as half-pixel, quarter-pixel and / or 1 / 16 interpolation, for example.

[0182] In addition to the above prediction modes, skip mode and / or direct mode may also be employed.

[0183] For example, extended merge prediction, the merge candidate list of this mode consists of the following five candidate types in order: spatial MVP from spatially adjacent CUs, temporal MVP from collocated CUs, history-based MVP from FIFO tables, pairwise average MVP, and zero MV. Decoder side motion vector refinement (DMVR) based on bilateral matching can be used to increase the accuracy of the MV of the merge mode. Merge mode with MVD (MMVD) is derived from the merge mode with motion vector difference. The MMVD flag is sent immediately after the skip flag and merge flag are sent to specify whether the CU uses the MMVD mode. The CU-level adaptive motion vector resolution (AMVR) scheme can be used. AMVR supports encoding the MVD of the CU with different precisions. The MVD of the current CU is adaptively selected according to the prediction mode of the current CU. When the CU is encoded in merge mode, the combined inter / intra prediction (CIIP) mode can be applied to the current CU. The CIIP prediction is obtained by weighted averaging the inter-frame and intra-frame prediction signals. For affine motion compensated prediction, the affine motion field of the block is described by the motion information of 2 control points (4 parameters) or 3 control points (6 parameters) motion vectors. Subblock-based temporal motion vector prediction (SbTMVP) is similar to the temporal motion vector prediction (TMVP) in HEVC, but the motion vector of the sub-CU within the current CU is predicted. Bidirectional optical flow (BDOF), formerly known as BIO, is a simplified version that reduces calculations, especially in terms of the number of multiplications and the size of the multipliers. In the triangle partitioning mode, the CU is evenly divided into two triangular parts in two ways: diagonal partitioning and anti-diagonal partitioning. In addition, the bidirectional prediction mode is extended on the basis of simple averaging to support weighted averaging of two prediction signals.

[0184] The inter-frame prediction unit 244 may include a motion estimation (ME) unit and a motion compensation (MC) unit (both in Figure 2). The motion estimation unit may be configured to receive or acquire the image block 203 (the current image block 203 of the current image 17) and the decoded image 231, or at least one or more previously reconstructed blocks, for example, reconstructed blocks of one or more other / different previously decoded images 231, for motion estimation. For example, a video sequence may include the current image and the previously decoded image 231, or in other words, the current image and the previously decoded image 231 may be part of or form a sequence of images forming the video sequence.

[0185] For example, the encoder 20 may be configured to select a reference block from a plurality of reference blocks of the same or different images in a plurality of other images, and provide the reference image (or reference image index) and / or the offset (spatial offset) between the position (x, y coordinates) of the reference block and the position of the current block as an inter-frame prediction parameter to the motion estimation unit. The offset is also referred to as a motion vector (MV).

[0186] The motion compensation unit is used to obtain, for example, receive, inter-frame prediction parameters, and perform inter-frame prediction based on or using the inter-frame prediction parameters to obtain an inter-frame prediction block 246. The motion compensation performed by the motion compensation unit may include extracting or generating a prediction block based on a motion / block vector determined by motion estimation, and may also include performing interpolation with sub-pixel accuracy. Interpolation filtering can generate pixel points of other pixels from pixel points of known pixels, thereby potentially increasing the number of candidate prediction blocks that can be used to encode the image block. Once a motion vector corresponding to a PU of a current image block is received, the motion compensation unit can locate the prediction block pointed to by the motion vector in one of the reference image lists.

[0187] The motion compensation unit may also generate syntax elements associated with blocks and video slices for use by the video decoder 30 when decoding image blocks of the video slice. In addition, or as an alternative to slices and corresponding syntax elements, coding block groups and / or coding blocks and corresponding syntax elements may be generated or used.

[0188] Entropy Coding

[0189] The entropy coding unit 270 is used to apply an entropy coding algorithm or scheme (e.g., a variable length coding (VLC) scheme, a context adaptive VLC scheme (CALVC), an arithmetic coding scheme, a binarization algorithm, a context adaptive binary arithmetic coding (CABAC), a syntax-based context-adaptive binary arithmetic coding (SBAC), a probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques) to the quantized residual coefficients 209, inter-frame prediction parameters, intra-frame prediction parameters, loop filter parameters, and / or other syntax elements to obtain coded image data 21 that can be output in the form of a coded bit stream 21 through an output terminal 272, so that the video decoder 30, etc. can receive and use the parameters for decoding. The coded bit stream 21 can be transmitted to the video decoder 30, or it can be stored in a memory for later transmission or retrieval by the video decoder 30.

[0190] Other structural variations of the video encoder 20 may be used to encode the video stream. For example, a non-transform based encoder 20 may directly quantize the residual signal without a transform processing unit 206 for certain blocks or frames. In another implementation, the encoder 20 may have a quantization unit 208 and an inverse quantization unit 210 combined into a single unit.

[0191] Decoder and decoding method

[0192] Figure 3 3 is an exemplary block diagram of a video decoder 30 according to an embodiment of the present application. The video decoder 30 is used to receive, for example, coded image data 21 (for example, coded bitstream 21) encoded by an encoder 20, and obtain a decoded image 331. The coded image data or bitstream includes information for decoding the coded image data, such as data representing image blocks of a coded video slice (and / or a coding block group or coding block) and related syntax elements.

[0193] exist Figure 3In the example of FIG. 3 , decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., a summer 314), a loop filter 320, a decoded picture buffer (DBP) 330, a mode application unit 360, an inter-frame prediction unit 344, and an intra-frame prediction unit 354. Inter-frame prediction unit 344 may be or include a motion compensation unit. In some examples, video decoder 30 may perform substantially the same as reference Figure 2 The decoding process of the video encoder 100 is the reverse of the encoding process described above.

[0194] As described for encoder 20, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded picture buffer DPB 230, inter prediction unit 344, and intra prediction unit 354 also constitute a “built-in decoder” of video encoder 20. Accordingly, inverse quantization unit 310 may be functionally identical to inverse quantization unit 110, inverse transform processing unit 312 may be functionally identical to inverse transform processing unit 122, reconstruction unit 314 may be functionally identical to reconstruction unit 214, loop filter 320 may be functionally identical to loop filter 220, and decoded picture buffer 330 may be functionally identical to decoded picture buffer 230. Therefore, the explanation of the corresponding units and functions of video encoder 20 applies accordingly to the corresponding units and functions of video decoder 30.

[0195] Entropy decoding

[0196] The entropy decoding unit 304 is used to parse the bit stream 21 (or generally the encoded image data 21) and perform entropy decoding on the encoded image data 21 to obtain the quantization coefficients 309 and / or the decoded encoding parameters ( Figure 3 The entropy decoding unit 304 may be used to provide the inter-frame prediction parameters, intra-frame prediction parameters and / or other syntax elements to the mode application unit 360, and to provide other parameters to other units of the decoder 30. The video decoder 30 may receive syntax elements at the video slice and / or video block level. In addition, or as an alternative to slices and corresponding syntax elements, coding block groups and / or coding blocks and corresponding syntax elements may be received or used.

[0197] Dequantization

[0198] The inverse quantization unit 310 may be configured to receive a quantization parameter (QP) (or generally information related to inverse quantization) and a quantization coefficient from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304), and inverse quantize the decoded quantization coefficient 309 based on the quantization parameter to obtain an inverse quantization coefficient 311, which may also be referred to as a transform coefficient 311. The inverse quantization process may include using the quantization parameter calculated by the video encoder 20 for each video block in the video slice to determine a degree of quantization, and also determine a degree of inverse quantization that needs to be performed.

[0199] Inverse Transform

[0200] The inverse transform processing unit 312 may be configured to receive the dequantized coefficients 311, also referred to as transform coefficients 311, and apply a transform to the dequantized coefficients 311 to obtain a reconstructed residual block 213 in the pixel domain. The reconstructed residual block 213 may also be referred to as a transform block 313. The transform may be an inverse transform, such as an inverse DCT, an inverse DST, an inverse integer transform, or a conceptually similar inverse transform process. The inverse transform processing unit 312 may also be configured to receive transform parameters or corresponding information from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304) to determine the transform applied to the dequantized coefficients 311.

[0201] reconstruction

[0202] The reconstruction unit 314 (eg, the summer 314 ) is used to add the reconstructed residual block 313 to the prediction block 365 to obtain the reconstructed block 315 in the pixel domain, for example, by adding the pixel point values ​​of the reconstructed residual block 313 and the pixel point values ​​of the prediction block 365 .

[0203] Filtering

[0204] The loop filter unit 320 (in or after the encoding loop) is used to filter the reconstruction block 315 to obtain a filter block 321, so as to smoothly perform pixel conversion or improve video quality, etc. The loop filter unit 320 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination. For example, the loop filter unit 220 may include a deblocking filter, an SAO filter, and an ALF filter. The order of the filtering process may be a deblocking filter, an SAO filter, and an ALF filter. For another example, a process called luma mapping with chromascaling (LMCS) (i.e., an adaptive in-loop shaper) is added. This process is performed before deblocking. For another example, the deblocking filtering process may also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Figure 3 3. Although shown as a loop filter in FIG. 3, in other configurations, the loop filter unit 320 may be implemented as a post-loop filter.

[0205] Decoded Image Buffer

[0206] The decoded video blocks 321 in one picture are then stored in a decoded picture buffer 330 which stores the decoded pictures 331 as reference pictures for subsequent motion compensation of other pictures and / or for respective output display.

[0207] The decoder 30 is used to output the decoded image 311 through the output terminal 312, etc., to display it to the user or for the user to view it.

[0208] predict

[0209] The inter-frame prediction unit 344 may be functionally the same as the inter-frame prediction unit 244 (particularly the motion compensation unit), and the intra-frame prediction unit 354 may be functionally the same as the inter-frame prediction unit 254, and may determine the division or segmentation and perform prediction based on the segmentation and / or prediction parameters or corresponding information received from the coded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304). The mode application unit 360 may be used to perform prediction (intra-frame or inter-frame prediction) of each block according to the reconstructed block, block or corresponding pixel point (filtered or unfiltered), and obtain a prediction block 365.

[0210] When the video slice is encoded as an intra coded (I) slice, the intra prediction unit 354 in the mode application unit 360 is used to generate a prediction block 365 for the image block of the current video slice based on the indicated intra prediction mode and data from the previously decoded block of the current image. When the video image is encoded as an inter coded (i.e., B or P) slice, the inter prediction unit 344 (e.g., motion compensation unit) in the mode application unit 360 is used to generate a prediction block 365 for the video block of the current video slice based on the motion vector and other syntax elements received from the entropy decoding unit 304. For inter prediction, these prediction blocks can be generated from one of the reference images in one of the reference image lists. The video decoder 30 can construct reference frame list 0 and list 1 using a default construction technique based on the reference images stored in the DPB 330. The same or similar processes may be applied to embodiments of coding block groups (e.g., video coding block groups) and / or coding blocks (e.g., video coding blocks) in addition to or as an alternative to slices (e.g., video slices), for example, a video may be encoded using I, P or B coding block groups and / or coding blocks.

[0211] The mode application unit 360 is used to determine prediction information for a video block of a current video slice by parsing motion vectors and other syntax elements, and use the prediction information to generate a prediction block for the current video block being decoded. For example, the mode application unit 360 uses some of the received syntax elements to determine the prediction mode (e.g., intra-frame prediction or inter-frame prediction) for encoding the video block of the video slice, the inter-frame prediction slice type (e.g., B slice, P slice, or GPB slice), the construction information of one or more reference picture lists for the slice, the motion vector for each inter-frame coded video block of the slice, the inter-frame prediction state for each inter-frame coded video block of the slice, and other information to decode the video block in the current video slice. In addition to slices (e.g., video slices) or as an alternative to slices, the same or similar process can be applied to embodiments of coding block groups (e.g., video coding block groups) and / or coding blocks (e.g., video coding blocks), for example, the video can be encoded using I, P, or B coding block groups and / or coding blocks.

[0212] In one embodiment, Figure 3 The illustrated video encoder 30 may also be used to partition and / or decode an image using slices (also referred to as video slices), where an image may be partitioned or decoded using one or more slices (usually non-overlapping). Each slice may include one or more blocks (e.g., CTUs) or one or more groups of blocks (e.g., coding blocks in the H.265 / HEVC / VVC standard and bricks in the VVC standard).

[0213] In one embodiment, Figure 3The video decoder 30 shown can also be used to segment and / or decode an image using slices / coding block groups (also called video coding block groups) and / or coding blocks (also called video coding blocks), wherein an image can be segmented or decoded using one or more slices / coding block groups (usually non-overlapping), each slice / coding block group may include one or more blocks (e.g., CTU) or one or more coding blocks, etc., wherein each coding block may be in a rectangular shape, etc., and may include one or more complete or partial blocks (e.g., CTU).

[0214] Other variations of the video decoder 30 may be used to decode the encoded image data 21. For example, the decoder 30 may generate an output video stream without the loop filter unit 320. For example, a non-transform based decoder 30 may directly dequantize the residual signal without the inverse transform processing unit 312 for certain blocks or frames. In another implementation, the video decoder 30 may have the dequantization unit 310 and the inverse transform processing unit 312 combined into a single unit.

[0215] It should be understood that the processing result of the current step can be further processed in the encoder 20 and the decoder 30 and then output to the next step. For example, after interpolation filtering, motion vector derivation or loop filtering, the processing result of interpolation filtering, motion vector derivation or loop filtering can be further operated, such as clipping or shifting operation.

[0216] It should be noted that the derived motion vector of the current block (including but not limited to the control point motion vector of the affine mode, the sub-block motion vector of the affine, plane, ATMVP mode, the temporal motion vector, etc.) can be further operated. For example, the value of the motion vector is limited to a predefined range according to the representation bit of the motion vector. If the representation bit of the motion vector is bitDepth, the range is -2^(bitDepth-1) to 2^(bitDepth-1)-1, where "^" represents the power. For example, if bitDepth is set to 16, the range is -32768 to 32767; if bitDepth is set to 18, the range is -131072 to 131071. For example, the value of the derived motion vector (e.g., the MV of 4 4×4 sub-blocks in an 8×8 block) is limited so that the maximum difference between the integer parts of the 4 4×4 sub-block MVs does not exceed N pixels, for example, not more than 1 pixel. Two methods of limiting motion vectors according to bitDepth are provided here.

[0217] Although the above embodiments mainly describe video coding and decoding, it should be noted that the embodiments of the decoding system 10, the encoder 20 and the decoder 30 and other embodiments described herein can also be used for still image processing or coding and decoding, that is, the processing or coding and decoding of a single image in video coding and decoding that is independent of any previous or consecutive images. In general, if the image processing is limited to a single image 17, the inter-frame prediction unit 244 (encoder) and the inter-frame prediction unit 344 (decoder) may not be available. All other functions (also called tools or techniques) of the video encoder 20 and the video decoder 30 can also be used for still image processing, such as residual calculation 204 / 304, transformation 206, quantization 208, inverse quantization 210 / 310, (inverse) transformation 212 / 312, segmentation 262 / 362, intra-frame prediction 254 / 354 and / or loop filtering 220 / 320, entropy coding 270 and entropy decoding 304.

[0218] Figure 4 FIG. 4 is an exemplary block diagram of a video decoding device 400 according to an embodiment of the present application. The video decoding device 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 400 may be an encoder, such as Figure 1 The video encoder 20 in the embodiment may also be a decoder, for example Figure 1 The video decoder 30 in.

[0219] The video decoding device 400 includes: an input port 410 (or input port 410) and a receiving unit (Rx) 420 for receiving data; a processor, a logic unit or a central processing unit (CPU) 430 for processing data; a transmitter unit (Tx) 440 and an output port 450 (or output port 450) for transmitting data; and a memory 460 for storing data. The video decoding device 400 may also include an optical-to-electrical (OE) component and an electrical-to-optical (EO) component coupled to the input port 410, the receiving unit 420, the transmitting unit 440 and the output port 450 for the output or output of optical signals or electrical signals.

[0220] The processor 430 is implemented by hardware and software. The processor 430 can be implemented as one or more processor chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 430 communicates with the input port 410, the receiving unit 420, the sending unit 440, the output port 450, and the memory 460. The processor 430 includes a decoding module 470. The decoding module 470 implements the embodiments disclosed above. For example, the decoding module 470 performs, processes, prepares, or provides various encoding operations. Therefore, the decoding module 470 provides substantial improvements to the functions of the video decoding device 400 and affects the switching of the video decoding device 400 to different states. Alternatively, the decoding module 470 is implemented with instructions stored in the memory 460 and executed by the processor 430.

[0221] The memory 460 includes one or more disks, tape drives, and solid-state hard disks, and can be used as an overflow data storage device for storing such programs when such programs are selected for execution, and for storing instructions and data read during program execution. The memory 460 can be volatile and / or non-volatile, and can be a read-only memory (ROM), a random access memory (RAM), a ternary content-addressable memory (TCAM), and / or a static random-access memory (SRAM).

[0222] To facilitate understanding, some technologies or terms used in the embodiments of the present application are explained below, and these technologies or terms are also considered as part of the content of the invention.

[0223] 1. Scalable Video Coding

[0224] Also known as scalable video coding, it is an extension of the current video coding standard (generally scalable video coding (SVC), an extension of advanced video coding (AVC) (H.264), or scalable high efficiency video coding (SHVC), an extension of high efficiency video coding (HEVC) (H.265). The emergence of scalable video coding is mainly to solve the problems of packet loss and delay jitter caused by real-time changes in network bandwidth in real-time video transmission.

[0225] The basic structure in scalable video coding can be called a hierarchy. Scalable video coding technology can obtain hierarchical code streams of different resolutions by spatially stratifying the original image blocks (resolution stratification). Resolution can refer to the size of the image block in pixels, the resolution of the lower layer is lower, and the resolution of the higher layer is not lower than the resolution of the lower layer; or, by temporally stratifying the original image blocks (frame rate stratification), hierarchical code streams of different frame rates can be obtained. Frame rate can refer to the number of image frames contained in the video per unit time, the frame rate of the lower layer is lower, and the frame rate of the higher layer is not lower than the frame rate of the lower layer; or, by quality stratifying the original image blocks, hierarchical code streams of different coding qualities can be obtained. Coding quality can refer to the quality of the video, the image distortion of the lower layer is greater, and the image distortion of the higher layer is not higher than the image distortion of the lower layer.

[0226] Generally, the layer called the base layer is the lowest layer in scalable video coding. In the spatial layer, the base layer image blocks are encoded using the lowest resolution; in the temporal layer, the base layer image blocks are encoded using the lowest frame rate; in the quality layer, the base layer image blocks are encoded using the highest QP or the lowest bit rate. That is, the base layer is the lowest quality layer in scalable video coding. The layer called the enhancement layer is the layer above the base layer in scalable video coding, and can be divided into multiple enhancement layers from low to high. The lowest enhancement layer encodes the merged bitstream based on the coding information obtained from the base layer, and its coding resolution is higher than that of the base layer, or the frame rate is higher than that of the base layer, or the bit rate is larger than that of the base layer. The higher enhancement layer can encode higher quality image blocks based on the coding information of the lower enhancement layer.

[0227] For example, SHVC uses a layered coding architecture, where each layer reuses the same HEVC encoder core, so that scalable coding can be implemented efficiently and simply without changing the single-layer HEVC encoder core. SHVC is also the industry's first scalable video coding standard built only on a high-level syntax level.

[0228] 1. SHVC provides a mechanism to encode video signals into several levels. The processing process is as follows:

[0229] At the encoding end, the mechanism encodes the video sequence only once to generate a compressed bitstream containing a base layer (BL) and multiple enhancement layers (EL), such as Figure 5a ( Figure 5a As shown in the schematic diagram of the SHVC encoding end architecture of the embodiment of the present application,

[0230] a) The base layer input image is input to the base layer encoder for encoding. The base layer encoder reuses the existing HEVC encoder for encoding. Each coding block can perform intra-frame prediction or inter-frame prediction. After obtaining the predicted value, it is subtracted from the original image pixel value to obtain the residual value. Finally, the transformed / quantized residual value is entropy encoded to obtain the base layer code stream.

[0231] It should be noted that during the encoding process, it is also necessary to output a reconstructed image, specifically, the residual after inverse quantization / inverse transformation and the predicted value are added, and then the reconstructed image is obtained after loop filtering. The reconstructed image needs to be placed in the decoded picture buffer (DPB) for inter-frame reference for subsequent image encoding of the base layer, and also for inter-layer prediction of the encoding of the enhancement layer image at the current moment. The inter-layer prediction in the existing SHVC architecture can reuse the inter-frame reference method in HEVC for prediction.

[0232] b) The enhanced layer input image is input to the enhanced layer encoder for encoding. The enhanced layer encoder also reuses the existing HEVC encoder for encoding. The encoding process is basically the same as the basic layer image. The difference is that the enhanced layer image refers to the reconstructed image output by the basic layer encoder during the encoding process.

[0233] At the decoding end, Figure 5b ( Figure 5b As shown in the schematic diagram of the SHVC decoding end architecture of the embodiment of the present application,

[0234] a) The base layer bitstream can be decoded alone to produce the lowest quality video.

[0235] b) The enhanced layer bitstream contains the additional information needed to reconstruct high-quality video. Therefore, the enhanced layer bitstream needs to be combined with the basic layer bitstream information to decode a high-quality video image. The more enhanced layer bitstreams are decoded, the higher the quality of the reconstructed video image. Figure 5b Only two layers of decoding are shown.

[0236] The video bitstream generated by single-layer HEVC encoding may cause a sudden drop in the quality of the reconstructed video due to network congestion or data packet loss, or even make the reconstructed video unwatchable. In the face of this situation, the bitstream generated by SHVC can be adaptively transmitted and decoded, so the degradation of the reconstructed video quality becomes more stable. SHVC has excellent compression efficiency, good adaptability, robustness, fault tolerance, and scalability, which provides a guarantee for effective video transmission in heterogeneous network environments. However, in order to obtain the same reconstructed video quality, SHVC usually requires more bits than single-layer HEVC encoding.

[0237] 2. SHVC layered coding is mainly reflected in three aspects: time, space and quality.

[0238] (1) Temporal layering: It means that the compressed bitstream contains several temporal layers, and different temporal layers correspond to different video frame rates, such as Figure 5c ( Figure 5c As shown in the schematic diagram of the SHVC temporal layered architecture of the embodiment of the present application, the BL layer only encodes low frame rate video, such as Figure 5c For even frames with a Picture Order Count (POC) of 0, 2, etc., the encoding process reuses the existing HEVC standard encoder, where each coding block may perform intra-frame prediction or inter-frame prediction. For example, inter-frame prediction can refer to Figure 5c The arrow in the figure indicates that the BL layer code stream decoding can only output the reconstructed video at a low frame rate; the EL layer only encodes the additional frame rate video, such as Figure 5c For odd frames with POC of 0, 3, etc., the encoding process reuses the existing HEVC standard encoder, in which each coding block may perform intra-frame prediction, inter-frame prediction, and inter-layer prediction. For example, inter-frame prediction and inter-layer prediction can refer to Figure 5c The output picture decoded by the EL layer is combined with the output picture of the BL layer to obtain a high frame rate reconstructed video.

[0239] (2) Spatial layering: refers to the compressed bitstream containing several spatial layers. Different spatial layers correspond to different video sizes, such as Figure 5d ( Figure 5d As shown in the schematic diagram of the SHVC spatial hierarchical architecture of the embodiment of the present application, the original image is downsampled to a low resolution and then input into the BL layer encoder for encoding. The encoding process reuses the existing HEVC standard encoder, wherein each coding block may perform intra-frame prediction or inter-frame prediction. For example, inter-frame prediction can refer to Figure 5d The arrow in the figure indicates that the BL layer code stream decoding can only output a low-resolution reconstructed video; the original un-downsampled original image is input into the EL layer encoder for encoding. The encoding process reuses the existing HEVC standard encoder, in which each coding block may perform intra-frame prediction, inter-frame prediction, and inter-layer prediction. For example, inter-frame prediction and inter-layer prediction can refer to the following. Figure 5d The arrow in the figure indicates that during the decoding process of the EL layer, a high-resolution reconstructed video can be obtained by referring to the output image of the BL layer.

[0240] (3) Quality layering: refers to the compression bitstream containing several quality layers, and different quality layers correspond to different video fidelity, such as Figure 5e ( Figure 5eAs shown in the schematic diagram of the SHVC quality layered architecture of the embodiment of the present application, the original image is input into the BL layer encoder for encoding, the frame level QP is set to 22 (the larger the QP, the worse the reconstructed video quality), and the encoding process reuses the existing HEVC standard encoder, wherein each coding block may perform intra-frame prediction or inter-frame prediction. For example, inter-frame prediction can refer to Figure 5e The arrow in the figure indicates that the BL layer code stream decoding can only output low-quality reconstructed video; the original image is input to the EL layer encoder for encoding, and the frame level QP is set to 20 (the smaller the QP, the better the reconstructed video quality). The encoding process reuses the existing HEVC standard encoder, in which each coding block may perform intra-frame prediction, inter-frame prediction, and inter-layer prediction. For example, inter-frame prediction and inter-layer prediction can refer to Figure 5e Indicated by the arrow in the figure. During the decoding process of the EL layer, high-quality reconstructed video can be obtained by referring to the output picture of the BL layer.

[0241] In addition, in order to provide viewers with a better viewing experience, high-definition television (HDTV) technology and ultra-high-definition television (UHDTV) technology have been booming around the world in recent years. UHDTV has a larger bit depth and a wider color gamut than HDTV. For backward compatibility, SHVC adds support for new scalability such as bit depth scalability and color gamut scalability.

[0242] 3. The code stream organization structure of layered coding is the interleaving of the code streams of the EL layer and the BL layer. In order to ensure that the inter-layer prediction of the EL layer can smoothly refer to the output of the BL layer at the same time to reconstruct the image, the BL layer code stream of the image at the same time must be decoded before the EL layer code stream. The specific code stream organization is as follows: Figure 5f ( Figure 5f As shown in the schematic diagram of the code stream organization structure of SHVC encoding in the embodiment of the present application, the header information of the BL layer image code stream can be identified as a BL layer code stream, and the header information of the EL layer image code stream can be identified as an EL layer code stream. After decoding the interleaved code stream, the BL layer header information is identified, the BL layer code stream is decoded, and the EL layer header information is identified. According to the situation of the client network and the device, the EL layer code stream is selectively received or discarded, so that the switching of adaptive frame rate, resolution and quality can be effectively realized at the terminal.

[0243] 2. Key Terms

[0244] 1. Keystream: In cryptography, a keystream is a stream of random or pseudo-random characters that is combined with a plaintext data stream to generate encrypted messages (ciphertext).

[0245] 2. Network Abstract Layer Unit (NALU): In video coding, the encoded data can be encapsulated into individual NALUs for transmission. It has a unified structure, that is, each NALU contains two parts: the NALU header and the NALU payload (Raw Byte Sequence Payload, RBSP). The length of the NALU Header is two bytes, reflecting the content payload of the NALU; the length of the RBSP is an integer number of bytes, carrying the bitstream segment after video coding. The Sequence Parameter Set (SPS), the Picture Parameter Set (PPS), and the video coding data can all be encapsulated in the form of NALUs, such as Figure 6a ( Figure 6a is a schematic diagram of the NALU structure of the embodiment of the present application).

[0246] 3. SPS / PPS: Contains the parameters required for encoding the entire sequence / whole image.

[0247] Optionally, in addition to the key parameters describing the characteristics of the encoded sequence, the SPS can also carry the control switches of most coding tools and the corresponding coding limit parameters. The scope of action of the parameters in the SPS is the entire encoded sequence, which means that during the entire video coding process, there is only one SPS and the parameters therein do not change.

[0248] Optionally, some tool parameters of the PPS can be allowed to change at the frame level, but they do not change frequently. The tool control parameters at the picture level can be repeatedly sent, which means that the parameters in the PPS are allowed to change with the change of image characteristics. Usually, the scope of action of the PPS is n pictures (0 < n < all images).

[0249] III. Related Technologies

[0250] 1. GB35114 video data encryption, such as Figure 6b ( Figure 6b is the GB35114 encryption flowchart of the embodiment of the present application).

[0251] a) The encoder randomly generates a 16-byte Video Encryption Key (VEK) periodically.

[0252] b) Read the current coding payload (i.e., the RSRP to be encrypted in the currently processed NALU, and the RSRP can include one of the SPS, PPS, and video coding data).

[0253] c) When both of the following conditions are met, the current VEK is invalidated and a new VEK is activated; otherwise, the current VEK continues to be used:

[0254] i. The RBSP is the first coded data starting from a group of pictures (GOP) (GOP, for example, may refer to a group of P frames starting from an I frame to the next I frame, and the embodiment of the present application does not specifically limit GOP);

[0255] ii. A new VEK is available.

[0256] d) The encoder randomly generates a 16-byte initial vector (IV).

[0257] e) Encrypt the currently used VEK and IV with the video key encryption key (VKEK) of the source, and encapsulate the corresponding ciphertext into the NALU of the security parameter set.

[0258] f) Use the output-feedback (OFB) mode of the agreed block encryption algorithm to generate a key stream using VEK and IV.

[0259] g) The key stream is bit-aligned and XORed with the current encoded payload to obtain the encrypted encoded payload.

[0260] h) Encapsulate the encrypted encoded payload into a VCL_NALU and set the encryption_idc of the NALU to 1.

[0261] If a new VEK is activated or a new IV is used, the security parameter set is updated first, and the NALU of the security parameter set is re-encapsulated. The encrypted encoded payload is generated with the new VEK or IV, and then the VCL_NALU is output.

[0262] 2. Decryption of GB35114 video data, such as Figure 6c ( Figure 6c As shown in the GB35114 decryption flow chart of the embodiment of the present application,

[0263] a) Get the ciphertext of IV and VEK from the NALU of the security parameter set, denoted as EVEK, and decrypt EVEK with VKEK to obtain VEK and IV.

[0264] b) Reading the current coded payload to be decrypted (ie, the RSRP to be decrypted in the NALU currently being processed, where the RSRP may include one of the SPS, PPS, and video coded data).

[0265] c) When both of the following conditions are met, the current VEK is invalidated and a new VEK is activated; otherwise, the current VEK continues to be used:

[0266] i. The RBSP is the first coded slice data at the beginning of the GOP;

[0267] ii. A new VEK is available.

[0268] d) Use the OFB mode of the agreed block encryption algorithm and generate the key stream using VEK and IV.

[0269] e) The key stream is bit-aligned with the current encoded payload and an XOR operation is performed to obtain the decrypted encoded payload.

[0270] It can be seen that GB35114 video encryption rules

[0271] 1) Do not encrypt the security parameter set (sec_parameter_set);

[0272] 2) After receiving the new key or new security parameter, the decoding end needs to update the security parameter set in time to replace the old key and parameter;

[0273] 3) The length of VEK needs to be 16 bytes and is randomly generated by a front-end device with security functions. The minimum period for VEK update is one GOP, and the generally used update period is 30 or 60 GOPs.

[0274] 4) The NALU carrying the security parameter set for the updated VEK and IV should be transmitted before the encrypted VCL_NALU after the new key is activated;

[0275] 5) The NALU of the security parameter set carrying device information should be transmitted before or simultaneously with the NALU carrying the authentication data set;

[0276] 6) The last byte of the encoded payload is not encrypted;

[0277] 7) When using the OFB mode of the block encryption algorithm for encryption, each GOP uses a different IV, which is randomly generated by the encoder.

[0278] 3. GB35114 video data signature, such as Figure 6d ( Figure 6d As shown in the GB35114 signature flow chart of the embodiment of the present application,

[0279] a) Read one or more NALU data that need to be authenticated (which may include NALUs such as SPS, PPS, and video encoding data);

[0280] b) Process the data in NALU (including NAL Header and RBSP) to generate a digest according to the hash algorithm specified by hash_type in the security parameter set (currently only the cryptographic hash function standard SM3 can be used in the security monitoring digital audio and video coding technology standard (Surveillance Video and Audio Coding, SVAC) 2, which belongs to the hash algorithm);

[0281] c) According to the signature algorithm specified by signature_type in the security parameter set (currently only SM2 can be used in SVAC2, which is an asymmetric encryption algorithm), the above summary is signed (i.e., the summary is encrypted) with the private key of the security device (such as a security camera, surveillance camera, etc.) to obtain a digital signature (i.e., video authentication data);

[0282] d) Encoding the digital signature using Base64 to represent binary data;

[0283] e) Encapsulate the Base64-encoded digital signature into the authentication data set in an independent NALU form, namely authentication_data_rbsp.

[0284] 4. GB35114 digital signature verification

[0285] a) Get the source device information from the NALU of the security parameter set (which can be identified based on camera_idc and camera_id), and find the public key of the source device (which is different from the private key used for encryption of the source device mentioned above);

[0286] b) According to the signature algorithm specified by signature_type in the NALU of the security parameter set (currently only SM2 can be used in SVAC2, which is an asymmetric encryption algorithm), the digital signature in the NALU of the authentication data set is decrypted with the public key of the source device to obtain the summary generated by the encoding end;

[0287] c) According to successive_hash_pictures_minus1 (the total number of authenticated consecutive images) in the security parameter set and frame_num (the last image number of all authenticated consecutive images) in the authentication data set, the NALU of all authenticated video encoding data can be located;

[0288] d) According to the hash algorithm specified by hash_type in the security parameter set (currently only SM3 can be used in SVAC2), starting from the first authenticated NALU, the data in the NALU of the authenticated video encoding data (including NAL Header and RBSP) is processed to generate a digest of the decoding end as a comparison item;

[0289] e) Compare the digest generated by the decoding end with the digest generated by the encoding end obtained after decrypting the digital signature. If the two are the same, the image corresponding to the above-mentioned authenticated video encoding data passes the verification, indicating that the data is true and complete; if the two are different, the verification fails.

[0290] Security parameter sets in SVAC3

[0291]

[0292]

[0293] in,

[0294] When encryption_flag is equal to 1, it means that encryption of video coding data, SPS, PPS or extended data unit is supported, that is, RBSP in NALU may be encrypted; when encryption_flag is equal to 0, it means that encryption of RBSP in NALU is not supported.

[0295] When authentication_flag is equal to 1, it means that authentication of the entire frame image data content is supported. The NALU for authentication can include video coding data, as well as SPS, PPS, security parameter set and / or extended data unit transmitted in the frame. When authentication of the above data content is supported, the absolute time extension information must be carried in the coded bitstream, and the authentication data carried in the coded bitstream should be Base64 encoded. The authentication data is transmitted through the NALU with nal_unit_type equal to 10. If the image contains video coding data, SPS, PPS, security parameter set, extended data unit, etc. with authentication_idc equal to 1, the NALUs with authentication_idc equal to 1 in an image are arranged in decoding order and authenticated to generate a summary of the image; when it is equal to 0, it means that authentication of the entire frame image data content is not supported, and the coded bitstream should not contain NALUs with nal_unit_type equal to 10.

[0296] If spatial_svc_flag is equal to 1, the base layer image and enhancement layer image of the same frame image are authenticated separately, the PPS and video coding data of the enhancement layer are authenticated according to the enhancement layer, and the SPS, security parameter set and extended data unit are authenticated together with the image parameter set and video coding data of the base layer according to the base layer.

[0297] encryption_type indicates the encryption algorithm used. The corresponding relationship is shown in Table 1.

[0298] Table 1 Correspondence between encryption_type and specific encryption algorithms

[0299] encryption_type Encryption Algorithm 0 SM1 1 SM4 2～15 reserve

[0300] vek_flag equal to 1 means carrying vek; equal to 0 means not carrying vek.

[0301] iv_flag equal to 1 indicates that IV is carried; equal to 0 indicates that IV is not carried.

[0302] camera_id is a 20-byte string indicating the camera ID of the image source.

[0303] vek_encryption_type indicates the algorithm used for key encryption. The specific correspondence is the same as encryption_type.

[0304] evek_length_minus1 is the length of the encrypted key minus 1, in bytes.

[0305] evek is the encrypted key used for encryption calculations, and its length is evek_length_minus1 plus 1 byte.

[0306] vkek_version length_minus1 is the length of the encryption key version number minus 1, in bytes.

[0307] vkek_version is the encryption key version number, and its length is vkek_version_length_minus1 plus 1 byte.

[0308] iv_length_minus1 is the initial vector length minus 1, in bytes.

[0309] iv is the initial vector, used for block encryption, and its length is iv_length_minus1 plus 1 byte.

[0310] hash_type indicates the algorithm used for authentication. The corresponding relationship is shown in Table 2.

[0311] Table 2 Correspondence between hash_type and specific algorithms

[0312] hash_type Authentication algorithm Summary data length byte 0 SM3 32 1～3 reserve reserve

[0313] hash_discard_p_pictures equal to 1 means that non-Instantaneous Decoding Refresh (IDR) images are not authenticated; equal to 0 means that non-IDR images are authenticated. If hash_discard_p_pictures is not in the bitstream, its default value is 1. The authentication_idc of each NALU in the image that is not authenticated should be equal to 0.

[0314] successive_hash_pictures_minus1 plus 1 indicates the number of consecutive pictures that are digitally signed in the decoding order, and these consecutive pictures are limited to one IDR picture interval. The value of successive_hash_pictures_minus1 can be 0-255.

[0315] If successive_hash_pictures_minus1 is greater than 0, first generate a tree summary for the digests of the consecutive SuccessiveHashPictures images in the decoding order, and then digitally sign the tree top summary. Figure 6e ( Figure 6e As shown in the schematic diagram of the tree summary of the embodiment of the present application, the tree-top summary of n images is a summary obtained by processing the tree-top summary of the first n-1 images and the summary of the nth image (for example, the video coding data of the nth image is hashed) according to the algorithm shown in hash_type. For example, for n images, the encoding stream of the first image and the encoding stream of the second image are hashed respectively to obtain the digests of the two, and then the digest of the first image and the digest of the second image are processed according to the algorithm shown in hash_type to obtain the digest, which can be the tree-top summary of the first image and the second image; then the encoding stream of the third image is hashed to obtain the digest of the third image, and the tree-top digests of the first image and the second image and the digest of the third image are processed according to the algorithm shown in hash_type to obtain the digest, which can be the tree-top summary of the first image, the second image and the third image; and so on.

[0316] SuccessiveHashPictures=successive_hash_pictures_minus1+1

[0317] Note: The first image to be authenticated after the security parameter set is activated shall be the first of the consecutive SuccessiveHashPictures images. An IDR image shall be the first of the consecutive SuccessiveHashPictures images. If the number of images to be authenticated in an IDR image interval is less than SuccessiveHashPictures, the digest corresponding to the digital signature shall be the digest of all images contained in the previous IDR image interval.

[0318] If successive_hash_pictures_minus1 is equal to 0, the digest of each image being authenticated is signed.

[0319] signature_type indicates the algorithm for digitally signing the summary data of the image. The corresponding relationship is shown in Table 3.

[0320] Table 3 Correspondence between signature_type and specific signature algorithms

[0321] signature_type Signature Algorithm 0 SM2 1～3 reserve

[0322] camera_idc is a 19-byte string used to indicate the certificate identifier of the camera from which the image is obtained.

[0323] Certification dataset in SVAC3

[0324]

[0325]

[0326] frame_num indicates the image that should contain the authentication data. This image is the image with the same frame_num as the authentication data's frame_num before the authentication data NALU. When successive_hash_pictures_minus1 is equal to 0, frame_num indicates the image corresponding to the authentication data; when it is greater than 0, frame_num indicates the last of the consecutive SuccessiveHashPictures images.

[0327] The spatial_el_flag is equal to 1, indicating that the authentication data is the enhanced layer signature data, and the spactial_el_flag is equal to 0, indicating that the authentication data is the base layer signature data. If spactial_el_flag does not exist in the bitstream, its value is 0 by default.

[0328] authentication_data_length_minus1 plus 1 indicates the length of the signature data in bytes, and the value can be 0-255.

[0329] authentication_data[i] is the i-th byte of the signature data.

[0330] The signature data may be encoded using Base64. For the Base64 encoding method, see Request For Comments (RFC) 3548 of the Internet Communication Protocol.

[0331] In recent years, relevant government administrative departments have successively issued a series of policies and regulations to guide the construction of network monitoring and security systems. On November 1, 2018, the Standardization Administration of China issued the national standard "Technical Requirements for Information Security of Public Security Video Surveillance Networks". As the first technical standard for information security of China's video surveillance networks, it has received great attention from public security organs, scientific research institutions and related enterprises. The standard clearly stipulates that public security video surveillance should use domestic cryptographic technology to implement front-end image acquisition equipment, access users, central servers, etc., and realize identity authentication, video signature, video stream encryption and decryption based on domestic cryptographic technology. Identity authentication is required, video data signature uses the cryptographic hash function standard SM2 national secret algorithm, and video stream encryption uses SM1 and SM4 national secret algorithms.

[0332] Among the above national standards, GB35114 is a highly integrated product of cryptography and video surveillance related technologies. The original video data is directly compressed and encrypted using a password. This encryption method is relatively simple and does not change the encrypted object due to the characteristics of any video data. The encryption process usually occurs after the compression encoding process, that is, the encryption and signing involved in the embodiments of the present application can occur after video encoding, while decryption and signature verification can occur before video decoding.

[0333] In order to ensure that video data will not be stolen or tampered with by others, and to ensure the security of video transmission, the embodiment of the present application provides an authentication method, encryption method and related devices for temporal scalable video coding (TSVC) technology. The technical solution of the present application will be described below.

[0334] Figure 7 Flowchart of process 700 of the authentication method for TSVC technology according to an embodiment of the present application. Process 700 may be performed by video encoder 20 (or encoder). Process 700 is described as a series of steps or operations, and it should be understood that process 700 may be performed in various orders and / or occur simultaneously, not limited to Figure 7 The execution order shown. Process 700 includes the following steps:

[0335] Step 701: Obtain a first summary corresponding to a first level.

[0336] The first level is any one of the multiple levels obtained through time stratification.

[0337] For example, Figure 8 This is a TSVC authentication flow chart of an embodiment of the present application, such as Figure 8 As shown, three levels can be obtained through time stratification, and the three levels are respectively represented as Temporal id 0, Temporal id 1, and Temporal id 2 from low to high, so the first level can refer to any one of the above three levels.

[0338] The first summary is obtained based on the encoded bitstreams of n1 images, where n1≥1; when the first layer is a base layer, the n1 images come from the first layer; when the first layer is an enhancement layer, the n1 images come from the first layer and at least one layer below the first layer.

[0339] like Figure 8 As shown in the figure, the video includes 9 images (POC is 0-8), among which images 0, 4, and 8 are from Temporal id 0; images 2 and 6 are from Temporal id 1; and images 1, 3, 5, and 7 are from Temporal id 2. The arrows in the figure indicate the prediction relationship between images, for example, image 0 is used to predict images 1, 2, and 4; image 2 is used to predict image 3; and so on.

[0340] When the first layer is Temporal id 0 (i.e., the base layer), the n1 images are all from Temporal id 0, including images 0, 4, and 8;

[0341] When the first level is Temporal id 1 (i.e., enhancement layer), n1 pictures come from Temporal id 1 and Temporal id 0, including pictures 0, 4, 8, 2, and 6;

[0342] When the first level is Temporal id 2 (ie, enhancement layer), n1 pictures come from Temporal id 2, Temporal id 1, and Temporal id 0, including pictures 0, 4, 8, 2, 6, 1, 3, 5, and 7.

[0343] It should be noted that, when the first level is Temporal id 2, the n1 images may also come from Temporal id 1 and Temporal id 0, or may also come from Temporal id 2 and Temporal id 0, and this embodiment of the present application does not make any specific limitation on this.

[0344] The encoder may obtain a digest acquisition algorithm according to the security parameter set; and then process the encoded code streams of the n1 images according to the digest acquisition algorithm to obtain a first digest.

[0345] For example, referring to the security parameter set in SVAC3 above, a parameter hash_type is included, which indicates the algorithm used for authentication (i.e., the digest acquisition algorithm mentioned above). As shown in Table 2, when hash_type=0, the algorithm can use SM3, and the digest output by it has 32 bytes; when hash_type=1-3, the algorithm is reserved. Therefore, SM3 can be used. Figure 6e The tree-top digests of n1 images are obtained in the order shown, and it can be seen that the first digest can be the tree-top digest of the coded bitstreams of n1 images. For example, when the first level is Temporal id 0 (i.e., the base layer), the n1 images include images 0, 4, and 8. The coded bitstream of image 0 and the coded bitstream of image 4 are first hashed to obtain their digests, and then the digests of image 0 and image 4 are processed according to SM3 to obtain the digests; then the coded bitstream of image 8 is hashed to obtain the digest of image 8, and the digests of image 0, image 4, and image 8 are processed according to SM3 to obtain the digests, which can be the tree-top digests of image 0, image 4, and image 8, and can be called the first digest of Temporal id 0.

[0346] It should be noted that, in addition to Figure 6e The tree-top digest is obtained in the order shown, and other orders can also be used, such as first obtaining the digests of image No. 4 and image No. 8, and then adding image No. 0. This embodiment of the present application does not specifically limit this. In addition, in addition to the hash algorithm in the example, the above digest acquisition algorithm can also use other algorithms to obtain the digest, which is also not specifically limited in the embodiment of the present application.

[0347] Step 702: Sign the first digest to obtain a first-level digital signature.

[0348] The encoder may obtain a signature algorithm according to the security parameter set; and then sign the first summary according to the signature algorithm to obtain a first-level digital signature.

[0349] For example, referring to the security parameter set in SVAC3 above, a parameter signature_type is included, which indicates the algorithm for digitally signing the summary data of the image (i.e., the above-mentioned signature algorithm). As shown in Table 3, when signature_type=0, the algorithm can adopt SM2; when signature_type=1-3, the algorithm is reserved. Therefore, SM2 can be adopted to encrypt (sign) the first summary with the private key of the security device (such as a security camera, a surveillance camera, etc.) to obtain a digital signature, which comes from the first summary, and the first summary is obtained from the n1 images corresponding to the first level, so the digital signature can be called the digital signature of the first level.

[0350] Step 703: Encapsulate the first-level digital signature into the first-level authentication data set.

[0351] The encoder can encode the first-level digital signature using 64 printable characters to represent binary data (Base64), and then encapsulate the Base64-encoded digital signature into the authentication data set, which can be encapsulated in the form of an independent NALU. Optionally, the NALU of the first-level authentication data set can be placed after the encrypted NALU to form a video stream for transmission.

[0352] The above process is the process of the encoder authenticating the first level. The digital signature obtained in the embodiment of the present application corresponds to the level, that is, if there are N levels, N authentications can be performed, and then N authentication data sets containing digital signatures can be obtained. Based on this, the identification of the first level can be carried in the authentication data set of the first level to distinguish multiple authentication data sets.

[0353] Exemplarily, additional syntax elements are added to the authentication data set in SVAC3 above, as shown below:

[0354]

[0355] temporal_id is a 3-bit unsigned integer indicating the TSVC temporal level to which the authentication data set belongs. The value range of temporal_id can be 0-7.

[0356] The embodiment of the present application integrates the TSVC technology and the GB35114 specification, and performs signature authentication on the TSVC time domain levels respectively, which can effectively cope with various complex and changeable network bandwidth changes to ensure that the encoded code streams at each time domain level can be correctly decrypted and authenticated.

[0357] Fig. 9Flowchart of process 900 of the authentication method for TSVC technology according to an embodiment of the present application. Process 900 may be performed by video decoder 30 (or decoder). Process 900 is described as a series of steps or operations. It should be understood that process 900 may be performed in various orders and / or occur simultaneously, not limited to Fig. 9 The execution order shown. Process 900 includes the following steps:

[0358] Step 901: Obtain a first authentication data set.

[0359] Corresponding to the encoding end, the first authentication data set may include a first digital signature and a layer identifier, and the first authentication data set comes from the received code stream.

[0360] Step 902: Decrypt the first digital signature to obtain a first summary.

[0361] The decoder can obtain the signature algorithm according to the security parameter set; and then decrypt the first digital signature according to the signature algorithm to obtain the first summary.

[0362] The decoder can obtain the source device information from the NALU of the security parameter set (can be identified by camera_idc and camera_id), find the public key of the source device (this public key is different from the private key used for encryption of the source device mentioned above). According to the signature algorithm specified by signature_type in the NALU of the security parameter set (currently only SM2 can be used in SVAC2, which is an asymmetric encryption algorithm), the first digital signature in the NALU of the authentication data set is decrypted with the public key of the source device to obtain the first summary, which is the summary generated by the encoding end.

[0363] Step 903: Obtain a second summary according to the encoded code streams of n1 images.

[0364] The method for the decoder to obtain the second summary can refer to Figure 7 In step 701 of the illustrated embodiment, the layer identifier in the first authentication data set indicates the layer currently being processed, and n1 images correspond to the aforementioned layers. When the first layer corresponding to the layer identifier is a basic layer, the n1 images come from the first layer; when the first layer is an enhanced layer, the n1 images come from the first layer and at least one layer below the first layer.

[0365] The decoder may obtain a digest acquisition algorithm according to the security parameter set; and then process the encoded code streams of the n1 images according to the digest acquisition algorithm to obtain a second digest.

[0366] The above process can refer to Figure 8 The embodiments shown will not be described in detail here.

[0367] Step 904: When the first digest and the second digest are the same, it is determined that n1 images have passed authentication.

[0368] Compare the first digest generated by the encoder and the second digest generated by the decoder using the same method. If the two are the same, it means that n1 images have passed authentication.

[0369] The embodiment of the present application integrates the TSVC technology and the GB35114 specification, and performs signature authentication on the TSVC time domain levels respectively, which can effectively cope with various complex and changeable network bandwidth changes to ensure that the encoded code streams at each time domain level can be correctly decrypted and authenticated.

[0370] Fig.10 1 is a flowchart of a process 1000 of an authentication method for TSVC technology according to an embodiment of the present application. The process 1000 may be performed by the video encoder 20 (or encoder). The process 1000 is described as a series of steps or operations. It should be understood that the process 1000 may be performed in various orders and / or may occur simultaneously, not limited to Fig.10 The execution order shown. Process 1000 includes the following steps:

[0371] Step 1001: Obtain third summaries of multiple levels.

[0372] Multiple levels are obtained through time stratification.

[0373] For example, Fig.11 This is a TSVC authentication flow chart of an embodiment of the present application, such as Fig.11 As shown, three levels can be obtained after time stratification, and the three levels are respectively represented as Temporal id 0, Temporal id 1, and Temporal id 2 from low to high, so the multiple levels can refer to the aforementioned three levels.

[0374] The third summary is obtained based on the encoded bitstreams of n2 images, n2≥1; the n2 images come from multiple levels.

[0375] like Fig.11 As shown in the figure, the video includes 9 images (POC is 0-8), among which images 0, 4, and 8 are from Temporal id 0; images 2 and 6 are from Temporal id 1; and images 1, 3, 5, and 7 are from Temporal id 2. The arrows in the figure indicate the prediction relationship between images, for example, image 0 is used to predict images 1, 2, and 4; image 2 is used to predict image 3; and so on.

[0376] The n2 images are all images from the three levels, including images 0-8.

[0377] The encoder may obtain a digest acquisition algorithm according to the security parameter set; and then process the encoded code streams of the n2 images according to the digest acquisition algorithm to obtain a third digest.

[0378] For example, referring to the security parameter set in SVAC3 above, a parameter hash_type is included, which indicates the algorithm used for authentication (i.e., the digest acquisition algorithm mentioned above). As shown in Table 2, when hash_type=0, the algorithm can use SM3, and the digest output by it has 32 bytes; when hash_type=1-3, the algorithm is reserved. Therefore, SM3 can be used. Figure 6e The tree-top digests of n2 images are obtained in the order shown, and it can be seen that the third digest can be the tree-top digest of the coded code streams of n2 images. For example, the coded code stream of image No. 0 and the coded code stream of image No. 4 are hashed to obtain their digests, and then the digest of image No. 0 and the digest of image No. 4 are processed according to SM3 to obtain the digests; then the coded code stream of image No. 8 is hashed to obtain the digest of image No. 8, and the digests of image No. 0, image No. 4 and image No. 8 are processed according to SM3 to obtain the digests, which can be the tree-top digests of image No. 0, image No. 4 and image No. 8; and so on, until all 9 images participate in the calculation of the digest, and the final digest is the tree-top digest of the 9 images, which can be called the third digest. In addition, in addition to the hash algorithm in the example, the above-mentioned summary acquisition algorithm can also use other algorithms to obtain the digest, and the embodiments of the present application do not make specific limitations on this.

[0379] It should be noted that, in addition to Figure 6e The tree-top summary is obtained in the order shown, and other orders may also be used, for example, obtaining the summaries of image No. 4 and image No. 8 first, and then adding image No. 0. This embodiment of the present application does not make specific limitations on this.

[0380] Step 1002: Sign the third digest to obtain multiple levels of digital signatures.

[0381] The encoder may obtain a signature algorithm according to the security parameter set; and then sign the third digest according to the signature algorithm to obtain a multi-level digital signature.

[0382] Exemplarily, referring to the security parameter set in SVAC3 above, a parameter signature_type is included, which indicates the algorithm for digitally signing the summary data of the image (i.e., the above-mentioned signature algorithm). As shown in Table 3, when signature_type=0, the algorithm can adopt SM2; when signature_type=1-3, the algorithm is reserved. Therefore, SM2 can be adopted, and the third summary can be encrypted (signed) with the private key of the security device (such as a security camera, a surveillance camera, etc.) to obtain a digital signature. The digital signature comes from the third summary, and the third summary is obtained from n2 images corresponding to all levels, so the digital signature can be called a digital signature of multiple levels.

[0383] Step 1003: Encapsulate multiple levels of digital signatures into an authentication data set.

[0384] The encoder can encode multiple levels of digital signatures using 64 printable characters to represent binary data (Base64), and then encapsulate the Base64-encoded digital signature into the authentication data set, which can be encapsulated in the form of an independent NALU. Optionally, the NALUs of multiple levels of authentication data sets can be placed after the encrypted NALU to form a video stream for transmission.

[0385] The embodiment of the present application does not distinguish between levels when obtaining the third summary. All levels correspond to only one summary, thereby obtaining one authentication data set. Therefore, there is no need for a level identifier to distinguish the authentication data set.

[0386] The embodiment of the present application integrates the TSVC technology and the GB35114 specification, and does not need to change the existing SVAC3 security parameter set, so that all TSVC time domain code streams can be more conveniently encrypted and authenticated uniformly.

[0387] Fig.12 1 is a flowchart of a process 1200 of an authentication method for TSVC technology according to an embodiment of the present application. The process 1200 may be performed by the video decoder 30 (or decoder). The process 1200 is described as a series of steps or operations. It should be understood that the process 1200 may be performed in various orders and / or may occur simultaneously, not limited to the following. Fig.12 The execution order shown. Process 1200 includes the following steps:

[0388] Step 1201: Obtain the authentication data set.

[0389] Corresponding to the encoding end, the authentication data set may include a digital signature, and the authentication data set comes from the received code stream.

[0390] Step 1202: Decrypt the digital signature to obtain a third summary.

[0391] The decoder can obtain the signature algorithm according to the security parameter set; and then decrypt the digital signature according to the signature algorithm to obtain the third summary.

[0392] The decoder can obtain the source device information from the NALU of the security parameter set (can be identified by camera_idc and camera_id), find the public key of the source device (this public key is different from the private key used for encryption of the source device mentioned above). According to the signature algorithm specified by signature_type in the NALU of the security parameter set (currently only SM2 can be used in SVAC2, which is an asymmetric encryption algorithm), the public key of the source device is used to decrypt the digital signature in the NALU of the authentication data set to obtain the third summary, which is the summary generated by the encoding end.

[0393] Step 1203: Obtain a fourth summary according to the encoded code streams of the n2 images.

[0394] The method for the decoder to obtain the fourth summary can refer to Fig.10 In step 1001 of the illustrated embodiment, n2 images correspond to a plurality of levels obtained through temporal stratification.

[0395] The decoder may obtain a digest acquisition algorithm according to the security parameter set; and then process the encoded code streams of the n2 images according to the digest acquisition algorithm to obtain a fourth digest.

[0396] The above process can refer to Fig.11 The embodiments shown will not be described in detail here.

[0397] Step 1204: When the third digest and the fourth digest are the same, it is determined that n2 images have passed authentication.

[0398] Compare the third digest generated by the encoder and the fourth digest generated by the decoder using the same method. If the two are the same, it means that n2 images have passed authentication.

[0399] The embodiment of the present application integrates the TSVC technology and the GB35114 specification, and does not need to change the existing SVAC3 security parameter set, so that all TSVC time domain code streams can be more conveniently encrypted and authenticated uniformly.

[0400] It should be noted that the above Figure 7-Figure 12 The authentication method for the TSVC technology in the illustrated embodiment may refer to the video signature / verification process of GB35114 above.

[0401] Fig.131300 is a flowchart of a process 1300 of an encryption method for TSVC technology according to an embodiment of the present application. Process 1300 may be performed by video encoder 20 (or encoder). Process 1300 is described as a series of steps or operations, and it should be understood that process 1300 may be performed in various orders and / or occur simultaneously, not limited to Fig.13 The execution order shown. Process 1300 includes the following steps:

[0402] Step 1301: Get the current NALU.

[0403] The object to be encrypted in the embodiment of the present application may be the RBSP of the NALU including the SPS, PPS or multiple levels of video coding data obtained through time stratification. The Header of the NALU does not need to be encrypted.

[0404] Step 1302: Obtain VEK according to the security parameter set.

[0405] The encoder can periodically randomly generate a 16-byte VEK and a 16-byte IV.

[0406] Step 1303: Encrypt the RBSP according to the VEK to obtain the ciphertext of the current NALU.

[0407] The encoder uses the output feedback OFB mode of the agreed block encryption algorithm to generate a key stream using VEK and IV. The key stream is bit-aligned with the RBSP of the current NALU and an XOR operation is performed to obtain the encrypted RBSP.

[0408] Step 1304: Transmit the ciphertext of the NALU.

[0409] Optionally, the encoder encrypts the above VEK and IV with the VKEK of the source end, and encapsulates the corresponding ciphertext into the NALU of the security parameter set.

[0410] The embodiment of the present application integrates the TSVC technology and the GB35114 specification to implement encryption processing of the time-domain layered RBSP of TSVC.

[0411] Fig.14 1400 is a flowchart of a decryption method for TSVC technology according to an embodiment of the present application. Process 1400 may be performed by video decoder 30 (or decoder). Process 1400 is described as a series of steps or operations. It should be understood that process 1400 may be performed in various orders and / or occur simultaneously, not limited to Fig.14 The execution order shown. Process 1400 includes the following steps:

[0412] Step 1401: Get the ciphertext of the current NALU.

[0413] Corresponding to the encoding end, the RBSP of the NALU carried in the code stream received by the decoder is encrypted, so what is obtained is the ciphertext of the current NALU.

[0414] Step 1402: Obtain VEK.

[0415] The decoder obtains the ciphertext of IV and VEK from the NALU of the security parameter set, denoted as EVEK, and decrypts EVEK with VKEK to obtain VEK and IV.

[0416] Step 1403: Decrypt the ciphertext of the current NALU according to the VEK to obtain the RBSP of the current NALU.

[0417] The decoder uses the output feedback OFB mode of the agreed block encryption algorithm and generates a key stream using VEK and IV. The key stream is XORed with the ciphertext of the current NALU bit by bit to obtain the decrypted RBSP.

[0418] Step 1404: Obtain a reconstructed image according to the RBSP of the current NALU.

[0419] The decompression process according to the RBSP may refer to the above embodiment, which will not be described in detail.

[0420] The embodiment of the present application integrates the TSVC technology and the GB35114 specification to implement the decryption processing of the time-domain layered RBSP of TSVC.

[0421] For example, Fig.15 This is the encryption / decryption flow chart of the embodiment of the present application, such as Fig.15 As shown, the embodiment of the present application can encrypt the data streams of all time domain layers (only RBSP) uniformly. Therefore, the data streams of all time domain layers can maintain one key and one set of encryption parameters, so that each time domain layer does not need to be encrypted separately.

[0422] On the encoding side, perform the encryption operation:

[0423] a) The encoder outputs video bitstreams with multiple temporal layers.

[0424] b) Parse the security parameter set in the video stream and obtain the VEK therein.

[0425] c) According to the provisions of GB35114 standard, the parsed VEK is used to encrypt the RBSP data (including SPS, PPS and video coding data) of all time domain layers except the security parameter set in sequence.

[0426] d) After encryption, the corresponding ciphertext is obtained and then transmitted.

[0427] On the decoding side, perform the decryption operation:

[0428] a) The decoder receives the ciphertext code stream and parses the security parameter set in the ciphertext code stream to obtain the VEK.

[0429] b) Read all encrypted RBSP data except the security parameter set in the corresponding time domain layer in the code stream, and use the parsed VEK to decrypt all ciphertexts in turn according to the provisions of the GB35114 standard.

[0430] c) After decryption, the original plaintext code stream corresponding to the time domain layer is obtained.

[0431] d) The decoder obtains the reconstructed image based on the plaintext code stream.

[0432] The encoding device 1600 can be used in a video encoder 20 (or encoder), and includes: an acquisition module 1601, a signature module 1602, an encryption module 1603, a packaging module 1604 and a transmission module 1605.

[0433] An acquisition module 1601 is used to acquire a first summary corresponding to a first level, where the first level is any one of multiple levels obtained through time stratification; the first summary is obtained based on a coded bitstream of n1 images, where n1≥1; when the first level is a base layer, the n1 images come from the first level; when the first level is an enhancement layer, the n1 images come from the first level and at least one level below the first level; a signature module 1602 is used to sign the first summary to obtain a digital signature of the first level; an encapsulation module 1604 is used to encapsulate the digital signature of the first level into an authentication data set of the first level.

[0434] In a possible implementation, the acquisition module 1601 is specifically configured to acquire a digest acquisition algorithm according to a security parameter set; and process the encoded code streams of the n1 images according to the digest acquisition algorithm to obtain the first digest.

[0435] In a possible implementation, the signature module 1602 is specifically configured to obtain a signature algorithm according to a security parameter set; and sign the first summary according to the signature algorithm to obtain a digital signature of the first level.

[0436] In a possible implementation manner, the first-level authentication data set also includes an identifier of the first level.

[0437] In a possible implementation, an acquisition module 1601 is used to obtain a third summary of multiple levels, where the multiple levels are obtained through time stratification; the third summary is obtained based on the encoded code stream of n2 images, n2≥1; the n2 images are from the multiple levels; a signature module 1602 is used to sign the third summary to obtain digital signatures of the multiple levels; and an encapsulation module 1604 is used to encapsulate the digital signatures of the multiple levels into an authentication data set.

[0438] In a possible implementation, the acquisition module 1601 is specifically configured to acquire a digest acquisition algorithm according to a security parameter set; and process the encoded code streams of the n2 images according to the digest acquisition algorithm to obtain the third digest.

[0439] In a possible implementation, the signature module 1602 is specifically configured to obtain a signature algorithm according to a security parameter set; and sign the third digest according to the signature algorithm to obtain the digital signatures of the multiple levels.

[0440] In one possible implementation, an acquisition module 1601 is used to obtain a current network abstraction layer unit NALU, wherein the original byte sequence payload RBSP of the current NALU includes a sequence parameter set SPS, an image parameter set PPS, or multiple levels of video encoding data obtained through time stratification; a video encryption key VEK is obtained according to a security parameter set; an encryption module 1603 is used to encrypt the RBSP according to the VEK to obtain a ciphertext of the current NALU; and a transmission module 1605 is used to transmit the ciphertext of the NALU.

[0441] In a possible implementation, the encryption module 1603 is specifically configured to generate a key stream according to the VEK; and perform an XOR operation on the key stream and the RBSP to obtain the ciphertext.

[0442] In a possible implementation, the encryption module 1603 is further configured to encrypt the VEK to obtain an encrypted VEK.

[0443] The device of this embodiment can be used to perform Figure 7 , 10 , The technical solution of the method embodiment shown in 13 has similar implementation principles and technical effects, which will not be repeated here.

[0444] Fig.17 1700 is a schematic diagram of the structure of a decoding device 1700 according to an embodiment of the present application. The decoding device 1700 can be used in a video decoder 30 (or decoder), and includes: an acquisition module 1701, a decryption module 1702, a verification module 1703 and a reconstruction module 1704.

[0445] The acquisition module 1701 is used to obtain a first authentication data set, the first authentication data set includes a first digital signature and a layer identifier, and the first authentication data set comes from a received code stream; the decryption module 1702 is used to decrypt the first digital signature to obtain a first summary; the acquisition module 1701 is also used to obtain a second summary based on the encoded code stream of n1 images, n1≥1; when the first layer corresponding to the layer identifier is the basic layer, the n1 images come from the first layer; when the first layer is the enhanced layer, the n1 images come from the first layer and at least one layer below the first layer; the verification module 1703 is used to determine that the n1 images have passed the authentication when the first summary and the second summary are the same.

[0446] In a possible implementation, the decryption module 1702 is specifically configured to obtain a signature algorithm according to a security parameter set; and decrypt the first digital signature according to the signature algorithm to obtain the first summary.

[0447] In a possible implementation, the acquisition module 1701 is specifically configured to acquire a digest acquisition algorithm according to a security parameter set; and process the encoded code streams of the n1 images according to the digest acquisition algorithm to obtain the second digest.

[0448] In a possible implementation, an acquisition module 1701 is used to acquire an authentication data set, the authentication data set includes a digital signature, and the authentication data set comes from a received code stream; a decryption module 1702 is used to decrypt the digital signature to obtain a third summary; the acquisition module 1701 is also used to acquire a fourth summary based on the encoded code streams of n2 images, n2 ≥ 1; the n2 images come from multiple levels obtained by time stratification; and a verification module 1703 is used to determine that the n2 images have passed authentication when the third summary and the fourth summary are the same.

[0449] In a possible implementation, the decryption module 1702 is specifically configured to obtain a signature algorithm according to a security parameter set; and decrypt the digital signature according to the signature algorithm to obtain the third summary.

[0450] In a possible implementation, the acquisition module 1701 is specifically configured to acquire a digest acquisition algorithm according to a security parameter set; and process the encoded code streams of the n2 images according to the digest acquisition algorithm to obtain the fourth digest.

[0451] In one possible implementation, an acquisition module 1701 is used to obtain the ciphertext of the current network abstraction layer unit NALU; obtain a video encryption key VEK; a decryption module 1702 is used to decrypt the ciphertext of the current NALU according to the VEK to obtain the original byte sequence payload RBSP of the current NALU; and a reconstruction module 1704 is used to obtain a reconstructed image according to the RBSP of the current NALU.

[0452] In a possible implementation, the decryption module 1702 is specifically configured to generate a key stream according to the VEK; and perform an XOR operation on the key stream and the ciphertext to obtain the RBSP.

[0453] The device of this embodiment can be used to perform Fig. 9 , 12 , The technical solution of the method embodiment shown in 14 has similar implementation principles and technical effects, which will not be repeated here.

[0454] In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The processor can be a general-purpose processor, a digital signal processor (digital signal processor, DSP), an application-specific integrated circuit (application-specific integrated circuit, ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware coding processor to be executed, or the hardware and software modules in the coding processor are combined to be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0455] The memory mentioned in the above embodiments may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0456] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0457] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0458] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0459] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0460] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0461] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0462] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An authentication method for TSVC technology, characterized in that: include: Obtaining a first summary corresponding to a first level, where the first level is any one of a plurality of levels obtained through time stratification; The first summary is obtained according to the coded bitstreams of n1 images, where n1≥1; when the first layer is a base layer, the n1 images are from the first layer; when the first layer is an enhancement layer, the n1 images are from the first layer and at least one layer below the first layer; Signing the first digest to obtain a digital signature at the first level; The digital signature of the first level is encapsulated into the authentication data set of the first level.

2. The method according to claim 1, characterized in that The obtaining of a first summary corresponding to the first level includes: Obtaining a digest acquisition algorithm according to a security parameter set; The encoded code streams of the n1 images are processed according to the summary acquisition algorithm to obtain the first summary.

3. The method according to claim 1 or 2, characterized in that: The step of signing the first summary to obtain the digital signature of the first level includes: Obtain the signature algorithm according to the security parameter set; The first digest is signed according to the signature algorithm to obtain a digital signature of the first level.

4. The method according to any one of claims 1 to 3, characterized in that The first-level authentication data set also includes an identification of the first level.

5. An authentication method for TSVC technology, characterized in that: include: Acquire a first authentication data set, the first authentication data set including a first digital signature and a layer identifier, the first authentication data set coming from a received code stream; Decrypting the first digital signature to obtain a first summary; Obtaining a second summary according to the coded bitstreams of n1 images, where n1≥1; when a first layer corresponding to the layer identifier is a base layer, the n1 images are from the first layer; when the first layer is an enhancement layer, the n1 images are from the first layer and at least one layer below the first layer; When the first digest and the second digest are the same, it is determined that the n1 images have passed authentication.

6. The method according to claim 5, characterized in that The decrypting the first digital signature to obtain a first summary includes: Obtain the signature algorithm according to the security parameter set; The first digital signature is decrypted according to the signature algorithm to obtain the first summary.

7. The method according to claim 5 or 6, characterized in that: The step of obtaining a second summary according to the encoded code streams of the n1 images includes: Obtaining a digest acquisition algorithm according to a security parameter set; The encoded code streams of the n1 images are processed according to the summary acquisition algorithm to obtain the second summary.

8. An authentication method for TSVC technology, characterized in that: include: Obtaining a third summary of a plurality of levels, wherein the plurality of levels are obtained by time stratification; The third summary is obtained based on the encoded bitstreams of n2 images, n2≥1; the n2 images are from the multiple levels; Signing the third digest to obtain the multiple levels of digital signatures; The digital signatures of the multiple levels are encapsulated into an authentication data set.

9. The method according to claim 8, characterized in that The obtaining of the third summaries of multiple levels includes: Obtaining a digest acquisition algorithm according to a security parameter set; The encoded code streams of the n2 images are processed according to the summary acquisition algorithm to obtain the third summary.

10. The method according to claim 8 or 9, characterized in that: The step of signing the third digest to obtain the multiple levels of digital signatures includes: Obtain the signature algorithm according to the security parameter set; The third digest is signed according to the signature algorithm to obtain the multiple levels of digital signatures.

11. An authentication method for TSVC technology, characterized in that: include: Acquire an authentication data set, the authentication data set including a digital signature, the authentication data set coming from a received code stream; decrypting the digital signature to obtain a third digest; Obtaining a fourth summary according to the coded bitstreams of n2 images, where n2≥1; the n2 images are from a plurality of levels obtained by time stratification; When the third digest and the fourth digest are identical, it is determined that the n2 images have passed authentication.

12. The method according to claim 11, characterized in that The decrypting the digital signature to obtain a third summary comprises: Obtain the signature algorithm according to the security parameter set; The digital signature is decrypted according to the signature algorithm to obtain the third digest.

13. The method according to claim 11 or 12, characterized in that: The step of obtaining a fourth summary according to the encoded code streams of the n2 images includes: Obtaining a digest acquisition algorithm according to a security parameter set; The encoded code streams of the n2 images are processed according to the summary acquisition algorithm to obtain the fourth summary.

14. An encryption method for TSVC technology, characterized in that: include: Obtain a current network abstraction layer unit NALU, wherein the original byte sequence payload RBSP of the current NALU includes a sequence parameter set SPS, a picture parameter set PPS, or multiple levels of video coding data obtained through time stratification; Obtain a video encryption key VEK according to a security parameter set; Encrypting the RBSP according to the VEK to obtain the ciphertext of the current NALU; The ciphertext of the NALU is transmitted.

15. The method according to claim 14, characterized in that The encrypting the RBSP according to the VEK to obtain the ciphertext of the current NALU includes: Generate a key stream according to the VEK; An XOR operation is performed on the key stream and the RBSP to obtain the ciphertext.

16. The method according to claim 14 or 15, characterized in that Also includes: The VEK is encrypted to obtain an encrypted VEK.

17. A decryption method for TSVC technology, characterized in that: include: Get the ciphertext of the current network abstraction layer unit NALU; Get the video encryption key VEK; Decrypting the ciphertext of the current NALU according to the VEK to obtain the original byte sequence payload RBSP of the current NALU; A reconstructed image is obtained according to the RBSP of the current NALU.

18. The method according to claim 17, characterized in that The decrypting the ciphertext of the current NALU according to the VEK to obtain the original byte sequence payload RBSP of the current NALU includes: Generate a key stream according to the VEK; An XOR operation is performed on the key stream and the ciphertext to obtain the RBSP.

19. An authentication device for TSVC technology, characterized in that: include: An acquisition module, configured to acquire a first summary corresponding to a first level, wherein the first level is any one of a plurality of levels obtained through time stratification; The first summary is obtained according to the coded bitstreams of n1 images, where n1≥1; when the first layer is a base layer, the n1 images are from the first layer; when the first layer is an enhancement layer, the n1 images are from the first layer and at least one layer below the first layer; A signing module, configured to sign the first summary to obtain a digital signature of the first level; The encapsulation module is used to encapsulate the digital signature of the first layer into the authentication data set of the first layer.

20. The device according to claim 19, characterized in that The acquisition module is specifically configured to acquire a digest acquisition algorithm according to a security parameter set; and process the encoded code streams of the n1 images according to the digest acquisition algorithm to obtain the first digest.

21. The device according to claim 19 or 20, characterized in that The signature module is specifically configured to obtain a signature algorithm according to a security parameter set; and sign the first summary according to the signature algorithm to obtain a digital signature of the first level.

22. The device according to any one of claims 19 to 21, characterized in that The first-level authentication data set also includes an identification of the first level.

23. An authentication device for TSVC technology, characterized in that: include: An acquisition module, configured to acquire a first authentication data set, wherein the first authentication data set includes a first digital signature and a layer identifier, and the first authentication data set comes from a received code stream; A decryption module, configured to decrypt the first digital signature to obtain a first summary; The acquisition module is further configured to acquire a second summary according to the coded bitstreams of n1 images, where n1≥1; when the first layer corresponding to the layer identifier is a base layer, the n1 images are from the first layer; when the first layer is an enhancement layer, the n1 images are from the first layer and at least one layer below the first layer; A verification module is used to determine that the n1 images have passed authentication when the first digest and the second digest are the same.

24. The device according to claim 23, characterized in that The decryption module is specifically configured to obtain a signature algorithm according to a security parameter set; and decrypt the first digital signature according to the signature algorithm to obtain the first summary.

25. The device according to claim 23 or 24, characterized in that The acquisition module is specifically configured to acquire a digest acquisition algorithm according to a security parameter set; and process the encoded code streams of the n1 images according to the digest acquisition algorithm to obtain the second digest.

26. An authentication device for TSVC technology, characterized in that: include: An acquisition module, used for acquiring a third summary of multiple levels, wherein the multiple levels are obtained by time stratification; The third summary is obtained based on the encoded bitstreams of n2 images, n2≥1; the n2 images are from the multiple levels; A signing module, used for signing the third summary to obtain the digital signatures of the multiple levels; The encapsulation module is used to encapsulate the digital signatures of the multiple levels into an authentication data set.

27. The device according to claim 26, characterized in that The acquisition module is specifically configured to acquire a summary acquisition algorithm according to a security parameter set; and process the encoded code streams of the n2 images according to the summary acquisition algorithm to obtain the third summary.

28. The device according to claim 26 or 27, characterized in that The signature module is specifically configured to obtain a signature algorithm according to a security parameter set; and sign the third summary according to the signature algorithm to obtain the digital signatures of the multiple levels.

29. An authentication device for TSVC technology, characterized in that: include: An acquisition module, used to acquire an authentication data set, the authentication data set including a digital signature, the authentication data set coming from a received code stream; A decryption module, used for decrypting the digital signature to obtain a third summary; The acquisition module is further used to acquire a fourth summary according to the coded code streams of n2 images, where n2≥1; the n2 images are from multiple levels obtained through time stratification; The verification module is used to determine that the n2 images have passed the authentication when the third digest and the fourth digest are the same.

30. The device according to claim 29, characterized in that The decryption module is specifically configured to obtain a signature algorithm according to a security parameter set; and decrypt the digital signature according to the signature algorithm to obtain the third summary.

31. The device according to claim 29 or 30, characterized in that The acquisition module is specifically configured to acquire a summary acquisition algorithm according to a security parameter set; and process the encoded code streams of the n2 images according to the summary acquisition algorithm to obtain the fourth summary.

32. An encryption device for TSVC technology, characterized in that: include: An acquisition module, used to acquire a current network abstraction layer unit NALU, wherein the original byte sequence payload RBSP of the current NALU includes a sequence parameter set SPS, a picture parameter set PPS, or multiple levels of video coding data obtained through time stratification; Obtain a video encryption key VEK according to a security parameter set; An encryption module, used for encrypting the RBSP according to the VEK to obtain the ciphertext of the current NALU; The transmission module is used to transmit the ciphertext of the NALU.

33. The device according to claim 32, characterized in that The encryption module is specifically used to generate a key stream according to the VEK; and perform an XOR operation on the key stream and the RBSP to obtain the ciphertext.

34. The device according to claim 32 or 33, characterized in that The encryption module is also used to encrypt the VEK to obtain an encrypted VEK.

35. A decryption device for TSVC technology, characterized in that: include: The acquisition module is used to obtain the ciphertext of the current network abstraction layer unit NALU; obtain the video encryption key VEK; A decryption module, used for decrypting the ciphertext of the current NALU according to the VEK to obtain the original byte sequence payload RBSP of the current NALU; The reconstruction module is used to obtain a reconstructed image according to the RBSP of the current NALU.

36. The device according to claim 35, characterized in that The decryption module is specifically used to generate a key stream according to the VEK; and perform an XOR operation on the key stream and the ciphertext to obtain the RBSP.

37. An encoder, characterized in that include: one or more processors; A non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the encoder to perform the method according to any one of claims 1-4, 8-10, 14-16.

38. A decoder, characterized in that include: one or more processors; A non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the decoder to perform the method according to any one of claims 5-7, 11-13, 17-18.

39. A non-transitory computer-readable storage medium, characterized in that: The method comprises a program code which, when executed by a computer device, is used to perform the method according to any one of claims 1 to 18.

Images