Video encoding and decoding method and apparatus based on intelligent image encoding and decoding

Abstract

This invention discloses a video encoding and decoding method and apparatus based on intelligent image encoding and decoding. The method uses intelligent image encoding and decoding to encode and decode a portion of images in the video, such as keyframes, and puts the reconstruction results of these images into a decoded image buffer as reference images for subsequent images. At the same time, the method uses traditional video encoding and decoding methods, such as VVC, to encode and decode another portion of images in the video, such as non-keyframes. The encoding and decoding of these images can refer to the reconstructed images obtained by intelligent image encoding and decoding.

Description

Technical Field

[0001] This invention relates to the field of video encoding and decoding technology, and more specifically, to a video encoding and decoding method and apparatus based on intelligent image encoding and decoding. Background Technology

[0002] Existing image and video encoding and decoding methods include traditional methods and neural network-based intelligent methods. Traditional methods eliminate redundant information in images and videos through operations such as prediction, transformation, quantization, entropy coding, and loop filtering. Intelligent methods, on the other hand, utilize neural networks to transform images and videos into feature tensors and then perform downsampling, quantization, and entropy coding to achieve compression encoding.

[0003] Image intelligent encoding and decoding methods have proposed many efficient neural network structures for image feature extraction. Convolutional Neural Networks (CNNs) were the earliest network structure used for image encoding and decoding. Based on CNNs, many improved network structures and probability estimation models have been derived. For example, network structures such as Generative Adversarial Networks (GANs) and Recurrent Neural Networks (RNNs) have significantly improved the end-to-end image compression performance based on neural networks. Among them, image encoding and decoding methods based on Generative Adversarial Networks (GANs) have achieved significant results in improving the subjective quality of images.

[0004] Intelligent video encoding and decoding methods mainly focus on three aspects: 1) hybrid neural network coding, 2) neural network rate-distortion optimization coding, and 3) end-to-end video coding. Hybrid neural network coding embeds neural networks into the video framework, replacing traditional coding modules. It is commonly used for inter-frame prediction, loop filtering, and entropy coding modules. Neural network rate-distortion optimization coding leverages the highly nonlinear characteristics of neural networks to train them into efficient discriminators and classifiers, for example, in the video coding mode decision-making stage. End-to-end video coding currently generally involves either replacing all modules of traditional coding methods with CNNs or expanding the input dimension of the neural network to perform end-to-end compression on all frames.

[0005] In image or video encoding / decoding methods using end-to-end neural networks, a common operation is to first extract features from the image or video, then encode and decode them, and finally restore them to the original image. For example... Figure 1 As shown, the process of extracting the feature tensor is E1, and the process of encoding the feature tensor obtained from E1 to obtain the bitstream is E2. The encoding method consists of E1 and E2. Correspondingly, the decoding method consists of D1 and D2, where D1 refers to the process of transforming the feature tensor into a decoded image, and D2 refers to the process of decoding the bitstream into a feature tensor.

[0006] After encoding and decoding, images and videos are not only used for human viewing in current mainstream applications, but are also frequently used to complete machine intelligence tasks. Intelligent task networks analyze images or videos to complete tasks such as object detection, object tracking, or behavior recognition.

[0007] Existing traditional video coding is typically designed for pixel-perfect fidelity and cannot effectively perform machine intelligence tasks. In contrast, neural network-based intelligent image coding and decoding can be trained to achieve higher accuracy for machine intelligence tasks, thus surpassing traditional image coding methods. However, current neural network-based intelligent video coding and decoding methods are not yet good enough to surpass the latest traditional video coding methods such as H.266 / VVC. This is mainly because neural network-based methods cannot fully eliminate temporal redundancy.

[0008] The technology disclosed in JVET-AA0063 replaces the traditional video coding method with the LIC encoding / decoding method for keyframes, achieving higher coding efficiency than VVC. However, this technology is still optimized for pixel fidelity and human visual perception, and cannot effectively address application scenarios involving machine intelligence tasks. Furthermore, this technology requires modification of the VVC standard and is not compatible with it. Summary of the Invention

[0009] To address the shortcomings of existing technologies, this invention proposes a video encoding and decoding method and apparatus based on intelligent image coding. This method uses intelligent image coding to encode and decode a portion of images in a video, such as keyframes, and places the reconstructed results of these images into a decoded image buffer as reference images for subsequent images. Simultaneously, this method uses traditional video encoding and decoding methods, such as VVC, to encode and decode another portion of images in the video, such as non-keyframes. The encoding and decoding of these images can refer to the reconstructed images obtained by intelligent image coding.

[0010] Therefore, the first objective of this invention is to propose a video decoding method based on intelligent image decoding, which performs the following operations for any image data to be decoded in the input video bitstream:

[0011] At least one stripe data is obtained from the image data to be decoded, and the type information of the stripe data is obtained. The type information is used to identify whether the stripe data is decoded using an intelligent image decoding network or a traditional video decoding operation.

[0012] If the type information of the strip data indicates that the strip data is decoded using an intelligent image decoding network, then the strip data is decoded using the intelligent image decoding network to obtain a reconstructed image of the image data to be decoded; the reconstructed image is placed in the decoded image buffer and marked as a short-term reference image;

[0013] If the type information of the strip data indicates that the strip data is decoded using conventional video decoding operations, then the strip data is decoded using conventional video decoding operations to obtain the reconstructed image of the image data to be decoded, and the reconstructed image is managed according to the reference image management method of conventional video decoding.

[0014] The reconstructed image obtained by the intelligent image decoding network and the reconstructed image obtained by the traditional video decoding operation are combined in the display order to obtain the output video.

[0015] Furthermore, the type information of the strip data is present in the header data of the strip data, and the type information also identifies whether the strip data is decoded using an intelligent intra-frame decoding network or an intelligent inter-frame decoding network.

[0016] Furthermore, the method also includes:

[0017] Obtain the sequence-level parameter set and image-level parameter set corresponding to the image data to be decoded from the input video bitstream;

[0018] Sequence-level intelligent image decoding extension information is obtained from the sequence-level parameter set. This information is used to represent a candidate list of network structures or network models of the intelligent image decoding network used by the strip data in the sequence to which the sequence-level parameter set belongs.

[0019] Image-level intelligent image decoding extension information is obtained from the image-level parameter set. This information is used to represent the network structure or network model of the intelligent image decoding network used for the strip data in several images to which the image-level parameter set belongs.

[0020] Furthermore, the striped data is contained in a network digest layer packet unit defined by a conventional video encoding and decoding method, and the syntax information carried in the header of the packet unit is defined by the conventional video encoding and decoding method.

[0021] The second objective of this invention is to propose a video encoding method based on intelligent image encoding, which performs the following operations for any image to be encoded in the input video:

[0022] The image to be encoded is encoded using one of the following two encoding methods:

[0023] 1) The image to be encoded is encoded using an intelligent image coding network to obtain image coding data, and the reconstructed image is placed in the decoded image buffer and marked as a short-term reference image;

[0024] 2) Encode the image to be encoded using conventional video encoding operations to obtain image encoding data, and manage the reconstructed image according to the reference image management method of conventional video encoding;

[0025] The image encoded data is placed into strip data, and the type information of the strip data is placed in the header data of the strip data. This type information is used to identify whether the strip data is decoded using an intelligent image decoding network or a traditional video decoding operation.

[0026] The output bitstream is obtained by combining the strip data encoded by the intelligent image coding network and the strip data encoded by the traditional video coding operation in the coding order.

[0027] Furthermore, the method also includes:

[0028] The sequence-level parameter set and image-level parameter set corresponding to the stripe data are placed into the output bitstream;

[0029] Sequence-level intelligent image decoding extension information is placed in the sequence-level parameter set. This information is used to represent a candidate list of network structures or network models of the intelligent image decoding network used by the strip data in the sequence to which the sequence-level parameter set belongs.

[0030] Image-level intelligent image decoding extension information is placed in the image-level parameter set. This information is used to represent the network structure or network model of the intelligent image decoding network used for the strip data in several images to which the image-level parameter set belongs.

[0031] Furthermore, the operation of placing the image encoded data into the strip data also includes:

[0032] The striped data is placed into a network digest layer packet unit defined by a conventional video encoding and decoding method. The syntax information carried in the header of this packet unit is defined by the conventional video encoding and decoding method.

[0033] Furthermore, if the strip data is decoded using an intelligent image decoding network, the type of the network digest layer packet unit to which it belongs is identified as a random access point packet unit.

[0034] A third objective of this invention is to provide a video decoding device based on intelligent image decoding, comprising:

[0035] processor;

[0036] A memory for storing bitstreams, neural network models, and image data to be decoded; and

[0037] One or more programs are used to perform the following methods:

[0038] The processor obtains at least one stripe data from the image data to be decoded and obtains the type information of the stripe data, which is used to identify whether the stripe data is decoded using an intelligent image decoding network or a traditional video decoding operation;

[0039] If the type information of the strip data indicates that the strip data is decoded using an intelligent image decoding network, the processor uses the intelligent image decoding network to decode the strip data to obtain a reconstructed image of the image data to be decoded; the processor puts the reconstructed image into the decoded image buffer and marks it as a short-term reference image;

[0040] If the type information of the strip data indicates that the strip data is decoded using conventional video decoding operations, the processor decodes the strip data using conventional video decoding operations to obtain the reconstructed image with the decoded image, and manages the reconstructed image according to the reference image management method of conventional video decoding.

[0041] The processor combines the reconstructed image obtained from the intelligent image decoding network and the reconstructed image obtained from the traditional video decoding operation in the display order to obtain the output video.

[0042] The fourth objective of this invention is to provide a video encoding device based on intelligent image encoding, comprising:

[0043] processor;

[0044] Memory used to store bitstreams, neural network models, and images to be encoded; and

[0045] One or more programs are used to perform the following methods:

[0046] For the image to be encoded, the processor selects one of the following two encoding methods for encoding:

[0047] 1) The processor uses an intelligent image coding network to encode the image to be encoded, obtains image coding data, and puts the reconstructed image into the decoded image buffer, marking it as a short-term reference image;

[0048] 2) The processor encodes the image to be encoded using conventional video encoding operations to obtain image encoding data, and manages the reconstructed image according to the reference image management method of conventional video encoding;

[0049] The processor puts the image encoded data into strip data and puts the type information of the strip data into the header data of the strip data. This information is used to identify whether the strip data is decoded using an intelligent image decoding network or a traditional video decoding operation.

[0050] The processor combines the strip data encoded by the intelligent image coding network and the strip data encoded by the traditional video coding operation in the coding order to obtain the output bitstream.

[0051] The beneficial effects of this invention are twofold: firstly, it enables the use of intelligent image coding methods to encode a portion of images in a video, such as keyframes, thereby ensuring efficient encoding of these images for machine intelligence tasks to a certain extent; secondly, it allows the reuse of standard-compliant traditional video coding methods to encode another portion of images in a video, such as non-keyframes, effectively eliminating temporal redundancy in the video. Simultaneously, this invention reuses the high-level syntax design and main encoding / decoding operations of traditional coding, enabling the fusion of traditional video encoding / decoding and intelligent image coding without affecting the existing encoding / decoding capabilities of traditional methods. This improves the encoding efficiency of traditional video encoding / decoding for machine intelligence tasks while maintaining the system's transport layer's encapsulation and transmission capabilities for video encoding / decoding. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is an existing image encoding and decoding system that uses an end-to-end neural network;

[0054] Figure 2 This is a schematic diagram of the framework of a video encoding and decoding method based on intelligent image encoding according to an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram illustrating the display order of reconstructed images obtained from a video encoding / decoding method based on intelligent image encoding according to an embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram of the bitstream structure obtained by a video encoding and decoding method based on intelligent image encoding according to an embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0058] Definition of noun:

[0059] Keyframes: Keyframes are usually a number of video frames that are automatically or manually selected. Keyframes are usually encoded intra-frame, support random access, and can be used as reference frames when non-keyframes are inter-frame encoded.

[0060] NNIC, short for Neural Network Based ImageCodec, is an end-to-end neural network. Its encoding network encodes and compresses the input image to obtain an encoded bitstream, and its decoding network decodes the bitstream to output a reconstructed image. In encoding and decoding applications for machine intelligence tasks, because the LIC neural network can be jointly trained with the backend neural network used to complete the machine intelligence task, LIC can achieve high encoding efficiency with the goal of achieving machine intelligence task accuracy.

[0061] VVC: Versatile Video Coding, also known as H.266, is a video coding standard that can compress input video into a bitstream. This bitstream can contain multiple sub-bitstreams, and there can be reference relationships between the sub-bitstreams.

[0062] DPB: Short for Decoded Picture Buffer, this DPB exists in the VVC codec and stores and manages decoded images for output or to provide reference frames for subsequent decoded video frames. When managing decoded images, the DPB attaches tags to them to identify their attributes, such as the image's index number and layer number, whether it is used for short-term or long-term reference, and whether it is no longer in use.

[0063] Reference image: This image exists in the DPB and can serve as reference information for encoding and decoding other images.

[0064] Current image: The current image refers to an image to be encoded or decoded that is being encoded or decoded by the video encoding / decoding method or its apparatus.

[0065] A slice is a data unit defined in traditional video encoding and decoding methods. An image typically contains at least one slice. There are several types of slices, including intra slices, single-hypothesis inter-frame slices (P slices), and double-hypothesis inter-frame slices (B slices), which are encoded and decoded using intra-prediction, single-hypothesis inter-frame prediction, and double-hypothesis inter-frame prediction methods, respectively.

[0066] NAL packets, short for Network Abstract Layer units, are a video stream encapsulation format. NAL packets consist of a header and payload. The header contains information such as layer identifiers, type identifiers, and temporal layer identifiers, allowing the system layer to quickly identify NAL packets and distinguish their functions and data types.

[0067] This embodiment discloses a video decoding method based on intelligent image encoding, the framework of which is as follows: Figure 2 As shown. For the input bitstream, the image data in the bitstream is processed one by one. According to the type identifier of the image data, the corresponding decoding operation is selected to decode the image data to obtain the reconstructed image. If the image data is marked as being encoded by an NNIC encoding network, then this method will call the corresponding NNIC decoding network to decode the image data. The intelligently reconstructed image obtained by decoding is marked as a reference image and put into the DPB; if the image data is marked as being encoded by a conventional encoding method, then this method will call the corresponding conventional decoding method to decode the image data. The implementation scheme of the method and its various extended implementation schemes will be described in detail below.

[0068] In one implementation of a video decoding method based on intelligent image coding, the decoding method uses intelligent coding stripes that differ from the existing stripes in traditional video coding and decoding methods to distinguish image data in the bitstream. Without loss of generality, the traditional video coding and decoding in this implementation is VVC.

[0069] Specifically, for the current image bitstream data in the input bitstream, the header and payload data of the NAL packets are parsed from the bitstream. The syntax structure of the NAL packets is defined by VVC, and its header contains the hierarchical identifier, type identifier, and temporal hierarchical identifier of the NAL packet. In one possible implementation, without increasing or decreasing the number of bits in the NAL packet header data, considering that video encoding and decoding for machine intelligence tasks does not require multi-level functionality in application scenarios such as security monitoring and vehicle networking, this implementation can use the number of bits occupied by the hierarchical identifier in the NAL packet header data to represent new syntax elements, such as a new NAL packet type identifier, to indicate that the payload data of the NAL packet contains bitstream data obtained using intelligent image coding or other image coding methods such as intra-frame image coding of JPEG and HEVC.

[0070] For the NAL packet of the current image, the type of the NAL packet is parsed from the NAL packet header, and the decision to decode the payload data of the NAL packet is made based on the actual playback operation, such as random access, fast forward, or rewind. If the current image needs decoding, the payload data of the NAL packet of the current image is parsed and decoded to obtain the slice data of the current image. Without loss of generality, assuming that the current image contains only one slice, this implementation obtains the slice header data and slice payload data from the slice data. The slice type is parsed from the slice header data, and the decoding method used to decode the slice is determined based on the slice type. If the slice type is an intra-frame type (Intra), a single-hypothesis inter-frame type (P Inter), or a double-hypothesis inter-frame type (B Inter) defined by VVC, this implementation uses the intra-frame prediction method or the inter-frame prediction method of VVC to decode the slice payload data. If the slice type is the intelligent image coding intra-frame type (NNIC-Intra) newly defined in this implementation, this implementation uses an intelligent image decoding network to decode the slice payload data. In one possible implementation, the intelligent image decoding network can be an existing intelligent image decoding network such as cheng2020, mbt, etc., or it can be a publicly available intelligent image decoding network structure loaded with a decoding network model trained for machine intelligence tasks, aiming to improve the coding efficiency of video coding methods on machine intelligence tasks.

[0071] In another implementation, the stripe type can also be an Intelligent Image Coding Inter-Frame Type (NNIC-Inter). This scheme uses the inter-frame image decoding network in the intelligent video decoding network to decode the stripe payload data. The inter-frame image decoding network can obtain prediction information from the reference image in the DPB using methods such as optical flow or motion compensation, and combine it with the residual information obtained from decoding the stripe payload data to decode the reconstructed image.

[0072] In another implementation, the syntax structure for intra-frame or inter-frame type striped data using intelligent image encoding is shown in Table 1.

[0073] Table 1 shows an example syntax structure for using Smart Image Encoding for intra- or inter-frame type striped data.

[0074]

[0075]

[0076] The syntax structure `vcmi_slice_header()` contains the slice header data, `vcmi_slice_data()` contains the slice payload data, `rbsp_slice_trailing_bits()` contains the slice padding data, and `picture_header_structure()` contains the image header data. The semantics are as follows:

[0077] vcmi_sh_picture_header_in_slice_header_flag: indicates whether the image header is recorded in the slice header. A value of 1 indicates that the image header is in the slice header, and a value of 0 indicates that the image header is not in the slice header.

[0078] vcmi_sh_slice_type: indicates the data type of the slice. For example, a value of 3 indicates an intra-frame type for Smart Picture Coding, and a value of 4 indicates an inter-frame type for Smart Picture Coding.

[0079] vcmi_sh_slice_header_extension_length and vcmi_sh_slice_header_extension_data_byte[i] represent the length and data content of the slice header extension data;

[0080] vcmi_sd_transform_data_length, vcmi_sd_transform_data_byte[i], vcmi_sd_hyper_data_length, and vcmi_sd_hyper_data_byte[i] represent the bitstream data generated by intelligent image encoding, which is used to decode and obtain the reconstructed image.

[0081] For reconstructed images decoded using the VVC method, the reconstructed images are placed in the DPB and marked for management according to the reference image management method defined by VVC. For reconstructed images decoded using intelligent image decoding, this scheme places the reconstructed image in the DPB and marks it as a short-term reference image. This reconstructed image is managed using the same management method as the reconstructed image decoded by VVC. For example, if the reconstructed image decoded by intelligent image decoding is an IDR image, the DPB is refreshed. In one implementation, the NAL packet type of intelligent image coding is limited to Intra-IRAP type, including IDR or CRA. In this case, the stripe data of the NAL packet only uses the intelligent image coding intra-frame type.

[0082] After obtaining the reconstructed image from the VVC method decoding and the reconstructed image from the intelligent image decoding, the reconstructed images are interleaved and output according to their display order. One possible output order is as follows: Figure 3 As shown.

[0083] In another implementation, extended data related to intelligent image coding is included in the Sequence Parameter Set (SPS) or Image Parameter Set (PPS) to describe common parameters of intelligent image coding at the sequence or image level. For example, an identifier indicating whether intelligent image coding is enabled is included in the extended data of the SPS, indicating whether the sequence uses intelligent image coding and whether it contains intelligent image coding stripe data; another example is including a candidate list of network structures or models used for intelligent image coding in the SPS, indicating which network result or model is used in the intelligent image coding used in the sequence; yet another example is including the network structure or model used for intelligent image coding in the PPS, indicating which network result or model is used in the intelligent image coding of the image using the PPS; still another example is including a filtering identifier and filtering method for intelligent image coding images in the PPS, indicating whether the reconstructed image from intelligent image decoding needs additional filtering before being used as a reference image. When there are multiple candidates for network structure or network model described by SPS or PPS, the image header data of the strip data or the image to which the strip data belongs contains network structure or network model identification information, which is used to identify which network structure or network model was selected for the strip data or the image.

[0084] In one implementation scheme, researchers in the relevant field can, without the need for creativity, extend the above implementation scheme based on HEVC (High Efficiency Video Coding) or AVC (Advanced Video Coding) to obtain one or more video encoding and decoding methods based on intelligent image coding.

[0085] In another implementation, to effectively maintain compatibility with existing video codec methods, this invention proposes a slice type identification method that reuses existing video codec methods. Unlike previous implementations, this method adds a new slice sub-type identifier to the existing slice type identifiers in existing video codec methods. This identifier indicates whether the slice uses an intelligent image decoding method for decoding. For example, taking the existing video codec method VVC as an example, VVC defines that the existing slice type identifier syntax element `sh_slice_type` in the slice header can identify whether the current slice is an intra-prediction slice (I slice), a single-hypothesis inter-prediction slice (P slice), or a double-hypothesis inter-prediction slice (B slice). The decoding method used by each slice can be further distinguished as an existing decoding method or an intelligent image decoding method. For example, a slice sub-type identifier syntax element `sh_slice_sub_type` is added to the slice header to identify the decoding method used by the current slice. The advantage of this method compared to previous implementations is that it is compatible with existing video encoding and decoding methods. It can maintain the series of judgments and operations made by existing video encoding and decoding methods based on the stripe type identifier. The decoder only needs to select and call the existing decoding method or the intelligent image decoding method to decode the current stripe data after completing the judgment and operation of the stripe head.

[0086] In one implementation, taking the existing video encoding and decoding method VVC as an example, the structure of a syntax element can be shown in the table below.

[0087]

[0088]

[0089] Here, `sh_slice_type` is an existing slice type identifier in the slice header defined by VVC, and `sh_slice_sub_type` is a newly defined slice subtype identifier. A value of 1 indicates that the slice uses the smart decoding method, and a value of 0 indicates that the slice uses the existing decoding method. `vcmi_slice_data()` is the set of syntax elements for slices using the smart decoding method, and `slice_data()` is the set of syntax elements for slices using the existing decoding method.

[0090] In another implementation, only intra-prediction slices can use the intelligent decoding method. The advantage of this is that current mainstream intelligent image encoding and decoding methods achieve significant improvements in coding efficiency for intra-frame encoding, but their efficiency for inter-frame encoding is not high or even suffers. Therefore, allowing only intra-prediction slices to use the intelligent decoding method can effectively improve the overall coding efficiency of video encoding and decoding. In this implementation, when intra-prediction slices only use the intelligent decoding method, the slice subtype identifier can be implicitly derived from the slice type identifier; that is, when the slice type identifier is intra-prediction slice, the slice uses the intelligent decoding method.

[0091] This embodiment discloses a video coding method based on intelligent image coding, the framework of which is as follows: Figure 2 As shown. For the input video, a portion of the images are encoded using intelligent image coding to obtain a bitstream, and then decoded to obtain a reconstructed image, which is placed into the DPB as a reference image. The other portion of the images is encoded using traditional video coding methods to obtain a bitstream, and the reconstructed image is marked and managed according to traditional video coding methods. To improve the compatibility of this video coding method, the image bitstream obtained using intelligent image coding adopts a high-level syntax similar to or the same as that of traditional video coding methods, and uses the intelligent image coding stripe type at the stripe level. The advantage of this is that, from a high-level perspective, the data interface between the video bitstream and the system layer encapsulation and transmission can remain consistent, and no additional data structure needs to be designed to use the existing system layer encapsulation and transmission of bitstream data obtained from intelligent image coding video coding. From a low-level perspective, the existing stripes and coding tools of traditional video coding are inherited. By adding new intelligent image coding stripes to introduce intelligent image coding to compress the stripe data in the image, it is compatible with existing traditional video coding methods and brings them additional high coding efficiency for machine intelligence tasks.

[0092] In one implementation, without loss of generality, it is assumed that only keyframes or random access point images are encoded using intelligent image coding methods, while non-keyframes are encoded using VVC video coding inter-frame coding methods, and the two bitstreams are combined as the final bitstream. A possible bitstream structure is as follows: Figure 4 As shown.

[0093] Specifically, for keyframes in the video, this scheme calls the intelligent image coding method to encode the keyframe image to obtain an intelligent image bitstream, which is then placed into stripe data and marked as an intelligent image coding intra-frame type. The stripe data is placed into NAL packets, where the bitstream structure and high-level syntax of the NAL packets conform to VVC's design and definition for NAL packets. For example, the NAL packet can be marked as an IRAP NAL packet type, such as an IDR NAL packet or a CRA NAL packet. This scheme also obtains the reconstructed image of the keyframe, places it in the DPB, and marks it as a short-term reference image for use as a reference image for subsequent images. For non-keyframes in the video, this scheme calls VVC's inter-frame coding method for encoding and manages the reconstructed images of non-keyframes according to VVC's reference image marking and management methods.

[0094] In another implementation, keyframe encoding can be selected between intelligent image coding and VVC intra-frame image coding. The selection can be based on the information content of the image for machine intelligence tasks or the complexity of the image content. Images containing rich machine intelligence task information or complex image content can choose intelligent image coding, improving the coding efficiency for machine intelligence tasks. The resulting video stream contains keyframes of both stripe types simultaneously.

[0095] In another implementation, the encoding of non-keyframes can also use the inter-frame image coding method in Smart Video Coding. When encoding non-keyframes, a similar approach to the keyframe encoding selection described above can be used to choose between Smart Video Coding's inter-frame coding and VVC inter-frame image coding.

[0096] In another implementation, this scheme incorporates additional information for intelligent image encoding into the SPS or PPS as extended data in the SPS or PPS.

[0097] This embodiment discloses a decoding device, which includes a processor and a memory, for executing the decoding method disclosed in this invention.

[0098] This embodiment discloses an encoding device, which includes a processor and a memory, for executing the encoding method disclosed in this invention.

[0099] The above description of the embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. A video decoding method based on intelligent image decoding, characterized in that, For any image data to be decoded in the input video stream, perform the following operations: At least one stripe data is obtained from the image data to be decoded, and the type information of the stripe data is obtained. The type information is used to identify whether the stripe data is decoded using an intelligent image decoding network or a traditional video decoding operation. If the type information of the strip data indicates that the strip data is decoded using an intelligent image decoding network, then the strip data is decoded using the intelligent image decoding network to obtain a reconstructed image of the image data to be decoded; the reconstructed image is placed in the decoded image buffer and marked as a short-term reference image; If the type information of the strip data indicates that the strip data is decoded using conventional video decoding operations, then the strip data is decoded using conventional video decoding operations to obtain the reconstructed image of the image data to be decoded, and the reconstructed image is managed according to the reference image management method of conventional video decoding. The reconstructed image obtained by the intelligent image decoding network and the reconstructed image obtained by the traditional video decoding operation are combined in the display order to obtain the output video.

2. The method according to claim 1, characterized in that, The type information of the strip data is contained in the header data of the strip data. The type information also identifies whether the strip data is decoded using an intelligent intra-frame decoding network or an intelligent inter-frame decoding network.

3. The method according to claim 1 or 2, characterized in that, Also includes: Obtain the sequence-level parameter set and image-level parameter set corresponding to the image data to be decoded from the input video bitstream; Sequence-level intelligent image decoding extension information is obtained from the sequence-level parameter set. This information is used to represent a candidate list of network structures or network models of the intelligent image decoding network used by the strip data in the sequence to which the sequence-level parameter set belongs. Image-level intelligent image decoding extension information is obtained from the image-level parameter set. This information is used to represent the network structure or network model of the intelligent image decoding network used for the strip data in several images to which the image-level parameter set belongs.

4. The method according to claim 1, characterized in that, The strip data is contained in a network digest layer packet unit defined by a conventional video coding and decoding method, and the syntax information carried in the header of the packet unit is defined by the conventional video coding and decoding method.

5. A video coding method based on intelligent image coding, characterized in that, For any image to be encoded in the input video, perform the following operations: The image to be encoded is encoded using one of the following two encoding methods: 1) The image to be encoded is encoded using an intelligent image coding network to obtain image coding data, and the reconstructed image is placed in the decoded image buffer and marked as a short-term reference image; 2) Encode the image to be encoded using conventional video encoding operations to obtain image encoding data, and manage the reconstructed image according to the reference image management method of conventional video encoding; The image encoded data is placed into strip data, and the type information of the strip data is placed in the header data of the strip data. This type information is used to identify whether the strip data is decoded using an intelligent image decoding network or a traditional video decoding operation. The output bitstream is obtained by combining the strip data encoded by the intelligent image coding network and the strip data encoded by the traditional video coding operation in the coding order.

6. The method according to claim 5, characterized in that, Also includes: The sequence-level parameter set and image-level parameter set corresponding to the stripe data are placed into the output bitstream; Sequence-level intelligent image decoding extension information is placed in the sequence-level parameter set. This information is used to represent a candidate list of network structures or network models of the intelligent image decoding network used by the strip data in the sequence to which the sequence-level parameter set belongs. Image-level intelligent image decoding extension information is placed in the image-level parameter set. This information is used to represent the network structure or network model of the intelligent image decoding network used for the strip data in several images to which the image-level parameter set belongs.

7. The method according to claim 5, characterized in that, The operation of placing the image encoded data into strip data further includes: The striped data is placed into a network digest layer packet unit defined by a conventional video encoding and decoding method. The syntax information carried in the header of this packet unit is defined by the conventional video encoding and decoding method.

8. The method according to claim 7, characterized in that, If the strip data is decoded using an intelligent image decoding network, the type of the network digest layer packet unit to which it belongs is identified as a random access point packet unit.

9. A video decoding device based on intelligent image decoding, comprising: processor; A memory used to store bitstreams, neural network models, and image data to be decoded; as well as One or more programs are used to perform the following methods: The processor obtains at least one stripe data from the image data to be decoded and obtains the type information of the stripe data, which is used to identify whether the stripe data is decoded using an intelligent image decoding network or a traditional video decoding operation; If the type information of the strip data indicates that the strip data is decoded using an intelligent image decoding network, the processor uses the intelligent image decoding network to decode the strip data to obtain a reconstructed image of the image data to be decoded; the processor puts the reconstructed image into the decoded image buffer and marks it as a short-term reference image; If the type information of the strip data indicates that the strip data is decoded using conventional video decoding operations, the processor decodes the strip data using conventional video decoding operations to obtain the reconstructed image of the image to be decoded, and manages the reconstructed image according to the reference image management method of conventional video decoding. The processor combines the reconstructed image obtained from the intelligent image decoding network and the reconstructed image obtained from the traditional video decoding operation in the display order to obtain the output video.

10. A video encoding device based on intelligent image encoding, comprising: processor; A memory used to store bitstreams, neural network models, and images to be encoded; as well as One or more programs are used to perform the following methods: For the image to be encoded, the processor selects one of the following two encoding methods for encoding: 1) The processor uses an intelligent image coding network to encode the image to be encoded, obtains image coding data, and puts the reconstructed image into the decoded image buffer, marking it as a short-term reference image; 2) The processor encodes the image to be encoded using conventional video encoding operations to obtain image encoding data, and manages the reconstructed image according to the reference image management method of conventional video encoding; The processor puts the image encoded data into strip data and puts the type information of the strip data into the header data of the strip data. This information is used to identify whether the strip data is decoded using an intelligent image decoding network or a traditional video decoding operation. The processor combines the strip data encoded by the intelligent image coding network and the strip data encoded by the traditional video coding operation in the coding order to obtain the output bitstream.

Images