Video coding and decoding method and device based on intelligent image coding and decoding

Abstract

The invention discloses a video coding and decoding method and device based on intelligent image coding and decoding, and the method comprises the steps: carrying out the coding and decoding of a part of images, such as key frames, in a video through employing an intelligent image coding and decoding method, putting the reconstruction results of the images into a decoding image cache, and enabling the reconstruction results to serve as reference images of subsequent images, and at the same time, carrying out the coding and decoding of the images. According to the method, a traditional video coding and decoding method such as VVC is used for coding and decoding another part of images, such as non-key frames, in a video, and the coding and decoding of the images can refer to reconstructed images obtained through intelligent image coding and decoding.

Description

Technical Field

[0001] The present invention relates to the technical field of video coding and decoding, and more specifically, to a video coding and decoding method and device based on intelligent image coding and decoding. Background Art

[0002] The existing image and video encoding and decoding processing includes traditional encoding and decoding methods and intelligent encoding and decoding methods based on neural networks. Traditional encoding and decoding methods eliminate redundant information of image and video through operations such as prediction, transformation, quantization, entropy coding and loop filtering. Intelligent encoding and decoding methods use neural networks to transform image and video, convert image and video into feature tensors, and perform operations such as downsampling, quantization and entropy coding to achieve compression coding.

[0003] The image intelligent coding and decoding method proposes many efficient neural network structures for image feature information extraction. Convolutional neural network (CNN) is the earliest network structure used for image coding and decoding. Based on CNN, many improved network structures and probability estimation models have been derived. Taking network structures as an example, including generative adversarial network (GAN) and recurrent neural network (RNN), they have greatly improved the end-to-end image compression performance based on neural networks. Among them, the image coding and decoding method based on generative adversarial network (GAN) has achieved significant results in improving the subjective quality of images.

[0004] Video intelligent coding and decoding methods are mainly concentrated in three aspects: 1) hybrid neural network coding, 2) neural network rate-distortion optimization coding, and 3) end-to-end video coding. Hybrid neural network coding embeds neural networks into the video framework instead of traditional coding modules. Generally, inter-frame prediction modules, loop filter modules, and entropy coding modules are used more. Neural network rate-distortion optimization coding uses the highly nonlinear characteristics of neural networks to train neural networks into efficient discriminators and classifiers, for example, in the video coding mode decision link. End-to-end video coding is currently generally divided into using CNN to replace all modules of traditional coding methods, or expanding the input dimension of the neural network to perform end-to-end compression on all frames.

[0005] In the image or video encoding and decoding method using end-to-end neural network, the common operation is to first extract features from the image or video, then encode and decode it, and then restore it to the image. Figure 1 As shown, the process of extracting feature tensors is E1, the process of encoding the feature tensors obtained by E1 and obtaining a code stream is E2, and the encoding method is composed of E1 and E2. Correspondingly, the decoding method is composed of D1 and D2, where D1 refers to the process of transforming feature tensors into decoded images, and D2 refers to the process of decoding code streams into feature tensors.

[0006] In the current mainstream application scenarios, the encoded and decoded images and videos are not only used for human viewing, but are also often used to complete machine intelligence tasks. The intelligent task network completes task objectives such as target detection, target tracking or behavior recognition by analyzing images or videos.

[0007] The existing traditional video coding is usually designed for pixel fidelity and cannot effectively complete machine intelligence tasks. However, the image intelligent coding and decoding based on neural networks can be trained for the accuracy of machine intelligence tasks, thereby achieving results that surpass traditional image coding methods in machine intelligence tasks. However, the current video intelligent coding and decoding methods based on neural networks are not good enough and cannot surpass the latest traditional video coding methods such as H.266 / VVC. This is mainly because the neural network-based methods cannot fully eliminate redundant information in the time domain.

[0008] In the technology disclosed in JVET-AA0063, the encoding of key frames is replaced by the LIC encoding and decoding method from the traditional video encoding method, achieving a higher encoding efficiency than VVC. However, this technology is still optimized for pixel fidelity and human eye viewing, and cannot effectively cope with the application scenarios of machine intelligence tasks. At the same time, this technology requires modification of the VVC standard and is not compatible with the VVC standard. Summary of the invention

[0009] In order to solve the above-mentioned shortcomings of the prior art, the present invention proposes a video encoding and decoding method and device based on intelligent image coding. The method uses the intelligent image coding method to encode and decode a part of images in the video, such as key frames, and puts the reconstructed results of these images into a decoded image cache as reference images for subsequent images; at the same time, the method uses traditional video encoding and decoding methods such as VVC to encode and decode another part of images in the video, such as non-key frames, and the encoding and decoding of these images can refer to the reconstructed images obtained by intelligent image coding.

[0010] To this end, the first object of the present invention is to propose a video decoding method based on intelligent image decoding, for any image data to be decoded in an input video code stream, the following operations are performed:

[0011] Acquire at least one stripe of data from the image data to be decoded and acquire type information of the stripe of data, where the type information is used to identify whether the stripe of data is decoded using an intelligent image decoding network or a traditional video decoding operation;

[0012] If the type information of the strip data identifies that the strip data is decoded using an intelligent image decoding network, the strip data is decoded using the intelligent image decoding network to obtain a reconstructed image of the image data to be decoded; the reconstructed image is placed in a decoded image cache and marked as a short-term reference image;

[0013] If the type information of the strip data identifies that the strip data is to be decoded using a conventional video decoding operation, the strip data is decoded using the conventional video decoding operation to obtain a reconstructed image of the image data to be decoded, and the reconstructed image is managed according to a reference image management method of conventional video decoding;

[0014] The reconstructed image decoded by the intelligent image decoding network and the reconstructed image decoded by the traditional video decoding operation are combined in a display order to obtain an output video.

[0015] Furthermore, the type information of the strip data exists in the header data of the strip data, and the type information also identifies whether the strip data is decoded using an intelligent intra-frame decoding network or an intelligent inter-frame decoding network.

[0016] Furthermore, the method also includes:

[0017] Acquire a sequence-level parameter set and a picture-level parameter set corresponding to the to-be-decoded picture data from an input video bitstream;

[0018] Acquire sequence-level intelligent image decoding extension information from a sequence-level parameter set, the information being used to represent a network structure or a candidate list of network models of an intelligent image decoding network used by strip data in a sequence to which the sequence-level parameter set belongs;

[0019] Image-level intelligent image decoding extension information is obtained from an image-level parameter set, where the information is used to represent a network structure or a network model of an intelligent image decoding network used by strip data in a number of images to which the image-level parameter set belongs.

[0020] Furthermore, the stripe data is contained in a network summary layer packet unit defined by a conventional video coding and decoding method, and the syntax information carried by the packet header of the packet unit is defined by the conventional video coding and decoding method.

[0021] The second object of the present invention is to provide a video encoding method based on intelligent image encoding, for any image to be encoded in an input video, the following operations are performed:

[0022] Select one of the following two encoding methods to encode the image to be encoded:

[0023] 1) Encoding the image to be encoded using an intelligent image coding network to obtain image coding data, and putting the reconstructed image into a decoded image cache and marking it as a short-term reference image;

[0024] 2) encoding the image to be encoded using a conventional video encoding operation to obtain image encoding data, and managing the reconstructed image according to a reference image management method of conventional video encoding;

[0025] Put the image coding data into the strip data, and put the type information of the strip data into the header data of the strip data, where the type information is used to identify whether the strip data is decoded using an intelligent image decoding network or a traditional video decoding operation;

[0026] The strip data obtained by the intelligent image coding network coding and the strip data obtained by the traditional video coding operation coding are combined in a coding order to obtain an output code stream.

[0027] Furthermore, the method also includes:

[0028] Putting the sequence level parameter set and the picture level parameter set corresponding to the stripe data into the output bitstream;

[0029] Putting sequence-level intelligent image decoding extension information into the sequence-level parameter set, the information being used to represent a candidate list of network structures or network models of an intelligent image decoding network used for strip data in a sequence to which the sequence-level parameter set belongs;

[0030] Image-level intelligent image decoding extension information is placed in the image-level parameter set, and the information is used to represent the network structure or network model of the intelligent image decoding network used by the strip data in several images to which the image-level parameter set belongs.

[0031] Furthermore, the operation of placing the image coding data into the strip data further includes:

[0032] The strip data is placed into a network summary layer packet unit defined by a conventional video coding and decoding method, and the syntax information carried by the packet header of the packet unit is defined by the conventional video coding and decoding method.

[0033] Further, if the strip data is decoded using an intelligent image decoding network, the type of the network summary layer packet unit to which it belongs is identified as a random access point packet unit.

[0034] A third object of the present invention is to provide a video decoding device based on intelligent image decoding, comprising:

[0035] processor;

[0036] A memory for storing a code stream, a neural network model, and image data to be decoded; and

[0037] One or more procedures are used to accomplish the following:

[0038] The processor obtains at least one strip of data from the image data to be decoded and obtains type information of the strip of data, where the information is used to identify whether the strip of data is decoded using an intelligent image decoding network or a traditional video decoding operation;

[0039] If the type information of the strip data identifies that the strip data is decoded using an intelligent image decoding network, the processor decodes the strip data using the intelligent image decoding network to obtain a reconstructed image of the image data to be decoded; the processor puts the reconstructed image into a decoded image cache and marks it as a short-term reference image;

[0040] If the type information of the strip data identifies that the strip data is decoded using a conventional video decoding operation, the processor decodes the strip data using the conventional video decoding operation to obtain a reconstructed image with the decoded image, and manages the reconstructed image according to a reference image management method of conventional video decoding;

[0041] The processor combines the reconstructed image decoded by the intelligent image decoding network and the reconstructed image decoded by the traditional video decoding operation in a display order to obtain an output video.

[0042] A fourth object of the present invention is to provide a video encoding device based on intelligent image coding, comprising:

[0043] processor;

[0044] A memory for storing a bitstream, a neural network model, and an image to be encoded; and

[0045] One or more procedures are used to accomplish the following:

[0046] For the image to be encoded, the processor selects one of the following two encoding methods for encoding:

[0047] 1) The processor encodes the image to be encoded using an intelligent image encoding network to obtain image encoding data, and puts the reconstructed image into a decoded image cache and marks it as a short-term reference image;

[0048] 2) The processor encodes the image to be encoded using a conventional video encoding operation to obtain image encoding data, and manages the reconstructed image according to a reference image management method of conventional video encoding;

[0049] The processor puts the image encoding data into the strip data, and puts the type information of the strip data into the header data of the strip data, where the information is used to identify whether the strip data is decoded using an intelligent image decoding network or a traditional video decoding operation;

[0050] The processor combines the strip data obtained by the intelligent image coding network coding and the strip data obtained by the traditional video coding operation coding in a coding order to obtain an output code stream.

[0051] The beneficial effects of the present invention are that, on the one hand, an intelligent image coding method can be used to encode a part of the images in the video, such as key frames, which to a certain extent ensures the efficient coding of these images when facing machine intelligence tasks, and on the other hand, a traditional video coding method that meets the standard can be reused to encode another part of the images in the video, such as non-key frames, effectively eliminating the time domain redundant information in the video. At the same time, the invention reuses the high-level syntax design and main encoding and decoding operations in traditional coding, and can achieve the fusion of traditional video coding and intelligent image coding without affecting the existing coding and decoding capabilities of traditional coding, thereby improving the coding efficiency of traditional video coding when facing machine intelligence tasks, while also maintaining the system transmission layer's encapsulation and transmission capabilities for video coding. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 It is an image encoding and decoding system using an end-to-end neural network in the prior art;

[0054] Figure 2 A schematic diagram of a framework of a video encoding and decoding method based on intelligent image coding according to an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of the display order of reconstructed images obtained by decoding a bit stream using a video encoding and decoding method based on intelligent image coding according to an embodiment of the present invention;

[0056] Figure 4 The present invention is a schematic diagram of a bitstream structure obtained by encoding a video encoding and decoding method based on intelligent image coding according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0058] Definition of terms:

[0059] Key frame: Key frames are usually several video frames that are automatically or manually selected in the video. Key frames usually use intra-frame coding, can support random access, and can be used as reference frames when non-key frames are inter-frame coded.

[0060] NNIC: The full name is Neural Network Based Image Codec, which is an end-to-end neural network. Its encoding network can encode compressed input images to obtain encoded bitstreams, and its decoding network can decode the bitstreams to output reconstructed images. In the encoding and decoding application scenarios for machine intelligence tasks, since the LIC neural network can be jointly trained with the neural network used to complete machine intelligence tasks at the back end, LIC can achieve high coding efficiency with the accuracy of machine intelligence tasks as the goal.

[0061] VVC: Versatile Video Coding, also known as H.266, is a video coding standard that can compress the input video to obtain a bitstream. The bitstream can contain multiple layers of sub-bitstreams, and the sub-bitstreams can have reference relationships.

[0062] DPB: Decoded Picture Buffer, which exists in the VVC codec and can store and manage decoded images for outputting images or providing reference frames for subsequently decoded video frames. When managing decoded images, DPB will add some tags to the decoded images to identify the attributes of the decoded images, such as the index number of the decoded image and the number of the layer to which it belongs, whether it is used for short-term reference or long-term reference, whether it is no longer used, etc.

[0063] Reference image: This image exists in the DPB and can provide reference information for encoding and decoding of other images.

[0064] Current image: The current image refers to an image to be encoded or an image to be decoded that is being encoded or decoded by a video coding or decoding method or device.

[0065] Slice: A data unit defined in traditional video coding and decoding methods. A picture usually contains at least one slice, and there are many types of slices, including intra slices, single-hypothesis inter slices (P slices) and double-hypothesis inter slices (B slices), which are coded and decoded using intra prediction, single-hypothesis inter prediction and double-hypothesis inter prediction methods respectively.

[0066] NAL packet: full name Network Abstract Layer units, is a video code stream encapsulation format. NAL packet consists of header data and payload data, where the header data contains information such as layer identification, type identification and time domain layer identification, which is used by the system layer to quickly identify NAL packets to distinguish the functions of different NAL packets and the data types they contain.

[0067] This embodiment discloses a video decoding method based on intelligent image coding, and its framework is as follows: Figure 2 As shown. For the input code stream, the image data in the code stream is processed one by one, and according to the type identification of the image data, the corresponding decoding operation is selected to decode the image data to obtain a reconstructed image. If the image data is marked as being encoded by the NNIC encoding network, then the method will call the corresponding NNIC decoding network to decode the image data, and the decoded intelligent reconstructed image is marked as a reference image and placed in the DPB; if the image data is marked as being encoded by a traditional encoding method, then the method will call the corresponding traditional decoding method to decode the image data. The implementation scheme of the method and its various extended implementation schemes will be described in detail below.

[0068] In one implementation of a video decoding method based on intelligent image coding, the decoding method uses intelligent coding strips different from existing strips in traditional video coding and decoding methods to distinguish image data in a bitstream. Without loss of generality, the traditional video codec in this implementation is VVC.

[0069] Specifically, for the bitstream data of the current image in the input bitstream, the header data and payload data of the NAL packet are parsed from the bitstream, wherein the syntax structure of the NAL packet is defined by VVC, and its header contains the layer identifier, type identifier, and time domain layer identifier of the NAL packet to which it belongs. In a possible implementation, without adding or deleting the number of bits of the NAL header data, considering that the video encoding and decoding for machine intelligence tasks does not require multi-level functions in application scenarios such as security monitoring and Internet of Vehicles, this implementation can use the number of bits occupied by the layer identifier in the NAL header data to represent new syntax elements, such as a new NAL packet type identifier, which is used to identify that the payload data of the NAL packet contains bitstream data obtained by using intelligent image coding or other image coding methods such as JPEG and HEVC intra-frame image coding.

[0070] For the NAL packet of the current image obtained, the type of the NAL packet is parsed from the NAL packet header, and whether to decode the payload data of the NAL packet is selected according to the actual playback operation, and the playback operation can be random access, fast forward and rewind, etc. If the current image needs to be decoded, the payload data of the NAL packet of the current image is parsed and decoded, and the stripe data of the current image is obtained therefrom. Without loss of generality, assuming that the current image contains only one stripe, this implementation scheme obtains the stripe header data and the stripe payload data from the stripe data. The stripe type is parsed from the stripe header data, and the decoding method used to decode the stripe is determined according to the stripe type. If the stripe type is an intra-frame type (Intra), a single-hypothesis inter-frame type (P Inter), or a double-hypothesis inter-frame type (B Inter) defined by VVC, this scheme uses the intra-frame prediction method or the inter-frame prediction method of VVC to decode the stripe payload data. If the stripe type is an intelligent image coding intra-frame type (NNIC-Intra) newly defined in this scheme, this scheme uses an intelligent image decoding network to decode the stripe payload data. In one possible implementation, the intelligent image decoding network can be an existing intelligent image decoding network such as cheng2020, mbt, etc., or it can be a public intelligent image decoding network structure loaded with a decoding network model trained for machine intelligence tasks, aiming to improve the coding efficiency of the video coding method on machine intelligence tasks.

[0071] In another embodiment, the slice type may also be an intelligent image coding inter-frame type (NNIC-Inter), and this solution uses an inter-frame image decoding network in an intelligent video decoding network to decode the slice payload data. The inter-frame image decoding network may obtain prediction information from a reference image in the DPB using methods such as optical flow or motion compensation, and decodes and obtains a reconstructed image in combination with the residual information decoded from the slice payload data.

[0072] In another embodiment, the syntax structure of slice data of intra-frame or inter-frame type using intelligent image coding is shown in Table 1.

[0073] Table 1 An example of a syntax structure for intra-frame or inter-frame type slice data using intelligent image coding

[0074]

[0075]

[0076] Among them, the syntax structure vcmi_slice_header() is the slice header data, vcmi_slice_data() is the slice payload data, rbsp_slice_trailing_bits() is the slice padding data, and picture_header_structure() is the picture header data. Among them, the syntax semantics are as follows:

[0077] vcmi_sh_picture_header_in_slice_header_flag: identifies whether the picture header is recorded in the slice header. A value of 1 indicates that the picture header is in the slice header, and a value of 0 indicates that the picture header is not in the slice header.

[0078] vcmi_sh_slice_type: indicates the data type of the slice. For example, a value of 3 indicates the smart image coding intra-frame type, and a value of 4 indicates the smart image coding inter-frame type.

[0079] vcmi_sh_slice_header_extension_length and vcmi_sh_slice_header_extension_data_byte[i] indicate the length and data content of the slice header extension data;

[0080] vcmi_sd_transform_data_length, vcmi_sd_transform_data_byte[i], vcmi_sd_hyper_data_length and vcmi_sd_hyper_data_byte[i] represent the code stream data generated by intelligent image coding, which is used for decoding to obtain the reconstructed image.

[0081] For the reconstructed image obtained by decoding using the VVC method, the reconstructed image is placed in the DPB and marked for management according to the reference image management method defined by VVC. For the reconstructed image obtained by intelligent image decoding, this scheme puts the reconstructed image into the DPB and marks it as a short-term reference image. The reconstructed image is managed according to the same management method as the reconstructed image obtained by VVC decoding. For example, if the reconstructed image obtained by intelligent image decoding is an IDR image, the DPB is refreshed. In one embodiment, the NAL packet type of intelligent image coding is limited to an intra-frame random access (IRAP) type, including IDR or CRA. At this time, the slice data of the NAL packet only uses the intelligent image coding intra-frame type.

[0082] After obtaining the reconstructed image decoded by the VVC method and the reconstructed image decoded by the intelligent image, the reconstructed images are interspersed and output according to the display order of the reconstructed images. A possible output order is as follows: Figure 3 shown.

[0083] In another embodiment, extended data related to intelligent image coding is placed in the sequence parameter set SPS or the picture parameter set PPS to describe common parameters of intelligent image coding at the sequence level or the picture level. For example, an identifier of whether intelligent image coding is turned on is placed in the extended data of the SPS to indicate whether the sequence uses intelligent image coding and whether it contains intelligent image coding strip data; for another example, a candidate list of network structures or network models used in intelligent image coding is placed in the SPS to indicate which one or several network results or network models are used in the intelligent image coding used in the sequence; for another example, a network structure or network model used in intelligent image coding is placed in the PPS to indicate which one or several network results or network models are used in the intelligent image coding used in the image using the PPS; for another example, a filtering identifier and filtering method of an intelligent image coded image is placed in the PPS to indicate whether the reconstructed image of intelligent image decoding needs to be subjected to additional filtering before being used as a reference image. When there are multiple network structure or network model candidates described by SPS or PPS, the strip data or the image header data of the image to which the strip data belongs contains network structure or network model identification information for identifying which network structure or network model is selected for the strip data or the image.

[0084] In one embodiment, researchers in related fields can, based on the above implementation scheme, without having to exert any creativity, similarly expand the above implementation scheme based on HEVC (High Efficiency Video Coding) or AVC (Advanced Video Coding) to obtain one or more video coding and decoding methods based on intelligent image coding.

[0085] In another embodiment, in order to be effectively compatible with existing video coding and decoding methods, the present invention proposes a slice type identification method that reuses existing video coding and decoding methods. Different from the previous implementation scheme, this method adds a new slice subtype identification on the basis of the existing slice type identification of the existing video coding and decoding method, which is used to identify whether the slice uses the intelligent image decoding method to be decoded. For example, taking the existing video coding and decoding method VVC as an example, the existing slice type identification syntax element sh_slice_type in the VVC definition slice header can identify whether the current slice is an intra-frame prediction slice (I slice), a single-hypothesis inter-frame prediction slice (P slice) or a double-hypothesis inter-frame prediction slice (B slice). The decoding method used for each slice can be further distinguished as an existing decoding method or an intelligent image decoding method. For example, a slice subtype identification syntax element sh_slice_sub_type is added to the slice header to identify the decoding method used by the current slice. The advantage of this method compared to the previous implementation method is that this method is compatible with the existing video coding and decoding method, and can maintain the series of judgments and operations made by the existing video coding and decoding method according to the strip type identifier. The decoder only needs to complete the judgment and operation of the strip header, and then choose to call the existing decoding method or the intelligent image decoding method according to the strip subtype identifier to decode the current strip data.

[0086] In one implementation, taking the existing video coding method VVC as an example, the structure of a syntax element may be as shown in the following table.

[0087]

[0088]

[0089] Among them, sh_slice_type is the existing slice type identifier in the slice header defined by VVC, sh_slice_sub_type is the newly defined slice subtype identifier, a value of 1 indicates that the slice uses the smart decoding method, and a value of 0 indicates that the slice uses the existing decoding method. vcmi_slice_data() is the syntax element set of the slice using the smart decoding method, and slice_data() is the syntax element set of the slice using the existing decoding method.

[0090] In another embodiment, only the intra-frame prediction strip can use the intelligent decoding method. The advantage of this is that the current mainstream intelligent image coding and decoding method can obtain a significant improvement in coding efficiency in coding within the image frame, but the coding efficiency obtained in coding between image frames is not high or even lossy. In this case, only allowing the intra-frame prediction strip to use the intelligent decoding method can effectively improve the overall coding efficiency of the video codec. In this embodiment, when the intra-frame prediction strip only uses the intelligent decoding method, the strip subtype identifier can be implicitly derived from the strip type identifier, that is, when the strip type identifier is an intra-frame prediction strip, the strip uses the intelligent decoding method.

[0091] This embodiment discloses a video encoding method based on intelligent image encoding, and its framework is as follows: Figure 2 As shown. For the input video, a part of the image is encoded using the intelligent image coding method to obtain a bitstream, and the decoded reconstructed image is placed in the DPB as a reference image; the other part of the image is encoded using the traditional video coding method to obtain a bitstream, and the reconstructed image is marked and managed according to the traditional video coding method. In order to improve the compatibility of the video coding method, the image bitstream obtained by using the intelligent image coding adopts a high-level syntax similar to or the same as the traditional video coding method, and the stripe type of the intelligent image coding is used at the stripe level. The advantage of this is that, from a high-level perspective, the data interface of the video bitstream and the system layer encapsulation transmission can be consistent, and the existing system layer encapsulation and transmission of the bitstream data obtained by the video coding based on the intelligent image coding can be used without designing an additional data structure. From a bottom-level perspective, the existing strips and their encoding tools of the traditional video coding are inherited. By adding new intelligent image coding strips to introduce intelligent image coding to compress the stripe data in the image, it can be compatible with the existing traditional video coding method and bring it additional high coding efficiency for machine intelligence tasks.

[0092] In one embodiment, without loss of generality, it is assumed that only key frames or random access point images are encoded using the intelligent image coding method, and non-key frames are encoded using the inter-frame coding method of VVC video coding, and the two parts of the bitstream are combined together as the final bitstream. A possible bitstream structure is as follows: Figure 4 shown.

[0093] Specifically, for key frames in a video, this solution calls an intelligent image coding method to encode the key frame image to obtain an intelligent image code stream, puts it into the strip data, and marks the strip as an intelligent image coding intra-frame type. The strip data is put into a NAL packet, wherein the code stream structure and high-level syntax of the NAL packet conform to the design and definition of the NAL packet by VVC. For example, the NAL packet can be marked as a type of IRAP NAL packet such as an IDR NAL packet or a CRA NAL packet. This solution also obtains the reconstructed image of the key frame, puts it into the DPB and marks it as a short-term reference image, which is used as a reference image for subsequent images. For non-key frames in the video, this solution calls the inter-frame coding method of VVC video coding for encoding, and manages the reconstructed images of non-key frames according to the reference image marking and management method specified by VVC.

[0094] In another embodiment, the encoding of key frames can be selected between intelligent image encoding and VVC intra-frame image encoding. The selection can be based on the information content in the image for machine intelligence tasks or the complexity of the image content. Images containing rich machine intelligence task information or complex image content can choose intelligent image encoding to improve the encoding efficiency for machine intelligence tasks. In this way, the video bitstream obtained has two types of key frames of strips at the same time.

[0095] In another embodiment, the coding of non-key frames can also use the inter-frame image coding method in intelligent video coding. When coding non-key frames, a method similar to the above-mentioned key frame coding selection can be used to select between the inter-frame coding of intelligent video coding and the VVC inter-frame image coding.

[0096] In another embodiment, the present scheme places additional information for intelligent image coding in the SPS or PPS as extended data in the SPS or PPS.

[0097] This embodiment discloses a decoding device, which includes a processor and a memory, and is used to execute the decoding method disclosed in the present invention.

[0098] This embodiment discloses a coding device, which includes a processor and a memory, and is used to execute the coding method disclosed in the present invention.

[0099] The above embodiments are only used to help understand the method and core idea of ​​the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A video decoding method based on intelligent image decoding, characterized in that: For any image data to be decoded in the input video code stream, perform the following operations: Acquire at least one stripe of data from the image data to be decoded and acquire type information of the stripe of data, where the type information is used to identify whether the stripe of data is decoded using an intelligent image decoding network or a traditional video decoding operation; If the type information of the strip data identifies that the strip data is decoded using an intelligent image decoding network, the strip data is decoded using the intelligent image decoding network to obtain a reconstructed image of the image data to be decoded; the reconstructed image is placed in a decoded image cache and marked as a short-term reference image; If the type information of the strip data identifies that the strip data is to be decoded using a conventional video decoding operation, the strip data is decoded using the conventional video decoding operation to obtain a reconstructed image of the image data to be decoded, and the reconstructed image is managed according to a reference image management method of conventional video decoding; The reconstructed image decoded by the intelligent image decoding network and the reconstructed image decoded by the traditional video decoding operation are combined in a display order to obtain an output video.

2. The method according to claim 1, characterized in that: The type information of the strip data exists in the header data of the strip data, and the type information also identifies whether the strip data is decoded using an intelligent intra-frame decoding network or an intelligent inter-frame decoding network.

3. The method according to claim 1 or 2, characterized in that: Also includes: Acquire a sequence-level parameter set and a picture-level parameter set corresponding to the to-be-decoded picture data from an input video bitstream; Acquire sequence-level intelligent image decoding extension information from a sequence-level parameter set, the information being used to represent a network structure or a candidate list of network models of an intelligent image decoding network used by strip data in a sequence to which the sequence-level parameter set belongs; Image-level intelligent image decoding extension information is obtained from an image-level parameter set, where the information is used to represent a network structure or a network model of an intelligent image decoding network used by strip data in a number of images to which the image-level parameter set belongs.

4. The method according to claim 1, characterized in that The slice data is contained in a network summary layer packet unit defined by a conventional video coding and decoding method, and the syntax information carried by the packet header of the packet unit is defined by the conventional video coding and decoding method.

5. A video encoding method based on intelligent image coding, characterized in that: For any image to be encoded in the input video, perform the following operations: Select one of the following two encoding methods to encode the image to be encoded: 1) Encoding the image to be encoded using an intelligent image coding network to obtain image coding data, and putting the reconstructed image into a decoded image cache and marking it as a short-term reference image; 2) encoding the image to be encoded using a conventional video encoding operation to obtain image encoding data, and managing the reconstructed image according to a reference image management method of conventional video encoding; Put the image coding data into the strip data, and put the type information of the strip data into the header data of the strip data, where the type information is used to identify whether the strip data is decoded using an intelligent image decoding network or a traditional video decoding operation; The strip data obtained by the intelligent image coding network coding and the strip data obtained by the traditional video coding operation coding are combined in a coding order to obtain an output code stream.

6. The method according to claim 5, characterized in that Also includes: Putting the sequence level parameter set and the picture level parameter set corresponding to the stripe data into the output bitstream; Putting sequence-level intelligent image decoding extension information into the sequence-level parameter set, the information being used to represent a candidate list of network structures or network models of an intelligent image decoding network used for strip data in a sequence to which the sequence-level parameter set belongs; Image-level intelligent image decoding extension information is placed in the image-level parameter set, and the information is used to represent the network structure or network model of the intelligent image decoding network used by the strip data in several images to which the image-level parameter set belongs.

7. The method according to claim 5, characterized in that The operation of placing the image coding data into the strip data further includes: The strip data is placed into a network summary layer packet unit defined by a conventional video coding and decoding method, and the syntax information carried by the packet header of the packet unit is defined by the conventional video coding and decoding method.

8. The method according to claim 7, characterized in that If the slice data is decoded using an intelligent image decoding network, the type of the network summary layer packet unit to which the slice data belongs is identified as a random access point packet unit.

9. A video decoding device based on intelligent image decoding, comprising: processor; A memory for storing code streams, neural network models, and image data to be decoded; as well as One or more procedures are used to accomplish the following: The processor obtains at least one strip of data from the image data to be decoded and obtains type information of the strip of data, where the information is used to identify whether the strip of data is decoded using an intelligent image decoding network or a traditional video decoding operation; If the type information of the strip data identifies that the strip data is decoded using an intelligent image decoding network, the processor decodes the strip data using the intelligent image decoding network to obtain a reconstructed image of the image data to be decoded; the processor puts the reconstructed image into a decoded image cache and marks it as a short-term reference image; If the type information of the strip data identifies that the strip data is decoded using a conventional video decoding operation, the processor decodes the strip data using the conventional video decoding operation to obtain a reconstructed image with the decoded image, and manages the reconstructed image according to a reference image management method of conventional video decoding; The processor combines the reconstructed image decoded by the intelligent image decoding network and the reconstructed image decoded by the traditional video decoding operation in a display order to obtain an output video.

10. A video encoding device based on intelligent image coding, comprising: processor; A memory for storing bitstreams, neural network models, and images to be encoded; as well as One or more procedures are used to accomplish the following: For the image to be encoded, the processor selects one of the following two encoding methods for encoding: 1) The processor encodes the image to be encoded using an intelligent image encoding network to obtain image encoding data, and puts the reconstructed image into a decoded image cache and marks it as a short-term reference image; 2) The processor encodes the image to be encoded using a conventional video encoding operation to obtain image encoding data, and manages the reconstructed image according to a reference image management method of conventional video encoding; The processor puts the image encoding data into the strip data, and puts the type information of the strip data into the header data of the strip data, where the information is used to identify whether the strip data is decoded using an intelligent image decoding network or a traditional video decoding operation; The processor combines the strip data obtained by the intelligent image coding network coding and the strip data obtained by the traditional video coding operation coding in a coding order to obtain an output code stream.

Images