Video decoding method and device, electronic equipment, storage medium and computer program product

Abstract

The invention discloses a video decoding method and device, electronic equipment, a storage medium and a computer program product, and the method comprises the steps: carrying out the intra-frame decoding based on a first code stream and a second code stream of a video, and generating a reference frame list; calling a first deep neural network DNN model to process a first reference frame and a second reference frame corresponding to the current frame selected from the reference frame list to obtain a first prediction frame corresponding to the current frame; updating the reference frame list, the updating of the reference frame list comprising: replacing a third reference frame in the reference frame list with the first prediction frame, the third reference frame representing a reference frame having the largest difference with the POC of the current frame; and generating a second prediction frame corresponding to the current frame based on the updated reference frame list, and reconstructing an original frame corresponding to the current frame in combination with the second prediction frame and residual information decoded from the second code stream.

Description

Technical Field

[0001] The present application relates to the field of video processing, and in particular, to a video decoding method, device, electronic device, storage medium and computer program product. Background Art

[0002] Video analysis is one of the commonly used basic technologies in application scenarios such as security monitoring and autonomous navigation. When human vision tasks such as video reconstruction and playback are required, the traffic will surge when transmitting the video; when machine vision tasks such as target detection are required, the video features are generally compressed and the complete video is not transmitted. When human vision tasks and machine vision tasks need to be performed simultaneously, the existing unified framework is not mature, and the bit rate during video analysis will still increase significantly. Summary of the invention

[0003] To solve the related technical problems, the embodiments of the present application provide a video decoding method, device, electronic device, storage medium and computer program product.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a video decoding method, which is applied to a decoding end, and the method includes:

[0006] Performing intra-frame decoding based on a first bitstream and a second bitstream of the video to generate a reference frame list; the first bitstream includes a bitstream formed by performing intra-frame encoding on one or more original frames of the video; the second bitstream includes a bitstream formed by performing intra-frame encoding and inter-frame encoding on the one or more original frames;

[0007] Calling a first deep neural network (DNN) model to process a first reference frame and a second reference frame corresponding to a current frame selected from the reference frame list to obtain a first predicted frame corresponding to the current frame;

[0008] Updating the reference frame list, wherein updating the reference frame list comprises: replacing a third reference frame in the reference frame list with the first predicted frame, wherein the third reference frame represents a reference frame with the largest difference in image display order POC from the current frame;

[0009] Based on the updated reference frame list, a second predicted frame corresponding to the current frame is generated, and the original frame corresponding to the current frame is reconstructed by combining the second predicted frame and residual information decoded from the second bitstream.

[0010] In the above solution, replacing the third reference frame in the reference frame list with the first predicted frame includes:

[0011] When the third reference frame is in the preceding list, or when the third reference frame is in the preceding list and the succeeding list, replacing the third reference frame in the preceding list with the first prediction frame;

[0012] When the third reference frame is in the subsequent list, replacing the third reference frame in the subsequent list with the first prediction frame;

[0013] The preceding list represents the part of the reference frames in the reference frame list whose POC is smaller than the POC of the current frame; the following list represents the part of the reference frames in the reference frame list whose POC is larger than the POC of the current frame.

[0014] In the above solution, the updating of the reference frame list further includes:

[0015] The first reconstructed frame is stored in the reference frame list; wherein,

[0016] The first reconstructed frame represents a frame having the smallest POC difference with the current frame among all reconstructed frames.

[0017] In the above scheme, a first time distance between the first reference frame corresponding to the current frame and the current frame, and a second time distance between the second reference frame corresponding to the current frame and the current frame do not exceed a set time distance.

[0018] In the above solution, replacing the third reference frame in the reference frame list with the first predicted frame includes:

[0019] In the case that the first predicted frame can be used as a reference frame for other original frames, the third reference frame in the reference frame list is replaced by the first predicted frame.

[0020] In the above solution, the first DNN model is used to process the input first reference frame and the second reference frame using affine transformation parameters; wherein,

[0021] The affine transformation parameters represent parameters decoded from the second code stream and used for motion estimation or motion compensation.

[0022] In the above solution, the first code stream includes:

[0023] The one or more original frames are resized and then intra-frame encoded to form a code stream.

[0024] In the above solution, the second code stream includes:

[0025] The one or more original frames are inter-coded based on the second DNN model to form a bit stream; wherein,

[0026] The second DNN model and the first DNN model represent associated models that are collaboratively trained using the same affine transformation parameters.

[0027] The embodiment of the present application also provides another video decoding method, which is applied to a decoding end, and the method includes:

[0028] In the case of performing the target detection task, decoding a first bitstream of the video, the first bitstream comprising a bitstream formed by intra-frame encoding of one or more original frames of the video; and / or,

[0029] In case of performing a video reconstruction task, the first code stream and the second code stream of the video are decoded, and the second code stream includes a code stream formed by performing intra-frame coding and inter-frame coding on the one or more original frames.

[0030] In the above solution, the first code stream includes:

[0031] The one or more original frames are resized and then intra-frame encoded to form a code stream.

[0032] In the above solution, the second code stream includes:

[0033] The one or more original frames are inter-coded based on the second DNN model to form a bit stream; wherein,

[0034] The second DNN model and the first DNN model represent associated models that are collaboratively trained using the same affine transformation parameters.

[0035] In the above scheme, the method further comprises:

[0036] Sending a first request to the encoder; the first request carries the task type of the video decoding task; wherein, when the video decoding task is a target detection task, the first request is used to request the encoder to send the first code stream; when the video decoding task is a video reconstruction task, the first request is used to request the encoder to send the first code stream and the second code stream; or,

[0037] The first code stream sent by the encoding end is received, or the first code stream and the second code stream sent by the encoding end are received; wherein the code stream sent by the encoding end is determined based on the task type of the video decoding task.

[0038] In the above solution, decoding the first code stream and the second code stream of the video includes:

[0039] The first code stream and the second code stream are decoded using any of the above-mentioned video decoding methods.

[0040] The embodiment of the present application also provides a video decoding device, including:

[0041] A first processing unit is configured to perform intra-frame decoding based on a first bitstream and a second bitstream of a video to generate a reference frame list; the first bitstream includes a bitstream formed by performing intra-frame encoding on one or more original frames of the video; the second bitstream includes a bitstream formed by performing intra-frame encoding and inter-frame encoding on the one or more original frames;

[0042] A second processing unit is configured to call a first deep neural network DNN model to process a first reference frame and a second reference frame corresponding to a current frame selected from the reference frame list to obtain a first predicted frame corresponding to the current frame;

[0043] An updating unit, configured to update the reference frame list, wherein the updating of the reference frame list comprises: replacing a third reference frame in the reference frame list with the first predicted frame, wherein the third reference frame represents a reference frame with the largest difference in image display order POC from the current frame;

[0044] A reconstruction unit is used to generate a second predicted frame corresponding to the current frame based on the updated reference frame list, and reconstruct an original frame corresponding to the current frame by combining the second predicted frame and residual information decoded from the second bitstream.

[0045] The embodiment of the present application also provides another video decoding device, including:

[0046] A first decoding unit and / or a second decoding unit;

[0047] The first decoding unit is used to decode a first bitstream of the video when performing the target detection task, wherein the first bitstream includes a bitstream formed by intra-frame encoding of one or more original frames of the video; and / or,

[0048] The second decoding unit is used to decode the first code stream and the second code stream of the video when performing a video reconstruction task, wherein the second code stream includes a code stream formed by intra-frame encoding and inter-frame encoding of the one or more original frames.

[0049] The embodiment of the present application further provides a first electronic device, comprising: a first processor and a first communication interface; wherein,

[0050] The first processor is configured to perform intra-frame decoding based on a first bitstream and a second bitstream of the video to generate a reference frame list; the first bitstream includes a bitstream formed by performing intra-frame encoding on one or more original frames of the video; the second bitstream includes a bitstream formed by performing intra-frame encoding and inter-frame encoding on the one or more original frames;

[0051] Calling a first deep neural network DNN model to process a first reference frame and a second reference frame corresponding to a current frame selected from the reference frame list to obtain a first predicted frame corresponding to the current frame;

[0052] Updating the reference frame list, wherein updating the reference frame list comprises: replacing a third reference frame in the reference frame list with the first predicted frame, wherein the third reference frame represents a reference frame with the largest difference in image display order POC from the current frame;

[0053] Based on the updated reference frame list, a second predicted frame corresponding to the current frame is generated, and the original frame corresponding to the current frame is reconstructed by combining the second predicted frame and residual information decoded from the second bitstream.

[0054] The embodiment of the present application further provides a second electronic device, including: a second processor and a second communication interface; wherein,

[0055] The second processor is used to decode a first bitstream of the video when performing the target detection task, where the first bitstream includes a bitstream formed by intra-frame encoding of one or more original frames of the video; and / or,

[0056] In case of performing a video reconstruction task, the first code stream and the second code stream of the video are decoded, and the second code stream includes a code stream formed by performing intra-frame coding and inter-frame coding on the one or more original frames.

[0057] The present application also provides an electronic device, including: a processor and a memory for storing a computer program that can be run on the processor.

[0058] Wherein, the processor is used to execute the steps of any one of the above-mentioned video decoding methods when running the computer program.

[0059] An embodiment of the present application further provides a storage medium on which a computer program is stored, wherein when the computer program is executed, the steps of any one of the above-mentioned video decoding methods are implemented.

[0060] An embodiment of the present application further provides a computer program product, including a computer program, characterized in that when the computer program is executed, the steps of any one of the above-mentioned video decoding methods are implemented.

[0061] In a video decoding method, device, electronic device, storage medium and computer program product provided by the embodiment of the present application, when video decoding is performed at the video decoding end, first, intra-frame decoding is performed based on the first bitstream and the second bitstream of the video to generate a reference frame list, and then the reference frame list is updated and inter-frame decoding is performed to reconstruct the original frame corresponding to the current frame; the first bitstream includes a bitstream formed by intra-frame encoding of one or more original frames of the video; the second bitstream includes a bitstream formed by intra-frame encoding and inter-frame encoding of the one or more original frames; specifically, before updating the reference frame list, the first DNN model is called to process the first reference frame and the second reference frame corresponding to the current frame selected from the reference frame list to obtain the first predicted frame corresponding to the current frame; updating the reference frame list includes: replacing the third reference frame in the reference frame list with the first predicted frame, and the third reference frame represents the reference frame with the largest POC difference with the current frame. Based on the first DNN model, the above-mentioned reference frame list can be updated, so the shape of the predicted graphic block can be unrestricted, and no additional block shape needs to be enabled, and no additional flag bit is required, so the transmitted bit stream can be reduced in the video analysis task, and the efficiency of video decoding can be significantly improved. In addition, when performing a target detection task, the decoding end decodes the first bitstream of the video, and when performing a video reconstruction task, the decoding end decodes the first bitstream and the second bitstream of the video, thereby completing the switching of machine vision and human vision tasks, thereby ensuring the efficiency of target detection and meeting the needs of high-quality video reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A schematic diagram of a group of pictures structure of a video encoding and decoding system according to an embodiment of the present application;

[0063] Figure 2 A schematic diagram of a video decoding method according to an embodiment of the present application;

[0064] Figure 3 A schematic diagram of another video decoding method according to an embodiment of the present application;

[0065] Figure 4 This is a flowchart of a third video decoding method according to an embodiment of the present application;

[0066] Figure 5 This is a flowchart of a fourth video decoding method according to an embodiment of the present application;

[0067] Figure 6 This is a flowchart of a fifth video decoding method according to an embodiment of the present application;

[0068] Figure 7 This is a schematic diagram of the structure of a video decoding device according to an embodiment of the present application;

[0069] Figure 8 This is a schematic diagram of the structure of another video decoding device according to an embodiment of the present application;

[0070] Fig. 9 This is a schematic diagram of a first electronic device according to an embodiment of the present application;

[0071] Fig.10 This is a schematic diagram of a second electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0072] In the early video analysis related technologies, a unified framework supporting human vision tasks and machine vision tasks was generally not established. Human vision tasks refer to tasks such as video reconstruction and playback; machine vision tasks refer to tasks such as target detection. For example, conventional video coding standard technologies such as High Efficiency Video Coding (H.265 / HEVC) and Versatile Video Coding (H.266 / VVC) are mainly aimed at human vision rather than machine vision, and do not consider how to apply to machine vision tasks in the future; when only processing intelligent analysis tasks based on machine vision, in order to save bandwidth, the characteristics of the video will be compressed instead of transmitting the complete video, but when the human eye needs to watch, the video still needs to be encoded and decoded, which leads to reduced transmission efficiency.

[0073] In recent years, although some related technologies have proposed some unified frameworks that support human and machine vision, for example, some studies have proposed scalable image compression schemes, that is, using base layer features to perform machine analysis and using additional information to reconstruct images in the enhancement layer, the encoding and decoding schemes for media data in a scalable manner are still in the ongoing research stage, and a unified expansion architecture for efficient content understanding and compression has not yet been formed; some work directly recovers the input image from the features without the need for additional bitstreams. However, the above schemes or technologies have a common disadvantage: focusing on image coding rather than video coding. First, in machine vision tasks, the extraction and analysis of key frames are often crucial, and video codecs introduce temporal redundancy removal technology when processing continuous images to reduce the amount of data. In other words, temporal redundancy removal may affect the integrity of these key frames, so if video codecs are used to process machine vision tasks, additional steps may be required to restore these key frames, resulting in an increase in bit rate; secondly, human vision pays more attention to the smoothness and coherence of the picture, and image codecs may not be able to meet this requirement when processing a single image. In practical applications, the smoothness of the video can be simulated by arranging and playing the image sequence in chronological order, but this will also increase the amount of data and bit rate. Therefore, the above scheme faces the problem of a significant increase in bit rate in video analysis tasks.

[0074] In a video decoding method, device, electronic device, storage medium and computer program product provided by the embodiment of the present application, when video decoding is performed at the video decoding end, first, intra-frame decoding is performed based on the first bitstream and the second bitstream of the video to generate a reference frame list, and then the reference frame list is updated and inter-frame decoding is performed to reconstruct the original frame corresponding to the current frame; the first bitstream includes a bitstream formed by intra-frame encoding of one or more original frames of the video; the second bitstream includes a bitstream formed by intra-frame encoding and inter-frame encoding of the one or more original frames; specifically, before updating the reference frame list, the first DNN model is called to process the first reference frame and the second reference frame corresponding to the current frame selected from the reference frame list to obtain the first predicted frame corresponding to the current frame; updating the reference frame list includes: replacing the third reference frame in the reference frame list with the first predicted frame, and the third reference frame represents the reference frame with the largest POC difference with the current frame. Based on the first DNN model, the above-mentioned reference frame list can be updated, so the shape of the predicted graphic block can be unrestricted, and no additional block shape needs to be enabled, and no additional flag bit is required, so the transmitted bit stream can be reduced in the video analysis task, and the efficiency of video decoding can be significantly improved. In addition, when performing a target detection task, the decoding end decodes the first bitstream of the video, and when performing a video reconstruction task, the decoding end decodes the first bitstream and the second bitstream of the video, thereby completing the switching of machine vision and human vision tasks, thereby ensuring the efficiency of target detection and meeting the needs of high-quality video reconstruction.

[0075] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0076] In order to better explain the embodiments of the present application, it is necessary to briefly explain the group of pictures (GoP) structure of the video encoding system or decoding system used before and after video transmission. Figure 1 As shown, the bit stream of video transmission includes multiple GoPs, each GoP includes multiple frames (or pictures), wherein each GoP includes an intra-coded frame (I frame, Intra-coded Frames), that is, Figure 1 The dark-colored elongated rectangular part in the middle, each GoP also includes one or more subsequent inter-coded frames (Inter-coded Frames), such as Figure 1 The light-colored elongated rectangular part in the middle. Inter-frame coded frames can be forward-predicted frames (P frames, Predictive-coded Frames) or bi-directional predicted frames (B frames, Bi-directional Predicted Frames). Intra-frame prediction and inter-frame prediction can reduce the spatial redundancy and temporal redundancy of video images respectively. Figure 1The role of the base layer is to implement video encoding and decoding for machine tasks, including generating potential representations through intra-frame coding Output the first bitstream; the role of the enhancement layer is to implement video encoding and decoding for human tasks, including generating potential representations through intra-frame coding And after encoding through DNN-based inter-frame prediction, the second bitstream is output. Generate a reconstructed frame I frame, and reconstruct a P frame or B frame based on the inter-frame prediction of the DNN. The potential representation represents the internal representation or intermediate result of the video data obtained by the encoding method.

[0077] The present application embodiment provides a video decoding method, which is applied to a decoding end. Figure 2 , the method comprising:

[0078] Step 201: Perform intra-frame decoding based on the first bitstream and the second bitstream of the video to generate a reference frame list.

[0079] The first code stream includes a code stream formed by performing intra-frame coding on one or more original frames of the video; and the second code stream includes a code stream formed by performing intra-frame coding and inter-frame coding on the one or more original frames.

[0080] Step 202: Call the first DNN model to process the first reference frame and the second reference frame corresponding to the current frame selected from the reference frame list to obtain a first predicted frame corresponding to the current frame.

[0081] Step 203: updating the reference frame list, where updating the reference frame list includes: replacing a third reference frame in the reference frame list with the first predicted frame, where the third reference frame represents a reference frame with the largest difference in image display order POC with the current frame.

[0082] Step 204: Generate a second predicted frame corresponding to the current frame based on the updated reference frame list, and reconstruct the original frame corresponding to the current frame by combining the second predicted frame and residual information decoded from the second bitstream.

[0083] The video decoding method in the embodiment of the present application is applied to the decoding end, and the decoding end can receive the code stream for intra-frame decoding and / or inter-frame decoding. In actual application, in order to identify the intra-frame coded frame, the decoding end can use the information in the network abstraction layer (NAL) header for identification. For example, when parsing the NAL header, the decoding end can check the value of the nal_unit_type field. If the value corresponds to the type of the intra-frame coded frame (I frame), the decoding end can know that the current NAL unit contains intra-frame coded frame (I frame) data.

[0084] In the embodiment of the present application, the decoding end performs intra-frame decoding based on the first bitstream and the second bitstream of the video to generate a reference frame list. The first bitstream includes a bitstream formed by intra-frame encoding of one or more original frames of the video; the second bitstream includes a bitstream formed by intra-frame encoding and inter-frame encoding of the one or more original frames. Different intra-frame encoding methods can be used when forming the first bitstream and the second bitstream.

[0085] In one embodiment, the first code stream includes:

[0086] The code stream is formed by scaling one or more original frames and then performing intra-frame encoding.

[0087] In an embodiment of the present application, one or more original frames of the video may be resized, and then the steps of intra-frame coding such as analysis transformation, quantization, and arithmetic coding may be performed in sequence to form a first bitstream. In other words, the first bitstream may be a compressed bitstream, so the first bitstream may also be referred to as a base layer bitstream. In actual application, the first bitstream may be suitable for machine vision tasks such as target detection. Since one or more original frames are resized when forming the first bitstream, when only the first bitstream is transmitted, the quality of the video can be basically guaranteed, less bandwidth resources are occupied, and machine vision tasks can be supported.

[0088] In the embodiment of the present application, when performing intra-frame coding to form the second bitstream, the original frame may not be resized, and the steps of intra-frame coding such as analysis transformation, quantization and arithmetic coding may be directly performed. Moreover, the second bitstream also includes a bitstream formed by performing inter-frame coding on one or more original frames of the video, that is, the second bitstream contains richer coding information than the first bitstream, so the second bitstream can also be called an enhancement layer bitstream.

[0089] In the embodiment of the present application, intra-frame decoding is performed based on the first bitstream and the second bitstream of the video to generate a reference frame list. In practical applications, the first bitstream and the second bitstream can be arithmetically decoded to obtain a potential representation Y1 corresponding to the first bitstream and a potential representation Y2 corresponding to the second bitstream, and after Y1 and Y2 are synthesized and transformed, a reference frame list is generated and written into a decoding buffer at the decoding end.

[0090] In the embodiment of the present application, after generating the reference frame list, the decoding end may perform inter-frame decoding based on the reference frame list to obtain the original frame corresponding to the current frame.

[0091] In an embodiment of the present application, inter-frame decoding includes: calling a first DNN model to process a first reference frame and a second reference frame corresponding to a current frame selected from the reference frame list to obtain a first predicted frame corresponding to the current frame. In practical applications, the first predicted frame can also be referred to as a deep reference frame generated based on the first reference frame and the second reference frame, that is, the first predicted frame can be understood as a reference frame with better reference quality, richer reference information, and can achieve higher precision inter-frame prediction.

[0092] In an embodiment of the present application, inter-frame decoding also includes: after obtaining the first predicted frame corresponding to the current frame, updating the reference frame list, and updating the reference frame list includes: replacing the third reference frame in the reference frame list with the first predicted frame, and the third reference frame represents the reference frame with the largest POC difference with the current frame. The difference with the POC of the current frame represents the time distance with the current frame, and the larger the POC difference, the farther the time distance, and the smaller the POC difference, the closer the time distance. In actual application, it can also be decided whether to update the reference frame list based on the similarity between the current frame and the reference frame, the coding efficiency and the required bit rate.

[0093] In the embodiment of the present application, based on the updated reference frame list, a second predicted frame corresponding to the current frame is generated, and the original frame corresponding to the current frame is reconstructed by combining the second predicted frame and the residual information decoded from the second bitstream. After the original frame corresponding to the current frame is reconstructed, the inter-frame decoding process corresponding to the current frame is completed, and the decoded frame is written into the decoding buffer, which is further used as a reference frame for other frames.

[0094] In one embodiment, replacing the third reference frame in the reference frame list with the first prediction frame includes:

[0095] When the third reference frame is in the preceding list, or in the preceding list and the succeeding list, the third reference frame in the preceding list is replaced by the first prediction frame;

[0096] When the third reference frame is in the subsequent list, replacing the third reference frame in the subsequent list with the first prediction frame;

[0097] The preceding list represents the part of the reference frames in the reference frame list whose POC is smaller than the POC of the current frame; the following list represents the part of the reference frames in the reference frame list whose POC is larger than the POC of the current frame.

[0098] In the embodiment of the present application, the reference frame may appear only in the preceding list or only in the succeeding list. Further, in practical applications, when the encoding order is different from the display order, or when there are B frames, a frame may appear in both the preceding list and the succeeding list. When the third reference frame is located in the preceding list and the succeeding list, updating the reference frame list is an update in the bidirectional prediction mode.

[0099] In the embodiment of the present application, when the third reference frame is in the preceding list, or when the third reference frame is in the preceding list and the following list, that is, as long as the third reference frame appears in the preceding list, the first prediction frame replaces the third reference frame in the preceding list; when the third reference frame is in the following list, but does not appear in the preceding list, the first prediction frame replaces the third reference frame in the following list. In practical applications, the third reference frame with the farthest time distance from the current frame can be found in the preceding list and the following list. According to the above situation, the third reference frame can be replaced with the first prediction frame with richer reference information, which can more flexibly update the reference frame, optimize the efficiency and bit rate in the video decoding process, and thus improve the quality and stability of video decoding.

[0100] In one embodiment, the updated reference frame list further includes:

[0101] The first reconstructed frame is stored in a reference frame list; wherein,

[0102] The first reconstructed frame represents a frame having the smallest POC difference with the current frame among all reconstructed frames.

[0103] In the embodiment of the present application, the frame with the smallest POC difference from the current frame can be understood as the frame closest to the current frame, and the frame with the smallest POC difference from the current frame among all reconstructed frames is stored in the reference frame list. At this time, updating the reference frame list is an update in the unidirectional prediction mode. By updating the reference list in the unidirectional prediction mode, the reference list can be updated more simply than in the bidirectional prediction mode, with lower complexity and decoding delay.

[0104] In one embodiment, a first time distance between the first reference frame corresponding to the current frame and the current frame, and a second time distance between the second reference frame corresponding to the current frame and the current frame do not exceed a set time distance.

[0105] In an embodiment of the present application, the time distance between the first reference frame and the second reference frame and the current frame can be set to not exceed the set time distance. Exemplarily, the set time distance can be ±2 frames. In actual use, when the time distance between the first reference frame or the second reference frame and the current frame exceeds the set time distance, the default reference frame list can be used without updating the reference frame list. By setting the time distance between the first reference frame and the second reference frame and the current frame, the availability of the obtained first predicted frame can be further guaranteed.

[0106] In one embodiment, replacing the third reference frame in the reference frame list with the first predicted frame includes:

[0107] In the case that the first predicted frame can be used as a reference frame for other original frames, the third reference frame in the reference frame list is replaced by the first predicted frame.

[0108] In the embodiment of the present application, when the first predicted frame can be used as a reference frame for other original frames, the first predicted frame is used to replace the third reference frame in the reference frame list; when the first predicted frame cannot be used as a reference frame for other original frames, the reference frame list may not be updated. By determining in advance whether the first predicted frame can be used as a reference frame for other original frames, it can be ensured that when the reference frame list is updated, the correct first predicted frame is used to replace the third reference frame, thereby preventing the quality of video decoding from being affected when the first predicted frame is unavailable or of poor quality.

[0109] In one embodiment, the second code stream includes:

[0110] A code stream is formed by performing inter-frame coding on one or more original frames based on the second DNN model; wherein,

[0111] The second DNN model and the first DNN model represent associated models that are collaboratively trained using the same affine transformation parameters.

[0112] In an embodiment of the present application, during inter-frame coding, one or more original frames can be inter-coded based on the second DNN model to form a part of the second bitstream. In actual application, the second bitstream can also include a bitstream formed by intra-frame coding, so the second bitstream can also be called an enhancement layer bitstream. The second DNN model for video encoding and the first DNN model for video decoding are associated models that are collaboratively trained using the same affine transformation parameters to facilitate correct video encoding and corresponding video decoding.

[0113] In an embodiment of the present application, the first DNN model and the second DNN model use the same affine transformation parameters and can generate an estimated prediction frame by calculating motion and filter kernels, thereby minimizing the encoded residual signal, reducing the bit stream of video transmission, and improving the efficiency of encoding and decoding.

[0114] In one embodiment, the first DNN model is used to process the input first reference frame and the second reference frame using affine transformation parameters; wherein,

[0115] The affine transformation parameters represent parameters decoded from the second code stream and used for motion estimation or motion compensation.

[0116] In an embodiment of the present application, the first DNN model is called to process the first reference frame and the second reference frame, that is, the affine transformation parameters are used to perform motion estimation or motion compensation, and finally the first predicted frame is obtained. In practical applications, the estimated affine transformation parameters can be used to perform local convolution operations on the first reference frame and / or the second reference frame to generate geometrically deformed image blocks, and then multiple motion-compensated blocks are merged through a synthesis network to form the final first predicted frame. By calling the first DNN model, that is, using the affine transformation parameters to process the input first reference frame and the second reference frame, it is beneficial to generate a first predicted frame with better quality.

[0117] The present application provides a video decoding method, which is applied to a decoding end. The method includes:

[0118] In the case of performing the target detection task, decoding a first bitstream of the video, the first bitstream comprising a bitstream formed by intra-frame encoding of one or more original frames of the video; and / or,

[0119] In case of performing a video reconstruction task, the first code stream and the second code stream of the video are decoded, and the second code stream includes a code stream formed by performing intra-frame coding and inter-frame coding on the one or more original frames.

[0120] In the embodiments of the present application, the target detection task refers to the detection task that can be performed by terminal devices such as cameras, such as the detection of unattended luggage and the detection of safety hazards; the video reconstruction task is to reconstruct the video of the scene for human vision so that humans can play it back and watch it. By receiving different decoding tasks, different bit streams can be decoded intra-frame or inter-frame to facilitate switching between target detection and video reconstruction tasks. In actual application, the target detection task can also be referred to as a machine vision task; the video reconstruction task can also be referred to as a human vision task. Through the process of transmission and decoding of switchable videos of machine vision and human vision, it is possible to perform video detection or analysis tasks and view videos of areas of interest, which not only ensures the efficiency of video detection or analysis tasks, but also meets the diverse needs of users.

[0121] In one embodiment, the first code stream includes:

[0122] The one or more original frames are resized and then intra-coded to form a code stream.

[0123] In one embodiment, the second code stream includes:

[0124] The code stream is formed by performing inter-frame coding on the one or more original frames based on the second DNN model; wherein,

[0125] The second DNN model and the first DNN model represent associated models that are collaboratively trained using the same affine transformation parameters.

[0126] In one embodiment, the method further comprises:

[0127] Sending a first request to the encoder; the first request carries the task type of the video decoding task; wherein, when the video decoding task is a target detection task, the first request is used to request the encoder to send the first code stream; when the video decoding task is a video reconstruction task, the first request is used to request the encoder to send the first code stream and the second code stream; or,

[0128] The first code stream sent by the encoding end is received, or the first code stream and the second code stream sent by the encoding end are received; wherein the code stream sent by the encoding end is determined based on the task type of the video decoding task.

[0129] In an embodiment of the present application, the decoding end may send a first request to the encoding end, and the encoding end may send different bitstreams for different video decoding tasks. For example, a terminal device such as a camera performs a target detection task under normal circumstances, and when a special situation is detected, it performs a video reconstruction task, that is, receives a first bitstream and a second bitstream, and then the manager can view the reconstructed live video and decide when to start the target detection task. By receiving different bitstreams for different video decoding tasks, the decoding end may occupy less bandwidth resources under normal circumstances to ensure the basic quality of the video, and may receive a second bitstream under special circumstances to obtain a better frame rate, resolution or video quality.

[0130] In one embodiment, decoding a first bitstream and a second bitstream of a video includes:

[0131] The first code stream and the second code stream are decoded by using any of the above-mentioned video decoding methods.

[0132] In the embodiment of the present application, using any of the above-mentioned video decoding methods to decode the first bit stream and the second bit stream helps to enrich the video decoding method, not only supporting target detection tasks, but also supporting video reconstruction tasks for human viewing.

[0133] The present application is further described in detail below in conjunction with application examples.

[0134] In the first application example of the present application, the entire end-to-end video encoding and decoding system architecture is described, including a model front end, a model back end, an intra-frame encoder module, an intra-frame decoder module, an inter-frame encoder module, and an inter-frame decoder module. Figure 3 As shown, in the first application embodiment of the present application, the intra-frame encoder module is referred to as module A for detailed description, the intra-frame decoding module is referred to as module B for detailed description, the inter-frame encoder is referred to as module C for detailed description, and the inter-frame decoder is referred to as module D for detailed description.

[0135] In module A, intra-frame coding of the video can be performed in terminal devices such as cameras. In order to obtain the base layer code stream (first code stream), step A1 can be performed; in order to obtain the enhancement layer code stream (second code stream), step A2 can be performed. The following is a detailed description of step A1 and step A2:

[0136] In step A1, the input source is the collected original video stream data x, which is input into the model front end to generate feature identifier s, where the model front end can be the first few layers of the backbone network of the machine vision model; secondly, the frame (or picture) in the original video stream data x is resized and used together with the feature identifier s as the input of the analysis and transformation module of the base layer to generate the potential representation y1 of the base layer. Next, the potential representation y1 is input into the quantization module to obtain the quantized data Finally, the quantified data The data is input into the arithmetic encoder module for entropy coding to generate a base layer inter-frame bit stream (also called a base layer code stream).

[0137] The following method can be used to generate the feature identifier s: the submodule at the front end of the model can generate C using the first 13 consecutive layers of the YOLOv3 backbone network of the machine vision model. s = 256-channel feature s, whose spatial dimension is H s =H / 8 and W s =W / 8, where H represents the height of the input image, W represents the width of the input image, and H s Represents the height of the feature map, W s Represents the width of the feature map. For example, the input source data x∈R 3×H×W , which can generate features Among them, 3 represents the three color channels of RGB.

[0138] The analysis transformation of the base layer can be performed by the following method: the channel concatenation of feature s and Resize(x) is used as input to generate the basic potential representation y1, where the Resize operation is to resize the original video frame to the new size H s ×W s , the interpolation method selected in the Resize operation is to scale the image by selecting a spatial bicubic interpolation filter.

[0139] Among them, the quantization module or arithmetic encoder module of the base layer can round the potential representation y1 to the nearest integer value to obtain the quantized data Exemplarily, the latent representation Y is rounded to the nearest integer value to obtain an integer-valued latent representation Right now Among them, Ω(Y) is the calculation method in the quantization module or the arithmetic encoder module, and rounding includes rounding, rounding up, and rounding down. Furthermore, the rounding operation can be simulated by adding uniformly distributed random noise during the training process to obtain the potential representation of additive noise. They are similar to This noise-adding operation ensures that the decoder will learn to reconstruct the data from the degraded latent representation that occurs during inference and is a differentiable operation.

[0140] In step A2, only the original video stream data x is used as the input of the analysis transformation module of the enhancement layer to generate the potential representation y2 of the enhancement layer. Then, the potential representation y2 is input into the quantization module to obtain the quantized data Finally, the quantified data The data is input into the arithmetic encoder module for entropy coding to generate an enhancement layer inter-frame bit stream (also called enhancement layer code stream).

[0141] In order to transmit the bit stream, the encoder can be trained according to the actual scene to generate the structured latent space, and then transmitted to the machine vision decoding end to reconstruct the decoding end. Specifically, the formula can be used in, Contains the information needed by the base layer, Contains enhancement layer information, and Reconstruct the input frame together. For example, Of the 192 channels, 128 are allocated to The remaining 64 are allocated to One of the two decoders is for machine vision and the other is for human viewing, so that the required information can be reconstructed efficiently. In order to ensure that the information related to object detection is concentrated in the first j channels of Y (j=128), it is necessary to train the entire end-to-end intra-frame model from scratch. By performing gradient-based updates on various loss terms, the model learns to import object detection related information into the first 128 channels, and learns to use the entire Y to reconstruct the input image.

[0142] In module B, intra-frame decoding of the video can be performed in the cloud or edge server, wherein in step B1, the base layer code stream can be processed; in step B2, the enhancement layer code stream can be processed. Steps B1 and B2 are described in detail below:

[0143] In step B1, the base layer bitstream is fed into an arithmetic decoder to reconstruct the base layer latent representation And use the synthetic transformation module to reconstruct the feature representation The latent space is then transformed and input into the back-end module of the machine vision model to generate inference results for machine vision tasks.

[0144] In step B2, the enhancement layer bitstream is fed into an arithmetic decoder to reconstruct the latent representation of the enhancement layer Will and After being input into the synthesis transformation module together, use and The channel cascade of the enhanced layer synthesizes the transform to reconstruct the input

[0145] Among them, in order to identify intra-frame coding, the Network Abstraction Layer (NAL) header is used with 2 bytes in traditional HEVC; in the header data, the reserved 6-bit word that supports HEVC scalability is also reused to distinguish the layer ID of task scalability.

[0146] Among them, the latent space reconstruction is completed, and the intra-frame bit stream is decoded to obtain the latent representation of the image high-dimensional features. When only the base layer of the intra-frame coded frame is received, After reconstruction, it is sent to the target detection pipeline (latent space transform (LST) + YOLOv3 backend) for target detection; when the enhancement layer of the intra-coded frame is received, it can be reconstructed and

[0147] Among them, By performing LST transformation, the purpose of transforming the size of the feature tensor from the reconstructed basic features to the feature tensor expected input by YOLOv3 can be achieved. For example, the transformation formula is:

[0148]

[0149] in, It is a feature tensor generated in the target latent space. In the network structure, it is generally composed of a transposed convolution layer (whose purpose is to match the spatial resolution of the target latent space), an inverse generalized divisive normalization (IGDN) layer, and a sequence of convolution layers. Since the feature tensor expected by YOLOv3 is The basic features of reconstruction in the embodiment of the present application are Therefore, a transformation from one latent space to another is needed.

[0150] Among them, at the back end of the model, the potential space of the encoder can be mapped to the potential space of the target detection network. For example, the formula can be used, Ο(·) is the backend of the target detection network, which produces the detection output T. In practical applications, the 13th layer of YOLOv3 can be selected as the target detection potential space, so Ο(·) consists of all YOLOv3 layers after the 13th layer, especially the part from the batch normalization input of the 12th layer to the model output.

[0151] Among them, in the enhanced layer synthesis transformation module, a synthesis transformation is performed on the potential representation of the high-dimensional image features to restore them to the high-dimensional image features, that is, from the complete potential representation Reconstruct the input frame, and the decoding formula is: where D is the decoder, is an approximation of the original input frame x. The architecture of the decoder D can be obtained from the corresponding end-to-end codec of the encoder. The reconstructed frame is registered in the decoded picture buffer (DPB) for use as an inter-frame reference.

[0152] In module C, the inter-frame coding of the video can also be performed in terminal devices such as cameras. The inter-frame coding in the embodiments of the present application can integrate DNN-based affine frame prediction into the coding process. Specifically, it can be divided into step C1, step C2, step C3, step C4, and step C5.

[0153] In step C1, perform depth frame prediction based on bidirectional affine transformation. Specifically, the frame prediction network takes two previously encoded frames and as inputs and predicts an intermediate frame between the input frames, where t1 < t < t2; then, the frame prediction network estimates the affine transformation information between the two input frames and the desired output frame, as well as the adaptive filter kernel. By calculating the motion and filter kernel, the network generates an estimated prediction frame to minimize the encoded residual signal.

[0154] In step C2, generate the reference signal. Specifically, calculate the difference between the original frame and the predicted frame to generate a residual signal, which will be used in the subsequent coding process.

[0155] In step C3, perform the coding process. Specifically, first perform transform coding on the residual signal, such as discrete cosine transform; secondly, quantize the transformed coefficients to reduce the data volume; finally, use entropy coding techniques (such as Huffman coding or arithmetic coding) to further compress the data.

[0156] In step C4, generate the bitstream. Specifically, assemble the encoded residual data, affine transformation parameters, and possible other coding information (such as quantization parameters) into a bitstream.

[0157] Step C5 is about the coding buffer. Specifically, after the encoder completes the coding of a frame, it combines the predicted frame and the coding residual to generate a reconstructed frame. The previously encoded and reconstructed frames are stored in the reference frame buffer of the encoder.

[0158] In module D, inter-frame decoding of the video can also be performed in a server in the cloud or edge. Specifically, it can be divided into step D1, step D2, step D3, step D4, step D5 and step D6.

[0159] In step D1, a code stream is received. Specifically, the decoder receives and identifies an inter-frame coded frame code stream.

[0160] In step D2, for the inter-coded frames, the encoded bit stream is parsed by an entropy decoding module.

[0161] In step D3, a reconstructed residual signal is generated through inverse quantization (arithmetic decoder) and synthesis transformation modules.

[0162] In step D4, an affine transformation based on the depth frame prediction is performed. Specifically, the decoder performs an inverse transformation on the reference frame according to the decoded affine transformation parameters to generate a predicted frame.

[0163] In step D5, the reference frame is updated. Specifically, in the related art, the frame generated by DNN is updated. Replace one of the frames to update the current reference list, and then use that frame as any other reference frame for mutual prediction. At this time, a flag is set during the encoding process so that the decoder knows to perform motion compensation from the square blocks in the frame generated by the DNN. However, the above-mentioned related technologies also limit the use of DNN frame information to square blocks, and enabling additional block shapes will require additional flags. In the embodiment of the present application, in order to overcome the limitations of the above-mentioned direct mode, the process shown by the dotted line from the "affine transformation based on depth frame prediction" module to the "decoding buffer" module is supplemented. The predictor can decide whether to update the reference list and how to select the reference frame based on the similarity between the current frame and the reference frame, the coding efficiency and the required bit rate. The update of the reference list is divided into two modes: unidirectional prediction and bidirectional prediction.

[0164] In the bidirectional prediction mode, illustratively, as Figure 4 As shown, you can set it to be performed only when the maximum time distance from the current frame is ±2 frames, so that you can create Otherwise the default reference list is used without updating. When available, first find the decoded frame with the largest POC difference with the current frame (the farthest reference frame) in the two lists. Three situations are possible: (1) the frame is in the preceding list; (2) the frame is in both the preceding list and the following list; (3) the frame is in the following list. In the first two cases, the frame in the preceding list is replaced by the frame generated by the DNN. In the third case, the frames in the subsequent list are replaced by Replacement. In some special cases, the same frame may appear in both lists, such as when the encoding order is different from the display order, or when there are B frames, a frame may appear in both the forward and backward reference lists of the current frame. In this case, the decoder needs to handle the update of the reference list specially to ensure that the available reference information is used correctly.

[0165] In unidirectional prediction mode, previously decoded frames are placed in the reference list starting from the frame closest to the current frame. Therefore, updating the reference list in unidirectional prediction is relatively simple.

[0166] In step D6, when there is no available reference frame to create the DNN predicted frame When , the mode can be directly marked as unavailable. Finally, after compensating a block from the predicted frame generated by the DNN, the decoded residual signal can be added to the predicted frame to reconstruct the original frame.

[0167] In the first application embodiment of the present application, the DNN-based neural network can generate a prediction frame including an affine transformation, so there is no need to transmit additional motion information. At the same time, unidirectional prediction and bidirectional prediction can be realized, which reduces the bit stream of video transmission and significantly improves the coding efficiency.

[0168] In the embodiments of the present application, an application embodiment 2 is also proposed, which can perform end-to-end model training on the model in the above-mentioned end-to-end video encoding and decoding system architecture, which can be specifically divided into step E1, step E2, step E3 and step E4.

[0169] In step E1, the model training of the intra-frame codec is performed. Specifically, the structure of the encoder can be taken from any end-to-end trainable image codec, and is achieved by retraining all components in the encoder except the object detection backend. The loss function used in this step is as follows:

[0170]

[0171] Where R is the estimated bit rate, which can be obtained using the above entropy estimation model; λ is a parameter that controls the trade-off between rate and distortion; is an approximation of the original input frame X; It is a feature tensor generated in the target latent space; is the feature tensor corresponding to the original input frame X; the parameter γ controls the potential space distortion of target detection and the pixel domain distortion of the input reconstruction The trade-off between.

[0172] In step E2, the backend network access is performed, that is, in order to use the pre-trained YOLOv3 backend to evaluate the target detection performance, the input resolution is adjusted to 512×512 using bilinear interpolation.

[0173] In step E3, the model training of the inter-frame codec is performed. Specifically, in the encoder network design, a deep neural network with a lightweight U-Net structure is adopted to extract multi-scale features through dilated convolution. Optionally, it is implemented through a spatial transformer network (STN, Spatial Transformer Network) or a similar mechanism, which can perform spatial affine transformation on the input feature map; in the decoder network design, using the estimated affine transformation parameters, the model performs local convolution operations on the reference frame to generate geometrically deformed image blocks, and merges multiple motion compensated blocks through a synthesis network (such as GridNet) to form the final predicted frame. Among them, the multi-scale mean square error (MSE) loss function can be used, and the formula is as follows:

[0174]

[0175] Among them, s indicates the scale, which is usually 1, 2 or 4, P 1 / s represents the image block generated at a scale of 1 / s (vertically and horizontally), Represents a real image patch at the same scale.

[0176] In step E4, post-processing is performed. Specifically, a loop filter may be applied to smooth the block boundaries of the reconstructed frame, reduce visible block effects, and improve visual quality.

[0177] In the embodiment of the present application, an application embodiment 3 is also proposed. In this application embodiment, the process is refined in the base layer and the enhancement layer intra-frame codec, which is specifically reflected in the use of a super-prior module and a side information bit stream. Figure 5 , which is a schematic diagram of the operation structure of the neural network compression model of the present application, in which arrows represent data flow and boxes represent data conversion. The left side of the box marked with ·|· represents the uniform noise added during training (generating vectors marked with waves), and the right side represents round quantization and arithmetic coding / decoding during inference (generating vectors marked with hats).

[0178] In intra-frame coding, a super priori entropy model can be used. The super priori entropy model includes a super priori analysis transformation module, a super priori synthesis transformation module, and a context module. The three work together to achieve a more accurate probability estimate of the distribution of potential representations by adding auxiliary information, reducing coding redundancy, corresponding to Figure 5The super-a priori analysis transformation, context analysis, super-a priori synthesis transformation, and bit rate estimation module on the upper side. The network structure of the super-a priori entropy model can be composed of a sequence of convolutional layers, etc. The super-a priori entropy model can include the following four modules:

[0179] Module 1: Transformation of hyper-prior analysis.

[0180] This module is used to capture and preserve global dependencies in the latent representation related to the side information. This application embodiment builds a super prior network on top of the decomposition entropy model to transform the latent representation into a super latent representation z. More specifically, in the analysis transformation module, each element of the latent representation is modeled as a zero-mean Gaussian distribution, whose standard deviation is predicted from the latent distribution by the super prior block.

[0181] Module 2: Contextual Analysis.

[0182] This module is based on an autoregressive entropy coding model of latent variables, which looks at the decoded neighboring pixels and predicts the distribution of the next pixel based on this context. It is designed to capture local dependencies in the image that are directly adjacent to the currently predicted latent variable, such as edges or textures. In order to (t) To encode, we can use Y (0) ,Y (1) ,...,Y (t-1) (i.e., the previous context). For practical applications, the super-latent representation must be included in the compressed bitstream together with the main compressed image data.

[0183] Module 3: Super-Prior Synthetic Transform

[0184] This module is used for the decompression process. The super-prior decoder uses the stored super-latent representation to reconstruct the mean and scale parameter (entropy parameter) of the Gaussian distribution of each element in the quantized latent representation.

[0185] Module 4: Bitrate Estimation

[0186] Through the above modules 1-3, the entropy model is constructed using Gaussian mixture likelihood, which is By analyzing the probability of occurrence of each element in the data To predict the code rate required to encode these data and reduce the required coding bits. For example, the formula for calculating the probability is:

[0187]

[0188] Among them, ω (k) is the scale parameter of the kth element, μ (k) is the Gaussian distribution mean of the kth element, σ (k) is the Gaussian distribution variance of the kth element, The symbols are represented as Gaussian distribution. Through the above four modules, more accurate probability estimation can be achieved and coding redundancy can be reduced.

[0189] In the embodiment of the present application, by adopting the super-prior module and the side information bit stream, the process is refined in the base layer and the enhancement layer intra-frame codec, which can reduce the spatial redundancy in the video, reduce the bit rate, and significantly improve the efficiency of video compression and overall encoding.

[0190] In the embodiments of the present application, an application embodiment 4 is also proposed. In this application embodiment, after completing the training of the end-to-end intra-frame and inter-frame models through the above-mentioned embodiment 2, the sending end and receiving end models can be obtained, and through the cloud-based collaborative video transmission and intelligent analysis process, the machine vision mode and the human vision display mode can be optionally switched.

[0191] Specifically, the transmitter can output multiple layers of code streams after one encoding, which are divided into a base layer and an enhancement layer, suitable for different terminals and network conditions. The base layer occupies relatively less bandwidth resources to ensure basic video quality. The base layer plus the enhancement layer can obtain a better frame rate, resolution or quality. After adding the enhancement layer, a higher frame rate can be obtained, and the video is smoother. The encoder at the transmitter can receive different layers of video information under certain bandwidth rate restrictions according to the needs of the receiver, so as to obtain the best video reception effect. Figure 6 As shown, the processing flow of this embodiment is as follows:

[0192] Step 1: After encoding and transmission, the sender sends the base layer code stream to the receiver;

[0193] Step 2: The receiver identifies intra-coded frames and inter-coded frames through the NAL layer and the two bytes of traditional HEVC, and sends them to the corresponding decoder deep neural network. Under normal circumstances, the receiver performs machine vision tasks;

[0194] Step 3: When a specific situation or user request is detected, the receiving end will send a video playback instruction to the sending end;

[0195] Step 4: The sender packages the base layer and enhancement layer bit streams in the next frame together and sends them to the cloud;

[0196] Step 5: The user can review the reconstructed video and decide when to switch to machine vision-only mode;

[0197] Step 6: The receiving end requests the video base layer again and requests to perform the video processing task;

[0198] Step 7: The sender sends the base layer bit stream in the next frame to the cloud;

[0199] Step 8: If no specific event is detected, the receiving end sends a video processing instruction to the sending end;

[0200] Step 9: Send to continue transmitting the base layer bit stream, and repeat this process;

[0201] Step 10: If the network condition is poor, the sender only sends the base layer bit stream.

[0202] In the application embodiment of the present application, under normal circumstances, only the machine vision task is active. For example, cloud-based video analysis can automatically monitor airport lounges through surveillance cameras, which also belongs to the public monitoring scenario in the field of visual networking. When this embodiment is applied, terminal devices such as cameras send the base layer bitstream to the cloud, and perform target detection tasks through the base layer. When a situation of interest is detected (for example, unattended luggage), the cloud will request the enhancement layer, and the base layer and enhancement layer bitstreams in the next frame (together with subsequent inter-frame code streams) will be packaged and sent to the cloud. Security personnel can then view the reconstructed live video and decide when to switch to machine vision mode only. Compared with traditional video encoding and decoding methods, it can be effectively applied to scenes with machine vision as the main task, while also having the ability to reconstruct high-quality videos that meet human viewing needs. The potential representation realizes the separation of base layer and enhancement layer information, enhances the interpretability of the network, reduces repeated transmission of features, and improves the accuracy of video analysis and the quality of video reconstruction.

[0203] In the application embodiment of the present application, further, under the cloud-based collaborative framework of the visual network, the switchable video encoding and decoding process of human vision and machine vision allows users to both perform video analysis tasks and view videos of areas of interest, thereby ensuring the efficiency of video analysis tasks and meeting the diverse needs of users.

[0204] The applications of the embodiments of the present application include but are not limited to the fields of public area monitoring, traffic monitoring, smart home security, etc.

[0205] In order to implement the video decoding method of the embodiment of the present application, the embodiment of the present application also provides a video decoding device, such as Figure 7 As shown, the device comprises:

[0206] The first processing unit 701 is configured to perform intra-frame decoding based on a first bitstream and a second bitstream of a video to generate a reference frame list; the first bitstream includes a bitstream formed by performing intra-frame coding on one or more original frames of the video; the second bitstream includes a bitstream formed by performing intra-frame coding and inter-frame coding on the one or more original frames;

[0207] The second processing unit 702 is used to call the first deep neural network DNN model to process the first reference frame and the second reference frame corresponding to the current frame selected from the reference frame list to obtain a first predicted frame corresponding to the current frame;

[0208] An updating unit 703 is configured to update the reference frame list, wherein the updating of the reference frame list includes: replacing a third reference frame in the reference frame list with the first predicted frame, wherein the third reference frame represents a reference frame having the largest difference in image display order POC with the current frame;

[0209] The reconstruction unit 704 is configured to generate a second predicted frame corresponding to the current frame based on the updated reference frame list, and reconstruct an original frame corresponding to the current frame by combining the second predicted frame and residual information decoded from the second bitstream.

[0210] In one embodiment, the updating unit 703 is configured to replace the third reference frame in the preceding list with the first prediction frame when the third reference frame is in the preceding list or when the third reference frame is in the preceding list and the succeeding list; and replace the third reference frame in the succeeding list with the first prediction frame when the third reference frame is in the succeeding list.

[0211] The preceding list represents the part of the reference frames in the reference frame list whose POC is smaller than the POC of the current frame; the following list represents the part of the reference frames in the reference frame list whose POC is larger than the POC of the current frame.

[0212] In one embodiment, the updating unit 703 is further configured to store the first reconstructed frame in the reference frame list; wherein,

[0213] The first reconstructed frame represents a frame having the smallest POC difference with the current frame among all reconstructed frames.

[0214] In one embodiment, a first time distance between the first reference frame corresponding to the current frame and the current frame, and a second time distance between the second reference frame corresponding to the current frame and the current frame do not exceed a set time distance.

[0215] In one embodiment, replacing the third reference frame in the reference frame list with the first predicted frame includes:

[0216] In the case that the first predicted frame can be used as a reference frame for other original frames, the third reference frame in the reference frame list is replaced by the first predicted frame.

[0217] In one embodiment, the first DNN model is used to process the input first reference frame and the second reference frame using affine transformation parameters; wherein,

[0218] The affine transformation parameters represent parameters decoded from the second code stream and used for motion estimation or motion compensation.

[0219] In one embodiment, the first code stream includes:

[0220] The one or more original frames are resized and then intra-coded to form a code stream.

[0221] In one embodiment, the second code stream includes:

[0222] The code stream is formed by performing inter-frame coding on the one or more original frames based on the second DNN model; wherein,

[0223] The second DNN model and the first DNN model represent associated models that are collaboratively trained using the same affine transformation parameters.

[0224] In actual application, the above units can be implemented by a processor in a video decoding device.

[0225] It should be noted that: the video decoding device provided in the above embodiment is only illustrated by the division of the above program modules. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the above-described processing. In addition, the video decoding device provided in the above embodiment and the video decoding method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0226] In order to implement the video decoding method of the embodiment of the present application, the embodiment of the present application also provides a video decoding device, such as Figure 8 As shown, the device comprises:

[0227] A first decoding unit 801 and / or a second decoding unit 802;

[0228] The first decoding unit 801 is used to decode a first bitstream of a video when performing a target detection task, where the first bitstream includes a bitstream formed by intra-frame encoding of one or more original frames of the video; and / or,

[0229] The second decoding unit 802 is used to decode the first code stream and the second code stream of the video when performing a video reconstruction task, where the second code stream includes a code stream formed by performing intra-frame coding and inter-frame coding on the one or more original frames.

[0230] In one embodiment, the first code stream includes:

[0231] The one or more original frames are resized and then intra-coded to form a code stream.

[0232] In one embodiment, the second code stream includes:

[0233] The code stream is formed by performing inter-frame coding on the one or more original frames based on the second DNN model; wherein,

[0234] The second DNN model and the first DNN model represent associated models that are collaboratively trained using the same affine transformation parameters.

[0235] In one embodiment, the device further comprises:

[0236] A sending unit 803 or a receiving unit 804;

[0237] The sending unit 803 is configured to send a first request to the encoder; the first request carries the task type of the video decoding task; wherein, when the video decoding task is a target detection task, the first request is used to request the encoder to send the first code stream; when the video decoding task is a video reconstruction task, the first request is used to request the encoder to send the first code stream and the second code stream;

[0238] The receiving unit 804 is configured to receive the first code stream sent by the encoding end, or receive the first code stream and the second code stream sent by the encoding end; wherein the code stream sent by the encoding end is determined based on the task type of the video decoding task.

[0239] In one embodiment, decoding the first bitstream and the second bitstream of the video includes:

[0240] The first code stream and the second code stream are decoded using any of the above video decoding methods.

[0241] In practical applications, the above-mentioned units can be implemented by a processor or a communication interface in a video decoding device.

[0242] It should be noted that: the video decoding device provided in the above embodiment is only illustrated by the division of the above program modules. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the above-described processing. In addition, the video decoding device provided in the above embodiment and the video decoding method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0243] Based on the hardware implementation of the above program module, the embodiment of the present application also provides a first electronic device, such as Fig. 9 As shown, the first electronic device 900 includes:

[0244] The first communication interface 901 is capable of exchanging information with other network nodes;

[0245] The first processor 902 is connected to the first communication interface 901 to implement information exchange with other network nodes, and is used to execute the method provided by one or more technical solutions when running a computer program. The computer program is stored in the first memory 903.

[0246] When the first processor 902 runs the computer program to perform the above one or more video decoding methods, specifically:

[0247] The first processor 902 is configured to:

[0248] Performing intra-frame decoding based on a first bitstream and a second bitstream of the video to generate a reference frame list; the first bitstream includes a bitstream formed by performing intra-frame coding on one or more original frames of the video; the second bitstream includes a bitstream formed by performing intra-frame coding and inter-frame coding on the one or more original frames;

[0249] Calling a first deep neural network DNN model to process a first reference frame and a second reference frame corresponding to a current frame selected from the reference frame list to obtain a first predicted frame corresponding to the current frame;

[0250] Updating the reference frame list, wherein updating the reference frame list includes: replacing a third reference frame in the reference frame list with the first predicted frame, wherein the third reference frame represents a reference frame with the largest difference in image display order POC from the current frame;

[0251] Based on the updated reference frame list, a second predicted frame corresponding to the current frame is generated, and the original frame corresponding to the current frame is reconstructed by combining the second predicted frame and residual information decoded from the second bitstream.

[0252] In one embodiment, replacing the third reference frame in the reference frame list with the first predicted frame includes:

[0253] When the third reference frame is in the preceding list, or when the third reference frame is in the preceding list and the succeeding list, the third reference frame in the preceding list is replaced by the first prediction frame;

[0254] When the third reference frame is in the subsequent list, replacing the third reference frame in the subsequent list with the first prediction frame;

[0255] The preceding list represents the part of the reference frames in the reference frame list whose POC is smaller than the POC of the current frame; the following list represents the part of the reference frames in the reference frame list whose POC is larger than the POC of the current frame.

[0256] In one embodiment, replacing the third reference frame in the reference frame list with the first predicted frame includes:

[0257] When the third reference frame is in the preceding list, or when the third reference frame is in the preceding list and the succeeding list, the third reference frame in the preceding list is replaced by the first prediction frame;

[0258] When the third reference frame is in the subsequent list, replacing the third reference frame in the subsequent list with the first prediction frame;

[0259] The preceding list represents the part of the reference frames in the reference frame list whose POC is smaller than the POC of the current frame; the following list represents the part of the reference frames in the reference frame list whose POC is larger than the POC of the current frame.

[0260] In one embodiment, updating the reference frame list further includes:

[0261] The first reconstructed frame is stored in the reference frame list; wherein,

[0262] The first reconstructed frame represents a frame having the smallest POC difference with the current frame among all reconstructed frames.

[0263] In one embodiment, a first time distance between the first reference frame corresponding to the current frame and the current frame, and a second time distance between the second reference frame corresponding to the current frame and the current frame do not exceed a set time distance.

[0264] In one embodiment, replacing the third reference frame in the reference frame list with the first predicted frame includes:

[0265] In the case that the first predicted frame can be used as a reference frame for other original frames, the third reference frame in the reference frame list is replaced by the first predicted frame.

[0266] In one embodiment, the first DNN model is used to process the input first reference frame and the second reference frame using affine transformation parameters; wherein,

[0267] The affine transformation parameters represent parameters decoded from the second code stream and used for motion estimation or motion compensation.

[0268] In one embodiment, the first code stream includes:

[0269] The one or more original frames are resized and then intra-coded to form a code stream.

[0270] In one embodiment, the second code stream includes:

[0271] The code stream is formed by performing inter-frame coding on the one or more original frames based on the second DNN model; wherein,

[0272] The second DNN model and the first DNN model represent associated models that are collaboratively trained using the same affine transformation parameters.

[0273] It should be noted that the specific processing process of the first processor 902 and the first communication interface 901 can be understood by referring to the above method.

[0274] Of course, in actual application, the various components in the first electronic device 900 are coupled together through the bus system 904. It can be understood that the bus system 904 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 904 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Fig. 9 Various buses are labeled as bus system 904.

[0275] The first memory 903 in the embodiment of the present application is used to store various types of data to support the operation of the first electronic device 900. Examples of such data include: any computer program used to operate on the first electronic device 900.

[0276] The method disclosed in the above embodiment of the present application can be applied to the first processor 902, or implemented by the first processor 902. The first processor 902 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the first processor 902 or an instruction in the form of software. The above-mentioned first processor 902 may be a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The first processor 902 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in the first memory 903, and the first processor 902 reads the information in the first memory 903, and completes the steps of the above method in combination with its hardware.

[0277] In an exemplary embodiment, the first electronic device 900 can be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD), field programmable gate array (FPGA), general processor, controller, microcontroller (MCU), microprocessor, or other electronic components to execute the aforementioned method.

[0278] Based on the hardware implementation of the above program module, the embodiment of the present application also provides a second electronic device, such as Fig.10 As shown, the second electronic device 1000 includes:

[0279] The second communication interface 1001 is capable of exchanging information with other network nodes;

[0280] The second processor 1002 is connected to the second communication interface 1001 to implement information exchange with other network nodes, and is used to execute the method provided by one or more technical solutions when running a computer program. The computer program is stored in the second memory 1003.

[0281] Specifically, the second processor 1002 is used to decode a first bitstream of the video when the video decoding task is a target detection task, where the first bitstream includes a bitstream formed by intra-coding one or more original frames of the video; and / or,

[0282] In the case that the video decoding task is a video reconstruction task, the first code stream and the second code stream of the video are decoded, and the second code stream includes a code stream formed by performing intra-frame encoding and inter-frame encoding on the one or more original frames.

[0283] In one embodiment, the first code stream includes:

[0284] The one or more original frames are resized and then intra-frame encoded to form a code stream.

[0285] In one embodiment, the second code stream includes:

[0286] The one or more original frames are inter-coded based on the second DNN model to form a bit stream; wherein,

[0287] The second DNN model and the first DNN model represent associated models that are collaboratively trained using the same affine transformation parameters.

[0288] In one embodiment, the second communication interface 1001 is used for:

[0289] Sending a first request to the encoder; the first request carries the task type of the video decoding task; wherein, when the video decoding task is a target detection task, the first request is used to request the encoder to send the first code stream; when the video decoding task is a video reconstruction task, the first request is used to request the encoder to send the first code stream and the second code stream; or,

[0290] The first code stream sent by the encoding end is received, or the first code stream and the second code stream sent by the encoding end are received; wherein the code stream sent by the encoding end is determined based on the task type of the video decoding task.

[0291] In one embodiment, the second processor 1002 is configured to:

[0292] The first code stream and the second code stream are decoded using any of the above video decoding methods.

[0293] It should be noted that the specific processing process of the second processor 1002 and the second communication interface 1001 can be understood by referring to the above method.

[0294] Of course, in actual application, the various components in the second electronic device 1000 are coupled together through the bus system 1004. It can be understood that the bus system 1004 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1004 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Fig.10 Various buses are labeled as bus system 1004 .

[0295] The method disclosed in the above embodiment of the present application can be applied to the second processor 1002, or implemented by the second processor 1002. The second processor 1002 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the second processor 1002. The above-mentioned second processor 1002 may be a general-purpose processor, DSP, or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The second processor 1002 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in the second memory 1003, and the second processor 1002 reads the information in the second memory 1003 and completes the steps of the above method in combination with its hardware.

[0296] In an exemplary embodiment, the second electronic device 1000 may be implemented by one or more ASICs, DSPs, PLDs, CPLDs, FPGAs, general processors, controllers, MCUs, Microprocessors, or other electronic components to perform the aforementioned methods.

[0297] It can be understood that the memory (first memory 903, second memory 1003) of the embodiment of the present application can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM).The memories described in the embodiments of the present application are intended to include, but are not limited to, these and any other suitable types of memories.

[0298] In an exemplary embodiment, the embodiment of the present application further provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, for example, including a first memory 903 storing a computer program, and the computer program can be executed by the first processor 902 of the first electronic device 900 to complete the steps of the aforementioned video decoding method. For another example, a second memory 1003 storing a computer program can be executed by the second processor 1002 of the second electronic device 1000 to complete the steps of the aforementioned video decoding method. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disk, or CD-ROM.

[0299] Exemplarily, the embodiment of the present application further provides a computer program product, including a computer program, which can be executed by the first processor 902 of the first electronic device 900 to complete the steps of the aforementioned video decoding method. Alternatively, the computer program can be executed by the second processor 1002 of the second electronic device 1000 to complete the steps of the aforementioned video decoding method.

[0300] It should be noted that: "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0301] The term "one or more" herein means any combination of at least two of any one or more of a plurality, for example, including at least one of A, B, and C, and may mean any one or more elements selected from the set consisting of A, B, and C.

[0302] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0303] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A video decoding method, applied to a decoding end, characterized in that: The method comprises: Performing intra-frame decoding based on a first bitstream and a second bitstream of the video to generate a reference frame list; the first bitstream includes a bitstream formed by performing intra-frame encoding on one or more original frames of the video; the second bitstream includes a bitstream formed by performing intra-frame encoding and inter-frame encoding on the one or more original frames; Calling a first deep neural network DNN model to process a first reference frame and a second reference frame corresponding to a current frame selected from the reference frame list to obtain a first predicted frame corresponding to the current frame; Updating the reference frame list, wherein updating the reference frame list comprises: replacing a third reference frame in the reference frame list with the first predicted frame, wherein the third reference frame represents a reference frame with the largest difference in image display order POC from the current frame; Based on the updated reference frame list, a second predicted frame corresponding to the current frame is generated, and the original frame corresponding to the current frame is reconstructed by combining the second predicted frame and residual information decoded from the second bitstream.

2. The method according to claim 1, characterized in that The replacing the third reference frame in the reference frame list with the first predicted frame comprises: When the third reference frame is in the preceding list, or when the third reference frame is in the preceding list and the succeeding list, replacing the third reference frame in the preceding list with the first prediction frame; When the third reference frame is in the subsequent list, replacing the third reference frame in the subsequent list with the first prediction frame; The preceding list represents the part of the reference frames in the reference frame list whose POC is smaller than the POC of the current frame; the following list represents the part of the reference frames in the reference frame list whose POC is larger than the POC of the current frame.

3. The method according to claim 1 or 2, characterized in that: The updating of the reference frame list further comprises: The first reconstructed frame is stored in the reference frame list; wherein, The first reconstructed frame represents a frame having the smallest POC difference with the current frame among all reconstructed frames.

4. The method according to claim 1 or 2, characterized in that: A first time distance between the first reference frame corresponding to the current frame and the current frame, and a second time distance between the second reference frame corresponding to the current frame and the current frame do not exceed a set time distance.

5. The method according to claim 1 or 2, characterized in that: The replacing the third reference frame in the reference frame list with the first predicted frame comprises: In the case that the first predicted frame can be used as a reference frame for other original frames, the third reference frame in the reference frame list is replaced by the first predicted frame.

6. The method according to claim 1 or 2, characterized in that: The first DNN model is used to process the input first reference frame and the second reference frame using affine transformation parameters; wherein, The affine transformation parameters represent parameters decoded from the second code stream and used for motion estimation or motion compensation.

7. The method according to claim 1 or 2, characterized in that: The first code stream includes: The one or more original frames are resized and then intra-frame encoded to form a code stream.

8. The method according to claim 1 or 2, characterized in that: The second code stream includes: The one or more original frames are inter-coded based on the second DNN model to form a bit stream; wherein, The second DNN model and the first DNN model represent associated models that are collaboratively trained using the same affine transformation parameters.

9. A video decoding method, applied to a decoding end, characterized in that: The method comprises: In the case of performing the target detection task, decoding a first bitstream of the video, the first bitstream comprising a bitstream formed by intra-frame encoding of one or more original frames of the video; and / or, In case of performing a video reconstruction task, the first code stream and the second code stream of the video are decoded, and the second code stream includes a code stream formed by performing intra-frame coding and inter-frame coding on the one or more original frames.

10. The method according to claim 9, characterized in that The first code stream includes: The one or more original frames are resized and then intra-frame encoded to form a code stream.

11. The method according to claim 9, characterized in that The second code stream includes: The one or more original frames are inter-coded based on the second DNN model to form a bit stream; wherein, The second DNN model and the first DNN model represent associated models that are collaboratively trained using the same affine transformation parameters.

12. The method according to any one of claims 9 to 11, characterized in that: The method further comprises: Sending a first request to the encoder; the first request carries the task type of the video decoding task; wherein, when the video decoding task is a target detection task, the first request is used to request the encoder to send the first code stream; when the video decoding task is a video reconstruction task, the first request is used to request the encoder to send the first code stream and the second code stream; or, The first code stream sent by the encoding end is received, or the first code stream and the second code stream sent by the encoding end are received; wherein the code stream sent by the encoding end is determined based on the task type of the video decoding task.

13. The method according to any one of claims 9 to 11, characterized in that: The decoding of the first code stream and the second code stream of the video includes: The first code stream and the second code stream are decoded using the video decoding method according to any one of claims 1 to 8.

14. A video decoding device, characterized in that: include: A first processing unit, configured to perform intra-frame decoding based on a first bitstream and a second bitstream of the video to generate a reference frame list; The first code stream includes a code stream formed by performing intra-frame coding on one or more original frames of the video; the second code stream includes a code stream formed by performing intra-frame coding and inter-frame coding on the one or more original frames; A second processing unit is configured to call a first deep neural network DNN model to process a first reference frame and a second reference frame corresponding to a current frame selected from the reference frame list to obtain a first predicted frame corresponding to the current frame; An updating unit, configured to update the reference frame list, wherein the updating of the reference frame list comprises: replacing a third reference frame in the reference frame list with the first predicted frame, wherein the third reference frame represents a reference frame with the largest difference in image display order POC from the current frame; A reconstruction unit is used to generate a second predicted frame corresponding to the current frame based on the updated reference frame list, and reconstruct an original frame corresponding to the current frame by combining the second predicted frame and residual information decoded from the second bitstream.

15. A video decoding device, characterized in that: include: A first decoding unit and / or a second decoding unit; The first decoding unit is used to decode a first bitstream of the video when performing the target detection task, where the first bitstream includes a bitstream formed by intra-frame encoding of one or more original frames of the video; and / or, The second decoding unit is used to decode the first code stream and the second code stream of the video when performing a video reconstruction task, wherein the second code stream includes a code stream formed by intra-frame encoding and inter-frame encoding of the one or more original frames.

16. A first electronic device, characterized in that: include: A first processor and a first communication interface; wherein, The first processor is configured to perform intra-frame decoding based on a first bitstream and a second bitstream of the video to generate a reference frame list; the first bitstream includes a bitstream formed by performing intra-frame encoding on one or more original frames of the video; the second bitstream includes a bitstream formed by performing intra-frame encoding and inter-frame encoding on the one or more original frames; Calling a first deep neural network DNN model to process a first reference frame and a second reference frame corresponding to a current frame selected from the reference frame list to obtain a first predicted frame corresponding to the current frame; Updating the reference frame list, wherein updating the reference frame list comprises: replacing a third reference frame in the reference frame list with the first predicted frame, wherein the third reference frame represents a reference frame with the largest difference in image display order POC from the current frame; Based on the updated reference frame list, a second predicted frame corresponding to the current frame is generated, and the original frame corresponding to the current frame is reconstructed by combining the second predicted frame and residual information decoded from the second bitstream.

17. A second electronic device, characterized in that: include: A second processor and a second communication interface; wherein, The second processor is used to decode a first bitstream of the video when performing the target detection task, where the first bitstream includes a bitstream formed by intra-frame encoding of one or more original frames of the video; and / or, In case of performing a video reconstruction task, the first code stream and the second code stream of the video are decoded, and the second code stream includes a code stream formed by performing intra-frame coding and inter-frame coding on the one or more original frames.

18. An electronic device, characterized in that: include: a processor and a memory for storing a computer program capable of being executed on the processor, Wherein, when the processor is used to run the computer program, it executes the steps of the method described in any one of claims 1 to 8, or executes the steps of the method described in any one of claims 9 to 13.

19. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the steps of the method according to any one of claims 1 to 8 are implemented, or the steps of the method according to any one of claims 9 to 13 are implemented.

20. A computer program product, comprising a computer program, characterized in that When the computer program is executed, the steps of the method according to any one of claims 1 to 8 are implemented, or the steps of the method according to any one of claims 9 to 13 are implemented.

Images