Video encoding and decoding method and device based on quality grading

Abstract

The present application discloses a video encoding and decoding method, comprising: i Compress to get the video frame P i The base layer image and M enhancement layer images, where the video frame P i is any frame other than the first frame of the video, M is an integer greater than 0; the video frame P is obtained according to the feedback information of the decoding device i The feedback information is used to indicate the information of the video frame received by the decoding device, according to the video frame P i The reference frame of the video frame P i The base layer image and M enhancement layer images are encoded to obtain a video frame P i The first code stream of the video frame P is obtained by obtaining the reference frame determined by the decoding device according to the information of the video frame. i Encode and get the video frame P i The code stream enables the decoding device to decode the video frame P even when the network bandwidth fluctuates and packet loss occurs. i The code stream is decoded normally to produce the projection screen, thereby ensuring that the projection device can play the projection screen smoothly.

Description

Technical Field

[0001] The present application relates to the field of scalable video coding and decoding, and in particular to a video coding and decoding method and device based on quality grading. Background Art

[0002] In recent years, with the rapid development of mobile internet, consumer electronics have seen explosive growth, and the demand for multi-screen interaction has become increasingly urgent. Wireless short-range projection, a fundamental capability for multi-screen interaction, provides a crucial technical means for creating a "1+8" intelligent, full-scenario experience by embedding smartphones into smart devices such as in-vehicle terminals, smart large screens, computer tablets, and smartwatches.

[0003] Multi-screen interaction requires real-time transmission of the video image from the current terminal to other terminals. During video transmission, due to varying network bandwidth conditions, terminal processing capabilities, and user quality requirements in different scenarios, the server encoding process requires multiple encodings. Scalable video coding (SVC) technology can generate adaptive video compression streams with different frame rates, resolutions, and image qualities through temporal, spatial, and quality grading in a single encoding. It has become an extension of the Advanced Video Coding (AVC) and High Efficiency Video Coding (HEVC) standards.

[0004] In scalable video coding, scalability refers to the adjustability of the bitrate. This means that while video data is compressed only once, it can be decoded at multiple frame rates, spatial resolutions, or video quality levels, thus supporting the diverse application requirements of various user types. Real-world network conditions are also complex and variable. Scenarios such as weak signal coverage, home Wi-Fi penetrating walls, and multiple users occupying public Wi-Fi can lead to sudden packet loss and network congestion, resulting in incomplete video data and resulting in image freezes, impacting the user experience. Summary of the Invention

[0005] The embodiment of the present application provides a video encoding and decoding method and device based on quality classification, which obtains the reference frame of the current frame image through the video frame information received by the decoding device, and then encodes and decodes the current frame image according to the reference frame of the current frame image. In this way, even when there is a risk of increased delay or packet loss in the projection screen due to large fluctuations in network bandwidth, it can be ensured that the terminal side can decode the projection screen normally. At the same time, it also improves the compression efficiency of the video stream and reduces chip power consumption.

[0006] In a first aspect, the present application provides a video encoding method based on quality grading, comprising:

[0007] For video frame P i Compress to get the video frame P i The base layer image and M enhancement layer images, where the video frame P i is any frame other than the first frame of the video, M is an integer greater than 0; the video frame P is obtained according to the feedback information of the decoding device i The feedback information is used to indicate the information of the video frame received by the decoding device, according to the video frame P i The reference frame of the video frame P i The base layer image and M enhancement layer images are encoded to obtain a video frame P i The first stream.

[0008] The reference frame determined by obtaining the information of the video frame from the decoding device is used to determine the video frame P. i Encode and get the video frame P i The code stream enables the decoding device to decode the video frame P even when the network bandwidth fluctuates and packet loss occurs. i The code stream is decoded normally to produce the projection screen, thereby ensuring that the projection device can play the projection screen smoothly.

[0009] In a feasible embodiment, the feedback information includes at least one video frame number and a layer number corresponding to each video frame number in the at least one video frame number, and the layer number corresponding to each video frame number is the layer number of the basic layer image or the enhanced layer image of the video frame indicated by the video frame number.

[0010] In a feasible embodiment, the feedback information includes the video frame number i-1 and the corresponding layer number W, where the layer number W is the number of the video frame P. i-1 The layer number of the image with the highest quality among the base layer images and M enhancement layer images obtained by the decoding device, the video frame P i-1 The video frame indicated by the video frame number i-1 is obtained according to the feedback information of the decoding device. i-1 The reference frame includes:

[0011] According to the video frame P i-1 The reconstructed frame of the base layer image and the image with layer number W among the M enhancement layer images is determined as the video frame P i reference frame.

[0012] Among them, the video frame P i-1 With video frame P i are adjacent images in the video stream, and the video frame P i-1 The timestamp is located at the video frame P i Before the timestamp.

[0013] By taking the reconstructed frame of the image corresponding to the frame number i-1 and the layer number W as the video frame Pi The reference frame enables the decoding device to decode a high-quality reconstructed frame.

[0014] In a feasible embodiment, the feedback information includes the video frame number i-1 and the corresponding layer number W, where the layer number W is the number of the video frame P. i-1 The layer number of the image with the highest quality among the base layer images and M enhancement layer images obtained by the decoding device, the video frame P i-1 The video frame indicated by the video frame number i-1 is obtained according to the feedback information of the decoding device. i The reference frame includes:

[0015] The first reconstruction frame queue Q i-1 , belongs to the video frame P i-1 The reconstructed frames of the images with layer numbers higher than layer number W are deleted to obtain the first reconstructed frame queue Q i ; Among them, the first reconstruction frame queue Q i-1 Including the video frame P obtained by the decoding device i-1 The reconstructed frame of the base layer image and the reconstructed frame of the enhancement layer image of the previous video frame, and the video frame P i-1 a reconstructed frame of the base layer image and a reconstructed frame of each enhancement layer image in the M enhancement layer images;

[0016] According to the video frame P i The reference relationship is from the first reconstructed frame queue Q i Get the video frame P i Reference frame; where the video frame P i The reference relationship is the video frame P i With the video frame P i The correspondence between the reference frames used during encoding and decoding.

[0017] By introducing the reconstructed frames of multiple video frames as reference frames, a decoding device can decode high-quality reconstructed frames.

[0018] In a feasible embodiment, according to the video frame P i The reference frame of the video frame P i The base layer image and M enhancement layer images are encoded to obtain a video frame P i After the first code stream, the method of the present application further includes:

[0019] According to the video frame P i The reference frame of the video frame P i The first code stream is decoded to obtain the video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each enhancement layer image in the M enhancement layer images; the video frame P iThe reconstructed frame of the base layer image and the reconstructed frame of each enhancement layer image in the M enhancement layer images are saved to the first reconstructed frame queue Q i middle.

[0020] By converting the video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each enhancement layer image in the M enhancement layer images are saved to the first reconstructed frame queue Q i In the subsequent encoding, the required reconstructed frame can be selected from the reconstructed frame queue as the reference frame for the subsequent video frame.

[0021] In a feasible embodiment, the feedback information includes the video frame P i The frame number and corresponding layer number of the previous video frame, the layer number corresponding to video frame j is j The layer number of the image with the highest quality among the base layer images and M enhancement layer images obtained by the decoding device, the video frame P j is the video frame indicated by the video frame number j, video frame P j is the video frame P i Any frame in the previous video frame, obtain the video frame P according to the feedback information of the decoding device i The reference frame includes:

[0022] According to the video frame P i The reference relationship is to obtain the video frame P from the first reconstruction frame queue i Candidate reconstructed frames; where the video frame P i The reference relationship is the video frame P i With the video frame P i The correspondence between the reference frames used in encoding and decoding; according to the video frame P i The frame number and corresponding layer number of the previous video frame are obtained from the video frame P i The candidate reconstructed frame is used to obtain the video frame P i The reference frame;

[0023] Among them, the video frame P i The reference frames include video frame P i Among the candidate reconstructed frames, the quality of the image corresponding to any reconstructed frame F is not higher than the quality of the image indicated by the layer number corresponding to the video frame number of the video frame to which the reconstructed frame F belongs.

[0024] In a feasible embodiment, according to the video frame P i The reference frame of the video frame P i The base layer image and M enhancement layer images are encoded to obtain a video frame P i After the first code stream, the method of the present application further includes:

[0025] According to the video frame Pi The reference frame of the video frame P i The first code stream is decoded to obtain the video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each enhancement layer image in the M enhancement layer images; the video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each enhancement layer image in the M enhancement layer images are saved in a first reconstructed frame queue.

[0026] In a feasible embodiment, the method of the present application further includes:

[0027] The video frame P i The first code stream is sent to the decoding device.

[0028] In a second aspect, an embodiment of the present application provides a video decoding method based on quality grading, including:

[0029] Get video frame P i The second stream;

[0030] According to the video frame P i The reference frame of the video frame P i The second code stream is decoded to obtain the video frame P i The first reconstructed frame and N second reconstructed frames; wherein the video frame P i The first reconstructed frame is the video frame P i The reconstructed frame of the base layer image, the video frame P i The second reconstructed frame is the video frame P i The reconstructed frame of the enhancement layer image, N is an integer greater than or equal to 0.

[0031] In one possible embodiment, the video frame P i The reference frame includes the video frame P indicated by the video frame number i-1. i-1 The reconstructed frame of the base layer image and the image with layer number W in the enhancement layer image;

[0032] Wherein, the layer number W is the video frame P obtained by the decoding device i-1 The layer number of the highest quality image among the N enhancement layer images corresponding to the N second reconstructed frames.

[0033] In one possible embodiment, the video frame P i The reference frame is based on the video frame P i The reference relationship is from the second reconstructed frame queue Q i-1 Obtained from

[0034] Among them, the video frame P i The reference relationship is the video frame P i With the video frame P iThe correspondence between the reference frames used in encoding and decoding, the second reconstructed frame queue Q i-1 Including the video frame P obtained by the decoding device i The reconstructed frame of the base layer image of the previous video frame and the reconstructed frame of each enhancement layer image in the N enhancement layer images.

[0035] In a feasible embodiment, the method of the present application further includes:

[0036] The video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each enhancement layer image in the N enhancement layer images are saved to the second reconstructed frame queue Q i-1 To obtain the second reconstructed frame queue Q i .

[0037] In a feasible embodiment, the method of the present application further includes:

[0038] A feedback message is sent to the encoding device, where the feedback message includes the video frame number i-1 and the layer number W.

[0039] In a third aspect, the present application provides a video encoding device, comprising:

[0040] Compression unit, used to compress video frame P i Compress to get the video frame P i The base layer image and M enhancement layer images, where the video frame P i is any non-first frame of the video, and M is an integer greater than 0;

[0041] The acquisition unit is used to obtain the video frame P according to the feedback information of the decoding device. i The feedback information is used to indicate information of the video frame received by the decoding device;

[0042] Coding unit, for encoding the video frame P i The reference frame of the video frame P i The base layer image and M enhancement layer images are encoded to obtain a video frame P i The first stream.

[0043] In a feasible embodiment, the feedback information includes at least one video frame number and a layer number corresponding to each video frame number in the at least one video frame number, and the layer number corresponding to each video frame number is the layer number of the basic layer image or the enhanced layer image of the video frame indicated by the video frame number.

[0044] In a feasible embodiment, the feedback information includes the video frame number i-1 and the corresponding layer number W, where the layer number W is the number of the video frame P. i-1The layer number of the image with the highest quality among the base layer images and M enhancement layer images obtained by the decoding device, the video frame P i-1 The video frame indicated by the video frame number i-1, the acquisition unit is specifically used to:

[0045] According to the video frame P i-1 The reconstructed frame of the base layer image and the image with layer number W among the M enhancement layer images is determined as the video frame P i reference frame.

[0046] In a feasible embodiment, the feedback information includes the video frame number i-1 and the corresponding layer number W, where the layer number W is the number of the video frame P. i-1 The layer number of the image with the highest quality among the base layer images and M enhancement layer images obtained by the decoding device, the video frame P i-1 The video frame indicated by the video frame number i-1, the acquisition unit is specifically used to:

[0047] The first reconstruction frame queue Q i-1 , belongs to the video frame P i-1 The reconstructed frames of the images with layer numbers higher than layer number W are deleted to obtain the first reconstructed frame queue Q i ; Among them, the first reconstruction frame queue Q i-1 Including the video frame P obtained by the decoding device i-1 The reconstructed frame of the base layer image and the reconstructed frame of the enhancement layer image of the previous video frame, and the video frame P i-1 a reconstructed frame of the base layer image and a reconstructed frame of each enhancement layer image in the M enhancement layer images;

[0048] According to the video frame P i The reference relationship is from the first reconstructed frame queue Q i Get the video frame P i Reference frame; where the video frame P i The reference relationship is the video frame P i With the video frame P i The correspondence between the reference frames used during encoding and decoding.

[0049] In a feasible embodiment, according to the video frame P i The reference frame of the video frame P i The base layer image and M enhancement layer images are encoded to obtain a video frame P i After the first code stream, the video encoding device further includes:

[0050] Reconstruction unit, for i The reference frame of the video frame P i The first code stream is decoded to obtain the video frame Pi a reconstructed frame of the base layer image and a reconstructed frame of each enhancement layer image in the M enhancement layer images;

[0051] Saving unit, used to save the video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each enhancement layer image in the M enhancement layer images are saved to the first reconstructed frame queue Q i middle.

[0052] In a feasible embodiment, the feedback information includes the video frame P i The frame number and corresponding layer number of the previous video frame, the layer number corresponding to the video frame number j is for the video frame P j The layer number of the image with the highest quality among the base layer images and M enhancement layer images obtained by the decoding device, the video frame P j is the video frame indicated by the video frame number j, video frame P j is the video frame P i For any frame in the previous video frame, the acquisition unit is specifically used to:

[0053] According to the video frame P i The reference relationship is to obtain the video frame P from the first reconstruction frame queue i Candidate reconstructed frames; where the video frame P i The reference relationship is the video frame P i With the video frame P i The correspondence between the reference frames used in encoding and decoding; according to the video frame P i The frame number and corresponding layer number of the previous video frame are obtained from the video frame P i The candidate reconstructed frame is used to obtain the video frame P i The reference frame;

[0054] Among them, the video frame P i The reference frames include video frame P i Among the candidate reconstructed frames, the quality of the image corresponding to any reconstructed frame F is not higher than the quality of the image indicated by the layer number corresponding to the video frame number of the video frame to which the reconstructed frame F belongs.

[0055] In a feasible embodiment, according to the video frame P i The reference frame of the video frame P i The base layer image and M enhancement layer images are encoded to obtain a video frame P i After the first code stream, the video encoding device further includes:

[0056] Reconstruction unit, for i The reference frame of the video frame P i The first code stream is decoded to obtain the video frame P ia reconstructed frame of the base layer image and a reconstructed frame of each enhancement layer image in the M enhancement layer images;

[0057] Saving unit, used to save the video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each enhancement layer image in the M enhancement layer images are saved in a first reconstructed frame queue.

[0058] In a feasible embodiment, the video encoding device further includes:

[0059] The sending unit is used to send the video frame P i The first code stream is sent to the decoding device.

[0060] In a fourth aspect, an embodiment of the present application provides a video decoding device, including:

[0061] Acquisition unit, used to obtain video frame P i The second stream;

[0062] Decoding unit, for decoding according to the video frame P i The reference frame of the video frame P i The second code stream is decoded to obtain the video frame P i The first reconstructed frame and N second reconstructed frames; wherein the video frame P i The first reconstructed frame is the video frame P i The reconstructed frame of the base layer image, the video frame P i The second reconstructed frame is the video frame P i The reconstructed frame of the enhancement layer image, N is an integer greater than or equal to 0.

[0063] In one possible embodiment, the video frame P i The reference frame includes the video frame P indicated by the video frame number i-1. i-1 The reconstructed frame of the base layer image and the image with layer number W in the enhancement layer image;

[0064] Wherein, the layer number W is the video frame P obtained by the decoding device i-1 The layer number of the highest quality image among the N enhancement layer images corresponding to the N second reconstructed frames.

[0065] In one possible embodiment, the video frame P i The reference frame is based on the video frame P i The reference relationship is from the second reconstructed frame queue Q i-1 Obtained from

[0066] Among them, the video frame P i The reference relationship is the video frame P i With the video frame P iThe correspondence between the reference frames used in encoding and decoding, the second reconstructed frame queue Q i-1 Including the video frame P obtained by the decoding device i The reconstructed frame of the base layer image of the previous video frame and the reconstructed frame of each enhancement layer image in the N enhancement layer images.

[0067] In a feasible embodiment, the video decoding device further includes:

[0068] Saving unit, used to save the video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each enhancement layer image in the N enhancement layer images are saved to the second reconstructed frame queue Q i-1 To obtain the second reconstructed frame queue Q i .

[0069] In a feasible embodiment, the video decoding device further includes:

[0070] The sending unit is configured to send a feedback message to the encoding device, where the feedback message includes the video frame number i-1 and the layer number W.

[0071] In a fifth aspect, an embodiment of the present application further provides a video encoding device, including:

[0072] A memory and a processor coupled to the memory, wherein the memory stores instructions, and when the processor executes the instructions, part or all of the first aspect is executed.

[0073] In a sixth aspect, an embodiment of the present application further provides a video decoding device, including:

[0074] A memory and a processor coupled to the memory, wherein the memory stores instructions, and when the processor executes the instructions, part or all of the method in the second aspect is executed.

[0075] In the seventh aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected by lines; the interface circuit is used to receive a signal from the memory of the electronic device and send the signal to the processor, the signal including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes part or all of the method described in the first aspect or the second aspect.

[0076] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes part or all of the method described in the first aspect or the second aspect.

[0077] In the ninth aspect, an embodiment of the present application provides a computer program product, characterized in that it includes computer instructions, which, when executed on an electronic device, enable the electronic device to execute part or all of the method described in the first aspect or the second aspect.

[0078] These and other aspects of the present application will become more readily apparent from the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0080] Figure 1a A schematic diagram of an application scenario provided in an embodiment of the present application;

[0081] Figure 1b A schematic diagram of an application scenario provided in an embodiment of the present application;

[0082] Figure 1c A schematic diagram of an application scenario provided in an embodiment of the present application;

[0083] Figure 2 A schematic diagram of a system architecture provided in an embodiment of the present application;

[0084] Figure 3 A schematic diagram of a video encoding method according to an embodiment of the present invention;

[0085] Figure 4 A schematic diagram of a base layer and an enhancement layer of an image provided in an embodiment of the present application;

[0086] Figure 5a A reference relationship diagram provided for an embodiment of the present application;

[0087] Figure 5b Another reference relationship diagram provided for an embodiment of the present application;

[0088] Figure 6 A schematic diagram of a video decoding method according to an embodiment of the present invention;

[0089] Figure 7 Another reference relationship diagram provided for an embodiment of the present application;

[0090] Figure 8 Schematic diagram for effect comparison;

[0091] Figure 9 Another reference relationship diagram provided for an embodiment of the present application;

[0092] Figure 10 Another reference relationship diagram provided for an embodiment of the present application;

[0093] Figure 11 A schematic diagram of the structure of a video encoding device provided in an embodiment of the present application;

[0094] Figure 12 A schematic diagram of the structure of a video decoding device provided in an embodiment of the present application;

[0095] Figure 13 A schematic diagram of the structure of another video encoding device provided in an embodiment of the present application;

[0096] Figure 14 A schematic diagram of the structure of another video decoding device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0097] The embodiments of the present application are described below with reference to the accompanying drawings.

[0098] First, the key terms in this application are explained.

[0099] A video frame can be divided into a base layer image and at least one enhancement layer image. The base layer image provides the basic image information of the video frame and can be encoded and decoded independently. The at least one enhancement layer image provides more detailed information of the video frame, and the encoding and decoding of the enhancement layer images requires the base layer image. The number of enhancement layer images is not fixed; the higher the number of layers, the higher the quality of the enhancement layer images.

[0100] Quantization parameter (QP): The compression coefficient used when compressing video frames. Generally, the smaller the quantization parameter, the higher the quality of the compressed video frame.

[0101] The basic layer image and enhanced layer image of a video frame are obtained by compressing the video frame using different quantization coefficients, wherein the quantization coefficient used when compressing the video frame to obtain the basic layer image is greater than the quantization coefficient used when compressing the video frame to obtain the enhanced layer image.

[0102] The quality of an image in this application is measured based on the structural similarity (SSIM) and peak signal to noise ratio (PSNR) of the image. Generally, the higher the SSIM and PSNR scores of an image, the higher the image quality.

[0103] SSIM: It is an indicator used to measure the similarity between the original image and the distorted image. The structural similarity between the two can be regarded as an image quality measurement indicator of the distorted image.

[0104] PSNR: It is an objective standard for evaluating images.

[0105] The reference frame in this application is an image frame referenced when encoding a video frame and decoding the code stream of the video frame. The reconstructed frame is an image frame obtained by decoding the code stream of the video frame, and the image frame is the reconstructed frame of the video frame.

[0106] See also Figure 1a , Figure 1a This is a schematic diagram of an application scenario provided by an embodiment of the present application. Figure 1a As shown, the application scenario includes a first terminal device 101 and a second terminal device 102.

[0107] Among them, the first terminal device 101 can be a smart phone, a laptop computer, a smart display or other devices with a display screen; the second terminal device 102 can also be a smart phone, a laptop computer, a smart display screen or other devices with a display screen.

[0108] When the content displayed on the display interface of the first terminal device 101 is projected to the second terminal device 102, the encoding device in the first terminal device 101 obtains the video frame P in real time. i , and according to the encoding method of this application, the video frame P i The base layer image and M enhancement layer images are encoded to obtain a video frame P i The code stream of the video frame P i The code stream is sent to the second terminal device 102; the decoding device of the second terminal device 102 decodes the received code stream according to the decoding method of the present application to obtain the video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each image in the N enhancement layer images, N is an integer greater than or equal to 0 and less than or equal to M, and the second terminal device 102 then selects the video frame P according to its display requirements. i Obtain a target reconstructed frame from the reconstructed frames of the base layer image and the N reconstructed frames of the enhancement layer images, and display the target reconstructed frame, thereby realizing screen projection of the content displayed on the display interface of the first terminal device 101 to the second terminal device 102;

[0109] When the content displayed on the display interface of the second terminal device 102 is projected to the first terminal device 101, the encoding device in the second terminal device 102 obtains the video frame P in real time. i , and according to the encoding method of this application, the video frame P iThe base layer image and M enhancement layer images are encoded to obtain a video frame P i The code stream of the video frame P i The code stream is sent to the second terminal device 102; the decoding device of the first terminal device 101 decodes the received code stream according to the decoding method of the present application to obtain the video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each image in the N enhancement layer images, N is an integer greater than or equal to 0 and less than or equal to M, and the first terminal device 101 then selects the video frame P from the video frame P according to its display requirements. i The target reconstructed frame is obtained from the reconstructed frames of the basic layer image and the N reconstructed frames of the enhanced layer images, and the target reconstructed frame is displayed, thereby realizing the projection of the content displayed on the display interface of the second terminal device 102 to the first terminal device 101.

[0110] The first terminal device 101 and the second terminal device 102 both include the encoding device and the decoding device of the present application.

[0111] See also Figure 1b , Figure 1b This is another application scenario diagram provided by the embodiment of the present application. Figure 1b As shown, the application scenario includes a first terminal device 101, multiple second terminal devices 102, an encoding device 103 and a decoding device 104.

[0112] The first terminal device 101 sends the video to be projected to the encoding device 103 in real time. The encoding device 103 obtains the video frame P i According to the encoding method of this application, the video frame P i The base layer image and M enhancement layer images are encoded to obtain a video frame P i The code stream of the video frame P i The code stream is sent to the decoding device 104; the decoding device 104 decodes the received code stream according to the decoding method of the present application to obtain the video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each image in the N enhancement layer images, where N is an integer greater than or equal to 0 and less than or equal to M; since the display requirements of the plurality of second terminal devices 102 are different, the decoding device 104 then decodes the video frame P according to the respective requirements of the plurality of second terminal devices 102. i Target reconstructed frames of multiple second terminal devices 102 are selected from the reconstructed frames of the basic layer image and the reconstructed frames of N enhanced layer images, and the multiple second terminal devices 102 display their respective target reconstructed frames, thereby realizing multi-screen projection.

[0113] See also Figure 1c , Figure 1cThis is another application scenario diagram provided by the embodiment of the present application. Figure 1c As shown, the application scenario includes a first terminal device 101 and multiple second terminal devices 102;

[0114] The encoding device 103 of the first terminal device 101 obtains the video frame P i According to the encoding method of this application, the video frame P i The base layer image and M enhancement layer images are encoded to obtain a video frame P i The code stream of the video frame P i The code stream is sent to multiple second terminal devices 102; the decoding device of the second terminal device 102 decodes the received code stream according to the decoding method of the present application to obtain the video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each image in the N enhancement layer images, where N is an integer greater than or equal to 0 and less than or equal to M; the second terminal device 102 then selects the video frame P from the video frame P according to its display requirements. i The target reconstructed frame is obtained from the reconstructed frames of the basic layer image and the N reconstructed frames of the enhanced layer images, and the target reconstructed frame is displayed, thereby realizing the projection of the content displayed on the display interface of the first terminal device 101 to multiple second terminal devices 102.

[0115] The following describes a screen projection system architecture. Figure 2 , Figure 2 A schematic diagram of a system architecture provided in an embodiment of the present application. Figure 2 As shown, first capture the video frame P of the projection video i The encoding device encodes the video frame P based on different quantization coefficients. i Compress and get the video frame P i Base layer image and M enhancement layer images; according to the decoding device video frame number i-1 and the corresponding layer number and video frame P i Get the video frame P from the reconstruction frame queue based on the reference relationship i The reference frame of video frame number i-1 and the corresponding layer number are for video frame P i-1 The layer number of the image with the highest quality among the base layer images and M enhancement layer images obtained by the decoding device; i The reference frame is saved to the reference frame queue of the encoding device, and the reference frame is saved according to the video frame P i The reference frame of the video frame P i-1 The base layer image and M enhancement layer images are encoded to obtain the video frame P i-1 The first code stream; according to the video frame P i The reference frame of the video frame P i Decode the first code stream to get the video frame Pi The reconstructed frame of the base layer image and the reconstructed frame of each image in the M enhancement layer images, and the video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each image in the M enhancement layer images are saved in the reconstructed frame queue; the video frame P is transmitted through the network device i The first code stream is sent to the decoding device; the decoding device receives the video frame P i Get the video frame P from the reference frame queue of the decoding device i The reference frame, and according to the video frame P i The reference frame of the video frame P i Decode the second code stream to get the video frame P i The reconstructed frame of the base layer image and the reconstructed frame of the enhancement layer image are decoded; and the corresponding reconstructed frame is displayed on the display according to the display requirements of the display; the decoded video frame P i Frame number i and video frame P i The layer number of the image with the highest quality among the reconstructed frames of the base layer image and the reconstructed frames of the enhancement layer image is sent to the encoding device through the network device, and the reconstructed frames of each layer image are saved; the reconstructed frame of the image with the highest quality is saved in the reference frame queue. Optionally, according to the video frame P i The reference relationship of subsequent video frames will delete the reconstructed frames in the reconstructed frame queue that will not be used as reference frames.

[0116] The following details the above Figure 1a 、 Figure 1b and Figure 1c The first terminal device 101, the second terminal device 102 and Figure 2 The specific process of encoding and decoding by the encoding device and decoding device.

[0117] See also Figure 3 , Figure 3 This is a flow chart of a video encoding method provided in an embodiment of the present application. Figure 3 As shown, the method includes:

[0118] S301, video frame P i Compress to get the video frame P i base layer image and M enhancement layer images.

[0119] Among them, the video frame P i is any frame other than the first frame of the video. i Among the base layer images and M enhancement layer images, the higher the number of layers of the image, the higher the quality of the image, and M is an integer greater than 1.

[0120] Optionally, M can be 1, 2, 3, 4, 5, 7, 9 or other positive integers.

[0121] Specifically, discrete cosine transform (DCT) is performed on the video frame to obtain the transformed video frame P i ; Then use different quantization coefficients to transform the video frame P i Process and get the video frame P i The base layer image and M enhancement layer images, where based on the video frame P i The quantization coefficient of the obtained base layer image is the largest, based on the video frame P i The quantization coefficients used to obtain the M enhancement layer images are different.

[0122] The larger the quantization coefficient, the lower the quality of the image obtained; that is, among the base layer image and the M enhancement layer images, the higher the layer number of the image, the smaller the quantization coefficient used, and the higher the image quality.

[0123] like Figure 4 As shown, based on the video frame P i A base layer image and two enhancement layer images (including enhancement layer image 1 and enhancement layer image 2) are obtained. The base layer image is also called the basic layer (BL) and has the lowest layer number. The enhancement layer image is also called the enhanced layer (EL). The two enhancement layer images can be called EL1 and EL2, and their corresponding layer numbers are 1 and 2. For the base layer image, the corresponding layer number can be 0 or other values, which are not limited here. The quality of EL1 is lower than that of EL2, and the quality of BL is lower than that of EL1.

[0124] It should be noted that the base layer image and the M enhancement layer images of the first frame of the video are all encoded using the intra-frame prediction method.

[0125] S302: Obtain video frame P according to feedback information from the decoding device. i The reference frame of the video frame is used to indicate the information of the video frame received by the decoding device; according to the video frame P i The reference frame of the video frame P i The base layer image and M enhancement layer images are encoded to obtain a video frame P i The first stream.

[0126] In an optional embodiment, the feedback information includes at least one video frame number and a layer number corresponding to each video frame number in the at least one video frame number, and the layer number corresponding to each video frame number is the layer number of the basic layer image or the enhanced layer image of the video frame indicated by the video frame number.

[0127] It should be noted that the feedback message from the decoding device is used to indicate that the decoding device can normally decode the video frame number and the corresponding layer number to which the image belongs.

[0128] In an optional embodiment, the feedback information includes the video frame number i-1 and the corresponding layer number W, where the layer number W is the number of the video frame P. i-1 The layer number of the image with the highest quality among the base layer images and M enhancement layer images obtained by the decoding device, the video frame P i-1 The video frame indicated by the video frame number i-1 is obtained according to the feedback information of the decoding device. i The reference frame includes:

[0129] According to the video frame P i-1 The reconstructed frame of the base layer image and the image with layer number W among the M enhancement layer images is determined as the video frame P i reference frame.

[0130] For example, the decoding device receives the video frame P i-1 The code stream gets the video frame P i-1 The reconstructed frame of the base layer image, the reconstructed frame of the enhancement layer image 1 and the reconstructed frame of the enhancement layer image 2, the image with the highest quality is the enhancement layer image 2, the decoding device feeds the video frame number i-1 and the corresponding layer number 2 back to the encoding device, and the encoding device obtains the video frame P according to the video frame number i-1 and the corresponding layer number 2. i-1 Enhance the reconstructed frame of layer image 2 and convert the video frame P i-1 The reconstructed frame of the enhancement layer image 2 is used as the video frame P i reference frame.

[0131] In an optional embodiment, the feedback information includes the video frame number i-1 and the corresponding layer number W, where the layer number W is the corresponding video frame P i-1 The layer number of the image with the highest quality among the base layer images and M enhancement layer images obtained by the decoding device, the video frame P i-1 The video frame indicated by the video frame number i-1 is obtained according to the feedback information of the decoding device. i The reference frame includes:

[0132] The first reconstruction frame queue Q i-1 , belongs to the video frame P i-1 The reconstructed frames with layer numbers higher than layer number W are deleted to obtain the first reconstructed frame queue Q i ; Among them, the first reconstruction frame queue Q i-1 Including the video frame P obtained by the decoding device i-1 The reconstructed frame of the base layer image and the reconstructed frame of the enhancement layer image of the previous video frame, and the video frame P i-1a reconstructed frame of the base layer image and a reconstructed frame of each enhancement layer image in the M enhancement layer images;

[0133] According to the video frame P i The reference relationship is from the first reconstructed frame queue Q i Get the video frame P i Reference frame; where the video frame P i The reference relationship is the video frame P i With the video frame P i The correspondence between the reference frames used during encoding and decoding.

[0134] In an optional embodiment, according to the video frame P i The reference frame of the video frame P i The base layer image and M enhancement layer images are encoded to obtain a video frame P i After the first code stream, the method of the present application further includes:

[0135] According to the video frame P i The reference frame of the video frame P i The first code stream is decoded to obtain the video frame P i a reconstructed frame of the base layer image and a reconstructed frame of each enhancement layer image in the M enhancement layer images;

[0136] The video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each enhancement layer image in the M enhancement layer images are saved to the first reconstructed frame queue Q i middle.

[0137] In a specific example, Figure 5a As shown, there are three frames of video images that need to be transmitted. For video frame P1, the encoding device encodes the base layer image, enhancement layer image 1 and enhancement layer image 2 of video frame P1 using intra-frame prediction to obtain the first code stream of video frame P1; decodes the first code stream of video frame P1 using intra-frame prediction to obtain the reconstructed frame of the base layer image of video frame P1, the reconstructed frame of enhancement layer image 1 and the reconstructed frame of enhancement layer image 2, and saves the reconstructed frame of the base layer image of video frame P1, the reconstructed frame of enhancement layer image 1 and the reconstructed frame of enhancement layer image 2 to the first reconstructed frame queue Q1; sends the first code stream of video frame P1 to the decoding device; receives the code stream containing video frame number 1 and corresponding layer number 1 sent by the decoding device, indicating that the enhancement layer image 2 is lost during transmission; the encoding device deletes the reconstructed frame of enhancement layer image 2 of video frame P1 from the first reconstructed frame queue Q1 to obtain the first reconstructed frame queue Q2;

[0138] For the video frame P2, the encoding device obtains the reference frame of the video frame P2 from the first reconstructed frame queue Q2 according to the reference relationship of the video frame P2. The reference relationship of the video frame P2 is as follows: Figure 5a As shown, the reference frame of the video frame P2 includes the reconstructed frame of the enhancement layer image 1 of the video frame P1; the base layer image, enhancement layer image 1 and enhancement layer image 2 of the video frame P2 are encoded according to the reference frame of the video frame P2 to obtain a first code stream of the video frame P2; the first code stream of the video frame P2 is decoded according to the reference frame of the video frame P2 to obtain a reconstructed frame of the base layer image, the reconstructed frame of the enhancement layer image 1 and the reconstructed frame of the enhancement layer image 2 of the video frame P2, and the reconstructed frame of the base layer image, the reconstructed frame of the enhancement layer image 1 and the reconstructed frame of the enhancement layer image 2 of the video frame P1 are saved in a first reconstructed frame queue; a code stream containing the video frame number 2 and the corresponding layer number 1, indicating that the enhancement layer image 2 is lost during transmission, is received from the decoding device; the encoding device deletes the reconstructed frame of the enhancement layer image 2 of the video frame P2 from the first reconstructed frame queue Q2 to obtain a first reconstructed frame queue Q3;

[0139] For the video frame P3, the encoding device obtains the reference frame of the video frame P3 from the first reconstructed frame queue Q3 according to the reference relationship of the video frame P3. The reference relationship of the video frame P3 is as follows: Figure 5a As shown, when encoding the video frame P3, it is necessary to refer to the reconstructed frame of the enhancement layer image 1 of the video frame P1, the reconstructed frame of the enhancement layer image 1 of the video frame P2, and the reconstructed frame of the enhancement layer image 2. However, the decoding device does not obtain the reconstructed frame of the enhancement layer image 2 of the video frame P2. In order to maintain the consistency between the encoding device and the decoding device during encoding and decoding, the reconstructed frame of the enhancement layer image 2 of the video frame P2 is removed from the reference frame of the video frame. The method used is that the encoding device determines that the decoding device has not obtained the code stream of the enhancement layer image of the video frame P2. , the reconstructed frame of the enhancement layer image of the video frame P2 is deleted from the first reconstructed frame queue Q2 to obtain the first reconstructed frame queue Q3, so that the reconstructed frame of the enhancement layer image 2 of the video frame P2 cannot be obtained from the first reconstructed frame queue Q3, and the reference frame of the video frame P3 includes the reconstructed frame of the enhancement layer image 1 of the video frame P1 and the reconstructed frame of the enhancement layer image 1 of the video frame P2; the basic layer image, enhancement layer image 1 and enhancement layer image 2 of the video frame P3 are encoded according to the reference frame of the video frame P3 to obtain the first code stream of the video frame P3.

[0140] In an optional embodiment, the feedback information includes the video frame P i The frame number and corresponding layer number of the previous video frame, the layer number corresponding to the video frame number j is for the video frame P j The layer number of the image with the highest quality among the base layer images and M enhancement layer images obtained by the decoding device, the video frame P j is the video frame indicated by the video frame number j, video frame Pj is the video frame P i Any frame in the previous video frame, obtain the video frame P according to the feedback information of the decoding device i Reference frames include:

[0141] According to the video frame P i The reference relationship is to obtain the video frame P from the first reconstruction frame queue i Candidate reconstructed frames; where the video frame P i The reference relationship is the video frame P i With the video frame P i The correspondence between the reference frames used in encoding and decoding; according to the video frame P i The frame number and corresponding layer number of the previous video frame are obtained from the video frame P i The candidate reconstructed frame is used to obtain the video frame P i The reference frame;

[0142] Among them, the video frame P i The reference frames include video frame P i Among the candidate reconstructed frames, the quality of the image corresponding to any reconstructed frame F is not higher than the quality of the image indicated by the layer number corresponding to the video frame number of the video frame to which the reconstructed frame F belongs.

[0143] Optionally, after receiving the feedback message sent by the decoding device, the encoding device saves the video frame number and the corresponding layer number in the feedback message for subsequent encoding. In other words, each feedback message sent by the decoding device only includes the frame number and layer number of the video frame to which the reconstructed frame obtained by the current decoding device belongs. For example, if the decoding device receives the video frame P i After the code stream is obtained, the video frame number i-1 and the corresponding layer number W are obtained based on the code stream, and the video frame number i-1 and the corresponding layer number W are sent to the encoding device through a feedback message. The encoding device obtains the video frame P based on the video frame number i-1 and the corresponding layer number W in the currently obtained feedback message and the video frame number and the corresponding layer number in the feedback message sent by the decoding device before. i The frame number of the previous video frame and the corresponding layer number.

[0144] In an optional embodiment, according to the video frame P i The reference frame of the video frame P i The base layer image and M enhancement layer images are encoded to obtain a video frame P i After the first code stream, the method of the present application further includes:

[0145] According to the video frame P i The reference frame of the video frame P i The first code stream is decoded to obtain the video frame P iThe reconstructed frame of the base layer image and the reconstructed frame of each enhancement layer image in the M enhancement layer images; the video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each enhancement layer image in the M enhancement layer images are saved in a first reconstructed frame queue.

[0146] It should be noted that the video frame P i-1 The previous video frame specifically refers to the time stamp in the video frame P in a video. i-1 The video frame before the timestamp of .

[0147] In a specific example, Figure 5b As shown, there are three frames of video images that need to be transmitted. For video frame P1, the encoding device encodes the base layer image, enhancement layer image 1 and enhancement layer image 2 of video frame P1 using intra-frame prediction to obtain a first code stream of video frame P1; decodes the first code stream of video frame P1 using intra-frame prediction to obtain a reconstructed frame of the base layer image of video frame P1, a reconstructed frame of enhancement layer image 1 and a reconstructed frame of enhancement layer image 2, and saves the reconstructed frame of the base layer image of video frame P1, the reconstructed frame of enhancement layer image 1 and the reconstructed frame of enhancement layer image 2 to a first reconstructed frame queue; sends the first code stream of video frame P1 to the decoding device; receives a code stream sent by the decoding device containing video frame number 1 and corresponding layer number 1, indicating that enhancement layer image 2 is lost during transmission;

[0148] For the video frame P2, the encoding device obtains the candidate reconstructed frame of the video frame P2 from the first reconstructed frame queue according to the reference relationship of the video frame P2. The reference relationship of the video frame P2 is as follows: Figure 5b As shown, the candidate reconstructed frames of the video frame P2 include the reconstructed frame of the enhancement layer image 1 of the video frame P1 and the reconstructed frame of the enhancement layer image 2; however, the decoding device does not obtain the reconstructed frame of the enhancement layer image 2 of the video frame P2. In order to maintain the consistency between the encoding device and the decoding device during encoding and decoding, the reference frame of the video frame P2 is obtained from the candidate reconstructed frames of the video frame P2 according to the video frame number 1 and the corresponding layer number 1. The reference frame of the video frame P2 includes the reconstructed frame of the enhancement layer image 1 of the video frame P1; the base layer image and the enhancement layer image of the video frame P2 are reconstructed according to the reference frame of the video frame P2. Image 1 and enhancement layer image 2 are encoded to obtain a first code stream of video frame P2; the first code stream of video frame P2 is decoded according to the reference frame of video frame P2 to obtain a reconstructed frame of the base layer image of video frame P2, a reconstructed frame of enhancement layer image 1, and a reconstructed frame of enhancement layer image 2, and the reconstructed frame of the base layer image of video frame P1, the reconstructed frame of enhancement layer image 1, and the reconstructed frame of enhancement layer image 2 are saved in a first reconstructed frame queue, and a code stream containing video frame number 2 and corresponding layer number 1, indicating that the enhancement layer image 2 is lost during transmission, is received from the decoding device;

[0149] For the video frame P3, the encoding device obtains the candidate reconstructed frame of the video frame P3 from the first reconstructed frame queue according to the reference relationship of the video frame P3. The reference relationship of the video frame P3 is as follows: Figure 5b As shown, the candidate reconstructed frames of video frame P3 include the reconstructed frame of the enhancement layer image 1 of video frame P1, the reconstructed frame of the enhancement layer image 1 of video frame P2 and the reconstructed frame of the enhancement layer image 2. However, the decoding device does not obtain the reconstructed frame of the enhancement layer image 2 of video frame P2. In order to maintain the consistency between the encoding device and the decoding device during encoding and decoding, the reference frame of video frame P3 is obtained from the candidate reconstructed frames of video frame P2 according to the video frame number 2 and the corresponding layer number 1. The reference frame of video frame P3 includes the reconstructed frame of the enhancement layer image 1 of video frame P1 and / or the reconstructed frame of the enhancement layer image 1 of video frame P2. The basic layer image, enhancement layer image 1 and / or enhancement layer image 2 of video frame P3 are encoded according to the reference frame of video frame P3 to obtain the first code stream of video frame P3.

[0150] Furthermore, when obtaining the video frame P i After the first code stream is sent, the video frame P is sent to the decoding device i The first stream.

[0151] In an optional embodiment, when the video frame P i The reference relationship of the subsequent frames determines the first reconstruction frame queue Q i-1 When any reconstructed frame will not be used as a reference frame in subsequent encoding, the reconstructed frame is removed from the first reconstructed frame queue Q i-1 Delete in.

[0152] It can be seen that in the solution of the present application, the video frame P obtained based on the decoding side i-1 The feedback information of the video frame P is obtained i The reference frame, video frame P i The reference frame includes the video frame P obtained by the decoding end i-1 The reconstructed frame of the highest quality image is obtained according to the video frame P i The reference frame of the video frame P i The base layer image and enhancement layer image are encoded. By using the high-quality enhancement layer image reconstructed frame as the reference frame, image compression efficiency is maximized, chip power consumption is reduced, and the reference frame is adjusted in real time based on channel feedback, avoiding the problem of screen distortion during decoding in the event of network packet loss.

[0153] See also Figure 6 , Figure 6 A flowchart of a video decoding method provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the method includes:

[0154] S601: Obtain video frame Pi The second stream.

[0155] Optionally, obtain the video frame P from the encoding device i The second stream.

[0156] It should be noted that the encoding device sends the video frame P to the decoding device. i However, due to the change of network bandwidth between the encoding device and the decoding device, packet loss may occur. Therefore, the video frame P is received by the decoding device. i The second code stream is the same as the video frame P i The first code stream is distinguished.

[0157] S602: Based on the video frame P i The reference frame of the video frame P i The second code stream is decoded to obtain the video frame P i The first reconstructed frame and N second reconstructed frames.

[0158] Among them, the video frame P i The first reconstructed frame is the video frame P i The reconstructed frame of the base layer image, the video frame P i The second reconstructed frame is the video frame P i The reconstructed frame of the enhancement layer image; N is an integer greater than or equal to 0.

[0159] Optionally, the video frame P i The reference frame includes the video frame P indicated by the video frame number i-1. i-1 The reconstructed frame of the base layer image and the image with layer number W in the enhancement layer image;

[0160] Wherein, the layer number W is the video frame P obtained by the decoding device i-1 The layer number of the highest quality image among the N enhancement layer images corresponding to the N second reconstructed frames.

[0161] For example, the decoding device receives the video frame P i-1 The code stream gets the video frame P i-1 The reconstructed frame corresponding to the base layer image, the reconstructed frame of the enhancement layer image 1 and the reconstructed frame of the enhancement layer image 2, the image with the highest quality is the enhancement layer image 2, and the feedback information sent to the encoding device includes the video frame number i-1 and the corresponding layer number 2. The video frame P i The reference frames include video frame P i-1 The reconstructed frame of the enhancement layer image 2.

[0162] Optionally, the video frame P i The reference frame is based on the video frame P i The reference relationship is obtained from the second reconstruction frame queue,

[0163] Among them, the video frame P i The reference relationship is the video frame P i With the video frame P i The correspondence between the reference frames used in encoding and decoding, the second reconstructed frame queue includes the video frame P obtained by the decoding device i The reconstructed frame of the base layer image of the previous video frame and the reconstructed frame of each enhancement layer image in the N enhancement layer images.

[0164] For example, the reference relationship of video frame P3 is as follows Figure 5a As shown, when decoding the code stream of the video frame P3, the second reconstructed frame queue Q2 includes the reconstructed frame of the base layer image and the reconstructed frame of the enhancement layer image 1 of the video frame P1, and the reconstructed frame of the base layer image and the reconstructed frame of the enhancement layer image 1 of the video frame P2; according to Figure 5a From the reference relationship of the video frame P3 shown, it can be seen that the reference frames required for decoding the code stream of the video frame P3 include the reconstructed frame of the enhancement layer image 1 of the video frame P1, the reconstructed frame of the enhancement layer image 1 of the video frame P2, and the reconstructed frame of the enhancement layer image 2; therefore, the reconstructed frame of the video frame P3 obtained from the second reconstructed frame queue Q2 based on the reference relationship of the video frame P3 includes the reconstructed frame of the enhancement layer image 1 of the video frame P1 and the reconstructed frame of the enhancement layer image 1 of the video frame P2.

[0165] In an optional embodiment, when the video frame P i The subsequent video frame reference relationship determines the first reconstruction frame queue Q i-1 When any reconstructed frame will not be used as a reference frame in subsequent decoding, the reconstructed frame is removed from the first reconstructed frame queue Q i-1 Delete in.

[0166] In an optional embodiment, the method of the present application further includes:

[0167] The video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each enhancement layer image in the N enhancement layer images are saved to the second reconstructed frame queue Q i-1 To obtain the second reconstructed frame queue Q i .

[0168] In an optional embodiment, the method of the present application further includes:

[0169] A feedback message is sent to the encoding device, where the feedback message includes the video frame number i-1 and the layer number W.

[0170] Optionally, the decoding device obtains the video frame P iAfter reconstructing the base layer image and the reconstructed frame of each of the N enhanced layer images, due to the different qualities of the reconstructed frames, the reconstructed frame of the corresponding quality is selected for display on the projection device according to the image quality required by the projection device.

[0171] It can be seen that in the embodiment, the video frame P obtained by the decoding device is i-1 The reconstructed frame of the high-quality image is used as a reference frame to decode the video frame P i The code stream avoids the problem of screen distortion during decoding in the event of network packet loss; and the decoding device only manages one reconstructed frame queue, reducing the cost and area of ​​the hardware decoding chip.

[0172] For example, in a bedroom scene, a mobile phone is used to project the screen to a computer or other terminal. Since the network channel conditions in the bedroom scene are relatively poor due to the closed space, data is very likely to be lost during transmission. Therefore, it is assumed that during the transmission of video data, the enhanced layer data may be lost. Figure 7 As shown, it is assumed that the video data includes 5 frames of images, and each frame of the 5 frames of images is compressed into a base layer image, an enhancement layer image 1 and an enhancement layer image 2; for the video frame P1, the encoding device adopts an intra-frame prediction encoding method to encode the base layer image, enhancement layer image 1 and enhancement layer image 2 of the video frame P1 to obtain a first code stream of the video frame P1; the encoding device decodes the first code stream of the video frame P1 according to the intra-frame prediction decoding method to obtain a reconstructed frame of the base layer image of the video frame P1, a reconstructed frame of the enhancement layer image 1 and a reconstructed frame of the enhancement layer image 2, and saves them to a first reconstructed frame queue; the first code stream of the video frame P1 is sent to the decoding device; the decoding device receives the second code stream of the video frame P1, and decodes the second code stream of the video frame P1 according to the intra-frame prediction decoding method to obtain a reconstructed frame of the base layer image of the video frame P1, a reconstructed frame of the enhancement layer image 1 and a reconstructed frame of the enhancement layer image 2, as shown Figure 7 As shown; the video frame number 1 and the corresponding layer number 2 (corresponding to the enhancement layer image 2 of the video frame P1) are sent to the encoding device;

[0173] For video frame P2, the encoding device obtains the reconstructed frame of the enhancement layer image 2 of video frame P1 from the first reconstructed frame queue according to the video frame number 1 and the corresponding layer number 2 sent by the decoding device, and encodes the basic layer image, enhancement layer image 1 and enhancement layer image 2 of video frame P2 according to the reconstructed frame of the enhancement layer image 2 of video frame P1 to obtain the first code stream of video frame P2; sends the first code stream of video frame P2 to the decoding device, and decodes the first code stream of video frame P2 according to the reconstructed frame of the enhancement layer image 2 of video frame P1 to obtain the basic layer image of video frame P2. The decoding device receives the second code stream of the video frame P2 and decodes the second code stream of the video frame P2 according to the reconstructed frame of the enhancement layer image 2 of the video frame P1 to obtain the reconstructed frame of the base layer image and the reconstructed frame of the enhancement layer image 1 of the video frame P2; the encoding device sends the video frame number 2 and the corresponding layer number 1 (corresponding to the enhancement layer image 1 of the video frame P2) to the encoding device;

[0174] For video frame P3, the encoding device obtains the reconstructed frame of the enhancement layer image 1 of video frame P2 from the first reconstructed frame queue according to the video frame number 2 and the corresponding layer number 1 sent by the decoding device, and encodes the base layer image, enhancement layer image 1 and enhancement layer image 2 of video frame P3 according to the reconstructed frame of the enhancement layer image 1 of video frame P2 to obtain the first code stream of video frame P3; sends the first code stream of video frame P3 to the decoding device, and decodes the first code stream of video frame P3 according to the reconstructed frame of the enhancement layer image 1 of video frame P2 to obtain the reconstructed frame of the base layer image, enhancement layer image 1 and enhancement layer image 2 of video frame P3. The decoding device receives the second code stream of the video frame P3 and decodes the second code stream of the video frame P3 according to the reconstructed frame of the enhancement layer image 1 of the video frame P2 to obtain the reconstructed frame of the base layer image, the reconstructed frame of the enhancement layer image 1, and the reconstructed frame of the enhancement layer image 2 of the video frame P3; the video frame number 3 and the corresponding layer number 2 (corresponding to the enhancement layer image 2 of the video frame P3) are sent to the encoding device;

[0175] For video frame P4, the encoding device obtains the reconstructed frame of the enhancement layer image 2 of video frame P3 from the first reconstructed frame queue according to the video frame number 3 and the corresponding layer number 2 sent by the decoding device, and encodes the basic layer image, enhancement layer image 1 and enhancement layer image 2 of video frame P4 according to the reconstructed frame of the enhancement layer image 2 of video frame P3 to obtain the first code stream of video frame P4; sends the first code stream of video frame P4 to the decoding device, and decodes the first code stream of video frame P4 according to the reconstructed frame of the enhancement layer image 2 of video frame P3 to obtain the first code stream of video frame P4. The decoding device receives the second bitstream of the video frame P4 and decodes the second bitstream of the video frame P4 based on the reconstructed frame of the enhancement layer image 2 of the video frame P3 to obtain the reconstructed frame of the base layer image of the video frame P4. The decoding device sends the video frame number 4 and the corresponding layer number 0 (corresponding to the base layer image of the video frame P4) to the encoding device.

[0176] For video frame P5, the encoding device obtains the reconstructed frame of the base layer image of video frame P4 from the first reconstructed frame queue according to the video frame number 4 and the corresponding layer number 0 sent by the decoding device, and encodes the base layer image, enhancement layer image 1 and enhancement layer image 2 of video frame P5 according to the reconstructed frame of the base layer image of video frame P4 to obtain the first code stream of video frame P5; sends the first code stream of video frame P5 to the decoding device, and decodes the first code stream of video frame P5 according to the reconstructed frame of the base layer image of video frame P4 to obtain the reconstructed frame of the base layer image, the reconstructed frame of enhancement layer image 1 and the reconstructed frame of enhancement layer image 2 of video frame P5, and saves the reconstructed frame of the base layer image, the reconstructed frame of enhancement layer image 1 and the reconstructed frame of enhancement layer image 2 of video frame P5 to the first reconstructed frame queue; the decoding device receives the second code stream of video frame P5, and decodes the second code stream of video frame P5 according to the reconstructed frame of the base layer image of video frame P4 to obtain the reconstructed frame of the base layer image and the reconstructed frame of enhancement layer image 1 of video frame P5.

[0177] Taking the 5th frame as an example, when the enhanced layer image of the 4th frame is lost, the code stream of the enhanced layer image of the 5th frame can still be decoded correctly, and the decoded reconstructed frame is as follows: Figure 8 As shown in Figure a, if the existing technology is used, a screen flower phenomenon will occur, such as Figure 8 As shown in Figure b.

[0178] Let’s take another example. Figure 9As shown, it is assumed that the video includes 5 frames of images, and each frame of the 5 frames is compressed into a base layer image, an enhancement layer image 1 and an enhancement layer image 2; for the video frame P1, the encoding device adopts an intra-frame prediction encoding method to encode the base layer image, enhancement layer image 1 and enhancement layer image 2 of the video frame P1 to obtain a first code stream of the video frame P1; the encoding device decodes the first code stream of the video frame P1 according to the intra-frame prediction decoding method to obtain a reconstructed frame of the base layer image of the video frame P1, a reconstructed frame of the enhancement layer image 1 and a reconstructed frame of the enhancement layer image 2, and saves them to a first reconstructed frame queue; the first code stream of the video frame P1 is sent to the decoding device; the decoding device receives the second code stream of the video frame P1, and decodes the second code stream of the video frame P1 according to the intra-frame prediction decoding method to obtain a reconstructed frame of the base layer image of the video frame P1, a reconstructed frame of the enhancement layer image 1 and a reconstructed frame of the enhancement layer image 2, as shown Figure 9 As shown; the video frame number 1 and the corresponding layer number 2 (corresponding to the enhancement layer image 2 of the video frame P1) are sent to the encoding device;

[0179] For video frame P2, the encoding device determines that no packet is lost during the transmission of the first code stream of video frame P1 based on the video frame number 1 and the corresponding layer number 2 sent by the decoding device, and therefore does not process the first reconstructed frame queue; obtains the reconstructed frame of the enhancement layer image 2 of video frame P1 from the first reconstructed frame queue according to the reference relationship of video frame P2, and encodes the basic layer image, enhancement layer image 1 and enhancement layer image 2 of video frame P2 according to the reconstructed frame of the enhancement layer image 2 of video frame P1 to obtain the first code stream of video frame P2; sends the first code stream of video frame P2 to the decoding device, and encodes the base layer image, enhancement layer image 1 and enhancement layer image 2 of video frame P1 according to the reconstructed frame of the enhancement layer image 2 of video frame P1 to obtain the first code stream of video frame P2. The frame construction device decodes the first code stream of the video frame P2 to obtain a reconstructed frame of the base layer image, a reconstructed frame of the enhancement layer image 1, and a reconstructed frame of the enhancement layer image 2 of the video frame P2, and saves the reconstructed frame of the base layer image, the reconstructed frame of the enhancement layer image 1, and the reconstructed frame of the enhancement layer image 2 of the video frame P2 to a first reconstructed frame queue; the decoding device receives the second code stream of the video frame P2, and decodes the second code stream of the video frame P2 according to the reconstructed frame of the enhancement layer image 2 of the video frame P1 to obtain a reconstructed frame of the base layer image and a reconstructed frame of the enhancement layer image 1 of the video frame P2; and sends the video frame number 2 and the corresponding layer number 1 to the encoding device;

[0180] For video frame P3, the encoding device deletes the reconstructed frame of the enhancement layer image 2 of video frame P2 from the first reconstructed frame queue according to the video frame number 2 and the corresponding layer number 1 (corresponding to the enhancement layer image 1 of video frame P2) sent by the decoding device, and then obtains the reconstructed frame of the enhancement layer image 2 of video frame P1 and the reconstructed frame of the enhancement layer image 1 of video frame P2 from the first reconstructed frame queue according to the reference relationship of video frame P3, and encodes the basic layer image, enhancement layer image 1 and enhancement layer image 2 of video frame P3 according to the reconstructed frame of the enhancement layer image 2 of video frame P1 and the reconstructed frame of the enhancement layer image 1 of video frame P2 to obtain the first code stream of video frame P3; sends the first code stream of video frame P3 to the decoding device, and obtains the reconstructed frame of the enhancement layer image 2 of video frame P1 according to the reconstructed frame of the enhancement layer image 2 of video frame P1. The decoding device receives the second code stream of the video frame P3 and decodes the second code stream of the video frame P3 according to the reconstructed frame of the enhancement layer image 1 of the video frame P2 to obtain the reconstructed frame of the base layer image, the reconstructed frame of the enhancement layer image 1, and the reconstructed frame of the enhancement layer image 2 of the video frame P3, and saves the reconstructed frame of the base layer image, the reconstructed frame of the enhancement layer image 1, and the reconstructed frame of the enhancement layer image 2 of the video frame P3 to the first reconstructed frame queue; the decoding device receives the second code stream of the video frame P3, and decodes the second code stream of the video frame P3 according to the reconstructed frame of the enhancement layer image 1 of the video frame P2 to obtain the reconstructed frame of the base layer image, the reconstructed frame of the enhancement layer image 1, and the reconstructed frame of the enhancement layer image 2 of the video frame P3; the video frame number 3 and the corresponding layer number 2 (corresponding to the enhancement layer image 2 of the video frame P3) are sent to the encoding device;

[0181] For video frame P4, the encoding device determines that there is packet loss during the transmission of the first code stream of video frame P3 based on the video frame number 3 and the corresponding layer number 2 sent by the decoding device, and therefore does not process the first reconstructed frame queue; obtains the reconstructed frame of the enhancement layer image 1 of video frame P2 and the reconstructed frame of the enhancement layer image 2 of video frame P3 from the first reconstructed frame queue according to the reference relationship of video frame P4, and encodes the basic layer image, enhancement layer image 1 and enhancement layer image 2 of video frame P4 according to the reconstructed frame of the enhancement layer image 1 of video frame P2 and the reconstructed frame of the enhancement layer image 2 of video frame P3 to obtain the first code stream of video frame P4; sends the first code stream of video frame P4 to the decoding device, and according to the enhancement layer image 2 of video frame P2, encodes the basic layer image, enhancement layer image 1 and enhancement layer image 2 of video frame P4 to obtain the first code stream of video frame P4; sends the first code stream of video frame P4 to the decoding device, and The first bitstream of the video frame P4 is decoded based on the reconstructed frame of the base layer image 1 of the video frame P4 and the reconstructed frame of the enhancement layer image 2 of the video frame P3 to obtain the reconstructed frame of the base layer image, the reconstructed frame of the enhancement layer image 1, and the reconstructed frame of the enhancement layer image 2 of the video frame P4, and the reconstructed frame of the base layer image, the reconstructed frame of the enhancement layer image 1, and the reconstructed frame of the enhancement layer image 2 of the video frame P4 are saved in the first reconstructed frame queue; the decoding device receives the second bitstream of the video frame P4, and decodes the second bitstream of the video frame P4 based on the reconstructed frame of the enhancement layer image 2 of the video frame P3 to obtain the reconstructed frame of the base layer image of the video frame P4; the video frame number 4 and the corresponding layer number 0 (corresponding to the base layer image of the video frame P4) are sent to the encoding device;

[0182] For video frame P5, the encoding device deletes the reconstructed frame of the enhancement layer image 1 of video frame P4 and the reconstructed frame of the enhancement layer image 2 of video frame P4 from the first reconstructed frame queue according to the video frame number 4 and the corresponding layer number 0 sent by the decoding device; obtains the reconstructed frame of the base layer image of video frame P4 from the first reconstructed frame queue according to the reference relationship of video frame P5, and encodes the base layer image, enhancement layer image 1 and enhancement layer image 2 of video frame P5 according to the reconstructed frame of the base layer image of video frame P4 to obtain the first code stream of video frame P5; sends the first code stream of video frame P5 to the decoding device, and The first code stream of video frame P5 is decoded according to the reconstructed frame of the basic layer image of video frame P4 to obtain the reconstructed frame of the basic layer image, the reconstructed frame of enhancement layer image 1 and the reconstructed frame of enhancement layer image 2 of video frame P5, and the reconstructed frame of the basic layer image, the reconstructed frame of enhancement layer image 1 and the reconstructed frame of enhancement layer image 2 of video frame P5 are saved in the first reconstructed frame queue; the decoding device receives the second code stream of video frame P5, and decodes the second code stream of video frame P5 according to the reconstructed frame of the basic layer image of video frame P4 to obtain the reconstructed frame of the basic layer image and the reconstructed frame of enhancement layer image 1 of video frame P5.

[0183] For example, consider using a mobile phone to project the screen onto a TV or other device in a living room. Compared to a bedroom or other complex scenes, a living room offers more open space, fewer obstructions, and a more stable network signal, making packet loss less likely during transmission. This assumes that both the base layer and enhancement layer of the video data can be sent and received normally.

[0184] like Figure 10 As shown, it is assumed that a video includes 5 frames of images, and each of the 5 frames of images is compressed into a base layer image, an enhancement layer image 1, and an enhancement layer image 2; for video frame P1, the encoding device encodes the base layer image, enhancement layer image 1, and enhancement layer image 2 of video frame P1 using an intra-frame prediction encoding method to obtain a first code stream of video frame P1; the encoding device decodes the first code stream of video frame P1 according to an intra-frame prediction decoding method to obtain a reconstructed frame of enhancement layer image 2 of video frame P1; the first code stream of video frame P1 is sent to a decoding device; the decoding device receives a second code stream of video frame P1, and decodes the second code stream of video frame P1 according to an intra-frame prediction decoding method to obtain a reconstructed frame of the base layer image, a reconstructed frame of enhancement layer image 1, and a reconstructed frame of enhancement layer image 2 of video frame P1;

[0185] For video frame P2, the encoding device encodes the base layer image, enhancement layer image 1, and enhancement layer image 2 of video frame P2 according to the reconstructed frame of enhancement layer image 2 of video frame P1 to obtain a first code stream of video frame P2; sends the first code stream of video frame P2 to the decoding device, and decodes the first code stream of video frame P2 according to the reconstructed frame of enhancement layer image 2 of video frame P1 to obtain a reconstructed frame of the base layer image, a reconstructed frame of enhancement layer image 1, and a reconstructed frame of enhancement layer image 2 of video frame P2; the decoding device receives the second code stream of video frame P2, and decodes the second code stream of video frame P2 according to the reconstructed frame of enhancement layer image 2 of video frame P1 to obtain a reconstructed frame of the base layer image, a reconstructed frame of enhancement layer image 1, and a reconstructed frame of enhancement layer image 2 of video frame P2;

[0186] For video frame P3, the encoding device encodes the base layer image, enhancement layer image 1, and enhancement layer image 2 of video frame P3 according to the reconstructed frame of enhancement layer image 2 of video frame P2 to obtain a first code stream of video frame P3; sends the first code stream of video frame P3 to the decoding device, and decodes the first code stream of video frame P3 according to the reconstructed frame of enhancement layer image 2 of video frame P2 to obtain a reconstructed frame of the base layer image, a reconstructed frame of enhancement layer image 1, and a reconstructed frame of enhancement layer image 2 of video frame P3; the decoding device receives a second code stream of video frame P3, and decodes the second code stream of video frame P3 according to the reconstructed frame of enhancement layer image 2 of video frame P2 to obtain a reconstructed frame of the base layer image, a reconstructed frame of enhancement layer image 1, and a reconstructed frame of enhancement layer image 2 of video frame P3;

[0187] For video frame P4, the encoding device encodes the base layer image, enhancement layer image 1, and enhancement layer image 2 of video frame P4 based on the reconstructed frame of enhancement layer image 2 of video frame P3 to obtain a first code stream of video frame P4; sends the first code stream of video frame P4 to the decoding device, and decodes the first code stream of video frame P4 based on the reconstructed frame of enhancement layer image 2 of video frame P3 to obtain a reconstructed frame of the base layer image, a reconstructed frame of enhancement layer image 1, and a reconstructed frame of enhancement layer image 2 of video frame P4; the decoding device receives the second code stream of video frame P4, and decodes the second code stream of video frame P4 based on the reconstructed frame of enhancement layer image 2 of video frame P3 to obtain a reconstructed frame of the base layer image, a reconstructed frame of enhancement layer image 1, and a reconstructed frame of enhancement layer image 2 of video frame P4;

[0188] For video frame P5, the encoding device encodes the base layer image, enhancement layer image 1 and enhancement layer image 2 of video frame P5 according to the reconstructed frame of the base layer image of video frame P4 to obtain the first code stream of video frame P5; sends the first code stream of video frame P5 to the decoding device, and decodes the first code stream of video frame P5 according to the reconstructed frame of the enhancement layer image 2 of video frame P4 to obtain the reconstructed frame of the base layer image, the reconstructed frame of enhancement layer image 1 and the reconstructed frame of enhancement layer image 2 of video frame P5; the decoding device receives the second code stream of video frame P5, and decodes the second code stream of video frame P5 according to the reconstructed frame of the enhancement layer image 2 of video frame P4 to obtain the reconstructed frame of the base layer image, the reconstructed frame of enhancement layer image 1 and the reconstructed frame of enhancement layer image 2 of video frame P5.

[0189] It should be noted here that the way in which the encoded bitstream is decoded in the encoding device is the same as the way in which the encoded bitstream is decoded in the decoding device, thereby ensuring that for the same video frame, the reconstructed frame obtained by the encoding device and the reconstructed frame obtained by the decoding device are the same.

[0190] It should be noted here that, in the present application, the encoding device determines the reference frame of the current frame based on the feedback information from the decoding device side, which can be regarded as the encoding device determining the reference frame of the current frame based on the network status between the encoding device and the decoding device; since the network bandwidth changes during the transmission of the code stream of the previous frame, part of the code stream content will be lost, making it impossible for the decoding device to completely decode all the enhancement layers of the previous frame; for the previous frame, the decoding device sends the decoded image as feedback information to the encoding device, and the encoding device determines the reference frame of the current frame based on the decoded image by the decoding device. Therefore, the encoding device determines the reference frame of the current frame based on the network status between the encoding device and the decoding device, so that when the network bandwidth changes, the decoding device can also decode normally, thereby making the screen projection smooth.

[0191] The solution of the present application can not only decode the enhancement layer normally in the case of network packet loss, but also use the enhancement layer of the highest quality layer as a reference frame to maximize the coding compression efficiency and reduce power consumption when the network is stable; and after the enhancement layer is lost, the quality difference between layers is reduced.

[0192] See also Figure 11 , Figure 11 This is a schematic diagram of the structure of an encoding device provided in an embodiment of the present application. Figure 11 As shown, the video encoding device 1100 includes:

[0193] Compression unit 1101, used to compress video frame P i Compress to get the video frame P i The base layer image and M enhancement layer images, where the video frame P i is any non-first frame of the video, and M is an integer greater than 0;

[0194] The acquisition unit 1102 is configured to acquire the video frame P according to the feedback information of the decoding device. i The feedback information is used to indicate information of the video frame received by the decoding device;

[0195] The encoding unit 1103 is configured to encode the video frame P i The reference frame of the video frame P i The base layer image and M enhancement layer images are encoded to obtain a video frame P i The first stream.

[0196] In a feasible embodiment, the feedback information includes at least one video frame number and a layer number corresponding to each video frame number in the at least one video frame number, and the layer number corresponding to each video frame number is the layer number of the basic layer image or the enhanced layer image of the video frame indicated by the video frame number.

[0197] In a feasible embodiment, the feedback information includes the video frame number i-1 and the corresponding layer number W, where the layer number W is the number of the video frame P. i-1 The layer number of the image with the highest quality among the base layer images and M enhancement layer images obtained by the decoding device, the video frame P i-1 For the video frame indicated by the video frame number i-1, the acquisition unit 1102 is specifically configured to:

[0198] According to the video frame P i-1 The reconstructed frame of the base layer image and the image with layer number W among the M enhancement layer images is determined as the video frame P i reference frame.

[0199] In a feasible embodiment, the feedback information includes the video frame number i-1 and the corresponding layer number W, where the layer number W is the number of the video frame P. i-1 The layer number of the image with the highest quality among the base layer images and M enhancement layer images obtained by the decoding device, the video frame P i-1 For the video frame indicated by the video frame number i-1, the acquisition unit 1102 is specifically configured to:

[0200] The first reconstruction frame queue Q i-1 , belongs to the video frame P i-1 The reconstructed frames of the images with layer numbers higher than layer number W are deleted to obtain the first reconstructed frame queue Q i ; Among them, the first reconstruction frame queue Q i-1 Including the video frame P obtained by the decoding device i-1 The reconstructed frame of the base layer image and the reconstructed frame of the enhancement layer image of the previous video frame, and the video frame P i-1 a reconstructed frame of the base layer image and a reconstructed frame of each enhancement layer image in the M enhancement layer images;

[0201] According to the video frame P i The reference relationship is from the first reconstructed frame queue Q i Get the video frame P i Reference frame; where the video frame P i The reference relationship is the video frame P i With the video frame P i The correspondence between the reference frames used during encoding and decoding.

[0202] In a feasible embodiment, according to the video frame P i The reference frame of the video frame P i The base layer image and M enhancement layer images are encoded to obtain a video frame P i After the first code stream, the video encoding device 1100 further includes:

[0203] The reconstruction unit 1104 is configured to reconstruct the video frame P i The reference frame of the video frame P i The first code stream is decoded to obtain the video frame P i a reconstructed frame of the base layer image and a reconstructed frame of each enhancement layer image in the M enhancement layer images;

[0204] The storage unit 1105 is used to save the video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each enhancement layer image in the M enhancement layer images are saved to the first reconstructed frame queue Q i middle.

[0205] In a feasible embodiment, the feedback information includes the video frame Pi The frame number and corresponding layer number of the previous video frame, the layer number corresponding to the video frame number j is for the video frame P j The layer number of the image with the highest quality among the base layer images and M enhancement layer images obtained by the decoding device, the video frame P j is the video frame indicated by the video frame number j, video frame P j is the video frame P i For any frame in the previous video frame, the acquisition unit 1102 is specifically configured to:

[0206] According to the video frame P i The reference relationship is to obtain the video frame P from the first reconstruction frame queue i Candidate reconstructed frames; where the video frame P i The reference relationship is the video frame P i With the video frame P i The correspondence between the reference frames used in encoding and decoding; according to the video frame P i The frame number and corresponding layer number of the previous video frame are obtained from the video frame P i The candidate reconstructed frame is used to obtain the video frame P i The reference frame;

[0207] Among them, the video frame P i The reference frames include video frame P i Among the candidate reconstructed frames, the quality of the image corresponding to any reconstructed frame F is not higher than the quality of the image indicated by the layer number corresponding to the video frame number of the video frame to which the reconstructed frame F belongs.

[0208] In a feasible embodiment, according to the video frame P i The reference frame of the video frame P i The base layer image and M enhancement layer images are encoded to obtain a video frame P i After the first code stream, the video encoding device 1100 further includes:

[0209] The reconstruction unit 1104 is configured to reconstruct the video frame P i The reference frame of the video frame P i The first code stream is decoded to obtain the video frame P i a reconstructed frame of the base layer image and a reconstructed frame of each enhancement layer image in the M enhancement layer images;

[0210] The storage unit 1105 is used to save the video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each enhancement layer image in the M enhancement layer images are saved in a first reconstructed frame queue.

[0211] In a feasible embodiment, the video encoding device 1100 further includes:

[0212] The sending unit 1106 is used to send the video frame P i The first code stream is sent to the decoding device.

[0213] It should be noted that the above-mentioned units (compression unit 1101, acquisition unit 1102, encoding unit 1103, reconstruction unit 1104, storage unit 1105 and sending unit 1106) are used to execute the relevant contents of steps S301 and S302 of the above-mentioned method.

[0214] In this embodiment, the video encoding device 1100 is presented in the form of a unit. The "unit" here may refer to an application-specific integrated circuit (ASIC), a processor and memory that executes one or more software or firmware programs, an integrated logic circuit, and / or other devices that can provide the above functions. In addition, the above compression unit 1101, acquisition unit 1102, encoding unit 1103, reconstruction unit 1104, and storage unit 1105 can be Figure 13 The encoding device shown is implemented by the processor 1301.

[0215] See also Figure 12 , Figure 12 This is a schematic diagram of the structure of a decoding device provided in an embodiment of the present application. Figure 12 As shown, the video decoding device 1200 includes:

[0216] Acquisition unit 1201, used to acquire video frame P i The second stream;

[0217] The decoding unit 1202 is configured to decode the video frame P i The reference frame of the video frame P i The second code stream is decoded to obtain the video frame P i The first reconstructed frame and N second reconstructed frames; wherein the video frame P i The first reconstructed frame is the video frame P i The reconstructed frame of the base layer image, the video frame P i The second reconstructed frame is the video frame P i The reconstructed frame of the enhancement layer image; N is an integer greater than or equal to 0.

[0218] In one possible embodiment, the video frame P i The reference frame includes the video frame P indicated by the video frame number i-1. i-1 The reconstructed frame of the base layer image and the image with layer number W in the enhancement layer image;

[0219] Wherein, the layer number W is the video frame P obtained by the decoding device i-1 The layer number of the highest quality image among the N enhancement layer images corresponding to the N second reconstructed frames.

[0220] In one possible embodiment, the video frame P i The reference frame is based on the video frame P i The reference relationship is from the second reconstructed frame queue Q i-1 Obtained from

[0221] Among them, the video frame P i The reference relationship is the video frame P i With the video frame P i The correspondence between the reference frames used in encoding and decoding, the second reconstructed frame queue Q i-1 Including the video frame P obtained by the decoding device i The reconstructed frame of the base layer image of the previous video frame and the reconstructed frame of each enhancement layer image in the N enhancement layer images.

[0222] In a feasible embodiment, the video decoding device 1200 further includes:

[0223] The storage unit 1203 is used to save the video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each enhancement layer image in the N enhancement layer images are saved to the second reconstructed frame queue Q i-1 To obtain the second reconstructed frame queue Q i .

[0224] In a feasible embodiment, the video decoding device 1200 further includes:

[0225] The sending unit 1204 is configured to send a feedback message to the encoding device, where the feedback message includes the video frame number i-1 and the layer number W.

[0226] It should be noted that the above-mentioned units (the acquisition unit 1201 , the decoding unit 1202 , the storage unit 1203 and the sending unit 1204 ) are used to execute the relevant contents of steps S601 and S602 of the above-mentioned method.

[0227] In this embodiment, the video decoding device 1200 is presented in the form of a unit. The "unit" here may refer to an application-specific integrated circuit (ASIC), a processor and memory that executes one or more software or firmware programs, an integrated logic circuit, and / or other devices that can provide the above functions. In addition, the above acquisition unit 1201, decoding unit 1202, and storage unit 1203 can be Figure 14The video decoding device shown is implemented by the processor 1404.

[0228] like Figure 13 The video encoding device 1300 shown can be Figure 13 The video encoding device 1300 includes at least one processor 1301, at least one memory 1302, and at least one communication interface 1303. The processor 1301, the memory 1302, and the communication interface 1303 are connected via the communication bus and communicate with each other.

[0229] The processor 1301 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the above program.

[0230] The communication interface 1303 is used to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Networks (WLAN), etc.

[0231] The memory 1302 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processor via a bus. The memory may also be integrated with the processor.

[0232] The memory 1302 is used to store application code for executing the above solution, and the execution is controlled by the processor 1301. The processor 1301 is used to execute the application code stored in the memory 1302.

[0233] The code stored in the memory 1302 can execute any of the above-mentioned video encoding methods, such as: i Compress to get the video frame P i The base layer image and M enhancement layer images, where the video frame P i is any frame other than the first frame of the video, M is an integer greater than 0; the video frame P is obtained according to the feedback information of the decoding device i The feedback information is used to indicate the information of the video frame received by the decoding device, according to the video frame P i The reference frame of the video frame P i The base layer image and M enhancement layer images are encoded to obtain a video frame P i The first stream.

[0234] like Figure 14 The video decoding device 1400 shown can be Figure 11 The video decoding device 1400 includes at least one processor 1401, at least one memory 1402, and at least one communication interface 1403. The processor 1401, the memory 1402, and the communication interface 1403 are connected via the communication bus and communicate with each other.

[0235] The processor 1401 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the above program.

[0236] The communication interface 1403 is used to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Networks (WLAN), etc.

[0237] The memory 1402 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processor via a bus. The memory may also be integrated with the processor.

[0238] The memory 1402 is used to store application code for executing the above solution, and the execution is controlled by the processor 1401. The processor 1401 is used to execute the application code stored in the memory 1402.

[0239] The code stored in the memory 1402 can execute any of the video encoding and decoding methods provided above, such as: obtaining a video frame P i The second code stream; according to the video frame P i The reference frame of the video frame P i The second code stream is decoded to obtain the video frame P i The first reconstructed frame and N second reconstructed frames; wherein the video frame P i The first reconstructed frame is the video frame P i The reconstructed frame of the base layer image, the video frame P i The second reconstructed frame is the video frame P i The reconstructed frame of the enhancement layer image, N is an integer greater than or equal to 0.

[0240] An embodiment of the present application further provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program includes part or all of the steps of any one of the video encoding and decoding methods described in the above method embodiments.

[0241] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0242] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0243] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0244] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0245] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0246] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0247] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0248] The above is a detailed introduction to the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A video encoding method based on quality grading, characterized in that: include: For video frame P i Compress to obtain the video frame P i The base layer image and M enhancement layer images; wherein the video frame P i is any frame other than the first frame of the video; M is an integer greater than 0; The video frame P is obtained according to the feedback information of the decoding device. i The feedback information is used to indicate information of a video frame received by the decoding device; the feedback information includes at least one video frame number and a layer number corresponding to each video frame number in the at least one video frame number, and the layer number corresponding to each video frame number is a layer number of a base layer image or an enhancement layer image of the video frame indicated by the video frame number; According to the video frame P i The reference frame of the video frame P i The base layer image and M enhancement layer images are encoded to obtain the video frame P i The first stream.

2. The method according to claim 1, characterized in that The feedback information includes the video frame number i-1 and the corresponding layer number W, the layer number W is for the video frame P i-1 The layer number of the image with the highest quality among the images corresponding to the reconstructed frame obtained by the decoding device in the base layer image and the M enhancement layer images, the video frame P i-1 The video frame number i-1 indicates the video frame, and the video frame P is obtained according to the feedback information of the decoding device. i The reference frame includes: The video frame P i-1 The reconstructed frame of the base layer image and the image with layer number W among the M enhancement layer images is determined as the video frame P i reference frame.

3. The method according to claim 1, characterized in that The feedback information includes the video frame number i-1 and the corresponding layer number W, the layer number W is for the video frame P i-1 The layer number of the image with the highest quality among the images corresponding to the reconstructed frame obtained by the decoding device in the base layer image and the M enhancement layer images, the video frame P i-1 The video frame number i-1 indicates the video frame, and the video frame P is obtained according to the feedback information of the decoding device. i The reference frame includes: The first reconstruction frame queue Q i-1 , belonging to the video frame P i-1 The reconstructed frames of the images whose layer numbers are higher than the layer number W are deleted to obtain the first reconstructed frame queue Q i ; Wherein, the first reconstruction frame queue Q i-1 Including the video frame P obtained by the decoding device i-1 The reconstructed frame of the base layer image and the reconstructed frame of the enhancement layer image of the previous video frame, and the video frame P i-1 a reconstructed frame of the base layer image and a reconstructed frame of each enhancement layer image in the M enhancement layer images; According to the video frame P i The reference relationship is from the first reconstructed frame queue Q i Get the video frame P i Reference frame; wherein the video frame P i The reference relationship is the video frame P i With respect to the video frame P i The correspondence between the reference frames used during encoding and decoding.

4. The method according to claim 3, characterized in that The video frame P i The reference frame of the video frame P i The base layer image and M enhancement layer images are encoded to obtain the video frame P i After the first code stream, the method further includes: According to the video frame P i The reference frame of the video frame P i The first code stream is decoded to obtain the video frame P i a reconstructed frame of the base layer image and a reconstructed frame of each enhancement layer image in the M enhancement layer images; The video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each enhancement layer image in the M enhancement layer images are saved to the first reconstructed frame queue Q i middle.

5. The method according to claim 1, wherein The feedback information includes the video frame P i The frame number and corresponding layer number of the previous video frame, the layer number corresponding to the video frame number j is for the video frame P j The layer number of the image with the highest quality among the images corresponding to the reconstructed frame obtained by the decoding device in the base layer image and the M enhancement layer images, the video frame P j is the video frame indicated by the video frame number j, the video frame P j The video frame P i Any frame in the previous video frame, the video frame P is obtained according to the feedback information of the decoding device i Reference frames include: According to the video frame P i The reference relationship is to obtain the video frame P from the first reconstruction frame queue. i Candidate reconstructed frame; wherein the video frame P i The reference relationship is the video frame P i With respect to the video frame P i The correspondence between the reference frames used in encoding and decoding; According to the video frame P i The frame number and corresponding layer number of the previous video frame are obtained from the video frame P i The candidate reconstructed frame obtains the video frame P i The reference frame; Wherein, the video frame P i The reference frame includes the video frame P i Among the candidate reconstructed frames, the quality of the image corresponding to any reconstructed frame F is not higher than the quality of the image indicated by the layer number corresponding to the video frame number of the video frame to which the reconstructed frame F belongs.

6. The method according to claim 5, characterized in that The video frame P i The reference frame of the video frame P i The base layer image and M enhancement layer images are encoded to obtain the video frame P i After the first code stream, the method further includes: According to the video frame P i The reference frame of the video frame P i The first code stream is decoded to obtain the video frame P i a reconstructed frame of the base layer image and a reconstructed frame of each enhancement layer image in the M enhancement layer images; The video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each enhancement layer image in the M enhancement layer images are saved in the first reconstructed frame queue.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: The video frame P i The first code stream is sent to the decoding device.

8. A video decoding method based on quality grading, characterized in that: include: Get video frame P i The second stream; According to the video frame P i The reference frame of the video frame P i The second code stream is decoded to obtain the video frame P i The first reconstructed frame and N second reconstructed frames; wherein the video frame P i The first reconstructed frame is the video frame P i The reconstructed frame of the base layer image, the video frame P i The second reconstructed frame is the video frame P i A reconstructed frame of the enhancement layer image, wherein N is an integer greater than or equal to 0; Feedback information is sent to the encoding device, where the feedback information is used to indicate information of the video frame received by the decoding device; the feedback information includes at least one video frame number and a layer number corresponding to each video frame number in the at least one video frame number, where the layer number corresponding to each video frame number is the layer number of the base layer image or the enhancement layer image of the video frame indicated by the video frame number.

9. The method according to claim 8, characterized in that The video frame P i The reference frame includes the video frame P indicated by the video frame number i-1. i-1 The reconstructed frame of the base layer image and the image with layer number W in the enhancement layer image; Wherein, the layer number W is the video frame P obtained by the decoding device i-1 The layer number of the highest quality image among the N enhancement layer images corresponding to the N second reconstructed frames.

10. The method according to claim 8, characterized in that The video frame P i The reference frame is based on the video frame P i The reference relationship is from the second reconstructed frame queue Q i-1 Obtained from Wherein, the video frame P i The reference relationship is the video frame P i With respect to the video frame P i The corresponding relationship between the reference frames used in encoding and decoding, the second reconstructed frame queue Q i-1 Including the video frame P obtained by the decoding device i The reconstructed frame of the base layer image of the previous video frame and the reconstructed frame of each enhancement layer image in the N enhancement layer images.

11. The method according to claim 10, characterized in that The method further comprises: The video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each enhancement layer image in the N enhancement layer images are saved to the second reconstructed frame queue Q i-1 To obtain the second reconstructed frame queue Q i .

12. The method according to claim 10, characterized in that The feedback information includes the video frame number i-1 and the layer number W.

13. A video encoding device, characterized in that include: Compression unit, used to compress video frame P i Compress to obtain the video frame P i The base layer image and M enhancement layer images; wherein the video frame P i is any frame other than the first frame of the video; M is an integer greater than 0; An acquisition unit is configured to acquire the video frame P according to feedback information from a decoding device. i The feedback information is used to indicate information of a video frame received by the decoding device; the feedback information includes at least one video frame number and a layer number corresponding to each video frame number in the at least one video frame number, and the layer number corresponding to each video frame number is a layer number of a base layer image or an enhancement layer image of the video frame indicated by the video frame number; The encoding unit is configured to encode the video frame P according to the encoding unit. i The reference frame of the video frame P i The base layer image and M enhancement layer images are encoded to obtain the video frame P i The first stream.

14. The encoding device according to claim 13, characterized in that The feedback information includes the video frame number i-1 and the corresponding layer number W, the layer number W is for the video frame P i-1 The layer number of the image with the highest quality among the images corresponding to the reconstructed frame obtained by the decoding device in the base layer image and the M enhancement layer images, the video frame P i-1 For the video frame indicated by the video frame number i-1, the acquiring unit is specifically configured to: According to the video frame P i-1 The reconstructed frame of the base layer image and the image with layer number W among the M enhancement layer images is determined as the video frame P i reference frame.

15. The encoding device according to claim 13, characterized in that The feedback information includes the video frame number i-1 and the corresponding layer number W, the layer number W is for the video frame P i-1 The layer number of the image with the highest quality among the images corresponding to the reconstructed frame obtained by the decoding device in the base layer image and the M enhancement layer images, the video frame P i-1 For the video frame indicated by the video frame number i-1, the acquiring unit is specifically configured to: The first reconstruction frame queue Q i-1 , belonging to the video frame P i-1 The reconstructed frames of the images whose layer numbers are higher than the layer number W are deleted to obtain the first reconstructed frame queue Q i ; Wherein, the first reconstruction frame queue Q i-1 Including the video frame P obtained by the decoding device i-1 The reconstructed frame of the base layer image and the reconstructed frame of the enhancement layer image of the previous video frame, and the video frame P i-1 a reconstructed frame of the base layer image and a reconstructed frame of each enhancement layer image in the M enhancement layer images; According to the video frame P i The reference relationship is from the first reconstructed frame queue Q i Get the video frame P i Reference frame; wherein the video frame P i The reference relationship is the video frame P i With respect to the video frame P i The correspondence between the reference frames used during encoding and decoding.

16. The encoding device according to claim 15, characterized in that The video frame P i The reference frame of the video frame P i The base layer image and M enhancement layer images are encoded to obtain the video frame P i After the first code stream, the encoding device further includes: A reconstruction unit is configured to reconstruct the video frame P i The reference frame of the video frame P i The first code stream is decoded to obtain the video frame P i a reconstructed frame of the base layer image and a reconstructed frame of each enhancement layer image in the M enhancement layer images; A storage unit is used to store the video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each enhancement layer image in the M enhancement layer images are saved to the first reconstructed frame queue Q i middle.

17. The encoding device according to claim 13, characterized in that The feedback information includes the video frame P i The frame number and corresponding layer number of the previous video frame, the layer number corresponding to the video frame number j is for the video frame P j The layer number of the image with the highest quality among the images corresponding to the reconstructed frame obtained by the decoding device in the base layer image and the M enhancement layer images, the video frame P j is the video frame indicated by the video frame number j, the video frame P j The video frame P i For any frame in the previous video frame, the acquisition unit is specifically configured to: According to the video frame P i The reference relationship is to obtain the video frame P from the first reconstruction frame queue. i Candidate reconstructed frame; wherein the video frame P i The reference relationship is the video frame P i With respect to the video frame P i The correspondence between the reference frames used in encoding and decoding; According to the video frame P i The frame number and corresponding layer number of the previous video frame are obtained from the video frame P i The candidate reconstructed frame obtains the video frame P i The reference frame; Wherein, the video frame P i The reference frame includes the video frame P i Among the candidate reconstructed frames, the quality of the image corresponding to any reconstructed frame F is not higher than the quality of the image indicated by the layer number corresponding to the video frame number of the video frame to which the reconstructed frame F belongs.

18. The encoding device according to claim 17, characterized in that The video frame P i The reference frame of the video frame P i The base layer image and M enhancement layer images are encoded to obtain the video frame P i After the first code stream, the encoding device further includes: A reconstruction unit is configured to reconstruct the video frame P i The reference frame of the video frame P i The first code stream is decoded to obtain the video frame P i a reconstructed frame of the base layer image and a reconstructed frame of each enhancement layer image in the M enhancement layer images; A storage unit is used to store the video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each enhancement layer image in the M enhancement layer images are saved in the first reconstructed frame queue.

19. The encoding device according to any one of claims 13 to 18, characterized in that: The encoding device further comprises: A sending unit, configured to send the video frame P i The first code stream is sent to the decoding device.

20. A video decoding device, characterized in that: include: Acquisition unit, used to obtain video frame P i The second stream; A decoding unit, configured to decode the video frame P i The reference frame of the video frame P i The second code stream is decoded to obtain the video frame P i The first reconstructed frame and N second reconstructed frames; wherein the video frame P i The first reconstructed frame is the video frame P i The reconstructed frame of the base layer image, the video frame P i The second reconstructed frame is the video frame P i The reconstructed frame of the enhancement layer image; N is an integer greater than or equal to 0; A sending unit is used to send feedback information to the encoding device, where the feedback information is used to indicate information of the video frame received by the decoding device; the feedback information includes at least one video frame number and a layer number corresponding to each video frame number in the at least one video frame number, where the layer number corresponding to each video frame number is the layer number of the basic layer image or the enhanced layer image of the video frame indicated by the video frame number.

21. The decoding device according to claim 20, characterized in that The video frame P i The reference frame includes the video frame P indicated by the video frame number i-1. i-1 The reconstructed frame of the base layer image and the image with layer number W in the enhancement layer image; Wherein, the layer number W is the video frame P obtained by the decoding device i-1 The layer number of the highest quality image among the N enhancement layer images corresponding to the N second reconstructed frames.

22. The decoding device according to claim 20, characterized in that The video frame P i The reference frame is based on the video frame P i The reference relationship is from the second reconstructed frame queue Q i-1 Obtained from Wherein, the video frame P i The reference relationship is the video frame P i With respect to the video frame P i The corresponding relationship between the reference frames used in encoding and decoding, the second reconstructed frame queue Q i-1 Including the video frame P obtained by the decoding device i The reconstructed frame of the base layer image of the previous video frame and the reconstructed frame of each enhancement layer image in the N enhancement layer images.

23. The decoding device according to claim 22, characterized in that The decoding device further comprises: A storage unit is used to store the video frame P i The reconstructed frame of the base layer image and the reconstructed frame of each enhancement layer image in the N enhancement layer images are saved to the second reconstructed frame queue Q i-1 To obtain the second reconstructed frame queue Q i .

24. The decoding device according to claim 23, characterized in that The feedback information includes the video frame number i-1 and the layer number W.

25. An encoding device, characterized in that include: a memory for storing instructions; as well as a processor coupled to the memory; When the processor executes the instruction, the method according to any one of claims 1 to 7 is performed.

26. A decoding device, characterized in that include: a memory for storing instructions; as well as a processor coupled to the memory; When the processor executes the instruction, the method according to any one of claims 8 to 12 is performed.

27. A chip system, characterized in that: The chip system is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through lines; the interface circuit is used to receive a signal from the memory of the electronic device and send the signal to the processor, the signal including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the method as described in any one of claims 1 to 12.

28. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 12.

29. A computer program product, characterized in that The method comprises computer instructions, which, when executed on an electronic device, enable the electronic device to execute the method according to any one of claims 1 to 12.

Images