Image encoding and decoding method and device

Abstract

The invention provides an image encoding and decoding method and device. The image encoding and decoding method comprises the following steps: receiving a code stream; obtaining a residual image corresponding to the current frame and a first reference frame list according to the code stream; acquiring a virtual reference frame corresponding to the current frame; acquiring a second reference frame list corresponding to the current frame according to the virtual reference frame and the first reference frame list; obtaining a prediction image corresponding to the current frame based on the second reference frame list; and performing reconstruction according to the prediction image and the residual image to obtain a reconstructed image of the current frame. According to the invention, the inter-frame coding performance can be improved, so that the code rate is further reduced.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular to an image encoding and decoding method and device. Background Art

[0002] Many scenarios (for example, games, virtual reality (VR) / augmented reality (AR), etc.) require rendering to generate images, which are more realistic and can improve the user experience. Rendering requires strong computing power, and the end-side device is limited by objective physical conditions such as device size and power consumption, and the computing power is often much weaker than that of the cloud-side server; therefore, rendering is usually deployed on the cloud-side server, which performs rendering and then compresses the rendered image / video and sends it to the end-side device for display. As people's requirements for rendering quality continue to increase, and the clarity of display devices continues to improve, the image quality and resolution of rendered images / videos are also constantly improving, which will lead to an increase in the bit rate overhead of the compressed rendered image / video, which in turn increases network bandwidth usage and causes greater interaction delays.

[0003] Therefore, reducing the bit rate is an important research direction. Summary of the invention

[0004] The present application provides an image encoding and decoding method and device to improve inter-frame encoding performance, thereby further reducing the bit rate.

[0005] In a first aspect, the present application provides an image decoding method, comprising: receiving a code stream; obtaining a residual image and a first reference frame list corresponding to a current frame according to the code stream; obtaining a virtual reference frame corresponding to the current frame; obtaining a second reference frame list corresponding to the current frame according to the virtual reference frame and the first reference frame list; obtaining a predicted image corresponding to the current frame based on the second reference frame list; and reconstructing according to the predicted image and the residual image to obtain a reconstructed image of the current frame.

[0006] In this embodiment, by modifying the reference frame list and adding the factor of the virtual reference frame to the reference frame list, the time domain correlation between the virtual reference frame and the current frame can be fully utilized, the inter-frame coding performance can be improved, and the bit rate can be further reduced.

[0007] The decoder can parse the bitstream to obtain the residual image of the current frame. In addition, when the encoder (such as the server of the end-cloud collaborative system) performs inter-frame prediction on the current frame, it creates a reference frame list for it according to certain rules, and marks the original reference frame in the list. The mark can be transmitted to the decoder along with the bitstream as a syntax element. Based on this, when processing the current frame, the decoder can obtain the aforementioned mark by parsing the bitstream, thereby adding the reconstructed image with the mark to the reference frame list of the current frame. At this time, the reference frame list only contains the reconstructed image, which can be called the first reference frame list.

[0008] Considering the diversity of video images, some image frames may use original reference frames to obtain better encoding / decoding effects, while some image frames may use virtual reference frames to obtain better encoding / decoding effects. Therefore, the video encoding of the present application can integrate the scheme of using original reference frames and the scheme of using virtual reference frames. For example, a new syntax element is added, and a value is assigned to the syntax element to indicate whether the decoder uses the virtual reference frame; or, a preset rule is used to make the decoder determine whether to use the virtual reference frame.

[0009] Optionally, when it is determined not to use the virtual reference frame, the decoder may refer to a standard video coding scheme and obtain an original reference frame from the first reference frame list for inter-frame prediction.

[0010] Optionally, when it is determined to use a virtual reference frame, the decoder can generate a virtual reference frame. A virtual reference frame refers to a reference frame generated based on prior information, which includes decoded information inside the encoder and information other than the video / image to be encoded; the virtual reference frame can be used as supplementary information for inter-frame coding to further remove video temporal redundancy.

[0011] In the present application, when a virtual reference frame is used, it is necessary to solve the problem of initializing the virtual reference frame, which involves the application, use and release of memory space.

[0012] In a possible implementation, the virtual reference frame is stored in a storage space of a non-decoded picture buffer DPB, and the storage space may be a brand new storage area that is newly applied for.

[0013] At this time, the operations of the storage space of the virtual reference frame include:

[0014] 1. Apply for a new cache as storage space for the virtual reference frame;

[0015] 2. The virtual reference frames in the above storage space are managed through the reference frame list and added to the reference frame list;

[0016] 3. Before writing the header file, release the cache of the virtual reference frame and restore the reference frame list;

[0017] 4. Process the next frame and repeat the above steps.

[0018] In one possible implementation, the virtual reference frame is stored in a designated space within the DPB. For example, the DPB can provide storage space for 16 original reference frames, and one or more spaces may remain. The virtual reference frame generated by the decoder may be stored in the remaining one or more spaces, and the virtual reference frame of the subsequent image frame may overwrite the previously stored content.

[0019] At this time, the operations of the storage space of the virtual reference frame include:

[0020] 1. Extend the reused DPB to add a reconstructed frame buffer and reuse the buffer to copy the virtual reference frame to the aforementioned buffer;

[0021] 2. The virtual reference frame in the above-mentioned extended buffer is added to the reference frame list through the reference frame management;

[0022] 3. Before writing the header file, release the cache of the virtual reference frame and restore the reference frame list;

[0023] 4. Process the next frame and repeat the above steps. The virtual reference frame of the current frame covers the virtual reference frame of the previous frame.

[0024] Optionally, the virtual reference frame can be set as a short-term reference frame, for example, its POC value is set to the POC value of the corresponding reconstructed frame; or, the virtual reference frame can be set as a long-term reference frame, for example, its POC value is set to the POC value of the current frame to be decoded.

[0025] The difference between the second reference frame list and the first reference frame list is that it not only includes the original reference frames, but also includes virtual reference frames. In this application, the decoder can obtain the second reference frame list by any of the following methods:

[0026] (1) Virtual reference frame insertion: Insert the virtual reference frame into the first position of the first reference frame list (e.g., n Position) to obtain a second reference frame list;

[0027] (2) fused reference frame insertion: inserting the fused reference frame into the first position of the first reference frame list to obtain a second reference frame list;

[0028] (3) Virtual reference frame replacement: replacing the first reference frame in the first reference frame list with the virtual reference frame to obtain a second reference frame list;

[0029] (4) fused reference frame replacement: replacing the first reference frame in the first reference frame list with the fused reference frame to obtain a second reference frame list;

[0030] The first reference frame is the reference frame located at the first position in the first reference frame list. The reference frames in the first reference frame list can be marked with different indexes to distinguish them, so the index can also represent the position in the first reference frame list. For example, n Can represent the index of the first reference frame, then I n The position may refer to the position where the first reference frame is located (ie, the first position mentioned above).

[0031] The above fused reference frame is obtained based on the virtual reference frame and the second reference frame in the first reference frame list, where the second reference frame is located at the second position in the first reference frame list (e.g., R n Position) of the reference frame. n is the index of the second reference frame in the first reference frame list.

[0032] In the above methods (1)-(4), methods (1) and (2) adopt an insertion method, in which a reference frame related to the virtual reference frame is inserted into the original first reference frame list to obtain a second reference frame list. The inserted reference frame is a virtual reference frame in method (1) and a fused reference frame in method (2).

[0033] Accordingly, because the new reference frame is inserted into the position of the first reference frame, the index of at least one reference frame in the first reference frame list is increased by 1, and the total number of reference frames in the first reference frame list is increased by 1, and the index of the aforementioned at least one reference frame is greater than or equal to the index of the first reference frame. n Position, then the first reference list is originally located at I n The index of the reference frame at the position and the reference frame after it are increased by 1, for example, I n ,I n+1 ,I n+2 , ..., after adding 1, the index becomes I n+1 ,I n+2 ,I n+3 ,… In addition, the total number of reference frames in the first reference frame list is increased by 1, thereby obtaining a second reference frame list. That is, each time a new reference frame is inserted, the total number of reference frames in the second reference frame list increases by one. In a possible implementation, when a new reference frame is inserted after the last reference frame in the first reference frame list, since it has no effect on the positions of all reference frames in the first reference frame list, in this case, the index of any other reference frame is not changed, and only the total number of reference frames in the first reference frame list is increased by 1, thereby obtaining a second reference frame list.

[0034] In the above methods (1)-(4), methods (3) and (4) adopt a replacement method, replacing a reference frame in the original first reference frame list with a reference frame related to the virtual reference frame to obtain a second reference frame list. The replaced reference frame is a virtual reference frame in method (3) and a fused reference frame in method (4).

[0035] Accordingly, because the first reference frame in the first reference frame list is replaced by a new reference frame, relevant information of the first reference frame is recorded, and the information may include the POC, pixel information, codec information, related settings, etc. of the first reference frame, so that when the first reference frame list is restored from the second reference frame list, the first reference frame can be re-acquired based on the information, so as to replace the virtual reference frame / fused reference frame in the second reference frame list with the first reference frame. The index of the original reference frame in the first reference frame list remains unchanged, and the number of reference frames remains unchanged.

[0036] It should be noted that, in addition to the above four methods, the present application may also adopt other methods to obtain the second reference frame list, and no specific limitation is made to this.

[0037] The present application can determine which of the above methods (1)-(4) to use to obtain the second reference frame list by the following methods:

[0038] 1. A logo

[0039] An identifier can be called a first identifier, which is used to indicate whether a virtual reference frame is used to obtain the second reference frame list, and when a virtual reference frame is used to obtain the second reference frame list, the method used to obtain the second reference frame list is carried in the bitstream as a syntax element, and the decoder can parse the bitstream to obtain it.

[0040] a. When the first identifier is the first value, method (1) is used to obtain the second reference frame list; or,

[0041] b. When the first identifier is the second value, use method (2) to obtain the second reference frame list; or,

[0042] c. When the first identifier is the third value, use method (3) to obtain the second reference frame list; or,

[0043] d. When the first identifier is the fourth value, method (4) is used to obtain the second reference frame list.

[0044] In the method, the first identifier is reused, and when the first identifier is the thirteenth value, it indicates that the second reference frame list is obtained without using a virtual reference frame.

[0045] The present application may add a new syntax element (first identifier), and by assigning different values ​​to the syntax element, the decoder may read the syntax element to determine which method to use to obtain the second reference frame list. At this time, the first identifier may have 5 valid values, for example, 0 (the thirteenth value, indicating that the virtual reference frame is not used), 1 (the first value, indicating the insertion of the virtual reference frame), 2 (the second value, indicating the insertion of the fused reference frame), 3 (the third value, indicating the replacement of the virtual reference frame), and 4 (the fourth value, indicating the replacement of the fused reference frame).

[0046] 2. Two logos

[0047] The two identifiers may be respectively referred to as the second identifier and the third identifier. The second identifier is used to indicate whether a virtual reference frame is used to obtain the second reference frame list, and when a virtual reference frame is used to obtain the second reference frame list, an insertion or replacement method is used to obtain the second reference frame list. The third identifier is used to indicate a virtual reference frame or a fused reference frame. Both are carried in the bitstream as syntax elements, and the decoder can parse the bitstream to obtain them.

[0048] a. When the second identifier is the fifth value and the third identifier is the sixth value, method (1) is used to obtain the second reference frame list; or,

[0049] b. When the second identifier is the fifth value and the third identifier is the seventh value, method (2) is used to obtain the second reference frame list; or,

[0050] c. When the second identifier is the eighth value and the third identifier is the sixth value, the second reference frame list is obtained by using method (3); or,

[0051] d. When the second identifier is the eighth value and the third identifier is the seventh value, method (4) is used to obtain the second reference frame list.

[0052] In the method, the second identifier is reused, and when the second identifier is the fourteenth value, it indicates that the second reference frame list is obtained without using a virtual reference frame.

[0053] The present application may add two new syntax elements (a second identifier and a third identifier), and by assigning different values ​​to the two syntax elements, the decoder may read the syntax elements to determine which method to use to obtain the second reference frame list. At this time, the second identifier may have three valid values, for example, 0 (the fourteenth value, indicating that the virtual reference frame is not used), 1 (the fifth value, indicating insertion), and 2 (the eighth value, indicating replacement), and the third identifier may have two valid values, for example, 0 (the sixth value, indicating a virtual reference frame) and 1 (the seventh value, indicating a fused reference frame).

[0054] 3. Three logos

[0055] The three identifiers can be respectively called the fourth identifier, the fifth identifier and the sixth identifier. The fourth identifier is used to indicate whether a virtual reference frame is used to obtain the second reference frame list. The fifth identifier is used to indicate whether the second reference frame list is obtained by insertion or replacement. The sixth identifier is used to indicate a virtual reference frame or a fused reference frame. The three are carried in the bitstream as syntax elements, and the decoder can parse the bitstream to obtain them.

[0056] a. When the fourth identifier is a ninth value, the fifth identifier is a tenth value, and the sixth identifier is an eleventh value, method (1) is used to obtain the second reference frame list; or,

[0057] b. When the fourth identifier is the ninth value, the fifth identifier is the tenth value, and the sixth identifier is the twelfth value, adopt method (2) to obtain the second reference frame list; or,

[0058] c. When the fourth identifier is the ninth value, the fifth identifier is the thirteenth value, and the sixth identifier is the eleventh value, adopt method (3) to obtain the second reference frame list; or,

[0059] d. When the fourth identifier is the ninth value, the fifth identifier is the thirteenth value, and the sixth identifier is the twelfth value, method (4) is used to obtain the second reference frame list.

[0060] In this method, the fourth identifier is reused, and when the fourth identifier is the fifteenth value, it indicates that the second reference frame list is obtained without using a virtual reference frame.

[0061] The present application may add three new syntax elements (a fourth identifier, a fifth identifier, and a sixth identifier), and by assigning different values ​​to the three syntax elements, the decoder may read the syntax elements to determine which method to use to obtain the second reference frame list. At this time, the fourth identifier may have two valid values, for example, 0 (the fifteenth value, indicating that the virtual reference frame is not used), 1 (the ninth value, indicating that the virtual reference frame is used), the fifth identifier may have two valid values, for example, 0 (the tenth value, indicating insertion), 1 (the thirteenth value, indicating replacement), and the sixth identifier may have two valid values, for example, 0 (the eleventh value, indicating a virtual reference frame), 1 (the twelfth value, indicating a fused reference frame).

[0062] It should be noted that, in addition to the above three methods, the present application may also adopt other methods. For example, the encoding end and the decoding end may pre-set which method to use to obtain the second reference frame list based on a preset rule, and both parties may perform corresponding steps based on the setting. Therefore, the present application embodiment does not specifically limit how to determine which of the above methods (1)-(4) is used to obtain the second reference frame list.

[0063] In the present application, one or more virtual reference frames can be obtained corresponding to the current frame, so the above process of obtaining the second reference frame list can be performed once for each virtual reference frame until all virtual reference frames of the current frame are incorporated into the reference frame list. In addition, according to the inter-frame prediction characteristics of the current frame, the virtual reference frame can also include forward and backward two types. Correspondingly, the first reference frame list and the second reference frame list include a forward reference frame list and a backward reference frame list, respectively. According to the forward and backward classifications, the virtual reference frames are incorporated into the corresponding reference frame lists.

[0064] Optionally, the newly added syntax elements may be stored in the following levels or structures, including but not limited to: Sequence Parameter Set (SPS), Video Parameter Set (VPS), Picture Parameter Set (PPS), Picture Header (PH), Slice Header (SH). This application does not make specific limitations on this.

[0065] In one possible implementation, when the decoder determines that the reference frame is a virtual reference frame based on the bitstream description information parsed from the bitstream, the corresponding virtual reference frame can be selected from the second reference frame list as the reference frame; then, the predicted image can be determined based on the virtual reference frame.

[0066] In one possible implementation, when the decoder determines that the reference frame is the original reference frame based on the bitstream description information parsed from the bitstream, the corresponding original reference frame can be selected as the reference frame from the second reference frame list; then, the predicted image can be determined based on the original reference frame.

[0067] Due to the movement of the camera and the object, the position of the same object in the current frame may be different from the position in the previous frame. The motion information generated by the rendering engine can more accurately describe the coordinate correspondence (geometric relationship) between the two frames. Therefore, the virtual reference frame obtained based on the aforementioned correspondence can more accurately predict the current frame than the original reference frame. Based on this, the factor of the virtual reference frame is introduced into the reference frame queue table, which can improve the inter-frame coding performance and further reduce the bit rate.

[0068] In a possible implementation manner, the second reference frame list may be restored to the first reference frame list.

[0069] Considering compatibility with various standards, after the reference frame list is changed to the second reference frame list, the decoder can also restore it to the first reference frame list. The restoration method may include the following two methods:

[0070] 1. When the second reference frame list is obtained by the insertion method of the above methods (1) and (2), the inserted reference frame can be deleted, and then the index of at least one reference frame in the second reference frame list is reduced by 1, and the total number of reference frames in the second reference frame list is reduced by 1, and the index of the aforementioned at least one reference frame is greater than or equal to the index of the aforementioned inserted reference frame. For example, the inserted reference frame is located at I of the second reference frame list. n Position, then the second reference list is originally located at I n+1 The index of the reference frame at the position and the index of the reference frame after it are respectively reduced by 1, for example, I n+1 ,I n+2 ,I n+3 , ..., after decrementing 1, the index becomes I n ,I n+1 ,I n+2 , ... In addition, the total number of reference frames in the second reference frame list is reduced by 1, thereby restoring the first reference frame list. In a possible implementation, when the deleted reference frame is the last reference frame in the second reference frame list, since it has no effect on the positions of other reference frames in the second reference frame list, in this case, the index of any other reference frame is not changed, and only the total number of reference frames in the second reference frame list is reduced by 1, thereby restoring the first reference frame list.

[0071] 2. When the second reference frame list is obtained by using the replacement method of the above methods (3) and (4), the replaced reference frame can be changed back to the original reference frame, which has been recorded before and can be directly read from the corresponding storage space. The index of the original reference frame in the second reference frame list remains unchanged, and the number of reference frames remains unchanged.

[0072] After the predicted image is obtained, the decoded residual image can be superimposed with the predicted image to obtain a reconstructed image. Then, the reconstructed image can be loop filtered to obtain a filtered reconstructed image, and the filtered reconstructed image can be output to a memory.

[0073] In a second aspect, the present application provides an image encoding method, including: obtaining a first reference frame list corresponding to a current frame; obtaining a virtual reference frame corresponding to the current frame; obtaining a second reference frame list corresponding to the current frame based on the virtual reference frame and the first reference frame list; obtaining a predicted image corresponding to the current frame based on the second reference frame list; predicting the current frame based on the predicted image to obtain a residual image corresponding to the current frame.

[0074] In a possible implementation manner, acquiring the second reference frame list corresponding to the current frame according to the virtual reference frame and the first reference frame list includes any one of the following methods:

[0075] (1) inserting the virtual reference frame into the first position of the first reference frame list to obtain the second reference frame list;

[0076] (2) inserting the fused reference frame into the first position of the first reference frame list to obtain the second reference frame list;

[0077] (3) replacing the first reference frame in the first reference frame list with the virtual reference frame to obtain the second reference frame list;

[0078] (4) replacing the first reference frame in the first reference frame list with the fused reference frame to obtain the second reference frame list;

[0079] Among them, the first reference frame is the reference frame located at the first position in the first reference frame list; the fused reference frame is obtained based on the virtual reference frame and the second reference frame in the first reference frame list, and the second reference frame is the reference frame located at the second position in the first reference frame list.

[0080] In a possible implementation manner, acquiring a second reference frame list corresponding to the current frame according to the virtual reference frame and the first reference frame list includes:

[0081] Acquire a first identifier from the bitstream, where the first identifier is used to indicate whether to use the virtual reference frame to acquire the second reference frame list, and a method for acquiring the second reference frame list when the virtual reference frame is used to acquire the second reference frame list;

[0082] When the first identifier is a first value, the second reference frame list is acquired by adopting method (1); or,

[0083] When the first identifier is the second value, method (2) is used to obtain the second reference frame list; or,

[0084] When the first identifier is the third value, method (3) is used to obtain the second reference frame list; or,

[0085] When the first identifier is the fourth value, method (4) is used to obtain the second reference frame list.

[0086] In a possible implementation manner, acquiring a second reference frame list corresponding to the current frame according to the virtual reference frame and the first reference frame list includes:

[0087] Acquire a second identifier and a third identifier from the bitstream, the second identifier being used to indicate whether to use the virtual reference frame to acquire the second reference frame list, and when the virtual reference frame is used to acquire the second reference frame list, the second reference frame list is acquired by an insertion or replacement method, and the third identifier is used to indicate the virtual reference frame or the fused reference frame;

[0088] When the second identifier is the fifth value and the third identifier is the sixth value, the second reference frame list is acquired by using method (1); or,

[0089] When the second identifier is the fifth value and the third identifier is the seventh value, the second reference frame list is acquired by using method (2); or,

[0090] When the second identifier is the eighth value and the third identifier is the sixth value, the second reference frame list is acquired by using method (3); or,

[0091] When the second identifier is the eighth value and the third identifier is the seventh value, method (4) is used to obtain the second reference frame list.

[0092] In a possible implementation manner, acquiring a second reference frame list corresponding to the current frame according to the virtual reference frame and the first reference frame list includes:

[0093] Acquire a fourth identifier, a fifth identifier, and a sixth identifier from the bitstream, wherein the fourth identifier is used to indicate whether to use the virtual reference frame to acquire the second reference frame list, the fifth identifier is used to indicate that the second reference frame list is acquired by an insertion or replacement method, and the sixth identifier is used to indicate the virtual reference frame or the fused reference frame;

[0094] When the fourth identifier is a ninth value, the fifth identifier is a tenth value, and the sixth identifier is an eleventh value, the method (1) is used to obtain the second reference frame list; or,

[0095] When the fourth identifier is a ninth value, the fifth identifier is a tenth value, and the sixth identifier is a twelfth value, method (2) is used to obtain the second reference frame list; or,

[0096] When the fourth identifier is a ninth value, the fifth identifier is a thirteenth value, and the sixth identifier is an eleventh value, the method (3) is used to obtain the second reference frame list; or,

[0097] When the fourth identifier is a ninth value, the fifth identifier is a thirteenth value, and the sixth identifier is a twelfth value, method (4) is used to obtain the second reference frame list.

[0098] In a possible implementation manner, when the first identifier is a thirteenth value, it indicates that the second reference frame list is acquired without using the virtual reference frame.

[0099] In a possible implementation manner, when the second identifier is the fourteenth value, it indicates that the second reference frame list is acquired without using the virtual reference frame.

[0100] In a possible implementation manner, when the fourth identifier is the fifteenth value, it indicates that the second reference frame list is acquired without using the virtual reference frame.

[0101] In a possible implementation, the method further includes:

[0102] When method (1) or method (2) is adopted, the index of at least one reference frame in the first reference frame list is increased by 1, and the total number of reference frames in the first reference frame list is increased by 1, and the index of the at least one reference frame is greater than or equal to the index of the first reference frame.

[0103] In a possible implementation, the method further includes:

[0104] When method (3) or method (4) is adopted, relevant information of the first reference frame is recorded.

[0105] In a possible implementation manner, after acquiring the predicted image corresponding to the current frame from the second reference frame list, the method further includes:

[0106] The second reference frame list is restored to the first reference frame list.

[0107] In a possible implementation, the virtual reference frame is stored in a storage space of a non-decoded picture buffer DPB; or, the virtual reference frame is stored in a designated space in the DPB.

[0108] In one possible implementation, the identifier obtained from the bitstream is stored in a sequence parameter set SPS, a video parameter set VPS, a picture parameter set PPS, a picture header PH or a slice header SH, and the identifier includes a first identifier, a second identifier, a third identifier, a fourth identifier, a fifth identifier and a sixth identifier.

[0109] In a possible implementation, obtaining the virtual reference frame corresponding to the current frame includes: obtaining rendering information from the bitstream; and obtaining the virtual reference frame according to the decoded reconstructed image and the rendering information, wherein the rendering information comes from a server.

[0110] The server can be a single server or a server cluster. The server can be a cloud server, a physical server, a cluster server, or a high-defense server. The server can be a tower server, a blade server, a rack server, or a cabinet server. The server can be a general-purpose server or a dedicated server. This application does not impose any restrictions on the type of server.

[0111] In a possible implementation, the method is applied to a terminal-cloud collaborative system, where the terminal-cloud collaborative system includes a server and a terminal device, the server includes an encoder, and the terminal device includes a decoder.

[0112] In a third aspect, the present application provides an image decoding device, comprising: a receiving module for receiving a code stream; a prediction module for obtaining a residual image and a first reference frame list corresponding to a current frame based on the code stream; obtaining a virtual reference frame corresponding to the current frame; obtaining a second reference frame list corresponding to the current frame based on the virtual reference frame and the first reference frame list; obtaining a predicted image corresponding to the current frame based on the second reference frame list; and a reconstruction module for reconstructing based on the predicted image and the residual image to obtain a reconstructed image of the current frame.

[0113] In a possible implementation, the prediction module is specifically used to implement any one of the following methods:

[0114] (1) inserting the virtual reference frame into the first position of the first reference frame list to obtain the second reference frame list;

[0115] (2) inserting the fused reference frame into the first position of the first reference frame list to obtain the second reference frame list;

[0116] (3) replacing the first reference frame in the first reference frame list with the virtual reference frame to obtain the second reference frame list;

[0117] (4) replacing the first reference frame in the first reference frame list with the fused reference frame to obtain the second reference frame list;

[0118] Among them, the first reference frame is the reference frame located at the first position in the first reference frame list; the fused reference frame is obtained based on the virtual reference frame and the second reference frame in the first reference frame list, and the second reference frame is the reference frame located at the second position in the first reference frame list.

[0119] In a possible implementation, the prediction module is specifically used to obtain a first identifier from the code stream, the first identifier being used to indicate whether the virtual reference frame is used to obtain the second reference frame list, and a method for obtaining the second reference frame list when the virtual reference frame is used to obtain the second reference frame list; when the first identifier is a first value, method (1) is used to obtain the second reference frame list; or, when the first identifier is a second value, method (2) is used to obtain the second reference frame list; or, when the first identifier is a third value, method (3) is used to obtain the second reference frame list; or, when the first identifier is a fourth value, method (4) is used to obtain the second reference frame list.

[0120] In a possible implementation, the prediction module is specifically used to obtain a second identifier and a third identifier from the code stream, the second identifier is used to indicate whether to use the virtual reference frame to obtain the second reference frame list, and when the virtual reference frame is used to obtain the second reference frame list, the second reference frame list is obtained by insertion or replacement, and the third identifier is used to indicate the virtual reference frame or the fused reference frame; when the second identifier is a fifth value and the third identifier is a sixth value, the second reference frame list is obtained by method (1); or, when the second identifier is a fifth value and the third identifier is a seventh value, the second reference frame list is obtained by method (2); or, when the second identifier is an eighth value and the third identifier is a sixth value, the second reference frame list is obtained by method (3); or, when the second identifier is an eighth value and the third identifier is a sixth value, the second reference frame list is obtained by method (4).

[0121] In a possible implementation manner, the prediction module is specifically used to obtain a fourth identifier, a fifth identifier and a sixth identifier from the code stream, the fourth identifier is used to indicate whether to use the virtual reference frame to obtain the second reference frame list, the fifth identifier is used to indicate whether to use the insertion or replacement method to obtain the second reference frame list, and the sixth identifier is used to indicate the virtual reference frame or the fused reference frame; when the fourth identifier is a ninth value, the fifth identifier is a tenth value, and the sixth identifier is an eleventh value, the second reference frame list is obtained by method (1); or, when the fourth identifier is a ninth value, the fifth identifier is a tenth value, and the sixth identifier is a twelfth value, the second reference frame list is obtained by method (2); or, when the fourth identifier is a ninth value, the fifth identifier is a thirteenth value, and the sixth identifier is an eleventh value, the second reference frame list is obtained by method (3); or, when the fourth identifier is a ninth value, the fifth identifier is a thirteenth value, and the sixth identifier is an eleventh value, the second reference frame list is obtained by method (4).

[0122] In a possible implementation manner, when the first identifier is a thirteenth value, it indicates that the second reference frame list is acquired without using the virtual reference frame.

[0123] In a possible implementation manner, when the second identifier is the fourteenth value, it indicates that the second reference frame list is acquired without using the virtual reference frame.

[0124] In a possible implementation manner, when the fourth identifier is the fifteenth value, it indicates that the second reference frame list is acquired without using the virtual reference frame.

[0125] In one possible implementation, the prediction module is further used to, when method (1) or method (2) is adopted, add 1 to the index of at least one reference frame in the first reference frame list, and add 1 to the total number of reference frames in the first reference frame list, and the index of the at least one reference frame is greater than or equal to the index of the first reference frame.

[0126] In a possible implementation manner, the prediction module is further used to record relevant information of the first reference frame when method (3) or method (4) is adopted.

[0127] In a possible implementation manner, the prediction module is further configured to restore the second reference frame list to the first reference frame list.

[0128] In a possible implementation, the virtual reference frame is stored in a storage space of a non-decoded picture buffer DPB; or, the virtual reference frame is stored in a designated space in the DPB.

[0129] In one possible implementation, the identifier obtained from the bitstream is stored in a sequence parameter set SPS, a video parameter set VPS, a picture parameter set PPS, a picture header PH or a slice header SH, and the identifier includes a first identifier, a second identifier, a third identifier, a fourth identifier, a fifth identifier and a sixth identifier.

[0130] In a possible implementation, the prediction module is specifically configured to obtain rendering information from the bitstream; and obtain the virtual reference frame according to the decoded reconstructed image and the rendering information, wherein the rendering information comes from a server.

[0131] In a possible implementation, the apparatus is applied to a terminal-cloud collaborative system, the terminal-cloud collaborative system includes a server and a terminal device, the server includes an encoder, and the terminal device includes a decoder.

[0132] In a fourth aspect, the present application provides a method for generating a bitstream based on the virtual reference frame described in any one of the first aspects above, to generate a bitstream.

[0133] In a possible implementation, the code stream includes a first identifier, which is used to indicate whether to use the virtual reference frame to obtain the second reference frame list, and when using the virtual reference frame to obtain the second reference frame list, a method for obtaining the second reference frame list.

[0134] In a possible implementation, the code stream includes a second identifier and a third identifier, the second identifier is used to indicate whether to use the virtual reference frame to obtain the second reference frame list, and when the virtual reference frame is used to obtain the second reference frame list, the second reference frame list is obtained by insertion or replacement, and the third identifier is used to indicate the virtual reference frame or the fused reference frame.

[0135] In a possible implementation, the code stream includes a fourth identifier, a fifth identifier and a sixth identifier, the fourth identifier is used to indicate whether the virtual reference frame is used to obtain the second reference frame list, the fifth identifier is used to indicate the use of insertion or replacement method to obtain the second reference frame list, and the sixth identifier is used to indicate the virtual reference frame or the fused reference frame.

[0136] In a fifth aspect, the present application provides a terminal device, comprising: a memory and a processor, wherein the memory is coupled to the processor; the memory stores program instructions, and when the program instructions are executed by the processor, the terminal device executes a method as described in any one of the above-mentioned first aspects.

[0137] In a sixth aspect, the present application provides a chip comprising one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of a terminal device and send the signal to the processor, wherein the signal comprises a computer instruction stored in the memory; when the processor executes the computer instruction, the terminal device executes a method as described in any one of the above-mentioned first aspects.

[0138] In a seventh aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes a method as described in any one of the above-mentioned first aspects.

[0139] In an eighth aspect, the present application provides a computer program product, wherein the computer program product comprises computing instructions, and when the computing instructions are executed by a computer or a processor, the steps of the method described in any one of the above-mentioned first aspects are executed.

[0140] In a ninth aspect, the present application provides a code stream, the code stream comprising a first identifier, the first identifier being used to indicate whether a virtual reference frame is used to obtain a second reference frame list, and a method for obtaining the second reference frame list when the virtual reference frame is used to obtain the second reference frame list;

[0141] When the first identifier is a first value, the second reference frame list is acquired by adopting method (1); or,

[0142] When the first identifier is the second value, method (2) is used to obtain the second reference frame list; or,

[0143] When the first identifier is the third value, method (3) is used to obtain the second reference frame list; or,

[0144] When the first identifier is a fourth value, adopting method (4) to obtain the second reference frame list;

[0145] The method for obtaining the second reference frame list includes:

[0146] (1) inserting a virtual reference frame into a first position of a first reference frame list to obtain the second reference frame list;

[0147] (2) inserting the fused reference frame into the first position of the first reference frame list to obtain the second reference frame list;

[0148] (3) replacing the first reference frame in the first reference frame list with the virtual reference frame to obtain the second reference frame list;

[0149] (4) Replace the first reference frame in the first reference frame list with the fused reference frame to obtain the second reference frame list.

[0150] In a possible implementation manner, when the first identification value is 0, it indicates that the second reference frame list is acquired without using the virtual reference frame.

[0151] In a tenth aspect, the present application provides a code stream, the code stream comprising a second identifier and a third identifier, the second identifier is used to indicate whether a virtual reference frame is used to obtain a second reference frame list, and when the virtual reference frame is used to obtain the second reference frame list, an insertion or replacement method is used to obtain the second reference frame list, and the third identifier is used to indicate the virtual reference frame or the fused reference frame;

[0152] When the second identifier is the fifth value and the third identifier is the sixth value, the second reference frame list is acquired by using method (1); or,

[0153] When the second identifier is the fifth value and the third identifier is the seventh value, the second reference frame list is acquired by using method (2); or,

[0154] When the second identifier is the eighth value and the third identifier is the sixth value, the second reference frame list is acquired by using method (3); or,

[0155] When the second identifier is an eighth value and the third identifier is a seventh value, the second reference frame list is acquired by using method (4);

[0156] The method for obtaining the second reference frame list includes:

[0157] (1) inserting a virtual reference frame into a first position of a first reference frame list to obtain the second reference frame list;

[0158] (2) inserting the fused reference frame into the first position of the first reference frame list to obtain the second reference frame list;

[0159] (3) replacing the first reference frame in the first reference frame list with the virtual reference frame to obtain the second reference frame list;

[0160] (4) Replace the first reference frame in the first reference frame list with the fused reference frame to obtain the second reference frame list.

[0161] In a possible implementation manner, when the second identification value is 0, it indicates that the second reference frame list is obtained without using the virtual reference frame.

[0162] In an eleventh aspect, the present application provides a code stream, the code stream comprising a fourth identifier, a fifth identifier and a sixth identifier, the fourth identifier being used to indicate whether a virtual reference frame is used to obtain a second reference frame list, the fifth identifier being used to indicate that an insertion or replacement method is used to obtain the second reference frame list, and the sixth identifier being used to indicate the virtual reference frame or the fused reference frame;

[0163] When the fourth identifier is a ninth value, the fifth identifier is a tenth value, and the sixth identifier is an eleventh value, the method (1) is used to obtain the second reference frame list; or,

[0164] When the fourth identifier is a ninth value, the fifth identifier is a tenth value, and the sixth identifier is a twelfth value, method (2) is used to obtain the second reference frame list; or,

[0165] When the fourth identifier is a ninth value, the fifth identifier is a thirteenth value, and the sixth identifier is an eleventh value, the method (3) is used to obtain the second reference frame list; or,

[0166] When the fourth identifier is a ninth value, the fifth identifier is a thirteenth value, and the sixth identifier is a twelfth value, adopting method (4) to obtain the second reference frame list;

[0167] The method for obtaining the second reference frame list includes:

[0168] (1) inserting a virtual reference frame into a first position of a first reference frame list to obtain the second reference frame list;

[0169] (2) inserting the fused reference frame into the first position of the first reference frame list to obtain the second reference frame list;

[0170] (3) replacing the first reference frame in the first reference frame list with the virtual reference frame to obtain the second reference frame list;

[0171] (4) Replace the first reference frame in the first reference frame list with the fused reference frame to obtain the second reference frame list.

[0172] In a possible implementation manner, when the fourth identification value is 0, it indicates that the second reference frame list is acquired without using the virtual reference frame.

[0173] In a twelfth aspect, the present application provides an apparatus for storing a code stream, the apparatus comprising: a receiver and at least one storage medium, the receiver being used to receive the code stream; the at least one storage medium being used to store the code stream; the code stream being a code stream recorded according to any possible implementation manner of aspects 9 to 11, or generated according to a method as described in any one of the above-mentioned aspects 4.

[0174] In a thirteenth aspect, the present application provides a device for transmitting a code stream, the device comprising: a transmitter and at least one storage medium, the at least one storage medium being used to store a code stream, the code stream being a code stream recorded according to any possible implementation method of aspects nine to eleven, or being generated according to a method as described in any one of the above-mentioned fourth aspects; the transmitter being used to obtain the code stream from the storage medium and send the code stream to a terminal side device via a transmission medium.

[0175] In a fourteenth aspect, the present application provides a system for distributing a code stream, the system comprising: at least one storage medium, used to store at least one code stream, the at least one code stream is a code stream recorded according to any possible implementation method of the ninth to eleventh aspects, or is generated according to a method as described in any one of the fourth aspects above; a streaming media device, used to obtain a target code stream from the at least one storage medium and send the target code stream to a terminal side device, wherein the streaming media device includes a content server or a content distribution server. BRIEF DESCRIPTION OF THE DRAWINGS

[0176] Figure 1a is a schematic diagram of an application scenario shown as an example;

[0177] Figure 1b is a schematic diagram of an exemplary end-cloud collaborative system;

[0178] Figure 2a is a schematic diagram of an exemplary encoding process;

[0179] Figure 2b is a schematic diagram of an exemplary coding framework;

[0180] Figure 3a is a schematic diagram of an exemplary decoding process;

[0181] Figure 3b is a schematic diagram of an exemplary decoding framework;

[0182] Figure 4 is a flowchart for exporting a virtual reference frame;

[0183] Figure 5 A flowchart of a process 500 of an image decoding method provided in an embodiment of the present application;

[0184] Figure 6 A schematic diagram of the decoding process of the decoder of the present application;

[0185] Figure 7 A schematic diagram of reference frame list management for this application;

[0186] Figure 8 is the statistical result of the inter-frame coding mode;

[0187] Fig. 9 A schematic diagram of reference frame list management for this application;

[0188] Fig.10 A schematic diagram of reference frame list management for this application;

[0189] Fig.11 A schematic diagram of reference frame list management for this application;

[0190] Fig.12 is a structural diagram of an image decoding device 1200 of the present application;

[0191] Fig.13 A schematic block diagram of a device of the present application. DETAILED DESCRIPTION

[0192] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0193] The terms "first", "second", etc. in the specification embodiments, claims, and drawings of the present application are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, including a series of steps or units. The method, system, product, or device is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0194] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0195] Figure 1a The figure is a schematic diagram of an exemplary application scenario. Figure 1a The application scenario shown is a terminal-cloud collaborative rendering scenario. This application can be applied to various terminal-cloud collaborative scenarios; for example, cloud games, VR / AR conferences, etc., which are not limited in this application. This application is explained using cloud games as an example.

[0196] Reference Figure 1a In one possible manner, terminal devices such as mobile phones, personal computers (PCs), VR glasses, AR helmets, tablet computers, and other terminal devices may be included, and the first rendering parameters may be sent to the central server (such as a game server) through the edge server. Then, the central server may render the three-dimensional scene based on the rendering parameters (the rendering parameters may only include the first rendering parameters uploaded by the terminal device, or include the first rendering parameters uploaded by the terminal device and the second rendering parameters generated by the central server) to obtain a rendered image (such as a game image). Afterwards, the central server encodes the rendered image based on part of the rendering intermediate results (hereinafter referred to as the first rendering intermediate results) in the rendering intermediate results generated during the rendering process to obtain a code stream and distribute the code stream to each terminal device through the edge server. Subsequently, the terminal device may render the three-dimensional scene based on the rendering parameters (the rendering parameters include the first rendering parameters and the fifth rendering parameters, the fifth rendering parameters are also generated by the terminal device, and the fifth rendering parameters may include the same parameters as the second rendering parameters), and generate the first rendering intermediate result during the rendering process; then the terminal device decodes the code stream based on the first rendering intermediate result to obtain a reconstructed image.

[0197] Reference Figure 1aIn one possible manner, terminal devices such as mobile phones, PCs, VR glasses, AR helmets, tablet computers, and other terminal devices may be included, and the first rendering parameters may be sent to the central server through the edge server. Then, the central server may render the three-dimensional scene based on the rendering parameters (the rendering parameters may only include the first rendering parameters uploaded by the terminal device, or include the first rendering parameters uploaded by the terminal device and the second rendering parameters generated by the server) to obtain a rendered image. After that, the central server encodes the rendered image based on the first rendering intermediate result generated during the rendering process; and encodes the second rendering intermediate result (the second rendering intermediate result is part of the first rendering intermediate result) to obtain a code stream and distribute the code stream to each terminal device through the edge server. Subsequently, the terminal device may render the three-dimensional scene based on the rendering parameters (the rendering parameters include the first rendering parameters and the fifth rendering parameters), and generate a third rendering intermediate result (the third rendering intermediate result is the part of the first rendering intermediate result other than the second rendering intermediate result) during the rendering process; then, the terminal device generates a first rendering intermediate result based on the second rendering intermediate result and the third rendering intermediate result, and decodes the code stream based on the first rendering intermediate result to obtain a reconstructed image.

[0198] Reference Figure 1a In one possible way, terminal devices such as mobile phones, PCs, VR glasses, AR helmets, tablets, and other terminal devices may send the first rendering parameters to the central server through the edge server. Then, the central server can render the three-dimensional scene based on the rendering parameters (the rendering parameters include the first rendering parameters uploaded by the terminal device and the second rendering parameters generated by the central server) to obtain a rendered image. After that, the central server encodes the rendered image based on the first rendering intermediate result generated in the rendering process to obtain a code stream and writes the third rendering parameters (the third rendering parameters include all or part of the parameters in the second rendering parameters) into the code stream; then, the central server distributes the code stream to each terminal device through the edge server. Subsequently, the terminal device may perform rendering processing on the three-dimensional scene based on the rendering parameters (when the third rendering parameters include all parameters of the second rendering parameters, the rendering parameters may include the first rendering parameters and the third rendering parameters parsed from the code stream; when the third rendering parameters include part of the parameters of the second rendering parameters, the rendering parameters may include the first rendering parameters, the third rendering parameters parsed from the code stream, and the fourth rendering parameters generated by the terminal device, and the fourth rendering parameters are the parameters of the second rendering parameters other than the third rendering parameters), and generate a first rendering intermediate result during the rendering process; then, the terminal device decodes the code stream based on the first rendering intermediate result to obtain a reconstructed image.

[0199] Reference Figure 1a In one possible way, terminal devices such as mobile phones, PCs, VR glasses, AR helmets, tablets, and other terminal devices may send the first rendering parameters to the central server through the edge server. Then, the central server can render the three-dimensional scene based on the rendering parameters (including the first rendering parameters uploaded by the terminal device and the second rendering parameters generated by the central server) to obtain a rendered image. After that, the central server encodes the rendered image based on the first rendering intermediate result generated during the rendering process, and encodes the second rendering intermediate result to obtain a code stream and write the third rendering parameter into the code stream; then, the central server distributes the code stream to each terminal device through the edge server. Subsequently, the terminal device can perform rendering processing on the three-dimensional scene based on the rendering parameters (when the third rendering parameters include all parameters of the second rendering parameters, the rendering parameters may include the first rendering parameters and the third rendering parameters parsed from the code stream; when the third rendering parameters include part of the parameters of the second rendering parameters, the rendering parameters may include the first rendering parameters, the third rendering parameters parsed from the code stream, and the fourth rendering parameters generated by the terminal device), and generate a third rendering intermediate result during the rendering process; then the terminal device generates a first rendering intermediate result based on the second rendering intermediate result and the third rendering intermediate result, and decodes the code stream based on the first rendering intermediate result to obtain a reconstructed image.

[0200] In this way, part or all of the rendering is executed on the terminal device, and the server may not send the rendering intermediate result or only send part of the rendering intermediate result to the terminal device; therefore, the present application can reduce the interaction delay while ensuring that the bit rate overhead of the data stream transmitted from the server to the terminal device is effectively reduced. And because the rendering intermediate result is highly correlated with the rendered image, the present application encodes the rendered image based on the rendering intermediate result to ensure the quality of image reconstruction. In addition, the data volume of the rendering parameters is much smaller than the data volume of the rendering intermediate result; therefore, even if the server of the present application also sends the rendering parameters to the terminal device, the bit rate overhead of the data stream transmitted from the server to the terminal device in the present application is also less than the bit rate overhead of the data stream transmitted in the prior art.

[0201] Exemplarily, rendering parameters may refer to all parameters that are input to the graphics rendering engine and are required for the graphics rendering engine to perform rendering processing, and may include various parameters used for rendering, the position and color vectors of all light sources, the position vectors of players or observers, the sampling method of each texture, the position coordinates of each scene object and other information, the motion trajectory of moving objects, skeletal animation parameters, etc., which are not limited in this application. Among them, the first rendering parameter and the second rendering parameter may constitute the rendering parameter (i.e., all parameters that are input to the graphics rendering engine and are required for the graphics rendering engine to perform rendering). It should be noted that the type of parameters contained in the fifth rendering parameter generated by the terminal device is the same as the type of parameters contained in the second rendering parameter generated by the central server, and the precision of the parameters contained in the fifth rendering parameter is less than or equal to the precision of the parameters contained in the second rendering parameter.

[0202] Exemplarily, the intermediate rendering result may refer to the intermediate data for generating the image / video to be displayed generated by the graphics rendering engine during the process of generating the image to be displayed (i.e., the rendered image) / video (i.e., the rendered video). Exemplarily, the intermediate rendering result may include, but is not limited to: Computer Graphics Motion Vector (CGMV), intermediate rendering image (intermediate rendering image refers to the image generated before generating the final rendered image (i.e., the rendered image mentioned above), and the computational complexity of the intermediate rendering image is lower than the computational complexity of the rendered image; wherein, the intermediate rendering image may be, for example, an intermediate rendering image without indirect lighting rendering, an intermediate rendering image without mirror reflection processing, an intermediate rendering image without highlight processing, etc.), position map, normal map, albedo map, specular intensity map, mesh ID, material ID (wherein, each material map corresponds to a Material ID), rendering ID (Render ID) (each object (or a three-dimensional object model) corresponds to a Render ID) ID, depth information, etc., which are not restricted in this application. Among them, the first rendering intermediate result is part of all rendering intermediate results generated in the rendering process, and the second rendering intermediate result and the third rendering intermediate result can constitute the first rendering intermediate result. It should be noted that the type of intermediate results included in the first rendering intermediate result generated by the terminal device is the same as the type of intermediate results included in the first rendering intermediate result generated by the central server, and the accuracy of the intermediate results included in the first rendering intermediate result generated by the terminal device is less than or equal to the accuracy of the intermediate results included in the first rendering intermediate result generated by the central server.

[0203] based on Figure 1a In this application scenario, this application proposes a terminal-cloud collaboration system that can be used as follows: Figure 1b shown.

[0204] Figure 1b The figure is a schematic diagram of an end-cloud collaborative system shown as an example. Figure 1b The end-cloud collaboration system shown may include a server and a terminal device.

[0205] Reference Figure 1b , for example, the server may refer to Figure 1a The central server in the application may be a single server or a server cluster, and this application does not impose any restrictions on this.

[0206] Exemplarily, the server may include a first rendering module, an encoder, and a first communication module. It should be understood that Figure 1b This is only an example of the present application. The server of the present application may include Figure 1b The present application is not limited to more or fewer modules as shown.

[0207] Exemplarily, the first rendering module may be used to perform rendering processing on the three-dimensional scene based on the rendering parameters to obtain a rendered image and an intermediate rendering result;

[0208] Exemplarily, the encoder may be configured to encode the rendered image based on the first rendering intermediate result, encode the second rendering intermediate result, and write the second rendering parameter into a bitstream.

[0209] Exemplarily, the first communication module can be used to communicate with other electronic devices, for example, it can send the code stream output by the encoder to the terminal device.

[0210] Exemplarily, terminal devices include, but are not limited to: personal computers, computer workstations, smart phones, tablet computers, servers, smart cameras, smart cars or other types of cellular phones, media consumption devices, wearable devices (such as VR / AR helmets, VR glasses), set-top boxes, game consoles, etc.

[0211] Continue to refer to Figure 1b , exemplarily, the terminal device may include a second communication module, a second rendering module, a decoder and a display module. It should be understood that, Figure 1b This is only an example of the present application. The terminal device of the present application may include Figure 1b The present application is not limited to more or fewer modules as shown.

[0212] Exemplarily, the second communication module can be used to communicate with other electronic devices, for example, to receive a code stream sent by a server.

[0213] Exemplarily, the second rendering module can be used to render the three-dimensional scene based on the rendering parameters to obtain the first rendering intermediate result or the third rendering intermediate result. It should be understood that the second rendering module can also generate a rendered image, but the rendered image is not used for display by the subsequent display module.

[0214] Exemplarily, the decoder may be configured to decode the code stream based on the first rendering intermediate result to obtain a reconstructed image.

[0215] Exemplarily, a display module may be used to display the reconstructed image.

[0216] It should be understood that the present application does not limit the video coding standards used by the encoder and decoder, for example, may include but is not limited to: H.264 / AVC (Advanced Video Coding), H.265 / HEVC (High Efficiency Video Coding), H.266 / VVC (Versatile Video Coding), AV1 (AOMedia Video 1, "AOMedia" refers to a video coding standard developed by the Open Media Video Alliance), etc., as well as extended standards of these video coding standards; in addition, new video coding standards and extended standards generated with the development of video coding and decoding technology may also be included.

[0217] The following is based on Figure 1a and Figure 1b , the encoding and decoding process is explained.

[0218] Figure 2a FIG. 1 is a schematic diagram of an exemplary encoding process. Figure 2a In the embodiment, the server does not send the first rendering intermediate result to the terminal device, nor does it send the second rendering parameter to the terminal device. Figure 2a In the embodiment, encoding a frame of rendered image is taken as an example for description.

[0219] S201, the server renders the three-dimensional scene based on rendering parameters to obtain a rendered image; wherein the rendering parameters include first rendering parameters obtained from the terminal device.

[0220] For example, the server may obtain rendering parameters; then, the rendering parameters may be input into a graphics rendering engine, and the graphics rendering engine may render the three-dimensional scene based on the rendering parameters to obtain a rendered image. Figure 1b The first rendering module in .

[0221] Exemplarily, the server may obtain a first rendering parameter from a terminal device (wherein the first rendering parameter is generated by the terminal device). In one possible manner, when the first rendering parameter generated by the terminal device is all the parameters required for the graphics rendering engine to perform rendering processing, the server may determine the first rendering parameter as the rendering parameter; that is, the rendering parameter obtained by the server may only include the first rendering parameter. In one possible manner, when the first rendering parameter generated by the terminal device is part of the parameters required for the graphics rendering engine to perform rendering processing, the server may also generate a second rendering parameter; then, the first rendering parameter and the second rendering parameter are determined as rendering parameters. That is, the rendering parameters obtained by the server may include the first rendering parameter and the second rendering parameter. The first rendering parameter and the second rendering parameter are described in detail in subsequent embodiments.

[0222] It should be noted that the server can obtain the first rendering parameter from the terminal device according to a preset period, wherein the preset period can be set as required, and this application does not limit this. It should be understood that the first rendering parameter obtained by the server from the terminal device each time can be used to render one or more frames of images.

[0223] Exemplarily, the rendering process of the graphics rendering engine can be as follows: the image engine can perform geometric transformation, projection transformation, perspective transformation and window clipping on the three-dimensional object model in the three-dimensional scene based on the input rendering parameters, and then generate a rendered image through the acquired material and light and shadow information.

[0224] S202: The server selects a first rendering intermediate result based on the rendering intermediate results generated during the rendering process.

[0225] Exemplarily, the graphics rendering engine in the first rendering module generates a rendering intermediate result during the rendering process (the rendering process refers to the time period from the moment when the rendering parameters are acquired from the image rendering engine to the moment when the rendered image is obtained), and a first rendering intermediate result can be selected from the rendering intermediate results; the rendered image can be subsequently encoded based on the first rendering intermediate result.

[0226] Exemplarily, part of the rendering intermediate results can be selected from the rendering intermediate results as the first rendering intermediate result. Exemplarily, the present application may encode the rendered image based on the first rendering intermediate result by generating a virtual reference frame based on the first rendering intermediate result, and then encoding the rendered image based on the virtual reference frame; further, part of the rendering intermediate results that can be used to generate the virtual reference frame can be selected from the rendering intermediate results as the first rendering intermediate result. For example, CGMV, a rendering intermediate image, etc. can be selected as the first rendering intermediate result, and the present application does not limit this.

[0227] S203: The server generates a virtual reference frame based on the first rendering intermediate result.

[0228] Exemplarily, a virtual reference frame refers to a reference frame generated based on prior information, the prior information including decoded information inside the encoder and information other than the video / image to be encoded; the virtual reference frame can be used as supplementary information for inter-frame coding to further remove video time domain redundancy.

[0229] Exemplarily, different first rendering intermediate results correspond to different types, for example, the type of CGMV is a motion vector type, the type of the rendering intermediate image is an image type, and so on. In one possible manner, the type of the first rendering intermediate result may be determined; then, based on the first rendering intermediate result and the type of the first rendering intermediate result, a virtual reference frame is generated. Specifically for different types of first rendering intermediate results, the manner of generating a virtual reference frame is described later.

[0230] It should be noted that S201-S202 can be Figure 1b The first rendering module in S203 can be executed by Figure 1b The first rendering module in the process can also be executed by the server. Figure 1b The present application does not limit the execution of other modules other than the modules shown in the figure (the present application takes the first rendering module executing S203 as an example for explanation). The following S204 to S205 can be performed by Figure 1b The encoder in is executed.

[0231] S204: The server predicts the rendered image based on the virtual reference frame to obtain a predicted image.

[0232] S205 , the server encodes a residual image between the predicted image and the rendered image, and writes the encoded data of the residual image into a bitstream; wherein the bitstream does not include the encoded data of the first rendering intermediate result.

[0233] Figure 2b A schematic diagram of an exemplary coding framework is shown in FIG. Figure 2b The embodiment shows Figure 1b The coding frame of the encoder in . Figure 2b The video to be encoded is a rendered video and may include multiple frames of rendered images.

[0234] Reference Figure 2b Exemplarily, the encoder may include: a blocking module, an intra-frame prediction module, an inter-frame prediction module, a transform module, a quantization module, an entropy coding module, an inverse quantization module, an inverse transform module, a loop filtering module and a memory.

[0235] Exemplarily, after the first rendering module generates the virtual reference frame, the virtual reference frame may be inserted into a reference frame list in the memory, and the length of the reference frame list may be increased by 1.

[0236] Continue to refer to Figure 2b , the encoding process of the encoder can be as follows: after the block module of the encoder obtains the video to be encoded, it can block each frame of the rendered image in the video to be encoded to obtain a block to be encoded. For each block to be encoded, the decision module ( Figure 2b (not shown) determines whether to perform inter-frame prediction or intra-frame prediction on the block to be encoded.

[0237] When it is determined to be an intra-frame prediction, the intra-frame prediction module can perform intra-frame prediction to determine the prediction block corresponding to the block to be encoded. After obtaining the prediction block, the residual block between the block to be encoded and the prediction block can be determined; then, the residual block is input into the transformation module, and the transformation module transforms the residual block to obtain the transformation result and outputs the transformation result to the quantization module. Subsequently, the quantization module can quantize the transformation result to obtain the quantization result and output the quantization result to the entropy coding module; then, the entropy coding module can entropy encode the quantization result to obtain the encoded data of the residual block, and write the encoded data of the residual block into the bitstream.

[0238] Exemplarily, the quantization module can also output the quantization result to the inverse quantization module, which performs inverse quantization to obtain the inverse quantization result and outputs the inverse quantization result to the inverse transformation module. Then, the inverse transformation module performs inverse transformation on the inverse quantization result to obtain a decoded residual block. Then, the decoded residual block can be superimposed with the prediction block output by the intra-frame prediction module to obtain a reconstructed block. Subsequently, the reconstructed block is input into the loop filtering module, which performs loop filtering on the reconstructed block to obtain a filtered reconstructed block and output the filtered reconstructed block to the memory. (Wherein, after obtaining the filtered reconstructed block, the loop filtering module can splice the filtered reconstructed block to the corresponding position of the image to be reconstructed; after obtaining a frame of reconstructed image, the reconstructed image can be output to the memory for storage).

[0239] When it is determined that the prediction is inter-frame prediction, the inter-frame prediction module may perform inter-frame prediction to determine the prediction block corresponding to the block to be encoded.

[0240] In one possible approach, the inter-frame prediction module may select a virtual reference frame from a reference frame list in a memory as a reference frame for the rendered image. For a block to be encoded in the rendered image, a matching prediction block (that is, the prediction image described above) may be searched from the virtual reference frame.

[0241] In one possible manner, the inter-frame prediction module can select both the virtual reference frame and the original reference frame from the reference frame list in the memory as candidate reference frames for the rendered image; wherein the original reference frame refers to the reconstructed image. For the block to be encoded in the rendered image, inter-frame prediction can be performed based on multiple candidate reference frames to determine multiple prediction blocks; wherein one candidate reference frame corresponds to one prediction block; and then an optimal prediction block can be selected from the multiple prediction blocks (for example, the rate-distortion cost can be used as an evaluation criterion, and the prediction block with the smallest rate-distortion cost can be determined as the optimal prediction block (that is, the above-mentioned prediction image)).

[0242] After obtaining the prediction block, the residual block between the block to be encoded and the prediction block (that is, the residual image mentioned above) can be determined; then, the residual block is input into the transformation module, which transforms the residual block to obtain the transformation result and outputs the transformation result to the quantization module. Subsequently, the quantization module can quantize the transformation result to obtain the quantization result and output the quantization result to the entropy coding module; then, the entropy coding module can entropy encode the quantization result to obtain the coded data of the residual block, and write the coded data of the residual block into the bitstream (that is, the bitstream obtained in the above S205).

[0243] Exemplarily, the quantization module can also output the quantization result to the inverse quantization module, which performs inverse quantization to obtain the inverse quantization result and outputs the inverse quantization result to the inverse transformation module. Then, the inverse transformation module performs inverse transformation on the inverse quantization result to obtain a decoded residual block. Then, the decoded residual block can be superimposed with the prediction block output by the intra-frame prediction module to obtain a reconstructed block. Subsequently, the reconstructed block is input into the loop filtering module, which performs loop filtering on the reconstructed block to obtain a filtered reconstructed block and output the filtered reconstructed block to the memory. (Wherein, after obtaining the filtered reconstructed block, the loop filtering module can splice the filtered reconstructed block to the corresponding position of the image to be reconstructed; after obtaining a frame of reconstructed image, the reconstructed image can be output to the memory for storage).

[0244] Exemplarily, after the rendered image is encoded, the virtual reference frame in the reference frame list in the memory may be deleted, and the length of the reference frame list may be reduced by 1, so as to avoid the reference frame list being too long and reducing the encoding efficiency.

[0245] Figure 3a FIG. 4 is a schematic diagram showing an exemplary decoding process. Figure 3a Shown is Figure 2a The corresponding decoding process. Figure 3a In an embodiment of the present invention, the terminal device performs rendering processing based on locally generated rendering parameters and generates a first rendering intermediate result. Figure 3a In the embodiment, decoding of one frame of image is taken as an example for description.

[0246] S301, the terminal device receives a code stream.

[0247] Exemplarily, after the server sends the code stream, the terminal device can receive the code stream.

[0248] S302: The terminal device parses the code stream to obtain a parsing result, where the parsing result includes a residual image corresponding to the current frame.

[0249] Figure 3b FIG. 1 is a schematic diagram of an exemplary decoding framework. Figure 3b The embodiment shows Figure 1b The decoding framework of the decoder in .

[0250] Reference Figure 3b Exemplarily, the decoder may include: an entropy decoding module, an inverse quantization module, an inverse transformation module, an intra-frame prediction module, an inter-frame prediction module, a loop filtering module and a memory.

[0251] Exemplarily, after receiving the code stream, the terminal device can parse the code stream to obtain the parsing result. The parsing process can be as follows: extract the encoded data of the residual block from the code stream, and then input the encoded data of the residual block into the entropy decoding module, and the entropy decoding module performs entropy decoding on the encoded data of the residual block to obtain entropy decoded data. Then, the entropy decoded data can be input into the inverse quantization module, and the inverse quantization module performs inverse quantization to obtain inverse quantized data and input the inverse quantized data into the inverse transformation module. Subsequently, the inverse transformation module performs inverse transformation on the inverse quantized data to obtain the decoded residual block (that is, the residual image corresponding to the current frame in S302).

[0252] It should be understood that when the encoder performs lossy encoding on the residual block, the residual block parsed by the decoder is different from the residual block encoded by the encoder. When the encoder performs lossless encoding on the residual block, the residual block parsed by the decoder is the same as the residual block encoded by the encoder.

[0253] It should be noted that the code stream description information for describing the code stream can also be extracted from the code stream, in which case there is no need to perform entropy decoding, inverse quantization and inverse transformation on the code stream description information. The code stream description information can be used in the subsequent decoding process.

[0254] S303, the terminal device renders the three-dimensional scene based on the rendering parameters corresponding to the current frame, and generates a first rendering intermediate result during the rendering process; wherein the rendering parameters include the first rendering parameters generated by the terminal device.

[0255] Exemplarily, after parsing and obtaining the residual image corresponding to the current frame, the second rendering module (which may be a graphics rendering engine included in the second rendering module) may render the three-dimensional scene based on the rendering parameters corresponding to the current frame.

[0256] Exemplarily, when the first rendering parameters uploaded by the terminal device to the server are all the parameters required by the graphics rendering engine for rendering processing, the second rendering module can render the three-dimensional scene based on the first rendering parameters corresponding to the current frame. When the first rendering parameters uploaded by the terminal device are part of the parameters required by the graphics rendering engine for rendering processing, the terminal device can also generate a fifth rendering parameter, and then the second rendering module can render the three-dimensional scene based on the first rendering parameters and the fifth rendering parameters corresponding to the current frame.

[0257] It should be noted that the type of parameters included in the fifth rendering parameter generated by the terminal device is the same as the type of parameters included in the second rendering parameter generated by the central server; the precision of the parameters included in the fifth rendering parameter is less than or equal to the precision of the parameters included in the second rendering parameter.

[0258] It should be noted that the type of intermediate results contained in the first rendering intermediate result generated by the terminal device is the same as the type of intermediate results contained in the first rendering intermediate result generated by the server; the precision of the intermediate results contained in the first rendering intermediate result generated by the terminal device is less than or equal to the precision of the intermediate results contained in the first rendering intermediate result generated by the server.

[0259] It should be noted that the graphics rendering engine of the terminal device can also obtain a rendered image by rendering processing; since the computing power of the terminal device is lower than that of the server, the image quality of the rendered image obtained by the graphics rendering engine of the terminal device through rendering processing (wherein the image quality may include subjective quality and objective quality (objective quality such as rate-distortion performance, PSNR (Peak Signal to Noise Ratio)) is lower than the image quality of the rendered image obtained by the graphics rendering engine of the server through rendering processing; therefore, the terminal device may not display the rendered image obtained by the graphics rendering engine of the terminal device through rendering processing, but may display the reconstructed image obtained by decoding the code stream.

[0260] S304: The terminal device generates a virtual reference frame based on the first rendering intermediate result.

[0261] Exemplarily, S304 may refer to the description of S203 above, which will not be repeated here.

[0262] Exemplarily, after generating the virtual reference frame, the terminal device may reconstruct the residual image corresponding to the current frame based on the virtual reference frame to obtain a reconstructed image. The image quality of the reconstructed image is higher than the rendered image obtained by the graphics rendering engine of the terminal device. Subsequently, the terminal device may display the reconstructed image.

[0263] It should be noted that S301 to S303 may be executed by the second rendering module in the terminal device, S304 may be executed by the second rendering module in the terminal device, or by the second rendering module in the terminal device. Figure 1b The following S305 to S306 can be executed by other modules other than the modules shown in the figure (the present application takes the second rendering module executing S304 as an example for explanation). Figure 1b The decoder in is executed.

[0264] S305: The terminal device predicts the current frame based on the virtual reference frame to obtain a predicted image.

[0265] S306: The terminal device reconstructs based on the predicted image and the residual image to obtain a reconstructed image of the current frame.

[0266] Refer again Figure 3b Exemplarily, after generating the virtual reference frame, the second rendering module may insert the virtual reference frame into a reference frame list in the memory and increase the length of the reference frame list by 1.

[0267] Refer again Figure 3b Exemplarily, when it is determined that it is an intra-frame prediction based on the code stream description information parsed from the code stream, the intra-frame prediction module can perform intra-frame prediction to obtain a prediction block. Afterwards, the decoded residual block can be superimposed with the prediction block to obtain a reconstructed block. Then, the reconstructed block can be loop filtered to obtain a filtered reconstructed block and the filtered reconstructed block can be output to a memory. (After obtaining the filtered reconstructed block, the loop filtering module can splice the filtered reconstructed block to the corresponding position of the image to be reconstructed; after obtaining a frame of reconstructed image, the reconstructed image can be output to the memory for storage).

[0268] Refer again Figure 3b For example, when it is determined that inter-frame prediction is performed according to the code stream description information parsed from the code stream, the inter-frame prediction module can perform inter-frame prediction to obtain a prediction block (that is, the prediction image in the above S305).

[0269] In one possible approach, when the inter-frame prediction module determines that the reference frame is a virtual reference frame based on the bitstream description information parsed from the bitstream, the inter-frame prediction module can select the corresponding virtual reference frame as the reference frame from the reference frame list in the memory; then, the prediction block can be determined based on the virtual reference frame.

[0270] In one possible approach, when the inter-frame prediction module determines from the code stream description information parsed from the code stream that the reference frame is an original reference frame, the inter-frame prediction module can select the corresponding original reference frame from the reference frame list in the memory as the reference frame; then, the prediction block can be determined based on the original reference frame.

[0271] After obtaining the prediction block, the decoded residual block can be superimposed with the prediction block to obtain a reconstructed block. Then, the reconstructed block can be loop filtered to obtain a filtered reconstructed block and the filtered reconstructed block can be output to the memory. (After obtaining the filtered reconstructed block, the loop filtering module can splice the filtered reconstructed block to the corresponding position of the image to be reconstructed; after obtaining a frame of reconstructed image, the reconstructed image can be output to the memory).

[0272] It should be understood that the terminal device can also generate a code stream based on the first rendering parameter, and then send it to the server according to a preset period. In this way, the server can obtain the first rendering parameter. In addition, since the data volume of the first rendering parameter is small, the terminal device can encode the first rendering parameter and write the encoded data of the first rendering parameter into the code stream before sending it to the server; the terminal device may not encode the first rendering parameter, but directly write the first rendering parameter into the code stream and send it to the server; this application does not limit this.

[0273] In this way, the present application performs all rendering on the terminal device, so that the server does not need to send the intermediate rendering results to the terminal device; therefore, the present application can reduce the interaction delay while ensuring that the bit rate overhead of the data stream transmitted from the server to the terminal device is effectively reduced. In addition, since the intermediate rendering results are highly correlated with the rendered image, the present application encodes the rendered image based on the intermediate rendering results to ensure the quality of image reconstruction.

[0274] Based on the above content, the keywords involved in this application are first explained.

[0275] Virtual Reference Frame

[0276] Video coding (video coding as used in this article can mean video encoding or video decoding) generally refers to processing a sequence of pictures to form a video or video sequence. In the field of video coding, the terms "picture", "frame" or "image" can be used as synonyms. A video sequence includes a series of pictures, which are further divided into slices, and slices are further divided into blocks. Video coding can be encoded / decoded in units of pictures, slices or blocks. Therefore, based on the processing unit of video coding, the virtual reference frame can also be replaced by a virtual reference block.

[0277] Standard video codecs usually use input / reconstructed videos to generate reference frames, which are referred to as original reference frames in this application. Virtual reference frames are reference frames obtained based on prior information, which includes decoded information inside the codec and information other than the video to be encoded / decoded. They can be used as supplementary information for inter-frame prediction and act on motion estimation and motion compensation processes to further remove video temporal redundancy.

[0278] For example, Figure 4 This is the export flow chart of the virtual reference frame, such as Figure 4 As shown, based on the decoded information, a virtual reference frame is derived for inter-frame prediction. In video coding, the reference frames used for motion estimation are stored in the decoded picture buffer (DPB). The current frame can have one or more reference frames in the forward and / or backward directions. A virtual reference frame is generated using one or more reference frames that are closer to the current frame using the picture order count (POC). For example, the original reference frame of the current frame is 0, and a virtual reference frame 0' is generated in combination with a game rendering engine filter. The virtual reference frame 0' can be placed in the DPB for motion estimation or motion compensation processes.

[0279] In traditional video coding, the inter-frame prediction mode refers to the decoded image and puts the decoded image into the reference frame list for management. This application is based on the standard reference frame list management method, introduces virtual reference frames, and improves the reference frame list management rules, which can effectively improve the encoding and decoding performance and efficiency of rendered video.

[0280] Figure 5 Flowchart of process 500 of the image decoding method provided in an embodiment of the present application. Process 500 may be performed by the terminal device (decoding end) described above, specifically, a video decoder (decoder for short) of the decoding end, and is applicable to the process of inter-frame prediction. Process 500 is described as a series of steps or operations, and it should be understood that process 500 may be performed in various orders and / or occur simultaneously, not limited to Figure 5 The execution order shown. Process 500 may include:

[0281] Step 501: Receive a code stream.

[0282] Step 502: Obtain a residual image and a first reference frame list corresponding to the current frame according to the code stream.

[0283] See above Figure 3aIn S302 of the illustrated embodiment, the decoder can parse the bitstream to obtain the residual image of the current frame. In addition, when the encoding end (such as a server in an end-cloud collaborative system) performs inter-frame prediction on the current frame, it creates a reference frame list for it according to certain rules, and marks the original reference frame in the list with a mark, which can be transmitted to the decoder along with the bitstream as a syntax element. Based on this, when processing the current frame, the decoder can obtain the aforementioned mark by parsing the bitstream, thereby adding the reconstructed image with the mark to the reference frame list of the current frame. At this time, the reference frame list only contains the reconstructed image, which can be called the first reference frame list.

[0284] Step 503: Obtain a virtual reference frame corresponding to the current frame.

[0285] Considering the diversity of video images, some image frames may use original reference frames to obtain better encoding / decoding effects, while some image frames may use virtual reference frames to obtain better encoding / decoding effects. Therefore, the video encoding of the present application can integrate the scheme of using original reference frames and the scheme of using virtual reference frames. For example, a new syntax element is added, and a value is assigned to the syntax element to indicate whether the decoder uses the virtual reference frame; or, a preset rule is used to make the decoder determine whether to use the virtual reference frame.

[0286] Optionally, when it is determined not to use the virtual reference frame, the decoder may refer to a standard video coding scheme and obtain an original reference frame from the first reference frame list for inter-frame prediction.

[0287] Optionally, when it is determined to use a virtual reference frame, the decoder may generate a virtual reference frame (the method for generating a virtual reference frame may refer to Figure 2a S203 or Figure 3a S304 of the illustrated embodiment).

[0288] A virtual reference frame refers to a reference frame generated based on prior information, which includes decoded information inside the encoder and information other than the video / image to be encoded. The virtual reference frame can be used as supplementary information for inter-frame coding to further remove video time domain redundancy.

[0289] In the present application, when a virtual reference frame is used, it is necessary to solve the problem of initializing the virtual reference frame, which involves the application, use and release of memory space.

[0290] In a possible implementation, the virtual reference frame is stored in a storage space of a non-decoded picture buffer DPB, and the storage space may be a brand new storage area that is newly applied for.

[0291] At this time, the operations of the storage space of the virtual reference frame include:

[0292] 1. Apply for a new cache as storage space for the virtual reference frame;

[0293] 2. The virtual reference frames in the above storage space are managed through the reference frame list and added to the reference frame list;

[0294] 3. Before writing the header file, release the cache of the virtual reference frame and restore the reference frame list;

[0295] 4. Process the next frame and repeat the above steps.

[0296] In one possible implementation, the virtual reference frame is stored in a designated space within the DPB. For example, the DPB can provide storage space for 16 original reference frames, and one or more spaces may remain. The virtual reference frame generated by the decoder may be stored in the remaining one or more spaces, and the virtual reference frame of the subsequent image frame may overwrite the previously stored content.

[0297] At this time, the operations of the storage space of the virtual reference frame include:

[0298] 1. Extend the reused DPB to add a reconstructed frame buffer and reuse the buffer to copy the virtual reference frame to the aforementioned buffer;

[0299] 2. The virtual reference frame in the above-mentioned extended buffer is added to the reference frame list through the reference frame management;

[0300] 3. Before writing the header file, release the cache of the virtual reference frame and restore the reference frame list;

[0301] 4. Process the next frame and repeat the above steps. The virtual reference frame of the current frame covers the virtual reference frame of the previous frame.

[0302] Optionally, the virtual reference frame can be set as a short-term reference frame, for example, its POC value is set to the POC value of the corresponding reconstructed frame; or, the virtual reference frame can be set as a long-term reference frame, for example, its POC value is set to the POC value of the current frame to be decoded.

[0303] Step 504: Obtain a second reference frame list corresponding to the current frame according to the virtual reference frame and the first reference frame list.

[0304] The difference between the second reference frame list and the first reference frame list is that it not only includes the original reference frames, but also includes virtual reference frames. In this application, the decoder can obtain the second reference frame list by any of the following methods:

[0305] (1) Virtual reference frame insertion: Insert the virtual reference frame into the first position of the first reference frame list (e.g., n Position) to obtain a second reference frame list;

[0306] (2) fused reference frame insertion: inserting the fused reference frame into the first position of the first reference frame list to obtain a second reference frame list;

[0307] (3) Virtual reference frame replacement: replacing the first reference frame in the first reference frame list with the virtual reference frame to obtain a second reference frame list;

[0308] (4) fused reference frame replacement: replacing the first reference frame in the first reference frame list with the fused reference frame to obtain a second reference frame list;

[0309] The first reference frame is the reference frame located at the first position in the first reference frame list. The reference frames in the first reference frame list can be marked with different indexes to distinguish them, so the index can also represent the position in the first reference frame list. For example, n Can represent the index of the first reference frame, then I n The position may refer to the position where the first reference frame is located (ie, the first position mentioned above).

[0310] The above fused reference frame is obtained based on the virtual reference frame and the second reference frame in the first reference frame list, where the second reference frame is located at the second position in the first reference frame list (e.g., R n Position) of the reference frame. n is the index of the second reference frame in the first reference frame list.

[0311] In the above methods (1)-(4), methods (1) and (2) adopt an insertion method, in which a reference frame related to the virtual reference frame is inserted into the original first reference frame list to obtain a second reference frame list. The inserted reference frame is a virtual reference frame in method (1) and a fused reference frame in method (2).

[0312] Accordingly, because the new reference frame is inserted into the position of the first reference frame, the index of at least one reference frame in the first reference frame list is increased by 1, and the total number of reference frames in the first reference frame list is increased by 1, and the index of the aforementioned at least one reference frame is greater than or equal to the index of the first reference frame. n Position, then the first reference list is originally located at I n The index of the reference frame at the position and the reference frame after it are increased by 1, for example, I n ,I n+1 ,I n+2 , ..., after adding 1, the index becomes I n+1 ,I n+2 ,I n+3,… In addition, the total number of reference frames in the first reference frame list is increased by 1, thereby obtaining a second reference frame list. That is, each time a new reference frame is inserted, the total number of reference frames in the second reference frame list increases by one. In a possible implementation, when a new reference frame is inserted after the last reference frame in the first reference frame list, since it has no effect on the positions of all reference frames in the first reference frame list, in this case, the index of any other reference frame is not changed, and only the total number of reference frames in the first reference frame list is increased by 1, thereby obtaining a second reference frame list.

[0313] In the above methods (1)-(4), methods (3) and (4) adopt a replacement method, replacing a reference frame in the original first reference frame list with a reference frame related to the virtual reference frame to obtain a second reference frame list. The replaced reference frame is a virtual reference frame in method (3) and a fused reference frame in method (4).

[0314] Accordingly, because the first reference frame in the first reference frame list is replaced by a new reference frame, relevant information of the first reference frame is recorded, and the information may include the POC, pixel information, codec information, related settings, etc. of the first reference frame, so that when the first reference frame list is restored from the second reference frame list, the first reference frame can be re-acquired based on the information, so as to replace the virtual reference frame / fused reference frame in the second reference frame list with the first reference frame. The index of the original reference frame in the first reference frame list remains unchanged, and the number of reference frames remains unchanged.

[0315] It should be noted that, in addition to the above four methods, the present application may also adopt other methods to obtain the second reference frame list, and no specific limitation is made to this.

[0316] The present application can determine which of the above methods (1)-(4) to use to obtain the second reference frame list by the following methods:

[0317] 1. A logo

[0318] An identifier can be called a first identifier, which is used to indicate whether a virtual reference frame is used to obtain the second reference frame list, and when a virtual reference frame is used to obtain the second reference frame list, the method used to obtain the second reference frame list is carried in the bitstream as a syntax element, and the decoder can parse the bitstream to obtain it.

[0319] a. When the first identifier is the first value, method (1) is used to obtain the second reference frame list; or,

[0320] b. When the first identifier is the second value, use method (2) to obtain the second reference frame list; or,

[0321] c. When the first identifier is the third value, use method (3) to obtain the second reference frame list; or,

[0322] d. When the first identifier is the fourth value, method (4) is used to obtain the second reference frame list.

[0323] In the method, the first identifier is reused, and when the first identifier is the thirteenth value, it indicates that the second reference frame list is obtained without using a virtual reference frame.

[0324] The present application may add a new syntax element (first identifier), and by assigning different values ​​to the syntax element, the decoder may read the syntax element to determine which method to use to obtain the second reference frame list. At this time, the first identifier may have 5 valid values, for example, 0 (the thirteenth value, indicating that the virtual reference frame is not used), 1 (the first value, indicating the insertion of the virtual reference frame), 2 (the second value, indicating the insertion of the fused reference frame), 3 (the third value, indicating the replacement of the virtual reference frame), and 4 (the fourth value, indicating the replacement of the fused reference frame).

[0325] 2. Two logos

[0326] The two identifiers may be respectively referred to as the second identifier and the third identifier. The second identifier is used to indicate whether a virtual reference frame is used to obtain the second reference frame list, and when a virtual reference frame is used to obtain the second reference frame list, an insertion or replacement method is used to obtain the second reference frame list. The third identifier is used to indicate a virtual reference frame or a fused reference frame. Both are carried in the bitstream as syntax elements, and the decoder can parse the bitstream to obtain them.

[0327] a. When the second identifier is the fifth value and the third identifier is the sixth value, method (1) is used to obtain the second reference frame list; or,

[0328] b. When the second identifier is the fifth value and the third identifier is the seventh value, method (2) is used to obtain the second reference frame list; or,

[0329] c. When the second identifier is the eighth value and the third identifier is the sixth value, the second reference frame list is obtained by using method (3); or,

[0330] d. When the second identifier is the eighth value and the third identifier is the seventh value, method (4) is used to obtain the second reference frame list.

[0331] In the method, the second identifier is reused, and when the second identifier is the fourteenth value, it indicates that the second reference frame list is obtained without using a virtual reference frame.

[0332] The present application may add two new syntax elements (a second identifier and a third identifier), and by assigning different values ​​to the two syntax elements, the decoder may read the syntax elements to determine which method to use to obtain the second reference frame list. At this time, the second identifier may have three valid values, for example, 0 (the fourteenth value, indicating that the virtual reference frame is not used), 1 (the fifth value, indicating insertion), and 2 (the eighth value, indicating replacement), and the third identifier may have two valid values, for example, 0 (the sixth value, indicating a virtual reference frame) and 1 (the seventh value, indicating a fused reference frame).

[0333] 3. Three logos

[0334] The three identifiers can be respectively called the fourth identifier, the fifth identifier and the sixth identifier. The fourth identifier is used to indicate whether a virtual reference frame is used to obtain the second reference frame list. The fifth identifier is used to indicate whether the second reference frame list is obtained by insertion or replacement. The sixth identifier is used to indicate a virtual reference frame or a fused reference frame. The three are carried in the bitstream as syntax elements, and the decoder can parse the bitstream to obtain them.

[0335] a. When the fourth identifier is a ninth value, the fifth identifier is a tenth value, and the sixth identifier is an eleventh value, method (1) is used to obtain the second reference frame list; or,

[0336] b. When the fourth identifier is the ninth value, the fifth identifier is the tenth value, and the sixth identifier is the twelfth value, adopt method (2) to obtain the second reference frame list; or,

[0337] c. When the fourth identifier is the ninth value, the fifth identifier is the thirteenth value, and the sixth identifier is the eleventh value, adopt method (3) to obtain the second reference frame list; or,

[0338] d. When the fourth identifier is the ninth value, the fifth identifier is the thirteenth value, and the sixth identifier is the twelfth value, method (4) is used to obtain the second reference frame list.

[0339] In this method, the fourth identifier is reused, and when the fourth identifier is the fifteenth value, it indicates that the second reference frame list is obtained without using a virtual reference frame.

[0340] The present application may add three new syntax elements (a fourth identifier, a fifth identifier, and a sixth identifier), and by assigning different values ​​to the three syntax elements, the decoder may read the syntax elements to determine which method to use to obtain the second reference frame list. At this time, the fourth identifier may have two valid values, for example, 0 (the fifteenth value, indicating that the virtual reference frame is not used), 1 (the ninth value, indicating that the virtual reference frame is used), the fifth identifier may have two valid values, for example, 0 (the tenth value, indicating insertion), 1 (the thirteenth value, indicating replacement), and the sixth identifier may have two valid values, for example, 0 (the eleventh value, indicating a virtual reference frame), 1 (the twelfth value, indicating a fused reference frame).

[0341] It should be noted that, in addition to the above three methods, the present application may also adopt other methods. For example, the encoding end and the decoding end may pre-set which method to use to obtain the second reference frame list based on a preset rule, and both parties may perform corresponding steps based on the setting. Therefore, the present application embodiment does not specifically limit how to determine which of the above methods (1)-(4) is used to obtain the second reference frame list.

[0342] In the present application, one or more virtual reference frames can be obtained corresponding to the current frame, so the above process of obtaining the second reference frame list can be performed once for each virtual reference frame until all virtual reference frames of the current frame are incorporated into the reference frame list. In addition, according to the inter-frame prediction characteristics of the current frame, the virtual reference frame can also include forward and backward two types. Correspondingly, the first reference frame list and the second reference frame list include a forward reference frame list and a backward reference frame list, respectively. According to the forward and backward classifications, the virtual reference frames are incorporated into the corresponding reference frame lists.

[0343] Optionally, the newly added syntax elements may be stored in the following levels or structures, including but not limited to: Sequence Parameter Set (SPS), Video Parameter Set (VPS), Picture Parameter Set (PPS), Picture Header (PH), Slice Header (SH). This application does not make specific limitations on this.

[0344] Step 505: Obtain a predicted image corresponding to the current frame according to the second reference frame list.

[0345] In one possible implementation, when the decoder determines that the reference frame is a virtual reference frame based on the bitstream description information parsed from the bitstream, the corresponding virtual reference frame can be selected from the second reference frame list as the reference frame; then, the predicted image can be determined based on the virtual reference frame.

[0346] In one possible implementation, when the decoder determines that the reference frame is the original reference frame based on the bitstream description information parsed from the bitstream, the corresponding original reference frame can be selected as the reference frame from the second reference frame list; then, the predicted image can be determined based on the original reference frame.

[0347] Due to the movement of the camera and the object, the position of the same object in the current frame may be different from the position in the previous frame. The motion information generated by the rendering engine can more accurately describe the coordinate correspondence (geometric relationship) between the two frames. Therefore, the virtual reference frame obtained based on the aforementioned correspondence can more accurately predict the current frame than the original reference frame. Based on this, the factor of the virtual reference frame is introduced into the reference frame queue table, which can improve the inter-frame coding performance and further reduce the bit rate.

[0348] In a possible implementation manner, the second reference frame list may be restored to the first reference frame list.

[0349] Considering compatibility with various standards, after the reference frame list is changed to the second reference frame list, the decoder can also restore it to the first reference frame list. The restoration method may include the following two methods:

[0350] 1. When the second reference frame list is obtained by the insertion method of the above methods (1) and (2), the inserted reference frame can be deleted, and then the index of at least one reference frame in the second reference frame list is reduced by 1, and the total number of reference frames in the second reference frame list is reduced by 1, and the index of the aforementioned at least one reference frame is greater than or equal to the index of the aforementioned inserted reference frame. For example, the inserted reference frame is located at I of the second reference frame list. n Position, then the second reference list is originally located at I n+1 The index of the reference frame at the position and the index of the reference frame after it are respectively reduced by 1, for example, I n+1 ,I n+2 ,I n+3 , ..., after decrementing 1, the index becomes I n ,I n+1 ,I n+2 , ... In addition, the total number of reference frames in the second reference frame list is reduced by 1, thereby restoring the first reference frame list. In a possible implementation, when the deleted reference frame is the last reference frame in the second reference frame list, since it has no effect on the positions of other reference frames in the second reference frame list, in this case, the index of any other reference frame is not changed, and only the total number of reference frames in the second reference frame list is reduced by 1, thereby restoring the first reference frame list.

[0351] 2. When the second reference frame list is obtained by using the replacement method of the above methods (3) and (4), the replaced reference frame can be changed back to the original reference frame, which has been recorded before and can be directly read from the corresponding storage space. The index of the original reference frame in the second reference frame list remains unchanged, and the number of reference frames remains unchanged.

[0352] Step 506: Reconstruct the predicted image and the residual image to obtain a reconstructed image of the current frame.

[0353] After the predicted image is obtained, the decoded residual image can be superimposed with the predicted image to obtain a reconstructed image. Then, the reconstructed image can be loop filtered to obtain a filtered reconstructed image, and the filtered reconstructed image can be output to a memory.

[0354] In this embodiment, by modifying the reference frame list and adding the factor of the virtual reference frame to the reference frame list, the time domain correlation between the virtual reference frame and the current frame can be fully utilized, the inter-frame coding performance can be improved, and the bit rate can be further reduced.

[0355] Several specific embodiments are used below. Figure 5 The technical solution of the method embodiment shown is described in detail.

[0356] Figure 6 A schematic diagram of the decoding process of the decoder of this application is shown in FIG. Figure 6 As shown above Figure 3a Based on the embodiment shown, a reference frame list management step S304' is added. This step can refer to Figure 5 In step 504 of the illustrated embodiment, before the decoder performs inter-frame prediction of the current frame, the decoder performs reference frame list management according to the parsed syntax elements or preset rules, and introduces the virtual reference frame into the reference frame list to facilitate subsequent inter-frame prediction. It should be understood that the reference frame list management of the present application can be applied to any video coding standard with reference frame list management.

[0357] This application extends the management rules of the reference frame list, adding a new syntax element 1 to indicate whether to use a virtual reference frame; a new syntax element 2 to indicate insertion or replacement; a new syntax element 3 to indicate whether to use a fused reference frame; a new syntax element 4 to indicate the position of insertion or replacement; and a new syntax element 5 to indicate the index of the original reference frame involved in the fusion. It should be noted that the new syntax elements added in this application may be more or less than the aforementioned syntax elements, and there is no specific limitation on this.

[0358] For example, Figure 7 This is a schematic diagram of reference frame list management for this application, such as Figure 7 As shown, the reference frame list management method may include:

[0359] 1. The decoder reads Flag1 and its decision process is as follows:

[0360] a) Flag1 = 0, end the process, compatible with the standard reference frame list management method;

[0361] b) Flag1 = 1, execute step 2.

[0362] 2. The decoder reads Flag2:

[0363] a) If it is an insertion, the virtual reference frame is inserted into the reference frame list I n Position, index in the reference frame list is greater than or equal to I n The index of the reference frame is increased by 1, and the number of reference frames in the reference frame list is increased by 1. After completing the inter-frame prediction, delete I n The virtual reference frame at the position, and the virtual reference frame settings associated with it, with an index greater than or equal to I in the reference frame list n The index of the reference frame is reduced by one, and the number of reference frames in the reference frame list is reduced by one;

[0364] b) If it is a replacement, the virtual reference frame replaces the I in the reference frame list n The original reference frame at the position is replaced, and the information of the original reference frame to be replaced is recorded. Other information in the reference frame list remains unchanged. After completing the inter-frame prediction, delete I n The virtual reference frame at the position, and the virtual reference frame settings associated with it, and insert the recorded original reference frame into I n Location;

[0365] c) If it is a fusion, decide whether to insert or replace after the fusion:

[0366] i. If it is inserted after fusion: add the virtual reference frame to the reference frame list indexed as R n The original reference frame is fused (the original reference frame is unchanged) to obtain a fused reference frame, and the fused reference frame is inserted into I in the reference frame list. n Position, index in the reference frame list is greater than or equal to I n The index of the reference frame is increased by 1, and the number of reference frames in the reference frame list is increased by 1. After completing the inter-frame prediction, delete I n The fused reference frame of the position, and the virtual reference frame settings associated with it, the index in the reference frame list is greater than or equal to I n The index of the reference frame is reduced by one, and the number of reference frames in the reference frame list is reduced by one;

[0367] ii. If it is a post-fusion replacement: index the virtual reference frame and the reference frame list as R n The original reference frame is fused (the original reference frame is unchanged) to obtain a fused reference frame, and the fused reference frame replaces I in the reference frame list n The original reference frame at the position is replaced, and the information of the original reference frame to be replaced is recorded. Other information in the reference frame list remains unchanged. After completing the inter-frame prediction, delete I n The fused reference frame of the position and the virtual reference frame setting related to it are inserted into the recorded original reference frame. n Location.

[0368] In a possible implementation, in the above steps 2-a) and 2-c)-i, when I n When the position is after the last reference frame in the reference frame list, since it has no effect on the positions of all reference frames in the reference frame list, in this case, the index of any other reference frame is not changed, and only the total number of reference frames in the reference frame list is increased by 1. n Position) is at the end of the reference frame list and has no effect on the positions of other reference frames in the reference frame list. Therefore, in this case, the index of any other reference frame is not changed, and the total number of reference frames in the reference frame list is reduced by 1.

[0369] This embodiment generates a virtual reference frame based on the rendering engine and the reconstructed prior information. The virtual reference frame has a stronger temporal correlation with the current frame, which can reduce the motion vector amplitude and improve the prediction accuracy, thereby improving the performance of inter-frame prediction. Compared with the original reference frame, the virtual reference frame can effectively increase the skip mode ratio in inter-frame prediction, thereby reducing the bit rate and improving the decoding performance. For example, Figure 8 is the statistical result of the inter-frame coding mode, such as Figure 8 As shown, the left side shows the statistical results with the virtual reference frame added, and the right side shows the statistical results without the virtual reference frame.

[0370] In one possible implementation, Figure 7 Based on the embodiment shown, the syntax elements can be deleted, and the encoder and decoder manage the reference frame list according to the preset principle. For example, both parties preset the virtual reference frame insertion / virtual frame replacement / fused reference frame insertion / fused reference frame replacement method to manage the reference frame list. In this case, no syntax elements need to be added, thereby further saving bit stream and reducing bit rate.

[0371] In one possible implementation, Figure 7 Based on the illustrated embodiment, the syntax elements may be refined to manage the reference frame list in a more specific implementation manner.

[0372] For example, Fig. 9 This is a schematic diagram of reference frame list management for this application, such as Fig. 9 As shown, the reference frame list management method may include:

[0373] 1. The decoder reads the syntax element Flag1:

[0374] 2. Execution based on the value of Flag1:

[0375] a) When Flag1 = 0, refer to Figure 7Step 1-a) of the illustrated embodiment is compatible with the standard reference frame list management method;

[0376] b) When Flag1 = 1, refer to Figure 7 Step 2-a) of the illustrated embodiment, virtual reference frame insertion;

[0377] c) When Flag1=3, refer to Figure 7 Step 2-b) of the illustrated embodiment, virtual reference frame replacement;

[0378] d) When Flag1 = 2 or 4, refer to Figure 7 In step 2-c) of the illustrated embodiment, the fused reference frame is inserted or replaced.

[0379] For example, Fig.10 This is a schematic diagram of reference frame list management for this application, such as Fig.10 As shown, the reference frame list management method may include:

[0380] 1. The decoder reads the syntax element Flag1;

[0381] 2. When Flag1=0, refer to Figure 7 Step 1-a) of the illustrated embodiment is compatible with the standard reference frame list management method;

[0382] 3. When Flag1=1, read Flag2:

[0383] a) When Flag2 = 0, refer to Figure 7 Step 2-a) of the illustrated embodiment, virtual reference frame insertion;

[0384] b) When Flag2 = 1, refer to Figure 7 Step 2-c)-i of the illustrated embodiment, fusion reference frame insertion;

[0385] 4. When Flag1=2, read Flag2:

[0386] a) When Flag2 = 0, refer to Figure 7 Step 2-b) of the illustrated embodiment, virtual reference frame replacement;

[0387] b) When Flag2 = 1, refer to Figure 7 Step 2-c)-ii of the illustrated embodiment is replacement of the fusion reference frame.

[0388] For example, Fig.11 This is a schematic diagram of reference frame list management for this application, such as Fig.11 As shown, the reference frame list management method may include:

[0389] 1. The decoder reads the syntax element Flag1;

[0390] 2. When Flag1=0, refer to Figure 7 Step 1-a) of the illustrated embodiment is compatible with the standard reference frame list management method;

[0391] 3. When Flag1=1, read Flag2:

[0392] a) When Flag2=0, read Flag3:

[0393] i. When Flag3 = 0, refer to Figure 7 Step 2-a) of the illustrated embodiment, virtual reference frame insertion;

[0394] ii. When Flag3 = 1, refer to Figure 7 Step 2-c)-i of the illustrated embodiment, fusion reference frame insertion;

[0395] b) When Flag2=1, read Flag3:

[0396] i. When Flag3 = 0, refer to Figure 7 Step 2-b) of the illustrated embodiment, virtual reference frame replacement;

[0397] ii. When Flag3 = 1, refer to Figure 7 Step 2-c)-ii of the illustrated embodiment is replacement of the fusion reference frame.

[0398] Fig.12 is a schematic diagram of the structure of the image decoding device 1200 of the present application, as shown in Fig.12 As shown, the image decoding device 1200 of this embodiment can be applied to the above terminal device. The image decoding device 1200 may include: a receiving module 1201, a prediction module 1202 and a reconstruction module 1203.

[0399] A receiving module 1201 is used to receive a code stream; a prediction module 1202 is used to obtain a residual image and a first reference frame list corresponding to a current frame according to the code stream; obtain a virtual reference frame corresponding to the current frame; obtain a second reference frame list corresponding to the current frame according to the virtual reference frame and the first reference frame list; obtain a predicted image corresponding to the current frame based on the second reference frame list; a reconstruction module 1203 is used to reconstruct according to the predicted image and the residual image to obtain a reconstructed image of the current frame.

[0400] In a possible implementation, the prediction module 1202 is specifically configured to implement any one of the following methods:

[0401] (1) inserting the virtual reference frame into the first position of the first reference frame list to obtain the second reference frame list;

[0402] (2) inserting the fused reference frame into the first position of the first reference frame list to obtain the second reference frame list;

[0403] (3) replacing the first reference frame in the first reference frame list with the virtual reference frame to obtain the second reference frame list;

[0404] (4) replacing the first reference frame in the first reference frame list with the fused reference frame to obtain the second reference frame list;

[0405] Among them, the first reference frame is the reference frame located at the first position in the first reference frame list; the fused reference frame is obtained based on the virtual reference frame and the second reference frame in the first reference frame list, and the second reference frame is the reference frame located at the second position in the first reference frame list.

[0406] In a possible implementation, the prediction module 1202 is specifically used to obtain a first identifier from the code stream, where the first identifier is used to indicate whether the virtual reference frame is used to obtain the second reference frame list, and when the virtual reference frame is used to obtain the second reference frame list, a method for obtaining the second reference frame list is used; when the first identifier is a first value, method (1) is used to obtain the second reference frame list; or, when the first identifier is a second value, method (2) is used to obtain the second reference frame list; or, when the first identifier is a third value, method (3) is used to obtain the second reference frame list; or, when the first identifier is a fourth value, method (4) is used to obtain the second reference frame list.

[0407] In a possible implementation, the prediction module 1202 is specifically used to obtain a second identifier and a third identifier from the code stream, the second identifier is used to indicate whether to use the virtual reference frame to obtain the second reference frame list, and when the virtual reference frame is used to obtain the second reference frame list, the second reference frame list is obtained by insertion or replacement, and the third identifier is used to indicate the virtual reference frame or the fused reference frame; when the second identifier is a fifth value and the third identifier is a sixth value, the second reference frame list is obtained by method (1); or, when the second identifier is a fifth value and the third identifier is a seventh value, the second reference frame list is obtained by method (2); or, when the second identifier is an eighth value and the third identifier is a sixth value, the second reference frame list is obtained by method (3); or, when the second identifier is an eighth value and the third identifier is a sixth value, the second reference frame list is obtained by method (4).

[0408] In a possible implementation, the prediction module 1202 is specifically used to obtain a fourth identifier, a fifth identifier and a sixth identifier from the code stream, the fourth identifier is used to indicate whether to use the virtual reference frame to obtain the second reference frame list, the fifth identifier is used to indicate whether to use the insertion or replacement method to obtain the second reference frame list, and the sixth identifier is used to indicate the virtual reference frame or the fused reference frame; when the fourth identifier is a ninth value, the fifth identifier is a tenth value, and the sixth identifier is an eleventh value, the second reference frame list is obtained by method (1); or, when the fourth identifier is a ninth value, the fifth identifier is a tenth value, and the sixth identifier is a twelfth value, the second reference frame list is obtained by method (2); or, when the fourth identifier is a ninth value, the fifth identifier is a thirteenth value, and the sixth identifier is an eleventh value, the second reference frame list is obtained by method (3); or, when the fourth identifier is a ninth value, the fifth identifier is a thirteenth value, and the sixth identifier is an eleventh value, the second reference frame list is obtained by method (4).

[0409] In a possible implementation manner, when the first identifier is a thirteenth value, it indicates that the second reference frame list is acquired without using the virtual reference frame.

[0410] In a possible implementation manner, when the second identifier is the fourteenth value, it indicates that the second reference frame list is acquired without using the virtual reference frame.

[0411] In a possible implementation manner, when the fourth identifier is the fifteenth value, it indicates that the second reference frame list is acquired without using the virtual reference frame.

[0412] In one possible implementation, the prediction module is further used to, when method (1) or method (2) is adopted, add 1 to the index of at least one reference frame in the first reference frame list, and add 1 to the total number of reference frames in the first reference frame list, and the index of the at least one reference frame is greater than or equal to the index of the first reference frame.

[0413] In a possible implementation, the prediction module 1202 is further configured to record relevant information of the first reference frame when method (3) or method (4) is adopted.

[0414] In a possible implementation manner, the prediction module 1202 is further configured to restore the second reference frame list to the first reference frame list.

[0415] In a possible implementation, the virtual reference frame is stored in a storage space of a non-decoded picture buffer DPB; or, the virtual reference frame is stored in a designated space in the DPB.

[0416] In one possible implementation, the identifier obtained from the bitstream is stored in a sequence parameter set SPS, a video parameter set VPS, a picture parameter set PPS, a picture header PH or a slice header SH, and the identifier includes a first identifier, a second identifier, a third identifier, a fourth identifier, a fifth identifier and a sixth identifier.

[0417] In a possible implementation, the prediction module 1202 is specifically configured to obtain rendering information from the bitstream; and obtain the virtual reference frame according to the decoded reconstructed image and the rendering information, wherein the rendering information comes from a server.

[0418] In a possible implementation, the apparatus is applied to a terminal-cloud collaborative system, the terminal-cloud collaborative system includes a server and a terminal device, the server includes an encoder, and the terminal device includes a decoder.

[0419] The device of this embodiment can be used to perform Figure 5 The technical solution of the method embodiment shown has similar implementation principles and technical effects, which will not be repeated here.

[0420] For example, Fig.13 A schematic block diagram of a device 1300 of the present application is shown. The device 1300 may include: a processor 1301 and a transceiver / transceiver pin 1302 , and optionally, a memory 1303 .

[0421] The components of the device 1300 are coupled together via a bus 1304 , wherein the bus 1304 includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, all buses are referred to as bus 1304 in the figure.

[0422] Optionally, the memory 1303 may be used for instructions in the above method embodiment. The processor 1301 may be used to execute instructions in the memory 1303, and control the receiving pin to receive a signal, and control the sending pin to send a signal.

[0423] The apparatus 1300 may be a terminal device or a chip of the terminal device in the above method embodiment.

[0424] In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The processor can be a general-purpose processor, a digital signal processor (digital signal processor, DSP), an application-specific integrated circuit (application-specific integrated circuit, ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware coding processor to be executed, or the hardware and software modules in the coding processor are combined to be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0425] The memory mentioned in the above embodiments may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0426] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0427] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0428] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0429] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0430] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0431] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0432] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image decoding method, characterized in that: include: Receive code stream; Acquire a residual image and a first reference frame list corresponding to the current frame according to the bitstream; Acquire a virtual reference frame corresponding to the current frame; Acquire a second reference frame list corresponding to the current frame according to the virtual reference frame and the first reference frame list; Acquire a predicted image corresponding to the current frame based on the second reference frame list; Reconstruction is performed according to the predicted image and the residual image to obtain a reconstructed image of the current frame.

2. The method according to claim 1, characterized in that The acquiring the second reference frame list corresponding to the current frame according to the virtual reference frame and the first reference frame list includes any one of the following methods: (1) inserting the virtual reference frame into the first position of the first reference frame list to obtain the second reference frame list; (2) inserting the fused reference frame into the first position of the first reference frame list to obtain the second reference frame list; (3) replacing the first reference frame in the first reference frame list with the virtual reference frame to obtain the second reference frame list; (4) replacing the first reference frame in the first reference frame list with the fused reference frame to obtain the second reference frame list; Wherein, the first reference frame is a reference frame located at the first position in the first reference frame list; The fused reference frame is obtained based on the virtual reference frame and a second reference frame in the first reference frame list, where the second reference frame is a reference frame located at a second position in the first reference frame list.

3. The method according to claim 2, characterized in that The acquiring a second reference frame list corresponding to the current frame according to the virtual reference frame and the first reference frame list includes: Acquire a first identifier from the bitstream, where the first identifier is used to indicate whether to use the virtual reference frame to acquire the second reference frame list, and a method for acquiring the second reference frame list when the virtual reference frame is used to acquire the second reference frame list; When the first identifier is a first value, the second reference frame list is acquired by using method (1); or, When the first identifier is the second value, method (2) is used to obtain the second reference frame list; or, When the first identifier is the third value, method (3) is used to obtain the second reference frame list; or, When the first identifier is the fourth value, method (4) is used to obtain the second reference frame list.

4. The method according to claim 2, characterized in that: The acquiring a second reference frame list corresponding to the current frame according to the virtual reference frame and the first reference frame list includes: Acquire a second identifier and a third identifier from the bitstream, the second identifier being used to indicate whether to use the virtual reference frame to acquire the second reference frame list, and when the virtual reference frame is used to acquire the second reference frame list, the second reference frame list is acquired by an insertion or replacement method, and the third identifier is used to indicate the virtual reference frame or the fused reference frame; When the second identifier is the fifth value and the third identifier is the sixth value, the second reference frame list is acquired by using method (1); or, When the second identifier is the fifth value and the third identifier is the seventh value, the second reference frame list is acquired by using method (2); or, When the second identifier is the eighth value and the third identifier is the sixth value, the second reference frame list is acquired by using method (3); or, When the second identifier is the eighth value and the third identifier is the seventh value, method (4) is used to obtain the second reference frame list.

5. The method according to claim 2, characterized in that: The acquiring a second reference frame list corresponding to the current frame according to the virtual reference frame and the first reference frame list includes: Acquire a fourth identifier, a fifth identifier, and a sixth identifier from the bitstream, wherein the fourth identifier is used to indicate whether to use the virtual reference frame to acquire the second reference frame list, the fifth identifier is used to indicate that the second reference frame list is acquired by an insertion or replacement method, and the sixth identifier is used to indicate the virtual reference frame or the fused reference frame; When the fourth identifier is a ninth value, the fifth identifier is a tenth value, and the sixth identifier is an eleventh value, the method (1) is used to obtain the second reference frame list; or, When the fourth identifier is a ninth value, the fifth identifier is a tenth value, and the sixth identifier is a twelfth value, method (2) is used to obtain the second reference frame list; or, When the fourth identifier is a ninth value, the fifth identifier is a thirteenth value, and the sixth identifier is an eleventh value, the method (3) is used to obtain the second reference frame list; or, When the fourth identifier is a ninth value, the fifth identifier is a thirteenth value, and the sixth identifier is a twelfth value, method (4) is used to obtain the second reference frame list.

6. The method according to claim 3, characterized in that When the first identifier is a thirteenth value, it indicates that the second reference frame list is acquired without using the virtual reference frame.

7. The method according to claim 4, characterized in that When the second identifier is the fourteenth value, it indicates that the second reference frame list is acquired without using the virtual reference frame.

8. The method according to claim 5, characterized in that When the fourth identifier is the fifteenth value, it indicates that the second reference frame list is acquired without using the virtual reference frame.

9. The method according to any one of claims 2 to 8, characterized in that: Also includes: When method (1) or method (2) is adopted, the index of at least one reference frame in the first reference frame list is increased by 1, and the total number of reference frames in the first reference frame list is increased by 1, and the index of the at least one reference frame is greater than or equal to the index of the first reference frame.

10. The method according to any one of claims 2 to 8, characterized in that: Also includes: When method (3) or method (4) is adopted, relevant information of the first reference frame is recorded.

11. The method according to any one of claims 1 to 10, characterized in that After acquiring the predicted image corresponding to the current frame from the second reference frame list, the method further includes: The second reference frame list is restored to the first reference frame list.

12. The method according to any one of claims 1 to 11, characterized in that The virtual reference frame is stored in a storage space of a non-decoded picture buffer DPB; or, the virtual reference frame is stored in a designated space in the DPB.

13. The method according to any one of claims 3 to 5, characterized in that: The identifier obtained from the bitstream is stored in a sequence parameter set SPS, a video parameter set VPS, a picture parameter set PPS, a picture header PH or a slice header SH, and the identifier includes a first identifier, a second identifier, a third identifier, a fourth identifier, a fifth identifier and a sixth identifier.

14. The method according to any one of claims 1 to 13, characterized in that The acquiring the virtual reference frame corresponding to the current frame includes: Obtaining rendering information from the code stream; The virtual reference frame is acquired according to the decoded reconstructed image and the rendering information, where the rendering information comes from a server.

15. The method according to any one of claims 1 to 14, characterized in that The method is applied to an end-cloud collaborative system, which includes a server and a terminal device, wherein the server includes an encoder and the terminal device includes a decoder.

16. A code stream generation method, characterized in that: Based on the virtual reference frame described in any one of claims 1 to 15, a code stream is generated.

17. The method according to claim 16, characterized in that The code stream includes a first identifier, and the first identifier is used to indicate whether to use the virtual reference frame to obtain the second reference frame list, and when the virtual reference frame is used to obtain the second reference frame list, a method used to obtain the second reference frame list.

18. The method according to claim 16, characterized in that The code stream includes a second identifier and a third identifier, the second identifier is used to indicate whether to use the virtual reference frame to obtain the second reference frame list, and when the virtual reference frame is used to obtain the second reference frame list, the second reference frame list is obtained by insertion or replacement, and the third identifier is used to indicate the virtual reference frame or the fused reference frame.

19. The method according to claim 16, characterized in that The code stream includes a fourth identifier, a fifth identifier and a sixth identifier, the fourth identifier is used to indicate whether the virtual reference frame is used to obtain the second reference frame list, the fifth identifier is used to indicate whether the second reference frame list is obtained by insertion or replacement, and the sixth identifier is used to indicate the virtual reference frame or the fused reference frame.

20. A terminal device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the terminal device executes the method according to any one of claims 1 to 15.

21. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of a terminal device and send the signal to the processor, the signal comprising a computer instruction stored in the memory; when the processor executes the computer instruction, the terminal device executes the method as described in any one of claims 1-15.

22. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the method according to any one of claims 1 to 15.

23. A computer program product, characterized in that The computer program product comprises computing instructions, which, when executed by a computer or a processor, cause the steps of the method according to any one of claims 1 to 15 to be performed.

24. A device for storing a code stream, characterized in that: The device comprises: a receiver and at least one storage medium, The receiver is used to receive a code stream; The at least one storage medium is used to store the code stream; The code stream is generated according to the method according to any one of claims 16-19.

25. A device for transmitting a code stream, characterized in that: The device comprises: a transmitter and at least one storage medium, The at least one storage medium is used to store a code stream, wherein the code stream is generated according to the method according to any one of claims 16 to 19; The transmitter is used to obtain the code stream from the storage medium and send the code stream to the terminal side device through the transmission medium.

26. A system for distributing code streams, characterized in that: The system comprises: At least one storage medium, used to store at least one code stream, wherein the at least one code stream is generated according to the method according to any one of claims 16 to 19; The streaming media device is used to obtain a target code stream from the at least one storage medium and send the target code stream to the terminal side device, wherein the streaming media device includes a content server or a content distribution server.

Images