Image decoding method, encoding method, electronic equipment and readable storage medium
By combining motion stream decoding with multi-reference frame information, the problem of insufficient stream compression rate in existing technologies is solved, achieving higher compression rate and more accurate prediction information.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG DAHUA TECH CO LTD
- Filing Date
- 2024-11-15
- Publication Date
- 2026-05-26
AI Technical Summary
In existing technologies, the prediction information in the video image encoding and decoding process is not accurate enough, resulting in insufficient compression rate of the bitstream.
By decoding the motion stream, the decoded motion information of the current frame relative to the preset reference frame is obtained. Combined with the reconstruction information and estimated motion information of multiple target reference frames, the initial prediction information is corrected to obtain more accurate target prediction information, thereby improving the compression rate of the stream.
It improves the compression rate of the bitstream, reduces redundant information, and enhances the accuracy of prediction information.
Smart Images
Figure CN122093558A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image encoding and decoding technology, and in particular to an image decoding method, encoding method, electronic device, and readable storage medium. Background Technology
[0002] Video image data is relatively large, and it is usually necessary to compress the video pixel data. The compressed data is called a bitstream. During the encoding and decoding process, it is usually necessary to predict video frames to reduce redundant data between video frames before encoding to obtain the bitstream, and then decode the bitstream to obtain reconstructed information. However, in current technologies, the prediction information in the encoding and decoding process is still not accurate enough, resulting in insufficient bitstream compression ratio. In view of this, how to improve the bitstream compression ratio has become an urgent problem to be solved. Summary of the Invention
[0003] The main technical problem addressed by this application is to provide an image decoding method, encoding method, electronic device, and readable storage medium that can improve the compression rate of the bitstream.
[0004] To address the aforementioned technical problems, this application provides an image decoding method, comprising: decoding a motion bitstream to obtain decoded motion information of a current frame relative to a preset reference frame; obtaining initial prediction information corresponding to the current frame based on the decoded motion information and information transmitted by the preset reference frame; wherein the motion bitstream is obtained by encoding motion information, and the motion information is obtained by motion estimation based on the current frame and its corresponding preset reference frame; acquiring a target reference frame corresponding to the current frame, and determining reference information corresponding to the target reference frame; wherein the number of target reference frames is at least one, and the reference information includes at least reconstruction information of the target reference frame; obtaining target prediction information corresponding to the current frame based on the initial prediction information and the reference information corresponding to the target reference frame; wherein the target prediction information is used to decode and obtain reconstruction information corresponding to the current frame.
[0005] To address the aforementioned technical problems, a second aspect of this application provides an image encoding method, comprising: acquiring a current frame and its corresponding preset reference frame; performing motion estimation on the current frame and the preset reference frame to obtain motion information of the current frame relative to the preset reference frame; encoding and decoding the motion information to obtain decoded motion information of the current frame relative to the preset reference frame; obtaining initial prediction information corresponding to the current frame based on the decoded motion information and information transmitted by the preset reference frame; acquiring a target reference frame corresponding to the current frame; determining reference information corresponding to the target reference frame; wherein the number of target reference frames is at least one, and the reference information includes at least reconstruction information of the target reference frames; obtaining target prediction information corresponding to the current frame based on the initial prediction information and the reference information corresponding to the target reference frames; wherein the motion information and the target prediction information corresponding to the current frame are used to encode a current frame bitstream of the current frame.
[0006] To address the aforementioned technical problems, a third aspect of this application provides an electronic device comprising: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor invokes the program data to execute the method described in the first or second aspect.
[0007] To address the aforementioned technical problems, a fourth aspect of this application provides a computer-readable storage medium storing program data thereon, wherein the program data, when executed by a processor, implements the method described in the first or second aspect.
[0008] The above scheme decodes the motion stream to obtain decoded motion information of the current frame relative to a preset reference frame. Based on the decoded motion information and the information transmitted by the preset reference frame, the current frame is predicted to obtain initial prediction information corresponding to the current frame. The motion stream is obtained by encoding motion information at the encoding end, and the motion information is obtained based on motion estimation between the current frame and the preset reference frame. At least one target reference frame corresponding to the current frame is acquired, and reference information corresponding to the target reference frame is determined, thereby obtaining more reference frames to correct the initial prediction information. The reference information includes at least the reconstructed information of the target reference frame. Based on the initial prediction information and the reference information corresponding to the target reference frame, the prediction information of the current frame is corrected to obtain the target prediction information corresponding to the current frame. By combining multiple reference frames to determine the target prediction information corresponding to the current frame, the accuracy of the prediction information is improved, and redundant information is reduced. The target prediction information is used to decode the reconstructed information corresponding to the current frame, thereby effectively improving the compression rate of the bitstream. Attached Figure Description
[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:
[0010] Figure 1 This is a flowchart illustrating one embodiment of the image decoding method of this application;
[0011] Figure 2 This is a schematic diagram of the framework of one implementation method of the encoding and decoding process of this application;
[0012] Figure 3 This is a schematic diagram of the topology of one embodiment of the image decoding method of this application;
[0013] Figure 4 This is a schematic diagram of an application scenario corresponding to one embodiment of the image decoding method of this application;
[0014] Figure 5 This is a flowchart illustrating another embodiment of the image decoding method of this application;
[0015] Figure 6 This is a schematic diagram of the topology of another embodiment of the image decoding method of this application;
[0016] Figure 7 This is a schematic diagram of an application scenario corresponding to another embodiment of the image decoding method of this application;
[0017] Figure 8 This is a schematic diagram of the topology of another embodiment of the image decoding method of this application;
[0018] Figure 9 This is a schematic diagram illustrating an application scenario of one embodiment of this application for obtaining estimated reference information corresponding to all target reference frames;
[0019] Figure 10 This is a schematic diagram illustrating an application scenario of one embodiment of this application for obtaining estimated motion information corresponding to a single target reference frame;
[0020] Figure 11 This is a flowchart illustrating one embodiment of the image encoding method of this application;
[0021] Figure 12 This is a schematic diagram of one embodiment of the image decoding end of this application;
[0022] Figure 13 This is a schematic diagram of the structure of one embodiment of the image encoding end of this application;
[0023] Figure 14This is a schematic diagram of the structure of one embodiment of the electronic device of this application;
[0024] Figure 15 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. Detailed Implementation
[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments, and different implementation methods can be adaptively combined. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0026] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper means two or more.
[0027] The image decoding method and image encoding method provided in this application are used to encode and decode video frames, and the corresponding execution subject is a terminal capable of processing video image data.
[0028] Please see Figure 1 , Figure 1 This is a flowchart illustrating one embodiment of the image decoding method of this application, which includes:
[0029] S101: Decode the motion bitstream to obtain the decoded motion information of the current frame relative to the preset reference frame. Based on the decoded motion information and the information transmitted by the preset reference frame, obtain the initial prediction information corresponding to the current frame. The motion bitstream is obtained by encoding motion information, and the motion information is obtained by motion estimation based on the current frame and its corresponding preset reference frame.
[0030] Specifically, the motion stream is decoded to obtain the decoded motion information of the current frame relative to the preset reference frame. Based on the decoded motion information and the information transmitted by the preset reference frame, the current frame is predicted to obtain the initial prediction information corresponding to the current frame. The motion stream is obtained by encoding the motion information at the encoding end, and the motion information is obtained by motion estimation between the current frame and the preset reference frame.
[0031] It should be noted that the encoding and decoding process specifically involves an encoding end and a decoding end; please refer to [link / reference needed]. Figure 2 , Figure 2 This is a schematic diagram illustrating the framework of one embodiment of the encoding and decoding process of this application. The encoding end encodes the current frame to obtain the current frame bitstream, and the decoding end obtains the current frame bitstream and decodes it to obtain the reconstructed information corresponding to the current frame. The current frame bitstream includes at least a motion bitstream and a residual bitstream, which can be used to decode and obtain the reconstructed information corresponding to the current frame, i.e., to decode the image. The image decoding method and image encoding method provided in this application are based on... Figure 2 The framework shown operates, and in this application for Figure 2 The key adjustments to the frame shown correspond to the positions circled by the elliptical dashed lines.
[0032] In some implementation scenarios, there is only one preset reference frame. The encoder performs motion estimation on the current frame and the preset reference frame to obtain motion information, and then encodes the motion information to obtain a motion bitstream. The decoder decodes the motion bitstream to obtain the decoded motion information of the current frame relative to the preset reference frame. Based on the decoded motion information, motion compensation is performed on the information transmitted by the preset reference frame to obtain the initial prediction information corresponding to the current frame.
[0033] In some implementation scenarios, there are multiple preset reference frames. The encoding end fuses these multiple preset reference frames to obtain a fused preset reference frame. Motion estimation is performed on the current frame and the fused preset reference frame to obtain motion information, which is then encoded to obtain a motion stream. The decoding end decodes the motion stream to obtain the decoded motion information of the current frame relative to the preset reference frames. Multiple preset reference frames are then fused to obtain a fused preset reference frame. Based on the decoded motion information, motion compensation is performed on the information transmitted in the fused preset reference frame to obtain the initial prediction information corresponding to the current frame.
[0034] It is understandable that the preset reference frame is usually a video frame that is located before the current frame in the time domain, that is, a video frame that is decoded before the current frame. When there is only one preset reference frame, it is usually a video frame that is adjacent to the current frame. The information conveyed by the preset reference frame includes at least one of the reference frame features and image information.
[0035] S102: Obtain the target reference frame corresponding to the current frame and determine the reference information corresponding to the target reference frame; wherein, the number of target reference frames is at least one, and the reference information includes at least the reconstruction information of the target reference frame.
[0036] Specifically, at least one target reference frame corresponding to the current frame is obtained, and the reference information corresponding to the target reference frame is determined, thereby obtaining more reference frames to correct the initial prediction information. The reference information includes at least the reconstruction information of the target reference frame.
[0037] In some implementation scenarios, the target reference frame corresponding to the current frame is obtained, and the reconstruction information corresponding to the target reference frame is used as the reference information corresponding to the target reference frame.
[0038] In some implementation scenarios, the target reference frame corresponding to the current frame is obtained, the reconstruction information of the target reference frame is obtained, and the estimated motion information between the current frame and the target reference frame is determined. The reconstruction information and estimated motion information corresponding to the target reference frame are used as the reference information corresponding to the target reference frame.
[0039] It is understandable that there can be one or more target reference frames. Target reference frames are usually different from preset reference frames. In some implementation scenarios, preset reference frames may also be included among multiple target reference frames.
[0040] S103: Based on the initial prediction information and the reference information corresponding to the target reference frame, obtain the target prediction information corresponding to the current frame; wherein, the target prediction information is used to decode and obtain the reconstruction information corresponding to the current frame.
[0041] Specifically, based on the initial prediction information and the reference information corresponding to the target reference frame, the prediction information of the current frame is corrected to obtain the target prediction information corresponding to the current frame.
[0042] It is understandable that by combining multiple reference frames to determine the target prediction information corresponding to the current frame, the accuracy of the prediction information is improved to reduce redundant information. The target prediction information is then used to decode and obtain the reconstruction information corresponding to the current frame, thereby effectively improving the compression rate of the bitstream.
[0043] In some implementation scenarios, the reference information corresponds to the reconstruction information of the target reference frame. The initial prediction information is fused with the reconstruction information of the target reference frame to obtain the target prediction information corresponding to the current frame.
[0044] In some implementation scenarios, the reference information corresponds to the reconstruction information and estimated motion information of the target reference frame. The initial prediction information is fused with the reconstruction information and estimated motion information of the target reference frame to obtain the target prediction information corresponding to the current frame.
[0045] It is understandable that when the reference information includes the estimated motion information of the target reference frame, the target prediction information corresponding to the current frame is obtained based on the initial prediction information and the reference information corresponding to the target reference frame. This includes: performing motion compensation on the reconstructed information of the target reference frame based on the estimated motion information corresponding to the target reference frame to obtain the reference prediction information corresponding to the target reference frame; and fusing the initial prediction information and the reference prediction information to obtain the target prediction information corresponding to the current frame.
[0046] Specifically, based on the estimated motion information corresponding to the target reference frame, motion compensation is performed on the reconstructed information of the target reference frame to obtain more accurate prediction information corresponding to the target reference frame as reference prediction information. The initial prediction information and the reference prediction information are fused together, and the initial prediction information is corrected by combining the prediction information of the target reference frame to obtain target prediction information with higher precision and accuracy corresponding to the current frame.
[0047] It should be noted that the target prediction information obtained in this application is... Figure 2 The result output by the multi-reference frame motion compensation and temporal prediction module is obtained by training multiple training video frames and their corresponding training information. The training information includes at least the reconstruction information of the training video frames, and may also include the motion information of the training video frames.
[0048] Understandably, with Figure 2 As shown in the encoding and decoding process, target prediction information can be used in the encoding and decoding process to be applicable to the end-to-end encoding and decoding process. Moreover, by using target prediction information with higher accuracy, the compression rate of the bitstream can be improved.
[0049] The above scheme decodes the motion stream to obtain decoded motion information of the current frame relative to a preset reference frame. Based on the decoded motion information and the information transmitted by the preset reference frame, the current frame is predicted to obtain initial prediction information corresponding to the current frame. The motion stream is obtained by encoding motion information at the encoding end, and the motion information is obtained based on motion estimation between the current frame and the preset reference frame. At least one target reference frame corresponding to the current frame is acquired, and reference information corresponding to the target reference frame is determined, thereby obtaining more reference frames to correct the initial prediction information. The reference information includes at least the reconstructed information of the target reference frame. Based on the initial prediction information and the reference information corresponding to the target reference frame, the prediction information of the current frame is corrected to obtain the target prediction information corresponding to the current frame. By combining multiple reference frames to determine the target prediction information corresponding to the current frame, the accuracy of the prediction information is improved, and redundant information is reduced. The target prediction information is used to decode the reconstructed information corresponding to the current frame, thereby effectively improving the compression rate of the bitstream.
[0050] In some embodiments, the reference information corresponds to the reconstruction information of the target reference frame. Based on the initial prediction information and the reference information corresponding to the target reference frame, the target prediction information corresponding to the current frame is obtained, including: fusing the initial prediction information with the reference information corresponding to the target reference frame to obtain the target prediction information corresponding to the current frame; or, fusing the initial prediction information with the information transmitted by the preset reference frame and the reference information corresponding to the target reference frame to obtain the target prediction information corresponding to the current frame.
[0051] Specifically, please refer to Figure 3 , Figure 3 This is a schematic diagram of the topology of an embodiment of the image decoding method of this application. At least the reconstruction information corresponding to the target reference frame is fused with the initial prediction information, so as to optimize the initial prediction information by at least using the reconstruction information corresponding to the target reference frame.
[0052] It is understandable that, at least based on the reconstruction information and initial prediction information corresponding to the target reference frame, the correction information corresponding to the initial prediction information is determined, and the target prediction information is obtained by fusing the initial prediction information and the correction information, thereby obtaining more accurate target prediction information efficiently with lower processing cost.
[0053] In some implementation scenarios, the initial prediction information is fused with the reference information corresponding to the target reference frame, and the initial prediction information is corrected using the reconstruction information corresponding to the target reference frame to obtain the target prediction information corresponding to the current frame.
[0054] In some implementation scenarios, the initial prediction information is fused with the information transmitted by the preset reference frame and the reference information corresponding to the target reference frame. The initial prediction information is then corrected by using the information corresponding to the preset reference frame and the target reference frame, and by combining the differences between the information transmitted by the preset reference frame and the reconstructed information of the target reference frame, so as to obtain the target prediction information corresponding to the current frame.
[0055] Optionally, the fusion method can be implemented in ways including but not limited to residual networks, channel / spatial attention, transformers, and neural networks formed by combinations of the above.
[0056] For clarity, please refer to Figure 4 , Figure 4 This is a schematic diagram of an application scenario corresponding to one embodiment of the image decoding method of this application. Taking a fusion network composed of two residual blocks as an example, the initial prediction information, the information transmitted by the preset reference frame, and the reconstruction information of the target reference frame are input into the fusion network to obtain the correction information. The correction information is superimposed and fused with the initial prediction information to obtain the target prediction information corresponding to the current frame.
[0057] Please see Figure 5 , Figure 5 This is a flowchart illustrating another embodiment of the image decoding method of this application, the method comprising:
[0058] S201: Decode the motion bitstream to obtain the decoded motion information of the current frame relative to the preset reference frame. Based on the decoded motion information and the information transmitted by the preset reference frame, obtain the initial prediction information corresponding to the current frame. The motion bitstream is obtained by encoding motion information, and the motion information is obtained by motion estimation based on the current frame and its corresponding preset reference frame.
[0059] Specifically, the motion stream is decoded to obtain the decoded motion information of the current frame relative to the preset reference frame. Based on the decoded motion information and the information transmitted by the preset reference frame, the current frame is predicted to obtain the initial prediction information corresponding to the current frame. The motion stream is obtained by encoding the motion information at the encoding end, and the motion information is obtained by motion estimation between the current frame and the preset reference frame.
[0060] S202: Obtain a preset number of target reference frames corresponding to the current frame, and determine the reconstruction information corresponding to each target reference frame.
[0061] Specifically, the number of target reference frames is set to a preset number, which can be adaptively selected. The preset number of target reference frames corresponding to the current frame is obtained, and the cached reconstruction information of each target reference frame is obtained.
[0062] It is understandable that the target reference frame is located before the current frame during the decoding process, so that the reconstruction information of the target reference frame can be obtained in the image buffer area. Specifically, after the current frame is decoded, its reconstruction information is buffered in the image buffer area. The buffer area stores the reconstruction information of at least a portion of the video frames in a first-in, first-out manner. There can be one or more preset reference frames. The target reference frame is usually distinct from the preset reference frame; however, in some implementation scenarios, multiple target reference frames may include the preset reference frame.
[0063] S203: Based on the decoded motion information, the information transmitted by the preset reference frame, and the reconstruction information corresponding to the target reference frame, the estimated motion information corresponding to the target reference frame is obtained; wherein, the reference information includes reconstruction information and estimated motion information.
[0064] Specifically, based on the decoded motion information, the information transmitted by the preset reference frame, and the reconstructed information corresponding to the target reference frame, the motion information of the target reference frame is estimated to obtain the estimated motion information of the target reference frame. The reference information includes the reconstructed information and the estimated motion information of the target reference frame. When the reference information is used to optimize the initial prediction information, it can perform motion compensation on the reconstructed information of the target reference frame, achieving a better optimization effect.
[0065] In some implementation scenarios, the image buffer region only caches the reconstructed information of the target reference frame, and the estimated motion information of the target reference frame is obtained using the optical flow compensation method. Specifically, obtaining the estimated motion information corresponding to the target reference frame based on the decoded motion information, the information transmitted from the preset reference frame, and the reconstructed information corresponding to the target reference frame includes: transforming the decoded motion information based on the information transmitted from the preset reference frame and the reconstructed information corresponding to the target reference frame to obtain the estimated motion information corresponding to the target reference frame.
[0066] Specifically, based on the information transmitted by the preset reference frame and the reconstruction information corresponding to the target reference frame, the decoded motion information is adaptively transformed to generate the estimated motion information obtained by the adaptive transformation for the target reference frame, thereby obtaining the estimated motion information that is adapted to each target reference frame. Moreover, there is no need to specifically transmit the motion information corresponding to each target reference frame in the bitstream, thus improving the compression rate of the bitstream.
[0067] Optionally, the adaptive transformation is based primarily on the difference between the information transmitted by the preset reference frame and the reconstructed information corresponding to the target reference frame.
[0068] It should be noted that, based on the information transmitted by the preset reference frame and the reconstructed information corresponding to the target reference frame, the decoded motion information is transformed to obtain the estimated motion information corresponding to the target reference frame. This includes: inputting the decoded motion information, the information transmitted by the preset reference frame, and the reconstructed information corresponding to the target reference frame into the transformation network to obtain the estimated motion information matched by the transformation network for the target reference frame; wherein, the transformation network is trained based on multiple training video frames.
[0069] Specifically, the decoded motion information, the information transmitted by the preset reference frame, and the reconstruction information corresponding to the target reference frame are input into the transform network to obtain the estimated motion information output by the transform network. Each target reference frame is matched with its own corresponding estimated motion information.
[0070] Furthermore, the transformation network is trained based on the information contained in multiple training video frames. Thus, the transformation network can transform motion information based on the differences between the information contained in different video frames, and adaptively generate matching predicted motion information for each target reference frame.
[0071] Understandably, please refer to Figure 6 , Figure 6 This is a schematic diagram of the topology of another embodiment of the image decoding method of this application. The decoded motion information, the information transmitted by the preset reference frame, and the reconstruction information of the target reference frame are input into the transform network. The transform network performs optical flow compensation for each target reference frame to generate the estimated motion information corresponding to the reconstruction information of each target reference frame.
[0072] Furthermore, by utilizing the estimated motion information corresponding to the target reference frame, motion compensation is performed on the reconstructed information of the target reference frame to obtain the reference prediction information corresponding to the target reference frame. The compensation network includes, but is not limited to, convolutional networks incorporating alignment methods such as deformable convolution and interpolation-based warp operations. Therefore, by fusing the initial prediction information and the reference prediction information corresponding to each target reference frame, the target prediction information for the current frame can be obtained. The fusion method includes, but is not limited to, neural networks such as attention networks, residual networks, and transformers.
[0073] For clarity, please refer to Figure 7 , Figure 7 This is a schematic diagram illustrating an application scenario corresponding to another embodiment of the image decoding method of this application. Taking two target reference frames as an example, and a transform network consisting of convolution and two residual blocks, the transformation and motion compensation of the motion information of the target reference frames are explained. The decoded motion information, the reconstructed information of the two reference frames, and the information transmitted by the preset reference frame are jointly input into the transform network to obtain the motion information transformation quantity output by the transform network. The decoded motion information and the motion information transformation quantity are superimposed to obtain the estimated motion information of the target reference frame. The estimated motion information is used to perform motion compensation on the reconstructed information of the target reference frame to obtain the reference prediction information corresponding to the target reference frame. Wherein, as... Figure 7 As shown, the first target reference frame and the second target reference frame perform the same process, that is, each target reference frame obtains the estimated motion information and uses the estimated motion information to perform motion compensation.
[0074] In some implementation scenarios, preset reference frames and a preset number of target reference frames are ordered sequentially, and reference motion information is matched between adjacent reference frames.
[0075] It is understandable that the preset reference frame and the preset number of target reference frames are all video frames that have been decoded before the current frame. The reference frames are sorted in order of decoding. There is motion information obtained by motion estimation between adjacent reference frames. Once the motion information is cached, the motion information between adjacent reference frames can be obtained as reference motion information.
[0076] Furthermore, based on the decoded motion information, the information transmitted by the preset reference frame, and the reconstructed information corresponding to the target reference frame, the estimated motion information corresponding to the target reference frame is obtained, including: for each target reference frame in sequence, determining the preceding reference frame; based on the motion information between the current frame and the preceding reference frame, the reference motion information matched between the target reference frame and the preceding reference frame, the information transmitted by the preset reference frame, and the reconstructed information corresponding to at least some of the target reference frames, the motion information matched between the current frame and the target reference frame is estimated to obtain the estimated motion information corresponding to the target reference frame.
[0077] Specifically, please refer to Figure 8 , Figure 8This is a schematic diagram of the topology of another embodiment of the image decoding method of this application. There is at least one reference frame between the current frame and the target reference frame; that is, motion information is determined across frames between the current frame and the target reference frame. Starting with the target reference frame closest to the current frame, the estimated motion information of the current frame relative to each target reference frame is estimated for each sequentially ordered target reference frame. Reference frames located between the current target reference frames are determined as forward reference frames. Based on the motion information between the current frame and the forward reference frames, the reference motion information matched between the target reference frames and the forward reference frames, the information transmitted by the preset reference frames, and the reconstruction information corresponding to at least some of the target reference frames, the motion information matched between the current frame and the target reference frames is estimated across frames to obtain the estimated motion information corresponding to the target reference frame.
[0078] For clarity, please refer to Figure 9 , Figure 9 This is a schematic diagram illustrating an application scenario of one embodiment of this application for obtaining estimated reference information corresponding to all target reference frames, wherein, as shown in the diagram... Figure 9 The target reference frames include A1 to A2. N For example, the preset reference frame A0 is the forward reference frame of the target reference frame A1. Based on the motion information F between the current frame and the forward reference frame, the reference motion information F0 matched between the target reference frame and the forward reference frame, the information transmitted by the preset reference frame and the reconstruction information corresponding to at least part of the target reference frame, the motion information between the current frame and the target reference frame A1 is estimated to obtain the estimated motion information C1.
[0079] Furthermore, when traversing to the target reference frame A2, the target reference frame A1 is the preceding reference frame of the target reference frame A2. Based on the motion information C1 between the current frame and the preceding reference frame, the reference motion information F1 matched between the target reference frame and the preceding reference frame, the information transmitted by the preset reference frame, and the reconstructed information corresponding to at least part of the target reference frame, the motion information between the current frame and the target reference frame A2 is estimated to obtain the estimated motion information C2. This process is repeated to determine the target reference frames A1 to A2. N Each has its corresponding estimated motion information.
[0080] It is understandable that the estimated motion information C1 to CN obtained across frames is determined by grouping the current frame, the forward reference frame, and the current target reference frame into three frames, thereby sequentially determining the estimated motion information corresponding to each target reference frame.
[0081] It should be noted that, based on the motion information between the current frame and the forward reference frame, the reference motion information matched between the target reference frame and the forward reference frame, the information transmitted by the preset reference frame, and the reconstruction information corresponding to at least part of the target reference frame, the motion information matched between the current frame and the target reference frame is estimated to obtain the estimated motion information corresponding to the target reference frame. This includes: using the motion information between the current frame and the forward reference frame to compensate the reference motion information matched between the target reference frame and the forward reference frame to obtain compensated motion information; and fusing the motion information between the current frame and the forward reference frame, the compensated motion information, the information transmitted by the preset reference frame, and the reconstruction information corresponding to at least part of the target reference frame to obtain the estimated motion information corresponding to the target reference frame.
[0082] Specifically, please refer to Figure 10 , Figure 10 This is a schematic diagram illustrating an application scenario of one embodiment of this application for obtaining estimated motion information corresponding to a single target reference frame, wherein... Figure 10 The motion information between the current frame and the preceding reference frame is labeled as motion information I, the motion information between the target reference frame and the preceding reference frame is labeled as motion information J, and the estimated motion information to be obtained is labeled as motion information K. Using the motion information I between the current frame and the preceding reference frame, the motion information J matched between the target reference frame and the preceding reference frame is compensated to obtain compensated reference information. This compensated reference information is then fused with the reconstructed information of at least a portion of the target reference frame, and can also be fused with information transmitted from a preset reference frame, thereby obtaining motion information K, which serves as the estimated motion information corresponding to the target reference frame. Therefore, by compensating first and then fusing, the estimated motion information matched between the current frame and the target reference frame is sequentially obtained, thereby improving the accuracy of the estimated motion information, and eliminating the need to transmit this cross-frame motion information during the motion coding stage of the current frame, thus reducing the amount of data transmission.
[0083] In a specific implementation scenario, Figure 9 The process of determining the estimated motion information for each target reference frame, as shown, can be formulated as follows:
[0084] K=func1(I, func(J,I),(A0,A1...A N ))(1)
[0085] Where func is the motion compensation method, func1 is the fusion operation, A0 is the information transmitted by the preset reference frame, A1...A NThe reconstruction information refers to the target reference frames. This application does not limit the number of target reference frames, meaning that the reconstruction information of all target reference frames may not be used. The motion compensation method can be implemented in ways including, but not limited to, alignment methods such as deformable convolutions and interpolation-based warp operations, or convolutional networks incorporating the aforementioned methods. The fusion method can be implemented in ways including, but not limited to, addition operations, concatenation + convolution, and concatenation + attention structures, among other convolutional networks.
[0086] Understandable Figure 9 A1 to A1 shown N The calculations for the corresponding estimated motion information are as follows:
[0087] C1=func1(F, func(F0,F),(A0,A1...A N ))(2)
[0088] By analogy, we can obtain:
[0089] C2=func1(C1, func(F1, C1), (A0, A1...A N ))(3)
[0090] C3=func1(C2, func(F2, C2), (A0, A1...A N ))(4)
[0091] C N =func1(C N-1 ,func(F N-1 C N-1 ), (A0, A1…A N ))(5)
[0092] Among them, C1~C N The motion compensation operation func and the fusion operation func1 can be set to the same way, or they can be set to at least some different ways.
[0093] In a specific implementation scenario, the compensation reference information is obtained using a compensation network, and the predicted motion information is obtained using a fusion network. Both the compensation network and the fusion network are trained based on multiple training video frames and their corresponding motion information.
[0094] Specifically, the compensation network and the fusion network are trained based on the information contained in multiple training video frames and the motion information of the training video frames. The compensation network can adaptively utilize the motion information between the current frame and the forward reference frame to compensate for the reference motion information matched between the target reference frame and the forward reference frame. The fusion network can adaptively fuse the compensated reference information with the information corresponding to the reference frame, thereby efficiently and accurately determining the estimated motion information corresponding to the target reference frame.
[0095] S204: Based on the initial prediction information and the reference information corresponding to the target reference frame, obtain the target prediction information corresponding to the current frame; wherein, the target prediction information is used to decode and obtain the reconstruction information corresponding to the current frame.
[0096] Specifically, based on the initial prediction information and the reference information corresponding to the target reference frame, the prediction information of the current frame is corrected to obtain the target prediction information corresponding to the current frame.
[0097] In some implementation scenarios, motion compensation is performed on the reconstructed information of the target reference frame based on the estimated motion information corresponding to the target reference frame to obtain the reference prediction information corresponding to the target reference frame; the initial prediction information and the reference prediction information are fused to obtain the target prediction information corresponding to the current frame.
[0098] Specifically, based on the estimated motion information corresponding to the target reference frame, motion compensation is performed on the reconstructed information of the target reference frame to obtain the reference prediction information corresponding to the target reference frame, thereby making the reference prediction information have high accuracy.
[0099] Furthermore, the initial prediction information is fused with the reference prediction information, thereby combining the prediction information of the target reference frame with the initial prediction information to obtain target prediction information with higher precision and accuracy for the current frame. This allows the reference prediction information of the target reference frame to correct the initial prediction information, thereby improving the precision and accuracy of the target prediction information.
[0100] Optionally, the motion compensation method can be implemented in ways including, but not limited to, alignment methods such as deformable convolutions and interpolation-based warp operations, or convolutional networks incorporating the above methods. The fusion method can be implemented in ways including, but not limited to, convolutional networks using addition operations, concatenation + convolution, or concatenation + attention structures.
[0101] In this embodiment, the motion stream is decoded to obtain decoded motion information. Based on the motion information and a preset reference frame, the current frame is predicted to obtain initial prediction information corresponding to the current frame. Based on the decoded motion information, the information transmitted by the preset reference frame, and the reconstructed information corresponding to the target reference frame, the motion information of the target reference frame is estimated to obtain estimated motion information of the target reference frame. Based on the estimated motion information corresponding to the target reference frame, motion compensation is performed on the reconstructed information of the target reference frame to obtain more accurate prediction information corresponding to the target reference frame, which is used as reference prediction information. The initial prediction information and the reference prediction information are fused, and the initial prediction information is corrected by combining it with the prediction information of the target reference frame to obtain target prediction information with higher precision and accuracy corresponding to the current frame.
[0102] Please see Figure 11 , Figure 11 This is a flowchart illustrating one embodiment of the image encoding method of this application, which includes:
[0103] S301: Obtain the current frame and its corresponding preset reference frame, perform motion estimation on the current frame and the preset reference frame to obtain the motion information of the current frame relative to the preset reference frame, encode and decode the motion information to obtain the decoded motion information of the current frame relative to the preset reference frame, and obtain the initial prediction information corresponding to the current frame based on the decoded motion information and the information transmitted by the preset reference frame.
[0104] Specifically, please refer to [the relevant document] again. Figure 2 The current frame and its corresponding preset reference frame are obtained. Motion estimation is performed on the current frame and the preset reference frame to obtain the motion information of the current frame relative to the preset reference frame.
[0105] Furthermore, the motion information of the current frame relative to the preset reference frame is encoded and decoded to obtain the decoded motion information of the current frame relative to the preset reference frame. Based on the decoded motion information and the information transmitted by the preset reference frame, the initial prediction information corresponding to the current frame is obtained.
[0106] It is understandable that the process of encoding motion information also involves, for example, [other factors]. Figure 2 The current frame features and preset reference frame features shown are shown, as well as contextual motion information, wherein the contextual motion information is generated by the decoder during decoding and then cached in the cache area.
[0107] S302: Obtain the target reference frame corresponding to the current frame and determine the reference information corresponding to the target reference frame; wherein, the number of target reference frames is at least one, and the reference information includes at least the reconstruction information of the target reference frame.
[0108] Specifically, at least one target reference frame corresponding to the current frame is obtained, and the reference information corresponding to the target reference frame is determined, thereby obtaining more reference frames to correct the initial prediction information. The reference information includes at least the reconstruction information of the target reference frame.
[0109] It is understandable that there can be one or more target reference frames. Target reference frames are usually different from preset reference frames. In some implementation scenarios, preset reference frames may also be included among multiple target reference frames.
[0110] S303: Based on the initial prediction information and the reference information corresponding to the target reference frame, obtain the target prediction information corresponding to the current frame; wherein, the decoded motion information and target prediction information are used to encode the current frame bitstream of the current frame.
[0111] Specifically, based on the initial prediction information and the reference information corresponding to the target reference frame, the prediction information of the current frame is corrected to obtain the target prediction information corresponding to the current frame. In this process, combining multiple reference frames to determine the target prediction information corresponding to the current frame improves the accuracy of the prediction information and reduces redundant information. The decoded motion information and target prediction information are then used to encode the current frame bitstream, thereby effectively improving the compression ratio of the bitstream.
[0112] Understandably, in combination Figure 2 As can be seen, the technical solution adopted by the encoding end to obtain the target prediction information of the current frame is the same as that described in any of the above embodiments, and this application will not repeat it here.
[0113] Furthermore, the current frame bitstream includes a motion bitstream and a residual bitstream. The motion bitstream is obtained by encoding the motion information of the current frame relative to a preset reference frame.
[0114] It is understandable that after obtaining the target prediction information corresponding to the current frame based on the initial prediction information and the reference information corresponding to the target reference frame, the process also includes: encoding the residual information of the current frame compared to the preset reference frame and the target prediction information using a preset encoding method to obtain the residual bitstream.
[0115] Specifically, the preset encoding method can include any of the following: individual encoding method, merged encoding method, and reference encoding method. Correspondingly, the preset decoding method can include any of the following: individual decoding method, split decoding method, and reference decoding method. By encoding motion information using the preset encoding method, a motion bitstream can be obtained. By encoding the residual information of the current frame compared to the preset reference frame and the target prediction information using the preset encoding method, a residual bitstream can be obtained. These two methods together form the current frame bitstream, enabling the current frame bitstream to transmit motion information and achieve a higher compression ratio.
[0116] The above scheme acquires the current frame and its corresponding preset reference frame, performs motion estimation on the current frame and the preset reference frame to obtain motion information of the current frame relative to the preset reference frame, encodes and decodes the motion information of the current frame relative to the preset reference frame to obtain decoded motion information of the current frame relative to the preset reference frame, and obtains initial prediction information corresponding to the current frame based on the decoded motion information and the information transmitted by the preset reference frame. It then acquires at least one target reference frame corresponding to the current frame, determines the reference information corresponding to the target reference frame, thereby obtaining more reference frames to correct the initial prediction information. The reference information includes at least the reconstruction information of the target reference frame. Based on the initial prediction information and the reference information corresponding to the target reference frame, the prediction information of the current frame is corrected to obtain the target prediction information corresponding to the current frame. Combining multiple reference frames to determine the target prediction information corresponding to the current frame improves the accuracy of the prediction information and reduces redundant information. The motion information and target prediction information are used to encode the current frame bitstream, thereby effectively improving the compression ratio of the bitstream.
[0117] Please see Figure 12 , Figure 12 This is a schematic diagram of an embodiment of the image decoding terminal 40 of this application. The image decoding terminal 40 includes a decoding module 401, a determining module 402, and a generating module 403. The decoding module 401 decodes the motion bitstream to obtain decoded motion information of the current frame relative to a preset reference frame. Based on the decoded motion information and the information transmitted by the preset reference frame, it obtains initial prediction information corresponding to the current frame. The motion bitstream is obtained by encoding motion information, and the motion information is obtained by motion estimation based on the current frame and its corresponding preset reference frame. The determining module 402 obtains the target reference frame corresponding to the current frame and determines the reference information corresponding to the target reference frame. The number of target reference frames is at least one, and the reference information includes at least the reconstruction information of the target reference frame. The generating module 403 obtains target prediction information corresponding to the current frame based on the initial prediction information and the reference information corresponding to the target reference frame. The target prediction information is used to decode and obtain the reconstruction information corresponding to the current frame.
[0118] It is understood that the image decoding end 40 can implement the image decoding method described in any of the above embodiments. For related explanations, please refer to the detailed description of the above method embodiments, which will not be repeated here.
[0119] Please see Figure 13 , Figure 13This is a schematic diagram of the structure of an embodiment of the image encoding terminal 50 of this application. The image encoding terminal 50 includes an encoding / decoding module 501, a determination module 502, and a generation module 503. The encoding / decoding module 501 is used to acquire the current frame and its corresponding preset reference frame, perform motion estimation on the current frame and the preset reference frame to obtain motion information of the current frame relative to the preset reference frame, encode and decode the motion information to obtain decoded motion information of the current frame relative to the preset reference frame, and obtain initial prediction information corresponding to the current frame based on the decoded motion information and the information transmitted by the preset reference frame. The determination module 502 is used to acquire the target reference frame corresponding to the current frame and determine the reference information corresponding to the target reference frame; wherein, the number of target reference frames is at least one, and the reference information includes at least the reconstruction information of the target reference frame. The generation module 503 is used to obtain target prediction information corresponding to the current frame based on the initial prediction information and the reference information corresponding to the target reference frame; wherein, the motion information and target prediction information corresponding to the current frame are used to encode the current frame bitstream.
[0120] It is understood that the image encoding end 50 can implement the image encoding method described in any of the above embodiments. For related explanations, please refer to the detailed description of the above method embodiments, which will not be repeated here.
[0121] Please see Figure 14 , Figure 14 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 60 includes a memory 601 and a processor 602 coupled to each other. The memory 601 stores program data (not shown in the figure), and the processor 602 calls the program data to implement the method in any of the above embodiments. For the description of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here.
[0122] Please see Figure 15 , Figure 15 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 70 stores program data 700. When the program data 700 is executed by a processor, it implements the method in any of the above embodiments. For a detailed description of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here.
[0123] It should be noted that the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0124] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0125] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0126] The above description is merely an embodiment of this application and does not limit the scope of protection of this application. Any equivalent structural or procedural transformations made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of protection of this application.
Claims
1. An image decoding method characterized by, The method comprises: decoding a motion code stream to obtain decoded motion information of a current frame relative to a preset reference frame, and obtaining initial prediction information corresponding to the current frame based on the decoded motion information and information transmitted by the preset reference frame; wherein the motion code stream is obtained by encoding motion information, and the motion information is obtained by motion estimation based on the current frame and the preset reference frame corresponding thereto; obtaining target reference frames corresponding to the current frame, and determining reference information corresponding to the target reference frames; wherein the number of the target reference frames is at least one, and the reference information at least comprises reconstructed information of the target reference frames; obtaining target prediction information corresponding to the current frame based on the initial prediction information and the reference information corresponding to the target reference frames; wherein the target prediction information is used to decode reconstructed information corresponding to the current frame.
2. The image decoding method according to claim 1, characterized by, The method comprises: fusing the initial prediction information and the reference information corresponding to the target reference frames to obtain the target prediction information corresponding to the current frame; or fusing the initial prediction information, information transmitted by the preset reference frame and the reference information corresponding to the target reference frames to obtain the target prediction information corresponding to the current frame.
3. The image decoding method according to claim 1, characterized by, The method comprises: obtaining a preset number of the target reference frames corresponding to the current frame, and determining reconstructed information corresponding to each of the target reference frames; obtaining estimated motion information corresponding to the target reference frames based on the decoded motion information, information transmitted by the preset reference frame and the reconstructed information corresponding to the target reference frames; wherein the reference information comprises the reconstructed information and the estimated motion information.
4. The image decoding method according to claim 3, characterized by, The method comprises: transforming the decoded motion information based on the information transmitted by the preset reference frame and the reconstructed information corresponding to the target reference frames to obtain the estimated motion information corresponding to the target reference frames.
5. The image decoding method according to claim 4, characterized by, The method comprises: inputting the decoded motion information, the information transmitted by the preset reference frame and the reconstructed information corresponding to the target reference frames into a transformation network to obtain estimated motion information matched by the transformation network for the target reference frames; wherein the transformation network is trained based on a plurality of training video frames.
6. The image decoding method according to claim 3, characterized by, The preset reference frame and the preset number of the target reference frames are sequentially sorted, and reference motion information is matched between adjacent reference frames. The method comprises: For each of the target reference frames in the order, a forward reference frame preceding the target reference frame in the order is determined; Based on the motion information between the current frame and the forward reference frame, the reference motion information matched between the target reference frame and the forward reference frame, and the information transmitted by the preset reference frame and the reconstruction information corresponding to at least part of the target reference frame, the motion information matched between the current frame and the target reference frame is estimated to obtain the estimated motion information corresponding to the target reference frame.
7. The image decoding method according to claim 6, characterized by, The estimation of the motion information matched between the current frame and the target reference frame based on the motion information between the current frame and the forward reference frame, the reference motion information matched between the target reference frame and the forward reference frame, and the information transmitted by the preset reference frame and the reconstruction information corresponding to at least part of the target reference frame to obtain the estimated motion information corresponding to the target reference frame comprises: Compensate the reference motion information matched between the target reference frame and the forward reference frame by using the motion information between the current frame and the forward reference frame to obtain compensated motion information; Fuse the motion information between the current frame and the forward reference frame, the compensated motion information, and the information transmitted by the preset reference frame and the reconstruction information corresponding to at least part of the target reference frame to obtain the estimated motion information corresponding to the target reference frame.
8. The image decoding method according to claim 7, characterized by, The compensation reference information is obtained by using a compensation network, and the estimated motion information is obtained by using a fusion network. The compensation network and the fusion network are trained based on a plurality of training video frames and corresponding motion information.
9. The image decoding method according to claim 3, characterized by, The target prediction information corresponding to the current frame is obtained based on the initial prediction information and the reference information corresponding to the target reference frame, comprising: Motion compensation is performed on the reconstruction information of the target reference frame based on the estimated motion information corresponding to the target reference frame to obtain the reference prediction information corresponding to the target reference frame; Fuse the initial prediction information and the reference prediction information to obtain the target prediction information corresponding to the current frame.
10. The image decoding method according to any one of claims 1 to 9, characterized by, The target reference frame is different from the preset reference frame.
11. An image coding method characterized by, The method comprises: Obtain a current frame and a corresponding preset reference frame, perform motion estimation on the current frame and the preset reference frame to obtain motion information of the current frame relative to the preset reference frame, encode and decode the motion information to obtain decoded motion information of the current frame relative to the preset reference frame, and based on the decoded motion information and information transmitted by the preset reference frame, obtain initial prediction information corresponding to the current frame; Obtain a target reference frame corresponding to the current frame and determine reference information corresponding to the target reference frame; wherein the number of target reference frames is at least one, and the reference information at least includes reconstruction information of the target reference frame; obtain target prediction information corresponding to the current frame based on the initial prediction information and reference information corresponding to the target reference frame; wherein the motion information and the target prediction information corresponding to the current frame are used to encode a current frame bitstream of the current frame.
12. The image coding method of claim 11, wherein, The current frame bitstream includes a motion bitstream and a residual bitstream, and the motion information of the current frame relative to the preset reference frame is encoded to obtain the motion bitstream. After the target prediction information corresponding to the current frame is obtained based on the initial prediction information and the reference information corresponding to the target reference frame, the method further includes: The residual information of the current frame relative to the preset reference frame and the target prediction information are encoded by using a preset encoding mode to obtain the residual bitstream.
13. The image coding method of any of claims 11-12, wherein, The target reference frame is different from the preset reference frame.
14. An electronic device, comprising: It includes: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor invokes the program data to execute the method of any one of claims 1-10 or 11-13.
15. A computer readable storage medium having stored thereon program data, wherein, The program data is executed by the processor to implement the method of any one of claims 1-10 or 11-13.