Encoding method, decoding method, and related apparatuses

Abstract

The present application relates to the field of data compression, and discloses an encoding method, a decoding method, and related apparatuses. The encoding method comprises: determining a global reference tensor corresponding to the current image and motion information of the current image; determining target prediction information of the current image on the basis of the global reference tensor corresponding to the current image and the motion information of the current image; determining residual information of the current image on the basis of the target prediction information of the current image and the current image; and encoding the residual information of the current image and the motion information of the current image into a bitstream. The global reference tensor is generated on the basis of a plurality of encoded images prior to the current image, so that the global reference tensor is capable of characterizing image information available for reference by the current image among the plurality of encoded images prior to the current image, thereby achieving globality across temporal domains. By means of the present application, the target prediction information obtained is more accurate, so that the data bit amount of the residual information of the current image determined subsequently is smaller, thereby effectively improving the encoding and decoding efficiency.

Description

Coding and decoding method and related device

[0001] This application claims priority to Chinese patent application No. 202410095535.5 filed on January 23, 2024, with invention name “Encoding and decoding method, device, equipment, storage medium and computer program”, and claims priority to Chinese patent application No. 202410445360.6 filed on April 12, 2024, with invention name “Encoding and decoding method and related device”, the entire contents of which are incorporated into this application by reference. Technical Field

[0002] The present application relates to the field of data compression, and in particular to a coding and decoding method and related devices. Background Art

[0003] Video compression is a technology that leverages redundant information in a video to represent the original video with less data (video bitstream). This reduces the network bandwidth requirements for video storage and transmission. Video compression involves encoding and decoding, with encoding and decoding performance (reflecting video quality) and encoding and decoding efficiency (reflecting encoding and decoding time) being key considerations. However, some current video compression schemes suffer from poor encoding and decoding performance and low efficiency.

[0004] Summary of the Invention

[0005] This application provides a coding and decoding method, apparatus, device, storage medium, and computer program that can solve the problems of poor coding and decoding performance and low efficiency in related technologies. The technical solution is as follows:

[0006] In a first aspect, a coding method is provided, the method comprising: determining a global reference tensor corresponding to a current image and motion information of the current image, the global reference tensor being determined based on multiple encoded images before the current image; determining target prediction information of the current image based on the global reference tensor corresponding to the current image and the motion information of the current image; determining residual information of the current image based on the target prediction information of the current image and the current image; and encoding the residual information and the motion information of the current image into a bitstream.

[0007] The present application can determine the target prediction information of the current image through the global reference tensor corresponding to the current image and the motion information of the current image. Since the global reference tensor is determined based on multiple encoded images before the current image, that is, the embodiment of the present application can generate a global reference tensor based on multiple encoded images before the current image, so that the global reference tensor can represent the image information that can be used for reference by the current image in multiple encoded images before the current image, and has globality across the time domain. In this way, it can be ensured that the target prediction information corresponding to the current image obtained based on the global reference tensor corresponding to the current image and the motion information of the current image is more accurate, so that the data bit amount of the residual information corresponding to the current image determined subsequently is smaller, thereby effectively improving the efficiency of encoding and decoding. In addition, by generating a global reference tensor based on multiple encoded images before the current image, the encoding end can ensure that the current image has more information for reference when encoding without having to cache multiple encoded images, thereby greatly reducing the cache pressure of the encoding end on the basis of improving the encoding and decoding efficiency.

[0008] Optionally, if the current image is in a video, and the current image is the first frame in the video, the encoder can determine that the global reference tensor corresponding to the current image is 0, that is, the global reference tensor corresponding to the first frame is a feature of all 0s. When the current image is the Mth frame in the video, the encoder can determine the global reference tensor corresponding to the current image and the motion information of the current image, where M is an integer greater than 1.

[0009] Optionally, if the current image is in a video, and the video includes at least one GOP, the encoder can determine that the global reference tensor corresponding to the current image is 0, that is, the global reference tensor corresponding to the I frame is a feature of all 0s.

[0010] When the current image is an I frame in the Nth GOP in the video, or a P frame in the video, the encoder can determine the global reference tensor corresponding to the current image and the motion information of the current image, where N is an integer greater than 1.

[0011] It should be noted that when the current image is an I frame in a GOP, if the encoding end determines the global reference tensor corresponding to the current image to be 0, that is, no matter which GOP the current image is in the video, the encoding end determines the global reference tensor corresponding to the current image to be 0. In this case, the global reference tensor is determined based on multiple encoded images before the current image, and the multiple encoded images are the encoded images in the GOP where the current image is located.

[0012] Optionally, if the encoding end determines the global reference tensor corresponding to the current image to 0 only when the current image is an I frame in the first GOP in the video, and when the current image is an I frame in the Nth GOP in the video, the encoding end determines the global reference tensor corresponding to the current image and the motion information of the current image. In this case, the global reference tensor corresponding to the current image is determined based on all encoded images before the current image. At this time, the global reference tensor is not limited to one encoded image or several encoded images, but can represent all information that can be used for reference in multiple encoded images before the current image, thereby increasing the information available for reference for the current image and further improving the efficiency of encoding and decoding.

[0013] Optionally, the encoding end determines reconstruction information of a first encoded image, where the first encoded image is an encoded image adjacent to the current image, and based on the reconstruction information of the first encoded image, updates the global reference tensor corresponding to the first encoded image to obtain the global reference tensor corresponding to the current image.

[0014] It should be noted that the first coded image is usually the previous coded image adjacent to the current image. For example, if the current image is the tth frame of the video, the first coded image is the t-1th frame.

[0015] Since the decoding end needs to update the global reference tensor corresponding to the first decoded image in the subsequent steps, the first decoded image is a decoded image adjacent to the current image, and the first decoded image of the decoding end and the first encoded image of the encoding end are the same image. For the convenience of description, the first decoded image and the first encoded image are subsequently referred to as the first image. When the decoding end performs decoding, it is unable to obtain the original first image of the encoding end, and can only obtain the reconstructed first image. That is to say, the first image reconstructed by the decoding end may have a certain deviation relative to the original first image. Therefore, the encoding end updates the global reference tensor corresponding to the first image through the reconstruction information of the first image to ensure that the global reference tensor corresponding to the current image subsequently used by the encoding end and the decoding end is the same, so that the target prediction information of the current image obtained by the encoding end and the decoding end is more accurate, thereby effectively improving the efficiency and performance of encoding and decoding.

[0016] If the current image is in a video that includes at least one GOP, if the current image is the first P frame in the GOP, the first coded image may be an I frame in the GOP. If the current image is the second P frame in the GOP, the first coded image may be the first P frame in the GOP.

[0017] If the current image is in the video, and the encoder determines that the global reference tensor corresponding to the current image is 0 when the current image is the first frame in the video, and determines the global reference tensor corresponding to the current image and the motion information of the current image when the current image is the Mth frame in the video. In this case, the global reference tensor corresponding to the image of the Nth frame in the video code stream is obtained based on the global reference tensor corresponding to the encoded image of the N-1th frame and the reconstruction information of the encoded image of the N-1th frame, and the global reference tensor corresponding to the second frame is the reconstruction information of the encoded image of the 1st frame.

[0018] Optionally, the reconstruction information of the first coded image includes the reconstructed first coded image, or image features of the reconstructed first coded image. In different situations, the implementation methods for determining the reconstruction information of the first coded image are different, which will be introduced below respectively.

[0019] If the reconstruction information of the first coded image includes a reconstructed first coded image, the encoding end can reconstruct the first coded image to obtain a reconstructed image of the first coded image, that is, obtain reconstructed motion information of the first coded image based on the code stream, and then determine target prediction information of the first coded image based on a global reference tensor corresponding to the first coded image and the reconstructed motion information of the first coded image; obtain reconstructed residual information of the first coded image based on the code stream; and reconstruct the first coded image based on the target prediction information of the first coded image and the reconstructed residual information of the first coded image to obtain a reconstructed image of the first coded image.

[0020] If the reconstruction information of the first encoded image includes a reconstructed first encoded image, the encoding end can reconstruct the first encoded image to obtain a reconstructed image of the first encoded image, and determine image features of the reconstructed first encoded image based on the reconstructed image of the first encoded image.

[0021] Optionally, the encoding end inputs the reconstructed image of the first encoded image into an image feature extraction network to obtain image features of the reconstructed first encoded image output by the image feature extraction network.

[0022] Optionally, based on the reconstruction information of the first encoded image, the global reference tensor corresponding to the first encoded image is updated to obtain the global reference tensor corresponding to the current image. The implementation process includes: inputting the reconstruction information of the first encoded image and the global reference tensor corresponding to the first encoded image into the global reference tensor update network to obtain the global reference tensor corresponding to the current image output by the global reference tensor update network.

[0023] Optionally, the motion information of the current image includes a motion vector, where the motion vector indicates an offset between a target block in the current image and a reference image for the current image, where the reference image is a reconstructed image of an adjacent coded image. In this case, the reference image for the current image is the reconstructed image of the first coded image.

[0024] Before determining the motion information of the current image based on the image features of the current image and the image features of the reconstructed first encoded image, the encoding end can extract image features from the current image to obtain the image features of the current image.

[0025] Optionally, the encoding end inputs the current image into an image feature extraction network to obtain image features of the current image output by the image feature extraction network.

[0026] Optionally, based on the global reference tensor corresponding to the current image and the motion information of the current image, the global prediction information corresponding to the current image is determined, and based on the global prediction information corresponding to the current image, the target prediction information of the current image is determined.

[0027] Optionally, the encoding end can determine the reconstructed motion information of the current image based on the motion information of the current image, and determine the global motion information corresponding to the current image based on the reconstructed motion information of the current image, where the global motion information includes a global motion vector, which indicates the offset of the target block in the current image and the global reference tensor corresponding to the current image, and determine the global prediction information corresponding to the current image based on the global reference tensor corresponding to the current image and the global motion information corresponding to the current image.

[0028] Since the decoding end also needs to determine the global motion information corresponding to the current image in the subsequent steps, the decoding end cannot obtain the original motion information of the current image during decoding, and can only obtain the reconstructed motion information of the current image. That is to say, the motion information reconstructed by the decoding end may have a certain deviation compared with the motion information corresponding to the original current image. Therefore, the encoding end determines the global motion information corresponding to the current image through the reconstructed motion information of the current image to ensure that the global motion information corresponding to the current image subsequently used by the encoding end and the decoding end is the same, so that the encoding end and the decoding end obtain the target prediction information of the current image more accurately, thereby effectively improving the efficiency and performance of encoding and decoding.

[0029] Optionally, the encoding end updates the global motion information corresponding to the first encoded image based on the reconstructed motion information of the current image to obtain the global motion information corresponding to the current image.

[0030] Optionally, the encoding end inputs the reconstructed motion information of the current image and the global motion information corresponding to the first encoded image into the global motion information update network to obtain the global motion information corresponding to the current image output by the global motion information update network.

[0031] Optionally, the encoding end determines the global prediction information corresponding to the current image based on the global reference tensor corresponding to the current image and the global motion information corresponding to the current image.

[0032] Optionally, the encoder may also directly determine the global prediction information corresponding to the current image based on the global reference tensor corresponding to the current image and the motion information of the current image.

[0033] Since the motion information of the current image can characterize the motion of the target block in the current image relative to the encoded image to a certain extent, the encoding end can directly determine the global prediction information corresponding to the current image based on the motion information of the current image and the global reference tensor corresponding to the current image. Compared with determining the global prediction information corresponding to the current image based on the global reference tensor corresponding to the current image and the global motion information corresponding to the current image, this can effectively reduce the computational amount of video compression and improve the efficiency of encoding and decoding while ensuring the accuracy of the subsequently determined target prediction information.

[0034] Optionally, the global prediction information corresponding to the current image may be directly determined as the target prediction information of the current image.

[0035] Optionally, the encoding end determines the local prediction information corresponding to the current image based on the reconstructed motion information of the current image and the reconstruction information of the first encoded image, and determines the target prediction information of the current image based on the local prediction information corresponding to the current image and the global prediction information corresponding to the current image.

[0036] The implementation process of determining the target prediction information of the current image based on the local prediction information corresponding to the current image and the global prediction information corresponding to the current image includes: fusing the local prediction information corresponding to the current image and the global prediction information corresponding to the current image to obtain the target prediction information of the current image.

[0037] It should be noted that, when the reconstruction information of the first coded image includes the reconstructed first coded image, the target prediction information of the current image is the same as the channel dimension and spatial dimension of the current image. When the reconstruction information of the first coded image includes the image features of the reconstructed first coded image, the target prediction information of the current image is the same as the channel dimension and spatial dimension of the image features of the current image.

[0038] Optionally, if the reconstruction information of the first encoded image includes image features of the reconstructed first encoded image, in this case, the encoding end can determine the residual information corresponding to the current image based on the target prediction information of the current image and the image features of the current image.

[0039] Optionally, if the reconstruction information of the first encoded image includes the reconstructed first encoded image, in this case, the encoding end can determine the residual information of the current image based on the target prediction information of the current image and the current image.

[0040] It should be noted that the residual information of the current image and the motion information of the current image can be encoded into different code streams respectively, or the residual information of the current image and the motion information of the current image can be encoded into the same code stream.

[0041] In a second aspect, a decoding method is provided, the method comprising: obtaining reconstructed motion information of a current image based on a code stream; determining a global reference tensor corresponding to the current image, the global reference tensor being determined based on multiple decoded images before the current image; determining target prediction information of the current image based on the global reference tensor corresponding to the current image and the reconstructed motion information of the current image; obtaining reconstructed residual information of the current image based on the code stream; and obtaining a reconstructed image of the current image based on the target prediction information of the current image and the reconstructed residual information of the current image.

[0042] During the decoding process of the present application, accurate target prediction information can be obtained through the reconstructed motion information of the current image and the global reference tensor corresponding to the current image. In this way, it can be ensured that the subsequent reconstructed image of the current image based on the target prediction information and the reconstructed residual information corresponding to the current image has better image quality, thereby greatly improving the encoding and decoding performance.

[0043] Optionally, the reconstructed motion information of the current image includes a motion vector, where the motion vector indicates an offset between a target block in the current image and a reference image for the current image, where the reference image is a reconstructed image of a decoded image adjacent to the current image. In this case, the reference image for the current image is a reconstructed image of a first decoded image described below, or in other words, the reference image for the current image is the reconstructed first decoded image.

[0044] If the current image is in a video, and the current image is the first frame in the video, the decoder can determine that the global reference tensor corresponding to the current image is 0, that is, the global reference tensor corresponding to the first frame is a feature of all 0s. If the current image is the Mth frame in the video, the decoder can determine the global reference tensor corresponding to the current image, where M is an integer greater than 1.

[0045] If the current image is in a video that includes at least one GOP, the decoder can determine that the global reference tensor corresponding to the current image is 0, that is, the global reference tensor corresponding to the I-frame is all 0s. When the current image is an I-frame in the Nth GOP of the video, or a P-frame in the video, the decoder can determine the global reference tensor corresponding to the current image, where N is an integer greater than 1.

[0046] If the decoding end determines the global reference tensor corresponding to the current image as 0 only when the current image is an I frame in the first GOP in the video, and when the current image is an I frame in the Nth GOP in the video, the decoding end determines the global reference tensor corresponding to the current image. In this case, the global reference tensor corresponding to the current image is determined based on all decoded images before the current image.

[0047] Optionally, reconstruction information of a first decoded image, where the first decoded image is a decoded image adjacent to the current image, is used to update the global reference tensor corresponding to the first decoded image based on the reconstruction information of the first decoded image to obtain the global reference tensor corresponding to the current image.

[0048] It should be noted that the first decoded image is usually the previous decoded image adjacent to the current image.

[0049] When the current image is in a video including at least one GOP, if the current image is the first P frame in the GOP, the first decoded image may be an I frame in the GOP. If the current image is the second P frame in the GOP, the first decoded image may be the first P frame in the GOP.

[0050] If the current image is in the video, and the decoding end determines that the global reference tensor corresponding to the current image is 0 when the current image is the first frame in the video, and determines the global reference tensor corresponding to the current image when the current image is the Mth frame in the video. In this case, the global reference tensor corresponding to the image of the Nth frame in the video code stream is obtained based on the global reference tensor corresponding to the decoded image of the N-1th frame and the reconstruction information of the decoded image of the N-1th frame. The global reference tensor corresponding to the second frame is the reconstruction information of the decoded image of the 1st frame.

[0051] Optionally, the reconstruction information of the first decoded image includes the reconstructed first decoded image, or image features of the reconstructed first decoded image.

[0052] In the case where the reconstruction information of the first decoded image includes the reconstructed first decoded image, the decoding end stores the reconstructed first decoded image, so that the decoding end can directly determine the reconstruction information of the first decoded image.

[0053] Optionally, the decoding end can also reconstruct the first decoded image, that is, obtain the reconstructed motion information of the first decoded image based on the code stream, and then determine the target prediction information of the first decoded image based on the global reference tensor corresponding to the first decoded image and the reconstructed motion information of the first decoded image; obtain the reconstructed residual information of the first decoded image based on the code stream; reconstruct the first decoded image based on the target prediction information of the first decoded image and the reconstructed residual information of the first decoded image to obtain a reconstructed image of the first decoded image.

[0054] If the reconstruction information of the first decoded image includes a reconstructed first decoded image, the decoding terminal can reconstruct the first decoded image to obtain a reconstructed first decoded image, and determine image features of the reconstructed first decoded image based on the reconstructed first decoded image.

[0055] Optionally, based on the reconstruction information of the first decoded image, the global reference tensor corresponding to the first decoded image is updated to obtain the global reference tensor corresponding to the current image. The implementation process includes: inputting the reconstruction information of the first decoded image and the global reference tensor corresponding to the first decoded image into the global reference tensor update network to obtain the global reference tensor corresponding to the current image output by the global reference tensor update network.

[0056] Optionally, based on the global reference tensor corresponding to the current image and the reconstructed motion information of the current image, the global prediction information corresponding to the current image is determined, and based on the global prediction information corresponding to the current image, the target prediction information of the current image is determined.

[0057] Optionally, the decoding end determines the reconstructed motion information of the current image based on the reconstructed motion information of the current image, and determines the global motion information corresponding to the current image based on the reconstructed motion information of the current image, where the global motion information is used to describe the position offset of the current image relative to the global reference tensor corresponding to the current image, and determines the global prediction information corresponding to the current image based on the global reference tensor corresponding to the current image and the global motion information corresponding to the current image.

[0058] Optionally, the decoding end may update the global motion information corresponding to the first decoded image based on the reconstructed motion information of the current image to obtain the global motion information corresponding to the current image.

[0059] Optionally, the decoding end may directly determine the global prediction information corresponding to the current image as the target prediction information of the current image.

[0060] Optionally, the decoding end determines the local prediction information corresponding to the current image based on the reconstructed motion information of the current image and the reconstruction information of the first decoded image, and determines the target prediction information of the current image based on the local prediction information corresponding to the current image and the global prediction information corresponding to the current image.

[0061] Optionally, if the reconstruction information of the first decoded image includes image features of the reconstructed first decoded image, in this case, the decoding end determines the image features of the reconstructed image of the current image based on the target prediction information of the current image and the reconstruction residual information of the current image according to relevant algorithms; and reconstructs the current image based on the image features of the reconstructed image of the current image according to relevant algorithms to obtain the reconstructed image of the current image.

[0062] Optionally, the decoding end inputs the target prediction information of the current image and the reconstruction residual information of the current image into the reconstruction network to obtain the image features of the reconstructed image of the current image output by the reconstruction network.

[0063] Optionally, if the reconstruction information of the first decoded image includes a reconstructed first decoded image, in this case, the decoding end can reconstruct the current image based on the target prediction information of the current image and the reconstruction residual information of the current image to obtain a reconstructed image of the current image.

[0064] In a third aspect, a coding device is provided, wherein the coding device has the function of implementing the coding method described in the first aspect. The coding device includes at least one module, wherein the at least one module is configured to implement the coding method described in the first aspect.

[0065] In a fourth aspect, a decoding device is provided, wherein the decoding device has the function of implementing the decoding method described in the first aspect. The decoding device includes at least one module, wherein the at least one module is configured to implement the decoding method described in the first aspect.

[0066] In a fifth aspect, a coding device is provided, comprising: a processor, the processor being coupled to a memory, the memory being used to store programs or instructions, and when the program or instructions are executed by the processor, the coding device executes the coding method described in the first aspect above.

[0067] In the sixth aspect, a decoding device is provided, comprising: a processor, the processor being coupled to a memory, the memory being used to store programs or instructions, and when the program or instructions are executed by the processor, the decoding device executes the decoding method described in the second aspect above.

[0068] In a seventh aspect, a coding and decoding system is provided, which includes the coding device described in the fifth aspect and / or the decoding device described in the sixth aspect.

[0069] In an eighth aspect, a computer-readable storage medium is provided, comprising a program code, wherein when the program code is run on a computer, the computer executes the method described in the first aspect.

[0070] In a ninth aspect, a computer-readable storage medium is provided, comprising a program code, wherein when the program code is run on a computer, the computer executes the method described in the second aspect.

[0071] In a tenth aspect, a computer program product is provided, comprising instructions, which, when executed on a computer, cause the computer to execute the method described in the first aspect.

[0072] In an eleventh aspect, a computer program product is provided, comprising instructions, which, when executed on a computer, cause the computer to execute the method described in the second aspect.

[0073] In a twelfth aspect, a computer-readable storage medium is provided, on which a code stream obtained by the method described in the first aspect above and executed by one or more processors is stored.

[0074] In the thirteenth aspect, a device for storing a code stream is provided, characterized in that it includes at least one storage medium and a communication interface; the communication interface is used to receive or send the code stream; the at least one storage medium is used to store the code stream; the code stream is encoded by the encoder according to the encoding method described in the first aspect above.

[0075] In a fourteenth aspect, a method for storing a code stream is provided, comprising: receiving a code stream through a communication interface; and storing the code stream in one or more storage media, wherein the code stream is encoded by an encoder according to the encoding method described in the first aspect.

[0076] In a fifteenth aspect, a system for distributing a bitstream is provided, comprising at least one storage medium and a video streaming device; the at least one storage medium is used to store the bitstream, the bitstream being encoded by an encoder according to the encoding method described in the first aspect;

[0077] The video stream device is configured to respond to a request from a decoder so that the target code stream in the at least one storage medium is sent to the decoder.

[0078] In a sixteenth aspect, a method for distributing a code stream is provided, comprising: receiving a first request; selecting a target code stream from at least one storage medium in response to the first request; and sending the target code stream to a destination device; the at least one storage medium is used to store the code stream, wherein the code stream is encoded by an encoder according to the encoding method described in the first aspect above.

[0079] In the seventeenth aspect, a system for processing a code stream is provided, characterized in that it includes an image source device, an encoder device, one or more storage media and a destination device; the image source device is used to provide image data; the encoder device is used to obtain the image data of the image source device through an interface, and encode the image data to obtain one or more code streams, wherein the code streams are encoded by the encoder according to the encoding method described in the first aspect above; the encoder device is used to store the one or more code streams in one or more storage media; or, the encoder device is used to encapsulate the one or more code streams to obtain a transmission code stream; the encoder device is used to transmit the transmission code stream to the destination device via a communication link or a communication network; the destination device is used to decapsulate the transmission code stream to obtain the one or more code streams; and the destination device is used to decode the one or more code streams to obtain decoded data.

[0080] The technical effects obtained in the above-mentioned second to seventeenth aspects are similar to the technical effects obtained by the corresponding technical means in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] FIG1 is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0082] FIG2 is a schematic diagram of another implementation environment provided by an embodiment of the present application;

[0083] FIG3 is a flowchart of an encoding method provided in an embodiment of the present application;

[0084] FIG4 is a schematic diagram of the structure of an image feature extraction network provided in an embodiment of the present application;

[0085] FIG5 is a schematic diagram of the structure of a ResBlock subnetwork provided in an embodiment of the present application;

[0086] FIG6 is a schematic diagram of the structure of a global reference tensor update network provided in an embodiment of the present application;

[0087] FIG7 is a schematic diagram of the structure of a global motion information update network provided in an embodiment of the present application;

[0088] FIG8 is a flowchart of a decoding method provided in an embodiment of the present application;

[0089] FIG9 is a schematic diagram of the structure of a reconstruction network provided in an embodiment of the present application;

[0090] FIG10 is a flowchart of a coding and decoding method provided in an embodiment of the present application;

[0091] FIG11 is a flowchart of another encoding and decoding method provided in an embodiment of the present application;

[0092] FIG12 is a schematic diagram of a test result provided in an embodiment of the present application;

[0093] FIG13 is a flowchart of another encoding and decoding method provided in an embodiment of the present application;

[0094] FIG14 is a schematic structural diagram of an encoding device provided in an embodiment of the present application;

[0095] FIG15 is a schematic structural diagram of a decoding device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0096] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0097] For ease of understanding, before explaining in detail the encoding and decoding method provided in the embodiment of the present application, the nouns, application scenarios and implementation environment involved in the embodiment of the present application are first introduced.

[0098] First, the nouns involved in the embodiments of the present application are introduced.

[0099] Video compression: includes intra-frame prediction coding and inter-frame prediction coding. Intra-frame prediction coding does not require the use of reference images, while inter-frame prediction coding needs to use the current image and the reference image to determine the inter-frame motion information, and use the inter-frame motion information to compress the video.

[0100] Group of Pictures (GOP): A video stream consists of multiple GOPs. A GOP is a group of consecutive pictures and is the basic unit of video image access by encoders and decoders. Each GOP contains I-frames and P-frames.

[0101] Reference picture: In video compression, a reference picture is an already coded or decoded picture that is used to encode or decode the current picture.

[0102] I-frames: I-frames are typically encoded using intra-frame prediction and are also called keyframes. I-frames are compressed without reference to other frames. They describe the image background and moving subjects in detail, and during decoding, the complete image can be reconstructed using only I-frame data. I-frames are typically the first frame in each GOP.

[0103] P-frames: P-frames are typically encoded using inter-frame prediction, also known as forward-predicted frames (forward reference frames). They represent the difference between a P-frame and the previous I-frame (or P-frame). P-frames use motion compensation to transmit the prediction residual and motion vector between the frame and the previous reference image (i.e., I- or P-frame).

[0104] Inter-frame prediction coding: It mainly includes two parts, one is the inter-frame prediction part, and the other is the residual compression part. The inter-frame prediction part includes a motion information prediction and compression module, and a transformation module. In some related technologies, motion information is reflected as optical flow. During the encoding process, the images of the reference frame and the current frame are input into the optical flow estimation network to obtain the predicted optical flow, and the optical flow is compressed. In other related technologies, motion information is reflected as motion features. During the encoding process, the image features of the current frame and the reference frame are extracted, and the image features of the current frame and the reference frame are input into the convolutional neural network to obtain the predicted motion features, and the motion features are compressed. The transformation module usually adopts a wrap operation. During the encoding process, the inter-frame edge information is used to transform the reference frame into the prediction result of the current frame.

[0105] In the embodiment of the present application, the above-mentioned motion information (also called motion vector) can be optical flow or motion feature, and the embodiment of the present application does not limit this.

[0106] IPPP encoding mode: A video compression encoding mode in which the first frame is an I-frame and subsequent frames are P-frames. P-frames use only one forward reference frame. For example, for the current image (i.e., frame t), the reference image for frame t is usually frame t-1.

[0107] Bit rate: In image compression, it refers to the encoding length required to encode a unit pixel. The higher the bit rate, the larger the transmitted file size and the better the image reconstruction quality.

[0108] RD-λ: R refers to the bitrate; D refers to the reconstruction quality; λ is a network parameter used to adjust the bitrate. There is usually a corresponding relationship between λ and R.

[0109] Rate-distortion curve (RD-Curve): The horizontal axis is the bit rate and the vertical axis is the PSNR. Generally, the higher the curve is, the better the codec performance.

[0110] Next, the application scenarios involved in the embodiments of this application are introduced.

[0111] Video compression is a technology that uses redundant information in videos to represent the original video with less data (video bitstream). This reduces the pressure on network bandwidth required for video storage and transmission. Video compression involves encoding and decoding, with encoding and decoding performance (reflecting video quality) and encoding and decoding efficiency (reflecting encoding and decoding time) being key considerations. With the development of society and advancements in technology, video content has accounted for over 80% of consumer internet traffic in recent years, and the file size of original videos has continued to grow, posing significant challenges to both encoding and decoding performance and efficiency.

[0112] In end-to-end video compression methods, a neural network model is trained using a single loss function to balance distortion and bitrate, and compression performance is improved through a multi-reference video frame mechanism. However, the number of reference video frames in this multi-reference video frame mechanism remains relatively fixed, meaning that after the neural network model is trained, the number of reference video frames is fixed, and the number and position of reference video frames cannot be freely adjusted or controlled. Furthermore, as the number of reference video frames increases, the computing power and cache size requirements for the device also increase.

[0113] Based on this, an embodiment of the present application provides a coding and decoding method that can determine the target prediction information of the current image through the global reference tensor corresponding to the current image and the motion information of the current image. Since the global reference tensor is determined based on multiple encoded images before the current image, that is, the embodiment of the present application can generate a global reference tensor based on multiple encoded images before the current image, so that the global reference tensor can represent the image information that can be used for reference by the current image in multiple encoded images before the current image, and has globality across the time domain. In this way, it can be ensured that the target prediction information corresponding to the current image obtained based on the global reference tensor corresponding to the current image and the motion information of the current image is more accurate, so that the data bit amount of the residual information corresponding to the current image determined subsequently is smaller, thereby effectively improving the efficiency of coding and decoding. In addition, by generating a global reference tensor based on multiple encoded images before the current image, the encoding end can ensure that the current image has more information for reference by the current image when encoding without having to cache multiple encoded images, thereby greatly reducing the cache pressure of the encoding end on the basis of improving coding and decoding efficiency. During the decoding process, accurate target prediction information can be obtained through the reconstructed motion information of the current image and the global reference tensor corresponding to the current image. In this way, it can be ensured that the subsequent reconstructed image of the current image based on the target prediction information and the reconstructed residual information corresponding to the current image has better image quality, thereby greatly improving the encoding and decoding performance.

[0114] Next, the implementation environment involved in the embodiments of this application is introduced.

[0115] Please refer to Figure 1, which is a schematic diagram of an implementation environment provided by an embodiment of the present application. The implementation environment includes a source device 10, a destination device 20, a link 30, and a storage device 40. Source device 10 can generate an encoded image, i.e., a code stream. Therefore, source device 10 can also be referred to as an encoding device. Destination device 20 can decode the code stream generated by source device 10. Therefore, destination device 20 can also be referred to as a decoding device. Link 30 can receive the encoded image generated by source device 10 and transmit the encoded image to destination device 20. Storage device 40 can receive the encoded image generated by source device 10 and store the encoded image. In this case, destination device 20 can directly obtain the encoded image from storage device 40. Alternatively, storage device 40 can correspond to a file server or another intermediate storage device that can store the encoded image generated by source device 10. In this case, destination device 20 can stream or download the encoded image stored by storage device 40.

[0116] The source device 10 and the destination device 20 may each include one or more processors and a memory coupled to the one or more processors, wherein the memory may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, any other medium that can be used to store desired program code in the form of instructions or data structures accessible by a computer, etc. For example, the source device 10 and the destination device 20 may each include a mobile phone, a smartphone, a personal digital assistant (PDA), a wearable device, a pocket PC (PPC), a tablet computer, a smart car computer, a smart TV, a smart speaker, a desktop computer, a mobile computing device, a notebook (e.g., laptop) computer, a tablet computer, a set-top box, a telephone handset such as a so-called "smart" phone, a television, a camera, a display device, a digital media player, a video game console, an in-vehicle computer, or the like.

[0117] Link 30 may include one or more media or devices capable of transmitting encoded images from source device 10 to destination device 20. In one possible implementation, link 30 may include one or more communication media that enable source device 10 to send encoded images directly to destination device 20 in real time. In an embodiment of the present application, source device 10 may modulate the encoded images based on a communication standard, such as a wireless communication protocol, and may transmit the modulated images to destination device 20. The one or more communication media may include wireless and / or wired communication media, such as radio frequency (RF) spectrum or one or more physical transmission lines. The one or more communication media may form part of a packet-based network, such as a local area network, a wide area network, or a global network (e.g., the Internet). The one or more communication media may include routers, switches, base stations, or other devices that facilitate communication from source device 10 to destination device 20, although this embodiment of the present application does not specifically limit this.

[0118] In one possible implementation, the storage device 40 may store the received encoded image sent by the source device 10, and the destination device 20 may directly obtain the encoded image from the storage device 40. Under such conditions, the storage device 40 may include any of a variety of distributed or locally accessible data storage media, for example, any of the various distributed or locally accessible data storage media may be a hard disk drive, a Blu-ray disc, a digital versatile disc (DVD), a compact disc read-only memory (CD-ROM), a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing a code stream.

[0119] In one possible implementation, storage device 40 may correspond to a file server or another intermediate storage device that can store the codestream generated by source device 10. Destination device 20 may stream or download the images stored on storage device 40. The file server may be any type of server capable of storing and transmitting encoded images to destination device 20. In one possible implementation, the file server may include a network server, a file transfer protocol (FTP) server, a network attached storage (NAS) device, or a local disk drive. Destination device 20 may obtain the encoded images via any standard data connection, including an internet connection. Any standard data connection may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL), a cable modem, etc.), or a combination of both suitable for obtaining encoded images stored on a file server. The transmission of the encoded images from storage device 40 may be streaming, downloading, or a combination of both.

[0120] The implementation environment shown in FIG1 is only one possible implementation method, and the technology of the embodiment of the present application is applicable not only to the source device 10 that can encode images and the destination device 20 that can decode encoded images shown in FIG1 , but also to other devices that can encode images and decode code streams, and the embodiment of the present application does not specifically limit this.

[0121] In the implementation shown in FIG1 , source device 10 includes a data source 120, an encoder 100, and an output interface 140. In some embodiments, output interface 140 may include a modem and / or a transmitter, where the transmitter may also be referred to as a transmitter. Data source 120 may include an image capture device (e.g., a camera), an archive containing previously captured images, a feed interface for receiving images from an image content provider, and / or a computer graphics system for generating images, or a combination of these sources of images.

[0122] The data source 120 may send an image to the encoder 100, and the encoder 100 may encode the image received from the data source 120 to generate an encoded image. The encoder may send the encoded image to an output interface. In some embodiments, the source device 10 directly sends the encoded image to the destination device 20 via the output interface 140. In other embodiments, the encoded image may also be stored on the storage device 40 for later retrieval by the destination device 20 for decoding and / or display.

[0123] In the implementation environment shown in FIG1 , the destination device 20 includes an input interface 240, a decoder 200, and a display device 220. In some embodiments, the input interface 240 includes a receiver and / or a modem. The input interface 240 may receive encoded images via the link 30 and / or from the storage device 40, and then transmit the encoded images to the decoder 200. The decoder 200 may decode the received encoded images to obtain decoded images. The decoder may transmit the decoded images to the display device 220. The display device 220 may be integrated with the destination device 20 or may be external to the destination device 20. Generally, the display device 220 displays the decoded images. The display device 220 may be any of a variety of types of display devices, for example, the display device 220 may be a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or other types of display devices.

[0124] Although not shown in FIG1 , in some aspects, the encoder 100 and decoder 200 can be integrated with an encoder and decoder, respectively, and can include appropriate multiplexer-demultiplexer (MUX-DEMUX) units or other hardware and software for encoding both audio and video in a common data stream or in separate data streams. In some embodiments, the MUX-DEMUX units can conform to the ITU H.223 multiplexer protocol, or other protocols such as the User Datagram Protocol (UDP), if applicable.

[0125] The encoder 100 and the decoder 200 can each be any of the following circuits: one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, or any combination thereof. If the technology of the embodiments of the present application is implemented in part by software, the device can store instructions for the software in a suitable non-volatile computer-readable storage medium, and can use one or more processors to execute the instructions in hardware to implement the technology of the embodiments of the present application. Any of the foregoing (including hardware, software, a combination of hardware and software, etc.) can be regarded as one or more processors. Each of the encoder 100 and the decoder 200 can be included in one or more encoders or decoders, and any of the encoders or decoders can be integrated as part of a combined encoder / decoder (encoder / decoder) in the corresponding device.

[0126] Embodiments of the present application may generally refer to encoder 100 as "signaling" or "sending" certain information to another device, such as decoder 200. The terms "signaling" or "sending" may generally refer to the transmission of syntax elements and / or other data used to decode a compressed image. This transmission may occur in real time or near real time. Alternatively, this communication may occur over time, such as when the syntax elements are stored in the encoded bitstream to a computer-readable storage medium during encoding, and the decoding device may then retrieve the syntax elements at any time after they are stored to this medium.

[0127] Figure 2 is a schematic diagram of another implementation environment provided by an embodiment of the present application. The implementation environment includes an encoding end and a decoding end. The encoding end includes an AI encoding module, an entropy encoding module, and a file saving module, and the decoding end includes a file loading module, an entropy decoding module, and an AI decoding module.

[0128] During the compression process, the encoder obtains the image to be compressed (for example, an image in a video captured by a camera), obtains the residual information and motion information of the current image through the AI ​​encoding module, and then performs entropy encoding on the residual information and motion information of the current image through the entropy encoding module to obtain a bitstream file. The bitstream file is saved through the file saving module to obtain a compressed file. The compressed file is input to the decoder, which loads the compressed file through the file loading module and obtains the reconstructed image through the entropy decoding module and the AI ​​decoding module.

[0129] Optionally, the data processing process of the AI ​​encoding module and the AI ​​decoding module is implemented on an embedded neural network processing unit (NPU) to improve data processing efficiency, and the processes such as entropy coding, saving files, and loading files are implemented on a central processing unit (CPU).

[0130] Optionally, the encoding end and the decoding end are one device, or the encoding end and the decoding end are two independent devices. If the encoding end and the decoding end are one device, the device can encode the image by the encoding method provided in the embodiment of the present application, and can also decode the image by the decoding method provided in the embodiment of the present application. If the encoding end and the decoding end are two independent devices, the encoding method provided in the embodiment of the present application can be applied to the encoding end of the two devices, and the decoding method provided in the embodiment of the present application can be applied to the decoding end of the two devices. That is, for one device, the device has both an image compression function and an image decompression function, or the device has an image compression function or an image decompression function.

[0131] The encoding and decoding method provided in the embodiments of the present application can be applied to a variety of scenarios, such as cloud storage, video surveillance, live broadcast, transmission and other business scenarios, and can be specifically applied to terminal recording, video albums, cloud storage, etc. The images encoded and decoded in various scenarios can be images included in the video file. It should be noted that, in conjunction with the implementation environment shown in Figure 1, any of the encoding methods below can be executed by the encoder 100 in the source device 10. Any of the decoding methods below can be executed by the decoder 200 in the destination device 20. In conjunction with the implementation environment shown in Figure 2, any of the encoding methods below can be executed by the encoding end. Any of the decoding methods below can be executed by the decoding end.

[0132] FIG3 is a flow chart of an encoding method provided in an embodiment of the present application, which is applied to an encoding end. Referring to FIG3 , the method includes the following steps.

[0133] Step 301: Determine a global reference tensor corresponding to a current image and motion information of the current image, wherein the global reference tensor is determined based on a plurality of encoded images before the current image.

[0134] In some embodiments, if the current image is in a video, the video includes at least one GOP, and the encoder can determine that the global reference tensor corresponding to the current image is 0, that is, the global reference tensor corresponding to the I frame is a feature of all 0s.

[0135] When the current image is an I frame in the Nth GOP in the video, or a P frame in the video, the encoder can determine the global reference tensor corresponding to the current image and the motion information of the current image, where N is an integer greater than 1.

[0136] That is, when the current image is an I-frame in the first GOP of the video, the encoder determines that the global reference tensor corresponding to the current image is 0, that is, the global reference tensor corresponding to the I-frame is a feature of all 0s. When the current image is an I-frame in the Nth GOP of the video, the encoder determines the global reference tensor corresponding to the current image and the motion information of the current image, or directly determines that the global reference tensor corresponding to the current image is 0.

[0137] It should be noted that when the current image is an I-frame in a GOP, if the encoder determines the global reference tensor corresponding to the current image to be 0, that is, regardless of which GOP the current image is in, the encoder determines the global reference tensor corresponding to the current image to be 0. In this case, the global reference tensor is determined based on multiple encoded images before the current image, and these multiple encoded images are the images encoded in the GOP where the current image is located. In other words, the global reference tensor is updated in units of GOPs. The global reference tensor is updated within each GOP and reset to 0 when crossing GOPs.

[0138] If the encoder determines the global reference tensor corresponding to the current image as 0 only when the current image is an I frame in the first GOP in the video, and when the current image is an I frame in the Nth GOP in the video, the encoder determines the global reference tensor corresponding to the current image and the motion information of the current image. In this case, the global reference tensor corresponding to the current image is determined based on all encoded images before the current image. At this time, the global reference tensor is not limited to one or several encoded images, but can represent all information that can be used for reference in multiple encoded images before the current image, thereby increasing the information available for reference for the current image and further improving the efficiency of encoding and decoding.

[0139] Optionally, when the current image is an I frame in a GOP, the encoding end can encode the current image according to a related I frame encoding technique. As an example, the encoding end can encode the current image according to an intra-frame prediction encoding technique.

[0140] In other embodiments, if the current image is in a video, and the current image is the first frame in the video, the encoder can determine that the global reference tensor corresponding to the current image is 0, that is, the global reference tensor corresponding to the first frame is a feature of all 0s. When the current image is the Mth frame in the video, the encoder can determine the global reference tensor corresponding to the current image and the motion information of the current image, where M is an integer greater than 1.

[0141] Optionally, when the current image is the first frame image in a GOP, the encoding end can encode the current image according to a related I-frame encoding technique. As an example, the encoding end can encode the current image according to an intra-frame prediction encoding technique.

[0142] Next, the implementation method of determining the global reference tensor corresponding to the current image and the implementation method of determining the motion information of the current image will be introduced respectively through (1)-(2).

[0143] (1) Determine the global reference tensor corresponding to the current image.

[0144] In some embodiments, the encoding end determines the reconstruction information of the first encoded image, where the first encoded image is an encoded image adjacent to the current image. Based on the reconstruction information of the first encoded image, the global reference tensor corresponding to the first encoded image is updated to obtain the global reference tensor corresponding to the current image.

[0145] It should be noted that the first coded image is usually the previous coded image adjacent to the current image. For example, if the current image is the tth frame of the video, the first coded image is the t-1th frame.

[0146] Since the decoding end needs to update the global reference tensor corresponding to the first decoded image in the subsequent steps, the first decoded image is a decoded image adjacent to the current image, and the first decoded image of the decoding end and the first encoded image of the encoding end are the same image. For the convenience of description, the first decoded image and the first encoded image are subsequently referred to as the first image. When the decoding end performs decoding, it is unable to obtain the original first image of the encoding end, and can only obtain the reconstructed first image. That is to say, the first image reconstructed by the decoding end may have a certain deviation relative to the original first image. Therefore, the encoding end updates the global reference tensor corresponding to the first image through the reconstruction information of the first image to ensure that the global reference tensor corresponding to the current image subsequently used by the encoding end and the decoding end is the same, so that the target prediction information of the current image obtained by the encoding end and the decoding end is more accurate, thereby effectively improving the efficiency and performance of encoding and decoding.

[0147] For example, if the current image is in a video that includes at least one GOP, if the current image is the first P frame in the GOP, the first coded image may be an I frame in the GOP. If the current image is the second P frame in the GOP, the first coded image may be the first P frame in the GOP.

[0148] For example, if the current image is in a video, and the encoder determines that the global reference tensor corresponding to the current image is 0 when the current image is the first frame in the video, and determines the global reference tensor corresponding to the current image and the motion information of the current image when the current image is the Mth frame in the video. In this case, the global reference tensor corresponding to the image of the Nth frame in the video code stream is obtained based on the global reference tensor corresponding to the encoded image of the N-1th frame and the reconstruction information of the encoded image of the N-1th frame, and the global reference tensor corresponding to the second frame is the reconstruction information of the encoded image of the 1st frame.

[0149] In some embodiments, the reconstruction information of the first encoded image includes the reconstructed first encoded image, or image features of the reconstructed first encoded image. In different situations, the implementation method for determining the reconstruction information of the first encoded image is different, and each will be described below.

[0150] If the reconstruction information of the first coded image includes a reconstructed first coded image, the encoding end can reconstruct the first coded image to obtain a reconstructed image of the first coded image, that is, obtain reconstructed motion information of the first coded image based on the code stream, and then determine target prediction information of the first coded image based on a global reference tensor corresponding to the first coded image and the reconstructed motion information of the first coded image; obtain reconstructed residual information of the first coded image based on the code stream; and reconstruct the first coded image based on the target prediction information of the first coded image and the reconstructed residual information of the first coded image to obtain a reconstructed image of the first coded image.

[0151] The implementation method of determining the target prediction information of the first encoded image based on the global reference tensor corresponding to the first encoded image and the reconstructed motion information of the first encoded image is similar to the implementation method of determining the target prediction information of the current image based on the global reference tensor corresponding to the current image and the reconstructed motion information of the current image in the subsequent encoding method; the implementation method of obtaining the reconstructed image of the first encoded image based on the target prediction information of the first encoded image and the reconstructed residual information of the first encoded image is similar to the implementation method of obtaining the reconstructed image of the current image based on the target prediction information of the current image and the reconstructed residual information of the current image in the subsequent decoding method. The details will be introduced later and will not be elaborated here.

[0152] If the reconstruction information of the first encoded image includes a reconstructed first encoded image, the encoding end can reconstruct the first encoded image to obtain a reconstructed image of the first encoded image, and determine image features of the reconstructed first encoded image based on the reconstructed image of the first encoded image.

[0153] In some embodiments, the encoding end can extract image features from the reconstructed image of the first encoded image according to relevant feature extraction technology to obtain image features of the reconstructed first encoded image.

[0154] Optionally, the encoding end inputs the reconstructed image of the first encoded image into an image feature extraction network to obtain image features of the reconstructed first encoded image output by the image feature extraction network.

[0155] For example, FIG4 is a schematic diagram of the structure of an image feature extraction network provided by an embodiment of the present application. The image feature extraction network includes a convolutional layer (Conv) and three consecutive ResBlock sub-networks with the same configuration. in , h in , w in ] indicates that the number of channels of the input tensor of the image feature extraction network is c in , the size of the input tensor of the image feature extraction network in the x-axis direction is h in , the size of the input tensor of the image feature extraction network in the y-axis direction is w in ;Conv(ks,c in , c out , 2) indicates that the convolution kernel size of the convolution layer is ks, and the number of channels of the input tensor of the convolution layer is c in , the number of channels of the output tensor of the convolutional layer is c out , the step length of the convolution kernel movement is 2. ResBlock(c out ) indicates that the number of channels of the output tensor of the ResBlock sub-network is c out . [c out , h in / 2,w in / 2] indicates that the number of channels of the output tensor of the image feature extraction network is c out， The output tensor of the image feature extraction network has a size of h in the x-axis direction. in / 2, the size of the output tensor of the image feature extraction network in the y-axis direction is w in / 2. In this case, the input tensor of the image feature extraction network is the reconstructed image of the first encoded image, and the output tensor of the image feature extraction network is the image feature of the reconstructed first encoded image.

[0156] For example, Figure 5 is a schematic diagram of the structure of a ResBlock subnetwork provided in an embodiment of the present application. The ResBlock subnetwork includes two convolutional layers (Conv) and an activation layer (such as an activation layer built based on Relu, LRelu, or other activation functions). Among them, the convolution kernel size of the convolution layer in Conv(3, c, c, 1) is 3×3, the number of channels of the input tensor of the convolution layer and the number of channels of the output tensor are both c, and the step size of the convolution kernel movement is 1.

[0157] It should be noted that all the network structure diagrams shown in the embodiments of the present application are only examples. In actual applications, other structures may be possible. The embodiments of the present application do not limit the structure and dimensions of the network.

[0158] In some embodiments, based on the reconstruction information of the first encoded image, the global reference tensor corresponding to the first encoded image is updated to obtain the global reference tensor corresponding to the current image. The implementation process includes: inputting the reconstruction information of the first encoded image and the global reference tensor corresponding to the first encoded image into the global reference tensor update network to obtain the global reference tensor corresponding to the current image output by the global reference tensor update network.

[0159] For example, FIG6 is a schematic diagram of the structure of a global reference tensor update network provided by an embodiment of the present application. The global reference tensor update network includes three convolutional layers (Conv) and three consecutive ResBlock sub-networks with the same configuration. The number of channels of the reconstructed information of the first coded image is 3, the size of the reconstructed information of the first coded image in the x-axis direction is h, and the size of the reconstructed information of the first coded image in the y-axis direction is w. Conv(ks,3,c,2) indicates that the convolution layer has a convolution kernel size of ks, the number of channels of the input tensor of the convolution layer is 3, the number of channels of the output tensor of the convolution layer is c, and the stride of the convolution kernel is 2. The number of channels of the global reference tensor corresponding to the first encoded image is c, and the size of the global reference tensor corresponding to the first encoded image in the x-axis direction is h ref The size of the global reference tensor corresponding to the first encoded image in the y-axis direction is w refConv(ks, c×2, c, 2) indicates that the convolution kernel size of the convolution layer is ks, the number of channels of the input tensor of the convolution layer is c×2, the number of channels of the output tensor of the convolution layer is c, and the stride length of the convolution kernel movement is 2. ResBlock(c) indicates that the number of channels of the output tensor of the ResBlock subnetwork is c. The structure of the ResBlock subnetwork is shown in Figure 5. Conv(ks, c, c, 2) indicates that the convolution kernel size of the convolution layer is ks, the number of channels of the input tensor of the convolution layer and the number of channels of the output tensor of the convolution layer are both c, and the stride length of the convolution kernel movement is 2. Indicates that the number of channels of the global reference tensor corresponding to the current image is c, and the size of the global reference tensor corresponding to the current image in the x-axis direction is h ref , the size of the global reference tensor corresponding to the current image in the y-axis direction is w ref .

[0160] (2) Determine the motion information of the current image.

[0161] The motion information of the current image includes a motion vector, which is used to indicate the offset of a target block in the current image and in a reference image for the current image, where the reference image is a reconstructed image of a coded image adjacent to the current image. In this case, the reference image for the current image is the reconstructed image of the first coded image, or in other words, the reference image for the current image is the reconstructed first coded image.

[0162] It should be noted that, in the embodiment of the present application, the reconstructed first coded image and the reconstructed image of the first coded image have the same meaning.

[0163] In some embodiments, the current image includes at least one target block, which may be a coding unit (CU) in the HEVC standard. In other embodiments, the target block may be a pixel or a set of multiple pixels in the current image. The embodiments of the present application do not limit the specific composition of the target block.

[0164] It should also be noted that if the motion information of the current image is determined through the network model, the current image needs to be input into the network model. In this case, the encoder can convert the current image into a tensor corresponding to the current image, and then input the tensor corresponding to the current image into the network model. At this time, the target block can also be an element in the tensor corresponding to the current image or a set of multiple elements.

[0165] Based on the above description, the reconstructed information of the first coded image includes the reconstructed first coded image or the image features of the reconstructed first coded image. In different situations, the implementation methods for determining the motion information of the current image are different, which will be introduced below.

[0166] When the reconstruction information of the first encoded image includes the reconstructed first encoded image, the encoding end may determine the motion information of the current image based on the current image and the reconstructed first encoded image according to a related motion estimation algorithm such as optical flow estimation.

[0167] In the case where the reconstruction information of the first encoded image includes image features of the reconstructed first encoded image, the encoding end can determine the motion information of the current image based on the image features of the current image and the image features of the reconstructed first encoded image according to relevant motion estimation algorithms such as optical flow estimation.

[0168] In some embodiments, before determining the motion information of the current image based on the image features of the current image and the image features of the reconstructed first encoded image, the encoding end can extract image features from the current image according to relevant feature extraction technology to obtain the image features of the current image.

[0169] Optionally, the encoding end inputs the current image into an image feature extraction network to obtain image features of the current image output by the image feature extraction network.

[0170] For example, the image feature extraction network can be as shown in FIG4 . In this case, the input tensor of the image feature extraction network is the above-mentioned current image, and the output tensor of the image feature extraction network is the image feature of the current image.

[0171] To summarize, when the reconstruction information of the first coded image includes the reconstructed first coded image, the motion vector is used to indicate the offset of the target block in the current image and in the reconstructed first coded image; and when the reconstruction information of the first coded image includes the image features of the reconstructed first coded image, the motion vector is used to indicate the offset of the target block in the image features of the current image and in the image features of the reconstructed first coded image.

[0172] Step 302: Determine target prediction information of the current image based on the global reference tensor corresponding to the current image and the motion information of the current image.

[0173] Next, through (1)-(2), the implementation method of determining the target prediction information of the current image based on the global reference tensor corresponding to the current image and the motion information of the current image will be introduced in detail.

[0174] (1) Determine global prediction information corresponding to the current image based on a global reference tensor corresponding to the current image and motion information of the current image.

[0175] In some embodiments, the encoding end is able to determine the reconstructed motion information of the current image based on the motion information of the current image, and determine the global motion information corresponding to the current image based on the reconstructed motion information of the current image, the global motion information including a global motion vector, which indicates the offset of the target block in the current image and the global reference tensor corresponding to the current image, and determine the global prediction information corresponding to the current image based on the global reference tensor corresponding to the current image and the global motion information corresponding to the current image.

[0176] Optionally, the encoding end can encode the motion information of the current image into a bitstream, and then parse the bitstream to obtain the reconstructed motion information of the current image.

[0177] Similarly to the above, since the decoding end also needs to determine the global motion information corresponding to the current image in the subsequent steps, the decoding end cannot obtain the original motion information of the current image during decoding, and can only obtain the reconstructed motion information of the current image. That is to say, the motion information reconstructed by the decoding end may have a certain deviation compared with the motion information corresponding to the original current image. Therefore, the encoding end determines the global motion information corresponding to the current image through the reconstructed motion information of the current image to ensure that the global motion information corresponding to the current image subsequently used by the encoding end and the decoding end is the same, so that the encoding end and the decoding end obtain the target prediction information of the current image more accurately, thereby effectively improving the efficiency and performance of encoding and decoding.

[0178] In some embodiments, the encoding end updates the global motion information corresponding to the first encoded image based on the reconstructed motion information of the current image to obtain the global motion information corresponding to the current image.

[0179] Optionally, the encoding end inputs the reconstructed motion information of the current image and the global motion information corresponding to the first encoded image into the global motion information update network to obtain the global motion information corresponding to the current image output by the global motion information update network.

[0180] For example, FIG7 is a schematic diagram of the structure of a global motion information update network provided by an embodiment of the present application. The global motion information update network includes three convolutional layers (Conv) and three consecutive ResBlock sub-networks with the same configuration. The number of channels representing the reconstructed motion information of the current image is c m The size of the reconstructed motion information of the current image in the x-axis direction is h refThe size of the reconstructed information of the first coded image in the y-axis direction is w ref . Conv(ks,3,c m , 1) indicates that the convolution kernel size of the convolution layer is ks, the number of channels of the input tensor of the convolution layer is 3, and the number of channels of the output tensor of the convolution layer is c m , the step size of the convolution kernel movement is 1. The number of channels representing the global motion information corresponding to the first coded image is c m The size of the global motion information corresponding to the first encoded image in the x-axis direction is h ref The size of the global motion information corresponding to the first encoded image in the y-axis direction is w ref . Conv(ks,c m ×2, c m , 2) indicates that the convolution kernel size of the convolution layer is ks, and the number of channels of the input tensor of the convolution layer is c m ×2, the number of channels of the output tensor of the convolutional layer is c m , the step length of the convolution kernel movement is 2. ResBlock(c m ) indicates that the number of channels of the output tensor of the ResBlock sub-network is c m , the structure of the ResBlock sub-network is shown in Figure 5. Conv(ks,c m , c m ,2) indicates that the convolution kernel size of the convolution layer is ks, the number of channels of the input tensor of the convolution layer and the number of channels of the output tensor of the convolution layer are both c m , the step size of the convolution kernel movement is 2. The number of channels representing the global motion information corresponding to the current image is c m The size of the global motion information corresponding to the current image in the x-axis direction is h ref The size of the global motion information corresponding to the current image in the y-axis direction is w ref .

[0181] In some embodiments, the encoding end determines the global prediction information corresponding to the current image based on the global reference tensor corresponding to the current image and the global motion information corresponding to the current image, according to relevant prediction techniques such as optical flow mapping (warping) and deformable convolution. The embodiments of the present application are not limited to this.

[0182] In other embodiments, the encoding end may also directly determine the global prediction information corresponding to the current image based on the global reference tensor corresponding to the current image and the motion information of the current image, according to relevant prediction techniques such as optical flow mapping (warping) and deformable convolution. The embodiments of the present application do not limit this.

[0183] Since the motion information of the current image can characterize the motion of the target block in the current image relative to the encoded image to a certain extent, the encoding end can directly determine the global prediction information corresponding to the current image based on the motion information of the current image and the global reference tensor corresponding to the current image. Compared with determining the global prediction information corresponding to the current image based on the global reference tensor corresponding to the current image and the global motion information corresponding to the current image, this can effectively reduce the computational amount of image compression and improve the efficiency of encoding and decoding while ensuring the accuracy of the subsequently determined target prediction information.

[0184] (2) Based on the global prediction information corresponding to the current image, determine the target prediction information of the current image.

[0185] In some embodiments, the encoder may directly determine the global prediction information corresponding to the current image as the target prediction information of the current image.

[0186] In other embodiments, the encoding end determines the local prediction information corresponding to the current image based on the reconstructed motion information of the current image and the reconstruction information of the first encoded image, and determines the target prediction information of the current image based on the local prediction information corresponding to the current image and the global prediction information corresponding to the current image.

[0187] Optionally, the encoding end determines local prediction information corresponding to the current image based on the reconstructed motion information of the current image and the reconstruction information of the first encoded image according to relevant prediction techniques such as optical flow mapping (warping) and deformable convolution.

[0188] The implementation process of determining the target prediction information of the current image based on the local prediction information corresponding to the current image and the global prediction information corresponding to the current image includes: fusing the local prediction information corresponding to the current image and the global prediction information corresponding to the current image to obtain the target prediction information of the current image.

[0189] It should be noted that, when the reconstruction information of the first coded image includes the reconstructed first coded image, the target prediction information of the current image is the same as the channel dimension and spatial dimension of the current image. When the reconstruction information of the first coded image includes the image features of the reconstructed first coded image, the target prediction information of the current image is the same as the channel dimension and spatial dimension of the image features of the current image.

[0190] In some embodiments, the channel dimensions and spatial dimensions of the target prediction information of the current image, the local prediction information corresponding to the current image, and the global prediction information corresponding to the current image are all the same, wherein the spatial dimension includes the dimension of the first tensor in the x-axis direction and the dimension in the y-direction, and the channel dimension includes the dimension of the first tensor in the z-direction.

[0191] For an element at any position in the target prediction information of the current image of any channel, target first data is determined based on the first data corresponding to the local prediction information and the global prediction information, respectively, and the target first data is used as the value of the element at that position in the target prediction information of the channel, where the first data is the value of the element at the corresponding position in the local prediction information or the global prediction information of the channel. The elements at each position in the target prediction information of each channel are processed in the same manner to obtain the target prediction information.

[0192] Optionally, based on the first data corresponding to the local prediction information and the global prediction information, determining the target first data corresponding to the multiple first data includes: taking a weighted average of the first data corresponding to the local prediction information and the global prediction information as the target first data. Of course, in actual applications, the maximum or minimum value of the first data corresponding to the local prediction information and the global prediction information may also be used as the target first data, and this embodiment of the present application is not limited to this.

[0193] In other embodiments, the encoder may fuse the local prediction information corresponding to the current image with the global prediction information corresponding to the current image using a related feature fusion technique to obtain target prediction information for the current image. For example, the local prediction information corresponding to the current image and the global prediction information corresponding to the current image may be input into a fusion network to obtain the target prediction information for the current image output by the fusion network.

[0194] Step 303: Determine residual information of the current image based on the target prediction information of the current image and the current image.

[0195] In some embodiments, if the reconstruction information of the first encoded image includes the image features of the reconstructed first encoded image, in this case, the encoding end can determine the residual information corresponding to the current image according to the relevant algorithm based on the target prediction information of the current image and the image features of the current image.

[0196] In other embodiments, if the reconstruction information of the first encoded image includes the reconstructed first encoded image, in this case, the encoding end can determine the residual information of the current image according to a relevant algorithm based on the target prediction information of the current image and the current image.

[0197] Step 304: Encode the residual information of the current image and the motion information of the current image into the bitstream.

[0198] In some embodiments, the encoding end can encode the residual information of the current image and the motion information of the current image into the bitstream according to the relevant algorithm, so that the subsequent decoding end can decode the current image based on the residual information of the current image and the motion information of the current image in the bitstream.

[0199] It should be noted that the residual information of the current image and the motion information of the current image can be encoded into different code streams respectively, or the residual information of the current image and the motion information of the current image can be encoded into the same code stream, which is not limited in this embodiment of the present application.

[0200] The embodiment of the present application determines the target prediction information of the current image by using the global reference tensor corresponding to the current image and the motion information of the current image. Since the global reference tensor is determined based on multiple encoded images before the current image, that is, the embodiment of the present application can generate a global reference tensor based on multiple encoded images before the current image, so that the global reference tensor can represent the image information that can be used for reference by the current image in multiple encoded images before the current image, and has globality across the time domain. In this way, it can be ensured that the target prediction information corresponding to the current image obtained based on the global reference tensor corresponding to the current image and the motion information of the current image is more accurate, so that the data bit amount of the residual information corresponding to the current image determined subsequently is smaller, thereby effectively improving the efficiency of encoding and decoding. In addition, by generating a global reference tensor based on multiple encoded images before the current image, the encoding end can ensure that the current image has more information for reference by the current image when encoding without having to cache multiple encoded images, thereby greatly reducing the cache pressure of the encoding end on the basis of improving the encoding and decoding efficiency.

[0201] Since the decoder also needs to determine the global motion information corresponding to the current image in subsequent steps, the decoder cannot obtain the original motion information of the current image during decoding, and can only obtain the reconstructed motion information of the current image. In other words, the motion information reconstructed by the decoder may have a certain deviation from the motion information corresponding to the original current image. Therefore, the encoder determines the global motion information corresponding to the current image based on the reconstructed motion information of the current image to ensure that the global motion information corresponding to the current image subsequently used by the encoder and decoder is the same. This makes the target prediction information obtained by the encoder and decoder more accurate, thereby effectively improving the efficiency and performance of encoding and decoding. The encoder directly determines the global prediction information corresponding to the current image based on the motion information of the current image and the global reference tensor corresponding to the current image, which can effectively reduce the computational cost of image compression and improve the efficiency of encoding and decoding. Since the motion information of the current image can, to a certain extent, represent the motion of the target block in the current image relative to the encoded image, the encoder can directly determine the global prediction information corresponding to the current image based on the motion information of the current image and the global reference tensor corresponding to the current image. This can effectively reduce the computational cost of image compression and improve the efficiency of encoding and decoding while ensuring the accuracy of the subsequently determined target prediction information.

[0202] FIG8 is a flowchart of a decoding method provided in an embodiment of the present application. The method is applied to a decoding end. Please refer to FIG8 . The method includes the following steps.

[0203] Step 801: Obtain reconstructed motion information of the current image based on the code stream.

[0204] The reconstructed motion information of the current image includes a motion vector, which is used to indicate the offset of a target block in the current image and in a reference image for the current image. The reference image is a reconstructed image of a decoded image adjacent to the current image. In this case, the reference image for the current image is the reconstructed image of the first decoded image described below, or in other words, the reference image for the current image is the reconstructed first decoded image.

[0205] It should be noted that, in the embodiment of the present application, the reconstructed first decoded image and the reconstructed image of the first decoded image have the same meaning.

[0206] In some embodiments, the current image includes at least one target block, which may be a coding unit (CU) in the HEVC standard. In other embodiments, the target block may be a pixel point or a set of multiple pixels in the current image. In other embodiments, the target block may be an element or a set of multiple elements in a tensor corresponding to the current image. The embodiments of the present application do not limit the specific composition of the target block.

[0207] Step 802: Determine a global reference tensor corresponding to a current image, where the global reference tensor is determined based on a plurality of decoded images prior to the current image.

[0208] In some embodiments, if the current image is in a video, the video includes at least one GOP, and the decoding end can determine that the global reference tensor corresponding to the current image is 0, that is, the global reference tensor corresponding to the I frame is a feature of all 0s.

[0209] When the current image is an I frame in the Nth GOP in the video, or a P frame in the video, the decoding end can determine the global reference tensor corresponding to the current image, where N is an integer greater than 1.

[0210] That is, when the current image is an I-frame in the first GOP of the video, the decoder determines that the global reference tensor corresponding to the current image is 0, that is, the global reference tensor corresponding to the I-frame is a feature of all 0s. When the current image is an I-frame in the Nth GOP of the video, the decoder determines the global reference tensor corresponding to the current image, or determines that the global reference tensor corresponding to the current image is 0.

[0211] It should be noted that when the current image is an I-frame in a GOP, if the decoder determines the global reference tensor corresponding to the current image to be 0, that is, regardless of the I-frame number in the GOP of the video, the decoder determines the global reference tensor corresponding to the current image to be 0. In this case, the global reference tensor is determined based on multiple decoded images before the current image, and these multiple decoded images are the decoded images in the GOP where the current image is located. In other words, the global reference tensor is updated in units of GOPs. The global reference tensor is updated within each GOP and reset to 0 across GOPs.

[0212] If the decoding end determines the global reference tensor corresponding to the current image as 0 only when the current image is an I frame in the first GOP in the video, and when the current image is an I frame in the Nth GOP in the video, the decoding end determines the global reference tensor corresponding to the current image. In this case, the global reference tensor corresponding to the current image is determined based on all decoded images before the current image.

[0213] Optionally, when the current image is an I frame in a GOP, the decoding end can decode the current image according to a related I frame decoding technology.

[0214] In other embodiments, if the current image is in a video, and the current image is the first frame in the video, the decoder can determine that the global reference tensor corresponding to the current image is 0, that is, the global reference tensor corresponding to the first frame is a feature of all 0s. When the current image is the Mth frame in the video, the decoder can determine the global reference tensor corresponding to the current image, where M is an integer greater than 1.

[0215] Optionally, when the current image is the first frame image in a GOP, the decoding end can decode the current image according to a related I-frame decoding technology.

[0216] In some embodiments, the decoding end determines the reconstruction information of a first decoded image, where the first decoded image is a decoded image adjacent to the current image, and based on the reconstruction information of the first decoded image, updates the global reference tensor corresponding to the first decoded image to obtain the global reference tensor corresponding to the current image.

[0217] It should be noted that the first decoded image is usually the previous decoded image adjacent to the current image. For example, if the current image is the tth frame of the video, the first decoded image is the t-1th frame.

[0218] For example, when the current image is in a video including at least one GOP, if the current image is the first P frame in the GOP, the first decoded image may be an I frame in the GOP. If the current image is the second P frame in the GOP, the first decoded image may be the first P frame in the GOP.

[0219] For example, if the current image is in a video, and the decoding end determines that the global reference tensor corresponding to the current image is 0 when the current image is the first frame in the video, and determines the global reference tensor corresponding to the current image when the current image is the Mth frame in the video. In this case, the global reference tensor corresponding to the image of the Nth frame in the video code stream is obtained based on the global reference tensor corresponding to the decoded image of the N-1th frame and the reconstruction information of the decoded image of the N-1th frame, and the global reference tensor corresponding to the second frame is the reconstruction information of the decoded image of the 1st frame.

[0220] In some embodiments, the reconstruction information of the first decoded image includes the reconstructed first decoded image, or image features of the reconstructed first decoded image.

[0221] In the case where the reconstruction information of the first decoded image includes the reconstructed first decoded image, in some embodiments, the decoding end stores the reconstructed first decoded image, so that the decoding end can directly determine the reconstruction information of the first decoded image.

[0222] In other embodiments, the decoding end is capable of reconstructing the first decoded image, that is, obtaining reconstructed motion information of the first decoded image based on the code stream, and then determining target prediction information of the first decoded image based on the global reference tensor corresponding to the first decoded image and the reconstructed motion information of the first decoded image; obtaining reconstructed residual information of the first decoded image based on the code stream; and reconstructing the first decoded image based on the target prediction information of the first decoded image and the reconstructed residual information of the first decoded image to obtain a reconstructed image of the first decoded image.

[0223] The implementation method of determining the target prediction information of the first decoded image based on the global reference tensor corresponding to the first decoded image and the reconstructed motion information of the first decoded image is similar to the implementation method of determining the target prediction information of the current image based on the global reference tensor corresponding to the current image and the reconstructed motion information of the current image in the above encoding method; the implementation method of obtaining the reconstructed image of the first decoded image based on the target prediction information of the first decoded image and the reconstructed residual information of the first decoded image is similar to the implementation method of obtaining the reconstructed image of the current image based on the target prediction information of the current image and the reconstructed residual information of the current image in the above encoding method. For details, please refer to the relevant content above and will not be repeated here.

[0224] If the reconstruction information of the first decoded image includes a reconstructed first decoded image, the decoding terminal can reconstruct the first decoded image to obtain a reconstructed first decoded image, and determine image features of the reconstructed first decoded image based on the reconstructed first decoded image.

[0225] In some embodiments, based on the reconstruction information of the first decoded image, the global reference tensor corresponding to the first decoded image is updated to obtain the global reference tensor corresponding to the current image. The implementation process includes: inputting the reconstruction information of the first decoded image and the global reference tensor corresponding to the first decoded image into the global reference tensor update network to obtain the global reference tensor corresponding to the current image output by the global reference tensor update network.

[0226] The implementation method of determining the image features of the reconstructed first decoded image based on the reconstructed first decoded image is similar to the implementation method of determining the image features of the reconstructed first coded image based on the reconstructed first coded image in the above encoding method; the implementation method of inputting the reconstruction information of the first decoded image and the global reference tensor corresponding to the first decoded image into the global reference tensor update network to obtain the global reference tensor corresponding to the current image output by the global reference tensor update network is similar to the implementation method of inputting the reconstruction information of the first coded image and the global reference tensor corresponding to the first coded image into the global reference tensor update network to obtain the global reference tensor corresponding to the current image output by the global reference tensor update network in the above encoding method. For details, please refer to the corresponding content above and will not be repeated here.

[0227] Step 803: Determine target prediction information of the current image based on the global reference tensor corresponding to the current image and the reconstructed motion information of the current image.

[0228] Next, through (1)-(2), the implementation method of determining the target prediction information of the current image based on the global reference tensor corresponding to the current image and the reconstructed motion information of the current image will be introduced in detail.

[0229] (1) determining global prediction information corresponding to the current image based on a global reference tensor corresponding to the current image and reconstructed motion information of the current image;

[0230] In some embodiments, the decoding end determines the global prediction information corresponding to the current image based on the global reference tensor corresponding to the current image and the reconstructed motion information of the current image, according to relevant prediction techniques such as optical flow mapping (warping) and deformable convolution. The embodiments of the present application do not limit this.

[0231] In other embodiments, the decoding end can determine the reconstructed motion information of the current image based on the reconstructed motion information of the current image, and determine the global motion information corresponding to the current image based on the reconstructed motion information of the current image, where the global motion information is used to describe the position offset of the current image relative to the global reference tensor corresponding to the current image, and determine the global prediction information corresponding to the current image based on the global reference tensor corresponding to the current image and the global motion information corresponding to the current image.

[0232] In some embodiments, the decoding end may update the global motion information corresponding to the first decoded image based on the reconstructed motion information of the current image to obtain the global motion information corresponding to the current image.

[0233] Based on the reconstructed motion information of the current image, the global motion information corresponding to the first decoded image is updated to obtain the implementation method of the global motion information corresponding to the current image. This is similar to the implementation method of updating the global motion information corresponding to the first encoded image based on the reconstructed motion information of the current image in the above encoding method to obtain the global motion information corresponding to the current image. For details, please refer to the corresponding content above and will not be repeated here.

[0234] (2) Based on the global prediction information corresponding to the current image, determine the target prediction information of the current image.

[0235] In some embodiments, the decoding end may directly determine the global prediction information corresponding to the current image as the target prediction information of the current image.

[0236] In other embodiments, the decoding end determines local prediction information corresponding to the current image based on the reconstructed motion information of the current image and the reconstructed information of the first decoded image, and determines target prediction information for the current image based on the local prediction information corresponding to the current image and the global prediction information corresponding to the current image. For detailed implementation, please refer to the corresponding content of the encoding method above and will not be repeated here.

[0237] Step 804: Obtain reconstruction residual information of the current image based on the code stream, and obtain a reconstructed image of the current image based on the target prediction information of the current image and the reconstruction residual information of the current image.

[0238] In some embodiments, the decoding end can parse the code stream to obtain reconstructed residual information of the current image.

[0239] In some embodiments, if the reconstruction information of the first decoded image includes image features of the reconstructed first decoded image, in this case, the decoding end determines the image features of the reconstructed image of the current image based on the target prediction information of the current image and the reconstruction residual information of the current image according to the relevant algorithm; and reconstructs the current image based on the image features of the reconstructed image of the current image according to the relevant algorithm to obtain the reconstructed image of the current image.

[0240] Optionally, the decoding end may input the target prediction information of the current image and the reconstruction residual information of the current image into the reconstruction network to obtain the image features of the reconstructed image of the current image output by the reconstruction network.

[0241] For example, FIG9 is a schematic diagram of a reconstruction network structure provided by an embodiment of the present application. The reconstruction network includes a deformable convolution layer (dconv) and three consecutive ResBlock sub-networks with the same configuration. in , h in , win ] indicates that the number of channels of the input tensor of the reconstruction network is c in , the size of the input tensor of the reconstruction network in the x-axis direction is h in , the size of the input tensor of the reconstruction network in the y-axis direction is w in ;dconv(ks,c in , c out , 2) indicates that the convolution kernel size of the deformable convolution layer is ks, and the number of channels of the input tensor of the deformable convolution layer is c in , the number of channels of the output tensor of the deformable convolution layer is c out , the step length of the convolution kernel movement is 2. ResBlock(c in ) indicates that the number of channels of the output tensor of the ResBlock sub-network is c in ,The structure of the ResBlock sub-network is shown in Figure 5. [c out , h in ×2, w in ×2] indicates that the number of channels of the reconstructed network output tensor is c out , the output tensor of the reconstruction network has a size of h in the x-axis direction in ×2, the size of the output tensor of the reconstruction network in the y-axis direction is w in ×2.

[0242] In other embodiments, if the reconstruction information of the first decoded image includes a reconstructed first decoded image, in this case, the decoding end can reconstruct the current image based on the target prediction information of the current image and the reconstruction residual information of the current image according to the relevant algorithm to obtain a reconstructed image of the current image.

[0243] During the decoding process, accurate target prediction information can be obtained through the reconstructed motion information of the current image and the global reference tensor corresponding to the current image. In this way, it can be ensured that the subsequent reconstructed image of the current image based on the target prediction information and the reconstructed residual information corresponding to the current image has better image quality, thereby greatly improving the encoding and decoding performance.

[0244] Next, the encoding and decoding method provided by the embodiment of the present application will be described in detail again through Figures 10-12. In Figures 10-12, it is assumed that the current image is x t , the first encoded image reconstructed by the encoding end and the first decoded image reconstructed by the decoding end are both The global reference tensor corresponding to the first encoded image at the encoding end and the global reference tensor corresponding to the first decoded image at the decoding end are both The global motion information corresponding to the first encoded image at the encoding end and the global motion information corresponding to the first decoded image at the decoding end are

[0245] Please refer to FIG10, which is a flowchart of a coding and decoding method provided by an embodiment of the present application. The reconstruction information of the first coded image reconstructed by the encoding end includes the reconstructed first coded image The reconstruction information of the first decoded image reconstructed by the decoding end includes the reconstructed first decoded image

[0246] During encoding:

[0247] Step 1: Reconstruct the first coded image The global reference tensor corresponding to the first encoded image Input the global reference tensor update network to obtain the global reference tensor corresponding to the current image output by the global reference tensor update network

[0248] Step 2: Based on the current image x t and the reconstructed first coded image Determine the motion information of the current image and encode the motion information of the current image into the code stream to obtain the motion information code stream of the current image Then the code stream Parse to obtain the reconstructed motion information of the current image

[0249] Step 3: Reconstruct motion information of the current image Global motion information corresponding to the first coded image Input the global motion information update network to obtain the global motion information corresponding to the current image output by the global motion information update network

[0250] Step 4: Based on the global reference tensor corresponding to the current image Global motion information corresponding to the current image Global prediction information corresponding to the current image is determined, and the global prediction information corresponding to the current image is determined as target prediction information of the current image.

[0251] Step 5: Based on the target prediction information of the current image and the current image x t , determine the residual information of the current image, and encode the residual information of the current image into the code stream to obtain the residual information code stream of the current image

[0252] During decoding:

[0253] Step (1): Reconstruct the first decoded image The global reference tensor corresponding to the first decoded image Input the global reference tensor update network to obtain the global reference tensor corresponding to the current image output by the global reference tensor update network

[0254] Step (2): Motion information code stream of the current image Parse to obtain the reconstructed motion information of the current image

[0255] Step (3): Reconstruct the motion information of the current image Global motion information corresponding to the first decoded image Input the global motion information update network to obtain the global motion information corresponding to the current image output by the global motion information update network

[0256] Step (4): Based on the global reference tensor corresponding to the current image Global motion information corresponding to the current image Global prediction information corresponding to the current image is determined, and the global prediction information corresponding to the current image is determined as target prediction information of the current image.

[0257] Step (5): Residual information code stream of the current image Analyze to obtain the reconstruction residual information of the current image

[0258] Step (6): Based on the target prediction information of the current image and the reconstruction residual information of the current image Reconstruct the current image to obtain the reconstructed image of the current image

[0259] Please refer to FIG11, which is a flowchart of another encoding and decoding method provided by an embodiment of the present application. The reconstruction information of the first encoded image reconstructed by the encoding end includes the reconstructed first encoded image The reconstruction information of the first decoded image reconstructed by the decoding end includes the reconstructed first decoded image image features.

[0260] During encoding:

[0261] Step 1: Reconstruct the first coded image Input the image feature extraction network to obtain the reconstructed first encoded image output by the image feature extraction network The image features of the current image x tInput the image feature extraction network to obtain the current image x output by the image feature extraction network t image features.

[0262] Step 2: Reconstruct the first coded image The image features and the global reference tensor corresponding to the first encoded image Input the global reference tensor update network to obtain the global reference tensor corresponding to the current image output by the global reference tensor update network

[0263] Step 3: Based on the current image x t Image features and the reconstructed first coded image The image features of the current image are used to determine the motion information of the current image, and the motion information of the current image is encoded into the code stream to obtain the motion information code stream of the current image. Then the code stream Parse to obtain the reconstructed motion information of the current image

[0264] Step 4: Reconstruct motion information of the current image Global motion information corresponding to the first coded image Input the global motion information update network to obtain the global motion information corresponding to the current image output by the global motion information update network

[0265] Step 5: Based on the global reference tensor corresponding to the current image Global motion information corresponding to the current image Determine the global prediction information corresponding to the current image.

[0266] Step 6: Reconstruct motion information based on the current image and the reconstructed first coded image The image features are used to determine the local prediction information corresponding to the current image.

[0267] Step 7: Fuse the local prediction information corresponding to the current image with the global prediction information corresponding to the current image to obtain the target prediction information of the current image.

[0268] Step 8: Based on the target prediction information of the current image and the current image x t The image features of the current image are used to determine the residual information of the current image, and the residual information of the current image is encoded into the code stream to obtain the residual information code stream of the current image.

[0269] During decoding:

[0270] Step (1): Reconstruct the first decoded image Input the image feature extraction network to obtain the reconstructed first decoded image output by the image feature extraction network image features.

[0271] Step (2): Reconstruct the first decoded image The image features and the global reference tensor corresponding to the first decoded image Input the global reference tensor update network to obtain the global reference tensor corresponding to the current image output by the global reference tensor update network

[0272] Step (3): Motion information code stream of the current image Parse to obtain the reconstructed motion information of the current image

[0273] Step (4): Reconstruct the motion information of the current image Global motion information corresponding to the first decoded image Input the global motion information update network to obtain the global motion information corresponding to the current image output by the global motion information update network

[0274] Step (5): Based on the global reference tensor corresponding to the current image Global motion information corresponding to the current image Determine the global prediction information corresponding to the current image.

[0275] Step (6): Reconstruct motion information based on the current image and reconstruct the first decoded image The image features are used to determine the local prediction information corresponding to the current image.

[0276] Step (7): Fusing the local prediction information corresponding to the current image and the global prediction information corresponding to the current image to obtain the target prediction information of the current image.

[0277] Step (8): Residual information code stream of the current image Analyze to obtain the reconstruction residual information of the current image

[0278] Step (9): Based on the target prediction information of the current image and the reconstruction residual information of the current image Reconstruct the current image The image features of the reconstructed current image are then used to reconstruct the current image to obtain a reconstructed image of the current image.

[0279] Please refer to Figure 12, which shows the test results of the performance test of the encoding and decoding method provided in Figure 11 of the embodiment of the present application using three standard test sequence sets, class B, class C, and class D. Among them, class B contains 5 1080p videos, class C contains 4 720p videos, and class D contains 4 416x240 size videos. It is not difficult to see from Figure 13 that the RD-Curve of the embodiment of the present application on the test set has an average compression efficiency improvement of 14%, 13%, and 15% in the three standard test sequence sets compared with the related art (that is, the transmission bandwidth can be reduced by 14%, 13%, and 15% respectively compared with the related art).

[0280] Please refer to FIG13, which is a flowchart of another encoding and decoding method provided by an embodiment of the present application. The reconstruction information of the first encoded image reconstructed by the encoding end includes the reconstructed first encoded image The reconstruction information of the first decoded image reconstructed by the decoding end includes the reconstructed first decoded image image features.

[0281] During encoding:

[0282] Step 1: Reconstruct the first coded image Input the image feature extraction network to obtain the reconstructed first encoded image output by the image feature extraction network The image features of the current image x t Input the image feature extraction network to obtain the current image x output by the image feature extraction network t image features.

[0283] Step 2: Reconstruct the first coded image The image features and the global reference tensor corresponding to the first encoded image Input the global reference tensor update network to obtain the global reference tensor corresponding to the current image output by the global reference tensor update network

[0284] Step 3: Based on the current image x t Image features and the reconstructed first coded image The image features of the current image are used to determine the motion information of the current image, and the motion information of the current image is encoded into the code stream to obtain the motion information code stream of the current image. Then the code stream Parse to obtain the reconstructed motion information of the current image

[0285] Step 4: Based on the global reference tensor corresponding to the current image and the reconstructed motion information of the current image Determine the global prediction information corresponding to the current image.

[0286] Step 5: Reconstruct motion information based on the current image and the reconstructed first coded image The image features are used to determine the local prediction information corresponding to the current image.

[0287] Step 6: Fuse the local prediction information corresponding to the current image with the global prediction information corresponding to the current image to obtain the target prediction information of the current image.

[0288] Step 7: Target prediction information based on the current image and the current image x t The image features of the current image are used to determine the residual information of the current image, and the residual information of the current image is encoded into the code stream to obtain the residual information code stream of the current image.

[0289] During decoding:

[0290] Step (1): Reconstruct the first decoded image Input the image feature extraction network to obtain the reconstructed first decoded image output by the image feature extraction network image features.

[0291] Step (2): Reconstruct the first decoded image The image features and the global reference tensor corresponding to the first decoded image Input the global reference tensor update network to obtain the global reference tensor corresponding to the current image output by the global reference tensor update network

[0292] Step (3): Motion information code stream of the current image Parse to obtain the reconstructed motion information of the current image

[0293] Step (4): Based on the global reference tensor corresponding to the current image and the reconstructed motion information of the current image Determine the global prediction information corresponding to the current image.

[0294] Step (5): Reconstruct motion information based on the current image and reconstruct the first decoded image The image features are used to determine the local prediction information corresponding to the current image.

[0295] Step (6): Fusing the local prediction information corresponding to the current image and the global prediction information corresponding to the current image to obtain the target prediction information of the current image.

[0296] Step (7): Residual information code stream of the current image Analyze to obtain the reconstruction residual information of the current image

[0297] Step (8): Based on the target prediction information of the current image and the reconstruction residual information of the current image Reconstruct the current image The image features of the reconstructed current image are then used to reconstruct the current image to obtain a reconstructed image of the current image.

[0298] FIG14 is a schematic diagram of the structure of an encoding device provided in an embodiment of the present application. The encoding device can be implemented as part or all of an encoding end by software, hardware, or a combination of both. Moreover, the encoding device can be encoder 100 in FIG1 . Referring to FIG14 , the device includes: a first determination module 1401, a second determination module 1402, a third determination module 1403, and an encoding module 1404.

[0299] First determination module 1401 is configured to determine a global reference tensor corresponding to a current image and motion information for the current image. The global reference tensor is determined based on multiple encoded images prior to the current image. The detailed implementation process is described in the corresponding embodiments above and will not be repeated here.

[0300] The second determination module 1402 is configured to determine target prediction information of the current image based on the global reference tensor corresponding to the current image and the motion information of the current image. The detailed implementation process is referred to the corresponding content of each of the above embodiments and will not be repeated here.

[0301] The third determining module 1403 is configured to determine the residual information of the current image based on the target prediction information of the current image and the current image. The detailed implementation process is referred to the corresponding content of each of the above embodiments and will not be repeated here.

[0302] The encoding module 1404 is used to encode the residual information and the motion information of the current image into the bitstream. The detailed implementation process is referred to the corresponding content of each embodiment above and will not be repeated here.

[0303] Optionally, the first determining module 1401 is specifically configured to:

[0304] Determining reconstruction information of a first coded image, the first coded image being a coded image adjacent to the current image;

[0305] Based on the reconstruction information of the first encoded image, a global reference tensor corresponding to the first encoded image is updated to obtain a global reference tensor corresponding to the current image.

[0306] Optionally, the global reference tensor corresponding to the current image is determined based on all encoded images before the current image.

[0307] Optionally, the global reference tensor corresponding to the image of the Nth frame in the video code stream is obtained based on the global reference tensor corresponding to the encoded image of the N-1th frame and the reconstruction information of the encoded image of the N-1th frame, and the global reference tensor corresponding to the second frame image is the reconstruction information of the encoded image of the first frame.

[0308] Optionally, the second determining module 1402 is specifically configured to:

[0309] Determining global prediction information corresponding to the current image based on a global reference tensor corresponding to the current image and motion information of the current image;

[0310] Based on the global prediction information corresponding to the current image, target prediction information of the current image is determined.

[0311] Optionally, the second determining module 1402 is specifically configured to:

[0312] Determining reconstructed motion information of the current image based on motion information of the current image;

[0313] Determining, based on reconstructed motion information of the current image, global motion information corresponding to the current image, the global motion information including a global motion vector indicating an offset of a target block in the current image and a global reference tensor corresponding to the current image;

[0314] Based on the global reference tensor corresponding to the current image and the global motion information corresponding to the current image, global prediction information corresponding to the current image is determined.

[0315] Optionally, the second determining module 1402 is specifically configured to:

[0316] Based on the reconstructed motion information of the current image, the global motion information corresponding to the first encoded image is updated to obtain the global motion information corresponding to the current image.

[0317] Optionally, the second determining module 1402 is specifically configured to:

[0318] Determining local prediction information corresponding to the current image based on the reconstructed motion information of the current image and the reconstructed information of the first coded image;

[0319] Target prediction information of the current image is determined based on local prediction information corresponding to the current image and global prediction information corresponding to the current image.

[0320] Optionally, the reconstruction information of the first coded image includes the reconstructed first coded image or an image feature of the reconstructed first coded image.

[0321] Optionally, the third determining module 1403 is specifically configured to:

[0322] Based on target prediction information of the current image and image features of the current image, residual information of the current image is determined.

[0323] The embodiment of the present application determines the target prediction information of the current image by using the global reference tensor corresponding to the current image and the motion information of the current image. Since the global reference tensor is determined based on multiple encoded images before the current image, that is, the embodiment of the present application can generate a global reference tensor based on multiple encoded images before the current image, so that the global reference tensor can represent the image information that can be used for reference by the current image in multiple encoded images before the current image, and has globality across the time domain. In this way, it can be ensured that the target prediction information corresponding to the current image obtained based on the global reference tensor corresponding to the current image and the motion information of the current image is more accurate, so that the data bit amount of the residual information corresponding to the current image determined subsequently is smaller, thereby effectively improving the efficiency of encoding and decoding. In addition, by generating a global reference tensor based on multiple encoded images before the current image, the encoding end can ensure that the current image has more information for reference by the current image when encoding without having to cache multiple encoded images, thereby greatly reducing the cache pressure of the encoding end on the basis of improving the encoding and decoding efficiency.

[0324] Since the decoder also needs to determine the global motion information corresponding to the current image in subsequent steps, the decoder cannot obtain the original motion information of the current image during decoding, and can only obtain the reconstructed motion information of the current image. In other words, the motion information reconstructed by the decoder may have a certain deviation from the motion information corresponding to the original current image. Therefore, the encoder determines the global motion information corresponding to the current image based on the reconstructed motion information of the current image to ensure that the global motion information corresponding to the current image subsequently used by the encoder and decoder is the same. This makes the target prediction information obtained by the encoder and decoder more accurate, thereby effectively improving the efficiency and performance of encoding and decoding. The encoder directly determines the global prediction information corresponding to the current image based on the motion information of the current image and the global reference tensor corresponding to the current image, which can effectively reduce the computational cost of image compression and improve the efficiency of encoding and decoding. Since the motion information of the current image can, to a certain extent, represent the motion of the target block in the current image relative to the encoded image, the encoder can directly determine the global prediction information corresponding to the current image based on the motion information of the current image and the global reference tensor corresponding to the current image. This can effectively reduce the computational cost of image compression and improve the efficiency of encoding and decoding while ensuring the accuracy of the subsequently determined target prediction information.

[0325] It should be noted that the encoding device provided in the above embodiment is merely illustrated by the division of the above functional modules during encoding. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the encoding device provided in the above embodiment and the encoding method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0326] FIG15 is a schematic diagram of the structure of a decoding device provided in an embodiment of the present application. The decoding device can be implemented by software, hardware, or a combination of both to form part or all of a decoding end. Furthermore, the decoding device can be decoder 200 in FIG1 . Referring to FIG15 , the device includes a first parsing module 1501, a first determination module 1502, a second determination module 1503, a second parsing module 1504, and a reconstruction module 1505.

[0327] The first parsing module 1501 is configured to obtain the reconstructed motion information of the current image based on the bitstream. Detailed implementation procedures are described in the corresponding contents of the above embodiments and will not be repeated here.

[0328] The first determination module 1502 is configured to determine a global reference tensor corresponding to the current image. The global reference tensor is determined based on multiple decoded images prior to the current image. The detailed implementation process is described in detail in the above embodiments and will not be repeated here.

[0329] The second determination module 1503 is configured to determine target prediction information of the current image based on the global reference tensor corresponding to the current image and the reconstructed motion information of the current image. Detailed implementation procedures are described in detail in the above embodiments and will not be repeated here.

[0330] The second parsing module 1504 is configured to obtain the reconstructed residual information of the current image based on the code stream. The detailed implementation process is referred to the corresponding contents of the above embodiments and will not be repeated here.

[0331] The reconstruction module 1505 is configured to obtain a reconstructed image of the current image based on the target prediction information of the current image and the reconstruction residual information of the current image. The detailed implementation process is referred to the corresponding content of each of the above embodiments and will not be repeated here.

[0332] Optionally, the first determining module 1502 is specifically configured to:

[0333] Determining reconstruction information of a first decoded image, where the first decoded image is a decoded image adjacent to the current image;

[0334] Based on the reconstruction information of the first decoded image, a global reference tensor corresponding to the first decoded image is updated to obtain a global reference tensor corresponding to the current image.

[0335] Optionally, the global reference tensor corresponding to the current image is determined based on all decoded images before the current image.

[0336] Optionally, the global reference tensor corresponding to the image of the Nth frame in the video code stream is obtained based on the global reference tensor corresponding to the decoded image of the N-1th frame and the reconstruction information of the decoded image of the N-1th frame, and the global reference tensor corresponding to the second frame image is the reconstruction information of the decoded image of the first frame.

[0337] Optionally, the second determining module 1503 is specifically configured to:

[0338] Determining global prediction information corresponding to the current image based on a global reference tensor corresponding to the current image and reconstructed motion information of the current image;

[0339] Based on the global prediction information corresponding to the current image, target prediction information of the current image is determined.

[0340] Optionally, the second determining module 1503 is specifically configured to:

[0341] Determining global motion information corresponding to the current image based on reconstructed motion information of the current image, the global motion information including a global motion vector indicating an offset of a target block in the current image from a global reference tensor corresponding to the current image;

[0342] Based on the global reference tensor corresponding to the current image and the global motion information corresponding to the current image, global prediction information corresponding to the current image is determined.

[0343] Optionally, the second determining module 1503 is specifically configured to:

[0344] Based on the reconstructed motion information of the current image, the global motion information corresponding to the first decoded image is updated to obtain the global motion information corresponding to the current image.

[0345] Optionally, the second determining module 1503 is specifically configured to:

[0346] Determining local prediction information corresponding to the current image based on reconstructed motion information of the current image and reconstruction information corresponding to the first decoded image;

[0347] Target prediction information of the current image is determined based on local prediction information corresponding to the current image and global prediction information corresponding to the current image.

[0348] Optionally, the reconstruction information of the first decoded image includes the reconstructed first decoded image or an image feature of the reconstructed first decoded image.

[0349] Optionally, the reconstruction module 1505 is specifically configured to:

[0350] Determining image features of a reconstructed image of the current image based on target prediction information of the current image and reconstruction residual information of the current image;

[0351] A reconstructed image of the current image is determined based on image features of the reconstructed image of the current image.

[0352] During the decoding process, accurate target prediction information can be obtained through the reconstructed motion information of the current image and the global reference tensor corresponding to the current image. In this way, it can be ensured that the subsequent reconstructed image of the current image based on the target prediction information and the reconstructed residual information corresponding to the current image has better image quality, thereby greatly improving the encoding and decoding performance.

[0353] It should be noted that the decoding device provided in the above embodiment is merely illustrated by the division of the above functional modules during decoding. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the decoding device provided in the above embodiment and the decoding method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0354] An embodiment of the present application also provides an encoding device, which includes: a processor, the processor is coupled to a memory, the memory is used to store programs or instructions, and when the program or instructions are executed by the processor, the encoding device executes the above-mentioned encoding method.

[0355] An embodiment of the present application also provides a decoding device, which includes: a processor, the processor is coupled to a memory, the memory is used to store programs or instructions, and when the program or instructions are executed by the processor, the decoding device executes the above-mentioned decoding method.

[0356] An embodiment of the present application further provides a coding and decoding system, which includes the coding device and / or the decoding device.

[0357] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the above-mentioned encoding method or the above-mentioned decoding method.

[0358] An embodiment of the present application also provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the steps of the above-mentioned encoding method are executed, or the steps of the above-mentioned decoding method are executed.

[0359] An embodiment of the present application further provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a code stream obtained according to the above encoding method.

[0360] An embodiment of the present application also provides a device for storing a code stream, comprising at least one storage medium and a communication interface; the communication interface is used to receive or send a code stream; the at least one storage medium is used to store the code stream; the code stream is encoded by an encoder according to the above-mentioned image encoding method.

[0361] An embodiment of the present application further provides a method for storing a code stream, comprising: receiving a code stream through a communication interface; and storing the code stream in one or more storage media, wherein the code stream is encoded by an encoder according to the above-mentioned image encoding method.

[0362] An embodiment of the present application also provides a system for distributing code streams, comprising at least one storage medium and a video streaming device; the at least one storage medium is used to store the code stream, which is encoded by an encoder according to the above-mentioned image encoding method; the video streaming device is used to respond to a request from a decoder so that the code stream in the at least one storage medium can be sent to the decoder.

[0363] An embodiment of the present application also provides a method for distributing a code stream, comprising: receiving a first request; selecting a code stream from at least one storage medium in response to the first request; and sending the code stream to a destination device; the at least one storage medium is used to store the code stream, wherein the code stream is encoded by an encoder according to the above-mentioned image encoding method.

[0364] An embodiment of the present application also provides a system for processing a code stream, including an image source device, an encoder, one or more storage media, and a destination device; the image source device is used to provide image data; the encoder is used to obtain the image data of the image source device through an interface, and encode the image data to obtain one or more code streams, wherein the code streams are encoded by the encoder according to the above-mentioned image encoding method; the encoder is used to store the one or more code streams in one or more storage media; or the encoder is used to encapsulate the one or more code streams to obtain a transmission code stream; the encoder is used to transmit the transmission code stream to the destination device via a communication link or a communication network; the destination device is used to decapsulate the transmission code stream to obtain the one or more code streams; and the destination device is used to decode the one or more code streams to obtain decoded data.

[0365] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiments of the present application may be a non-volatile storage medium, in other words, a non-transient storage medium.

[0366] It should be understood that the "plurality" mentioned herein refers to two or more. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is merely a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate a clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0367] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the current images involved in the embodiments of this application were all obtained with full authorization.

[0368] The above description is an embodiment provided for this application and is not intended to limit this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application should be included in the scope of protection of this application.

Claims

1. A coding method, characterized in that, The method includes: Determining a global reference tensor corresponding to the current image and motion information of the current image, where the global reference tensor is determined based on a plurality of previously encoded images before the current image; Determining target prediction information of the current image based on the global reference tensor corresponding to the current image and the motion information of the current image; Determining residual information of the current image based on the target prediction information of the current image and the current image; Encoding the residual information and the motion information of the current image into a bitstream.

2. The method according to claim 1, wherein The determining the global reference tensor corresponding to the current image includes: Determining reconstruction information of a first encoded image, where the first encoded image is an encoded image adjacent to the current image; Updating the global reference tensor corresponding to the first encoded image based on the reconstruction information of the first encoded image to obtain the global reference tensor corresponding to the current image.

3. The method according to claim 1 or 2, characterized in that The global reference tensor corresponding to the current image is determined based on all previously encoded images before the current image.

4. The method according to claim 1, wherein The global reference tensor corresponding to the image of the Nth frame in the video bitstream is obtained based on the global reference tensor corresponding to the encoded image of the (N - 1)th frame and the reconstruction information of the encoded image of the (N - 1)th frame, and the global reference tensor corresponding to the second frame image is the reconstruction information of the encoded image of the first frame.

5. The method according to claim 2, wherein The determining the target prediction information of the current image based on the global reference tensor corresponding to the current image and the motion information of the current image includes: Determining global prediction information corresponding to the current image based on the global reference tensor corresponding to the current image and the motion information of the current image; Determining the target prediction information of the current image based on the global prediction information corresponding to the current image.

6. The method according to claim 5, characterized in that, The determining the global prediction information corresponding to the current image based on the global reference tensor corresponding to the current image and the motion information of the current image includes: Determining reconstruction motion information of the current image based on the motion information of the current image; Determining global motion information corresponding to the current image based on the reconstruction motion information of the current image, where the global motion information includes a global motion vector, and the global motion vector indicates the offset of a target block in the current image from that in the global reference tensor corresponding to the current image; Determining global prediction information corresponding to the current image based on the global reference tensor corresponding to the current image and the global motion information corresponding to the current image.

7. The method according to claim 6, characterized in that, The determining the global motion information corresponding to the current image based on the reconstruction motion information of the current image includes: Updating the global motion information corresponding to the first encoded image based on the reconstruction motion information of the current image to obtain the global motion information corresponding to the current image.

8. The method according to any one of claims 5 to 7, characterized in that, The determining the target prediction information of the current image based on the global prediction information corresponding to the current image includes: Determining local prediction information corresponding to the current image based on the reconstruction motion information of the current image and the reconstruction information of the first encoded image; Determine the target prediction information of the current image based on the local prediction information corresponding to the current image and the global prediction information corresponding to the current image. The target prediction information.

9. The method according to any one of claims 2-8, characterized in that, The reconstruction information of the first encoded image includes the reconstructed first encoded image or the image features of the reconstructed first encoded image.

10. The method according to claim 1, characterized in that, The determining the residual information of the current image based on the target prediction information of the current image and the current image includes: Determine the residual information of the current image based on the target prediction information of the current image and the image features of the current image.

11. A decoding method, characterized in that, The method includes: Obtain the reconstructed motion information of the current image based on the bitstream; Determine the global reference tensor corresponding to the current image, where the global reference tensor is determined based on multiple decoded images before the current image; Determine the target prediction information of the current image based on the global reference tensor corresponding to the current image and the reconstructed motion information of the current image; Obtain the reconstructed residual information of the current image based on the bitstream; Obtain the reconstructed image of the current image based on the target prediction information of the current image and the reconstructed residual information of the current image.

12. The method according to claim 11, wherein The determining the global reference tensor corresponding to the current image includes: Determine the reconstruction information of the first decoded image, where the first decoded image is a decoded image adjacent to the current image; Update the global reference tensor corresponding to the first decoded image based on the reconstruction information of the first decoded image to obtain the global reference tensor corresponding to the current image.

13. The method according to claim 11, wherein The global reference tensor corresponding to the current image is determined based on all decoded images before the current image.

14. The method according to claim 11, wherein The global reference tensor corresponding to the image of the Nth frame in the video bitstream is obtained based on the global reference tensor corresponding to the decoded image of the (N - 1)th frame and the reconstruction information of the decoded image of the (N - 1)th frame, and the global reference tensor corresponding to the second frame image is the reconstruction information of the decoded image of the first frame.

15. The method according to claim 12, wherein The determining the target prediction information of the current image based on the global reference tensor corresponding to the current image and the reconstructed motion information of the current image includes: Determine the global prediction information corresponding to the current image based on the global reference tensor corresponding to the current image and the reconstructed motion information of the current image; Determine the target prediction information of the current image based on the global prediction information corresponding to the current image.

16. The method according to claim 15, wherein The determining the global prediction information corresponding to the current image based on the global reference tensor corresponding to the current image and the reconstructed motion information of the current image includes: Determine the global motion information corresponding to the current image based on the reconstructed motion information of the current image, where the global motion information includes a global motion vector, and the global motion vector indicates the offset of the target block in the current image from that in the global reference tensor corresponding to the current image; Determine the global prediction information corresponding to the current image based on the global reference tensor corresponding to the current image and the global motion information corresponding to the current image.

17. The method according to claim 16, wherein Determining the global motion information corresponding to the current image based on the reconstructed motion information of the current image includes: Updating the global motion information corresponding to the first decoded image based on the reconstructed motion information of the current image to obtain the global motion information corresponding to the current image.

18. The method according to any one of claims 15-17, characterized in that, Determining the target prediction information of the current image based on the global prediction information corresponding to the current image includes: Determining the local prediction information corresponding to the current image based on the reconstructed motion information of the current image and the reconstruction information corresponding to the first decoded image; Determining the target prediction information of the current image based on the local prediction information corresponding to the current image and the global prediction information corresponding to the current image.

19. The method according to any one of claims 12-18, characterized in that, The reconstruction information of the first decoded image includes the reconstructed first decoded image or the image features of the reconstructed first decoded image.

20. The method according to claim 11, characterized in that, Obtaining the reconstructed image of the current image based on the target prediction information of the current image and the reconstruction residual information of the current image includes: Determining the image features of the reconstructed image of the current image based on the target prediction information of the current image and the reconstruction residual information of the current image; Determining the reconstructed image of the current image based on the image features of the reconstructed image of the current image.

21. A coding device, characterized in that, The apparatus includes: A first determination module, configured to determine the global reference tensor corresponding to the current image and the motion information of the current image, where the global reference tensor is determined based on a plurality of previously encoded images before the current image; A second determination module, configured to determine the target prediction information of the current image based on the global reference tensor corresponding to the current image and the motion information of the current image; A third determination module, configured to determine the residual information of the current image based on the target prediction information of the current image and the current image; An encoding module, configured to encode the residual information and the motion information of the current image into a bitstream.

22. A decoding device, characterized in that, The apparatus includes: A first parsing module, configured to obtain the reconstructed motion information of the current image based on the bitstream; A first determination module, configured to determine the global reference tensor corresponding to the current image, where the global reference tensor is determined based on a plurality of previously decoded images before the current image; A second determination module, configured to determine the target prediction information of the current image based on the global reference tensor corresponding to the current image and the reconstructed motion information of the current image; A second parsing module, configured to obtain the reconstruction residual information of the current image based on the bitstream; A reconstruction module, configured to obtain the reconstructed image of the current image based on the target prediction information of the current image and the reconstruction residual information of the current image.

23. A coding device, characterized in that, Includes: A processor, the processor is coupled to a memory, and the memory is configured to store programs or instructions. When the programs or instructions are executed by the processor, the encoding device is caused to execute the method according to any one of claims 1 to 10.

24. A decoding device, characterized in that, Includes: A processor, the processor being coupled to a memory for storing programs or instructions which, when executed by the processor, cause the decoding device to perform the method according to any one of claims 11 to 20.

25. A codec system, characterized in that, The encoding and decoding system includes the encoding device according to claim 23, and / or the decoding device according to claim 24.

26. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program which, when running on a computer or a processor, causes the computer or the processor to perform the method according to any one of claims 1 to 10, or to perform the method according to any one of claims 11 to 20.

27. A computer program product, characterized in that, The computer program product contains computer instructions which, when executed by a computer or a processor, cause the steps of the method according to any one of claims 1 to 10 to be executed, or cause the steps of the method according to any one of claims 11 to 20 to be executed.

28. A computer-readable storage medium, characterized in that, A bitstream obtained by performing the method according to any one of claims 1-10 is stored on the computer-readable storage medium and is executed by one or more processors.

29. A device for storing a bitstream, characterized in that, Comprising at least one storage medium and a communication interface; The communication interface is used for receiving or sending a bitstream; The at least one storage medium is used for storing the bitstream; The bitstream is encoded by an encoder according to any one of the encoding methods of claims 1 to 10.

30. A method for storing a bitstream, characterized in that, Comprising: Receiving a bitstream through the communication interface; Storing the bitstream into one or more storage media, the bitstream being an encoder Encoded according to any one of the encoding methods of claims 1 to 10.

31. A system for distributing a bitstream, characterized in that, Comprising at least one storage medium and a video stream device; The at least one storage medium is used for storing a bitstream, the bitstream being an encoder Encoded according to any one of the encoding methods of claims 1 to 10; The video stream device is configured to, in response to a request from a decoder, cause the target bitstream in the at least one storage medium to be sent to the decoder.

32. A method for distributing a bitstream, characterized in that, Comprising: Receiving a first request; In response to the first request, selecting a target bitstream from at least one storage medium; Sending the target bitstream to a destination device; The at least one storage medium is used for storing a bitstream, the bitstream being an encoder Encoded according to any one of the encoding methods of claims 1 to 10.

33. A system for processing a bitstream, characterized in that, Comprising an image source device, an encoder device, one or more storage media and a destination device; The image source device is configured to provide image data; The encoder device is configured to obtain the image data of the image source device through an interface and encode the image data to obtain one or more bitstreams, the bitstreams being encoded by the encoder according to any one of the encoding methods of claims 1 to 10; The encoder device is configured to store the one or more bitstreams into one or more storage media; or, The encoder device is configured to encapsulate the one or more bitstreams to obtain a transport bitstream; The encoder device is configured to transmit the transport bitstream to the destination device through a communication link or a communication network; The target device is used to unpack the transmission bitstream to obtain the one or more bitstreams; The target device is used to decode the one or more bitstreams to obtain decoded data.

Images