Video coding and decoding method, device, equipment, system and storage medium
Patent Information
- Application Number
- CN202280094956.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-29
- Publication Date
- 2024-11-26
AI Technical Summary
The existing video compression technology based on neural networks has poor compression effect and is difficult to effectively improve the compression rate of video data.
This paper proposes a video encoding and decoding method that generates a hybrid spatiotemporal representation through multi-level time domain fusion of quantized feature information, and performs motion compensation based on this representation to improve the video compression effect. Specific steps include feature fusion, quantization and encoding of the current image and the previous reconstructed image, and multi-level time domain fusion after decoding to generate a hybrid spatiotemporal representation, which is used to generate high-precision predicted images, thereby improving the video compression effect.
Through multi-level time domain fusion and motion compensation technology, the video compression effect is significantly improved, the generated predicted images are more accurate, the video data can be compressed more effectively, and the performance of video encoding and decoding is improved.
Smart Images

Figure CN119032566A_ABST
Abstract
Description
Video encoding and decoding method, device, equipment, system and storage medium Technical Field
[0001] The present application relates to the field of video coding and decoding technology, and in particular to a video coding and decoding method, apparatus, device, system and storage medium. Background Art
[0002] Digital video technology can be incorporated into a variety of video devices, such as digital televisions, smartphones, computers, e-readers, or video players. With the development of video technology, the amount of video data included has increased. To facilitate the transmission of video data, video devices implement video compression technology to enable more efficient transmission or storage of video data.
[0003] With the rapid development of neural network technology, neural network technology has been widely used in video compression technology, for example, in loop filtering, coding block division and coding block prediction. However, the current video compression technology based on neural network has poor compression effect.
[0004] Summary of the Invention
[0005] The embodiments of the present application provide a video encoding and decoding method, apparatus, device, system and storage medium to improve video compression effect.
[0006] In a first aspect, the present application provides a video decoding method, comprising:
[0007] Decoding the first bitstream to determine quantized first feature information, where the first feature information is obtained by performing feature fusion on a current image and a reconstructed image preceding the current image;
[0008] Performing multi-level time-domain fusion on the quantized first feature information to obtain a hybrid spatiotemporal representation;
[0009] Performing motion compensation on the previous reconstructed image according to the hybrid spatiotemporal representation to obtain P predicted images of the current image, where P is a positive integer;
[0010] A reconstructed image of the current image is determined according to the P predicted images.
[0011] In a second aspect, an embodiment of the present application provides a video encoding method, including:
[0012] Performing feature fusion on a current image and a reconstructed image previous to the current image to obtain first feature information;
[0013] quantizing the first feature information to obtain quantized first feature information;
[0014] The quantized first feature information is encoded to obtain the first code stream.
[0015] In a third aspect, the present application provides a video encoder configured to execute the method of the first aspect or its respective implementations. Specifically, the encoder includes a functional unit configured to execute the method of the first aspect or its respective implementations.
[0016] In a fourth aspect, the present application provides a video decoder for executing the method of the second aspect or its respective implementations. Specifically, the decoder includes a functional unit for executing the method of the second aspect or its respective implementations.
[0017] In a fifth aspect, a video encoder is provided, comprising a processor and a memory, wherein the memory is configured to store a computer program, and the processor is configured to call and execute the computer program stored in the memory to execute the method of the first aspect or its respective implementations.
[0018] In a sixth aspect, a video decoder is provided, comprising a processor and a memory, wherein the memory is configured to store a computer program, and the processor is configured to call and execute the computer program stored in the memory to perform the method of the second aspect or its respective implementations.
[0019] In a seventh aspect, a video encoding and decoding system is provided, comprising a video encoder and a video decoder. The video encoder is configured to execute the method of the first aspect or its respective implementations, and the video decoder is configured to execute the method of the second aspect or its respective implementations.
[0020] In an eighth aspect, a chip is provided for implementing the method described in any one of the first and second aspects above, or their respective implementations. Specifically, the chip includes a processor configured to load and execute a computer program from a memory, causing a device equipped with the chip to perform the method described in any one of the first and second aspects above, or their respective implementations.
[0021] In a ninth aspect, a computer-readable storage medium is provided for storing a computer program, which enables a computer to execute the method of any one of the first to second aspects or their respective implementations.
[0022] In a tenth aspect, a computer program product is provided, comprising computer program instructions, which enable a computer to execute the method of any one of the first to second aspects or their respective implementations.
[0023] In an eleventh aspect, a computer program is provided, which, when executed on a computer, enables the computer to execute the method in any one of the first to second aspects or their respective implementations.
[0024] In the twelfth aspect, a code stream is provided, including the code stream generated by any aspect of the second aspect.
[0025] Based on the above technical solution, in order to improve the accuracy of the reconstructed image, the present application performs multi-level time-domain fusion on the quantized first feature information, that is, the quantized first feature information is not only fused with the feature information of the previous reconstructed image of the current image, but also the quantized first feature information is fused with the feature information of multiple reconstructed images before the current image. This can avoid the situation where certain information in the previous reconstructed image of the current image is blocked, and the blocked information can be obtained from several reconstructed images before the current image, thereby making the generated mixed spatiotemporal representation include more accurate, rich and detailed feature information. In this way, when motion compensation is performed on the previous reconstructed image based on the mixed spatiotemporal representation, high-precision P predicted images can be generated. Based on the high-precision P predicted images, the reconstructed image of the current image can be accurately obtained, thereby improving the video compression effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] FIG1 is a schematic block diagram of a video encoding and decoding system according to an embodiment of the present application;
[0027] FIG2 is a schematic diagram of a flow chart of a video decoding method provided in an embodiment of the present application;
[0028] FIG3 is a schematic diagram of the network structure of the inverse transformation module involved in an embodiment of the present application;
[0029] FIG4 is a schematic diagram of the network structure of the recursive aggregation module involved in an embodiment of the present application;
[0030] FIG5 is a schematic diagram of a network structure of a first decoder involved in an embodiment of the present application;
[0031] FIG6 is a schematic diagram of the network structure of a second decoder involved in an embodiment of the present application;
[0032] FIG7 is a schematic diagram of a network structure of a third decoder involved in an embodiment of the present application;
[0033] FIG8 is a schematic diagram of the network structure of the fourth decoder involved in the embodiment of the present application
[0034] FIG9 is a schematic diagram of a network structure of a neural network-based decoder according to an embodiment of the present application;
[0035] FIG10 is a schematic diagram of a video decoding process according to an embodiment of the present application;
[0036] FIG11 is a schematic diagram of a flow chart of a video encoding method provided in an embodiment of the present application;
[0037] FIG12 is a schematic diagram of a network structure of an encoder based on a neural network according to an embodiment of the present application;
[0038] FIG13 is a schematic diagram of a video encoding process according to an embodiment of the present application;
[0039] FIG14 is a schematic block diagram of a video decoding device provided in an embodiment of the present application;
[0040] FIG15 is a schematic block diagram of a video encoding apparatus according to an embodiment of the present application;
[0041] FIG16 is a schematic block diagram of an electronic device provided in an embodiment of the present application;
[0042] FIG17 is a schematic block diagram of a video encoding system provided in an embodiment of the present application. DETAILED DESCRIPTION
[0043] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.
[0044] The present application can be applied to the fields of image coding and decoding, video coding and decoding, hardware video coding and decoding, dedicated circuit video coding and decoding, real-time video coding and decoding, etc. Alternatively, the solution of the present application can be combined with other proprietary or industry standards and operate, such as ITU-TH.261, ISO / IEC MPEG-1 Visual, ITU-TH.262 or ISO / IEC MPEG-2 Visual, ITU-TH.263, ISO / IEC MPEG-4 Visual, ITU-TH.264 (also known as ISO / IEC MPEG-4 AVC), including Scalable Video Coding (SVC) and Multi-View Video Coding (MVC) extensions. It should be understood that the technology of the present application is not limited to any specific codec standard or technology.
[0045] For ease of understanding, the video encoding and decoding system involved in the embodiment of the present application is first introduced with reference to FIG1 .
[0046] FIG1 is a schematic block diagram of a video encoding and decoding system involved in an embodiment of the present application. It should be noted that FIG1 is only an example, and the video encoding and decoding system of the embodiment of the present application includes but is not limited to that shown in FIG1. As shown in FIG1, the video encoding and decoding system 100 includes an encoding device 110 and a decoding device 120. The encoding device is used to encode (which can be understood as compressing) the video data to generate a code stream, and transmit the code stream to the decoding device. The decoding device decodes the code stream generated by the encoding device to obtain decoded video data.
[0047] The encoding device 110 of the embodiment of the present application can be understood as a device with a video encoding function, and the decoding device 120 can be understood as a device with a video decoding function, that is, the embodiment of the present application includes a wider range of devices for the encoding device 110 and the decoding device 120, such as smartphones, desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, car computers, etc.
[0048] In some embodiments, the encoding device 110 may transmit the encoded video data (eg, a code stream) to the decoding device 120 via a channel 130. The channel 130 may include one or more media and / or devices capable of transmitting the encoded video data from the encoding device 110 to the decoding device 120.
[0049] In one example, the channel 130 includes one or more communication media that enable the encoding device 110 to transmit the encoded video data directly to the decoding device 120 in real time. In this example, the encoding device 110 can modulate the encoded video data according to a communication standard and transmit the modulated video data to the decoding device 120. The communication media includes wireless communication media, such as radio frequency spectrum. Optionally, the communication media may also include wired communication media, such as one or more physical transmission lines.
[0050] In another example, channel 130 includes a storage medium that can store the video data encoded by encoding device 110. The storage medium includes various locally accessible data storage media, such as optical disks, DVDs, and flash memories. In this example, decoding device 120 can retrieve the encoded video data from the storage medium.
[0051] In another example, the channel 130 may include a storage server that can store the video data encoded by the encoding device 110. In this example, the decoding device 120 can download the stored encoded video data from the storage server. Alternatively, the storage server can store the encoded video data and transmit the encoded video data to the decoding device 120, such as a web server (e.g., for a website), a file transfer protocol (FTP) server, etc.
[0052] In some embodiments, the encoding device 110 includes a video encoder 112 and an output interface 113. The output interface 113 may include a modulator / demodulator (modem) and / or a transmitter.
[0053] In some embodiments, the encoding device 110 may further include a video source 111 in addition to the video encoder 112 and the input interface 113 .
[0054] The video source 111 may include at least one of a video acquisition device (eg, a video camera), a video archive, a video input interface, and a computer graphics system, wherein the video input interface is used to receive video data from a video content provider, and the computer graphics system is used to generate video data.
[0055] The video encoder 112 encodes the video data from the video source 111 to generate a bitstream. The video data may include one or more pictures or a sequence of pictures. The bitstream contains the coding information of the picture or picture sequence in the form of a bitstream. The coding information may include the coded picture data and associated data. The associated data may include a sequence parameter set (SPS), a picture parameter set (PPS), and other syntax structures. The SPS may contain parameters that apply to one or more sequences. The PPS may contain parameters that apply to one or more pictures. The syntax structure refers to a set of zero or more syntax elements arranged in a specified order in the bitstream.
[0056] The video encoder 112 transmits the encoded video data directly to the decoding device 120 via the output interface 113. The encoded video data may also be stored in a storage medium or a storage server for subsequent reading by the decoding device 120.
[0057] In some embodiments, the decoding device 120 includes an input interface 121 and a video decoder 122 .
[0058] In some embodiments, the decoding device 120 may further include a display device 123 in addition to the input interface 121 and the video decoder 122 .
[0059] The input interface 121 includes a receiver and / or a modem and can receive the encoded video data via the channel 130 .
[0060] The video decoder 122 is configured to decode the encoded video data to obtain decoded video data, and transmit the decoded video data to the display device 123 .
[0061] The decoded video data is displayed on the display device 123. The display device 123 may be integrated with the decoding device 120 or external to the decoding device 120. The display device 123 may include various display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or other types of display devices.
[0062] In addition, Figure 1 is only an example, and the technical solution of the embodiment of the present application is not limited to Figure 1. For example, the technology of the present application can also be applied to unilateral video encoding or unilateral video decoding.
[0063] In some embodiments, the video encoder 112 can be applied to image data in a luminance and chrominance (YCbCr, YUV) format. For example, the YUV ratio can be 4:2:0, 4:2:2, or 4:4:4, where Y represents brightness (Luma), Cb (U) represents blue chrominance, Cr (V) represents red chrominance, and U and V represent chrominance (Chroma) for describing color and saturation. For example, in terms of color format, 4:2:0 means that every 4 pixels have 4 luminance components and 2 chrominance components (YYYYCbCr), 4:2:2 means that every 4 pixels have 4 luminance components and 4 chrominance components (YYYYCbCrCbCr), and 4:4:4 represents full pixel display (YYYYCbCrCbCrCbCrCbCr).
[0064] Because there is a strong correlation between adjacent pixels in a video frame, intra-frame prediction is used in video coding and decoding technology to eliminate spatial redundancy between adjacent pixels. Because there is a strong similarity between adjacent frames in a video, inter-frame prediction is used in video coding and decoding technology to eliminate temporal redundancy between adjacent frames, thereby improving coding efficiency.
[0065] The embodiments of the present application can be used for inter-frame coding to improve the efficiency of inter-frame coding.
[0066] Video coding technology primarily encodes serialized video data, primarily used for data storage, transmission, and presentation in the internet era. Video currently accounts for over 85% of traffic and entry points. As user demands for video data resolution, frame rate, and dimensionality increase, the role and value of video coding technology will also significantly increase. This presents significant opportunities and challenges for both technical advancement and demand for video coding. Traditional video coding technology has undergone decades of development and evolution, significantly satisfying and serving the needs of global video services in each era. Traditional video coding technology has been iteratively updated and continues to be used today within a hybrid coding framework based on multi-scale block-level coding. With the rapid advancement of hardware technology, video coding has significantly improved coding performance through sub-technique advancements, at the expense of complexity. However, the approach of exchanging complexity for performance has become increasingly limited due to hardware bottlenecks, placing higher demands on hardware design and upgrades. Consequently, existing commercially available traditional codecs often require some simplification.
[0067] At the same time, deep learning technology, especially deep neural network technology, is becoming increasingly mature and has been widely researched and applied to various video image tasks, including video enhancement, video detection, and video segmentation. Initially, deep learning technology applied to video coding focused on the research and replacement of sub-techniques in traditional video coding. By studying relevant modules in traditional video coding, the original video coding framework was used as a data generation tool to generate paired training data for training the corresponding neural network. After the neural network converged, the corresponding modules were replaced with the corresponding modules. These replaceable modules include in-loop filtering, out-of-loop filtering, coding block partitioning, and coding block prediction. However, current neural network-based video compression technologies have poor compression performance.
[0068] In order to further improve the video compression effect, this application proposes a purely data-driven neural network coding framework, that is, the entire coding and decoding system is designed, trained and ultimately used for video coding based on deep neural networks, and a new hybrid lossy motion expression method is adopted to realize the neural network-based inter-frame coding and decoding technology.
[0069] The technical solutions provided in the embodiments of the present application are described in detail below in conjunction with specific embodiments.
[0070] First, referring to FIG2 , the decoding end is taken as an example for introduction.
[0071] FIG2 is a flow chart of a video decoding method provided by an embodiment of the present application, which is applied to the video decoder shown in FIG1. As shown in FIG2, the method of the embodiment of the present application includes:
[0072] S201: Decode a first code stream to determine quantized first feature information.
[0073] The first feature information is obtained by performing feature fusion on the current image and a reconstructed image before the current image.
[0074] An embodiment of the present application proposes a neural network-based decoder, which is obtained by end-to-end training of a neural network-based decoder and a neural network-based encoder.
[0075] In the embodiment of the present application, the previous reconstructed image of the current image can be understood as the previous frame image before the current image in the video sequence, and the previous frame image has been decoded and reconstructed.
[0076] Because there is a strong similarity between the two adjacent frames, the current image and the reconstructed image preceding the current image, the encoder performs feature fusion on the current image and the reconstructed image preceding the current image during encoding to obtain first feature information. For example, the encoder concatenates the current image and the reconstructed image preceding the current image and performs feature extraction on the concatenated image to obtain the first feature information. Exemplarily, the encoder uses a feature extraction module to extract features from the concatenated image to obtain the first feature information. This application does not limit the specific network structure of the feature extraction module. The obtained first feature information is of floating-point type, for example, a 32-bit floating-point representation. Furthermore, to reduce encoding cost, the encoder quantizes the obtained first feature information to obtain quantized first feature information. Next, the quantized first feature information is encoded to obtain a first bitstream. For example, the encoder performs arithmetic encoding on the first feature information to obtain the first bitstream. Thus, after obtaining the first bitstream, the decoder decodes the first bitstream to obtain the quantized first feature information and, based on the quantized first feature information, obtains a reconstructed image of the current image.
[0077] In the embodiment of the present application, the decoding end decodes the first bit stream in S201 and determines the quantized first feature information in the following manners, but not limited to:
[0078] In the first approach, if the encoder directly uses the probability distribution of the quantized first feature information to encode the quantized first feature information to obtain the first bitstream, then the decoder directly decodes the first bitstream to obtain the quantized first feature information.
[0079] The quantized first feature information includes a large amount of redundant information. Directly encoding the quantized first feature information requires many codewords, resulting in a high encoding cost. To reduce the encoding cost, in some embodiments, the encoder performs feature transformation based on the first feature information to obtain second feature information, quantizes the second feature information, and then encodes it to obtain a second bitstream. The second bitstream is decoded to obtain quantized second feature information, and the probability distribution of the quantized first feature information is determined based on the quantized second feature information. The quantized first feature information is then encoded based on the probability distribution of the quantized first feature information to obtain the first bitstream. In other words, to reduce the encoding cost, the encoder determines the super-a priori feature information corresponding to the first feature information, i.e., the second feature information, and determines the probability distribution of the quantized first feature information based on the second feature information. Since the second feature information is the super-a priori feature information of the first feature information and includes less redundancy, determining the probability distribution of the quantized first feature information based on the second feature information with less redundancy and encoding the first feature information using this probability distribution can reduce the encoding cost of the first feature information.
[0080] Based on the above description, the decoding end may determine the quantized first feature information through the following steps of the second method.
[0081] Method 2: The above S201 includes the following steps S201-A to S201-C:
[0082] S201-A: Decode the second code stream to obtain quantized second feature information.
[0083] The second feature information is obtained by performing feature transformation on the first feature information.
[0084] As can be seen from the above, to reduce the coding cost, the encoder performs a feature transformation on the first feature information to obtain super-prior feature information of the first feature information, namely the second feature information. This second feature information is used to determine the probability distribution of the quantized first feature information, and this probability distribution is used to encode the quantized first feature information to obtain the first bitstream. At the same time, to enable the decoder to decode the first bitstream using the same probability distribution as the encoding, the second feature information is encoded to obtain the second bitstream. In other words, in this second method, the encoder generates two bitstreams: the first bitstream and the second bitstream.
[0085] After obtaining the first and second bitstreams, the decoder first decodes the second bitstream to determine the probability distribution of the quantized first feature information. Specifically, the decoder decodes the second bitstream to obtain the quantized second feature information. Based on this quantized second feature information, the decoder then determines the probability distribution of the quantized first feature information. Next, the decoder uses the determined probability distribution to decode the first bitstream to obtain the quantized first feature information, thereby accurately decoding the first feature information.
[0086] In this application, because the second feature information is the super-a priori feature information of the first feature information and includes less redundant information, the encoder can directly use the probability distribution of the quantized second feature information during encoding to encode the quantized second feature information and obtain the second bitstream. Correspondingly, the decoder can directly decode the second bitstream to obtain the quantized second feature information.
[0087] S201-B. Determine the probability distribution of the quantized first feature information based on the quantized second feature information.
[0088] After determining the quantized second feature information according to the above steps, the decoding end determines the probability distribution of the quantized first feature information based on the quantized second feature information.
[0089] In the embodiment of the present application, there is no limitation on the specific method of determining the probability distribution of the quantized first feature information based on the quantized second feature information in the above S201-B.
[0090] In some embodiments, since the second feature information is obtained by performing feature transformation on the first feature information, based on this, S201-B includes the following steps S201-B1 to S201-B3:
[0091] S201-B1. Perform inverse transformation on the quantized second feature information to obtain reconstructed feature information.
[0092] In this implementation, the decoder performs an inverse transformation on the quantized second feature information to obtain reconstructed feature information. The inverse transformation method used by the decoder can be understood as the inverse operation of the transformation method used by the encoder. For example, the encoder performs feature extraction N times on the first feature information to obtain the second feature information. Correspondingly, the decoder performs feature extraction N times in the opposite direction on the quantized second feature information to obtain inversely transformed feature information, which is recorded as the reconstructed feature information.
[0093] The embodiment of the present application does not limit the inverse transformation method adopted by the decoding end.
[0094] In some embodiments, the inverse transformation method adopted by the decoding end includes N times of feature extraction. That is, the decoding end performs N times of feature extraction on the obtained quantized second feature information to obtain reconstructed feature information.
[0095] In some embodiments, the inverse transformation method used by the decoding end includes N times of feature extraction and N times of upsampling. In other words, the decoding end performs N times of feature extraction and N times of upsampling on the obtained quantized second feature information to obtain reconstructed feature information.
[0096] The embodiment of the present application does not limit the specific execution order of the above-mentioned N feature extractions and N upsamplings.
[0097] In an example, the decoding end may first perform N consecutive feature extractions on the quantized second feature information, and then perform N consecutive upsamplings.
[0098] In another example, the above-mentioned N feature extractions and N upsamplings are performed alternately, that is, upsampling is performed once after performing one feature extraction. For example, assuming N=2, the decoding end performs an inverse transform on the quantized second feature information to obtain the reconstructed feature information. The specific process is: inputting the quantized second feature information into the first feature extraction module for the first feature extraction to obtain feature information 1, upsampling feature information 1 to obtain feature information 2, inputting feature information 2 into the second feature extraction module for the second feature extraction to obtain feature information 3, upsampling feature information 3 to obtain feature information 4, and recording feature information 4 as the reconstructed feature information.
[0099] It should be noted that the embodiment of the present application does not limit the N-time feature extraction method adopted by the decoding end, for example, at least one of the feature extraction methods including multi-layer convolution, residual connection, dense connection, etc.
[0100] In some embodiments, the decoding end performs feature extraction using a non-local attention method. In this case, the above S201-B1 includes the following steps S201-B11:
[0101] S201-B11. Perform N non-local attention transformations and N upsamplings on the quantized second feature information to obtain reconstructed feature information, where N is a positive integer.
[0102] Since the non-local attention method can achieve more efficient feature extraction, it can make the extracted features retain more information and has high computational efficiency. Therefore, in the embodiment of the present application, the decoding end uses the non-local attention method to extract the features of the quantized second feature information to achieve fast and accurate feature extraction of the quantized second feature information. In addition, when the encoding end generates the second feature information based on the first feature information, it performs N downsamplings. Therefore, the decoding end performs N upsamplings accordingly to make the reconstructed feature information consistent with the size of the first feature information.
[0103] In some embodiments, as shown in FIG3 , the decoding end obtains reconstructed feature information through an inverse transformation module, and the inverse transformation module includes N non-local attention modules and N upsampling modules. Among them, the non-local attention module is used to implement non-local attention transformation, and the upsampling module is used to implement upsampling. Exemplarily, as shown in FIG3 , an upsampling module is connected after a non-local attention module. In actual application, the decoding end inputs the quantized second feature information obtained by decoding into the inverse transformation module. The first non-local attention module in the inverse transformation module performs non-local attention feature transformation extraction on the quantized second feature information to obtain feature information 1, and then inputs feature information 1 into the first upsampling module for upsampling to obtain feature information 2. Next, feature information 2 is input into the second non-local attention module for non-local attention feature transformation extraction to obtain feature information 3, and then inputs feature information 3 into the second upsampling module for upsampling to obtain feature information 4. By analogy, the feature information output by the Nth upsampling module is obtained, and the feature information is determined as the reconstructed feature information.
[0104] S201-B2: Determine the probability distribution of reconstructed feature information.
[0105] From the above, it can be seen that the second quantized feature information is obtained by transforming the first feature information. The decoding end dequantizes the quantized second feature information through the above steps to obtain reconstructed feature information. Therefore, the reconstructed feature information can be understood as the reconstructed information of the first feature information, that is, the probability distribution of the reconstructed feature information is similar to or related to the probability distribution of the quantized first feature information. In this way, the decoding end can first determine the probability distribution of the reconstructed feature information, and then predict the probability distribution of the first feature information after quantization based on the probability distribution of the reconstructed feature information.
[0106] In some embodiments, the probability distribution of the reconstructed feature information is a normal distribution or a Gaussian distribution. In this case, the process of determining the probability distribution of the reconstructed feature information is to determine the mean and variance matrix of the reconstructed feature information based on each eigenvalue in the reconstructed feature information, and generate the Gaussian distribution of the reconstructed feature information based on the mean and variance matrix.
[0107] S201-B3: Predict the probability distribution of the quantized first feature information based on the probability distribution of the reconstructed feature information.
[0108] Since the reconstructed feature information is the reconstructed information of the first feature information, the probability distribution of the reconstructed feature information is similar to or related to the probability distribution of the quantized first feature information. Therefore, the embodiment of the present application can achieve accurate prediction of the probability distribution of the quantized first feature information through the probability distribution of the reconstructed feature information.
[0109] The embodiment of the present application does not limit the specific implementation method of the above S201-B3.
[0110] In a possible implementation manner, the probability distribution of the reconstructed feature information is determined as the probability distribution of the quantized first feature information.
[0111] In another possible implementation, the probability of the encoded pixels in the quantized first feature information is predicted based on the probability distribution of the reconstructed feature information; and the probability distribution of the quantized first feature information is obtained based on the probability of the encoded pixels in the quantized first feature information.
[0112] S201-C. Decode the first bitstream according to the probability distribution of the quantized first feature information to obtain the quantized first feature information.
[0113] According to the above steps, after the probability distribution of the quantized first feature information is determined, the first code stream is decoded using the probability distribution, thereby achieving accurate decoding of the quantized first feature information.
[0114] In the embodiment of the present application, the decoding end decodes the first bit stream according to the above-mentioned method 1 or method 2, determines the quantized first feature information, and then performs the following step S202.
[0115] S202 , performing multi-level time-domain fusion on the quantized first feature information to obtain a mixed spatiotemporal representation.
[0116] In an embodiment of the present application, in order to improve the accuracy of the reconstructed image, the quantized first feature information is subjected to multi-level time domain fusion, that is, the quantized first feature information is not only fused with the feature information of the previous reconstructed image of the current image, but also the quantized first feature information is fused with the feature information of multiple reconstructed images before the current image, for example, the reconstructed images at multiple times such as time t-1, time t-2..., time tk are fused with the quantized first feature information. This can avoid the situation where certain information in the previous reconstructed image of the current image is blocked, and the blocked information can be obtained from several reconstructed images before the current image, thereby making the generated mixed spatiotemporal representation include more accurate, rich and detailed feature information. In this way, when motion compensation is performed on the previous reconstructed image based on the mixed spatiotemporal representation to generate P predicted images of the current image, the accuracy of the generated predicted image can be improved, and then based on the accurate predicted image, the reconstructed image of the current image can be accurately obtained, thereby improving the video compression effect.
[0117] The embodiment of the present application does not limit the specific method of performing multi-level time-domain fusion on the quantized first feature information at the decoding end to obtain a mixed spatiotemporal representation.
[0118] In some embodiments, the decoding end mixes the spatiotemporal representations through a recursive aggregation module, that is, the above S202 includes the following step S202-A:
[0119] S202-A: The decoding end fuses the quantized first feature information with the implicit feature information of the recursive aggregation module at the previous moment through the recursive aggregation module to obtain a mixed spatiotemporal representation.
[0120] The recursive aggregation module of the embodiment of the present application learns and retains the deep-level feature information learned from the feature information each time it generates a mixed spatiotemporal representation, and uses the learned deep-level features as implicit feature information to act on the next generation of the mixed spatiotemporal representation, thereby improving the accuracy of the generated mixed spatiotemporal representation. That is to say, in the embodiment of the present application, the implicit feature information of the recursive aggregation module at the previous moment includes the feature information of multiple reconstructed images before the current image learned by the recursive aggregation module. In this way, the decoding end fuses the quantized first feature information with the implicit feature information of the recursive aggregation module at the previous moment through the recursive aggregation module, thereby generating a more accurate, rich and detailed mixed spatiotemporal representation.
[0121] The embodiment of the present application does not limit the specific network structure of the recursive aggregation module, for example, it can be any network structure that can implement the above functions.
[0122] In some embodiments, the recursive aggregation module is formed by stacking at least one spatiotemporal recursive network ST-LSTM. In this case, the expression formula of the hybrid spatiotemporal representation Gt is as shown in formula (1):
[0123]
[0124] in, is the first feature information after quantization, and h is the implicit feature information included in ST-LSTM.
[0125] In one example, assuming that the recursive aggregation module consists of two ST-LSTMs, as shown in Figure 4, the decoding end reconstructs the quantized first feature information Input into the recursive aggregation module, the two ST-LSTM in the recursive aggregation module sequentially quantize the first feature information Processing is performed to generate feature information. Specifically, as shown in Figure 4, the implicit feature information h1 generated by the first ST-LSTM is used as the input of the next ST-LSTM, and the two ST-LSTMs generate transmission band update values c1 and c2 respectively during this operation to update their respective transmission band values, where m is the memory information, which is transferred between the two ST-LSTMs, and finally the feature information h2 output by the second ST-LSTM is obtained. Furthermore, in order to improve the accuracy of the generated hybrid spatiotemporal representation, the feature information h2 generated by the second ST-LSTM is combined with the quantized first feature information. Perform residual connection, that is, the feature information h generated by the second ST-LSTM is connected with the quantized first feature information Add them together to generate a mixed spatiotemporal representation Gt.
[0126] After the decoder obtains the mixed spatiotemporal representation according to the above method, it executes the following S203.
[0127] S203 : Perform motion compensation on the previous reconstructed image according to the hybrid spatiotemporal representation to obtain P predicted images of the current image.
[0128] Wherein, P is a positive integer.
[0129] From the above, it can be seen that the hybrid spatiotemporal representation of the embodiment of the present application fuses the feature information of the current image and multiple reconstructed images before the current image, so that motion compensation is performed on the previous reconstructed image based on the hybrid spatiotemporal representation, and P accurate predicted images of the current image can be obtained.
[0130] The embodiment of the present application does not limit the specific number of the generated P predicted images. That is, in the embodiment of the present application, the decoding end can use different methods to perform motion compensation on the previous reconstructed image based on the hybrid spatiotemporal representation to obtain P predicted images of the current image.
[0131] The embodiment of the present application does not limit the specific manner in which the decoding end performs motion compensation on the previous reconstructed image based on the mixed spatiotemporal representation.
[0132] In some embodiments, the P predicted images include a first predicted image, which is obtained by the decoding end using optical flow motion compensation. In this case, S203 includes the following steps S203-A1 and S203-A2:
[0133] S203-A1, determining optical flow motion information based on the mixed spatiotemporal representation;
[0134] S203-A2: Perform motion compensation on the previous reconstructed image according to the optical flow motion information to obtain a first predicted image.
[0135] The embodiment of the present application does not limit the specific manner in which the decoding end determines the optical flow motion information based on the mixed spatiotemporal representation.
[0136] In some embodiments, the decoding end obtains the optical flow motion information through a pre-trained neural network model, that is, the neural network model can predict the optical flow motion information based on the mixed spatiotemporal representation. In some embodiments, the neural network model can be called a first decoder, or an optical flow signal decoder Df. The decoding end inputs the mixed spatiotemporal representation Gt into the optical flow signal decoder Df to predict the optical flow motion information, and obtains the optical flow motion information f output by the optical flow signal decoder Df. x,y . Optionally, the f x,y is the optical flow motion information with channel 2.
[0137] For example, f x,y The generation formula is shown in formula (2):
[0138] f x,y =D f (G t ) (2)
[0139] The embodiment of the present application does not limit the specific network structure of the above-mentioned optical flow signal decoder Df.
[0140] In some embodiments, the optical flow signal decoder Df is composed of multiple NLAMs and multiple upsampling modules. For example, as shown in Figure 5, the optical flow signal decoder Df includes 1 NLAM, 3 LAMs and 4 downsampling modules, wherein one NLAM is connected to a downsampling module, and one LAM is connected to a downsampling module. Optionally, the NLAM includes multiple convolutional layers, for example, including 3 convolutional layers, each of which has a convolution kernel size of 3*3 and a channel number of 192. Optionally, the three LAMs each include multiple convolutional layers, for example, each including 3 convolutional layers, each of which has a convolution kernel size of 3*3, and the number of channels of the convolution layers included in the three LAMs is 128, 96 and 64, respectively. Optionally, the four downsampling modules each include a convolution layer Conv, the convolution kernel size of the convolution layer is 5*5, and the number of channels of the convolution layers included in the four downsampling modules is 128, 96, 64 and 2, respectively. In this way, the decoder inputs the mixed spatiotemporal representation Gt into the optical flow signal decoder Df. The NLAM performs feature extraction on the spatiotemporal representation Gt, obtaining feature information a with 192 channels. This feature information a is then input into the first downsampling module for downsampling, resulting in feature information b with 128 channels. Feature information b is then input into the first LAM for feature re-extraction, obtaining feature information c with 128 channels. This feature information c is then input into the second downsampling module for downsampling, resulting in feature information d with 96 channels. Feature information d is then input into the second LAM for feature re-extraction, obtaining feature information e with 96 channels. This feature information e is then input into the third downsampling module for downsampling, resulting in feature information f with 64 channels. Feature information f is then input into the third LAM for feature re-extraction, obtaining feature information g with 64 channels. This feature information g is then input into the fourth downsampling module for downsampling, resulting in feature information j with 2 channels. Feature information j is the optical flow motion information.
[0141] It should be noted that FIG5 is only an example, and the settings of the parameters in FIG5 are also only examples. The network structure of the optical flow signal decoder Df in the embodiment of the present application includes but is not limited to that shown in FIG5 .
[0142] The decoding end generates optical flow motion information f x,y Then, use the optical flow motion information f x,y Reconstruct the previous image Perform motion compensation to obtain the first predicted image X1.
[0143] In the embodiment of the present application, the decoding end performs motion compensation on the previous reconstructed image according to the optical flow motion information to obtain the first predicted image without limiting the specific method. For example, the decoding end uses the optical flow motion information f x,y Reconstruct the previous image Perform linear interpolation and record the interpolated image as the first predicted image X1.
[0144] In a possible implementation, the decoding end obtains the first predicted image X1 by the following formula (3):
[0145]
[0146] In this implementation, as shown in FIG5 , the decoding end uses the optical flow motion information fxy to reconstruct the previous image by warping operation. Perform motion compensation to obtain the first predicted image X1.
[0147] In some embodiments, the P predicted images include a second predicted image, and the second predicted image is obtained by the decoding end using an offset motion compensation method. In this case, the above S203 includes the following steps S203-B1 to S203-B3:
[0148] S203-B1, obtaining an offset corresponding to the current image according to the mixed spatiotemporal representation;
[0149] S203-B2, extracting spatial features from the previous reconstructed image to obtain reference feature information;
[0150] S203-B3. Use the offset to perform motion compensation on the reference feature information to obtain a second predicted image.
[0151] The embodiment of the present application does not limit the specific manner in which the decoding end obtains the offset corresponding to the current image based on the mixed spatiotemporal representation.
[0152] In some embodiments, the decoder obtains the offset corresponding to the current image using a pre-trained neural network model. Specifically, the neural network model can predict the offset based on the hybrid spatiotemporal representation. This offset is lossy offset information. In some embodiments, the neural network model can be referred to as a second decoder, or a variable convolutional decoder Dm. The decoder inputs the hybrid spatiotemporal representation Gt into the variable convolutional decoder Dm to predict the offset information.
[0153] At the same time, the decoding end performs spatial feature extraction on the previous reconstructed image to obtain reference feature information. For example, the decoding end performs spatial feature extraction on the previous reconstructed image through a spatial feature extraction module SFE to obtain reference feature information.
[0154] Next, the decoding end uses the offset to perform motion compensation on the extracted reference feature information to obtain a second predicted image of the current image.
[0155] In the embodiment of the present application, the decoding end uses the offset to perform motion compensation on the extracted reference feature information to obtain the second predicted image of the current image without limiting the specific method.
[0156] In a possible implementation, the decoding end uses the offset to perform deformable convolution-based motion compensation on the reference feature information to obtain a second predicted image.
[0157] In some embodiments, since the transformable convolution can generate an offset corresponding to the current image based on the mixed spatiotemporal representation, in an embodiment of the present application, the decoding end inputs the mixed spatiotemporal representation Gt and the reference feature information into the transformable convolution. The transformable convolution generates an offset corresponding to the current image based on the mixed spatiotemporal representation Gt, and applies the offset to the reference feature information for motion compensation, thereby obtaining a second predicted image.
[0158] Based on this, for example, as shown in FIG6 , the variable convolution decoder Dm of the embodiment of the present application includes a transformable convolution DCN, and the decoding end reconstructs the previous image The mixed spatiotemporal representation Gt and the reference feature information are then input into the transformable convolution DCN for offset extraction and motion compensation to obtain the second predicted image X2.
[0159] Exemplarily, the decoding end generates the second predicted image X2 by formula (4):
[0160]
[0161] The embodiment of the present application does not limit the specific network structure of the above-mentioned optical flow signal decoder Df.
[0162] In some embodiments, as shown in FIG6 , in order to further improve the accuracy of the second predicted image, the variable convolution decoder Dm includes, in addition to the transformable convolution DCN, 1 NLAM, 3 LAMs, and 4 downsampling modules, wherein one NLAM is followed by a downsampling module, and one LAM is followed by a downsampling module. Optionally, the network structure of the 1 NLAM, 3 LAMs, and the first 3 downsampling modules included in the variable convolution decoder Dm is the same as the network structure of the 1 NLAM, 3 LAMs, and the first 3 downsampling modules included in the above-mentioned optical flow signal decoder Df, and will not be repeated here. Optionally, the number of channels included in the last downsampling module included in the variable convolution decoder Dm is 5.
[0163] It should be noted that the above Figure 6 is only an example, and the settings of the various parameters in Figure 6 are also only examples. The network structure of the variable convolution decoder Dm in the embodiment of the present application includes but is not limited to that shown in Figure 6.
[0164] In the embodiment of the present application, as shown in FIG6 , the decoding end first reconstructs the previous image The mixed spatiotemporal representation Gt and the reference feature information are then fed into the transformable convolutional network (DCN) in the variable convolutional decoder (Dm) for offset extraction and motion compensation. This generates a feature, which is then fed into the NLAM. After feature extraction by the NLAM, three LAMs, and four downsampling modules, the second predicted image X2 is finally restored.
[0165] According to the above method, the decoding end can determine P predicted images, for example, determine the first predicted image and the second predicted image, and then perform the following step S204.
[0166] S204: Determine a reconstructed image of the current image based on the P predicted images.
[0167] In some embodiments, if the P predicted images include one predicted image, a reconstructed image of the current image is determined based on the predicted image.
[0168] For example, the predicted image is compared with one or several previous reconstructed images of the current image to calculate the loss. If the loss is small, it means that the prediction accuracy of the predicted image is high, and the predicted image can be determined as the reconstructed image of the current image.
[0169] For another example, if the above-mentioned loss is large, it means that the prediction accuracy of the predicted image is low. At this time, the reconstructed image of the current image can be determined based on the previous one or several reconstructed images of the current image and the predicted image. For example, the predicted image and the previous one or several reconstructed images of the current image are input into a neural network to obtain the reconstructed image of the current image.
[0170] In some embodiments, the above S204 includes the following steps S204-A and S204-B:
[0171] S204-A: Determine a target predicted image for the current image based on the P predicted images.
[0172] In this implementation, the decoding end first determines the target prediction image of the current image based on P prediction images, and then realizes the reconstructed image of the current image based on the target prediction image of the current image, thereby improving the accuracy of determining the reconstructed image.
[0173] The embodiment of the present application does not limit the specific method of determining the target predicted image of the current image based on P predicted images.
[0174] In some embodiments, if P=1, the one predicted image is determined as the target predicted image of the current image.
[0175] In some embodiments, if P is greater than 1, S204-A includes S204-A11 and S204-A12:
[0176] S204-A11, determining a weighted image based on the P predicted images;
[0177] In this implementation, if multiple predicted images of the current image are generated according to the above method, for example, when a first predicted image and a second predicted image are generated, these P predicted images are weighted to generate a weighted image, and then the target predicted image is obtained based on the weighted image.
[0178] The embodiment of the present application does not limit the specific method of determining the weighted image based on P predicted images.
[0179] For example, weights corresponding to P predicted images are determined; and according to the weights corresponding to the P predicted images, the P predicted images are weighted to obtain a weighted image.
[0180] Exemplarily, if the P predicted images include a first predicted image and a second predicted image, the decoding end determines a first weight corresponding to the first predicted image and a second weight corresponding to the second predicted image, and weights the first predicted image and the second predicted image according to the first weight and the second weight to obtain a weighted image.
[0181] The methods for determining the weights corresponding to the P prediction images include but are not limited to the following:
[0182] In a first approach, the weights corresponding to the P predicted images are preset weights. Assuming P = 2, the first weight corresponding to the first predicted image and the second weight corresponding to the second predicted image can be equal to the second weight, or the ratio of the first weight to the second weight can be 1 / 2, 1 / 4, 1 / 2, 1 / 3, 2 / 1, 3 / 1, 4 / 1, and so on.
[0183] In the second method, the decoding end performs adaptive masking based on the mixed spatiotemporal representation to obtain the weights corresponding to P predicted images.
[0184] Exemplarily, the decoding end generates weights corresponding to the P predicted images through a neural network model. The neural network model is pre-trained and can be used to generate weights corresponding to the P predicted images. In some embodiments, the neural network model is also called a third decoder or an adaptive mask compensation decoder D w Specifically, the decoding end inputs the mixed spatiotemporal representation into the adaptive mask compensation decoder D wFor example, the decoder inputs the mixed spatiotemporal representation Gt into the adaptive mask compensation decoder D w Adaptive mask compensation decoder D w Output the first weight w1 of the first predicted image and the second weight w2 of the second predicted image, perform the above-obtained first predicted image X1 and second predicted image X2 according to the first weight w1 and second weight w2, and adaptively select the corresponding information representing different areas in the predicted frame to generate a weighted image.
[0185] Exemplarily, the weighted image X3 is generated according to the following formula (5):
[0186] X3=w1*X1+w2*X2 (5)
[0187] In some embodiments, the weights corresponding to the above-mentioned P predicted images are a matrix, which includes the weights corresponding to each pixel in the predicted image. In this way, when generating a weighted image, for each pixel in the current image, the predicted values and their weights corresponding to the pixel in the P predicted images are weighted to obtain the weighted predicted value of the pixel. In this way, the weighted predicted values corresponding to each pixel in the current image constitute the weighted image of the current image.
[0188] The embodiment of the present application is to improve the adaptive mask compensation decoder D w There is no restriction on the specific network structure.
[0189] In some embodiments, as shown in FIG7 , the adaptive mask compensation decoder D w It includes 1 NLAM, 3 LAMs, 4 downsampling modules and a sigmoid function, where one NLAM is followed by a downsampling module and one LAM is followed by a downsampling module. w The network structure of 1 NLAM, 3 LAMs, and 4 downsampling modules included is consistent with the network structure of 1 NLAM, 3 LAMs, and 4 downsampling modules included in the above-mentioned variable convolution decoder Dm, and will not be repeated here.
[0190] It should be noted that FIG7 is only an example, and the settings of the parameters in FIG7 are also only examples. The adaptive mask compensation decoder D in the embodiment of the present application w The network structure includes but is not limited to that shown in FIG7 .
[0191] In this implementation, the decoding end weights the P predicted images according to the above method, obtains the weighted image, and then executes the following S204-A12.
[0192] S204-A12: Obtain a target predicted image based on the weighted image.
[0193] For example, the weighted image is determined as the target prediction image.
[0194] In some embodiments, the decoding end may also obtain a residual image of the current image based on the mixed spatiotemporal representation.
[0195] Exemplarily, the decoding end obtains a residual image of the current image through a neural network model. The neural network model is pre-trained and can be used to generate the residual image of the current image. In some embodiments, the neural network model is also called a fourth decoder or a spatial texture enhancement decoder Dt. Specifically, the decoding end inputs the mixed spatiotemporal representation into the spatial texture enhancement decoder Dt for spatial texture enhancement to obtain a residual image X of the current image. r =D_t(G t ), the residual image X r The predicted image can be texture enhanced.
[0196] In the embodiment of the present application, there is no limitation on the specific network structure of the above-mentioned spatial texture enhancement decoder Dt.
[0197] In some embodiments, as shown in FIG8 , the spatial texture enhancement decoder Dt includes one NLAM, three LAMs, and four downsampling modules, wherein one NLAM is followed by a downsampling module, and one LAM is followed by a downsampling module. Optionally, the network structure of the one NLAM, three LAMs, and the first three downsampling modules included in the spatial texture enhancement decoder Dt is consistent with the network structure of the one NLAM, three LAMs, and the first three downsampling modules included in the above-mentioned optical flow signal decoder Df, and will not be repeated here. The number of channels included in the last downsampling module of the spatial texture enhancement decoder Dt is 3.
[0198] It should be noted that the above FIG8 is only an example, and the settings of the parameters in FIG8 are also only examples. The network structure of the spatial texture enhancement decoder Dt in the embodiment of the present application includes but is not limited to that shown in FIG8.
[0199] Since the above residual image X r Texture enhancement can be performed on the predicted image. Based on this, in some embodiments, determining the target predicted image of the current image based on the P predicted images in S204-A includes the following steps S204-A21:
[0200] S204-A21. Obtain a target prediction image based on the P prediction images and the residual image.
[0201] For example, if P=1, a target predicted image is obtained based on the predicted image and the residual image. For example, the predicted image and the residual image are added to generate the target predicted image.
[0202] For another example, if P is greater than 1, a weighted image is first determined based on the P predicted images; and then a target predicted image is determined based on the weighted image and the residual image.
[0203] The specific process of determining the weighted image based on the P predicted images at the decoding end can refer to the specific description of S204-A11 above, which will not be repeated here.
[0204] For example, taking P = 2 as an example, according to the above method, the first weight w1 corresponding to the first predicted image and the second weight w2 corresponding to the second predicted image are determined. Optionally, the first predicted image and the second predicted image are weighted according to the above formula (5) to obtain a weighted image X3. Then, the residual image X is used r The weighted image X3 is enhanced to obtain the target predicted image.
[0205] Exemplarily, the target predicted image X4 is generated according to the following formula (6):
[0206] X4=X3+X r (6)
[0207] According to the above method, after the decoding end determines the target predicted image of the current image, it executes the following step S204-B.
[0208] S204-B: Determine a reconstructed image of the current image based on the target predicted image.
[0209] In some embodiments, the target predicted image is compared with one or more reconstructed images preceding the current image to calculate a loss. If the loss is small, it indicates that the prediction accuracy of the target predicted image is high, and the target predicted image can be determined as the reconstructed image of the current image. If the loss is large, it indicates that the prediction accuracy of the target predicted image is low. In this case, the reconstructed image of the current image can be determined based on one or more reconstructed images preceding the current image and the target predicted image. For example, the target predicted image and one or more reconstructed images preceding the current image are input into a neural network to obtain the reconstructed image of the current image.
[0210] In some embodiments, in order to further improve the accuracy of determining the reconstructed image, the embodiment of the present application further includes residual decoding. In this case, the above S204-B includes the following steps S204-B1 and S204-B2:
[0211] S204-B1, decoding the residual code stream to obtain a residual value of the current image;
[0212] S204-B2: Obtain a reconstructed image based on the target predicted image and the residual value.
[0213] In this embodiment of the present application, to improve the quality of the reconstructed image, the encoder also generates a residual bitstream through residual coding. Specifically, the encoder determines the residual value of the current image and encodes this residual value to generate a residual bitstream. Correspondingly, the decoder decodes the residual bitstream to obtain the residual value of the current image, and then obtains the reconstructed image based on the target predicted image and the residual value.
[0214] The embodiment of the present application does not limit the specific representation of the residual value of the current image.
[0215] In one possible implementation, the residual value of the current image is a matrix, and each element in the matrix is the residual value corresponding to each pixel in the current image. In this way, the decoding end can add the residual value and the predicted value corresponding to each pixel in the target predicted image pixel by pixel to obtain the reconstruction value of each pixel, and then obtain the reconstructed image of the current image. Taking the i-th pixel in the current image as an example, in the target predicted image, the predicted value corresponding to the i-th pixel is obtained, and the residual value corresponding to the i-th pixel is obtained from the residual value of the current image. Then, the predicted value and the residual value corresponding to the i-th pixel are added to obtain the reconstructed value corresponding to the i-th pixel. For each pixel in the current image, referring to the above i-th pixel, the reconstruction value corresponding to each pixel in the current image can be obtained, and the reconstruction value corresponding to each pixel in the current image constitutes the reconstructed image of the current image.
[0216] The embodiment of the present application does not limit the specific method for the decoding end to obtain the residual value of the current image. That is to say, the embodiment of the present application does not limit the residual encoding and decoding method adopted by both ends of the encoding and decoding.
[0217] In one example, the encoding end determines the target prediction image of the current image in the same manner as the decoding end, and then obtains the residual value of the current image based on the current image and the target prediction image. For example, the difference between the current image and the target prediction image is determined as the residual value of the current image. Then, the residual value of the current image is encoded to generate a residual code. Optionally, the residual value of the current image can be transformed to obtain a transformation coefficient, the transformation coefficient can be quantized to obtain a quantization coefficient, and the quantization coefficient can be encoded to obtain a residual code stream. Correspondingly, the decoding end decodes the residual code stream to obtain the residual value of the current image, for example, decodes the residual code stream to obtain a quantization coefficient, and dequantizes and detransforms the quantization coefficient to obtain the residual value of the current image. Then, according to the above method, the residual values corresponding to the target prediction image and the current image are added to obtain a reconstructed image of the current image.
[0218] In some embodiments, the encoder can use a neural network approach to process the current image and its target predicted image to generate a residual value for the current image, and then encode the residual value of the current image to generate a residual bitstream. Correspondingly, the decoder decodes the residual bitstream to obtain the residual value of the current image. Then, according to the aforementioned method, the residual values corresponding to the target predicted image and the current image are added together to obtain a reconstructed image of the current image.
[0219] In the embodiment of the present application, the decoding end can obtain a reconstructed image of the current image according to the above method.
[0220] Optionally, the reconstructed image can be displayed directly.
[0221] Optionally, the reconstructed image may be stored in a cache for use in subsequent image decoding.
[0222] The video decoding method provided in the embodiment of the present application is as follows: the decoding end determines the quantized first feature information by decoding the first bit stream, and the first feature information is obtained by feature fusion of the current image and the previous reconstructed image of the current image; the quantized first feature information is subjected to multi-level time domain fusion to obtain a mixed spatiotemporal representation; motion compensation is performed on the previous reconstructed image based on the mixed spatiotemporal representation to obtain P predicted images of the current image, where P is a positive integer; and the reconstructed image of the current image is determined based on the P predicted images. In order to improve the accuracy of the reconstructed image, the present application performs multi-level time domain fusion on the quantized first feature information, that is, the quantized first feature information is not only fused with the feature information of the previous reconstructed image of the current image, but also the quantized first feature information is feature fused with multiple reconstructed images before the current image. This can avoid the situation where certain information in the previous reconstructed image of the current image is blocked, and the blocked information can be obtained from several reconstructed images before the current image, thereby making the generated mixed spatiotemporal representation include more accurate, rich and detailed feature information. In this way, when motion compensation is performed on the previous reconstructed image based on the mixed spatiotemporal representation, P high-precision predicted images can be generated. Based on the P high-precision predicted images, the reconstructed image of the current image can be accurately obtained, thereby improving the video compression effect.
[0223] In an embodiment of the present application, an end-to-end neural network-based encoding and decoding framework is proposed, which includes a neural network-based encoder and a neural network-based decoder. The decoding process of the embodiment of the present application is introduced below in conjunction with a possible neural network-based decoder of the present application.
[0224] FIG9 is a schematic diagram of the network structure of a neural network-based decoder according to an embodiment of the present application, including an inverse transformation module, a recursive aggregation module, and a hybrid motion compensation module.
[0225] The inverse transformation module is used to perform inverse transformation on the quantized second feature information to obtain reconstructed feature information of the first feature information. Exemplarily, its network structure is shown in FIG3 .
[0226] The recursive aggregation module is used to perform multi-level time-domain fusion on the quantized first feature information to obtain a mixed spatiotemporal representation. Exemplarily, its network structure is shown in FIG4 .
[0227] The hybrid motion compensation module is used to perform hybrid motion compensation on the hybrid spatiotemporal representation to obtain a target predicted image of the current image. Exemplarily, the hybrid motion compensation module may include the first decoder shown in FIG5 and / or the second decoder shown in FIG6. Optionally, if the hybrid motion compensation module includes the first decoder and the second decoder, the hybrid motion compensation module may also include the third decoder shown in FIG7. In some embodiments, the hybrid motion compensation module may also include a fourth decoder shown in FIG8.
[0228] Illustratively, the embodiment of the present application is described by taking an example where the motion compensation module includes a first decoder, a second decoder, a third decoder, and a fourth decoder.
[0229] Based on the neural network-based decoder shown in FIG9 above, a possible video decoding method of an embodiment of the present application is introduced in combination with FIG10.
[0230] FIG10 is a schematic diagram of a video decoding process according to an embodiment of the present application, as shown in FIG10 , including:
[0231] S301: Decode the second code stream to obtain quantized second feature information.
[0232] The specific implementation process of the above S301 refers to the description of the above S201-A and will not be repeated here.
[0233] S302 : Perform an inverse transformation on the quantized second feature information through an inverse transformation module to obtain reconstructed feature information.
[0234] Exemplarily, the specific network structure of the inverse transformation module is shown in Figure 3, including two non-local self-attention modules and two upsampling modules.
[0235] For example, the decoding end inputs the quantized second feature information into the inverse transformation module for inverse transformation, and the inverse transformation module outputs the reconstructed feature information. The specific implementation process of the above S302 refers to the description of the above S201-B1, which will not be repeated here.
[0236] S303: Determine the probability distribution of the reconstructed feature information.
[0237] S304: Predict the probability distribution of the quantized first feature information according to the probability distribution of the reconstructed feature information.
[0238] S305 : Decode the first bitstream according to the probability distribution of the quantized first feature information to obtain the quantized first feature information.
[0239] The specific implementation process of the above S303 to S305 refers to the specific description of the above S201-B2, S201-B3 and S201-C, which will not be repeated here.
[0240] S306 , performing multi-level time-domain fusion on the quantized first feature information through a recursive aggregation module to obtain a mixed spatiotemporal representation.
[0241] Optionally, the recursive aggregation module is formed by stacking at least one spatiotemporal recursive network.
[0242] Exemplarily, the network structure of the recursive aggregation module is shown in FIG4 .
[0243] For example, the decoding end inputs the quantized first feature information into the recursive aggregation module, so that the recursive aggregation module fuses the quantized first feature information with the implicit feature information of the recursive aggregation module at the previous moment, thereby outputting a hybrid spatiotemporal representation. The specific implementation process of the above S306 is described in the above S202-A and is not repeated here.
[0244] S307 : Process the mixed spatiotemporal representation through a first decoder to obtain a first predicted image.
[0245] After the hybrid spatiotemporal representation is obtained according to the above S306, the hybrid spatiotemporal representation and the previous reconstructed image are input into the hybrid motion compensation module for motion hybrid compensation to obtain the target predicted image of the current image.
[0246] Specifically, the mixed spatiotemporal representation is processed by a first decoder to determine optical flow motion information, and motion compensation is performed on a previous reconstructed image according to the optical flow motion information to obtain a first predicted image.
[0247] Optionally, the network structure of the first decoder is shown in FIG5 .
[0248] The specific implementation process of the above S307 refers to the specific description of the above S203-A1 and S203-A2, which will not be repeated here.
[0249] S308 : Process the mixed spatiotemporal representation through a second decoder to obtain a second predicted image.
[0250] Specifically, spatial features of the previous reconstructed image are extracted through SFE to obtain reference feature information; the reference feature information and the mixed spatiotemporal representation are input into the second decoder so that the offset performs motion compensation on the reference feature information to obtain a second predicted image.
[0251] Optionally, the network structure of the second decoder is shown in FIG6 .
[0252] The specific implementation process of the above S308 refers to the specific description of the above S203-B1 to S203-B3, which will not be repeated here.
[0253] S309 : Process the mixed spatiotemporal representation through a third decoder to obtain a first weight corresponding to the first predicted image and a second weight corresponding to the second predicted image.
[0254] Specifically, the mixed spatiotemporal representation is input into the third decoder for adaptive masking to obtain a first weight corresponding to the first predicted image and a second weight corresponding to the second predicted image.
[0255] Optionally, the network structure of the third decoder is shown in FIG7 .
[0256] The specific implementation process of the above S309 refers to the specific description of the second method in the above S204-A11, which will not be repeated here.
[0257] S310 : Weighting the first predicted image and the second predicted image according to the first weight and the second weight to obtain a weighted image.
[0258] For example, the product of the first weight and the first predicted image is added to the product of the second weight and the second predicted image to obtain a weighted image.
[0259] S311 . Process the mixed spatiotemporal representation through a fourth decoder to obtain a residual image of the current image.
[0260] Specifically, the mixed spatiotemporal representation is input into the fourth decoder for processing to obtain a residual image of the current image.
[0261] Optionally, the network structure of the fourth decoder is shown in FIG8 .
[0262] The specific implementation process of the above S311 refers to the specific description of the above S204-A12 and will not be repeated here.
[0263] S312: Determine a target prediction image based on the weighted image and the residual image.
[0264] For example, the weighted image and the residual image are added together to determine the target prediction image.
[0265] S313: Decode the residual code stream to obtain a residual value of the current image.
[0266] S314: Obtain a reconstructed image based on the target predicted image and the residual value.
[0267] The specific implementation process of the above S313 and S314 refers to the specific description of the above S204-B1 and S204-B2, which will not be repeated here.
[0268] In an embodiment of the present application, when decoding is performed by the neural network-based decoder shown in FIG9 , the quantized first feature information is subjected to multi-level time-domain fusion, that is, the quantized first feature information is fused with multiple reconstructed images before the current image, so that the generated hybrid spatiotemporal representation includes more accurate, rich, and detailed feature information. In this way, motion compensation is performed on the previous reconstructed image based on the hybrid spatiotemporal representation to generate multiple decoding information. For example, the multiple decoding information includes the first predicted image, the second predicted image, the weights corresponding to the first predicted image and the second predicted image, and the residual image. In this way, when the target predicted image of the current image is determined based on the multiple decoding information, the accuracy of the target predicted image can be effectively improved, and then based on the accurate predicted image, the reconstructed image of the current image can be accurately obtained, thereby improving the video compression effect.
[0269] The above describes the video decoding method involved in the embodiment of the present application. On this basis, the following describes the video encoding method involved in the present application for the encoding end.
[0270] Figure 11 is a flow chart of a video encoding method provided by an embodiment of the present application. The execution subject of the embodiment of the present application may be the encoder shown in Figure 1 above.
[0271] As shown in FIG11 , the method of the embodiment of the present application includes:
[0272] S401 : Perform feature fusion on a current image and a reconstructed image before the current image to obtain first feature information.
[0273] An embodiment of the present application proposes a neural network-based encoder, which is obtained by end-to-end training of the neural network-based encoder and the neural network-based decoder.
[0274] In the embodiment of the present application, the previous reconstructed image of the current image can be understood as the previous frame image before the current image in the video sequence, and the previous frame image has been decoded and reconstructed.
[0275] Since the current image X t and the previous reconstructed image of the current image There is a strong similarity between these two adjacent frames. Therefore, when encoding, the encoder converts the current image X t and the previous reconstructed image of the current image Perform feature fusion to obtain the first feature information. For example, the encoder converts the current image X t and the previous reconstructed image of the current image Cascade connection between channels Get the cascaded input data X cat , and X t For 3-channel video frame input in SRGB domain, X cat The two frames of video are synthesized by stacking channels one by one to obtain an input signal with 6 channels. Then, the cascaded image X cat Perform feature extraction to obtain first feature information.
[0276] In this embodiment of the present application, the encoding end is cat The specific method of feature extraction is not limited, and may include at least one of multi-layer convolution, residual connection, dense connection, and other feature extraction methods.
[0277] In some embodiments, the encoding end performs Q non-local attention transformations and Q downsampling on the cascaded image to obtain first feature information, where Q is a positive integer.
[0278] For example, the encoder converts the cascaded 6-channel high-dimensional input signal X cat , input into the spatiotemporal feature extraction module (STFE) for multi-layer feature transformation and extraction.
[0279] Optionally, the spatiotemporal feature extraction module includes Q non-local attention modules and Q downsampling modules. The non-local attention module is used to implement non-local attention transformation, and the downsampling module is used to implement downsampling. For example, as shown in FIG12 , a downsampling module is connected after a non-local attention module. In actual application, the encoder converts the cascaded 6-channel high-dimensional input signal X cat In the input STFE, the first non-local attention module in STFE pays attention to X catPerform non-local attention feature transformation extraction to obtain feature information 11, and then input feature information 11 into the first downsampling module for downsampling to obtain feature information 12. Next, input feature information 12 into the second non-local attention module for non-local attention feature transformation extraction to obtain feature information 13, and then input feature information 13 into the second downsampling module for downsampling to obtain feature information 14. And so on, obtain the feature information output by the Q-th downsampling module, and determine the feature information as the first feature information X F .
[0280] The embodiment of the present application does not limit the specific value of Q.
[0281] Optionally, Q=4.
[0282] S402: quantize the first feature information to obtain quantized first feature information.
[0283] The first feature information obtained above is of floating point type, for example, represented by a 32-bit floating point number. Furthermore, in order to reduce the encoding cost, the encoding end quantizes the first feature information obtained above to obtain the quantized first feature information.
[0284] Exemplarily, the encoding end uses a rounding function Round(.) to quantize the first feature information.
[0285] In some embodiments, during the model training process, during forward propagation, the first feature information is quantized using the method shown in the following formula (7):
[0286]
[0287] Among them, U(-0.5,0.5) is a uniform noise distribution of plus or minus 0.5, which is used to approximate the actual rounding quantization function Round(.).
[0288] During the training process, the derivative of formula (7) is used to obtain the corresponding back-propagation gradient of 1, which is used as the back-propagation gradient to update the model.
[0289] S403: Encode the quantized first feature information to obtain a first code stream.
[0290] Method 1: The encoding end directly uses the probability distribution of the quantized first feature information to encode the quantized first feature information to obtain the first bit stream
[0291] The quantized first feature information includes a large amount of redundant information. Directly encoding the quantized first feature information requires many codewords, resulting in a high encoding cost. To reduce the encoding cost, in some embodiments, the encoder performs feature transformation based on the first feature information to obtain second feature information, quantizes the second feature information, and then encodes it to obtain a second bitstream. The second bitstream is decoded to obtain quantized second feature information, and the probability distribution of the quantized first feature information is determined based on the quantized second feature information. The quantized first feature information is then encoded based on the probability distribution of the quantized first feature information to obtain the first bitstream. In other words, to reduce the encoding cost, the encoder determines the super-a priori feature information corresponding to the first feature information, i.e., the second feature information, and determines the probability distribution of the quantized first feature information based on the second feature information. Since the second feature information is the super-a priori feature information of the first feature information and includes less redundancy, determining the probability distribution of the quantized first feature information based on the second feature information with less redundancy and encoding the first feature information using this probability distribution can reduce the encoding cost of the first feature information.
[0292] Based on the above description, the encoding end may encode the quantized first feature information through the following steps of the second method to obtain a first code stream.
[0293] Method 2: The above S403 includes the following steps S403-A1 to S403-A4:
[0294] S403-A1. Perform feature transformation based on the first feature information to obtain second feature information.
[0295] In this second approach, to reduce coding costs, the encoder performs feature transformation on the first feature information to obtain super-a priori feature information of the first feature information, namely the second feature information. This second feature information is used to determine the probability distribution of the quantized first feature information, and this probability distribution is used to encode the quantized first feature information to obtain a first bitstream. Simultaneously, to enable the decoder to decode the first bitstream using the same probability distribution as used for encoding, the second feature information is encoded to obtain a second bitstream. In other words, in this second approach, the encoder generates two bitstreams: the first bitstream and the second bitstream.
[0296] In the embodiment of the present application, the encoding end performs feature transformation based on the first feature information to obtain the second feature information in the following ways, but not limited to:
[0297] Method 1: Perform N non-local attention transformations and N downsamplings on the first feature information to obtain the second feature information.
[0298] Method 2: Perform N non-local attention transformations and N downsamplings on the quantized first feature information to obtain the second feature information.
[0299] That is to say, the encoding end can perform N non-local attention transformations and N downsamplings on the first feature information or the quantized first feature information to obtain the second feature information.
[0300] S403-A2: quantize and then encode the second feature information to obtain a second bit stream.
[0301] For example, the second feature information is quantized to obtain quantized second feature information; the probability distribution of the quantized second feature information is determined; and the quantized second feature information is encoded according to the probability distribution of the quantized second feature information to obtain a second code stream.
[0302] In the present application, since the second feature information is the super-prior feature information of the first feature information and includes less redundant information, the encoding end directly uses the probability distribution of the quantized second feature information during encoding to encode the quantized second feature information to obtain a second code stream.
[0303] S403-A3: Decode the second code stream to obtain quantized second feature information, and determine the probability distribution of the quantized first feature information based on the quantized second feature information.
[0304] In the embodiment of the present application, the encoding end performs arithmetic decoding on the second super-prior code stream to restore the quantized super-prior spatiotemporal features. That is, the quantized second feature information. Then, based on the quantized second feature information, the probability distribution of the quantized first feature information is determined, and then the quantized first feature information is encoded according to the probability distribution of the quantized first feature information to obtain the first code stream.
[0305] The following describes the process of determining the probability distribution of the quantized first feature information based on the quantized second feature information in S403-A3.
[0306] In some embodiments, determining the probability distribution of the quantized first feature information according to the quantized second feature information in S403-A3 includes the following steps:
[0307] S403-A31. Perform inverse transformation on the quantized second feature information to obtain reconstructed feature information.
[0308] In this implementation, the encoder performs an inverse transform on the quantized second feature information to obtain reconstructed feature information. The inverse transform method employed by the encoder can be understood as the inverse operation of the transform method employed by the encoder. For example, the encoder performs feature extraction N times on the first feature information to obtain the second feature information. Correspondingly, the encoder performs feature extraction N times in the reverse direction on the quantized second feature information to obtain inversely transformed feature information, which is recorded as the reconstructed feature information.
[0309] The embodiment of the present application does not limit the inverse transformation method adopted by the encoding end.
[0310] In some embodiments, the inverse transformation method adopted by the encoding end includes N times of feature extraction. That is, the encoding end performs N times of feature extraction on the obtained quantized second feature information to obtain reconstructed feature information.
[0311] In some embodiments, the inverse transformation method used by the encoding end includes N times of feature extraction and N times of upsampling. In other words, the encoding end performs N times of feature extraction and N times of upsampling on the obtained quantized second feature information to obtain reconstructed feature information.
[0312] The embodiment of the present application does not limit the specific execution order of the above-mentioned N feature extractions and N upsamplings.
[0313] In an example, the encoding end may first perform N consecutive feature extractions on the quantized second feature information, and then perform N consecutive upsamplings.
[0314] In another example, the N times of feature extraction and N times of upsampling are performed alternately, that is, one upsampling is performed after one feature extraction.
[0315] It should be noted that the embodiment of the present application does not limit the N-time feature extraction method adopted by the encoding end, for example, at least one of the feature extraction methods including multi-layer convolution, residual connection, dense connection, etc.
[0316] In some embodiments, the encoding end performs N non-local attention transformations and N upsamplings on the quantized second feature information to obtain reconstructed feature information, where N is a positive integer.
[0317] Since the non-local attention method can achieve more efficient feature extraction, it can make the extracted features retain more information and has high computational efficiency. Therefore, in the embodiment of the present application, the encoding end uses the non-local attention method to extract features from the quantized second feature information to achieve fast and accurate feature extraction of the quantized second feature information. In addition, when the encoding end generates the second feature information based on the first feature information, it performs N downsamplings. Therefore, at this time, the encoding end performs N upsamplings correspondingly during the inverse transformation to make the reconstructed feature information consistent with the size of the first feature information.
[0318] In some embodiments, as shown in FIG3 , the encoding end obtains reconstructed feature information through an inverse transformation module, where the inverse transformation module includes N non-local attention modules and N upsampling modules.
[0319] S403-A32: Determine the probability distribution of the reconstructed feature information.
[0320] From the above, it can be seen that the second quantized feature information is obtained by transforming the first feature information. The encoding end dequantizes the quantized second feature information through the above steps to obtain reconstructed feature information. Therefore, the reconstructed feature information can be understood as the reconstructed information of the first feature information. That is to say, the probability distribution of the reconstructed feature information is similar to or related to the probability distribution of the quantized first feature information. In this way, the encoding end can first determine the probability distribution of the reconstructed feature information, and then predict the probability distribution of the first feature information after quantization based on the probability distribution of the reconstructed feature information.
[0321] In some embodiments, the probability distribution of the reconstructed feature information is a normal distribution or a Gaussian distribution. In this case, the process of determining the probability distribution of the reconstructed feature information is to determine the mean and variance matrix of the reconstructed feature information based on each eigenvalue in the reconstructed feature information, and generate the Gaussian distribution of the reconstructed feature information based on the mean and variance matrix.
[0322] S403-A33: Determine the probability distribution of the quantized first feature information according to the probability distribution of the reconstructed feature information.
[0323] For example, the probability of the coded pixels in the quantized first feature information is predicted based on the probability distribution of the reconstructed feature information; and the probability distribution of the quantized first feature information is obtained based on the probability of the coded pixels in the quantized first feature information.
[0324] S403-A4: Encode the quantized first feature information according to the probability distribution of the quantized first feature information to obtain a first code stream.
[0325] According to the above steps, after the probability distribution of the quantized first feature information is determined, the quantized first feature information is encoded using the probability distribution to obtain a first code stream.
[0326] In some embodiments, the embodiment of the present application further includes a step of determining a reconstructed image of the current image, that is, the embodiment of the present application further includes the following S404:
[0327] S404: Determine a reconstructed image of the current image.
[0328] In some embodiments, the above S404 includes the following steps:
[0329] S404-A: Perform multi-level time-domain fusion on the quantized first feature information to obtain a mixed spatiotemporal representation.
[0330] In some embodiments, the quantized first feature information is feature information obtained by quantizing the first feature information at the encoding end.
[0331] In some embodiments, the quantized first feature information is reconstructed by the encoding end. For example, the encoding end decodes the second bitstream to obtain the quantized second feature information, and determines the probability distribution of the quantized first feature information based on the quantized second feature information. Exemplarily, the encoding end obtains the probability distribution of the quantized first feature information according to the above-mentioned methods S403-A31 to S403-A33, and then uses the probability distribution of the quantized first feature information to decode the first bitstream to obtain the quantized first feature information.
[0332] Next, the encoding end performs multi-level time-domain fusion on the quantized first feature information obtained above to obtain a mixed time-space representation.
[0333] In an embodiment of the present application, in order to improve the accuracy of the reconstructed image, the quantized first feature information is subjected to multi-level time domain fusion, that is, the quantized first feature information is not only fused with the feature information of the previous reconstructed image of the current image, but also the quantized first feature information is fused with the feature information of multiple reconstructed images before the current image, for example, the reconstructed images at multiple times such as time t-1, time t-2..., time tk are fused with the quantized first feature information. This can avoid the situation where certain information in the previous reconstructed image of the current image is blocked, and the blocked information can be obtained from several reconstructed images before the current image, thereby making the generated mixed spatiotemporal representation include more accurate, rich and detailed feature information. In this way, when motion compensation is performed on the previous reconstructed image based on the mixed spatiotemporal representation to generate P predicted images of the current image, the accuracy of the generated predicted image can be improved, and then based on the accurate predicted image, the reconstructed image of the current image can be accurately obtained, thereby improving the video compression effect.
[0334] The embodiment of the present application does not limit the specific method of performing multi-level time-domain fusion on the quantized first feature information at the encoding end to obtain a mixed spatiotemporal representation.
[0335] In some embodiments, the encoding end mixes the spatiotemporal representations through a recursive aggregation module, that is, the above S404-A includes the following step S404-A1:
[0336] S404-A1: The encoding end fuses the quantized first feature information with the implicit feature information of the recursive aggregation module at the previous moment through the recursive aggregation module to obtain a mixed spatiotemporal representation.
[0337] The recursive aggregation module of the embodiment of the present application will learn and retain the deep-level feature information learned from the feature information each time it generates a mixed spatiotemporal representation, and use the learned deep-level features as implicit feature information to act on the next generation of the mixed spatiotemporal representation, thereby improving the accuracy of the generated mixed spatiotemporal representation. That is to say, in the embodiment of the present application, the implicit feature information of the recursive aggregation module at the previous moment includes the feature information of multiple reconstructed images before the current image learned by the recursive aggregation module. In this way, the encoding end fuses the quantized first feature information with the implicit feature information of the recursive aggregation module at the previous moment through the recursive aggregation module, thereby generating a more accurate, rich and detailed mixed spatiotemporal representation.
[0338] The embodiment of the present application does not limit the specific network structure of the recursive aggregation module, for example, it can be any network structure that can implement the above functions.
[0339] In some embodiments, the recursive aggregation module is formed by stacking at least one spatiotemporal recursive network ST-LSTM. In this case, the expression formula of the above-mentioned mixed spatiotemporal representation Gt is as shown in the above formula (1).
[0340] S404-B. Perform motion compensation on the previous reconstructed image according to the hybrid spatiotemporal representation to obtain P predicted images of the current image, where P is a positive integer.
[0341] From the above, it can be seen that the hybrid spatiotemporal representation of the embodiment of the present application fuses the feature information of the current image and multiple reconstructed images before the current image, so that motion compensation is performed on the previous reconstructed image based on the hybrid spatiotemporal representation, and P accurate predicted images of the current image can be obtained.
[0342] The embodiment of the present application does not limit the specific number of P predicted images generated. That is, in the embodiment of the present application, the encoder can use different methods to perform motion compensation on the previous reconstructed image based on the hybrid spatiotemporal representation to obtain P predicted images of the current image.
[0343] The embodiment of the present application does not limit the specific manner in which the encoding end performs motion compensation on the previous reconstructed image based on the mixed spatiotemporal representation.
[0344] In some embodiments, the P predicted images include a first predicted image, which is obtained by the encoder using optical flow motion compensation. In this case, S404-B includes the following steps S404-B1 and S404-B2:
[0345] S404-B1, determining optical flow motion information based on the mixed spatiotemporal representation;
[0346] S404-B2: Perform motion compensation on the previous reconstructed image according to the optical flow motion information to obtain a first predicted image.
[0347] The embodiment of the present application does not limit the specific manner in which the encoder determines the optical flow motion information based on the mixed spatiotemporal representation.
[0348] In some embodiments, the encoder obtains optical flow motion information through a pre-trained neural network model, that is, the neural network model can predict optical flow motion information based on the mixed spatiotemporal representation. In some embodiments, the neural network model can be called a first decoder, or an optical flow signal decoder Df. The encoder inputs the mixed spatiotemporal representation Gt into the optical flow signal decoder Df to predict the optical flow motion information, and obtains the optical flow motion information f output by the optical flow signal decoder Df. x,y . Optionally, the f x,y is the optical flow motion information with channel 2.
[0349] For example, f x,y The generation formula of is shown in the above formula (2).
[0350] The embodiment of the present application does not limit the specific network structure of the above-mentioned optical flow signal decoder Df.
[0351] In some embodiments, the optical flow signal decoder Df is composed of multiple NLAMs and multiple upsampling modules. For example, as shown in Figure 5, the optical flow signal decoder Df includes 1 NLAM, 3 LAMs and 4 downsampling modules, where one NLAM is connected to a downsampling module, and one LAM is connected to a downsampling module.
[0352] It should be noted that FIG5 is only an example, and the settings of the parameters in FIG5 are also only examples. The network structure of the optical flow signal decoder Df in the embodiment of the present application includes but is not limited to that shown in FIG5 .
[0353] The encoder generates optical flow motion information f x,y Then, use the optical flow motion information f x,y Reconstruct the previous image Perform motion compensation to obtain the first predicted image X1.
[0354] In the embodiment of the present application, the encoder performs motion compensation on the previous reconstructed image according to the optical flow motion information to obtain the first predicted image without limiting the specific method. For example, the encoder uses the optical flow motion information f x,y Reconstruct the previous image Perform linear interpolation and record the interpolated image as the first predicted image X1.
[0355] In a possible implementation, the encoding end obtains the first predicted image X1 through the following formula (3).
[0356] In this implementation, as shown in FIG5 , the encoder uses the optical flow motion information f through the Warping operation. x,y Reconstruct the previous image Perform motion compensation to obtain the first predicted image X1.
[0357] In some embodiments, the P predicted images include a second predicted image, and the second predicted image is obtained by the decoding end using an offset motion compensation method. In this case, the above S404-B includes the following steps S404-B-1 to S404-B-3:
[0358] S404-B-1. Obtain an offset corresponding to the current image based on the mixed spatiotemporal representation;
[0359] S404-B-2, extracting spatial features from the previous reconstructed image to obtain reference feature information;
[0360] S404-B-3. Use the offset to perform motion compensation on the reference feature information to obtain a second predicted image.
[0361] The embodiment of the present application does not limit the specific manner in which the encoding end obtains the offset corresponding to the current image based on the mixed spatiotemporal representation.
[0362] In some embodiments, the encoder obtains the offset corresponding to the current image using a pre-trained neural network model. Specifically, the neural network model can predict the offset based on the hybrid spatiotemporal representation. This offset is lossy offset information. In some embodiments, the neural network model can be referred to as a second decoder, or a variable convolutional decoder Dm. The encoder inputs the hybrid spatiotemporal representation Gt into the variable convolutional decoder Dm to predict the offset information.
[0363] At the same time, the encoding end performs spatial feature extraction on the previous reconstructed image to obtain reference feature information. For example, the encoding end performs spatial feature extraction on the previous reconstructed image through a spatial feature extraction module SFE to obtain reference feature information.
[0364] Next, the encoder performs motion compensation on the extracted reference feature information using the offset to obtain a second predicted image of the current image.
[0365] In the embodiment of the present application, the encoding end uses the offset to perform motion compensation on the extracted reference feature information, and the specific method of obtaining the second predicted image of the current image is not limited.
[0366] In a possible implementation, the encoder uses the offset to perform deformable convolution-based motion compensation on the reference feature information to obtain a second predicted image.
[0367] In some embodiments, since the transformable convolution can generate an offset corresponding to the current image based on the mixed spatiotemporal representation, in an embodiment of the present application, the encoding end inputs the mixed spatiotemporal representation Gt and the reference feature information into the transformable convolution. The transformable convolution generates an offset corresponding to the current image based on the mixed spatiotemporal representation Gt, and applies the offset to the reference feature information for motion compensation, thereby obtaining a second predicted image.
[0368] Based on this, for example, as shown in FIG6 , the variable convolution decoder Dm of the embodiment of the present application includes a transformable convolution DCN, and the encoding end reconstructs the previous image The mixed spatiotemporal representation Gt and the reference feature information are then input into the transformable convolution DCN for offset extraction and motion compensation to obtain the second predicted image X2.
[0369] Exemplarily, the encoding end generates the second predicted image X2 using the above formula (4).
[0370] The embodiment of the present application does not limit the specific network structure of the above-mentioned optical flow signal decoder Df.
[0371] In some embodiments, as shown in Figure 6, in order to further improve the accuracy of the second predicted image, the variable convolution decoder Dm includes, in addition to the transformable convolution DCN, 1 NLAM, 3 LAMs and 4 downsampling modules, where one NLAM is connected to a downsampling module, and one LAM is connected to a downsampling module.
[0372] It should be noted that the above Figure 6 is only an example, and the settings of the various parameters in Figure 6 are also only examples. The network structure of the variable convolution decoder Dm in the embodiment of the present application includes but is not limited to that shown in Figure 6.
[0373] In the embodiment of the present application, as shown in FIG6 , the encoding end first reconstructs the previous image The mixed spatiotemporal representation Gt and the reference feature information are then fed into the transformable convolutional network (DCN) in the variable convolutional decoder (Dm) for offset extraction and motion compensation. This generates a feature, which is then fed into the NLAM. After feature extraction by the NLAM, three LAMs, and four downsampling modules, the second predicted image X2 is finally restored.
[0374] According to the above method, the encoding end may determine P predicted images, for example, a first predicted image and a second predicted image, and then perform the following step S204.
[0375] S404-C. Determine a reconstructed image of the current image based on the P predicted images.
[0376] In some embodiments, if the P predicted images include one predicted image, a reconstructed image of the current image is determined based on the predicted image.
[0377] For example, the predicted image is compared with one or several previous reconstructed images of the current image to calculate the loss. If the loss is small, it means that the prediction accuracy of the predicted image is high, and the predicted image can be determined as the reconstructed image of the current image.
[0378] For another example, if the above-mentioned loss is large, it means that the prediction accuracy of the predicted image is low. At this time, the reconstructed image of the current image can be determined based on the previous one or several reconstructed images of the current image and the predicted image. For example, the predicted image and the previous one or several reconstructed images of the current image are input into a neural network to obtain the reconstructed image of the current image.
[0379] In some embodiments, the above S404-C includes the following steps S404-CA and S404-CB:
[0380] S404-CA: Determine a target predicted image for the current image based on the P predicted images.
[0381] In this implementation, the encoding end first determines the target prediction image of the current image based on P prediction images, and then realizes the reconstructed image of the current image based on the target prediction image of the current image, thereby improving the determination accuracy of the reconstructed image.
[0382] The embodiment of the present application does not limit the specific method of determining the target predicted image of the current image based on P predicted images.
[0383] In some embodiments, if P=1, the one predicted image is determined as the target predicted image of the current image.
[0384] In some embodiments, if P is greater than 1, S404-CA includes S404-C-A11 and S404-C-A12:
[0385] S404-C-A11, determining a weighted image based on the P predicted images;
[0386] In this implementation, if multiple predicted images of the current image are generated according to the above method, for example, when a first predicted image and a second predicted image are generated, these P predicted images are weighted to generate a weighted image, and then the target predicted image is obtained based on the weighted image.
[0387] The embodiment of the present application does not limit the specific method of determining the weighted image based on P predicted images.
[0388] For example, weights corresponding to P predicted images are determined; and according to the weights corresponding to the P predicted images, the P predicted images are weighted to obtain a weighted image.
[0389] Exemplarily, if the P predicted images include a first predicted image and a second predicted image, the encoding end determines a first weight corresponding to the first predicted image and a second weight corresponding to the second predicted image, and weights the first predicted image and the second predicted image according to the first weight and the second weight to obtain a weighted image.
[0390] The methods for determining the weights corresponding to the P prediction images include but are not limited to the following:
[0391] In a first approach, the weights corresponding to the P predicted images are preset weights. Assuming P = 2, the first weight corresponding to the first predicted image and the second weight corresponding to the second predicted image can be equal to the second weight, or the ratio of the first weight to the second weight can be 1 / 2, 1 / 4, 1 / 2, 1 / 3, 2 / 1, 3 / 1, 4 / 1, and so on.
[0392] In the second method, the encoder performs adaptive masking based on the mixed spatiotemporal representation to obtain weights corresponding to P predicted images.
[0393] Exemplarily, the encoder generates weights corresponding to the P predicted images through a neural network model. The neural network model is pre-trained and can be used to generate weights corresponding to the P predicted images. In some embodiments, the neural network model is also called a third decoder or an adaptive mask compensation decoder D w Specifically, the encoder inputs the mixed spatiotemporal representation into the adaptive mask compensation decoder D w For example, the encoder inputs the mixed spatiotemporal representation Gt into the adaptive mask compensation decoder D wAdaptive mask compensation decoder D w Output the first weight w1 of the first predicted image and the second weight w2 of the second predicted image, perform the above-obtained first predicted image X1 and second predicted image X2 according to the first weight w1 and second weight w2, and adaptively select the corresponding information representing different areas in the predicted frame to generate a weighted image.
[0394] Exemplarily, the weighted image X3 is generated according to the above formula (5).
[0395] In some embodiments, the weights corresponding to the above-mentioned P predicted images are a matrix, which includes the weights corresponding to each pixel in the predicted image. In this way, when generating a weighted image, for each pixel in the current image, the predicted values and their weights corresponding to the pixel in the P predicted images are weighted to obtain the weighted predicted value of the pixel. In this way, the weighted predicted values corresponding to each pixel in the current image constitute the weighted image of the current image.
[0396] The embodiment of the present application is to improve the adaptive mask compensation decoder D w There is no restriction on the specific network structure.
[0397] In some embodiments, as shown in FIG7 , the adaptive mask compensation decoder D w It includes 1 NLAM, 3 LAMs, 4 downsampling modules and a sigmoid function, where a downsampling module is connected after one NLAM and a downsampling module is connected after one LAM.
[0398] It should be noted that FIG7 is only an example, and the settings of the parameters in FIG7 are also only examples. The adaptive mask compensation decoder D in the embodiment of the present application w The network structure includes but is not limited to that shown in FIG7 .
[0399] In this implementation, the encoder performs weighting on the P predicted images according to the above method, obtains the weighted image, and then executes the following S404-C-A12.
[0400] S404-C-A12. Obtain a target prediction image based on the weighted image.
[0401] For example, the weighted image is determined as the target prediction image.
[0402] In some embodiments, the encoding end may further obtain a residual image of the current image based on the mixed spatiotemporal representation.
[0403] Exemplarily, the encoder obtains a residual image of the current image through a neural network model. The neural network model is pre-trained and can be used to generate the residual image of the current image. In some embodiments, the neural network model is also called a fourth decoder or a spatial texture enhancement decoder Dt. Specifically, the encoder inputs the mixed spatiotemporal representation into the spatial texture enhancement decoder Dt for spatial texture enhancement to obtain a residual image X of the current image. r =D_t(G t ), the residual image X r The predicted image can be texture enhanced.
[0404] In the embodiment of the present application, there is no limitation on the specific network structure of the above-mentioned spatial texture enhancement decoder Dt.
[0405] In some embodiments, as shown in FIG8 , the spatial texture enhancement decoder Dt includes 1 NLAM, 3 LAMs, and 4 downsampling modules, wherein one NLAM is followed by a downsampling module, and one LAM is followed by a downsampling module.
[0406] It should be noted that the above FIG8 is only an example, and the settings of the parameters in FIG8 are also only examples. The network structure of the spatial texture enhancement decoder Dt in the embodiment of the present application includes but is not limited to that shown in FIG8.
[0407] Since the above residual image X r Texture enhancement can be performed on the predicted image. Based on this, in some embodiments, determining the target predicted image of the current image based on the P predicted images in the above S404-CA includes the following step S404-C-A21:
[0408] S404-C-A21. Obtain a target prediction image based on the P prediction images and the residual image.
[0409] For example, if P=1, a target predicted image is obtained based on the predicted image and the residual image. For example, the predicted image and the residual image are added to generate the target predicted image.
[0410] For another example, if P is greater than 1, a weighted image is first determined based on the P predicted images; and then a target predicted image is determined based on the weighted image and the residual image.
[0411] The specific process of the encoder determining the weighted image based on the P predicted images can refer to the specific description of S204-A11 above, which will not be repeated here.
[0412] For example, taking P = 2 as an example, according to the above method, the first weight w1 corresponding to the first predicted image and the second weight w2 corresponding to the second predicted image are determined. Optionally, the first predicted image and the second predicted image are weighted according to the above formula (5) to obtain a weighted image X3. Then, the residual image X is used r The weighted image X3 is enhanced to obtain the target predicted image.
[0413] Exemplarily, the target predicted image X4 is generated according to the above formula (6).
[0414] According to the above method, after the encoder determines the target predicted image of the current image, it executes the following steps S404-CB.
[0415] S404-CB: Determine a reconstructed image of the current image according to the target predicted image.
[0416] In some embodiments, the target predicted image is compared with one or more reconstructed images preceding the current image to calculate a loss. If the loss is small, it indicates that the prediction accuracy of the target predicted image is high, and the target predicted image can be determined as the reconstructed image of the current image. If the loss is large, it indicates that the prediction accuracy of the target predicted image is low. In this case, the reconstructed image of the current image can be determined based on one or more reconstructed images preceding the current image and the target predicted image. For example, the target predicted image and one or more reconstructed images preceding the current image are input into a neural network to obtain the reconstructed image of the current image.
[0417] In some embodiments, to further improve the accuracy of determining the reconstructed image, the encoder determines a residual value of the current image based on the current image and the target predicted image; encodes the residual value to obtain a residual bitstream. In this case, the embodiment of the present application also includes residual decoding, and the above S404-CB includes the following steps S404-C-B1 and S404-C-B2:
[0418] S404-C-B1, decoding the residual code stream to obtain a residual value of the current image;
[0419] S404-C-B2. Obtain a reconstructed image based on the target predicted image and the residual value.
[0420] In this embodiment of the present application, to improve the quality of the reconstructed image, the encoder also generates a residual bitstream through residual coding. Specifically, the encoder determines the residual value of the current image and encodes this residual value to generate a residual bitstream. Correspondingly, the encoder decodes the residual bitstream to obtain the residual value of the current image, and then obtains a reconstructed image based on the target predicted image and the residual value.
[0421] The embodiment of the present application does not limit the specific representation of the residual value of the current image.
[0422] In one possible implementation, the residual value of the current image is a matrix, and each element in the matrix is the residual value corresponding to each pixel in the current image. In this way, the encoding end can add the residual value and the predicted value corresponding to each pixel in the target predicted image pixel by pixel to obtain the reconstructed value of each pixel, and then obtain the reconstructed image of the current image. Taking the i-th pixel in the current image as an example, in the target predicted image, the predicted value corresponding to the i-th pixel is obtained, and the residual value corresponding to the i-th pixel is obtained from the residual value of the current image. Then, the predicted value and the residual value corresponding to the i-th pixel are added to obtain the reconstructed value corresponding to the i-th pixel. For each pixel in the current image, referring to the above i-th pixel, the reconstructed value corresponding to each pixel in the current image can be obtained. The reconstructed value corresponding to each pixel in the current image constitutes the reconstructed image of the current image.
[0423] The embodiment of the present application does not limit the specific method in which the encoding end obtains the residual value of the current image. That is, the embodiment of the present application does not limit the residual encoding and decoding method adopted by both ends of the encoding and decoding.
[0424] In one example, the encoding end determines the target prediction image of the current image, and then obtains the residual value of the current image based on the current image and the target prediction image. For example, the difference between the current image and the target prediction image is determined as the residual value of the current image. Then, the residual value of the current image is encoded to generate a residual code. Optionally, the residual value of the current image can be transformed to obtain a transformation coefficient, the transformation coefficient can be quantized to obtain a quantization coefficient, and the quantization coefficient can be encoded to obtain a residual code stream. Correspondingly, the encoding end decodes the residual code stream to obtain the residual value of the current image, for example, decodes the residual code stream to obtain a quantization coefficient, dequantizes and detransforms the quantization coefficient to obtain the residual value of the current image. Then, according to the above method, the residual values corresponding to the target prediction image and the current image are added to obtain a reconstructed image of the current image.
[0425] In some embodiments, the encoding end may use a neural network method to process the current image and the target predicted image of the current image to generate a residual value of the current image, and encode the residual value of the current image to generate a residual code stream.
[0426] In the embodiment of the present application, the encoding end can obtain a reconstructed image of the current image according to the above method.
[0427] Optionally, the reconstructed image can be displayed directly.
[0428] Optionally, the reconstructed image may be stored in a cache for use in encoding subsequent images.
[0429] The video encoding method provided in an embodiment of the present application comprises: an encoder performs feature fusion on a current image and a reconstructed image preceding the current image to obtain first feature information; quantizes the first feature information to obtain quantized first feature information; encodes the quantized first feature information to obtain a first bitstream, so that a decoder decodes the first bitstream, determines the quantized first feature information, performs multi-level temporal fusion on the quantized first feature information to obtain a hybrid spatiotemporal representation; performs motion compensation on the previous reconstructed image based on the hybrid spatiotemporal representation to obtain P predicted images of the current image; and then determines a reconstructed image of the current image based on the P predicted images. Specifically, in order to improve the accuracy of the reconstructed image, the present application performs multi-level temporal fusion on the quantized first feature information, for example, by fusing the quantized first feature information with multiple reconstructed images preceding the current image. This prevents occlusion of information in the reconstructed image preceding the current image from being obtained from several reconstructed images preceding the current image, thereby ensuring that the generated hybrid spatiotemporal representation includes more accurate, rich, and detailed feature information. In this way, when motion compensation is performed on the previous reconstructed image based on the mixed spatiotemporal representation, P high-precision predicted images can be generated. Based on the P high-precision predicted images, the reconstructed image of the current image can be accurately obtained, thereby improving the video compression effect.
[0430] In an embodiment of the present application, an end-to-end neural network-based encoding and decoding framework is proposed, which includes a neural network-based encoder and a neural network-based decoder. The encoding process of the embodiment of the present application is introduced below in conjunction with a possible neural network-based encoder of the present application.
[0431] Figure 12 is a schematic diagram of the network structure of a neural network-based encoder involved in an embodiment of the present application, including: a spatiotemporal feature extraction module, an inverse transformation module, a recursive aggregation module and a hybrid motion compensation module.
[0432] The spatiotemporal feature extraction module is used to perform feature extraction and down-sampling on the cascaded current image and the previous reconstructed image to obtain first feature information.
[0433] The inverse transformation module is used to perform inverse transformation on the quantized second feature information to obtain reconstructed feature information of the first feature information. Exemplarily, its network structure is shown in FIG3 .
[0434] The recursive aggregation module is used to perform multi-level time-domain fusion on the quantized first feature information to obtain a mixed spatiotemporal representation. Exemplarily, its network structure is shown in FIG4 .
[0435] The hybrid motion compensation module is used to perform hybrid motion compensation on the hybrid spatiotemporal representation to obtain a target predicted image of the current image. Exemplarily, the hybrid motion compensation module may include the first decoder shown in FIG5 and / or the second decoder shown in FIG6. Optionally, if the hybrid motion compensation module includes the first decoder and the second decoder, the hybrid motion compensation module may also include the third decoder shown in FIG7. In some embodiments, the hybrid motion compensation module may also include a fourth decoder shown in FIG8.
[0436] Illustratively, the embodiment of the present application is described by taking an example where the motion compensation module includes a first decoder, a second decoder, a third decoder, and a fourth decoder.
[0437] Based on the neural network-based encoder shown in FIG12 above, a possible video encoding method of an embodiment of the present application is introduced in combination with FIG13.
[0438] FIG13 is a schematic diagram of a video encoding process according to an embodiment of the present application, as shown in FIG13 , including:
[0439] S501: Perform feature fusion on a current image and a reconstructed image before the current image to obtain first feature information.
[0440] For example, the encoder converts the current image X t and the previous reconstructed image of the current image Cascade between channels to obtain X cat , then, for the cascaded image X cat Perform feature extraction to obtain first feature information.
[0441] The specific implementation process of the above S501 refers to the description of the above S401 and will not be repeated here.
[0442] S502: Quantize the first feature information to obtain quantized first feature information.
[0443] The specific implementation process of the above S502 refers to the description of the above S402 and will not be repeated here.
[0444] S503: Perform feature transformation according to the first feature information to obtain second feature information.
[0445] The specific implementation process of the above S503 refers to the description of the above S403-A1 and will not be repeated here.
[0446] S504: quantize and then encode the second feature information to obtain a second bit stream.
[0447] The specific implementation process of the above S504 refers to the description of the above S403-A2 and will not be repeated here.
[0448] S505: Decode the second code stream to obtain quantized second feature information.
[0449] The specific implementation process of the above S505 refers to the description of the above S403-A3 and will not be repeated here.
[0450] S506 : Perform an inverse transformation on the quantized second feature information through an inverse transformation module to obtain reconstructed feature information.
[0451] Exemplarily, the specific network structure of the inverse transformation module is shown in Figure 3, including two non-local self-attention modules and two upsampling modules.
[0452] For example, the decoding end inputs the quantized second feature information into the inverse transformation module for inverse transformation, and the inverse transformation module outputs the reconstructed feature information.
[0453] The specific implementation process of the above S506 refers to the description of the above S403-A31 and will not be repeated here.
[0454] S507: Determine the probability distribution of the reconstructed feature information.
[0455] S508 : Predicting the probability distribution of the quantized first feature information according to the probability distribution of the reconstructed feature information.
[0456] S509 : Encode the quantized first feature information according to the probability distribution of the quantized first feature information to obtain a first code stream.
[0457] The specific implementation process of the above S507 to S509 refers to the description of the above S403-A32, S403-A33 and S403-A4, which will not be repeated here.
[0458] The embodiment of the present application also includes a process of determining the reconstructed image.
[0459] S510 : Decode the first bitstream according to the probability distribution of the quantized first feature information to obtain the quantized first feature information.
[0460] S511 , performing multi-level time-domain fusion on the quantized first feature information through a recursive aggregation module to obtain a mixed spatiotemporal representation.
[0461] Optionally, the recursive aggregation module is formed by stacking at least one spatiotemporal recursive network.
[0462] Exemplarily, the network structure of the recursive aggregation module is shown in FIG4 .
[0463] For example, the decoding end inputs the quantized first feature information into the recursive aggregation module, so that the recursive aggregation module fuses the quantized first feature information with the implicit feature information of the recursive aggregation module at the previous moment, thereby outputting a hybrid spatiotemporal representation. The specific implementation process of the above S511 is described in the above S404-A and is not repeated here.
[0464] S512: Process the mixed spatiotemporal representation through a first decoder to obtain a first predicted image.
[0465] After the hybrid spatiotemporal representation is obtained according to the above S512, the hybrid spatiotemporal representation and the previous reconstructed image are input into the hybrid motion compensation module for motion hybrid compensation to obtain the target predicted image of the current image.
[0466] Specifically, the mixed spatiotemporal representation is processed by a first decoder to determine optical flow motion information, and motion compensation is performed on a previous reconstructed image according to the optical flow motion information to obtain a first predicted image.
[0467] Optionally, the network structure of the first decoder is shown in FIG5 .
[0468] The specific implementation process of the above S512 refers to the specific description of the above S404-B1 and S404-B2, which will not be repeated here.
[0469] S513: Process the mixed spatiotemporal representation through a second decoder to obtain a second predicted image.
[0470] Specifically, spatial features of the previous reconstructed image are extracted through SFE to obtain reference feature information; the reference feature information and the mixed spatiotemporal representation are input into the second decoder so that the offset performs motion compensation on the reference feature information to obtain a second predicted image.
[0471] Optionally, the network structure of the second decoder is shown in FIG6 .
[0472] The specific implementation process of the above S513 refers to the specific description of the above S404-B-1 to S404-B-3, which will not be repeated here.
[0473] S514 : Process the mixed spatiotemporal representation through a third decoder to obtain a first weight corresponding to the first predicted image and a second weight corresponding to the second predicted image.
[0474] Specifically, the mixed spatiotemporal representation is input into the third decoder for adaptive masking to obtain a first weight corresponding to the first predicted image and a second weight corresponding to the second predicted image.
[0475] Optionally, the network structure of the third decoder is shown in FIG7 .
[0476] The specific implementation process of the above S514 refers to the specific description of the second method in the above S404-C-A11, which will not be repeated here.
[0477] S515 . Weight the first predicted image and the second predicted image according to the first weight and the second weight to obtain a weighted image.
[0478] For example, the product of the first weight and the first predicted image is added to the product of the second weight and the second predicted image to obtain a weighted image.
[0479] S516: Process the mixed spatiotemporal representation through a fourth decoder to obtain a residual image of the current image.
[0480] Specifically, the mixed spatiotemporal representation is input into the fourth decoder for processing to obtain a residual image of the current image.
[0481] Optionally, the network structure of the fourth decoder is shown in FIG8 .
[0482] The specific implementation process of the above S516 refers to the specific description of the above S404-C-A12 and will not be repeated here.
[0483] S517: Determine a target prediction image according to the weighted image and the residual image.
[0484] For example, the weighted image and the residual image are added together to determine the target prediction image.
[0485] S518: Decode the residual code stream to obtain a residual value of the current image.
[0486] S519: Obtain a reconstructed image based on the target predicted image and the residual value.
[0487] For the specific implementation process of the above S518 and S519, please refer to the specific description of the above S404-C-B1 and S404-C-B2, which will not be repeated here.
[0488] In an embodiment of the present application, when encoding is performed by the neural network-based encoder shown in FIG12 , the quantized first feature information is subjected to multi-level time-domain fusion, that is, the quantized first feature information is subjected to feature fusion with multiple reconstructed images before the current image, so that the generated hybrid spatiotemporal representation includes more accurate, rich, and detailed feature information. In this way, motion compensation is performed on the previous reconstructed image based on the hybrid spatiotemporal representation to generate multiple decoding information. For example, the multiple decoding information includes the first predicted image, the second predicted image, the weights corresponding to the first predicted image and the second predicted image, and the residual image. In this way, when the target predicted image of the current image is determined based on the multiple decoding information, the accuracy of the target predicted image can be effectively improved, and then based on the accurate predicted image, the reconstructed image of the current image can be accurately obtained, thereby improving the video compression effect.
[0489] It should be understood that Figures 2 to 13 are merely examples of the present application and should not be understood as limiting the present application.
[0490] The preferred embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the above embodiments. Within the technical concept of the present application, a variety of simple modifications can be made to the technical solution of the present application, and these simple modifications all fall within the scope of protection of the present application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner unless there is any contradiction. In order to avoid unnecessary repetition, the present application will not further explain various possible combinations. For another example, the various different embodiments of the present application can also be arbitrarily combined, and as long as they do not violate the ideas of the present application, they should also be regarded as the contents disclosed in the present application.
[0491] It should also be understood that in the various method embodiments of the present application, the size of the sequence numbers of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, in the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that three types of relationships can exist. Specifically, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the related objects before and after are in an "or" relationship.
[0492] The above describes in detail the method embodiment of the present application in conjunction with Figures 2 to 13, and the following describes in detail the device embodiment of the present application in conjunction with Figures 14 to 17.
[0493] FIG14 is a schematic block diagram of a video decoding device according to an embodiment of the present application.
[0494] As shown in FIG14 , the video decoding apparatus 10 includes:
[0495] A decoding unit 11 is configured to decode a first bitstream and determine quantized first feature information, where the first feature information is obtained by performing feature fusion on a current image and a reconstructed image preceding the current image;
[0496] a fusion unit 12, configured to perform multi-level time-domain fusion on the quantized first feature information to obtain a hybrid spatiotemporal representation;
[0497] a compensation unit 13, configured to perform motion compensation on the previous reconstructed image according to the hybrid spatiotemporal representation to obtain P predicted images of the current image, where P is a positive integer;
[0498] The reconstruction unit 14 is configured to determine a reconstructed image of the current image based on the P predicted images.
[0499] In some embodiments, the fusion unit 12 is specifically configured to fuse the quantized first feature information with the implicit feature information of the recursive aggregation module at a previous moment through a recursive aggregation module to obtain the mixed spatiotemporal representation.
[0500] Optionally, the recursive aggregation module is formed by stacking at least one spatiotemporal recursive network.
[0501] In some embodiments, the P predicted images include a first predicted image, and the compensation unit 13 is specifically used to determine optical flow motion information based on the mixed spatiotemporal representation; and perform motion compensation on the previous reconstructed image based on the optical flow motion information to obtain the first predicted image.
[0502] In some embodiments, the P predicted images include a second predicted image, and the compensation unit 13 is specifically used to obtain the offset corresponding to the current image based on the mixed spatiotemporal representation; perform spatial feature extraction on the previous reconstructed image to obtain reference feature information; and use the offset to perform motion compensation on the reference feature information to obtain the second predicted image.
[0503] In some embodiments, the compensation unit 13 is specifically configured to use the offset to perform deformable convolution-based motion compensation on the reference feature information to obtain the second predicted image.
[0504] In some embodiments, the reconstruction unit 14 is configured to determine a target predicted image of the current image based on the P predicted images; and determine a reconstructed image of the current image based on the target predicted image.
[0505] In some embodiments, the reconstruction unit 14 is configured to determine a weighted image based on the P predicted images; and obtain the target predicted image based on the weighted image.
[0506] In some embodiments, the reconstruction unit 14 is further configured to obtain a residual image of the current image according to the mixed spatiotemporal representation; and obtain the target predicted image according to the P predicted images and the residual image.
[0507] In some embodiments, the reconstruction unit 14 is specifically configured to determine a weighted image based on the P predicted images; and determine the target predicted image based on the weighted image and the residual image.
[0508] In some embodiments, the reconstruction unit 14 is specifically configured to determine weights corresponding to the P predicted images; and weight the P predicted images according to the weights corresponding to the P predicted images to obtain the weighted image.
[0509] In some embodiments, the reconstruction unit 14 is specifically configured to perform adaptive masking according to the mixed spatiotemporal representation to obtain weights corresponding to the P predicted images.
[0510] In some embodiments, if the P predicted images include a first predicted image and a second predicted image, the reconstruction unit 14 is specifically used to determine a first weight corresponding to the first predicted image and a second weight corresponding to the second predicted image; and weight the first predicted image and the second predicted image according to the first weight and the second weight to obtain the weighted image.
[0511] In some embodiments, the reconstruction unit 14 is specifically configured to decode the residual code stream to obtain a residual value of the current image; and obtain the reconstructed image according to the target predicted image and the residual value.
[0512] In some embodiments, the decoding unit 11 is specifically used to decode the second code stream to obtain quantized second feature information, where the second feature information is obtained by performing feature transformation on the first feature information; determine the probability distribution of the quantized first feature information based on the quantized second feature information; decode the first code stream based on the probability distribution of the quantized first feature information to obtain the quantized first feature information.
[0513] In some embodiments, the decoding unit 11 is specifically used to perform an inverse transformation on the quantized second feature information to obtain reconstructed feature information; determine the probability distribution of the reconstructed feature information; and predict the probability distribution of the quantized first feature information based on the probability distribution of the reconstructed feature information.
[0514] In some embodiments, the decoding unit 11 is specifically used to perform N non-local attention transformations and N upsamplings on the quantized second feature information to obtain the reconstructed feature information, where N is a positive integer.
[0515] In some embodiments, the decoding unit 11 is specifically used to predict the probability of the encoded pixels in the quantized first feature information based on the probability distribution of the reconstructed feature information; and obtain the probability distribution of the quantized first feature information based on the probability of the encoded pixels in the quantized first feature information.
[0516] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions can refer to the method embodiment. To avoid repetition, no further description is given here. Specifically, the video decoding device 10 shown in Figure 14 can correspond to the corresponding subject in the method of the embodiment of the present application, and the aforementioned and other operations and / or functions of the various units in the video decoding device 10 are respectively for implementing the corresponding processes in each method such as the method, and for the sake of brevity, no further description is given here.
[0517] FIG15 is a schematic block diagram of a video encoding device according to an embodiment of the present application.
[0518] As shown in FIG15 , the video encoding apparatus 20 includes:
[0519] A fusion unit 21 is configured to perform feature fusion on a current image and a reconstructed image preceding the current image to obtain first feature information;
[0520] a quantization unit 22, configured to quantize the first feature information to obtain quantized first feature information;
[0521] The encoding unit 23 is configured to encode the quantized first feature information to obtain the first code stream.
[0522] In some embodiments, the fusion unit 21 is specifically configured to perform channel cascade on the current image and the reconstructed image to obtain a cascaded image; and perform feature extraction on the cascaded image to obtain the first feature information.
[0523] In some embodiments, the fusion unit 21 is specifically used to perform Q non-local attention transformations and Q downsampling on the cascaded image to obtain the first feature information, where Q is a positive integer.
[0524] In some embodiments, the encoding unit 23 is further configured to perform feature transformation based on the first feature information to obtain second feature information; quantize and then encode the second feature information to obtain a second code stream; decode the second code stream to obtain quantized second feature information, and determine the probability distribution of the quantized first feature information based on the quantized second feature information; and encode the quantized first feature information based on the probability distribution of the quantized first feature information to obtain a first code stream.
[0525] In some embodiments, the encoding unit 23 is specifically used to perform N non-local attention transformations and N downsamplings on the first feature information to obtain the second feature information, where N is a positive integer.
[0526] In some embodiments, the encoding unit 23 is specifically configured to perform N non-local attention transformations and N downsamplings on the quantized first feature information to obtain the second feature information.
[0527] In some embodiments, the encoding unit 23 is further used to quantize the second feature information to obtain quantized second feature information; determine the probability distribution of the quantized second feature information; and encode the quantized second feature information according to the probability distribution of the quantized second feature information to obtain the second code stream.
[0528] In some embodiments, the encoding unit 23 is specifically configured to perform an inverse transformation on the quantized second feature information to obtain reconstructed feature information; determine a probability distribution of the reconstructed feature information; and determine a probability distribution of the quantized first feature information based on the probability distribution of the reconstructed feature information.
[0529] In some embodiments, the encoding unit 23 is specifically used to perform N non-local attention transformations and N upsamplings on the quantized second feature information to obtain the reconstructed feature information, where N is a positive integer.
[0530] In some embodiments, the encoding unit 23 is specifically used to determine the probability of the encoded pixels in the quantized first feature information based on the probability distribution of the reconstructed feature information; and obtain the probability distribution of the quantized first feature information based on the probability of the encoded pixels in the quantized first feature information.
[0531] In some embodiments, the encoding unit 23 is further configured to determine a reconstructed image of the current image.
[0532] In some embodiments, the encoding unit 23 is specifically used to perform multi-level time-domain fusion on the quantized first feature information to obtain a mixed time-space representation; perform motion compensation on the previous reconstructed image according to the mixed time-space representation to obtain P predicted images of the current image, where P is a positive integer; and determine the reconstructed image of the current image based on the P predicted images.
[0533] In some embodiments, the encoding unit 23 is specifically configured to fuse the quantized first feature information with the implicit feature information of the recursive aggregation module at a previous moment through a recursive aggregation module to obtain the mixed spatiotemporal representation.
[0534] Optionally, the recursive aggregation module is formed by stacking at least one spatiotemporal recursive network.
[0535] In some embodiments, the P predicted images include a first predicted image, and the encoding unit 23 is specifically used to determine optical flow motion information based on the mixed spatiotemporal representation; and perform motion compensation on the previous reconstructed image based on the optical flow motion information to obtain the first predicted image.
[0536] In some embodiments, the P predicted images include a second predicted image, and the encoding unit 23 is specifically used to obtain the offset corresponding to the current image based on the mixed spatiotemporal representation; perform spatial feature extraction on the previous reconstructed image to obtain reference feature information; and use the offset to perform motion compensation on the reference feature information to obtain the second predicted image.
[0537] In some embodiments, the encoding unit 23 is specifically configured to use the offset to perform deformable convolution-based motion compensation on the reference feature information to obtain the second predicted image.
[0538] In some embodiments, the encoding unit 23 is specifically configured to determine a target predicted image of the current image based on the P predicted images; and determine a reconstructed image of the current image based on the target predicted image.
[0539] In some embodiments, the encoding unit 23 is specifically configured to determine a weighted image according to the P predicted images; and obtain the target predicted image according to the weighted image.
[0540] In some embodiments, the encoding unit 23 is further configured to obtain a residual image of the current image according to the mixed spatiotemporal representation; and obtain the target predicted image according to the P predicted images and the residual image.
[0541] In some embodiments, if P is greater than 1, the encoding unit 23 is specifically configured to determine a weighted image based on the P predicted images; and determine the target predicted image based on the weighted image and the residual image.
[0542] In some embodiments, the encoding unit 23 is specifically configured to determine weights corresponding to the P predicted images; and weight the P predicted images according to the weights corresponding to the P predicted images to obtain the weighted image.
[0543] In some embodiments, the encoding unit 23 is specifically configured to perform adaptive masking according to the mixed spatiotemporal representation to obtain weights corresponding to the P predicted images.
[0544] In some embodiments, if the P predicted images include a first predicted image and a second predicted image, the encoding unit 23 is specifically used to determine the P predicted images, determine a first weight corresponding to the first predicted image and a second weight corresponding to the second predicted image; and weight the first predicted image and the second predicted image according to the first weight and the second weight to obtain the weighted image.
[0545] In some embodiments, the encoding unit 23 is further configured to determine a residual value of the current image according to the current image and the target predicted image; and encode the residual value to obtain a residual code stream.
[0546] In some embodiments, the encoding unit 23 is specifically configured to decode the residual code stream to obtain a residual value of the current image; and obtain the reconstructed image according to the target predicted image and the residual value.
[0547] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions can refer to the method embodiment. To avoid repetition, no further description is given here. Specifically, the video encoding device 20 shown in Figure 15 can correspond to the corresponding subject in the method of the embodiment of the present application, and the aforementioned and other operations and / or functions of each unit in the video encoding device 20 are respectively for implementing the corresponding processes in each method such as the method. For the sake of brevity, no further description is given here.
[0548] The above describes the apparatus and system of the embodiment of the present application from the perspective of functional units in conjunction with the accompanying drawings. It should be understood that the functional unit can be implemented in the form of hardware, can be implemented by instructions in the form of software, or can be implemented by a combination of hardware and software units. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software units in the decoding processor. Optionally, the software unit can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in conjunction with its hardware.
[0549] FIG16 is a schematic block diagram of an electronic device provided in an embodiment of the present application.
[0550] As shown in FIG16 , the electronic device 30 may be a video encoder or a video decoder as described in an embodiment of the present application. The electronic device 30 may include:
[0551] The memory 33 and the processor 32 are configured to store a computer program 34 and transmit the program code 34 to the processor 32. In other words, the processor 32 can call and run the computer program 34 from the memory 33 to implement the method in the embodiment of the present application.
[0552] For example, the processor 32 may be configured to execute the steps of the above method according to the instructions in the computer program 34 .
[0553] In some embodiments of the present application, the processor 32 may include but is not limited to:
[0554] General-purpose processor, Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.
[0555] In some embodiments of the present application, the memory 33 includes but is not limited to:
[0556] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).
[0557] In some embodiments of the present application, the computer program 34 may be divided into one or more units, which are stored in the memory 33 and executed by the processor 32 to implement the method provided by the present application. The one or more units may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program 34 in the electronic device 30.
[0558] As shown in FIG16 , the electronic device 30 may further include:
[0559] The transceiver 33 may be connected to the processor 32 or the memory 33 .
[0560] The processor 32 can control the transceiver 33 to communicate with other devices. Specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 33 may include a transmitter and a receiver. The transceiver 33 may further include an antenna, and the number of antennas may be one or more.
[0561] It should be understood that the various components in the electronic device 30 are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.
[0562] FIG17 is a schematic block diagram of a video encoding and decoding system 40 provided in an embodiment of the present application.
[0563] As shown in Figure 17, the video encoding and decoding system 40 may include: a video encoder 41 and a video decoder 42, wherein the video encoder 41 is used to execute the video encoding method involved in the embodiment of the present application, and the video decoder 42 is used to execute the video decoding method involved in the embodiment of the present application.
[0564] In some embodiments, the present application also provides a code stream, which is obtained by the above encoding method.
[0565] The present application also provides a computer storage medium having a computer program stored thereon, which, when executed by a computer, enables the computer to perform the method of the above-mentioned method embodiment. In other words, the present application also provides a computer program product containing instructions, which, when executed by a computer, enables the computer to perform the method of the above-mentioned method embodiment.
[0566] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid state drive (SSD)).
[0567] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0568] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the unit is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0569] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment. For example, the functional units in the various embodiments of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0570] The above content is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.
Claims
1. A video decoding method, characterized in that: include: Decoding the first bitstream to determine quantized first feature information, where the first feature information is obtained by performing feature fusion on a current image and a reconstructed image preceding the current image; Performing multi-level time-domain fusion on the quantized first feature information to obtain a hybrid spatiotemporal representation; Performing motion compensation on the previous reconstructed image according to the hybrid spatiotemporal representation to obtain P predicted images of the current image, where P is a positive integer; A reconstructed image of the current image is determined according to the P predicted images.
2. The method according to claim 1, characterized in that The performing multi-level time-domain fusion on the quantized first feature information to obtain a mixed spatiotemporal representation includes: The quantized first feature information is fused with the implicit feature information of the recursive aggregation module at the previous moment through a recursive aggregation module to obtain the mixed spatiotemporal representation.
3. The method according to claim 2, characterized in that The recursive aggregation module is formed by stacking at least one spatiotemporal recursive network.
4. The method according to claim 1, wherein The P predicted images include a first predicted image, and the performing motion compensation on the previous reconstructed image according to the hybrid spatiotemporal representation to obtain the P predicted images of the current image includes: determining optical flow motion information based on the hybrid spatiotemporal representation; Motion compensation is performed on the previous reconstructed image according to the optical flow motion information to obtain the first predicted image.
5. The method according to claim 1, wherein The P predicted images include a second predicted image, and the performing motion compensation on the previous reconstructed image according to the hybrid spatiotemporal representation to obtain the P predicted images of the current image includes: Obtaining an offset corresponding to the current image according to the mixed spatiotemporal representation; performing spatial feature extraction on the previous reconstructed image to obtain reference feature information; The reference feature information is motion compensated using the offset to obtain the second predicted image.
6. The method according to claim 5, characterized in that The performing motion compensation on the reference feature information using the offset to obtain the second predicted image includes: Using the offset, deformable convolution-based motion compensation is performed on the reference feature information to obtain the second predicted image.
7. The method according to any one of claims 1 to 6, characterized in that Determining a reconstructed image of the current image according to the P predicted images includes: Determining a target predicted image for the current image based on the P predicted images; A reconstructed image of the current image is determined according to the target predicted image.
8. The method according to claim 7, characterized in that If P is greater than 1, determining the target predicted image of the current image based on the P predicted images includes: Determining a weighted image based on the P predicted images; The target predicted image is obtained according to the weighted image.
9. The method according to claim 7, characterized in that The method further comprises: Obtaining a residual image of the current image according to the mixed spatiotemporal representation; Determining a target predicted image of the current image based on the P predicted images includes: The target prediction image is obtained according to the P prediction images and the residual image.
10. The method according to claim 9, characterized in that If P is greater than 1, obtaining the target predicted image according to the P predicted images and the residual image includes: Determining a weighted image based on the P predicted images; The target prediction image is determined according to the weighted image and the residual image.
11. The method according to claim 8 or 10, characterized in that Determining a weighted image based on the P predicted images includes: Determining weights corresponding to the P predicted images; The P predicted images are weighted according to the weights corresponding to the P predicted images to obtain the weighted image.
12. The method according to claim 11, characterized in that Determining the weights corresponding to the P predicted images includes: Adaptive masking is performed according to the hybrid spatiotemporal representation to obtain weights corresponding to the P predicted images.
13. The method according to claim 11, characterized in that If the P predicted images include a first predicted image and a second predicted image, determining weights corresponding to the P predicted images includes: determining a first weight corresponding to the first predicted image and a second weight corresponding to the second predicted image; The step of weighting the P predicted images according to the weights corresponding to the P predicted images to obtain the weighted image includes: The first predicted image and the second predicted image are weighted according to the first weight and the second weight to obtain the weighted image.
14. The method according to claim 7, wherein: The step of determining a reconstructed image of the current image according to the target predicted image includes: Decoding the residual code stream to obtain a residual value of the current image; The reconstructed image is obtained according to the target predicted image and the residual value.
15. The method according to any one of claims 1 to 6, characterized in that The decoding of the first code stream to determine the quantized first feature information includes: Decoding the second bit stream to obtain quantized second feature information, where the second feature information is obtained by performing feature transformation on the first feature information; determining a probability distribution of the quantized first feature information according to the quantized second feature information; The first code stream is decoded according to the probability distribution of the quantized first feature information to obtain the quantized first feature information.
16. The method according to claim 15, characterized in that The determining, based on the quantized second feature information, probability distribution information of the quantized first feature information includes: Performing an inverse transformation on the quantized second feature information to obtain reconstructed feature information; determining a probability distribution of the reconstructed feature information; According to the probability distribution of the reconstructed feature information, the probability distribution of the quantized first feature information is predicted.
17. The method according to claim 16, characterized in that The performing an inverse transformation on the quantized second feature information to obtain reconstructed feature information includes: Perform N non-local attention transformations and N upsamplings on the quantized second feature information to obtain the reconstructed feature information, where N is a positive integer.
18. The method according to claim 16, characterized in that The predicting, based on the probability distribution of the reconstructed feature information, a probability distribution of the quantized first feature information includes: Predicting the probability of coded pixels in the quantized first feature information according to the probability distribution of the reconstructed feature information; According to the probability of the encoded pixels in the quantized first feature information, a probability distribution of the quantized first feature information is obtained.
19. A video encoding method, characterized in that: include: Performing feature fusion on a current image and a reconstructed image previous to the current image to obtain first feature information; quantizing the first feature information to obtain quantized first feature information; The quantized first feature information is encoded to obtain a first code stream.
20. The method according to claim 19, characterized in that The performing feature fusion on the current image and the reconstructed image before the current image to obtain first feature information includes: Perform channel cascading on the current image and the reconstructed image to obtain a cascaded image; Feature extraction is performed on the concatenated image to obtain the first feature information.
21. The method according to claim 20, characterized in that The step of extracting features from the concatenated images to obtain the first feature information includes: Perform Q non-local attention transformations and Q downsamplings on the cascaded image to obtain the first feature information, where Q is a positive integer.
22. The method according to claim 19, wherein The encoding of the quantized first feature information to obtain the first code stream includes: Performing feature transformation according to the first feature information to obtain second feature information; quantizing and then encoding the second feature information to obtain a second bit stream; Decoding the second code stream to obtain quantized second feature information, and determining a probability distribution of the quantized first feature information based on the quantized second feature information; The quantized first feature information is encoded according to the probability distribution of the quantized first feature information to obtain a first code stream.
23. The method according to claim 22, characterized in that The performing feature transformation according to the first feature information to obtain second feature information includes: Perform N non-local attention transformations and N downsamplings on the first feature information to obtain the second feature information, where N is a positive integer.
24. The method according to claim 22, characterized in that The performing feature transformation according to the first feature information to obtain second feature information includes: Perform N non-local attention transformations and N downsamplings on the quantized first feature information to obtain the second feature information.
25. The method according to claim 22, wherein The quantizing and then encoding the second feature information to obtain a second bit stream includes: quantizing the second feature information to obtain quantized second feature information; Determining a probability distribution of the quantized second feature information; The quantized second feature information is encoded according to the probability distribution of the quantized second feature information to obtain the second code stream.
26. The method according to claim 22, characterized in that The determining, based on the quantized second feature information, probability distribution information of the quantized first feature information includes: Performing an inverse transformation on the quantized second feature information to obtain reconstructed feature information; determining a probability distribution of the reconstructed feature information; The probability distribution of the quantized first feature information is determined according to the probability distribution of the reconstructed feature information.
27. The method according to claim 26, characterized in that The performing an inverse transformation on the quantized second feature information to obtain reconstructed feature information includes: Perform N non-local attention transformations and N upsamplings on the quantized second feature information to obtain the reconstructed feature information, where N is a positive integer.
28. The method according to claim 26, characterized in that The determining, according to the probability distribution of the reconstructed feature information, the probability distribution of the quantized first feature information includes: determining, according to the probability distribution of the reconstructed feature information, the probability of the encoded pixel in the quantized first feature information; According to the probability of the encoded pixels in the quantized first feature information, a probability distribution of the quantized first feature information is obtained.
29. The method according to any one of claims 19 to 28, characterized in that The method further comprises: A reconstructed image of the current image is determined.
30. The method according to claim 29, wherein The determining of the reconstructed image of the current image includes: Performing multi-level time-domain fusion on the quantized first feature information to obtain a hybrid spatiotemporal representation; Performing motion compensation on the previous reconstructed image according to the hybrid spatiotemporal representation to obtain P predicted images of the current image, where P is a positive integer; A reconstructed image of the current image is determined according to the P predicted images.
31. The method according to claim 30, wherein The performing multi-level time-domain fusion on the quantized first feature information to obtain a mixed spatiotemporal representation includes: The quantized first feature information is fused with the implicit feature information of the recursive aggregation module at the previous moment through a recursive aggregation module to obtain the mixed spatiotemporal representation.
32. The method according to claim 31, characterized in that The recursive aggregation module is formed by stacking at least one spatiotemporal recursive network.
33. The method according to claim 30, wherein The P predicted images include a first predicted image, and the performing motion compensation on the previous reconstructed image according to the hybrid spatiotemporal representation to obtain the P predicted images of the current image includes: determining optical flow motion information based on the hybrid spatiotemporal representation; Motion compensation is performed on the previous reconstructed image according to the optical flow motion information to obtain the first predicted image.
34. The method according to claim 30, wherein The P predicted images include a second predicted image, and the performing motion compensation on the previous reconstructed image according to the hybrid spatiotemporal representation to obtain the P predicted images of the current image includes: Obtaining an offset corresponding to the current image according to the mixed spatiotemporal representation; performing spatial feature extraction on the previous reconstructed image to obtain reference feature information; The reference feature information is motion compensated using the offset to obtain the second predicted image.
35. The method according to claim 34, wherein The performing motion compensation on the reference feature information using the offset to obtain the second predicted image includes: Using the offset, deformable convolution-based motion compensation is performed on the reference feature information to obtain the second predicted image.
36. The method according to any one of claims 30 to 35, characterized in that Determining a reconstructed image of the current image based on the P predicted images includes: Determining a target predicted image for the current image based on the P predicted images; A reconstructed image of the current image is determined according to the target predicted image.
37. The method according to claim 36, wherein If P is greater than 1, determining the target predicted image of the current image based on the P predicted images includes: Determining a weighted image based on the P predicted images; The target predicted image is obtained according to the weighted image.
38. The method according to claim 36, characterized in that The method further comprises: Obtaining a residual image of the current image according to the mixed spatiotemporal representation; Determining a target predicted image of the current image based on the P predicted images includes: The target predicted image is obtained according to the P predicted images and the residual image.
39. The method according to claim 38, characterized in that If P is greater than 1, obtaining the target predicted image according to the P predicted images and the residual image includes: Determining a weighted image based on the P predicted images; The target prediction image is determined according to the weighted image and the residual image.
40. The method according to claim 37 or 39, characterized in that Determining a weighted image based on the P predicted images includes: Determining weights corresponding to the P predicted images; The P predicted images are weighted according to the weights corresponding to the P predicted images to obtain the weighted image.
41. The method according to claim 40, characterized in that Determining the weights corresponding to the P predicted images includes: Adaptive masking is performed according to the hybrid spatiotemporal representation to obtain weights corresponding to the P predicted images.
42. The method according to claim 41, wherein If the P predicted images include a first predicted image and a second predicted image, determining weights corresponding to the P predicted images includes: Determine the P predicted images, and determine a first weight corresponding to the first predicted image and a second weight corresponding to the second predicted image; The step of weighting the P predicted images according to the weights corresponding to the P predicted images to obtain the weighted image includes: The first predicted image and the second predicted image are weighted according to the first weight and the second weight to obtain the weighted image.
43. The method according to claim 36, wherein The method further comprises: Determining a residual value of the current image according to the current image and the target predicted image; The residual value is encoded to obtain a residual code stream.
44. The method according to claim 43, wherein The step of determining a reconstructed image of the current image according to the target predicted image includes: Decoding the residual code stream to obtain a residual value of the current image; The reconstructed image is obtained according to the target predicted image and the residual value.
45. A video decoding device, characterized in that include: a decoding unit, configured to decode the first code stream and determine quantized first feature information, where the first feature information is obtained by performing feature fusion on a current image and a reconstructed image preceding the current image; a fusion unit, configured to perform multi-level time-domain fusion on the quantized first feature information to obtain a hybrid spatiotemporal representation; a compensation unit, configured to perform motion compensation on the previous reconstructed image according to the hybrid spatiotemporal representation to obtain P predicted images of the current image, where P is a positive integer; A reconstruction unit is used to determine a reconstructed image of the current image based on the P predicted images.
46. A video encoding device, characterized in that include: a fusion unit, configured to perform feature fusion on a current image and a reconstructed image preceding the current image to obtain first feature information; a quantization unit, configured to quantize the first feature information to obtain quantized first feature information; The encoding unit is configured to encode the quantized first feature information to obtain a first code stream.
47. A video encoding and decoding system, characterized in that Including video encoder and video decoder; The video decoder is configured to execute the video decoding method according to any one of claims 1 to 19; The video encoder is used to execute the video encoding method according to any one of claims 20-44.
48. An electronic device, characterized in that include: Memory, processor; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the method according to any one of claims 1 to 19 or 20 to 44.
49. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 19 or 20 to 44 when executed by a processor.
50. A code stream, characterized in that The method comprises a code stream obtained by the method according to any one of claims 20 to 44.
