Video encoding methods, video decoding methods and apparatus

Abstract

This application provides a video encoding method, a video decoding method, and an apparatus thereof. The video encoding method includes: acquiring image semantic information of an encoded frame; acquiring residuals of the encoded frame; determining multiple regions in the encoded frame based on the image semantic information; simplifying the residuals corresponding to the multiple regions in the encoded frame to obtain candidate residuals of the encoded frame, wherein the bitrate of the candidate residuals corresponding to at least one of the multiple regions is less than the bitrate of the residuals corresponding to the at least one region; and encoding the image semantic information and the candidate residuals of the encoded frame. The method in this application embodiment can reduce the bitrate of video compression.

Description

Technical Field

[0001] This application relates to the field of video encoding and decoding, and more specifically, to a method for video encoding, a method for video decoding, and an apparatus thereof. Background Technology

[0002] Digital video capabilities can be incorporated into a wide variety of devices, including digital television, digital live broadcast systems, wireless broadcasting systems, personal digital assistants (PDAs), laptops or desktop computers, tablets, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite radio phones (so-called "smartphones"), video conferencing devices, video streaming devices, and the like. Digital video devices can implement video compression techniques, such as those described in standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4 Part 10 High-Level Video Coding (AVC), the H.265 / HEVC video coding standard, and extensions to such standards. Digital video devices can more efficiently transmit, receive, encode, decode, and / or store digital video information by implementing such video compression techniques. Video compression techniques can perform temporal (inter-frame) prediction to reduce or remove redundancy inherent in video sequences. Here, images can be referred to as frames.

[0003] Currently, people have increasingly higher requirements for video quality, with video continuously evolving from high definition to ultra-high definition (4Kx2K, 8Kx4K). This continuous improvement in clarity leads to a constant increase in video compression bitrate, which in turn increases costs related to bandwidth, transmission latency, and storage. Therefore, reducing the bitrate of video compression has become a pressing technical problem that needs to be solved. Summary of the Invention

[0004] This application provides a video encoding method, a video decoding method, and an apparatus thereof, which can reduce the bit rate of video compression.

[0005] Firstly, a video encoding method is provided, the method comprising:

[0006] Obtain image semantic information of the encoded frame; obtain the residual of the encoded frame; determine multiple regions in the encoded frame based on the image semantic information; simplify the residuals corresponding to the multiple regions in the encoded frame to obtain candidate residuals of the encoded frame, wherein the bitrate of the candidate residuals corresponding to at least one of the multiple regions is less than the bitrate of the residuals corresponding to at least one region; encode the image semantic information and the candidate residuals of the encoded frame.

[0007] In this embodiment of the application, multiple regions in the encoded frame are determined based on the image semantic information, and the residuals corresponding to the multiple regions in the encoded frame are simplified so that the bitrate of the candidate residuals corresponding to at least one of the multiple regions is less than the bitrate of the residuals corresponding to the at least one region. At this time, encoding the image semantic information and the candidate residuals of the encoded frame can reduce the bitrate of video compression.

[0008] The image semantic information may include the image semantics of the encoded frame, instance segmentation of the encoded frame, semantic segmentation of the encoded frame, edge information of the encoded frame, and key point information in the encoded frame.

[0009] The plurality of regions may include regions of interest (ROIs) in the encoded frame. The ROIs may be determined based on the purpose of the video to be encoded.

[0010] For example, if the video to be encoded is used to detect the driver's driving behavior, then the region of interest may refer to the driver region or the vehicle region in the encoded frame (the driver region is located in the vehicle region); or, if the video to be encoded is used to detect vehicle violations, then the region of interest may refer to both the vehicle region and the road surface region in the encoded frame.

[0011] Optionally, the image semantic information may include the image semantics of the encoded frame, instance segmentation of the encoded frame, semantic segmentation of the encoded frame, edge information of the encoded frame, and key point information in the encoded frame.

[0012] The simplification process helps reduce the code rate after residual block encoding.

[0013] For example, regions other than the ROI can be simplified in the multiple regions so that the bitrate of the candidate residuals of the other regions is less than the bitrate of the residuals of the other regions. In this case, the ROI can be left unsimplified, that is, the candidate residuals of the ROI are actually the same as the residuals of the ROI.

[0014] Optionally, the simplification process can be quantization. Quantization can reduce the candidate set of pixel values ​​in an image.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, the simplification processing of the residuals corresponding to multiple regions in the encoded frame includes: simplifying the residuals corresponding to the multiple regions using a variety of different simplification strategies.

[0016] In this embodiment, based on the different importance of each of the multiple regions, various simplification strategies are adopted to perform different simplification processing on the residuals corresponding to the multiple regions, which can more flexibly reduce the bitrate of video compression.

[0017] The importance here can be determined based on the intended use of the video to be encoded.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, the encoded frame is one of a plurality of video frames included in the video to be encoded, and the video to be encoded further includes at least one keyframe; wherein, the method further includes: obtaining video semantic information of the video to be encoded, the video semantic information representing the changes between the plurality of video frames included in the video to be encoded.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, obtaining the residual of the encoded frame includes: using a first neural network model to generate a predicted image of the encoded frame based on a reference frame of the encoded frame and the video semantic information; and obtaining the residual of the encoded frame based on the original image of the encoded frame and the predicted image of the encoded frame.

[0020] In this embodiment of the application, the video semantic information represents the changes between multiple video frames included in the video to be encoded. By generating a predicted image of the encoded frame based on the reference frame of the encoded frame and the video semantic information, the changes between the generated predicted images can be smoother or more coherent.

[0021] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: adjusting the parameters of the first neural network model based on the residual of the encoded frame and the predicted image of the encoded frame.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, obtaining the residual of the encoded frame based on the original image of the encoded frame and the predicted image of the encoded frame includes: using a second neural network model to post-process the predicted images of multiple video frames included in the video to be encoded based on the video semantic information to obtain candidate predicted images of the multiple video frames, wherein the post-processing enables smooth transitions between the candidate predicted images of the multiple video frames; and subtracting the original image of the encoded frame from the candidate predicted image of the encoded frame to obtain the residual of the encoded frame.

[0023] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: adjusting the parameters of the second neural network model based on the original image of the encoded frame and the candidate predicted image of the encoded frame.

[0024] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: determining a video frame that satisfies at least one of the following as a key frame of the video to be encoded: the video frame contains a preset image, the difference between the video frame and its previous frame is greater than or equal to a first preset value, and the residual of the video frame is greater than or equal to a second preset value.

[0025] In this embodiment of the application, the keyframes of the video to be encoded are flexibly determined using the above method according to the requirements of the video to be encoded, thereby improving the video encoding effect.

[0026] In conjunction with the first aspect, in some implementations of the first aspect, encoding the image semantic information and the candidate residual of the encoded frame includes: encoding the reference frame, the image semantic information of the reference frame, the image semantic information and the candidate residual of the encoded frame.

[0027] Secondly, a video decoding method is provided, the method comprising:

[0028] The image semantic information of the encoded frame and the candidate residual of the encoded frame are obtained by parsing the bitstream. The candidate residual is obtained by simplifying the residuals corresponding to multiple regions in the encoded frame. The bitrate of the candidate residual corresponding to at least one of the multiple regions is less than the bitrate of the residual corresponding to the at least one region. The target image of the encoded frame is determined based on the image semantic information and the candidate residual of the encoded frame.

[0029] In this embodiment, the candidate residual is obtained by simplifying the residuals corresponding to multiple regions in the encoded frame. The bitrate of the candidate residual encoded by at least one of the multiple regions is less than the bitrate of the residual encoded by the at least one region. In this case, the target image of the encoded frame can be determined according to the image semantic information and the candidate residual of the encoded frame, which can reduce the bitrate of video compression.

[0030] The image semantic information may include the image semantics of the encoded frame, instance segmentation of the encoded frame, semantic segmentation of the encoded frame, edge information of the encoded frame, and key point information in the encoded frame.

[0031] The plurality of regions may include regions of interest (ROIs) in the encoded frame. The ROIs may be determined based on the purpose of the video to be encoded.

[0032] For example, if the video to be encoded is used to detect the driver's driving behavior, then the region of interest may refer to the driver region or the vehicle region in the encoded frame (the driver region is located in the vehicle region); or, if the video to be encoded is used to detect vehicle violations, then the region of interest may refer to both the vehicle region and the road surface region in the encoded frame.

[0033] Optionally, the image semantic information may include the image semantics of the encoded frame, instance segmentation of the encoded frame, semantic segmentation of the encoded frame, edge information of the encoded frame, and key point information in the encoded frame.

[0034] The simplification process helps reduce the code rate after residual block encoding.

[0035] For example, regions other than the ROI can be simplified in the multiple regions so that the bitrate of the candidate residuals of the other regions is less than the bitrate of the residuals of the other regions. In this case, the ROI can be left unsimplified, that is, the candidate residuals of the ROI are actually the same as the residuals of the ROI.

[0036] Optionally, the simplification process can be quantization. Quantization can reduce the candidate set of pixel values ​​in an image.

[0037] In conjunction with the second aspect, in some implementations of the second aspect, the candidate residual is obtained by simplifying the residuals corresponding to multiple regions in the coded frame using various different simplification strategies.

[0038] In this embodiment, based on the different importance of each of the multiple regions, various simplification strategies are adopted to perform different simplification processing on the residuals corresponding to the multiple regions, which can more flexibly reduce the bitrate of video compression.

[0039] The importance here can be determined based on the intended use of the video to be encoded.

[0040] In conjunction with the second aspect, in some implementations of the second aspect, the encoded frame is one of a plurality of video frames included in the video to be encoded, and the video to be encoded also includes at least one keyframe; wherein, the method further includes: parsing the bitstream to obtain video semantic information of the video to be encoded, the video semantic information representing the changes between the plurality of video frames included in the video to be encoded.

[0041] In conjunction with the second aspect, in some implementations of the second aspect, determining the target image of the encoded frame based on the image semantic information and the candidate residual includes: using a third neural network model to generate a predicted image of the encoded frame based on a reference frame of the encoded frame and the video semantic information; and obtaining the target image of the encoded frame based on the predicted image of the encoded frame and the candidate residual of the encoded frame.

[0042] In this embodiment of the application, the video semantic information represents the changes between multiple video frames included in the video to be encoded. By generating a predicted image of the encoded frame based on the reference frame of the encoded frame and the video semantic information, the changes between the generated predicted images can be smoother or more coherent.

[0043] In conjunction with the second aspect, in some implementations of the second aspect, the method further includes: adjusting the parameters of the third neural network model based on the target image of the encoded frame and the predicted image of the encoded frame.

[0044] In conjunction with the second aspect, in some implementations of the second aspect, obtaining the target image of the encoded frame based on the predicted image of the encoded frame and the candidate residual includes: using a fourth neural network model to post-process the predicted images of multiple video frames included in the video to be encoded based on the video semantic information to obtain candidate predicted images of the multiple video frames, wherein the post-processing enables smooth transitions between the candidate predicted images of the multiple video frames; and adding the candidate predicted image of the encoded frame to the candidate residual of the encoded frame to obtain the target image of the encoded frame.

[0045] In conjunction with the second aspect, in some implementations of the second aspect, the method further includes: adjusting the parameters of the fourth neural network model based on the target image of the encoded frame and the candidate prediction image of the encoded frame.

[0046] Thirdly, a video encoding apparatus is provided, comprising:

[0047] A first acquisition unit is used to acquire image semantic information of an encoded frame; a second acquisition unit is used to acquire the residual of the encoded frame; a first determination unit is used to determine multiple regions in the encoded frame based on the image semantic information; a simplification processing unit is used to simplify the residuals corresponding to the multiple regions in the encoded frame to obtain candidate residuals of the encoded frame, wherein the bitrate of the candidate residuals corresponding to at least one of the multiple regions is less than the bitrate of the residuals corresponding to at least one region; and an encoding unit is used to encode the image semantic information and the candidate residuals of the encoded frame.

[0048] In this embodiment of the application, multiple regions in the encoded frame are determined based on the image semantic information, and the residuals corresponding to the multiple regions in the encoded frame are simplified so that the bitrate of the candidate residuals corresponding to at least one of the multiple regions is less than the bitrate of the residuals corresponding to the at least one region. At this time, encoding the image semantic information and the candidate residuals of the encoded frame can reduce the bitrate of video compression.

[0049] The image semantic information may include the image semantics of the encoded frame, instance segmentation of the encoded frame, semantic segmentation of the encoded frame, edge information of the encoded frame, and key point information in the encoded frame.

[0050] The plurality of regions may include regions of interest (ROIs) in the encoded frame. The ROIs may be determined based on the purpose of the video to be encoded.

[0051] For example, if the video to be encoded is used to detect the driver's driving behavior, then the region of interest may refer to the driver region or the vehicle region in the encoded frame (the driver region is located in the vehicle region); or, if the video to be encoded is used to detect vehicle violations, then the region of interest may refer to both the vehicle region and the road surface region in the encoded frame.

[0052] Optionally, the image semantic information may include the image semantics of the encoded frame, instance segmentation of the encoded frame, semantic segmentation of the encoded frame, edge information of the encoded frame, and key point information in the encoded frame.

[0053] The simplification process helps reduce the code rate after residual block encoding.

[0054] For example, regions other than the ROI can be simplified in the multiple regions so that the bitrate of the candidate residuals of the other regions is less than the bitrate of the residuals of the other regions. In this case, the ROI can be left unsimplified, that is, the candidate residuals of the ROI are actually the same as the residuals of the ROI.

[0055] Optionally, the simplification process can be quantization. Quantization can reduce the candidate set of pixel values ​​in an image.

[0056] In conjunction with the third aspect, in some implementations of the third aspect, the simplification processing unit is specifically used to: simplify the residuals corresponding to the multiple regions using a variety of different simplification strategies.

[0057] In this embodiment, based on the different importance of each of the multiple regions, various simplification strategies are adopted to perform different simplification processing on the residuals corresponding to the multiple regions, which can more flexibly reduce the bitrate of video compression.

[0058] The importance here can be determined based on the intended use of the video to be encoded.

[0059] In conjunction with the third aspect, in some implementations of the third aspect, the encoded frame is one of a plurality of video frames included in the video to be encoded, and the video to be encoded also includes at least one keyframe; wherein, the apparatus further includes a third acquisition unit, configured to: acquire video semantic information of the video to be encoded, the video semantic information representing the changes between the plurality of video frames included in the video to be encoded.

[0060] In conjunction with the third aspect, in some implementations of the third aspect, the second acquisition unit is specifically used to: use a first neural network model to generate a predicted image of the encoded frame based on a reference frame of the encoded frame and the video semantic information; and obtain the residual of the encoded frame based on the original image of the encoded frame and the predicted image of the encoded frame.

[0061] In this embodiment of the application, the video semantic information represents the changes between multiple video frames included in the video to be encoded. By generating a predicted image of the encoded frame based on the reference frame of the encoded frame and the video semantic information, the changes between the generated predicted images can be smoother or more coherent.

[0062] In conjunction with the third aspect, in some implementations of the third aspect, the apparatus further includes an adjustment unit for: adjusting the parameters of the first neural network model based on the residual of the encoded frame and the predicted image of the encoded frame.

[0063] In conjunction with the third aspect, in some implementations of the third aspect, the second acquisition unit is specifically used to: use a second neural network model to post-process the predicted images of multiple video frames included in the video to be encoded based on the video semantic information to obtain candidate predicted images of the multiple video frames, wherein the post-processing enables smooth transitions between the candidate predicted images of the multiple video frames; and subtract the original image of the encoded frame from the candidate predicted image of the encoded frame to obtain the residual of the encoded frame.

[0064] In conjunction with the third aspect, in some implementations of the third aspect, the apparatus further includes an adjustment unit for: adjusting the parameters of the second neural network model based on the original image of the encoded frame and the candidate predicted image of the encoded frame.

[0065] In conjunction with the third aspect, in some implementations of the third aspect, the apparatus further includes a second determining unit, configured to: determine a video frame that satisfies at least one of the following as a key frame of the video to be encoded: the video frame contains a preset image, the difference between the video frame and its previous frame is greater than or equal to a first preset value, and the residual of the video frame is greater than or equal to a second preset value.

[0066] In this embodiment of the application, the keyframes of the video to be encoded are flexibly determined using the above method according to the requirements of the video to be encoded, thereby improving the video encoding effect.

[0067] In conjunction with the third aspect, in some implementations of the third aspect, the encoding unit is specifically used to: encode the reference frame, the image semantic information of the reference frame, the image semantic information, and the candidate residual of the encoded frame.

[0068] Fourthly, a video decoding apparatus is provided, comprising:

[0069] A decoding unit is used to parse the bitstream to obtain the image semantic information of the encoded frame and the candidate residuals of the encoded frame, wherein the candidate residuals are obtained by simplifying the residuals corresponding to multiple regions in the encoded frame, and the bitrate of the candidate residuals corresponding to at least one of the multiple regions is less than the bitrate of the residuals corresponding to at least one region; a determining unit is used to determine the target image of the encoded frame based on the image semantic information and the candidate residuals of the encoded frame.

[0070] In this embodiment, the candidate residual is obtained by simplifying the residuals corresponding to multiple regions in the encoded frame. The bitrate of the candidate residual encoded by at least one of the multiple regions is less than the bitrate of the residual encoded by the at least one region. In this case, the target image of the encoded frame can be determined according to the image semantic information and the candidate residual of the encoded frame, which can reduce the bitrate of video compression.

[0071] The image semantic information may include the image semantics of the encoded frame, instance segmentation of the encoded frame, semantic segmentation of the encoded frame, edge information of the encoded frame, and key point information in the encoded frame.

[0072] The plurality of regions may include regions of interest (ROIs) in the encoded frame. The ROIs may be determined based on the purpose of the video to be encoded.

[0073] For example, if the video to be encoded is used to detect the driver's driving behavior, then the region of interest may refer to the driver region or the vehicle region in the encoded frame (the driver region is located in the vehicle region); or, if the video to be encoded is used to detect vehicle violations, then the region of interest may refer to both the vehicle region and the road surface region in the encoded frame.

[0074] Optionally, the image semantic information may include the image semantics of the encoded frame, instance segmentation of the encoded frame, semantic segmentation of the encoded frame, edge information of the encoded frame, and key point information in the encoded frame.

[0075] The simplification process helps reduce the code rate after residual block encoding.

[0076] For example, regions other than the ROI can be simplified in the multiple regions so that the bitrate of the candidate residuals of the other regions is less than the bitrate of the residuals of the other regions. In this case, the ROI can be left unsimplified, that is, the candidate residuals of the ROI are actually the same as the residuals of the ROI.

[0077] Optionally, the simplification process can be quantization. Quantization can reduce the candidate set of pixel values ​​in an image.

[0078] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the candidate residual is obtained by simplifying the residuals corresponding to multiple regions in the coded frame using various different simplification strategies.

[0079] In this embodiment, based on the different importance of each of the multiple regions, various simplification strategies are adopted to perform different simplification processing on the residuals corresponding to the multiple regions, which can more flexibly reduce the bitrate of video compression.

[0080] The importance here can be determined based on the intended use of the video to be encoded.

[0081] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the encoded frame is one of a plurality of video frames included in the video to be encoded, and the video to be encoded also includes at least one keyframe; wherein, the decoding unit is further configured to: parse the bitstream to obtain video semantic information of the video to be encoded, the video semantic information representing the changes between the plurality of video frames included in the video to be encoded.

[0082] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the determining unit is further configured to: use a third neural network model to generate a predicted image of the encoded frame based on a reference frame of the encoded frame and the video semantic information; and obtain a target image of the encoded frame based on the predicted image of the encoded frame and the candidate residual of the encoded frame.

[0083] In this embodiment of the application, the video semantic information represents the changes between multiple video frames included in the video to be encoded. By generating a predicted image of the encoded frame based on the reference frame of the encoded frame and the video semantic information, the changes between the generated predicted images can be smoother or more coherent.

[0084] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the apparatus further includes an adjustment unit for: adjusting the parameters of the third neural network model based on the target image of the encoded frame and the predicted image of the encoded frame.

[0085] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the determining unit is further configured to: use a fourth neural network model to post-process the predicted images of multiple video frames included in the video to be encoded based on the video semantic information to obtain candidate predicted images of the multiple video frames, wherein the post-processing enables smooth transitions between the candidate predicted images of the multiple video frames; and add the candidate predicted image of the encoded frame to the candidate residual of the encoded frame to obtain the target image of the encoded frame.

[0086] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the apparatus further includes an adjustment unit for: adjusting the parameters of the fourth neural network model based on the target image of the encoded frame and the candidate prediction image of the encoded frame.

[0087] Fifthly, a video encoding apparatus is provided, the apparatus comprising a storage medium and a central processing unit, the storage medium being a non-volatile storage medium storing a computer-executable program, the central processing unit being connected to the non-volatile storage medium and executing the computer-executable program to implement the method of the first aspect or any possible implementation thereof.

[0088] In a sixth aspect, a video decoding apparatus is provided, the apparatus comprising a storage medium and a central processing unit, the storage medium being a non-volatile storage medium storing a computer-executable program, the central processing unit being connected to the non-volatile storage medium and executing the computer-executable program to implement the method of the second aspect or any possible implementation thereof.

[0089] In a seventh aspect, a chip is provided, the chip including a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface and executes the method of the first aspect or any possible implementation thereof.

[0090] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is configured to execute the instructions stored in the memory. When the instructions are executed, the processor is configured to perform the method in the first aspect or any possible implementation thereof.

[0091] Eighthly, a chip is provided, the chip including a processor and a data interface, the processor reading instructions stored in a memory through the data interface to execute the method of the second aspect or any possible implementation thereof.

[0092] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is configured to execute the instructions stored in the memory. When the instructions are executed, the processor is configured to perform the method in the second aspect or any possible implementation of the second aspect.

[0093] A ninth aspect provides a computer-readable storage medium storing program code for execution by a device, the program code including instructions for performing the method of the first aspect or any possible implementation thereof.

[0094] In a tenth aspect, a computer-readable storage medium is provided, the computer-readable medium storing program code for execution by a device, the program code including instructions for performing the method of the second aspect or any possible implementation thereof.

[0095] In this embodiment of the application, multiple regions in the encoded frame are determined based on the image semantic information, and the residuals corresponding to the multiple regions in the encoded frame are simplified so that the bitrate of the candidate residuals corresponding to at least one of the multiple regions is less than the bitrate of the residuals corresponding to the at least one region. At this time, encoding the image semantic information and the candidate residuals of the encoded frame can reduce the bitrate of video compression. Attached Figure Description

[0096] Figure 1 This is a schematic block diagram illustrating an example of a video encoding system used to implement embodiments of this application.

[0097] Figure 2 This is a schematic structural block diagram of a video decoding device example used to implement the embodiments of this application.

[0098] Figure 3 This is a schematic block diagram for implementing an example of an encoding or decoding device in the embodiments of this application.

[0099] Figure 4 This is a schematic block diagram of a video communication system used to implement the embodiments of this application.

[0100] Figure 5 This is a schematic flowchart of a video encoding method according to an embodiment of this application.

[0101] Figure 6 This is a schematic flowchart of a video decoding method according to an embodiment of this application.

[0102] Figure 7 This is a schematic flowchart of a video encoding and decoding method according to an embodiment of this application.

[0103] Figure 8 This is a schematic block diagram illustrating the determination of an I-frame in a video segment in one embodiment of this application.

[0104] Figure 9 This is a schematic block diagram illustrating the extraction of semantic information from video segments in one embodiment of this application.

[0105] Figure 10 This is a schematic block diagram illustrating the generation of a predicted image in one embodiment of this application.

[0106] Figure 11 This is a schematic block diagram illustrating the generation of a predicted image in another embodiment of this application.

[0107] Figure 12 This is a schematic block diagram illustrating the generation of a predicted image in another embodiment of this application.

[0108] Figure 13 This is a schematic block diagram illustrating post-processing of video segments in one embodiment of this application.

[0109] Figure 14 This is a schematic block diagram of a video encoding apparatus according to an embodiment of this application.

[0110] Figure 15 This is a schematic block diagram of a video decoding apparatus according to an embodiment of this application.

[0111] Figure 16 This is a schematic block diagram of a video encoding / decoding apparatus according to an embodiment of this application. Detailed Implementation

[0112] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0113] In the following description, reference will be made to the accompanying drawings, which form part of this application and illustrate specific aspects of embodiments of this application or from which specific aspects of embodiments of this application may be used. It should be understood that embodiments of this application may also be used in other aspects and may include structural or logical variations not depicted in the drawings. Therefore, the following detailed description should not be construed in a limiting sense, and the scope of this application should be defined by the appended claims.

[0114] For example, it should be understood that the disclosure of the described methods can be equally applied to the corresponding devices or systems that perform the methods, and vice versa.

[0115] For example, if one or more specific method steps are described, the corresponding device may include one or more units, such as functional units, to perform the described one or more method steps (e.g., one unit performs one or more steps, or multiple units, each of which performs one or more of the multiple steps), even if such one or more units are not explicitly described or illustrated in the accompanying drawings.

[0116] Furthermore, if a specific apparatus is described based on one or more units such as functional units, the corresponding method may include a step to perform the function of one or more units (e.g., a step to perform the function of one or more units, or multiple steps, wherein each performs the function of one or more units among a plurality of units), even if such one or more steps are not explicitly described or illustrated in the accompanying drawings. Further, it should be understood that, unless otherwise expressly stated, features of the various exemplary embodiments and / or aspects described herein can be combined with each other.

[0117] The technical solutions involved in the embodiments of this application can be applied to the H.266 standard and future video coding standards. The terminology used in the implementation section of this application is only for explaining specific embodiments and is not intended to limit the application. A brief introduction to some concepts that may be involved in the embodiments of this application is given below.

[0118] Video coding generally refers to the processing of a sequence of images that form a video or video sequence. In the field of video coding, the terms "picture," "frame," or "image" can be used synonymously. As used herein, video coding refers to either video encoding or video decoding. Video coding is performed on the source side and typically involves processing (e.g., by compression) the raw video images to reduce the amount of data required to represent them, thereby enabling more efficient storage and / or transmission. Video decoding is performed on the destination side and typically involves inverse processing relative to the encoder to reconstruct the video images. The "encoding" of video images in the embodiments should be understood as involving the "encoding" or "decoding" of a video sequence. The combination of encoding and decoding portions is also referred to as encoding and decoding (encoding and decoding).

[0119] The basic unit of video encoding and decoding can be called a group of pictures (GOP). Each GOP consists of an I-frame and several P-frames or B-frames. The I-frame is an internally coded frame, also known as a keyframe. The P-frame is a forward predictive coded frame, and the B-frame is a bidirectional predictive coded frame.

[0120] An I-frame is a complete image that can be directly encoded and decoded. A P-frame (or B-frame) represents the motion changes of the P-frame (or B-frame) relative to its reference frame. A P-frame (or B-frame) needs to be encoded and decoded based on its reference frame.

[0121] A video codec can encode and decode multiple Groups of Pictures (GOPs) in a video sequence sequentially until the video ends. At the encoding end, the codec encodes multiple images, generating multiple GOPs. These GOPs are transmitted over a network to the decoding end or stored on hard drive for the decoder to access. The video decoder reads these GOPs, decodes them, and generates a continuous video stream.

[0122] A video sequence consists of a series of pictures, which are further divided into slices, and slices into blocks. Video coding is performed on a block-by-block basis. In some newer video coding standards, the concept of a block has been further expanded. For example, the H.264 standard uses macroblocks (MBs), which can be further divided into multiple prediction blocks (partitions) for predictive coding. The High Efficiency Video Coding (HEVC) standard uses basic concepts such as coding units (CUs), prediction units (PUs), and transform units (TUs) to functionally divide various block units, and employs a novel tree-based structure for description. For instance, a CU can be divided into smaller CUs using a quadtree, and these smaller CUs can be further divided, forming a quadtree structure. The CU is the basic unit for partitioning and encoding the image. Similar tree structures exist for PUs and TUs. A PU corresponds to a prediction block and is the basic unit for predictive coding. CUs are further divided into multiple PUs according to partitioning patterns. TU can correspond to a transform block, which is the basic unit for transforming the prediction residual. However, whether it is CU, PU or TU, they all essentially belong to the concept of a block (or image block).

[0123] For example, in HEVC, the CTU is split into multiple CUs using a quadtree structure represented as a coding tree. At the CU level, a decision is made on whether to use inter-picture (temporal) or intra-picture (spatial) prediction to encode picture regions. Each CU can be further split into one, two, or four PUs based on the PU splitting type. The same prediction process is applied within a PU, and relevant information is passed to the decoder based on the PU. After obtaining residual blocks by applying the prediction process based on the PU splitting type, the CU can be segmented into transform units (TUs) according to other quadtree structures similar to the coding tree used for CUs. In recent developments in video compression technology, quad-tree and binary tree (QTBT) frame segmentation is used to divide coding blocks. In the QTBT block structure, CUs can be square or rectangular in shape.

[0124] In this paper, for ease of description and understanding, the image block to be encoded in the current encoded image is referred to as the current image block. For example, in encoding, it refers to the block currently being encoded; in decoding, it refers to the block currently being decoded. The decoded image block in the reference image used to predict the current image block is called the reference block. That is, the reference block is the block that provides a reference signal for the current image block, where the reference signal represents the pixel value within the image block. The block in the reference image that provides the prediction signal for the current image block can be called the prediction block, where the prediction signal represents the pixel value, sampled value, or sampled signal within the prediction block. For example, after traversing multiple reference blocks, an optimal reference block is found. This optimal reference block will provide prediction for the current image block; this block is called the prediction block.

[0125] In lossless video coding, the original video images can be reconstructed, meaning the reconstructed video images have the same quality as the original (assuming no transmission loss or other data loss during storage or transmission). In lossy video coding, further compression is performed, for example, through quantization, to reduce the amount of data required to represent the video images. However, the decoder cannot fully reconstruct the video images, meaning the quality of the reconstructed video images is lower or worse than the original.

[0126] Several H.261 video coding standards fall under the category of "lossy hybrid video coding and decoding" (i.e., combining spatial and temporal prediction in the sample domain with 2D transform coding in the transform domain for applying quantization). Each frame in a video sequence is typically segmented into a set of non-overlapping blocks, which are usually encoded at the block level. In other words, the encoder typically processes, i.e., encodes the video at the block (video block) level, for example, by generating prediction blocks through spatial (intra-frame) and temporal (inter-frame) predictions, subtracting the prediction blocks from the current image block (the block currently being processed or to be processed) to obtain residual blocks, transforming and quantizing the residual blocks in the transform domain to reduce the amount of data to be transmitted (compressed), while the decoder applies the inverse processing relative to the encoder to the encoded or compressed blocks to reconstruct the current image block for representation. Additionally, the encoder replicates the decoder processing loop, causing the encoder and decoder to generate the same predictions (e.g., intra-frame and inter-frame predictions) and / or reconstructions for processing, i.e., encoding subsequent blocks.

[0127] The system architecture used in the embodiments of this application is described below. See also Figure 1 , Figure 1 A schematic block diagram of the video encoding and decoding system 10 used in the embodiments of this application is provided as an example. Figure 1 As shown, the video encoding and decoding system 10 may include a source device 12 and a destination device 14. The source device 12 generates encoded video data; therefore, the source device 12 may be referred to as a video encoding device. The destination device 14 can decode the encoded video data generated by the source device 12; therefore, the destination device 14 may be referred to as a video decoding device. Various embodiments of the source device 12, the destination device 14, or both may include one or more processors and memory coupled to said one or more processors. The memory may include, but is not limited to, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), flash memory, or any other media that can be used to store desired program code in the form of computer-accessible instructions or data structures, as described herein. The source device 12 and the destination device 14 may include a variety of devices, including desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, handsets such as so-called "smart" phones, televisions, cameras, display devices, digital media players, video game consoles, in-vehicle computers, wireless communication devices, or the like.

[0128] Although Figure 1Source device 12 and destination device 14 are illustrated as separate devices, but device embodiments may also include the functionality of both source device 12 and destination device 14, or both; that is, the functionality of source device 12 or its corresponding function and the functionality of destination device 14 or its corresponding function. In such embodiments, the functionality of source device 12 or its corresponding function and the functionality of destination device 14 or its corresponding function may be implemented using the same hardware and / or software, or using separate hardware and / or software, or any combination thereof.

[0129] A communication connection can be established between source device 12 and destination device 14 via link 13, through which destination device 14 can receive encoded video data from source device 12. Link 13 may include one or more media or devices capable of moving encoded video data from source device 12 to destination device 14. In one example, link 13 may include one or more communication media enabling source device 12 to transmit encoded video data directly to destination device 14 in real time. In this example, source device 12 may modulate the encoded video data according to a communication standard (e.g., a wireless communication protocol) and transmit the modulated video data to destination device 14. The one or more communication media may include wireless and / or wired communication media, such as radio frequency (RF) spectrum or one or more physical transmission lines. The one or more communication media may form part of a packet-based network, such as a local area network, wide area network, or global network (e.g., the Internet). The one or more communication media may include routers, switches, base stations, or other devices facilitating communication from source device 12 to destination device 14.

[0130] The source device 12 includes an encoder 20. Optionally, the source device 12 may also include an image source 16, an image preprocessor 18, and a communication interface 22. In specific implementations, the encoder 20, image source 16, image preprocessor 18, and communication interface 22 may be hardware components or software programs in the source device 12.

[0131] The descriptions are as follows:

[0132] Image source 16 may include or be any type of image capture device for, for example, capturing real-world images, and / or any type of image or commentary (for screen content encoding, some text on the screen is also considered part of the image or picture to be encoded) generation device, such as a computer graphics processor for generating computer-animated images, or any type of device for acquiring and / or providing real-world images, computer-animated images (e.g., screen content, virtual reality (VR) images), and / or any combination thereof (e.g., augmented reality (AR) images). Image source 16 may be a camera for capturing images or a memory for storing images. Image source 16 may also include any type of (internal or external) interface for storing previously captured or generated images and / or acquiring or receiving images. When image source 16 is a camera, image source 16 may be, for example, a local or integrated camera integrated into a source device; when image source 16 is a memory, image source 16 may be a local or, for example, integrated memory integrated into a source device. When the image source 16 includes an interface, the interface may be, for example, an external interface for receiving images from an external video source. The external video source may be, for example, an external image capture device, such as a camera, external storage, or an external image generation device. The external image generation device may be, for example, an external computer graphics processor, a computer, or a server. The interface may be any type of interface according to any proprietary or standardized interface protocol, such as a wired or wireless interface, or an optical interface.

[0133] An image can be viewed as a two-dimensional array or matrix of pixels. Pixels in the array are also called sampling points. The number of sampling points in the array or image along the horizontal and vertical directions (or axes) defines the image's size and / or resolution. To represent color, three color components are typically used; that is, an image can be represented as or contain three sampling arrays. For example, in RGB format or color space, an image includes corresponding red, green, and blue sampling arrays. However, in video coding, each pixel is typically represented in a luma / chroma format or color space. For example, for a YUV format image, this includes a luma component indicated by Y (sometimes also indicated by L) and two chroma components indicated by U and V. The luma component Y represents the brightness or grayscale level intensity (e.g., both are the same in grayscale images), while the two chroma components U and V represent chroma or color information components. Accordingly, a YUV format image includes a luma sampling array of luma sample values ​​(Y) and two chroma sampling arrays of chroma values ​​(U and V). An RGB format image can be converted or transformed to a YUV format, and vice versa; this process is also called color transformation or conversion. If the image is black and white, it may only include a luminance sampling array. In this embodiment, the image transmitted from image source 16 to image processor can also be referred to as raw image data 17.

[0134] Image preprocessor 18 is configured to receive raw image data 17 and perform preprocessing on the raw image data 17 to obtain a preprocessed image 19 or preprocessed image data 19. For example, the preprocessing performed by image preprocessor 18 may include retouching, color format conversion (e.g., from RGB format to YUV format), color correction, or noise reduction.

[0135] Encoder 20 (or video encoder 20) is used to receive preprocessed image data 19 and process the preprocessed image data 19 using a relevant prediction mode (such as the prediction mode in the various embodiments of this document), thereby providing encoded image data 21 (which will be further based on...). Figure 2 (Description of structural details of encoder 20). In some embodiments, encoder 20 can be used to perform the various embodiments described below to implement the video encoding method described in this application on the encoding side.

[0136] Communication interface 22 can be used to receive encoded image data 21 and transmit the encoded image data 21 to destination device 14 or any other device (such as a memory) via link 13 for storage or direct reconstruction. The other device can be any device used for decoding or storage. Communication interface 22 can, for example, be used to encapsulate the encoded image data 21 into a suitable format, such as data packets, for transmission over link 13.

[0137] Destination device 14 includes decoder 30. Optionally, destination device 14 may also include communication interface 28, image post-processor 32, and display device 34. These are described below:

[0138] Communication interface 28 can be used to receive encoded image data 21 from source device 12 or any other source, such as a storage device, for example, an encoded image data storage device. Communication interface 28 can be used to transmit or receive encoded image data 21 via link 13 between source device 12 and destination device 14 or via any type of network, such as a direct wired or wireless connection, any type of network such as wired or wireless networks or any combination thereof, or any type of private and public network, or any combination thereof. Communication interface 28 can be used, for example, to decapsulate data packets transmitted by communication interface 22 to obtain encoded image data 21.

[0139] Both communication interface 28 and communication interface 22 can be configured as unidirectional or bidirectional communication interfaces, and can be used, for example, to send and receive messages to establish connections, acknowledge and exchange any other information related to the communication link and / or data transmission, such as encoded image data transmission.

[0140] Decoder 30 (or decoder 30) is used to receive encoded image data 21 and provide decoded image data 31 or decoded image 31 (hereinafter further based on...) Figure 2 (Description of structural details of decoder 30). In some embodiments, decoder 30 can be used to perform the various embodiments described below to implement the video decoding method described in this application on the decoding side.

[0141] Image post-processor 32 is used to perform post-processing on decoded image data 31 (also known as reconstructed image data) to obtain post-processed image data 33. The post-processing performed by image post-processor 32 may include: color format conversion (e.g., from YUV format to RGB format), color correction, retouching or resampling, or any other processing, and may also be used to transmit post-processed image data 33 to display device 34.

[0142] Display device 34 is used to receive post-processed image data 33 to display the image to, for example, a user or viewer. Display device 34 can be or may include any class of displays for presenting reconstructed images, such as integrated or external displays or monitors. For example, the display may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other class of display.

[0143] Although, Figure 1 In this illustration, source device 12 and destination device 14 are depicted as separate devices. However, device embodiments may also include the functionality of both source device 12 and destination device 14, or both. That is, the functionality of source device 12 or its corresponding functionality and the functionality of destination device 14 or its corresponding functionality may be included simultaneously. In such embodiments, the functionality of source device 12 or its corresponding functionality and the functionality of destination device 14 or its corresponding functionality may be implemented using the same hardware and / or software, or using separate hardware and / or software, or any combination thereof.

[0144] Based on the description, it is obvious to those skilled in the art that the functionality of different units is different. Figure 1 The presence and (precise) division of the functionality of the source device 12 and / or destination device 14 shown may vary depending on the actual device and application. The source device 12 and destination device 14 may include any of a variety of devices, encompassing any category of handheld or stationary devices, such as laptops or notebooks, mobile phones, smartphones, tablets or tablet computers, cameras, desktop computers, set-top boxes, televisions, cameras, in-vehicle devices, display devices, digital media players, video game consoles, video streaming devices (e.g., content service servers or content distribution servers), broadcast receiver devices, broadcast transmitter devices, etc., and may or may not use any type of operating system.

[0145] Both encoder 20 and decoder 30 can be implemented as any of a variety of suitable circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, or any combination thereof. If the technology is implemented in part in software, the device can store the software instructions in a suitable non-transitory computer-readable storage medium, and one or more processors can be used to execute the instructions in hardware to perform the technology of this application. Any of the foregoing (including hardware, software, combinations of hardware and software, etc.) can be considered as one or more processors.

[0146] In some cases, Figure 1 The video encoding and decoding system 10 shown is merely an example, and the technology of this application can be applied to video encoding setups (e.g., video encoding or video decoding) that do not necessarily involve any data communication between the encoding and decoding devices. In other instances, data may be retrieved from local storage, streamed over a network, etc. A video encoding device may encode data and store it in storage, and / or a video decoding device may retrieve data from storage and decode it. In some instances, encoding and decoding are performed by devices that do not communicate with each other but only encode data to storage and / or retrieve data from storage and decode the data.

[0147] See Figure 2 , Figure 2 This is a schematic diagram of the structure of a video decoding device 400 (e.g., a video encoding device 400 or a video decoding device 400) provided in an embodiment of this application. The video decoding device 400 is suitable for implementing the embodiments described herein. In one embodiment, the video decoding device 400 may be a video decoder or a video encoder.

[0148] The video decoding device 400 includes: an input port 410 and a receiving unit (Rx) 420 for receiving data; a processor, logic unit, or central processing unit (CPU) 430 for processing data; a transmitter unit (Tx) 440 and an output port 450 for transmitting data; and a memory 460 for storing data. The video decoding device 400 may also include photoelectric conversion components and electro-optical (EO) components coupled to the input port 410, receiver unit 420, transmitter unit 440, and output port 450 for the input or output of optical or electrical signals.

[0149] Processor 430 is implemented in both hardware and software. Processor 430 can be implemented as one or more CPU chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. Processor 430 communicates with ingress port 410, receiver unit 420, transmitter unit 440, egress port 450, and memory 460. Processor 430 includes a decoding module 470 (e.g., encoding module 470 or decoding module 470). Encoding / decoding module 470 implements the embodiments disclosed herein to implement the video encoding or video decoding methods provided in the embodiments of this application. For example, encoding / decoding module 470 implements, processes, or provides various encoding operations. Therefore, encoding / decoding module 470 provides a substantial improvement to the functionality of video decoding device 400 and affects the transitions of video decoding device 400 to different states. Alternatively, encoding / decoding module 470 can be implemented with instructions stored in memory 460 and executed by processor 430.

[0150] Memory 460 includes one or more disks, tape drives, and solid-state drives, which can be used as overflow data storage devices to store programs while they are selectively executed, and to store instructions and data read during program execution. Memory 460 can be volatile and / or non-volatile, and can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random access memory (SRAM).

[0151] See Figure 3 , Figure 3 It is available as an exemplary embodiment. Figure 1 A simplified block diagram of apparatus 500 comprising one or both of the source device 12 and the destination device 14. Apparatus 500 can implement the video encoding method or the video decoding method of the embodiments of this application. In other words, Figure 3 This is a schematic block diagram illustrating one implementation of an encoding or decoding device (referred to as decoding device 500) according to an embodiment of this application. The decoding device 500 may include a processor 510, a memory 530, and a bus system 550. The processor and memory are connected via the bus system. The memory stores instructions, and the processor executes the instructions stored in the memory. The decoding device's memory stores program code, and the processor can invoke the program code stored in the memory to execute the various video encoding or video decoding methods described in this application. To avoid repetition, further details are omitted here.

[0152] In this embodiment, the processor 510 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0153] The memory 530 may include a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may also be used as memory 530. Memory 530 may include code and data 531 accessed by processor 510 using bus 550. Memory 530 may further include an operating system 533 and an application program 535, which includes at least one program that allows processor 510 to execute the video encoding or decoding methods described in this application. For example, application program 535 may include applications 1 to N, which further include a video encoding or decoding application (hereinafter referred to as a video decoding application) that executes the video encoding or decoding methods described in this application.

[0154] In addition to the data bus, the bus system 550 may also include a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 550 in the diagram.

[0155] Optionally, the decoding device 500 may also include one or more output devices, such as a display 570. In one example, the display 570 may be a haptic display that combines a display with a haptic unit capable of operatively sensing touch input. The display 570 may be connected to the processor 510 via a bus 550.

[0156] Figure 4 This is a schematic block diagram of a video communication system according to an embodiment of this application.

[0157] Figure 4 The video communication system 500 shown includes a source device 600 and a destination device 700. The source device 600 is capable of encoding the acquired video and transmitting the encoded video stream to the receiving device 700. The destination device 700 is capable of parsing the received video stream to obtain a video image and displaying the video through a display device.

[0158] like Figure 4As shown, the video encoding method of this application embodiment can be executed by the source device 600, and the video decoding method of this application embodiment can be executed by the destination device 700. Specifically, the video encoding method of this application embodiment can be executed by the video encoder 603, and the video decoding method of this application embodiment can be executed by the video decoder 702.

[0159] The aforementioned video communication system 500 can also be called a video encoding and decoding system, the source device 600 can also be called a video encoding device or video encoding equipment, and the destination device 700 can also be called a video decoding device or video decoding equipment.

[0160] exist Figure 4 In this embodiment, source device 600 includes a video capture device 601, a video memory 602, a video encoder 603, and a transmitter 604. The video memory 602 can store the video acquired by the video capture device 601, and the video encoder 603 can encode the video data from the video memory 602 and the video capture device 601. In some instances, source device 600 directly transmits the encoded video data to destination device 700 via transmitter 604. The encoded video data can also be stored on storage media or a file server for later retrieval by destination device 700 for decoding and / or playback.

[0161] exist Figure 4 In this system, destination device 700 includes a receiver 701, a video decoder 702, and a display device 703. In some instances, receiver 701 may receive encoded video data via channel 800. Display device 703 may be integrated with destination device 700 or may be external to destination device 700. Generally, display device 700 displays decoded video data. Display device 700 may include various display devices, such as liquid crystal displays, plasma displays, organic light-emitting diode displays, or other types of display devices.

[0162] The source device 600 and the destination device 700 can be implemented in any of the following forms: desktop computer, mobile computing device, notebook (e.g., laptop) computer, tablet computer, set-top box, smartphone, handheld device, television, camera, display device, digital media player, video game console, vehicle computer, or other similar device.

[0163] Destination device 700 can receive encoded video data from source device 600 via channel 800. Channel 800 may include one or more media and / or devices capable of moving encoded video data from source device 600 to destination device 700. In one example, channel 800 may include one or more communication media enabling source device 600 to transmit encoded video data directly to destination device 700 in real time. In this example, source device 600 may modulate the encoded video data according to a communication standard (e.g., a wireless communication protocol) and transmit the modulated video data to destination device 700. The one or more communication media may include wireless and / or wired communication media, such as radio frequency (RF) spectrum or one or more physical transmission lines. The one or more communication media may form part of a packet-based network (e.g., a local area network, a wide area network, or a global network (e.g., the Internet)). The one or more communication media may include routers, switches, base stations, or other devices that enable communication from source device 600 to destination device 700.

[0164] In another example, channel 800 may include a storage medium for storing encoded video data generated by source device 600. In this example, destination device 700 may access the storage medium via disk access or card access. The storage medium may include various locally accessible data storage media, such as Blu-ray discs, digital video discs (DVDs), compact disc read-only memory (CD-ROMs), flash memory, or other suitable digital storage media for storing encoded video data.

[0165] In another example, channel 800 may include a file server or another intermediate storage device that stores the encoded video data generated by source device 600. In this example, destination device 700 may access the encoded video data stored on the file server or other intermediate storage device via streaming or downloading. The file server may be a server type capable of storing and transmitting the encoded video data to destination device 700. For example, the file server may include a world wide area network (Web) server (e.g., for a website), a file transfer protocol (FTP) server, a network attached storage (NAS) device, and a local disk drive.

[0166] The destination device 700 can access the encoded video data via a standard data connection (e.g., an Internet connection). Examples of data connection types include wireless channels, wired connections (e.g., cable modems, etc.), or combinations thereof, suitable for accessing encoded video data stored on a file server. The transmission of encoded video data from the file server can be streaming, downloading, or a combination of both.

[0167] Currently, people have increasingly higher requirements for video quality, with video continuously evolving from high definition to ultra-high definition (4Kx2K, 8Kx4K). This continuous improvement in clarity leads to a constant increase in video compression bitrate, which in turn increases costs related to bandwidth, transmission latency, and storage. Therefore, reducing the bitrate of video compression has become a pressing technical problem that needs to be solved.

[0168] To address the aforementioned problems, this application proposes a video encoding method, a video decoding method, and an apparatus thereof, which can reduce the bit rate of video compression.

[0169] The video encoding method, video decoding method, and apparatus of the present application embodiments are described in detail below with reference to the accompanying drawings.

[0170] Figure 5 This is a schematic flowchart of a video encoding method according to an embodiment of this application. Figure 5 The video encoding method shown can be performed by a video encoding device (system). For example, Figure 5 The method shown can Figure 1 Encoder 20 in Figure 2 The video encoder 20 in Figure 3 The processor 510 or Figure 4 The video encoder 603 in the middle is used to perform the operation. Figure 5 The method 500 shown includes steps 510, 520, 530, 540 and 550, which are described in detail below.

[0171] S510, acquires the image semantic information of the encoded frame.

[0172] The image semantic information may include the image semantics of the encoded frame, instance segmentation of the encoded frame, semantic segmentation of the encoded frame, edge information of the encoded frame, and key point information in the encoded frame.

[0173] The method for obtaining the image semantic information of the encoded frame in this embodiment is not limited. For example, a preset neural network model (i.e., an image semantic extractor) can be used to extract the image semantic information of the encoded frame. The specific method can be referred to the prior art, and will not be described in detail here.

[0174] Optionally, the video to be encoded may be one of a plurality of video frames included in the video to be encoded, and the video to be encoded may also include at least one keyframe.

[0175] For example, the video to be encoded can be a group of pictures (GOP) in a video sequence. The video sequence can be understood as a complete video that needs to be encoded and decoded. The GOP can include an I-frame (e.g., the I-frame can be a keyframe) and multiple P-frames or B-frames.

[0176] Optionally, the method 500 may further include step 512.

[0177] S512, Obtain the video semantic information of the video to be encoded.

[0178] The video semantic information can represent the changes between multiple video frames included in the video to be encoded.

[0179] For example, the video semantic information may include the orientation change of the target object in the multiple video frames between the multiple video frames, the speed change of the target object in the multiple video frames between the multiple video frames, the background change between the multiple video frames, and the lighting change between the multiple video frames.

[0180] The method for obtaining the video semantic information in this application embodiment is not limited. For example, a preset neural network model (i.e., a video semantic extractor) can be used to extract the video semantic information. Specific methods can be found in existing technologies and will not be elaborated here.

[0181] Optionally, the method 500 may further include step 514.

[0182] S514, determine the video frame that satisfies at least one of the following as the key frame of the video to be encoded:

[0183] The video frame includes a preset image, the difference between the video frame and its previous frame is greater than or equal to a first preset value, the residual of the video frame is greater than or equal to a second preset value, and the time between the video frame and the previous keyframe is greater than or equal to a third preset value.

[0184] Wherein, if the video to be encoded is a group of movies (GOPs) in a video sequence, the keyframe can be an I-frame within that GOP.

[0185] Optionally, the preset frame can be determined according to the purpose of the video to be encoded, and the video frame containing the preset frame can be determined as the key frame of the video to be encoded.

[0186] The purpose of the video to be encoded can refer to the needs of the business (or task) performed using the video to be encoded.

[0187] For example, if the video to be encoded is used to detect the driver's driving behavior, then the preset screen may include the driver's line of sight, whether the driver is making a phone call, whether the driver is smoking, etc.; or, if the video to be encoded is used to detect vehicle violations, then the preset screen may include whether the vehicle runs a red light, whether the vehicle crosses double yellow lines, and whether the vehicle does not follow the prescribed route, etc.

[0188] Optionally, if the difference between the video frame and its previous frame is greater than or equal to the first preset value, the video frame can be determined as the key frame of the video to be encoded.

[0189] For example, if the difference between the video frame and its previous frame is greater than or equal to the first preset value, it can be considered that when the video frame is compared with the previous frame (or other frames before the video frame), there is a sudden change in the content of the screen, such as a sudden change in environment, a screen switch, a sudden appearance of an object, or a sudden disappearance of an object, the video frame is determined as the key frame of the video to be encoded.

[0190] Optionally, video frames with excessively large residuals can be identified as keyframes of the video to be encoded. For example, a second preset value can be pre-set, and video frames with residuals greater than or equal to the second preset value can be identified as keyframes of the video to be encoded.

[0191] It should be noted that the video frame and the video to be encoded both belong to the same video sequence, which can be understood as the complete video that needs to be encoded and decoded.

[0192] S520, Obtain the residual of the encoded frame.

[0193] Optionally, motion estimation and motion compensation can be performed based on the reference frame of the coded frame to predict the coded frame, that is, to generate a predicted image of the coded frame. The residual of the coded frame can be obtained by subtracting the original image of the coded frame (or the real image of the coded frame) from the predicted image of the coded frame.

[0194] It can be seen that the residual of the encoded frame obtained in S520 is actually the original residual of the encoded frame.

[0195] The reference frame of the encoded frame may refer to other video frames that need to be referenced when generating the predicted image of the encoded frame.

[0196] For example, when the encoded frame is a P-frame, the reference frame is the video frame preceding the P-frame, which can be an I-frame or a P-frame; when the encoded frame is a B-frame, the reference frame is the video frames in both directions before and after the B-frame, and the video frames preceding the B-frame can be I-frames or B-frames.

[0197] The specific methods for generating the predicted image and calculating the residual of the coded frame described above can be referred to in the prior art, and will not be repeated here.

[0198] Optionally, obtaining the residual of the encoded frame may include:

[0199] Using a first neural network model, a predicted image of the encoded frame is generated based on the reference frame of the encoded frame and the video semantic information; the residual of the encoded frame is obtained based on the original image of the encoded frame and the predicted image of the encoded frame.

[0200] The first neural network model can be a long short-term memory (LSTM) model, or it can be other deep learning (DL) models. This application does not limit the specific model.

[0201] Optionally, the method 500 may further include step 522.

[0202] S522, Adjust the parameters of the first neural network model based on the residual of the encoded frame and the predicted image of the encoded frame.

[0203] It should be noted that in S522, by adjusting the parameters of the first neural network model, the predicted image of the encoded frame generated by the adjusted first neural network model (based on the reference frame of the encoded frame and the video semantic information) can be closer to (or more similar to) the original image of the encoded frame.

[0204] Further, obtaining the residual of the encoded frame based on the original image of the encoded frame and the predicted image of the encoded frame may include:

[0205] Using a second neural network model, the predicted images of multiple video frames included in the video to be encoded are post-processed based on the video semantic information to obtain candidate predicted images of the multiple video frames; the original image of the encoded frame is subtracted from the candidate predicted image of the encoded frame to obtain the residual of the encoded frame.

[0206] The post-processing can make the changes between the candidate prediction images of the multiple video frames smooth, or the post-processing can make the changes between the candidate prediction images of the multiple video frames approximate the changes between the original images of the multiple video frames.

[0207] The second neural network model can be a long short-term memory (LSTM) model, or it can be other deep learning (DL) models. This application does not limit the specific model.

[0208] Optionally, the method 500 may further include step 524.

[0209] S524, adjust the parameters of the second neural network model based on the original image of the encoded frame and the candidate predicted image of the encoded frame.

[0210] It should be noted that in S524, by adjusting the parameters of the second neural network model, the changes between the candidate prediction images of the multiple video frames obtained by the adjusted second neural network model (which performs the post-processing on the prediction images of the multiple video frames of the video to be encoded based on the video semantic information) can be made smoother.

[0211] Alternatively, by adjusting the parameters of the second neural network model, the changes between the candidate predicted images of the multiple video frames obtained by the adjusted second neural network model (which performs the post-processing on the predicted images of the multiple video frames of the video to be encoded based on the video semantic information) can be made closer to the changes between the original images of the multiple video frames.

[0212] S530, determine multiple regions in the encoded frame based on the image semantic information.

[0213] The plurality of regions may include regions of interest (ROIs) in the encoded frame. The ROIs may be determined based on the purpose of the video to be encoded.

[0214] For example, if the video to be encoded is used to detect the driver's driving behavior, then the region of interest may refer to the driver region or the vehicle region in the encoded frame (the driver region is located in the vehicle region); or, if the video to be encoded is used to detect vehicle violations, then the region of interest may refer to both the vehicle region and the road surface region in the encoded frame.

[0215] Optionally, the image semantic information may include the image semantics of the encoded frame, instance segmentation of the encoded frame, semantic segmentation of the encoded frame, edge information of the encoded frame, and key point information in the encoded frame.

[0216] As an example and not a limitation, S530 will be described below using the semantic segmentation map of the encoded frame (i.e., indicating the semantic segmentation in the encoded frame) as an example of the image semantic information.

[0217] Typically, by dividing the pixels in an image into multiple regions according to certain rules (e.g., lighting or category), a semantic segmentation map of the image can be obtained.

[0218] Optionally, the encoded frame can be divided into multiple regions at the pixel level according to category. For example, different colors can be used to distinguish different regions in the encoded frame.

[0219] For example, the area where vehicles are located in the encoded frame can be marked as gray, the area where drivers are located in the encoded frame as red, the area where buildings are located in the encoded frame as blue, the area where the sky is located in the encoded frame as green, and the area where plants are located in the encoded frame as yellow, according to categories.

[0220] At this point, the resulting image, which uses different colors to mark different categories in the encoded frame, can be called a semantic segmentation map.

[0221] Furthermore, if the video to be encoded is used to detect the driver's driving behavior, the red area in the encoded frame (e.g., the driver area in the encoded frame) can be determined as the first area, and the area in the encoded frame other than the red area can be determined as the second area.

[0222] If the video to be encoded is used to detect vehicle violations, then the gray area in the encoded frame (e.g., the vehicle area in the encoded frame) can be determined as the first area, and the area in the encoded frame other than the gray area can be determined as the second area.

[0223] S540, simplify the residuals corresponding to multiple regions in the encoded frame to obtain candidate residuals of the encoded frame.

[0224] Wherein, the bitrate of candidate residual coding for at least one of the plurality of regions is less than the bitrate of residual coding for the at least one region.

[0225] The simplification process helps reduce the code rate after residual block encoding.

[0226] For example, regions other than the ROI can be simplified in the multiple regions so that the bitrate of the candidate residuals of the other regions is less than the bitrate of the residuals of the other regions. In this case, the ROI can be left unsimplified, that is, the candidate residuals of the ROI are actually the same as the residuals of the ROI.

[0227] Optionally, the simplification process can be quantization.

[0228] A pixel value in an image can be a red-green-blue (RGB) color value, which can be a long integer representing the color. For example, a pixel value of 256*Red+100*Green+76Blue, where * indicates multiplication, Blue represents the blue component, Green represents the green component, and Red represents the red component. Within each color component, the smaller the value, the lower the brightness; the larger the value, the higher the brightness. For grayscale images, the pixel value can be a grayscale value.

[0229] Quantization can reduce the candidate set of pixel values ​​in an image.

[0230] For example, an image may contain a*b pixels, each containing N distinct pixel values, where (a*b) > N. After quantization, these a*b pixels will contain M distinct pixel values, where N > M, where a, b, N, and M are all positive integers. In this case, compared to the image before quantization, the bitrate of the encoded image after quantization will be reduced.

[0231] It can be seen that quantization can make the values ​​of pixels in an image more concentrated, that is, reduce the candidate set of pixel values ​​in the image.

[0232] Optionally, the simplification process for the residuals corresponding to multiple regions in the encoded frame may include: simplifying the residuals corresponding to the multiple regions using a variety of different simplification strategies.

[0233] For example, different quantization strategies can be adopted to quantify the residuals corresponding to the multiple regions, based on the different importance of each region, so that the residuals corresponding to the multiple regions can be simplified to different degrees.

[0234] The importance here can be determined based on the intended use of the video to be encoded.

[0235] It should be noted that the above embodiments are merely examples and not limitations, and the simplification process described in this application is not limited to quantitative processing.

[0236] S550, the semantic information of the image and the candidate residuals of the encoded frame are encoded.

[0237] Optionally, encoding the image semantic information, the residual of the first region, and the candidate residual of the second region may include:

[0238] The reference frame, the image semantic information of the reference frame, the image semantic information, and the candidate residual of the coded frame are encoded.

[0239] Figure 6 This is a schematic flowchart of a video decoding method according to an embodiment of this application. Figure 6 The video encoding method shown can be performed by a video encoding device (system). For example, Figure 6 The method shown can Figure 1 Decoder 30 in Figure 2 Video decoder 30 in Figure 3 The processor 510 or Figure 4 The video decoder 702 in the middle is used to perform the operation. Figure 6 The method 600 shown includes steps 610 and 620, which are described in detail below.

[0240] S610, parse the bitstream to obtain the image semantic information of the encoded frame and the candidate residual of the encoded frame.

[0241] The image semantic information may include the image semantics of the encoded frame, instance segmentation of the encoded frame, semantic segmentation of the encoded frame, edge information of the encoded frame, and key point information in the encoded frame.

[0242] For a detailed description of the image semantic information, please refer to the embodiments in the above method 500, which will not be repeated here.

[0243] It should be noted that the specific methods for parsing the bitstream in S610 can refer to existing technologies, and will not be elaborated here.

[0244] Optionally, the video to be encoded may be one of a plurality of video frames included in the video to be encoded, and the video to be encoded may also include at least one keyframe.

[0245] For example, the video to be encoded can be a group of pictures (GOP) in a video sequence. The video sequence can be understood as a complete video that needs to be encoded and decoded. The GOP can include an I-frame (e.g., the I-frame can be a keyframe) and multiple P-frames or B-frames.

[0246] The encoded frame may include multiple regions, and the multiple regions may include the region of interest (ROI) in the encoded frame.

[0247] Optionally, the region of interest may be determined based on the intended use of the video to be encoded. A detailed description of the region of interest can be found in the embodiments of method 500 described above, and will not be repeated here.

[0248] The candidate residual can be obtained by simplifying the residuals corresponding to multiple regions in the encoded frame, wherein the bitrate of the candidate residual corresponding to at least one of the multiple regions is less than the bitrate of the residual corresponding to the at least one region.

[0249] For example, the candidate residual can be obtained by simplifying the residuals corresponding to multiple regions in the encoded frame using various simplification strategies.

[0250] Optionally, the simplification process can be a quantization process. A detailed description of the simplification process can be found in the embodiments of method 500 above, and will not be repeated here.

[0251] The method 600 may further include step 612.

[0252] S612, parse the bitstream to obtain the video semantic information of the video to be encoded, the video semantic information representing the changes between multiple video frames included in the video to be encoded.

[0253] For a detailed description of the video semantic information, please refer to the embodiments in the above method 500, which will not be repeated here.

[0254] S620, determine the target image of the encoded frame based on the image semantic information and the candidate residual.

[0255] Optionally, determining the target image of the encoded frame based on the image semantic information and the candidate residual may include:

[0256] Using a third neural network model, a predicted image of the encoded frame is generated based on the reference frame of the encoded frame and the video semantic information; based on the predicted image of the encoded frame and the candidate residual of the encoded frame, the target image of the encoded frame is obtained.

[0257] The third neural network model can be a long short-term memory (LSTM) model, or it can be other deep learning (DL) models. This application does not limit the specific model.

[0258] Optionally, the method 600 may further include step 622.

[0259] S622, Adjust the parameters of the third neural network model based on the target image of the encoded frame and the predicted image of the encoded frame.

[0260] It should be noted that in S622, by adjusting the parameters of the third neural network model, the target image of the encoded frame generated by the adjusted third neural network model (based on the image semantic information and the candidate residual) can be closer to (or more similar to) the original image of the encoded frame.

[0261] Optionally, obtaining the target image of the encoded frame based on the predicted image of the encoded frame and the candidate residual may include:

[0262] Using a fourth neural network model, the predicted images of multiple video frames included in the video to be encoded are post-processed based on the video semantic information to obtain candidate predicted images of the multiple video frames; the candidate predicted images of the encoded frames are added to the candidate residuals of the encoded frames to obtain the target image of the encoded frames.

[0263] The post-processing can make the changes between the candidate prediction images of the multiple video frames smooth, or the post-processing can make the changes between the candidate prediction images of the multiple video frames approximate the changes between the original images of the multiple video frames.

[0264] The fourth neural network model can be a long short-term memory (LSTM) model, or it can be other deep learning (DL) models. This application does not limit the specific model.

[0265] Optionally, the method 600 may further include step 624.

[0266] S624, adjust the parameters of the fourth neural network model based on the target image of the encoded frame and the candidate prediction image of the encoded frame.

[0267] It should be noted that in S624, by adjusting the parameters of the fourth neural network model, the changes between the candidate prediction images of the multiple video frames obtained by the adjusted fourth neural network model (which performs post-processing on the prediction images of the multiple video frames included in the video to be encoded based on the video semantic information) can be made smoother.

[0268] Alternatively, by adjusting the parameters of the fourth neural network model, the changes between the candidate predicted images of the multiple video frames obtained by the adjusted fourth neural network model (which performs post-processing on the predicted images of the multiple video frames included in the video to be encoded based on the video semantic information) can be made closer to the changes between the original images of the multiple video frames.

[0269] Figure 7 This is a schematic flowchart of a video encoding and decoding method according to an embodiment of this application.

[0270] Figure 7 The method 700 shown is jointly executed by a video encoding device and a video decoding device. The video encoding device executes steps 701 to 714, and the video decoding device executes steps 721 to 726. These steps are described in detail below.

[0271] The following steps 701 to 714 can be performed by a video encoding device:

[0272] S701 identifies the target frame in a video clip.

[0273] Optionally, target frames in video clips can be identified according to business needs. Target frames can refer to video frames including preset frames and video frames with sudden changes in screen content.

[0274] like Figure 8 As shown, target frames in video clips can be identified using techniques such as detection, tracking, segmentation, behavior recognition, and attribute recognition. Specific methods can be found in existing technologies and will not be elaborated upon here.

[0275] The business requirement mentioned above can refer to the purpose of the video clip, and the video clip mentioned here can refer to the aforementioned Figure 5 The video sequence that needs to be encoded and decoded in Method 500.

[0276] For example, if the video clip is used to detect the driver's driving behavior, the preset screen may include the driver's line of sight, whether the driver is making a phone call, whether the driver is smoking, etc.; or, if the video clip is used to detect vehicle violations, the preset screen may include whether the vehicle runs a red light, whether the vehicle crosses double yellow lines, and whether the vehicle does not follow the prescribed route, etc.

[0277] The sudden change in the image content can refer to the difference between the video frame and the previous frame being greater than or equal to a first preset value.

[0278] For example, if the difference between the video frame and its previous frame is greater than or equal to the first preset value, it can be considered that, compared with the previous frame (or other frames before the video frame), there is a sudden change in the content of the screen, such as a sudden change in environment, a screen switch, a sudden appearance of an object, or a sudden disappearance of an object.

[0279] S702, determine the I-frame (i.e. keyframe) in the video segment.

[0280] For example, such as Figure 8 As shown, the video clip can include video frames with preset scenes, video frames with sudden changes in scene content, and residuals. Frames with a time greater than the second preset value (residual analysis) and frames with a time interval greater than or equal to the third preset value (time interval analysis) are identified as I-frames.

[0281] For ease of description, an I-frame (or the value of an I-frame) can be denoted as x0.

[0282] S703 uses image encoding to encode the I-frame x0.

[0283] For example, S703 can be executed by the image encoding module, specifically represented by the following formula:

[0284] y0 = Encoder1(x0)

[0285] Here, Encoder1() represents the image encoding performed by the image decoding module, x0 represents an I-frame, and y0 represents the value after image encoding of x0.

[0286] S704, decodes the bitstream y0 obtained after image encoding to obtain the decoded I-frame.

[0287] For example, S704 can be executed by the image decoding module, specifically represented by the following formula:

[0288]

[0289] Here, Decoder1() represents the image decoding performed by the image encoding module, and y0 represents the value of x0 after image encoding. This represents the decoded I-frame obtained after decoding the image of y0.

[0290] S705 extracts semantic information from video segments.

[0291] For example, S705 can be executed by the semantic extraction module. For ease of description, the semantic information can be denoted as L. i .

[0292] Semantic information can include image semantic information and video semantic information.

[0293] like Figure 9 As shown, the image semantic information may include the image semantics of each video frame in the video segment, the instance segmentation of each video frame, the semantic segmentation of each video frame, the edge information of each video frame, and the key point information in each video frame.

[0294] Video semantic information can be used to represent changes between multiple video frames within the video segment. For example... Figure 9 As shown, the video semantic information may include motion semantics and behavioral semantics in the video clips.

[0295] For example, the video semantic information may include the orientation change of a target object in multiple video frames in a video segment between the multiple video frames, the speed change of the target object in the multiple video frames between the multiple video frames, the background change between the multiple video frames, and the lighting change between the multiple video frames, etc.

[0296] S706, regarding semantic information L i Encoding is performed to obtain semantic code F i .

[0297] For example, S706 can be executed by the semantic coding module, specifically represented by the following formula:

[0298] F i =Encoder2(L i )

[0299] Where Encoder2() indicates that the semantic encoding module performs encoding, L i Representing semantic information, F i Indicates L i The value after encoding.

[0300] S707, the semantically encoded value F i Decoding is performed to obtain the target semantic information.

[0301] For example, S707 can be executed by the semantic decoding module, specifically represented by the following formula:

[0302]

[0303] Where Decoder2() indicates that the semantic decoding module performs decoding, Fi Indicates L i The encoded value, Indicates F i The decoded target semantic information obtained after decoding.

[0304] S708, based on the reference frame, the semantic information of the reference frame, and the semantic information F of the current encoded frame. i Perform image prediction to obtain the predicted image of the current coded frame.

[0305] S708 can be executed by the B / P frame generation module, specifically by the first neural network model in the B / P frame generation module.

[0306] Predicted image generated by the B / P frame generation module It can be a B-frame or a P-frame.

[0307] For example, in predicting images In the case of B-frames, such as Figure 10 As shown, two adjacent I-frames x0 and x1 in a video segment, as well as the frames between these two I-frames, can be considered as a GOP.

[0308] x0, x1, x 0-1 ... x 0-k L i As input to the B / P frame generation module.

[0309] Motion estimation of x0 and x1 is performed using the first neural network model, and L is extracted. i The high-level semantic information in the image is used to generate each B-frame (i.e., x in the GOP) by overlaying the motion estimation results with the high-level semantic information. 0-1 ... x 0-k The characteristics of ) are used to generate multiple B-frames in a GOP.

[0310] For example, in predicting images In the case of P-frames, such as Figure 11 As shown, an LSTM model (i.e., the first neural network model) can be used to estimate the motion of x0 and generate multiple P-frames in a GOP.

[0311] For example, in predicting images In the case of P-frames, such as Figure 12 As shown, a behavior-based video generation model (i.e., the first neural network model) can be used to estimate the motion of x0 and generate multiple P frames in a GOP.

[0312] The specific methods for generating B-frames or P-frames by image prediction described above can be found in existing technologies and will not be repeated here.

[0313] S709, for and Post-processing yields the candidate prediction image frame.

[0314] For example, S709 can be executed by a post-processing module, specifically by a second neural network model within the post-processing module.

[0315] The post-processing can make the changes between the candidate prediction images of the multiple video frames smooth, or the post-processing can make the changes between the candidate prediction images of the multiple video frames approximate the changes between the original images of the multiple video frames.

[0316] For example, such as Figure 13 As shown, an LSTM model (i.e., a second neural network model) can be used to post-process each GOP in a video clip.

[0317] S710, Calculate the post-processed predicted image. Its corresponding original video frame x i residual r i .

[0318] For example, S710 can be executed by the residual module.

[0319] S711, based on semantic information F i Determine video frame x i The region of interest (ROI) in the data.

[0320] S712, based on video frame x i ROI determines semantic residuals

[0321] For example, S712 can be executed by the semantic residual module.

[0322] Wherein, the semantic residual It can include video frame x i The residuals of the ROI and the video frame x i Candidate residuals in regions outside the ROI.

[0323] The candidate residuals here can be the candidate residuals in method 500, that is, for video frame x i The residuals outside the ROI are obtained after simplification. The bitrate of the candidate residual after encoding can be lower than the bitrate of the residual after encoding the region; that is, the simplification process can reduce the bitrate of the residual block after encoding.

[0324] Optionally, the simplification process can be quantization. A detailed description of the candidate residuals and simplification process can be found in method 500, and will not be repeated here.

[0325] S713, through the AC module, converts the image-encoded y0 and the semantically encoded F... i and semantic residuals The encoded bitstream is then transmitted.

[0326] S714 adjusts the parameters of the B / P frame generation module and the post-processing module.

[0327] For example, it can be based on the original image and I-frame of a video clip. Adjust the parameters of the B / P frame generation module and the post-processing module.

[0328] The following steps 721 to 726 can be performed by the video decoding device:

[0329] First, the AC module can parse the bitstream to obtain the image-encoded y0 and the semantically encoded F. i and semantic residuals

[0330] S721 decodes the image-encoded value y0 into a target image approximating an I-frame.

[0331]

[0332] Here, Decoder1() represents the image decoding module performing image decoding, and y0 represents the value after image encoding of x0. This represents the target image of the I-frame obtained after image decoding of y0.

[0333] S722, the semantically encoded value F i Decoded into target semantic information

[0334]

[0335] Where Decoder2() indicates that the semantic decoding module performs decoding, F i Indicates L i The encoded value, Indicates F i The decoded target semantic information obtained after decoding.

[0336] S723, based on the reference frame, the semantic information of the reference frame, and the semantic information F of the current encoded frame. iPerform image prediction to obtain the target prediction image of the current coded frame.

[0337] For example, S723 can be executed by the B / P frame generation module, specifically by the third neural network model within the B / P frame generation module.

[0338] S724, for and Post-processing yields the candidate prediction image frame.

[0339] For example, S709 can be executed by the post-processing module, specifically by the fourth neural network model in the post-processing module.

[0340] S725, Decoding Semantic Residuals

[0341] S726, semantic residuals Superimposed onto the generated frame The reconstructed target video frame is obtained.

[0342]

[0343] For example, S726 can be executed by the video frame reconstruction module.

[0344] S727 adjusts the parameters of the B / P frame generation module and the post-processing module.

[0345] For example, it can be based on the target image Target semantic information and target video frames Adjust the parameters of the B / P frame generation module and the post-processing module.

[0346] It should be noted that the numbering of each step in the above method 700 does not limit the execution order of each step in method 700. That is to say, the execution order of each step in method 700 is not limited in the embodiments of this application.

[0347] Figure 14 This is a schematic block diagram of a video encoding apparatus 1400 provided in one embodiment of this application. It should be understood that... Figure 14 The video encoding apparatus 1400 shown is merely an example; the apparatus 1400 of this application embodiment may also include other modules or units. It should be understood that the apparatus 1400 is capable of performing… Figure 5 or Figure 7 To avoid repetition, the steps in the method will not be described in detail here.

[0348] In one possible implementation of this application embodiment, the video encoding apparatus 1400 may include:

[0349] The first acquisition unit 1410 is used to acquire the image semantic information of the encoded frame;

[0350] The second acquisition unit 1420 is used to acquire the residual of the encoded frame;

[0351] The first determining unit 1430 is configured to determine multiple regions in the encoded frame based on the image semantic information;

[0352] The simplification processing unit 1440 is used to simplify the residuals corresponding to multiple regions in the encoded frame to obtain candidate residuals of the encoded frame, wherein the bit rate of the candidate residuals corresponding to at least one of the multiple regions is less than the bit rate of the residuals corresponding to at least one region.

[0353] The encoding unit 1450 is used to encode the image semantic information and the candidate residuals of the encoded frame.

[0354] The first acquisition unit and the second acquisition unit may be the same unit or module, and this is not limited in this embodiment.

[0355] Optionally, the simplification processing unit 1440 is specifically used to: simplify the residuals corresponding to the multiple regions using a variety of different simplification strategies.

[0356] Optionally, the encoded frame is one of a plurality of video frames included in the video to be encoded, and the video to be encoded further includes at least one keyframe; wherein, the apparatus further includes a third acquisition unit 1460, configured to:

[0357] Obtain the video semantic information of the video to be encoded, wherein the video semantic information represents the changes between multiple video frames included in the video to be encoded.

[0358] The third acquisition unit and the first acquisition unit may be the same unit or module, or the third acquisition unit and the second acquisition unit may be the same unit or module. This application embodiment does not limit this.

[0359] Optionally, the second acquisition unit 1420 is specifically used to: use a first neural network model to generate a predicted image of the encoded frame based on the reference frame of the encoded frame and the video semantic information; and obtain the residual of the encoded frame based on the original image of the encoded frame and the predicted image of the encoded frame.

[0360] Optionally, the apparatus further includes an adjustment unit 1460, configured to: adjust the parameters of the first neural network model based on the residual of the encoded frame and the predicted image of the encoded frame.

[0361] Optionally, the second acquisition unit 1420 is specifically used to: use a second neural network model to post-process the predicted images of multiple video frames included in the video to be encoded based on the video semantic information to obtain candidate predicted images of the multiple video frames, wherein the post-processing enables smooth transitions between the candidate predicted images of the multiple video frames; and subtract the original image of the encoded frame from the candidate predicted image of the encoded frame to obtain the residual of the encoded frame.

[0362] Optionally, the apparatus further includes an adjustment unit 1460, configured to: adjust the parameters of the second neural network model based on the original image of the encoded frame and the candidate prediction image of the encoded frame.

[0363] Optionally, the apparatus further includes a second determining unit 1470, configured to: determine video frames that satisfy at least one of the following as keyframes of the video to be encoded:

[0364] The video frame includes a preset image, the difference between the video frame and its previous frame is greater than or equal to a first preset value, and the residual of the video frame is greater than or equal to a second preset value.

[0365] The first determining unit and the second determining unit may be the same unit or module, and this is not limited in this embodiment.

[0366] Optionally, the encoding unit 1450 is specifically used for:

[0367] The reference frame, the image semantic information of the reference frame, the image semantic information, and the candidate residual of the coded frame are encoded.

[0368] It should be understood that the video encoding device 1400 described herein is embodied in the form of a functional module. The term "module" here can be implemented in software and / or hardware, without specific limitation. For example, a "module" can be a software program, hardware circuitry, or a combination of both that implements the functions described above. The hardware circuitry may include application-specific integrated circuits (ASICs), electronic circuitry, a processor (e.g., a shared processor, a proprietary processor, or a group processor, etc.) and memory for executing one or more software or firmware programs, combined logic circuitry, and / or other suitable components supporting the described functions.

[0369] As an example, the video encoding apparatus 1400 provided in this application embodiment may be... Figure 1 Encoder 20 in Figure 2 The video encoder 20 in Figure 3 The processor 510 or Figure 4 The video encoder 603 is used to perform the methods described in the embodiments of this application.

[0370] Figure 15 This is a schematic block diagram of a video decoding apparatus 1500 provided in one embodiment of this application. It should be understood that... Figure 15 The video decoding apparatus 1500 shown is merely an example; the apparatus 1500 in this embodiment may also include other modules or units. It should be understood that the apparatus 1500 is capable of performing… Figure 6 or Figure 7 To avoid repetition, the steps in the method will not be described in detail here.

[0371] In one possible implementation of this application embodiment, the video decoding apparatus 1500 may include:

[0372] The decoding unit 1510 is used to parse the bitstream to obtain the image semantic information of the encoded frame and the candidate residual of the encoded frame. The candidate residual is obtained by simplifying the residuals corresponding to multiple regions in the encoded frame. The bit rate of the candidate residual encoded by at least one region is less than the bit rate of the residual encoded by at least one region.

[0373] The determining unit 1520 is used to determine the target image of the encoded frame based on the image semantic information and the candidate residual of the encoded frame.

[0374] Optionally, the candidate residual is obtained by simplifying the residuals corresponding to multiple regions in the encoded frame using various simplification strategies.

[0375] Optionally, the encoded frame is one of a plurality of video frames included in the video to be encoded, and the video to be encoded further includes at least one keyframe; wherein, the decoding unit 1510 is further configured to: parse the bitstream to obtain video semantic information of the video to be encoded, the video semantic information representing the changes between the plurality of video frames included in the video to be encoded.

[0376] Optionally, the determining unit 1520 is further configured to: use a third neural network model to generate a predicted image of the encoded frame based on a reference frame of the encoded frame and the video semantic information; and obtain a target image of the encoded frame based on the predicted image of the encoded frame and the candidate residual of the encoded frame.

[0377] Optionally, the device further includes an adjustment unit 1530, configured to: adjust the parameters of the third neural network model based on the target image of the encoded frame and the predicted image of the encoded frame.

[0378] Optionally, the determining unit 1520 is further configured to: use a fourth neural network model to post-process the predicted images of multiple video frames included in the video to be encoded based on the video semantic information to obtain candidate predicted images of the multiple video frames, wherein the post-processing enables smooth transitions between the candidate predicted images of the multiple video frames; and add the candidate predicted image of the encoded frame to the candidate residual of the encoded frame to obtain the target image of the encoded frame.

[0379] Optionally, the apparatus further includes an adjustment unit 1530, configured to: adjust the parameters of the fourth neural network model based on the target image of the encoded frame and the candidate prediction image of the encoded frame.

[0380] It should be understood that the video decoding device 1500 described herein is embodied in the form of a functional module. The term "module" here can be implemented in software and / or hardware, without specific limitation. For example, a "module" can be a software program, hardware circuit, or a combination of both that implements the above-described functions. The hardware circuit may include application-specific integrated circuits (ASICs), electronic circuits, a processor (e.g., a shared processor, a proprietary processor, or a group processor, etc.) and memory for executing one or more software or firmware programs, integrated logic circuitry, and / or other suitable components supporting the described functions.

[0381] As an example, the video decoding apparatus 1500 provided in this application embodiment may be... Figure 1 Decoder 30 in Figure 2 Video decoder 30 in Figure 3 The processor 510 or Figure 4 The video decoder 702 in the present application is used to perform the methods described in the embodiments of this application.

[0382] Figure 16 This is a schematic block diagram of a video encoding / decoding apparatus 900 according to an embodiment of this application. Figure 16 The illustrated device 900 includes a memory 901, a processor 902, a communication interface 903, and a bus 904. The memory 901, processor 902, and communication interface 903 are interconnected via the bus 904.

[0383] The memory 901 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 901 can store programs. When the program stored in the memory 901 is executed by the processor 902, the processor 902 performs various steps of the video encoding or video decoding method of the embodiments of this application. For example, it can execute... Figure 5 , Figure 6 or Figure 7 The various steps of the illustrated embodiment.

[0384] The processor 902 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, used to execute relevant programs to implement the video encoding and decoding method of the method embodiment of this application.

[0385] The processor 902 can also be an integrated circuit chip with signal processing capabilities. In implementation, the various steps of the video encoding or video decoding method of this application embodiment can be completed by the integrated logic circuitry in the hardware of the processor 902 or by software instructions.

[0386] The processor 902 described above can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0387] The steps of the method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 901. The processor 902 reads the information in memory 901 and, in conjunction with its hardware, completes the functions required by the units included in the video encoding or video decoding apparatus in the embodiments of this application, or executes the video encoding or video decoding method of the method embodiments of this application. For example, it can execute... Figure 5 , Figure 6 or Figure 7 The various steps / functions of the illustrated embodiment.

[0388] The communication interface 903 can use, but is not limited to, transceivers to enable communication between the device 900 and other devices or communication networks.

[0389] Bus 904 may include a pathway for transmitting information between various components of device 900 (e.g., memory 901, processor 902, communication interface 903).

[0390] It should be understood that the device 900 shown in the embodiments of this application can be Figure 1 Encoder 20 in Figure 2 The video encoder 20 in Figure 3 The processor 510 or Figure 4 The video encoder 603 is used to perform the video encoding method described in the embodiments of this application.

[0391] Alternatively, the device 900 shown in the embodiments of this application can also be Figure 1 Decoder 30 in Figure 2 Video decoder 30 in Figure 3 The processor 510 or Figure 4 The video decoder 702 is used to perform the video decoding method described in the embodiments of this application.

[0392] It should be understood that the processor in the embodiments of this application can be a central processing unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0393] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0394] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0395] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0396] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0397] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0398] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0399] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0400] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0401] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0402] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0403] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0404] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video encoding method, characterized in that, include: Obtain the image semantic information of the encoded frame; Obtain the residual of the encoded frame; Multiple regions in the encoded frame are determined based on the image semantic information; The residuals corresponding to multiple regions in the encoded frame are simplified to obtain candidate residuals of the encoded frame. The bitrate of the candidate residuals corresponding to at least one of the multiple regions is less than the bitrate of the residuals corresponding to at least one region. The semantic information of the image and the candidate residuals of the encoded frame are encoded.

2. The method according to claim 1, characterized in that, The simplification process for the residuals corresponding to multiple regions in the encoded frame includes: The residuals corresponding to the multiple regions are simplified using a variety of different simplification strategies.

3. The method according to claim 1, characterized in that, The encoded frame is one of a plurality of video frames included in the video to be encoded, and the video to be encoded also includes at least one keyframe; The step of obtaining the residual of the encoded frame includes: Using a first neural network model, a predicted image of the encoded frame is generated based on the reference frame of the encoded frame and video semantic information, wherein the video semantic information represents the changes between multiple video frames included in the video to be encoded; The residual of the encoded frame is obtained based on the original image of the encoded frame and the predicted image of the encoded frame.

4. The method according to claim 3, characterized in that, The method further includes: The parameters of the first neural network model are adjusted based on the residual of the encoded frame and the predicted image of the encoded frame.

5. The method according to claim 4, characterized in that, The process of obtaining the residual of the coded frame based on the original image and the predicted image of the coded frame includes: Using a second neural network model, the predicted images of multiple video frames included in the video to be encoded are post-processed based on the video semantic information to obtain candidate predicted images of the multiple video frames. The post-processing enables smooth transitions between the candidate predicted images of the multiple video frames. The residual of the encoded frame is obtained by subtracting the original image of the encoded frame from the candidate predicted image of the encoded frame.

6. The method according to claim 5, characterized in that, The method further includes: The parameters of the second neural network model are adjusted based on the original image of the encoded frame and the candidate predicted image of the encoded frame.

7. The method according to any one of claims 3 to 6, characterized in that, The method further includes: Video frames that satisfy at least one of the following conditions are identified as keyframes of the video to be encoded: The video frame includes a preset image, the difference between the video frame and its previous frame is greater than or equal to a first preset value, and the residual of the video frame is greater than or equal to a second preset value.

8. A method for video decoding, characterized in that, include: The image semantic information of the encoded frame and the candidate residual of the encoded frame are obtained by parsing the bitstream. The candidate residual is obtained by simplifying the residuals corresponding to multiple regions in the encoded frame. The bitrate of the candidate residual corresponding to at least one of the multiple regions is less than the bitrate of the residual corresponding to at least one region. The target image of the encoded frame is determined based on the image semantic information and the candidate residual of the encoded frame.

9. The method according to claim 8, characterized in that, The candidate residuals are obtained by simplifying the residuals corresponding to multiple regions in the encoded frame using various simplification strategies.

10. The method according to claim 8 or 9, characterized in that, The encoded frame is one of a plurality of video frames included in the video to be encoded, and the video to be encoded also includes at least one keyframe; The step of determining the target image of the encoded frame based on the image semantic information and the candidate residual includes: Using a third neural network model, a predicted image of the encoded frame is generated based on the reference frame and video semantic information of the encoded frame, wherein the video semantic information represents the changes between multiple video frames included in the video to be encoded; Based on the predicted image of the encoded frame and the candidate residual of the encoded frame, the target image of the encoded frame is obtained.

11. The method according to claim 10, characterized in that, The method further includes: The parameters of the third neural network model are adjusted based on the target image and the predicted image of the encoded frame.

12. The method according to claim 11, characterized in that, The step of obtaining the target image of the encoded frame based on the predicted image of the encoded frame and the candidate residual includes: Using a fourth neural network model, the predicted images of multiple video frames included in the video to be encoded are post-processed based on the video semantic information to obtain candidate predicted images of the multiple video frames. The post-processing enables smooth transitions between the candidate predicted images of the multiple video frames. The candidate predicted image of the encoded frame is added to the candidate residual of the encoded frame to obtain the target image of the encoded frame.

13. The method according to claim 12, characterized in that, The method further includes: The parameters of the fourth neural network model are adjusted based on the target image of the encoded frame and the candidate prediction image of the encoded frame.

14. A video encoding apparatus, characterized in that, include: The first acquisition unit is used to acquire the image semantic information of the encoded frame; The second acquisition unit is used to acquire the residual of the encoded frame; The first determining unit is configured to determine multiple regions in the encoded frame based on the image semantic information; A simplification processing unit is used to simplify the residuals corresponding to multiple regions in the encoded frame to obtain candidate residuals of the encoded frame, wherein the bitrate of the candidate residuals corresponding to at least one of the multiple regions is less than the bitrate of the residuals corresponding to at least one region. The encoding unit is used to encode the semantic information of the image and the candidate residuals of the encoded frame.

15. The apparatus according to claim 14, characterized in that, The simplification processing unit is specifically used for: The residuals corresponding to the multiple regions are simplified using a variety of different simplification strategies.

16. The apparatus according to claim 14, characterized in that, The encoded frame is one of a plurality of video frames included in the video to be encoded, and the video to be encoded also includes at least one keyframe; Specifically, the second acquisition unit is used for: Using a first neural network model, a predicted image of the encoded frame is generated based on the reference frame of the encoded frame and video semantic information, wherein the video semantic information represents the changes between multiple video frames included in the video to be encoded; The residual of the encoded frame is obtained based on the original image of the encoded frame and the predicted image of the encoded frame.

17. The apparatus according to claim 16, characterized in that, The device further includes an adjustment unit for: The parameters of the first neural network model are adjusted based on the residual of the encoded frame and the predicted image of the encoded frame.

18. The apparatus according to claim 17, characterized in that, The second acquisition unit is specifically used for: Using a second neural network model, the predicted images of multiple video frames included in the video to be encoded are post-processed based on the video semantic information to obtain candidate predicted images of the multiple video frames. The post-processing enables smooth transitions between the candidate predicted images of the multiple video frames. The residual of the encoded frame is obtained by subtracting the original image of the encoded frame from the candidate predicted image of the encoded frame.

19. The apparatus according to claim 18, characterized in that, The device further includes an adjustment unit for: The parameters of the second neural network model are adjusted based on the original image of the encoded frame and the candidate predicted image of the encoded frame.

20. The apparatus according to any one of claims 16 to 19, characterized in that, The device further includes a second determining unit, used for: Video frames that satisfy at least one of the following conditions are identified as keyframes of the video to be encoded: The video frame includes a preset image, the difference between the video frame and its previous frame is greater than or equal to a first preset value, and the residual of the video frame is greater than or equal to a second preset value.

21. A video decoding apparatus, characterized in that, include: The decoding unit is used to parse the bitstream to obtain the image semantic information of the encoded frame and the candidate residual of the encoded frame. The candidate residual is obtained by simplifying the residuals corresponding to multiple regions in the encoded frame. The bitrate of the candidate residual corresponding to at least one of the multiple regions is less than the bitrate of the residual corresponding to at least one region. The determining unit is used to determine the target image of the encoded frame based on the image semantic information and the candidate residual of the encoded frame.

22. The apparatus according to claim 21, characterized in that, The candidate residuals are obtained by simplifying the residuals corresponding to multiple regions in the encoded frame using various simplification strategies.

23. The apparatus according to claim 21 or 22, characterized in that, The encoded frame is one of a plurality of video frames included in the video to be encoded, and the video to be encoded also includes at least one keyframe; The determining unit is further configured to: Using a third neural network model, a predicted image of the encoded frame is generated based on the reference frame and video semantic information of the encoded frame, wherein the video semantic information represents the changes between multiple video frames included in the video to be encoded; Based on the predicted image of the encoded frame and the candidate residual of the encoded frame, the target image of the encoded frame is obtained.

24. The apparatus according to claim 23, characterized in that, The device further includes an adjustment unit for: The parameters of the third neural network model are adjusted based on the target image and the predicted image of the encoded frame.

25. The apparatus according to claim 24, characterized in that, The determining unit is further configured to: Using a fourth neural network model, the predicted images of multiple video frames included in the video to be encoded are post-processed based on the video semantic information to obtain candidate predicted images of the multiple video frames. The post-processing enables smooth transitions between the candidate predicted images of the multiple video frames. The candidate predicted image of the encoded frame is added to the candidate residual of the encoded frame to obtain the target image of the encoded frame.

26. The apparatus according to claim 25, characterized in that, The device further includes an adjustment unit for: The parameters of the fourth neural network model are adjusted based on the target image of the encoded frame and the candidate prediction image of the encoded frame.

27. A video encoding apparatus, characterized in that, It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to invoke the program instructions to perform the method of any one of claims 1 to 7.

28. A video decoding apparatus, characterized in that, It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to invoke the program instructions to perform the method of any one of claims 8 to 13.

29. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that, when executed by a processor, implement the method of any one of claims 1 to 7 or 8 to 13.

30. A chip, characterized in that, The chip includes a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute the method as described in any one of claims 1 to 7 or 8 to 13.

31. A computer program product, characterized in that, The computer program product includes instructions for performing the method as described in any one of claims 1 to 7 or 8 to 13.

Images