Video encoding and decoding processing method and apparatus, computer device and storage medium

By encoding, decoding and model optimization of video frame sequences, the problem of poor high-definition and ultra-high-definition video encoding and decoding effects in the existing technology is solved, and higher compression quality and lower code rate are achieved, and video transmission is improved. and storage efficiency.

WO2024230330A1PCT designated stage expired Publication Date: 2024-11-14TENCENT TECHNOLOGY (SHENZHEN) CO LTD

Patent Information

Application Number
PCT/CN2024/082916
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-10
Filing Date
2024-03-21
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

The existing video codec models have problems such as degradation in high-definition video and ultra-high-definition video codecs, resulting in poor encoding and decoding effects.

Method used

By extracting the video frame sequence from the sample video, using the pre-trained video codec model to code the keyframe and estimated frames, and optimizing the model based on the obtained coded frame and reconstructed frames, the target video codec model is obtained. To improve the encoding and decoding effect.

Benefits of technology

The compression quality and compression rate of the video encoding and decoding model for high-definition and ultra-high-definition videos is improved, the encoding and decoding effect is improved, and the storage and transmission cost of video data is reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024082916_14112024_PF_FP_ABST
    Figure CN2024082916_14112024_PF_FP_ABST
Patent Text Reader

Abstract

The present application can be applied in the field of artificial intelligence, and relates to a video encoding and decoding processing method and apparatus, a computer device and a storage medium. The method comprises: extracting a video frame sequence from a sample video, wherein the video frame sequence comprises a key frame and an estimated frame (S202); performing encoding and decoding processing on the key frame by means of a pre-trained key frame network of a video encoding and decoding model to obtain a first encoded frame and a corresponding first reconstructed frame (S204); performing encoding and decoding processing on the estimated frame by means of a pre-trained estimated frame network of the video encoding and decoding model to obtain a second encoded frame and a corresponding second reconstructed frame (S206); performing model optimization on the video encoding and decoding model on the basis of the first encoded frame, the first reconstructed frame, the second encoded frame and the second reconstructed frame to obtain a target video encoding and decoding model (S208); and when a target video is obtained, performing encoding and decoding processing on the target video by means of the target video encoding and decoding model (S210). By means of the method, the effect of video encoding and decoding can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Video encoding and decoding processing method, device, computer equipment and storage medium

[0001] Related applications

[0002] This application claims priority to Chinese patent application number 2023105192609, filed on May 10, 2023, entitled “Video encoding and decoding processing method, device, computer equipment and storage medium”, the entire text of which is hereby incorporated by reference. Technical Field

[0003] The present application relates to the field of computer technology, and in particular to a video encoding and decoding processing method, apparatus, computer equipment, and storage medium. Background Art

[0004] Video data typically has a large size. Directly transmitting raw video data would consume a significant amount of network bandwidth and storage space. Video codec technology compresses and decompresses video data, enabling efficient transmission and storage. With the continuous development of artificial intelligence (AI), deep learning video codec technology based on neural networks has been gradually applied to video transmission.

[0005] However, existing video coding and decoding models have problems such as video quality degradation and bit rate increase in the coding and decoding of high-definition and ultra-high-definition videos, resulting in poor coding and decoding effects of existing video coding and decoding models.

[0006] Summary of the Invention

[0007] According to various embodiments provided in the present application, a video encoding and decoding processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product are provided.

[0008] In a first aspect, the present application provides a video encoding and decoding processing method, which is performed by a computer device, and the method includes:

[0009] Extracting a video frame sequence from a sample video, wherein the video frame sequence includes a key frame and an estimated frame;

[0010] Performing encoding and decoding processing on the key frame through a pre-trained key frame network of a video encoding and decoding model to obtain a first encoded frame and a corresponding first reconstructed frame;

[0011] Performing encoding and decoding processing on the estimated frame through a pre-trained estimated frame network of the video encoding and decoding model to obtain a second encoded frame and a corresponding second reconstructed frame;

[0012] Optimizing the video encoding and decoding model based on the first encoded frame, the first reconstructed frame, the second encoded frame, and the second reconstructed frame to obtain a target video encoding and decoding model;

[0013] When the target video is obtained, encoding and decoding processing is performed on the target video through the target video encoding and decoding model.

[0014] In a second aspect, the present application further provides a video encoding and decoding processing device. The device includes:

[0015] A video frame extraction module, configured to extract a video frame sequence from a sample video, wherein the video frame sequence includes a key frame and an estimated frame;

[0016] A key frame encoding module, configured to perform encoding and decoding processing on the key frame using a pre-trained key frame network of a video encoding and decoding model to obtain a first encoded frame and a corresponding first reconstructed frame;

[0017] an estimated frame encoding module, configured to perform encoding and decoding processing on the estimated frame through a pre-trained estimated frame network of the video encoding and decoding model to obtain a second encoded frame and a corresponding second reconstructed frame;

[0018] a model optimization module, configured to optimize the video codec model based on the first encoded frame, the first reconstructed frame, the second encoded frame, and the second reconstructed frame to obtain a target video codec model;

[0019] The model application module is used to perform encoding and decoding processing on the target video through the target video encoding and decoding model when the target video is obtained.

[0020] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the video encoding and decoding method when executing the computer-readable instructions.

[0021] In a fourth aspect, the present application further provides a computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by a processor, implement the steps of the video encoding and decoding processing method.

[0022] In a fifth aspect, the present application further provides a computer program product, comprising computer-readable instructions, which, when executed by a processor, implement the steps of the video encoding and decoding method.

[0023] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without any creative work.

[0025] FIG1 is a diagram illustrating an application environment of a video encoding and decoding method according to an embodiment;

[0026] FIG2a is a schematic flow chart of a method for encoding and decoding a video in one embodiment;

[0027] FIG2 b is a schematic flow chart of a video encoding and decoding method according to another embodiment;

[0028] FIG3 is a schematic diagram of a sample video in one embodiment;

[0029] FIG4 is a schematic diagram of a process of key frame network training steps in one embodiment;

[0030] FIG5 is a schematic diagram of a flow chart of the estimation frame network training steps in one embodiment;

[0031] FIG6 is a schematic flow chart of a model parameter optimization step in one embodiment;

[0032] FIG7 is a schematic flow chart of a loss value determination step in one embodiment;

[0033] FIG8 is a schematic flow chart of a model testing step in one embodiment;

[0034] FIG9 is a schematic diagram of reconstruction evaluation results in one embodiment;

[0035] FIG10 is a schematic diagram of a flow chart of target video encoding and decoding steps in one embodiment;

[0036] FIG11 is a schematic flow chart of a video encoding and decoding method according to another embodiment;

[0037] FIG12 is a schematic flow chart of a video encoding and decoding method according to another embodiment;

[0038] FIG13 is a flow chart of a method for training a video codec model according to one embodiment;

[0039] FIG14 is a flow chart of a method for training a video codec model according to another embodiment;

[0040] FIG15 is a structural block diagram of a video encoding and decoding processing apparatus according to an embodiment;

[0041] FIG16 is a structural block diagram of a video encoding and decoding processing apparatus according to another embodiment;

[0042] FIG17 is a diagram showing the internal structure of a computer device according to one embodiment;

[0043] FIG18 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0045] It should be noted that in the following description, the terms "first, second and third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first, second and third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0046] The video encoding and decoding processing method provided in the embodiments of the present application can be applied in the application environment shown in Figure 1. In this embodiment, terminal 102 communicates with server 104 via a network. A data storage system can store data that server 104 needs to process. The data storage system can be integrated with server 104, or placed in the cloud or on another server. The video encoding and decoding processing method can be executed by terminal 102 or server 104 alone, or by terminal 102 and server 104 in collaboration. In some embodiments, the video encoding and decoding processing method is executed by terminal 102, which extracts a video frame sequence from a sample video, and the video frame sequence includes key frames and estimated frames; the key frames are encoded and decoded by a pre-trained key frame network of a video encoding and decoding model to obtain a first encoded frame and a corresponding first reconstructed frame; the estimated frame is encoded and decoded by a pre-trained estimated frame network of a video encoding and decoding model to obtain a second encoded frame and a corresponding second reconstructed frame; based on the first encoded frame, the first reconstructed frame, the second encoded frame and the second reconstructed frame, the video encoding and decoding model is optimized to obtain a target video encoding and decoding model; when obtaining the target video, the target video is encoded and decoded by the target video encoding and decoding model.

[0047] Among them, the terminal 102 can be, but is not limited to, various desktop computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 102 and the server 104 can be directly or indirectly connected through wired or wireless communication, and this application does not impose any restrictions on this.

[0048] In one embodiment, as shown in FIG. 2a and FIG. 2b , a video encoding and decoding method is provided. The method is described by taking the computer device (terminal 102 or server 104) in FIG. 1 as an example, and includes the following steps:

[0049] S202: Extract a video frame sequence from the sample video, where the video frame sequence includes key frames and estimated frames.

[0050] Sample videos are video data used to train machine learning models. Sample videos typically consist of multiple video frames, each containing information about the video content, such as color, shape, and motion. Sample videos can come from a variety of sources, such as real-life recordings, simulated videos, and videos from the internet. Sample videos can also meet specific conditions, such as preset clarity requirements. This means that the clarity of each video frame in the sample video meets a certain standard or requirement.

[0051] In addition, the scenes in the sample videos of the embodiments of the present application can be continuous scenes. Continuous scenes refer to continuous and similar scene contents in the video, such as the shooting content of multiple cameras in the same room, natural scenery within a certain period of time, the content of a speaker's podium speech, etc. Continuous scenes can help analyze changes in scene content, identify scene transitions, extract scene information, etc., which are of great significance for video analysis and application.

[0052] It is understandable that a video frame sequence contains multiple continuous video frames. In the actual processing process, key frames and estimated frames can be determined according to needs. For example, the first video frame in the video frame sequence can be determined as a key frame, and other video frames after the first video frame in the video frame sequence can be determined as estimated frames. The video frame sequence can be called a GOP (Group of Pictures, i.e., a group of images). The key frame can also be called an I frame (Intra-coded frame), and the estimated frame can also be called a P frame (Predicted In an embodiment of the present application, encoding may be performed in an alternating manner of I frames and P frames. The I frame may also be called an intra-coded frame. The encoding result obtained by encoding it contains the complete picture information in the original video frame. It is self-contained, which means that it can be decoded independently of other frames and the image of the frame can be reconstructed without any external information. A GOP may contain an I frame and several P frames, wherein the first P frame is encoded relative to the I frame, and the encoding result only carries the difference information with the I frame. The subsequent P frames are encoded relative to the previous P frame, and the encoding result only carries the difference information with the previous P frame, that is, the encoding result obtained by encoding the P frame only carries the difference information with the previous frame. This encoding method can effectively reduce the bit rate of the video while ensuring the quality and smoothness of the video.

[0053] Specifically, after obtaining the sample video, the terminal extracts a video frame sequence from the sample video at a certain time interval, determines the first frame in the video frame sequence as a key frame, and determines other video frames except the first frame in the video frame sequence as estimated frames.

[0054] S204 , performing encoding and decoding processing on the key frame through a pre-trained key frame network of a video encoding and decoding model to obtain a first encoded frame and a corresponding first reconstructed frame.

[0055] Among them, the video codec model is a neural network model based on deep learning, which is used to compress, decompress, and reconstruct videos. In the deep learning codec model, models such as Convolutional Neural Network (CNN) or Recurrent Neural Network (RNN) are usually used.

[0056] The pre-trained keyframe network is a branch of the video codec model, used to encode and decode keyframes in a video frame sequence. In video codecs, keyframes are important frames in a video sequence because they can independently represent the video content without relying on other frames. Efficient encoding and decoding of keyframes can significantly improve the efficiency and quality of video compression. The pre-trained keyframe network is obtained by pre-training the keyframe network using deep learning technology.

[0057] Specifically, after obtaining the video frame sequence, the terminal inputs the key frames in the video frame sequence into the pre-trained key frame network of the video codec model, and encodes and decodes the key frames through the pre-trained key frame network to obtain the first encoded frame corresponding to the key frame, and the first reconstructed frame corresponding to the first encoded frame.

[0058] S206 , performing encoding and decoding processing on the estimated frame through a pre-trained estimated frame network of a video encoding and decoding model to obtain a second encoded frame and a corresponding second reconstructed frame.

[0059] The pre-trained estimated frame network is another branch of the video encoding and decoding model, which is used to encode and decode non-key frames in a video frame sequence. The pre-trained estimated frame network is obtained by pre-training the estimated frame network using deep learning technology.

[0060] Specifically, after obtaining the video frame sequence, the terminal inputs the estimated frame in the video frame sequence into the pre-trained estimated frame network of the video codec model, and encodes and decodes the estimated frame through the pre-trained estimated frame network to obtain a second encoded frame corresponding to the estimated frame, and a second reconstructed frame corresponding to the second encoded frame.

[0061] S208 , optimizing the video coding model based on the first coded frame, the first reconstructed frame, the second coded frame, and the second reconstructed frame to obtain a target video coding model.

[0062] Specifically, after obtaining the first encoded frame, the first reconstructed frame, the second encoded frame and the second reconstructed frame, the terminal optimizes the parameters of the video codec model based on the first encoded frame, the first reconstructed frame, the second encoded frame and the second reconstructed frame, and stops training when the convergence condition is reached to obtain the target video codec model.

[0063] Convergence means that the model training process has become stable, that is, the video codec model has learned the characteristics of the data and no longer has significant improvement. Convergence conditions include a fixed number of training rounds, a fixed threshold of the loss function, etc. When the model reaches this condition, training is stopped to avoid overfitting.

[0064] S210 , when the target video is obtained, encoding and decoding the target video is performed using the target video encoding and decoding model.

[0065] The target video refers to the video to be encoded and decoded, and the target video can be a video from different sources and different scenes.

[0066] Specifically, the terminal can be the sender or the receiver of the target video. In the scenario where the terminal is the sender of the target video, after obtaining the target video, the terminal encodes the target video through the target video codec model to obtain the encoded byte stream, and sends the encoded byte stream to the receiver; in the scenario where the terminal is the receiver of the target video, after receiving the encoded byte stream, the terminal reconstructs the encoded byte stream through the target video codec model to obtain the reconstructed target video.

[0067] In one embodiment, a terminal encodes a target video using a target video codec model to obtain an encoded byte stream, including the following steps: extracting each video frame sequence from the target video, encoding the key frames in each video frame sequence using an encoder of a pre-trained key frame network of the target video codec model to obtain a first encoded byte stream, encoding the estimated frames in multiple video frame sequences using an encoder of a pre-trained estimated frame network of the target video codec model to obtain a second encoded byte stream, and merging the first encoded byte stream and the second encoded byte stream into an encoded byte stream. The first encoded byte stream may also be referred to as a first processed encoded frame, and the second encoded byte stream may also be referred to as a second processed encoded frame.

[0068] In one embodiment, a terminal reconstructs a coded byte stream using a target video codec model to obtain a reconstructed target video, including the following steps: decoding a first coded byte stream in the coded byte stream using a decoder of a pre-trained key frame network of the target video codec model to obtain a reconstructed key frame; decoding a second coded byte stream in the coded byte stream using a decoder of a pre-trained estimated frame network of the target video codec model to obtain a reconstructed estimated frame; and generating a reconstructed target video based on the reconstructed key frame and the reconstructed estimated frame. The reconstructed key frame may also be referred to as a first processed reconstructed frame, and the reconstructed estimated frame may also be referred to as a second processed reconstructed frame.

[0069] In the above embodiment, after obtaining the video codec model including the pre-trained key frame network and the pre-trained estimated frame network, the terminal does not directly use the video codec model to process the video codec task, but extracts a video frame sequence from the sample video, where the video frame sequence includes key frames and estimated frames, so that different encoding and decoding methods are used for the key frames and the estimated frames, respectively, to improve the video compression rate while ensuring the video compression quality; for the key frames, the pre-trained key frame network of the video codec model is used to encode and decode them to obtain a first encoded frame and a corresponding first reconstructed frame; for the estimated frames, the pre-trained estimated frame network of the video codec model is used to encode and decode them to obtain a second The encoded frame and the corresponding second reconstructed frame, so that based on the first encoded frame, the first reconstructed frame, the second encoded frame and the second reconstructed frame, the pre-trained key frame network and the pre-trained estimated frame network of the video codec model can be jointly trained, that is, the parameters of the model are further optimized, so that the trained target video codec model has better encoding and decoding capabilities for videos that meet specific conditions. For example, when the sample video used is a video that meets specific clarity conditions (high definition and ultra-high definition), the target video codec model can improve the compression quality and compression rate of the video when processing the encoding and decoding task of the target video that meets the specific clarity conditions (high definition and ultra-high definition), that is, the encoding and decoding effect of the video is improved.

[0070] In one embodiment, the above-mentioned video encoding and decoding processing method also includes a process of obtaining a sample video, and the process of obtaining the sample video specifically includes the following steps: obtaining an original video that meets the clarity conditions; performing boundary detection on the original video to obtain the scene boundaries in the original video; based on the scene boundaries, extracting a video clip containing continuous scenes from the original video as a sample video.

[0071] Among them, clarity conditions refer to a set of rules or indicators used to ensure that the selected content meets certain visual quality standards when processing video or image data. The original video that meets the clarity conditions means that the clarity of the original video meets certain standards or requirements, for example, the original video is a high-definition video; boundary detection refers to the process of detecting and locating the boundaries between different scenes in the video, aiming to determine where the scene transition occurs in the video. It is usually used to detect and segment the boundary positions of continuous scenes. The scene boundary is the boundary position of continuous scenes in the video, that is, the location of the scene switch. During video playback, when the video screen shows obvious changes and jumps, this is the location of the scene boundary.

[0072] Specifically, the terminal obtains an authorized and reliable video website or video sharing platform, determines the original video that meets the clarity requirements on the video website or video sharing platform, obtains the video link of the original video, and uses a video download tool to download the original video that meets the clarity requirements from the video website or video sharing platform based on the obtained video link. After obtaining the original video, the terminal performs boundary detection on the original video based on a preset boundary detection algorithm to obtain scene boundaries in the original video. After obtaining the scene boundaries, the terminal determines the start time and end time of each continuous scene based on the scene boundaries, and extracts video segments containing the continuous scenes from the original video based on the start time and end time. Sub-videos of a target length are extracted from the video segments of each continuous scene, and each sub-video is used as a sample video. The target length is a preset length, such as 10 frames or 30 frames.

[0073] Among them, the download tool used can be Internet Download Manager, Free Download Manager, etc. Specifically, the obtained video link can be copied and pasted into the download tool, and the original video corresponding to the video link can be downloaded through the download tool; the boundary detection algorithm adopted can be inter-frame difference method, inter-frame similarity method, machine learning method, optical flow method, etc. The inter-frame difference method detects dynamic objects and scene changes in the video by comparing the pixel differences between adjacent frames, thereby determining the scene boundary; the inter-frame similarity method determines the change points and scene boundaries in the video by calculating the similarity and difference between adjacent frames; the machine learning method classifies and segments the video frames by using machine learning algorithms, such as neural networks, support vector machines, etc., thereby realizing scene boundary detection; the optical flow method detects object movement and scene changes in the video by calculating the pixel displacement and change between adjacent frames, thereby determining the scene boundary.

[0074] In one embodiment, the terminal uses a scene detection tool to detect scene boundaries in the original video and obtain the start and end times of each scene. Based on the scene boundary and scene time information, a video segment containing continuous scenes is extracted from the original video as a sample video. The scene detection tool can specifically be the Scene Detect tool, which is a Python-based video processing tool that is mainly used to detect and segment scene boundaries in videos. It can automatically identify scene switching points in the video, including switching special effects, scene changes, and screen dimming, and cut the video into continuous scene segments.

[0075] Figure 3 shows 9 consecutive frames of a video clip, where video frames 0 to 4 are pictures of horse racing scenes, and video frames 5 to 8 are pictures of sports scenes. The Scene detect tool can be used to detect that the scene boundary of the video clip is the time point where video frame 4 ends and video frame 5 begins. By cutting the video clip at this time point, sample video 1 and sample video 2 can be obtained, where sample video 1 is a continuous scene clip composed of video frames 0 to 4, and sample video 2 is a continuous scene clip composed of video frames 5 to 8.

[0076] In the above embodiment, the terminal obtains the original video that meets the clarity conditions, performs boundary detection on the original video, obtains the scene boundaries in the original video, and extracts video clips containing continuous scenes from the original video based on the scene boundaries as sample videos, thereby making the sample video higher in clarity, continuity and stability. When the sample video is used to train the video codec model, the training effect of the model can be improved.

[0077] In one embodiment, the process of extracting video segments containing continuous scenes from the original video as sample videos by the terminal based on scene boundaries specifically includes the following steps: extracting video segments containing continuous scenes from the original video based on scene boundaries; and performing artifact cleaning processing on the video segments to obtain sample videos.

[0078] Among them, artifact cleaning processing refers to the process of removing artifacts and noise in the video by adjusting video parameters such as color, contrast, and sharpness to improve the quality and clarity of the video.

[0079] Specifically, after obtaining the video clips, the terminal extracts sub-videos of target length from the video clips of each continuous scene, and uses a preset artifact cleaning algorithm to clean artifacts on each video frame in the sub-video to obtain a sample video.

[0080] The preset artifact cleaning algorithm can be Artifacts cleaning. Artifacts cleaning is a video processing technology that aims to remove artifacts, noise, distortion and other factors that affect video quality in the video, and improve the clarity and quality of the video. Artifacts usually refer to any distortion or abnormality of non-original scenes in images or videos caused by data compression, transmission errors, algorithm defects in the processing process, etc. Artifacts may appear as block noise, blur, banding, mosaics, etc., which reduce the visual quality of the video.

[0081] In the above embodiment, the terminal extracts video segments containing continuous scenes from the original video based on the scene boundaries; the video segments are cleaned of artifacts to obtain sample videos with clarity and quality assurance, avoiding the use of low-quality video samples to train the model, thereby improving the accuracy and robustness of the video encoding and decoding model, and further improving the video encoding and decoding efficiency and visual quality.

[0082] In one embodiment, the pre-trained key frame network of the video codec model is obtained by training the initial key frame network. Before training the video codec model, the terminal may also pre-train the key frame network of the video codec model separately to obtain the pre-trained key frame network of the video codec model. That is, before the terminal encodes and decodes the key frames through the pre-trained key frame network of the video codec model, the terminal pre-trains the key frame network of the video codec model separately to obtain the pre-trained key frame network of the video codec model. Referring to FIG4 , the process of training the initial key frame network specifically includes the following steps:

[0083] S402 : performing encoding and decoding processing on the video frames in the first initial video frame sequence through an initial key frame network to obtain a third encoded frame and a corresponding third reconstructed frame.

[0084] The first initial video frame sequence is extracted from a first initial sample video. The first initial sample video may be the same as the above sample video or may be different from the above sample video.

[0085] Specifically, the terminal can extract a first initial video frame sequence from the first initial sample video, and input each video frame in the first initial video frame sequence into the initial key frame network in sequence, encode the input video frame through the encoder of the initial key frame network to obtain a third encoded frame, and reconstruct the third encoded frame through the decoder of the initial key frame network to obtain a third reconstructed frame corresponding to the input video frame.

[0086] S404 , optimizing parameters of the initial key frame network based on the third encoded frame and the third reconstructed frame to obtain a pre-trained key frame network.

[0087] In one embodiment, S404 specifically includes the following steps: determining a first pre-training loss value based on the third encoded frame and the third reconstructed frame; optimizing the parameters of the initial key frame network based on the first pre-training loss value to obtain a pre-trained key frame network.

[0088] Among them, the first pre-training loss value is an indicator for measuring the compression bit rate of the video frame and the difference between the third reconstructed frame and the original video frame. This indicator can be used to evaluate the compression effect of the initial key frame network on the video frame, and then adjust the parameters of the initial key frame network to improve the compression effect of the initial key frame network on the video frame.

[0089] Specifically, after obtaining the third encoded frame and the third reconstructed frame, the terminal can also determine the byte stream size of the third reconstructed frame, and determine the first video frame compression loss value based on the byte stream size of the third encoded frame, and determine the first video frame reconstruction loss value based on the third reconstructed frame and the original video frame, and then determine the first pre-trained loss value based on the first video frame compression loss value and the first video frame reconstruction loss value, and use the back propagation algorithm to adjust the network parameters of the initial key frame network based on the first pre-trained loss value to obtain the adjusted initial key frame network, and re-execute step S402 until the training stops when the training meets the convergence condition to obtain the pre-trained key frame network.

[0090] Among them, the compression loss value of the first video frame reflects the degree of information loss of the third encoded frame during the video compression process, which can be specifically determined based on at least one of the byte stream sizes of the original frame and the compressed frame. The reconstruction loss value of the first video frame reflects the degree of difference between the reconstructed third encoded frame and the original third encoded frame, which can be specifically determined by evaluating the pixel-level difference between the reconstructed frame and the original frame.

[0091] In one embodiment, after obtaining the first video frame compression loss value and the first video frame reconstruction loss value, the terminal inputs the first video frame compression loss value and the first video frame reconstruction loss value into the loss function represented by the following formula to determine the first pre-training loss value by the following formula: Loss I =mse_loss I +bpp_loss I

[0092] Among them, Loss I is the first pre-training loss value corresponding to the video frame, mse_loss I is the first video frame reconstruction loss value corresponding to the video frame, bpp_loss1 is the first video frame compression loss value corresponding to the video frame, and the first video frame compression loss value may specifically be the byte stream size of the third coded frame.

[0093] In the above embodiment, the terminal encodes and decodes the video frames in the first initial video frame sequence through the initial key frame network to obtain a third encoded frame and a corresponding third reconstructed frame. The parameters of the initial key frame network are optimized based on the third encoded frame and the third reconstructed frame, so that the parameters of the network can more accurately capture the key information in the video, and a pre-trained key frame network is obtained. The pre-trained key frame network can be used as a basic model for subsequent tasks, accelerate the training process of subsequent tasks and improve model performance.

[0094] In one embodiment, the pre-trained estimated frame network of the video codec model is obtained by training the initial estimated frame network; before training the video codec model, the terminal may also pre-train the estimated frame network of the video codec model separately to obtain the pre-trained estimated frame network of the video codec model. That is, before the terminal performs encoding and decoding processing on the estimated frame through the pre-trained estimated frame network of the video codec model, the terminal pre-trains the estimated frame network of the video codec model separately to obtain the pre-trained estimated frame network of the video codec model. Referring to FIG5, the process of training the initial estimated frame network specifically includes the following steps:

[0095] S502 : Perform encoding and decoding processing on the second initial video frame sequence through an initial estimation frame network to obtain a fourth encoded frame and a corresponding fourth reconstructed frame.

[0096] The second initial video frame sequence is extracted from the second initial sample video. The second initial sample video can be the same as the above-mentioned sample video or a different video from the above-mentioned sample video. The second initial sample video can be the same as or different from the first initial sample video.

[0097] Specifically, the terminal can extract a second initial video frame sequence from the second initial sample video, and input each video frame in the second initial video frame sequence into the initial estimation frame network in turn, encode the input video frame through the encoder of the initial estimation frame network to obtain a fourth encoded frame, and reconstruct the fourth encoded frame through the decoder of the initial estimation frame network to obtain a fourth reconstructed frame corresponding to the input video frame.

[0098] S504: Optimize the initial estimated frame network of the video encoding and decoding model based on the fourth encoded frame and the fourth reconstructed frame to obtain a pre-trained estimated frame network.

[0099] In one embodiment, S504 specifically includes the following steps: determining a second pre-training loss value based on the fourth encoded frame and the fourth reconstructed frame; optimizing the parameters of the initial estimated frame network based on the second pre-training loss value to obtain a pre-trained estimated frame network.

[0100] Among them, the second pre-training loss value is an indicator for measuring the compression bit rate of the video frame and the difference between the fourth reconstructed frame and the original video frame. This indicator can be used to evaluate the compression effect of the initial estimated frame network on the video frame, and then adjust the parameters of the initial estimated frame network to improve the compression effect of the initial estimated frame network on the video frame.

[0101] Specifically, after obtaining the fourth encoded frame and the fourth reconstructed frame, the terminal can also determine the byte stream size of the fourth reconstructed frame, and determine the second video frame compression loss value based on the byte stream size of the fourth encoded frame, and determine the second video frame reconstruction loss value based on the fourth reconstructed frame and the original video frame, and then determine the second pre-trained loss value based on the second video frame compression loss value and the second video frame reconstruction loss value, and use the back propagation algorithm to adjust the network parameters of the initial estimated frame network based on the second pre-trained loss value to obtain the adjusted initial estimated frame network, and re-execute step S502 until the training stops when the training meets the convergence condition to obtain the pre-trained estimated frame network.

[0102] Among them, the second video frame compression loss value reflects the degree of information loss of the fourth coded frame during the video compression process, which can be specifically determined based on at least one of the byte stream sizes of the original frame and the compressed frame. The second video frame reconstruction loss value reflects the degree of difference between the reconstructed fourth coded frame and the original fourth coded frame, which can be specifically determined by evaluating the pixel-level difference between the reconstructed frame and the original frame.

[0103] In one embodiment, after obtaining the second video frame compression loss value and the second video frame reconstruction loss value, the terminal inputs the second video frame compression loss value and the second video frame reconstruction loss value into the loss function represented by the following formula, and determines the second pre-training loss value by the following formula: Loss P =mse_loss P +bpp_loss P

[0104] Among them, Loss P is the second pre-training loss value corresponding to the video frame, mse_loss P bpp_loss is the reconstruction loss value of the second video frame corresponding to the video frame P is a second video frame compression loss value corresponding to the video frame, and the second video frame compression loss value may specifically be a byte stream size of the third encoded frame.

[0105] In the above embodiment, the terminal encodes and decodes the second initial video frame sequence through the initial estimated frame network to obtain a fourth encoded frame and a corresponding fourth reconstructed frame, and optimizes the parameters of the initial estimated frame network based on the fourth encoded frame and the fourth reconstructed frame, so that the parameters of the network can more accurately capture the key information in the video, and obtain a pre-trained estimated frame network. The pre-trained estimated frame network can be used as the basic model for subsequent tasks, accelerate the training process of subsequent tasks and improve model performance.

[0106] In one embodiment, the pre-trained key frame network of the video codec model includes an encoder and a decoder; the terminal encodes and decodes the key frame through the pre-trained key frame network of the video codec model to obtain a first encoded frame and a corresponding first reconstructed frame. The process specifically includes the following steps: encoding the key frame through the encoder to obtain a first encoded frame; decoding the first encoded frame through the decoder to obtain a first reconstructed frame.

[0107] Among them, encoding is a process of compressing video signals into a smaller amount of data for easy storage, transmission and processing. The encoder processes and compresses the original video frames to generate a series of encoded data, which can be transmitted or stored so that it can be decoded into the original video when needed.

[0108] The decoder is used to decode the compressed code stream and restore it into image frames. The image quality of the restored image frames may be different from that of the original frames. This is because part of the information of the original frames is compressed during the compression encoding process, and the decoder needs to restore this part of the information through estimation and other technologies during the decoding process. Therefore, the image quality of the restored image frames is usually lower than that of the original frames. However, by adopting an appropriate compression ratio, the transmission and storage costs of the video can be reduced while ensuring the video quality.

[0109] Specifically, the terminal inputs the key frames in the video frame sequence into the encoder of the pre-trained key frame network, and the encoder of the pre-trained key frame network divides the key frames into blocks to obtain each key frame image block, and compresses each key frame image block to obtain a first coded frame, which is a compressed code stream; then the first coded frame is input into the decoder of the pre-trained key frame network, and the decoder decodes the first coded frame, that is, the compressed code stream, to obtain a decoding result, and restores each image block based on the decoding result to obtain each restored image block, and splices each restored image block to obtain a restored image corresponding to the key frame, which is the first reconstructed frame.

[0110] In the above embodiment, the terminal encodes the key frame through the encoder to obtain a first encoded frame, and decodes the first encoded frame through the decoder to obtain a first reconstructed frame, so that the model loss value can be determined based on the first encoded frame and the first reconstructed frame, and the loss value can be minimized by optimizing the model parameters, thereby improving the encoding and decoding effect of the video encoding and decoding model.

[0111] In one embodiment, the pre-trained estimated frame network of the video codec model includes an encoder and a decoder. The terminal encodes and decodes the estimated frame through the pre-trained estimated frame network of the video codec model to obtain a second encoded frame and a corresponding second reconstructed frame. The process specifically includes the following steps: encoding the estimated frame through the encoder to obtain a second encoded frame; decoding the second encoded frame through the decoder to obtain a second reconstructed frame.

[0112] Specifically, the terminal inputs the estimated frame to be processed in the video frame sequence and the reference frame corresponding to the estimated frame into the encoder of the pre-trained estimated frame network, performs motion estimation and motion compensation on the reference frame and the estimated frame through the encoder of the pre-trained estimated frame network to obtain a difference frame, and compresses and encodes the difference frame to obtain a second encoded frame, which is a compressed code stream; then the second encoded frame is input into the decoder of the pre-trained estimated frame network, and the decoder decodes the second encoded frame, that is, the compressed code stream, to obtain a decoding result, and restores the pixel information of the difference frame based on the decoding result, and performs motion compensation on the difference frame based on the restored image obtained by previously restoring the reference frame to obtain a restored image corresponding to the estimated frame, which is the second reconstructed frame.

[0113] Among them, the reference frame can be the reconstructed frame corresponding to the previous video frame of the current estimated frame to be processed. For example, if the estimated frame to be processed is the first estimated frame in the video frame sequence (that is, the second video frame in the video frame sequence), then the reference frame of the estimated frame to be processed is the reconstructed frame corresponding to the key frame (that is, the reconstructed frame corresponding to the first video frame in the video frame sequence); if the estimated frame to be processed is the second estimated frame in the video frame sequence (that is, the third video frame in the video frame sequence), then the reference frame of the estimated frame to be processed is the reconstructed frame corresponding to the first key frame in the video frame sequence (that is, the reconstructed frame corresponding to the second video frame in the video frame sequence).

[0114] In the above embodiment, the terminal encodes the key frame through the encoder to obtain a second encoded frame, and decodes the second encoded frame through the decoder to obtain a second reconstructed frame, so that the model loss value can be determined based on the second encoded frame and the second reconstructed frame, and the loss value can be minimized by optimizing the model parameters, thereby improving the encoding and decoding effect of the video encoding and decoding model.

[0115] In one embodiment, as shown in FIG6 , the terminal optimizes the video codec model based on the first coded frame, the first reconstructed frame, the second coded frame, and the second reconstructed frame to obtain the target video codec model, including the following steps:

[0116] S602 , determining a model loss value based on the first encoded frame, the first reconstructed frame, the second encoded frame, and the second reconstructed frame.

[0117] Among them, the model loss value is an indicator to measure the compression bit rate of the video frame and the difference between the reconstructed frame and the original frame. This indicator can be used to evaluate the compression effect of the video codec model on the video frame, and then adjust the parameters of the video codec model to make the codec model have a better compression effect on the video frame.

[0118] Specifically, after obtaining the first coded frame, the first reconstructed frame, the second coded frame and the second reconstructed frame, the terminal determines the model loss value based on the byte stream size of the first coded frame, the first reconstructed frame, the byte stream size of the second coded frame and the second reconstructed frame.

[0119] The byte stream size of the first coded frame is used to represent the compression code rate of the first coded frame, and the byte stream size of the second coded frame is used to represent the compression code rate of the second coded frame.

[0120] S604: Optimize the parameters of the video codec model based on the model loss value until the convergence condition is reached and stop training to obtain the target video codec model.

[0121] Convergence means that the model training process has become stable, that is, the video codec model has learned the characteristics of the data and no longer has significant improvement. Convergence conditions include a fixed number of training rounds, a fixed threshold of the loss function, etc. When the model reaches this condition, training is stopped to avoid overfitting.

[0122] Specifically, after obtaining the model loss value, the terminal uses the back propagation algorithm to adjust the values ​​of the weight parameters and bias parameters of the video codec model to obtain the adjusted video codec model, and re-executes step S202 until the training stops when the convergence condition is met, and the target video codec model is obtained.

[0123] In the above embodiment, the terminal determines the model loss value based on the first encoded frame, the first reconstructed frame, the second encoded frame and the second reconstructed frame, and optimizes the parameters of the video codec model based on the model loss value until the training is stopped when the convergence condition is reached. By continuously optimizing the model parameters, the trained target video codec model can encode and decode the video data more accurately, thereby improving the encoding and decoding effect of the target video codec model.

[0124] In one embodiment, as shown in FIG7 , the process of determining the model loss value based on the first coded frame, the first reconstructed frame, the second coded frame, and the second reconstructed frame by the terminal specifically includes the following steps:

[0125] S702: Determine a key frame loss value based on the first encoded frame and the first reconstructed frame.

[0126] Among them, the key frame loss value is an indicator used to measure the compression bit rate of the key frame and the difference between the first reconstructed frame and the original key frame. This indicator can be used to evaluate the compression effect of the pre-trained key frame network of the video codec model on the key frame, and then adjust the parameters of the video codec model to improve the compression effect of the video codec model on the video frame.

[0127] Specifically, after obtaining the first encoded frame and the first reconstructed frame, the terminal can also determine the byte stream size of the first encoded frame, and determine the key frame compression loss value based on the byte stream size of the first encoded frame, and determine the key frame reconstruction loss value based on the first reconstructed frame and the key frame, and then determine the key frame loss value based on the key frame compression loss value and the key frame reconstruction loss value.

[0128] Among them, the key frame compression loss value reflects the degree of information loss of the key frame during the video compression process, which can be specifically determined based on at least one of the byte stream sizes of the original frame and the compressed frame. The key frame reconstruction loss value reflects the degree of difference between the reconstructed key frame and the original key frame, which can be specifically determined by evaluating the pixel-level difference between the reconstructed frame and the original frame.

[0129] In one embodiment, after obtaining the key frame compression loss value and the key frame reconstruction loss value, the terminal inputs the key frame compression loss value and the key frame reconstruction loss value into the loss function represented by the following formula to determine the key frame loss value by the following formula: i =mse_loss i +bpp_loss i

[0130] Among them, Loss i is the key frame loss value corresponding to the key frame, mse_loss i The key frame reconstruction loss value corresponding to the key frame, bpp_loss i is a key frame compression loss value corresponding to the key frame, and the key frame compression loss value may specifically be the byte stream size of the first coded frame.

[0131] S704: Determine an estimated frame loss value based on the second coded frame and the second reconstructed frame estimation frame.

[0132] Among them, the estimated frame loss value is an indicator used to measure the compression bit rate of the estimated frame, and the difference between the first reconstructed frame and the original estimated frame. This indicator can be used to evaluate the compression effect of the pre-trained estimated frame network of the video codec model on the estimated frame, and then adjust the parameters of the video codec model to improve the compression effect of the video codec model on the video frame.

[0133] Specifically, after obtaining the second encoded frame and the second reconstructed frame, the terminal can also determine the byte stream size of the second encoded frame, and determine the estimated frame compression loss value based on the byte stream size of the second encoded frame, and determine the estimated frame reconstruction loss value based on the second reconstructed frame and the estimated frame, and then determine the estimated frame loss value based on the estimated frame compression loss value and the estimated frame reconstruction loss value.

[0134] In one embodiment, after obtaining the estimated frame compression loss value and the estimated frame reconstruction loss value, the terminal inputs the estimated frame compression loss value and the estimated frame reconstruction loss value into the loss function represented by the following formula to determine the estimated frame loss value by the following formula: Loss p =mse_loss p +bpp_loss p

[0135] Among them, Loss p is the estimated frame loss value corresponding to any estimated frame, mse_loss p bpp_loss is the estimated frame reconstruction loss value corresponding to any estimated frame, p It is an estimated frame compression loss value corresponding to any estimated frame. The estimated frame compression loss value may specifically be the byte stream size of the second coded frame.

[0136] S706 , determining a model loss value based on the key frame loss value and the estimated frame loss value.

[0137] Specifically, after obtaining the key frame loss value and the estimated frame loss value, the terminal obtains a preset loss function, inputs the key frame loss value and the estimated frame loss value into the preset loss function, and determines the model loss value based on the preset loss function.

[0138] In one embodiment, after obtaining the key frame loss value and the estimated loss value, the terminal inputs the key frame loss value and the estimated frame loss value into the loss function represented by the following formula to determine the model loss value by the following formula:

[0139] Among them, Loss i is the key frame loss value corresponding to the key frame (the first frame in the video frame sequence), is the estimated frame loss value corresponding to the j-1th estimated frame (the jth frame in the video frame sequence). There are n frames in the video frame sequence. Loss is the model loss value determined based on the n video frames in the video frame sequence.

[0140] In one embodiment, after obtaining the key frame loss value and the estimated frame loss value, the terminal can also obtain the first loss weight corresponding to the key frame loss value and the second loss weight corresponding to the estimated frame loss value, and determine the model loss value based on the first loss weight, the key frame loss value, the second loss weight and the estimated frame loss value.

[0141] In the above embodiment, the terminal determines the key frame loss value based on the first encoded frame and the first reconstructed frame, determines the estimated frame loss value based on the second encoded frame and the second reconstructed frame, and determines the model loss value based on the key frame loss value and the estimated frame loss value, so that the pre-trained key frame network and the pre-trained estimated frame network of the video codec model can be jointly trained based on the model loss value. By continuously optimizing the model parameters, the trained target video codec model can encode and decode the video data more accurately, thereby improving the encoding and decoding effect of the target video codec model.

[0142] In one embodiment, as shown in FIG8 , the video encoding and decoding processing method further includes a testing process for testing the trained target video encoding and decoding model. The testing process specifically includes the following steps:

[0143] S802: Extract a test video frame sequence from the test video, where the test video frame sequence includes a test key frame and a test estimation frame.

[0144] Among them, the test video is video data used to test the performance of the target video codec model. The test video contains various types of video frame sequences with different resolutions, different encoding qualities, different scenes, etc., to comprehensively evaluate the encoding and decoding effects of the model under different situations.

[0145] Specifically, after obtaining the test video, the terminal extracts a test video frame sequence from the test video at a certain time interval, determines the first frame in the test video frame sequence as the test key frame, and determines other video frames other than the first frame in the test video frame sequence as test estimated frames.

[0146] S804 , performing encoding and decoding processing on the test key frame through the pre-trained key frame network of the target video encoding and decoding model to obtain a first test encoded frame and a corresponding first test reconstructed frame.

[0147] Specifically, after obtaining the test video frame sequence, the terminal inputs the test key frame in the test video frame sequence into the pre-trained key frame network of the target video codec model, and encodes and decodes the test key frame through the pre-trained key frame network, thereby obtaining the first test encoding frame corresponding to the test key frame, and the first test reconstructed frame corresponding to the first test encoding frame.

[0148] S806 , performing encoding and decoding processing on the test estimated frame through the pre-trained estimated frame network of the target video encoding and decoding model to obtain a second test encoded frame and a corresponding second test reconstructed frame.

[0149] Specifically, after obtaining the test video frame sequence, the terminal inputs the test estimation frame in the test video frame sequence into the pre-trained estimation frame network of the target video codec model, and encodes and decodes the test estimation frame through the pre-trained estimation frame network, thereby obtaining a second encoded test frame corresponding to the test estimation frame, and a second reconstructed test frame corresponding to the second encoded test frame.

[0150] S808 : Determine the encoding and decoding effect of the target video encoding and decoding model based on the first test encoding frame, the second test encoding frame, the first test reconstructed frame, and the second test reconstructed frame.

[0151] Specifically, after obtaining the first test coding frame, the second test coding frame, the first test reconstruction frame and the second test reconstruction frame, the terminal determines the compression evaluation result of the target video codec model based on the first test coding frame and the second test coding frame, determines the reconstruction evaluation result of the target video codec model based on the first test reconstruction frame and the second test reconstruction frame, and determines the coding and decoding effect of the target video codec model based on the compression evaluation result and the reconstruction evaluation result.

[0152] The compression evaluation results include the size of the compressed byte stream (bpp). Bpp (bits per pixel) is a video coding efficiency indicator that indicates the number of bits required for each pixel. It can be understood that the lower the bpp value, the higher the video coding efficiency, that is, the fewer bits are required at the same visual quality.

[0153] In one embodiment, after obtaining the first coded frame and the second coded frame, the terminal determines the size of the compressed byte stream of the first coded frame and the size of the compressed byte stream of the second coded frame respectively, and determines the size of the compressed byte stream of the test video frame sequence based on the size of the compressed byte stream of the first coded frame and the size of the compressed byte stream of the second coded frame, and determines the size of the compressed byte stream of the test video frame sequence as the compression evaluation result of the target video codec model.

[0154] The reconstruction evaluation results include the quality of the reconstructed image (PNSR). PSNR (Peak Signal-to-Noise Ratio) is a video quality evaluation indicator used to compare the similarity between the original video frame image and the encoded and decoded video frame image. It can be understood that the higher the PSNR value, the better the reconstructed video quality, that is, the better the encoding and decoding effect.

[0155] Specifically, after obtaining the first test reconstructed frame and the second test reconstructed frame, the terminal determines the quality of the reconstructed image of the first test reconstructed frame based on the first test reconstructed frame and the test key frame, determines the quality of the reconstructed image of the second test reconstructed frame based on the second test reconstructed frame and the test estimation frame, and determines the quality of the reconstructed image of the test video frame sequence based on the quality of the reconstructed image of the first test reconstructed frame and the quality of the reconstructed image of the second test reconstructed frame, and determines the quality of the reconstructed image of the test video frame sequence as the reconstruction evaluation result of the target video codec model.

[0156] In one embodiment, after obtaining the compression evaluation result and the reconstruction evaluation result of the target video coding model, the terminal generates a reconstruction evaluation score based on the compression evaluation result and the reconstruction evaluation result. The reconstruction evaluation score is the reconstruction evaluation result.

[0157] As shown in Table 1 below, these are the reconstruction evaluation results of the traditional video codec model trained using the traditional scheme and the reconstruction evaluation results of the target video codec model trained using the scheme of the present application. It can be seen from the table that for the encoding and decoding of I frames, P0 frames, P1 frames and P2 frames, the reconstruction evaluation scores of the traditional video codec model are 80 points, 70 points, 70 points and 70 points respectively, and the overall reconstruction evaluation score is 72.5 points, while the reconstruction evaluation scores of the target video codec model are 75 points, 75 points, 75 points and 75 points respectively, and the overall reconstruction evaluation score is 75 points. Therefore, the reconstruction evaluation results of the target video codec model are better overall.

[0158] Table 1

[0159] In one embodiment, after obtaining the compression evaluation result and the reconstruction evaluation result, the terminal generates a reconstruction evaluation graph based on the compression evaluation result and the reconstruction evaluation result, and determines the reconstruction evaluation result of the target video codec model based on the reconstruction evaluation graph.

[0160] Referring to Figure 9, Figure 9 shows the reconstruction evaluation results obtained by processing the test video using different video codec models. The test video is the UVG dataset (dataset), which is a video quality assessment dataset provided by the University of Texas at Austin. The dataset includes 20 720p videos of different themes and content, each video contains 5 versions with different compression levels, for a total of 100 videos. Wherein the horizontal axis in Figure 9 is bpp, the vertical axis is PSNR, H264, H265, and H266 represent three generations of typical traditional codec technologies for reference. A, B, and C respectively represent the traditional video codec models (machine learning models) obtained by training the initial models of different structures using the traditional scheme, and A-ours, B-ours, and C-ours respectively represent the target video codec models obtained by training the initial models of different structures using the scheme of the present application. It can be seen from the figure that the reconstruction evaluation results obtained by the target video codec model trained using the scheme of the present application are better than the corresponding traditional video codec model and the corresponding traditional codec technology.

[0161] In the above embodiment, the terminal extracts a test video frame sequence from the test video, where the test video frame sequence includes a test key frame and a test estimated frame. The test key frame is encoded and decoded by the pre-trained key frame network of the target video codec model to obtain a first test coded frame and a corresponding first test reconstructed frame. The test estimated frame is encoded and decoded by the pre-trained estimated frame network of the target video codec model to obtain a second test coded frame and a corresponding second test reconstructed frame. The encoding and decoding effect of the target video codec model on the new data is determined based on the first test coded frame, the second test coded frame, the first test reconstructed frame and the second test reconstructed frame, so as to evaluate the performance of the model in actual application scenarios, timely discover problems with the model and make adjustments and optimizations, thereby improving the generalization ability and practicality of the model. At the same time, testing can provide reliable support for the application of the model and improve the credibility of its application.

[0162] In one embodiment, as shown in FIG10 , the process of encoding and decoding the target video by the terminal using the target video encoding and decoding model specifically includes the following steps:

[0163] S1002: extracting a to-be-processed video frame sequence from the target video, where the to-be-processed video frame sequence includes to-be-processed key frames and to-be-processed estimated frames.

[0164] Specifically, the terminal can be a transmitter or a receiver. After obtaining the target video, the transmitter extracts a sequence of video frames to be processed from the target video at a certain time interval, and determines the first frame in the sequence of video frames to be processed as a key frame to be processed, and determines other video frames other than the first frame in the sequence of video frames to be processed as estimated frames to be processed.

[0165] S1004 , encoding the key frame to be processed and the estimated frame to be processed respectively through the pre-trained key frame network and the pre-trained estimated frame network of the target video codec model to obtain a first processed encoded frame and a second processed encoded frame.

[0166] Specifically, after obtaining the sequence of video frames to be processed, the terminal inputs the key frames to be processed in the sequence of video frames to be processed into the pre-trained key frame network of the target video codec model, and encodes the key frames to be processed through the encoder of the pre-trained key frame network, thereby obtaining the first processed encoded frame corresponding to the key frames to be processed; the estimated frames to be processed in the sequence of video frames to be processed are input into the pre-trained estimated frame network of the target video codec model, and encodes the estimated frames to be processed through the encoder of the pre-trained estimated frame network, thereby obtaining the second processed encoded frame corresponding to the estimated frames to be processed.

[0167] In one embodiment, the terminal inputs the estimated frame to be processed in the video frame sequence to be processed into the pre-trained estimated frame network of the target video codec model, and the process of encoding the estimated frame to be processed through the encoder of the pre-trained estimated frame network specifically includes the following steps: inputting the estimated frame to be processed in the video frame sequence to be processed and its corresponding reference frame to be processed into the pre-trained estimated frame network of the video codec model, performing motion estimation and motion compensation on the reference frame to be processed and the estimated frame to be processed through the encoder of the pre-trained estimated frame network to obtain a difference frame, and compressing and encoding the difference frame to obtain a second processed encoded frame.

[0168] Among them, the reference frame to be processed can be the reconstructed frame corresponding to the previous video frame of the estimated frame to be processed. For example, if the estimated frame to be processed is the first estimated frame in the video frame sequence to be processed (that is, the second video frame in the video frame sequence to be processed), then the reference frame to be processed of the estimated frame to be processed is the reconstructed frame corresponding to the key frame in the video frame sequence to be processed (that is, the reconstructed frame corresponding to the first video frame in the video frame sequence to be processed); if the estimated frame to be processed is the second estimated frame in the video frame sequence (that is, the third video frame in the video frame sequence), then the reference frame to be processed of the estimated frame to be processed is the reconstructed frame corresponding to the first estimated frame in the video frame sequence to be processed (that is, the reconstructed frame corresponding to the second video frame in the video frame sequence to be processed).

[0169] It can be understood that after obtaining the first processed coded frame, the terminal can also input the first processed coded frame into the decoder of the pre-trained key frame network, decode the first processed coded frame through the decoder to obtain the first processed reconstructed frame, and use the first processed reconstructed frame as the reference frame of the first estimated frame in the video frame sequence to be processed, and input it and the first estimated frame together into the encoder of the pre-trained estimated frame network, perform motion estimation and motion compensation on the reference frame and the first estimated frame through the encoder to obtain a difference frame, and compress and encode the difference frame to obtain the first estimated frame corresponding to the frame. The second processed coded frame is obtained, and the reference frame of the first estimated frame and its corresponding second processed coded frame are input into the decoder of the pre-trained estimated frame network, and the second processed coded frame corresponding to the first estimated frame is decoded by the decoder to obtain a decoding result, and the second processed reconstructed frame corresponding to the first estimated frame is restored based on the decoding result and the reference frame of the first estimated frame, and the second processed reconstructed frame corresponding to the first estimated frame is used as the reference frame of the second estimated frame in the video frame sequence to be processed, and so on, until the second processed coded frame corresponding to each video frame in the video frame sequence to be processed is obtained.

[0170] It should be noted that after encoding the video frame sequence to be processed and obtaining the first processed encoded frame and the second processed encoded frame, the first processed encoded frame and the second processed encoded frame are transmitted to the receiving end so that the receiving end decodes the first processed encoded frame and the second processed encoded frame after receiving them to obtain the restored target video.

[0171] S1006, decoding the first processed coded frame and the second processed coded frame respectively through the pre-trained key frame network and the pre-trained estimated frame network of the target video codec model to obtain a first processed reconstructed frame and a second processed reconstructed frame.

[0172] Specifically, after obtaining the first processed coded frame and the second processed coded frame, the terminal inputs the first processed coded frame into the pre-trained key frame network of the target video codec model, decodes the first processed coded frame through the decoder of the pre-trained key frame network, and obtains the first processed reconstructed frame; inputs the second processed coded frame into the pre-trained estimated frame network of the target video codec model, decodes the second processed coded frame through the decoder of the pre-trained estimated frame network, and obtains the second processed reconstructed frame.

[0173] In the above embodiment, the terminal encodes the key frames to be processed and the estimated frames to be processed respectively through the pre-trained key frame network and the pre-trained estimated frame network of the target video processing model, and can compress the video data; by decoding the encoded data, the compressed video data can be restored to the original video data, and video decompression is achieved, thereby effectively reducing the storage and transmission costs of the video data, improving the transmission efficiency of the video data, and maintaining the high definition and good visual quality of the video.

[0174] In one embodiment, as shown in FIG11 , a video encoding and decoding processing method is further provided. The method is described by taking the computer device in FIG1 as an example, and includes the following steps:

[0175] S1102, obtaining an original video that meets the clarity condition; performing boundary detection on the original video to obtain scene boundaries in the original video; extracting video segments containing continuous scenes from the original video based on the scene boundaries; performing artifact cleaning processing on the video segments to obtain sample videos.

[0176] S1104 , extracting a video frame sequence from the sample video, where the video frame sequence includes key frames and estimated frames.

[0177] S1106, encoding the key frame through the encoder of the pre-trained key frame network of the video codec model to obtain a first encoded frame; decoding the first encoded frame through the decoder of the pre-trained key frame network of the video codec model to obtain a first reconstructed frame.

[0178] S1108, encoding the estimated frame through the encoder of the pre-trained estimated frame network of the video codec model to obtain a second encoded frame; decoding the second encoded frame through the decoder of the pre-trained estimated frame network to obtain a second reconstructed frame.

[0179] S1110, determining a key frame loss value based on the first encoded frame and the first reconstructed frame; determining an estimated frame loss value based on the second encoded frame and the second reconstructed frame; and determining a model loss value based on the key frame loss value and the estimated frame loss value.

[0180] S1112, optimizing the parameters of the video codec model based on the model loss value until the convergence condition is reached and stopping the training to obtain the target video codec model.

[0181] S1114: When the target video is obtained, encoding and decoding the target video is performed using the target video encoding and decoding model.

[0182] The present application also provides an application scenario, which applies the above-mentioned video encoding and decoding processing method. Specifically, the video encoding and decoding processing method can be integrated into the software system, and a corresponding interface is provided. By calling the interface, the encoding and decoding processing of the video data can be realized by applying the above-mentioned video encoding and decoding processing method.

[0183] Referring to Figure 12, the above-mentioned video encoding and decoding processing method can be applied to the encoding end and the decoding end. The encoding end inputs the video stream to be encoded and outputs the encoded byte stream. The decoding end inputs the encoded byte stream and outputs the decoded video. The encoding end is usually the server end and the decoding end is usually the client end, thereby minimizing the amount of data transmitted for the video to save costs.

[0184] The present application also provides an application scenario, which applies the above-mentioned video encoding and decoding processing method, and the video encoding and decoding method specifically includes the following steps:

[0185] 1. Training data acquisition

[0186] Collecting the dataset mainly includes the following steps: obtaining public HD video links on the Internet; using tools to download the HD video data corresponding to all HD video links; using the tool scene detect to obtain continuous scenes in all HD videos; extracting video frames of fixed length (10-30) from the continuous scenes, each fixed-length continuous frame is called a clip; cleaning the video frame image (H, W) of artifacts, and also adjusting the video frame image size to (H×2 / 3, W×2 / 3).

[0187] In this embodiment, more than 220,000 high-definition video scene clips were collected through the above strategy, providing better training data for model training.

[0188] 2. Model training

[0189] Referring to Figure 13, in the model training stage, it is first preferred to train the I frame model (key frame network) and the P frame model (estimated frame network) separately, so that the I frame model and the P frame model reach the optimal state respectively, and then the I frame model and the P frame model are jointly optimized so that the video encoding and decoding indicators of the target video encoding and decoding model obtained by the combination of the I frame model and the P frame model are optimal.

[0190] in:

[0191] a. Train the I-frame model separately

[0192] For the I-frame model that has been preliminarily trained, the collected high-definition training data is used to collect and train the I-frame model. The input is a single-frame original image, and the output is a single-frame reconstructed image and a compressed byte stream. The I-frame model loss value is determined based on the single-frame original image, the single-frame reconstructed image and the compressed byte stream. Based on the I-frame model loss value, the parameters of the preliminarily trained I-frame model are adjusted until the training is stopped when the convergence conditions are met, and the pre-trained I-frame model is obtained.

[0193] b. Train the P frame model separately

[0194] For the preliminarily trained P-frame model, we use larger input images for training and increase the number of consecutive input frames n-1, such as an input size of 512*512 (the image size used in traditional training is 256*256) and a single training frame number of 6 (the number of single training frames used in traditional training is 5). For each P-frame, the reconstructed frame of the previous frame is selected as the reference frame of the P-frame. The input is continuous P-frames, and the output is the reconstructed image and compressed byte stream of each P-frame. The P-frame model loss value is determined based on each P-frame, the reconstructed image of each P-frame, and the compressed byte stream. Based on the P-frame model loss value, the parameters of the preliminarily trained P-frame model are adjusted until the convergence condition is met and the training is stopped to obtain the pre-trained P-frame model.

[0195] c. Joint training of I-frame model and P-frame model

[0196] Referring to Figure 14, the I frame model and the P frame model are taken as a whole, and the input is a complete original GOP video frame (n frames in total). The output is the result of encoding and decoding n video frames by the I frame model and the P frame model, including 1 I frame model reconstructed image and the corresponding compressed byte stream, as well as n-1 P frame model reconstructed images and the corresponding compressed byte stream. During training, n frame inputs are considered, and the I frame model and the P frame model are taken as a whole for joint training. Based on 1 I frame model reconstructed image and the corresponding compressed byte stream (bits), as well as n-1 P frame model reconstructed images and the corresponding compressed byte stream (bits), the model loss value of the video encoding and decoding model obtained by the combination of the I frame model and the P frame model is determined. Based on the model loss value, the parameters of the video encoding and decoding model obtained by the combination of the I frame model and the P frame model are optimized until the convergence condition is reached, and the target video encoding and decoding model of the combination of the I frame model and the P frame model is obtained.

[0197] 3. Model testing

[0198] In order to avoid the loss of effect evaluation indicators caused by the existing scheme using lossless I frames (original I frames) as reference frames when training P frames, and using lossy I frames (reconstructed I frames) as reference frames during the testing process, the embodiment of the present application treats the I frame model and the P frame model as a whole, and uses the reconstructed lossy image of the previous I frame as a reference frame for the first P frame during training, while the logic during testing is completely consistent with that during training, thereby avoiding the loss of effect evaluation indicators. For all subsequent P frames, the logic of using reference frames during training and testing is also completely consistent.

[0199] 4. Model application stage

[0200] After obtaining the target video codec model, the target video codec model is widely used in scenarios that require video codec processing, such as scenarios where video is transmitted, stored or displayed, such as video conferencing, live video broadcasting, video monitoring, online education, digital entertainment and other fields. Specifically, the target video codec model can also be integrated into the software system as a video codec processing method. By executing the video codec processing method, the efficiency of video transmission and storage is improved, the data transmission cost and storage cost are reduced, and the video display quality is improved. Corresponding interfaces are provided to facilitate software system developers to carry out integrated development, and can be customized according to actual needs to meet the needs of different customers.

[0201] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0202] Based on the same inventive concept, the embodiments of the present application also provide a video codec processing device for implementing the video codec processing method involved above. The implementation solution provided by this device is similar to the implementation solution described in the above method. Therefore, the specific limitations of the one or more video codec processing device embodiments provided below can be found in the limitations of the video codec processing method above, and will not be repeated here.

[0203] In one embodiment, as shown in FIG15 , a video encoding and decoding apparatus is provided, comprising: a video frame extraction module 1502 , a key frame encoding module 1504 , an estimated frame encoding module 1506 , a model optimization module 1508 , and a model application module 1510 , wherein:

[0204] A video frame extraction module 1502 is configured to extract a video frame sequence from a sample video, where the video frame sequence includes key frames and estimated frames;

[0205] A key frame encoding module 1504 is configured to perform encoding and decoding processing on the key frame using a pre-trained key frame network of a video encoding and decoding model to obtain a first encoded frame and a corresponding first reconstructed frame;

[0206] An estimated frame encoding module 1506 is configured to perform encoding and decoding processing on the estimated frame using a pre-trained estimated frame network of a video encoding and decoding model to obtain a second encoded frame and a corresponding second reconstructed frame;

[0207] A model optimization module 1508 is configured to optimize the video codec model based on the first encoded frame, the first reconstructed frame, the second encoded frame, and the second reconstructed frame to obtain a target video codec model;

[0208] The model application module 1510 is used to perform encoding and decoding processing on the target video by using the target video encoding and decoding model when the target video is obtained.

[0209] In the above embodiment, after obtaining the video codec model including the pre-trained key frame network and the pre-trained estimated frame network, the terminal does not directly use the video codec model to process the video codec task, but extracts a video frame sequence from the sample video, where the video frame sequence includes key frames and estimated frames, so that different encoding and decoding methods are used for the key frames and the estimated frames, respectively, to improve the video compression rate while ensuring the video compression quality; for the key frames, the pre-trained key frame network of the video codec model is used to encode and decode them to obtain a first encoded frame and a corresponding first reconstructed frame; for the estimated frames, the pre-trained estimated frame network of the video codec model is used to encode and decode them to obtain a second The encoded frame and the corresponding second reconstructed frame, so that based on the first encoded frame, the first reconstructed frame, the second encoded frame and the second reconstructed frame, the pre-trained key frame network and the pre-trained estimated frame network of the video codec model can be jointly trained, that is, the parameters of the model are further optimized, so that the trained target video codec model has better encoding and decoding capabilities for videos that meet specific conditions. For example, when the sample video used is a video that meets specific clarity conditions (high definition and ultra-high definition), the target video codec model can improve the compression quality and compression rate of the video when processing the encoding and decoding task of the target video that meets the specific clarity conditions (high definition and ultra-high definition), that is, the encoding and decoding effect of the video is improved.

[0210] In one embodiment, as shown in FIG16 , the apparatus further includes a sample acquisition module 1512 for: acquiring an original video that meets a clarity condition; performing boundary detection on the original video to obtain scene boundaries in the original video; and extracting a video segment containing continuous scenes from the original video as a sample video based on the scene boundaries.

[0211] In one embodiment, the sample acquisition module 1512 is further configured to: extract video segments containing continuous scenes from the original video based on scene boundaries; and perform artifact cleaning on the video segments to obtain sample videos.

[0212] In one embodiment, the pre-trained key frame network is obtained by training the initial key frame network; the device also includes a first pre-training module 1514, which is used to: encode and decode the video frames in the first initial video frame sequence through the initial key frame network to obtain a third encoded frame and a corresponding third reconstructed frame; optimize the parameters of the initial key frame network based on the third encoded frame and the third reconstructed frame to obtain a pre-trained key frame network.

[0213] In one embodiment, as shown in Figure 16, the pre-trained estimated frame network of the video codec model is obtained by training the initial estimated frame network; the device also includes a second pre-training module 1516, which is used to: encode and decode the second initial video frame sequence through the initial estimated frame network to obtain a fourth encoded frame and a corresponding fourth reconstructed frame; optimize the parameters of the initial estimated frame network based on the fourth encoded frame and the fourth reconstructed frame to obtain a pre-trained estimated frame network.

[0214] In one embodiment, the pre-trained key frame network of the video codec model includes an encoder and a decoder: the key frame encoding module 1504 is further used to: encode the key frame through the encoder to obtain a first encoded frame; and decode the first encoded frame through the decoder to obtain a first reconstructed frame.

[0215] In one embodiment, the pre-trained estimated frame network of the video codec model includes an encoder and a decoder; the estimated frame encoding module 1506 is further used to: encode the estimated frame through the encoder to obtain a second encoded frame; and decode the second encoded frame through the decoder to obtain a second reconstructed frame.

[0216] In one embodiment, the model optimization module 1508 is further used to: determine a model loss value based on the first encoded frame, the first reconstructed frame, the second encoded frame, and the second reconstructed frame; optimize the parameters of the video codec model based on the model loss value, stop training until the convergence condition is reached, and obtain the target video codec model.

[0217] In one embodiment, the model optimization module 1508 is further used to: determine a key frame loss value based on the first encoded frame and the first reconstructed frame; determine an estimated frame loss value based on the second encoded frame and the second reconstructed frame; and determine a model loss value based on the key frame loss value and the estimated frame loss value.

[0218] In one embodiment, as shown in FIG16 , the apparatus further includes a testing module 1518 for: extracting a test video frame sequence from a test video, the test video frame sequence including a test key frame and a test estimated frame; performing encoding and decoding processing on the test key frame through a pre-trained key frame network of a target video codec model to obtain a first test encoded frame and a corresponding first test reconstructed frame; performing encoding and decoding processing on the test estimated frame through a pre-trained estimated frame network of a target video codec model to obtain a second test encoded frame and a corresponding second test reconstructed frame; and determining the encoding and decoding effect of the target video codec model based on the first test encoded frame, the second test encoded frame, the first test reconstructed frame and the second test reconstructed frame.

[0219] In one embodiment, the model application module 1510 is further used to: extract a sequence of video frames to be processed from the target video, the sequence of video frames to be processed including key frames to be processed and estimated frames to be processed; encode the key frames to be processed and the estimated frames to be processed respectively through the pre-trained key frame network and the pre-trained estimated frame network of the target video codec model to obtain a first processed encoded frame and a second processed encoded frame; decode the first processed encoded frame and the second processed encoded frame respectively through the pre-trained key frame network and the pre-trained estimated frame network of the target video codec model to obtain a first processed reconstructed frame and a second processed reconstructed frame.

[0220] Each module in the video encoding and decoding apparatus described above may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0221] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in Figure 17. The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, memory, and I / O interface are connected via a system bus, and the communication interface is connected to the system bus via the I / O interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer-readable instructions, and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the non-volatile storage medium. The database of the computer device is used to store video data. The I / O interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via a network connection. When the computer-readable instructions are executed by the processor, a video encoding and decoding processing method is implemented.

[0222] In one embodiment, a computer device is provided, which may be a terminal. Its internal structure diagram may be as shown in FIG18 . The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer-readable instructions. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the non-volatile storage medium. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals via wired or wireless communication, where the wireless communication may be achieved via Wi-Fi, a mobile cellular network, NFC (near-field communication), or other technologies. When executed by the processor, the computer-readable instructions implement a video encoding and decoding method. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0223] Those skilled in the art will understand that the structure shown in Figure 17 or Figure 18 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0224] In one embodiment, a computer device is further provided, including a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps in the above-mentioned method embodiments when executing the computer-readable instructions.

[0225] In one embodiment, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0226] In one embodiment, a computer program product is provided, comprising computer-readable instructions, which implement the steps in the above-mentioned method embodiments when executed by a processor.

[0227] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0228] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0229] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0230] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. The scope of protection of the present application shall be determined by the appended claims.

Claims

1. A video encoding and decoding processing method, executed by a computer device, the method comprising: Extracting a video frame sequence from a sample video, wherein the video frame sequence includes a key frame and an estimated frame; Performing encoding and decoding processing on the key frame through a pre-trained key frame network of a video encoding and decoding model to obtain a first encoded frame and a corresponding first reconstructed frame; Performing encoding and decoding processing on the estimated frame through a pre-trained estimated frame network of the video encoding and decoding model to obtain a second encoded frame and a corresponding second reconstructed frame; Based on the first encoded frame, the first reconstructed frame, the second encoded frame and the second reconstructed frame, optimizing the video encoding and decoding model to obtain a target video encoding and decoding model; When the target video is obtained, the target video is encoded and decoded by using the target video encoding and decoding model.

2. The method according to claim 1, further comprising: Obtaining the original video that meets the definition requirement; Performing boundary detection on the original video to obtain scene boundaries in the original video; Based on the scene boundary, a video segment containing continuous scenes is extracted from the original video as the sample video.

3. The method according to claim 2, wherein extracting a video segment containing continuous scenes from the original video as the sample video based on the scene boundary comprises: Extracting a video segment containing continuous scenes from the original video based on the scene boundary; Performing artifact cleaning processing on the video clip to obtain the sample video.

4. According to the method of claim 1, the pre-trained key frame network is obtained by training the initial key frame network; The training of the initial key frame network includes: Performing encoding and decoding processing on the video frames in the first initial video frame sequence through the initial key frame network to obtain a third encoded frame and a corresponding third reconstructed frame; Parameters of the initial key frame network are optimized based on the third encoded frame and the third reconstructed frame to obtain the pre-trained key frame network.

5. The method according to claim 1, wherein the pre-trained estimated frame network is obtained by training the initial estimated frame network; The training of the initial estimation frame network includes: Performing encoding and decoding processing on the second initial video frame sequence through the initial estimation frame network to obtain a fourth encoded frame and a corresponding fourth reconstructed frame; Parameters of the initial estimated frame network are optimized based on the fourth encoded frame and the fourth reconstructed frame to obtain the pre-trained estimated frame network.

6. The method according to claim 1, wherein the pre-trained key frame network comprises an encoder and a decoder: The step of encoding and decoding the key frame through a pre-trained key frame network of a video encoding and decoding model to obtain a first encoded frame and a corresponding first reconstructed frame includes: Encoding the key frame by the encoder to obtain a first encoded frame; The first encoded frame is decoded by the decoder to obtain a first reconstructed frame.

7. The method according to claim 1, wherein the pre-trained estimated frame network comprises an encoder and a decoder; The method of performing encoding and decoding processing on the estimated frame through the pre-trained estimated frame network of the video encoding and decoding model to obtain a second encoded frame and a corresponding second reconstructed frame includes: Performing encoding processing on the estimated frame by the encoder to obtain a second encoded frame; The second encoded frame is decoded by the decoder to obtain a second reconstructed frame.

8. The method according to any one of claims 1 to 7, wherein optimizing the video codec model based on the first coded frame, the first reconstructed frame, the second coded frame and the second reconstructed frame to obtain a target video codec model comprises: determining a model loss value based on the first encoded frame, the first reconstructed frame, the second encoded frame, and the second reconstructed frame; The parameters of the video codec model are optimized based on the model loss value, and the training is stopped until a convergence condition is reached to obtain a target video codec model.

9. The method according to claim 8, wherein determining the model loss value based on the first encoded frame, the first reconstructed frame, the second encoded frame, and the second reconstructed frame comprises: determining a key frame loss value based on the first encoded frame and the first reconstructed frame; determining an estimated frame loss value based on the second encoded frame and the second reconstructed frame; A model loss value is determined based on the key frame loss value and the estimated frame loss value.

10. The method according to claim 1, further comprising: Extracting a test video frame sequence from a test video, wherein the test video frame sequence includes a test key frame and a test estimation frame; Performing encoding and decoding processing on the test key frame through the pre-trained key frame network of the target video encoding and decoding model to obtain a first test encoded frame and a corresponding first test reconstructed frame; Performing encoding and decoding processing on the test estimated frame through the pre-trained estimated frame network of the target video encoding and decoding model to obtain a second test encoded frame and a corresponding second test reconstructed frame; The encoding and decoding effect of the target video encoding and decoding model is determined based on the first test encoding frame, the second test encoding frame, the first test reconstructed frame and the second test reconstructed frame.

11. The method according to claim 1, wherein encoding and decoding the target video using the target video encoding and decoding model comprises: Extracting a to-be-processed video frame sequence from a target video, wherein the to-be-processed video frame sequence includes a to-be-processed key frame and a to-be-processed estimated frame; Through the pre-trained key frame network and the pre-trained estimated frame network of the target video codec model, encoding the key frame to be processed and the estimated frame to be processed is performed respectively to obtain a first processed encoded frame and a second processed encoded frame; The first processed coded frame and the second processed coded frame are decoded respectively by the pre-trained key frame network and the pre-trained estimated frame network of the target video codec model to obtain a first processed reconstructed frame and a second processed reconstructed frame.

12. A video encoding and decoding processing device, characterized in that: The device comprises: A video frame extraction module, used to extract a video frame sequence from a sample video, wherein the video frame sequence includes a key frame and an estimated frame; A key frame encoding module, used for encoding and decoding the key frame through a pre-trained key frame network of a video encoding and decoding model to obtain a first encoded frame and a corresponding first reconstructed frame; An estimated frame encoding module, used for performing encoding and decoding processing on the estimated frame through a pre-trained estimated frame network of the video encoding and decoding model to obtain a second encoded frame and a corresponding second reconstructed frame; A model optimization module, configured to optimize the video coding model based on the first coded frame, the first reconstructed frame, the second coded frame and the second reconstructed frame to obtain a target video coding model; The model application module is used to encode and decode the target video through the target video encoding and decoding model when the target video is obtained.

13. A computer device, comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the method according to any one of claims 1 to 11 when executing the computer-readable instructions.

14. A computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 11.

15. A computer program product comprising computer readable instructions, characterized in that: When the computer readable instructions are executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Citation Information

Patent Citations

  • Signal processing method based on deep neural network

    CN112203093A

  • Video coding method based on video frame generation

    CN113132735A

  • Video coding method, model training method, equipment and storage medium

    CN116095328A

  • Video coding and decoding processing method and device, computer equipment and storage medium

    CN116233445A

  • Online training of hierarchical algorithms

    GB2548749A

Cited By

  • Archive management system based on distributed storage

    CN119669150A