Video encoding method, video decoding method, and apparatus

Abstract

The present application relates to the technical field of video encoding / decoding, and provides a video encoding method, a video decoding method, and an apparatus. The video encoding method comprises: acquiring a student model, wherein the student model is a model obtained by guiding training of a first network model by means of a teacher model, the teacher model is a model obtained by training a second network model, and when a same video is encoded separately on the basis of the first network model and the second network model, a bit rate of an encoding result from the first network model is lower than a bit rate of an encoding result from the second network model; and on the basis of the student model, acquiring an encoding result of a video to be encoded. In some embodiments of the present application, in a video encoding solution in which a deep learning network model is used for video encoding, a bit rate of a video encoding result is reduced while video reconstruction quality is ensured.

Description

Video encoding method, video decoding method and device

[0001] The present application claims priority to the Chinese patent application No. 202411391447.6, filed on September 30, 2024, and entitled "Video encoding method, video decoding method and device", the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] Some embodiments of the present application relate to the technical field of video coding. More specifically, the present application relates to a video encoding method, a video decoding method and a device. BACKGROUND

[0003] With the development of deep learning technology, video encoding schemes using deep learning network models for video encoding have also been widely applied and researched.

[0004] In the video encoding scheme using deep learning network models for video encoding, it is also necessary to balance distortion and code rate to determine the reconstruction quality and code rate size of the video encoding result. Generally, the larger the code rate of the video encoding result, the more information can be provided for video reconstruction, and the better the video reconstruction quality; on the contrary, the smaller the code rate of the video encoding result, the less information can be provided for video reconstruction, and the worse the video reconstruction quality. Therefore, in the video encoding scheme using deep learning network models for video encoding, how to reduce the code rate of the video encoding result while ensuring the video reconstruction quality is a problem to be solved. SUMMARY

[0005] The exemplary embodiments of the present application provide a video encoding method, a video decoding method and a device for reducing the code rate of the video encoding result while ensuring the video reconstruction quality in the video encoding scheme using deep learning network models for video encoding.

[0006] Some embodiments of the present application provide the technical solutions as follows:

[0007] In a first aspect, some embodiments of the present application provide a video encoding method, comprising:

[0008] obtaining a student model, the student model being a model obtained by training a first network model guided by a teacher model, the teacher model being a model obtained by training a second network model, and the code rate of the encoding result of the first network model being smaller than the code rate of the encoding result of the second network model when the same video is encoded based on the first network model and the second network model respectively;

[0009] obtaining the encoding result of the to-be-encoded video according to the student model.

[0010] In a second aspect, some embodiments of the present application provide a video decoding method, comprising:

[0011] obtaining encoding data of a to-be-decoded video;

[0012] constructing a student model according to the encoding data of the to-be-decoded video, the student model being a model obtained by guiding training of a first network model by a teacher model, the teacher model being a model obtained by training a second network model, and a code rate of an encoding result of the first network model being less than a code rate of an encoding result of the second network model when the first network model and the second network model encode a same video respectively;

[0013] obtaining a reconstructed video corresponding to the to-be-decoded video according to the student model.

[0014] In a third aspect, some embodiments of the present application provide a video decoding method, comprising:

[0015] obtaining encoding data of a to-be-decoded video;

[0016] obtaining representation information of each video frame of the to-be-decoded video according to the encoding data;

[0017] inputting the representation information of each video frame of the to-be-decoded video into a student model to obtain a reconstructed video frame corresponding to each video frame of the to-be-decoded video output by the student model;

[0018] wherein the student model is a model obtained by guiding training of a first network model by a teacher model, the teacher model being a model obtained by training a second network model, and a code rate of an encoding result of the first network model being less than a code rate of an encoding result of the second network model when the first network model and the second network model encode a same video respectively.

[0019] In a fourth aspect, some embodiments of the present application provide a video encoding apparatus, comprising:

[0020] a model training unit configured to obtain a student model, the student model being a model obtained by guiding training of a first network model by a teacher model, the teacher model being a model obtained by training a second network model, and a code rate of an encoding result of the first network model being less than a code rate of an encoding result of the second network model when the first network model and the second network model encode a same video respectively;

[0021] a video encoding unit configured to obtain an encoding result of a to-be-encoded video according to the student model.

[0022] In a fifth aspect, some embodiments of the present application provide a video decoding apparatus, comprising:

[0023] an obtaining unit, configured to obtain encoding data of a to-be-decoded video;

[0024] a constructing unit, configured to construct a student model according to the encoding data of the to-be-decoded video, the student model being a model obtained by guiding training of a first network model by a teacher model, the teacher model being a model obtained by training a second network model, and a code rate of an encoding result of the first network model being less than a code rate of an encoding result of the second network model when the first network model and the second network model encode a same video respectively;

[0025] a reconstructing unit, configured to obtain a reconstructed video corresponding to the to-be-decoded video according to the student model.

[0026] In a sixth aspect, some embodiments of the present application provide a video decoding apparatus, including:

[0027] an obtaining unit, configured to obtain encoding data of a to-be-decoded video;

[0028] a processing unit, configured to obtain representation information of each video frame of the to-be-decoded video according to the encoding data;

[0029] a reconstructing unit, configured to input the representation information of each video frame of the to-be-decoded video into a student model, to obtain a reconstructed video frame corresponding to each video frame of the to-be-decoded video output by the student model;

[0030] wherein the student model is a model obtained by guiding training of a first network model by a teacher model, the teacher model is a model obtained by training a second network model, and a code rate of an encoding result of the first network model is less than a code rate of an encoding result of the second network model when the first network model and the second network model encode a same video respectively.

[0031] a reconstructing unit, configured to input the encoding data of the to-be-decoded video into a student model, to obtain a reconstructed video corresponding to the to-be-decoded video output by the student model;

[0032] wherein the student model is a model obtained by guiding training of a first network model by a teacher model, the teacher model is a model obtained by training a second network model, and a code rate of an encoding result of the first network model is less than a code rate of an encoding result of the second network model when the first network model and the second network model encode a same video respectively.

[0033] In a seventh aspect, some embodiments of the present application provide an electronic device, comprising: a memory and a processor, the memory storing a computer program, and the processor being configured to implement the video encoding method of the first aspect or the video decoding method of the second aspect when executing the computer program.

[0034] In an eighth aspect, some embodiments of the present application provide a computer readable storage medium, the computer readable storage medium storing a computer program, and the computer program, when executed by a computing device, causing the computing device to implement the video encoding method of the first aspect or the video decoding method of the second aspect.

[0035] In a ninth aspect, some embodiments of the present application provide a chip, the chip comprising a processor and a memory, the memory being configured to store a program or instructions executable on the processor, and the processor being configured to execute the program or instructions to cause the video encoding method of the first aspect or the video decoding method of the second aspect to be executed.

[0036] In a tenth aspect, some embodiments of the present application provide a computer program product, the computer program product, when executed on a computer, causing the computer to implement the video encoding method of the first aspect or the video decoding method of the second aspect.

[0037] According to the above technical solutions, the video encoding method provided by some embodiments encodes a to-be-encoded video by first obtaining a student model and then obtaining an encoding result of the to-be-encoded video according to the student model. Since the student model is obtained by training a first network model, the teacher model is obtained by training a second network model, and the first network model has a smaller code rate than the second network model when encoding the same video based on the first network model and the second network model respectively, the encoding result of the to-be-encoded video obtained according to the student model can have a smaller code rate than the encoding result of the to-be-encoded video obtained according to the teacher model. Moreover, the student model is obtained by training the first network model guided by the teacher model, and the student model can learn more feature information of videos under the guidance of the teacher model, so that the encoding result obtained according to the student model can achieve a similar video reconstruction effect to the encoding result obtained according to the teacher model. Therefore, the video encoding method provided by some embodiments can reduce the code rate of the video encoding result while ensuring the video reconstruction quality. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the implementation in the related art, the drawings required to be used in the description of the embodiments or the related art will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art.

[0039] FIG. 1 shows a structural schematic diagram of a knowledge distillation system according to some embodiments;

[0040] FIG. 2 shows a structural schematic diagram of a deep learning network model in a deep learning-based video coding scheme according to some embodiments;

[0041] FIG. 3 shows a structural schematic diagram of a motion estimation unit according to some embodiments;

[0042] FIG. 4 shows a structural schematic diagram of a motion compression unit according to some embodiments;

[0043] FIG. 5 shows a structural schematic diagram of a motion compensation unit according to some embodiments;

[0044] FIG. 6 shows a structural schematic diagram of a multi-feature fusion module according to some embodiments;

[0045] FIG. 7 shows a step flowchart of a video coding method according to some embodiments;

[0046] FIG. 8 shows a step flowchart of a video decoding method according to some embodiments;

[0047] FIG. 9 shows a step flowchart of a video coding method according to some embodiments;

[0048] FIG. 10 shows a structural schematic diagram of a video coding apparatus according to some embodiments of the present application;

[0049] FIG. 11 shows a structural schematic diagram of a video decoding apparatus according to some embodiments of the present application;

[0050] FIG. 12 shows a structural schematic diagram of a video decoding apparatus according to some embodiments of the present application;

[0051] FIG. 13 shows a hardware structural schematic diagram of an electronic device according to some embodiments of the present application. DETAILED DESCRIPTION

[0052] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the following described embodiments, and in order to make the purpose and embodiments of the present application more clear, the following will clearly and completely describe the exemplary embodiments of the present application by combining the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only some embodiments of the present application, not all embodiments.

[0053] It is not intended to limit the embodiments of the present application. These terms should be understood in their ordinary and customary meaning unless otherwise specified.

[0054] The terms "comprising" and "having" and any variations thereof are intended to cover a non-exclusive inclusion, for example, a product or apparatus that comprises a list of components does not necessarily

[0055] Reference in the specification to "some implementations", "some embodiments", etc. means that a particular feature, structure, or characteristic described is included in some implementations, embodiments, or examples, but not necessarily every implementation, embodiment, or example. Furthermore, this

[0056] Knowledge distillation is a model compression and acceleration technique, the main purpose of which is to migrate the knowledge in a complex and better-performing teacher model to a relatively simple student model, so that the student model approaches or even surpasses the teacher model in performance, while having a smaller model size and faster inference speed.

[0057] Referring to FIG. 1, FIG. 1 is a structural schematic diagram of a knowledge distillation system in some embodiments of the present application. As shown in FIG. 1, the knowledge distillation system includes a teacher model 11 and a student model 12. The teacher model 11 is a model obtained by training a second network model, and the student model 12 is a model obtained by guiding a first network model to train through the teacher model 11. When the same video is encoded based on the first network model and the second network model respectively, the code rate of the encoding result of the first network model is less than the code rate of the encoding result of the second network model, and the reconstructed video quality of the encoding result of the second network model is higher than the reconstructed video quality of the encoding result of the network model.

[0058] The teacher model 11 includes at least one layer structure corresponding to the layer structure in the student model 12. In FIG. 1, the layer structure in the teacher model 11 corresponds to the layer structure in the student model 12 corresponds to the layer structure in the student model 12 corresponds to the layer structure in the student model 12 Corresponding to each other, the layer structure in the teacher model 11 Corresponding to each other, the layer structure in the student model 12 Corresponding to each other, the layer structure in the teacher model 11 Corresponding to each other, the layer structure in the student model 12 s 4 Corresponding to each other, the layer structure in the teacher model 11

[0059] In some embodiments, the teacher model 11 is a model obtained by training a second network model only through sample data.

[0060] In other embodiments, the teacher model 11 is a model obtained by training a second network model through sample data under the guidance of another model. For example, there are three network models with different code rates of encoding results, and the code rate of the encoding result based on network model A is greater than that of the encoding result based on network model B, and the code rate of the encoding result based on network model B is greater than that of the encoding result based on network model C. Network model A can be trained first to obtain video encoding model A, then network model B is trained under the guidance of video encoding model A to obtain the teacher model, and then network model C is trained under the guidance of the teacher model to obtain the student model.

[0061] When the sample video sequence is input into the teacher model 11, the layer structure of the teacher model 11 The features of each video frame of the output sample video sequence are respectively: When the sample video sequence is input into the student model 12, the layer structure of the student model 12 The features of each video frame of the output sample video sequence are respectively: The calculation formula of the distillation loss of the layer structure

[0062] Similarly, the distillation loss of the layer structure and the layer structure The features of each video frame of the output sample video sequence are calculated by the layer structure The distillation loss of the layer structure and the layer structure The features of each video frame of the output sample video sequence are calculated by the layer structure The distillation loss of the layer structure and the layer structure The features of each video frame of the output sample video sequence are calculated by the layer structure ​​​​

[0063] In addition, the sample video sequence is denoted as v1, v2, …, v T The reconstructed video sequence of the sample video sequence output by the student model 12 is denoted as f θ (1), f θ (2), …, f θ (T), then the ground truth loss function for calculating the ground truth loss of the student model 12 can be:

[0064] wherein, ||f θ (t)-v t ||1 is the L1 loss of f θ (t) and v t , SSIM(f θ (t), v t ) is the structural similarity index of f θ (t) and v t .

[0065] It should be noted that in the above embodiment, the ground truth loss of the student model 12 is calculated by the weighted sum of the L1 loss and the SSIM loss between the sample video sequence and the reconstructed video sequence, but the present application is not limited thereto, and the ground truth loss of the student model 12 can also be obtained by other loss functions, for example: the ground truth loss of the student model 12 can also be obtained by the Mean Squared Error (MSE) loss function, the Mean Absolute Error (MAE), the Hinge loss function, etc.

[0066] The mixed loss of the student model 11 can be calculated by the formula:

[0067] wherein, a, b, c, d, e are hyperparameters, and a+b+c+d+e=1.

[0068] The training process of the student model 12 guided by the teacher model 11 is: based on the above process, the mixed loss of the student model 11 is obtained, and the model parameters of the student model 11 are adjusted and optimized based on the mixed loss of the student model 11.

[0069] In some embodiments, the teacher model 11 and / or the student model 12 in the knowledge distillation system shown in FIG. 1 are deep learning network models in a deep learning-based video coding scheme. The deep learning-based video coding scheme encodes the video in the following manner: inputting the video to be coded into the deep learning network model, obtaining the representation information of each video frame of the video to be coded output by the deep learning network model, and encoding the representation information of each video frame of the video to be coded to obtain the encoded data of the video to be coded.

[0070] Referring to FIG. 2, in some embodiments, the deep learning network model in the deep learning based video coding scheme comprises a feature extraction module 21. The feature extraction module 21 is configured to perform feature extraction on a current video frame X t to obtain a feature F t of the current video frame, and perform feature extraction on a reconstructed video frame of a previous video frame of the current video frame to obtain a reference feature

[0071] Referring to FIG. 2, in some embodiments, the deep learning network model in the deep learning based video coding scheme comprises a deformable compensation module 22. The deformable compensation module 22 comprises a motion estimation unit 221, a motion compression unit 222 and a motion compensation unit 223, which are configured to obtain a predicted feature t of the current frame according to the feature F and the reference feature .

[0072] Referring to FIG. 3, in some embodiments, the motion estimation unit 221 comprises a first feature concatenation layer 31, a first convolution layer 32 and a second convolution layer 33 connected in sequence, which are configured to obtain an offset map O t of the current video frame according to the feature F and the reference feature . The first feature concatenation layer 31 is configured to concatenate the feature F t and the reference feature ; the first convolution layer 32 is configured to perform convolution operation on the output feature of the first feature concatenation layer 31, and the second convolution layer 33 is configured to perform convolution operation on the output feature of the first convolution layer 32 to obtain the offset map O t .

[0073] In some embodiments, the first convolution layer 32 and the second convolution layer 33 have 64 output channels, a convolution kernel size of 3*3 and a convolution step of 1.

[0074] Referring to FIG. 4, in some embodiments, the motion compression unit 222 comprises an encoder 41 and a decoder 42, which are configured to encode the offset map O t by the encoder 41, and decode the encoding result by the decoder 42 to obtain a reconstructed offset map The decoder 42 includes multiple cascaded first residual networks 410. Each first residual network 410 includes a third convolutional layer 411, a first residual block 412, a second residual block 413, a third residual block 414, and a first fusion layer 415. The third convolutional layer 411 is used as the input to the first residual network 410, and the input to the first residual block 412 is the output of the third convolutional layer 411. The first residual block 412, the second residual block 413, and the third residual block 414 are cascaded, and the input to the first residual block 412 is the output of the third convolutional layer 411. The first fusion layer 415 is used to add and fuse the output of the third residual block 414 and the output of the third convolutional layer 411 to obtain the output of the first residual network 410. The encoder 42 includes multiple cascaded second residual networks 420. Each second residual network 420 includes a fourth residual block 421, a fifth residual block 422, a sixth residual block 423, a second fusion layer 424, and a fourth convolutional layer 425. The fourth residual block 421, the fifth residual block 422, and the sixth residual block 423 are cascaded. The input of the fourth residual block 421 is the input of the second residual network 420. The second fusion layer 424 is used to add and fuse the output of the sixth residual block 423 and the input of the second residual network 420. The fourth convolutional layer 425 is used to perform a convolution operation on the output of the second fusion layer 424 to obtain the output of the second residual network 420.

[0075] Referring to Figure 5, in some embodiments, the motion compensation unit 223 includes: a deformable convolutional layer 51, a second feature stitching layer 52, a fifth convolutional layer 53, a sixth convolutional layer 54, and a third fusion layer 55, used to reconstruct the offset map. and reference features Obtain the predicted features of the current video frame Deformable convolutional layer 51 is used to reconstruct the offset map The corresponding filter pairs the reference features Perform deformable convolution operation; the second feature stitching layer 52 is used to stitch together the output and reference features of the deformable convolution layer 51. The fifth convolutional layer 53 performs a convolution operation on the output of the second feature concatenation layer 52, the sixth convolutional layer 54 performs a convolution operation on the output of the fifth convolutional layer 53, and the third fusion layer 55 concatenates the output of the sixth convolutional layer 54 and the output of the deformable convolutional layer 51 to obtain the predicted features of the current video frame.

[0076] Referring to FIG. 2, in some embodiments, the deep learning network model in the deep learning based video coding scheme includes a residual calculation module 23 configured to calculate a residual feature R t of the current video frame according to a feature F of the current video frame and a predicted feature t .

[0077] Referring to FIG. 2, in some embodiments, the deep learning network model in the deep learning based video coding scheme includes a residual compression module 24. The residual compression module is configured to compress the residual feature R t by an auto-encoding network to obtain a compressed residual feature

[0078] Referring to FIG. 2, in some embodiments, the deep learning network model in the deep learning based video coding scheme includes an entropy coding module 25 configured to compress an encoding result of the offset map O t output by the motion compression unit and the residual feature R t to obtain encoding data of the current video frame.

[0079] Referring to FIG. 2, in some embodiments, the deep learning network model in the deep learning based video coding scheme includes a fusion module 26 configured to fuse a preset feature of the current video frame and the compressed residual feature to obtain an initial reconstructed feature

[0080] Referring to FIG. 2, in some embodiments, the deep learning network model in the deep learning based video coding scheme includes a multi-frame feature fusion module 27. The multi-frame feature fusion module is configured to fuse the initial reconstructed feature and features of reconstructed video frames of the previous three consecutive video frames of the current video frame to obtain a final reconstructed feature of the current video frame.

[0081] Referring to FIG. 6, in some embodiments, the multi-feature fusion module 27 includes a first deformable compensation module 61, a second deformable compensation module 62, a third deformable compensation module 63, a first non-local attention module 64, a second non-local attention module 65, a third non-local attention module 66, a fourth non-local attention module 67, a third concatenation layer 68, a seventh convolutional layer 69, and a fourth fusion layer 610. The multi-feature fusion module 27 generates the final reconstructed feature of the current video frame The process includes: first, using the first deformable compensation module 61, the second deformable compensation module 62, and the third deformable compensation module 63 to generate the prediction feature of the current video frame respectively Then, through the first non-local attention module 64, the second non-local attention module 65, the third non-local attention module 66, and the fourth non-local attention module 67, the attention feature of the current video frame is generated respectively Then, through the third concatenation layer 68, the outputs of the first non-local attention module 64, the second non-local attention module 65, the third non-local attention module 66, and the fourth non-local attention module 67 are concatenated, and through the seventh convolutional layer 69, the output of the third concatenation layer 68 is convoluted, and finally through the fourth fusion layer 610, the initial reconstructed feature and the output of the seventh convolutional layer 69 are added and fused to obtain the final reconstructed feature of the current video frame

[0082] Referring to FIG. 2, in some embodiments, the deep learning network model in the deep learning-based video coding scheme includes a reconstruction module 28, which is used to reconstruct the final reconstructed feature of the current video frame to obtain the reconstructed video frame of the current video frame

[0083] ​In some embodiments, the teacher model 11 and the student model 12 in the knowledge distillation system shown in FIG. 1 are both models trained on the deep learning model 2. The difference between the teacher model 11 and the student model 12 is that the rate-distortion loss function used in the training of the teacher model 11 is different from that used in the training of the student model 12. The rate-distortion loss function can be expressed as RD = R + λD, where R represents the rate, which is the number of bits required to encode a video signal and is an indicator of the amount of data, usually measured in bits per second (bps), and in video encoding, a higher rate means more bits are used to represent a video frame; D represents the distortion, which is the difference between the reconstructed video after encoding and the original video, reflecting the impact of the encoding process on the quality of the original signal, and a smaller distortion means that the encoded signal is closer to the original signal and has higher quality. λ represents the Lagrange multiplier, which is used to balance the relationship between the rate and the distortion. The value of λ determines the relative importance of the rate and the distortion in the optimization process. When λ is large, more attention is paid to reducing the rate in the optimization process, and a certain degree of distortion increase may be accepted in exchange for a reduction in the rate. Conversely, when λ is small, more attention is paid to reducing the distortion, and a higher rate may be chosen to achieve a lower distortion. When the model structure of the teacher model 11 is the same as that of the student model 12, the value of λ of the teacher model 11 can be smaller than that of the student model 12, so that the distortion of the teacher model 12 is smaller than that of the student model, and the rate of the student model 12 is smaller than that of the teacher model 11.

[0084] In some examples, the teacher model 11 and / or the student model 12 in the knowledge distillation system shown in FIG. 1 are deep learning network models in an implicit neural representation (INR) scheme for video. The implicit neural representation scheme encodes a video in the following way: a deep learning network model is trained on the video to be encoded, and after the training of the deep learning network model is completed, the model parameters of the deep learning network model are encoded to obtain the encoded data of the video to be encoded.

[0085] Implicit neural representation is a method of converting signals into model parameterization of deep learning network model. Traditional signal representation is usually discrete, for example: an image is a discrete grid of pixels, an audio signal is a discrete sample of amplitude, and a three-dimensional shape is usually parameterized as a voxel, a point cloud or a mesh; while INR can parameterize the signal as a continuous function. Since INR can parameterize the signal as a continuous function, the memory required to parameterize the signal is independent of the spatial resolution and is proportional only to the complexity of the hidden signal, and the signal can be sampled at any spatial resolution to obtain more detailed signal details. Neural Representations for Videos (NeRV) is a kind of implicit neural representation scheme for video.

[0086] NeRV represents video as the parameter of neural network, converts video encoding into fitting video into neural network model, converts video compression problem into model compression problem, and can achieve performance comparable to conventional video compression methods. The proposal of NeRV provides a new perspective and solution for video representation, compression and other video related tasks, and compared with traditional video compression methods and learning based video compression methods, this method has a simpler training process and higher decoding speed.

[0087] In NeRV, the time index t of the video frame is learned through a multilayer perceptron (MLP) network to obtain a feature, and then the feature is sent to a subsequent reconstruction network composed of convolution and upsampling modules, and finally a reconstructed frame is obtained. Through this method, the video is fitted into the entire neural network model. Specifically, first, the index t of each time frame is normalized, and the normalized value is processed by a position encoding formula to obtain a set of vectors similar to Fourier transform, which is used to preserve the high frequency information brought by the change of index with time. Then the vector is input as the feature of the current time frame into the subsequent reconstruction network.

[0088] When the teacher model 11 and the student model 12 in the knowledge distillation system shown in FIG. 1 are both models trained by the deep learning network model in the implicit neural representation scheme for video, the model structures of the teacher model 11 and the student model 12 are different, the data amount of the model parameters of the teacher model 11 is greater than that of the student model 12, and the training data of the teacher model 11 and the student model 12 are both actual video to be encoded.

[0089] In some embodiments, when the teacher model 11 and the student model 12 are models trained by deep learning network models in an implicit neural representation (INR) scheme of a video, the deep learning network model used to train the student model 12 can be a network model pruned from a deep learning network model used to train the teacher model 11.

[0090] In some embodiments, the teacher model 11 and the student model 12 in the knowledge distillation system shown in FIG. 1 can be network models in different video coding schemes based on deep learning network. For example, the teacher model 11 is a deep learning network model in a deep learning based video coding scheme, and the student model 12 is a deep learning network model in an implicit neural representation scheme of a video, or the teacher model 11 is a deep learning network model in an implicit neural representation scheme of a video, and the student model 12 is a deep learning network model in a deep learning based video coding scheme. The embodiments of the present application do not limit this, and the teacher model 11 includes at least one layer structure corresponding to the layer structure in the student model 12. The student model 12 can learn the feature details that cannot be learned according to the true value loss according to the at least one layer structure.

[0091] It should be further noted that, in addition to being a deep learning network model in a deep learning based video coding scheme and a deep learning network model in an implicit neural representation scheme of a video, the teacher model 11 and / or the student model 12 in the knowledge distillation system shown in FIG. 1 can also be a network model in other video coding schemes based on deep learning network for video coding. The embodiments of the present application do not limit this.

[0092] The embodiments of the present application provide a video coding method. Referring to FIG. 7, the video coding method includes the following steps:

[0093] S701, obtaining a student model.

[0094] The student model is a model obtained by guiding a first network model training by a teacher model, the teacher model is a model obtained by training a second network model, and when the same video is encoded based on the first network model and the second network model respectively, the code rate of the encoding result of the first network model is less than the code rate of the encoding result of the second network model.

[0095] When the first network model and the second network model are deep learning network models in a deep learning based video coding scheme, the implementation of obtaining the student model can include the following steps a and b:

[0096] Step a, training the second network model based on the first training dataset to obtain the teacher model.

[0097] In some embodiments, the first training dataset is a dataset composed of a plurality of sample video sequences selected from scikit-video.

[0098] In some embodiments, the training of the second network model based on the first training dataset comprises steps a1 to a3 as follows:

[0099] Step a1, inputting each sample video sequence in the training dataset into the second network model and obtaining a reconstructed video sequence corresponding to each sample video sequence output by the second network model.

[0100] Step a2, calculating a loss value corresponding to each sample video sequence according to each sample video sequence, the reconstructed video sequence corresponding to each sample video sequence, and a loss function.

[0101] Step a3, adjusting the model parameters of the second network model according to the loss value corresponding to each sample video sequence.

[0102] In some embodiments, the loss function for training the teacher model is a loss function obtained by combining an L1 loss function and a Structural Similarity Index (SSIM) loss function, which is specifically as follows:

[0103] wherein T is the number of video frames constituting the sample video sequence, f θ (t) is a reconstructed video frame corresponding to the tth video frame in the sample video sequence, v t is the tth video frame in the sample video sequence, and a is a hyperparameter, ||f θ (t) - v t ||1 is the L1 loss of f θ (t) and v t , and SSIM(f θ (t), v t ) is the structural similarity index of f θ (t) and v t .

[0104] In some embodiments, a is 0.7. That is, the weights of the L1 loss and the SSIM loss are 0.7 and 0.3, respectively.

[0105] Step b, training the first network model based on the second training dataset under the guidance of the teacher model to obtain the student model.

[0106] The second training data set is also composed of a plurality of sample video sequences selected from scikit-video, and the first training data set and the second training data set can be the same or different.

[0107] In some embodiments, under the guidance of the teacher model, the first network model is trained based on the second training data set, including steps b1 to b3 as follows:

[0108] Step b1, obtaining a first feature set and a second feature set according to a sample video sequence.

[0109] The sample video sequence is any video sequence in the second training data set, the first feature set is a set composed of features of each video frame of the sample video sequence output by a target layer structure of the first network model, and the second feature set is a set composed of features of each video frame of the sample video sequence output by a layer structure corresponding to the target layer structure in the teacher model.

[0110] In some embodiments, the target layer structure is one layer structure of the first network model except the last layer structure.

[0111] In other embodiments, the target layer structure is the last layer structure of the first network model. When the target layer structure is the last layer structure of the first network model, the first feature set is a set composed of reconstructed video frames of each video frame of the sample video sequence output by the first network model.

[0112] Step b2, calculating a distillation loss corresponding to the target layer structure according to the first feature set, the second feature set, and a distillation loss function.

[0113] In some embodiments, the distillation loss function is:

[0114] wherein T is the number of video frames constituting the sample video sequence, is a feature of the tth video frame of the sample video sequence output by the target layer structure, is a feature of the tth video frame of the sample video sequence output by the layer structure corresponding to the target layer structure in the teacher model, and L distill is the distillation loss corresponding to the target layer structure.

[0115] Step b3, adjusting model parameters of the first network model based on the distillation loss corresponding to the target layer structure.

[0116] In some embodiments, when guiding the first network model training by the teacher model, only the distillation loss of one layer structure (the distillation loss of the target layer structure of the first network model) is obtained, and the model parameters of the first network model are adjusted based on the distillation loss corresponding to the target layer structure, including the following steps b31 to b34:

[0117] Step b31, obtaining the reconstructed video sequence corresponding to the sample video sequence.

[0118] Wherein, the reconstructed video sequence is a video obtained by video reconstruction according to the encoding data of the sample video sequence, which is obtained by encoding the sample video based on the first network model.

[0119] That is, the reconstructed video sequence is the output of the first network model when the sample video sequence is taken as the input of the first network model.

[0120] In some embodiments, the output of the target layer structure of the first network model and the output of the last layer structure of the first network model can be obtained at the same time when the sample video sequence is taken as the input of the first network model, so as to obtain the first feature set and the reconstructed video sequence.

[0121] Step b32, obtaining the ground truth loss according to the sample video sequence, the reconstructed video sequence corresponding to the sample video sequence, and the ground truth loss function.

[0122] In some embodiments, the ground truth loss function is:

[0123] Wherein, T is the number of video frames constituting the sample video sequence, f θ (t) is the reconstructed video frame corresponding to the tth video frame in the sample video sequence, v t is the tth video frame in the sample video sequence, and a is a hyperparameter, ||f θ (t)-v t ||1 is the L1 loss of f θ (t) and v t , SSIM(f θ (t), v t ) is the structural similarity index of f θ (t) and v t , and L org is the ground truth loss.

[0124] That is, the ground truth loss function is the same as the loss function of the training teacher model.

[0125] In some embodiments, the true value loss function can also be different from the loss function for training the teacher model.

[0126] Step b33, performing weighted summation on the distillation loss corresponding to the target layer structure and the true value loss to obtain a mixed loss.

[0127] The distillation loss corresponding to the target layer structure is denoted as L distill , the true value loss is denoted as L org , and the mixed loss is denoted as L, then when the weight of the true value loss is β, the calculation formula of the mixed loss can be as follows: L = β * L org + (1-β) L distill

[0128] In some embodiments, the weight values of the distillation loss and the true value loss are equal, both being 0.5. That is, β = 0.5.

[0129] Step b34, adjusting the model parameters of the first network model according to the mixed loss.

[0130] In some embodiments, when the first network model is trained by the teacher model, the distillation loss of multiple layer structures of the first network model is obtained, and the model parameters of the first network model are adjusted based on the distillation loss corresponding to the target layer structure, including the following steps b35 to b38:

[0131] Step b35, taking a reconstructed video sequence corresponding to the sample video sequence.

[0132] Wherein, the reconstructed video sequence is a video obtained by video reconstruction based on the encoding data of the sample video sequence, which is obtained by encoding the sample video based on the first network model.

[0133] Step b36, obtaining a true value loss according to the sample video sequence, the reconstructed video sequence corresponding to the sample video sequence, and a true value loss function.

[0134] The implementation manners of step b35 and step b36 can be respectively referred to the above-mentioned step b31 and step b32, and will not be repeated here to avoid redundancy.

[0135] Step b37, performing weighted summation on the distillation loss corresponding to the target layer structure, the distillation loss corresponding to at least one layer structure other than the target layer structure in the first network model, and the true value loss to obtain a mixed loss.

[0136] The implementation of obtaining the distillation loss corresponding to at least one layer structure other than the target layer structure in the first network model can refer to the implementation of obtaining the distillation loss corresponding to the target layer structure in the above-mentioned embodiments, except that the feature set obtained according to the sample video sequence is different when obtaining the distillation loss corresponding to different layer structures.

[0137] For example, when the first network model is trained by the teacher model, the distillation loss of four layer structures of the first network model is obtained, and the distillation loss of the four layer structures of the first network model is respectively The calculation formula of the mixed loss can be as follows:

[0138] Wherein, a, b, c, d, e are hyperparameters, and a+b+c+d+e=1.

[0139] In some embodiments, a=0.5, b+c+d+e=0.5.

[0140] Step b38, adjusting the model parameters of the first network model according to the mixed loss.

[0141] When the first network model and the second network model are deep learning network models in the implicit neural representation scheme of the video, the implementation of obtaining the student model can include the following steps 1 and 2:

[0142] Step 1, training the second network model based on the to-be-encoded video to obtain the teacher model.

[0143] In some embodiments, training the second network model based on the to-be-encoded video includes the following steps 11 to 13:

[0144] Step 11, taking the frame index of each video frame of the to-be-encoded video as the input of the second network model, and obtaining the reconstructed video sequence corresponding to the to-be-encoded video output by the second network model.

[0145] Step 12, calculating the loss value according to the to-be-encoded video, the reconstructed video sequence, and the loss function.

[0146] In some embodiments, the loss function is:

[0147] Wherein, T is the number of video frames constituting the sample video sequence, f θ (t) is the reconstructed video frame corresponding to the tth video frame in the sample video sequence, v t is the tth video frame in the sample video sequence, and a is a hyperparameter. ||f θ (t)-v t ||1 is fθ (t) and v t L1 loss of f θ (t), v t ) is a structural similarity index of f θ (t) and v t .

[0148] Step 13, adjusting according to the model parameters of the second network model.

[0149] In some embodiments, the above steps 11 to 13 can be repeated for a preset number of times to obtain the teacher model. For example, the preset number of times can be 300 times.

[0150] Step 2, training the first network model based on the to-be-encoded video under the guidance of the teacher model to obtain the student model.

[0151] The implementation of training the first network model based on the to-be-encoded video under the guidance of the teacher model can refer to the implementation of training the first network model based on the second training data set under the guidance of the teacher model, and the difference is only that the training data is different.

[0152] S702, obtaining an encoding result of the to-be-encoded video according to the student model.

[0153] When the first network model and the second network model are deep learning network models in a deep learning-based video encoding scheme, obtaining the encoding result of the to-be-encoded video according to the student model includes:

[0154] inputting the to-be-encoded video into the student model, obtaining representation information of each video frame of the to-be-encoded video output by the student model, and encoding the representation information of each video frame of the to-be-encoded video to obtain the encoding result of the to-be-encoded video.

[0155] In some embodiments, the representation information of the to-be-encoded video output by the student model includes motion features and residual features of each video frame of the to-be-encoded video.

[0156] When the first network model and the second network model are deep learning network models in a video implicit neural representation scheme, obtaining the encoding result of the to-be-encoded video according to the student model includes:

[0157] encoding the student model to obtain the encoding result of the to-be-encoded video.

[0158] In some embodiments, the student model is encoded to obtain the encoding result of the to-be-encoded video, including: encoding the model parameters of the student model to obtain the encoding result of the to-be-encoded video.

[0159] In some embodiments, the student model is encoded to obtain the encoding result of the to-be-encoded video, including: encoding the model parameters and the model structure of the student model to obtain the encoding result of the to-be-encoded video.

[0160] The video encoding method provided by the above embodiments first obtains a student model, and then obtains the encoding result of the to-be-encoded video according to the student model. Since the student model is a model obtained by training a first network model, the teacher model is a model obtained by training a second network model, and the first network model has a smaller code rate than the second network model when encoding the same video based on the first network model and the second network model respectively, the code rate of the encoding result of the to-be-encoded video can be reduced by obtaining the encoding result of the to-be-encoded video according to the student model compared with obtaining the encoding result of the to-be-encoded video according to the teacher model. Since the student model is a model obtained by training the first network model guided by the teacher model, the student model can learn more feature information of the video under the guidance of the teacher model, so that the encoding result obtained according to the student model can achieve a similar video reconstruction effect to the encoding result obtained according to the teacher model. Therefore, the video encoding method provided by some embodiments of the present application can reduce the code rate of the video encoding result while ensuring the video reconstruction quality.

[0161] When the first network model and the second network model are deep learning network models in the implicit neural representation scheme of the video, referring to FIG. 8, the video decoding method corresponding to the video encoding method provided by the above embodiments includes the following steps:

[0162] S801, obtaining the encoding data of the to-be-decoded video.

[0163] As described above, when the first network model and the second network model are deep learning network models in the implicit neural representation scheme of the video, the encoding data of the to-be-decoded video is the encoding data obtained by encoding the model parameters of the student network model.

[0164] In some embodiments, the encoding data of the to-be-encoded video is obtained by receiving the encoding data of the to-be-decoded video sent by the media resource server.

[0165] S802, constructing a student model according to the encoding data of the to-be-decoded video.

[0166] The student model is a model obtained by guiding the first network model to train, and the teacher model is a model obtained by training the second network model. When the same video is encoded based on the first network model and the second network model respectively, the code rate of the encoding result of the first network model is less than the code rate of the encoding result of the second network model.

[0167] Since the encoding data of the to-be-decoded video is encoding data obtained by encoding the model parameters of the student network model, the model parameters of the student network model can be obtained by decoding the encoding data of the to-be-decoded video, and therefore the student model can be constructed according to the encoding data of the to-be-decoded video.

[0168] S803, obtaining a reconstructed video corresponding to the to-be-decoded video according to the student model.

[0169] In some embodiments, obtaining the reconstructed video corresponding to the to-be-decoded video according to the student model includes: inputting the frame index of each video frame of the to-be-decoded video into the student model one by one, and obtaining the reconstructed video frame corresponding to each video frame of the to-be-decoded video output by the student model.

[0170] When the first network model and the second network model are deep learning network models in the implicit neural representation scheme of the video, referring to FIG. 9, the video decoding method corresponding to the video encoding method provided in the above embodiments includes the following steps:

[0171] S901, obtaining encoding data of a to-be-decoded video.

[0172] As described above, when the first network model and the second network model are deep learning network models in the implicit neural representation scheme of the video, the encoding data of the to-be-decoded video is encoding data obtained by encoding the representation information of each video frame of the to-be-decoded video output by the student network model.

[0173] Similarly, obtaining the encoding data of the to-be-encoded video can include: receiving the encoding data of the to-be-decoded video sent by the media resource server.

[0174] S902, obtaining representation information of each video frame of the to-be-decoded video according to the encoding data.

[0175] In some embodiments, obtaining the representation information of each video frame of the to-be-decoded video according to the encoding data can include: performing entropy decoding, inverse quantization, and the like on the encoding data to obtain the representation information of each video frame.

[0176] S903, input the representation information of each video frame of the to-be-decoded video into the student model to obtain a reconstructed video frame corresponding to each video frame of the to-be-decoded video output by the student model.

[0177] The student model is a model obtained by guiding training of a first network model by a teacher model, and the teacher model is a model obtained by training a second network model. When the same video is encoded based on the first network model and the second network model respectively, a code rate of an encoding result of the first network model is less than a code rate of an encoding result of the second network model.

[0178] In some embodiments, the student model can be integrated in a video player, so that the video playing device can obtain the student model and perform the step S903.

[0179] In some other embodiments, model parameters of the student model can be encoded, and an encoding result of the model parameters of the student model can be added to encoding data of the video, so that the video playing device can construct the student model according to the model parameters of the student model and perform the step S903.

[0180] Peak Signal-to-Noise Ratio (PSNR) is used as an evaluation index for evaluating the quality of the reconstructed video. NeRV-L is used as the second network model for training the teacher model, and NeRV-S is used as the first network model for training the student model. The code rates and PSNRs of the encoding results of the teacher model, the student model obtained by guiding training of the NeRV-S by the teacher model, and the original model obtained by directly training the first network model by the NeRV-S are shown in Table 1.

[0181] Table 1

[0182] The code rate unit is bits per pixel (bpp), and the calculation formula of the encoding result of each model is: modelparameter*(1-modelSparsity)*QuantBit / pixelNum.

[0183] The model parameter is the data amount of the model, the modelSparsity is the sparsity of the pruned model, the QuantBit is the quantization bit, and the pixelNum is the total pixel number of the video.

[0184] Compared with the code rate (0.2 bpp) of the encoding result of the teacher model in Table 1 and the code rate (0.05 bpp) of the encoding result of the student model, it can be obtained that the video encoding using the student model can reduce the code rate of the encoding result. Compared with the code rate (0.05 bpp) of the encoding result of the student model, the code rate (0.05 bpp) of the encoding result of the original model, the PSNR (32.64) of the encoding result of the student model, and the PSNR (32.11) of the encoding result of the original model in Table 1, it can be obtained that the student model obtained in the video encoding method provided by the embodiment of the present application can improve the quality of the reconstructed video under the same code rate compared with directly training the network model to obtain the video encoding model.

[0185] Referring to FIG. 10, some embodiments of the present application provide a video encoding apparatus, the video encoding apparatus 100 can include:

[0186] The model training unit 101 is configured to obtain a student model, the student model is a model obtained by guiding training of a first network model by a teacher model, the teacher model is a model obtained by training a second network model, and the code rate of the encoding result of the first network model is less than the code rate of the encoding result of the second network model when the same video is encoded based on the first network model and the second network model respectively.

[0187] The video encoding unit 102 is configured to obtain an encoding result of a to-be-encoded video according to the student model.

[0188] In some embodiments, the video encoding unit 102 is specifically configured to encode the model parameters of the student model to obtain the encoding result of the to-be-encoded video.

[0189] In some embodiments, the video encoding unit 102 is specifically configured to input the to-be-encoded video into the student model and obtain representation information of each video frame of the to-be-encoded video output by the student model; and encode the representation information of each video frame of the to-be-encoded video to obtain the encoding result of the to-be-encoded video.

[0190] In some embodiments, the model training unit is specifically configured to obtain a first feature set and a second feature set according to a sample video sequence; the first feature set is a set of features of each video frame of the sample video sequence output by a target layer structure of the first network model, and the second feature set is a set of features of each video frame of the sample video sequence output by a layer structure corresponding to the target layer structure in the teacher model; a distillation loss corresponding to the target layer structure is calculated according to the first feature set, the second feature set, and a distillation loss function; and model parameters of the first network model are adjusted based on the distillation loss corresponding to the target layer structure.

[0191] In some embodiments, the distillation loss function is:

[0192] wherein T is a number of video frames constituting the sample video sequence, is a feature of a tth video frame of the sample video sequence output by the target layer structure, is a feature of the tth video frame of the sample video sequence output by the layer structure corresponding to the target layer structure in the teacher model, and L distill is the distillation loss corresponding to the target layer structure.

[0193] In some embodiments, the model training unit is specifically configured to obtain a reconstructed video sequence corresponding to the sample video sequence; the reconstructed video sequence is a video obtained by video reconstruction based on encoded data of the sample video sequence obtained by encoding the sample video based on the first network model; a true value loss is obtained according to the sample video sequence, the reconstructed video sequence corresponding to the sample video sequence, and a true value loss function; a mixed loss is obtained by weighted summation of the distillation loss corresponding to the target layer structure and the true value loss; and model parameters of the first network model are adjusted according to the mixed loss.

[0194] In some embodiments, the model training unit is specifically configured to obtain a reconstructed video sequence corresponding to the sample video sequence; the reconstructed video sequence is a video obtained by video reconstruction based on encoded data of the sample video sequence obtained by encoding the sample video based on the first network model; a true value loss is obtained according to the sample video sequence, the reconstructed video sequence corresponding to the sample video sequence, and a true value loss function; a mixed loss is obtained by weighted summation of the distillation loss corresponding to the target layer structure, a distillation loss corresponding to at least one layer structure other than the target layer structure in the first network model, and the true value loss; and model parameters of the first network model are adjusted according to the mixed loss.

[0195] In some embodiments, the ground truth loss function is:

[0196] where T is the number of video frames that make up the sample video sequence, f θ (t) is a reconstructed video frame corresponding to the t-th video frame in the sample video sequence, v t is the t-th video frame in the sample video sequence, a is a hyperparameter, and ||f θ (t) - v t ||1 is the L1 loss of f θ (t) and v t , SSIM(f θ (t), v t ) is the structural similarity index of f θ (t) and v t , and L org is the ground truth loss.

[0197] The video encoding apparatus provided in the above examples can perform the video encoding method provided in any of the above examples, and the implementation principle and the achieved technical effects are the same. To avoid redundancy, no further details are provided.

[0198] Referring to FIG. 11, some embodiments of the present application provide a video decoding apparatus 110, which can include:

[0199] An acquisition unit 111, configured to acquire encoding data of a video to be decoded;

[0200] A construction unit 112, configured to construct a student model according to the encoding data of the video to be decoded, the student model being a model obtained by guiding training of a first network model by a teacher model, the teacher model being a model obtained by training a second network model, and a code rate of an encoding result of the first network model being less than a code rate of an encoding result of the second network model when the same video is encoded based on the first network model and the second network model respectively.

[0201] A reconstruction unit 113, configured to acquire a reconstructed video corresponding to the video to be decoded according to the student model.

[0202] Referring to FIG. 12, some embodiments of the present application further provide another video decoding apparatus 120, which can include:

[0203] An acquisition unit 121, configured to acquire encoding data of a video to be decoded;

[0204] A processing unit 122, configured to acquire representation information of each video frame of the video to be decoded according to the encoding data.

[0205] a reconstruction unit 123, configured to input the representation information of each video frame of the to-be-decoded video into the student model to obtain a reconstructed video frame corresponding to each video frame of the to-be-decoded video output by the student model;

[0206] The student model is a model obtained by guiding training of a first network model by a teacher model, and the teacher model is a model obtained by training a second network model. When the same video is encoded based on the first network model and the second network model respectively, a code rate of an encoding result of the first network model is less than a code rate of an encoding result of the second network model.

[0207] The video decoding apparatus provided in the above examples can perform the video decoding method provided in the above examples, and the implementation principle and the achieved technical effects are the same. To avoid redundancy, no longer detailed description.

[0208] Some embodiments of the present application provide an electronic device, which can include a memory and a processor, the memory having stored therein a computer program, and the processor configured to implement the video encoding method according to any one of the above embodiments or the video decoding method according to any one of the above embodiments when executing the computer program.

[0209] FIG. 13 exemplarily shows a configuration block diagram of an electronic device 1300 in some embodiments described above. As shown in FIG. 13, the display device 1300 can include at least one of a tuner and demodulator 131, a communicator 132, a detector 133, an external device interface 134, a processor 135, a display 136, an audio output interface 137, a memory 138, a power supply 139, and a user interface.

[0210] In some embodiments, the processor 135 includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), a RAM (Random Access Memory), a ROM (Read-Only Memory), a first interface to an n-th interface available for input / output, a communication bus, and the like.

[0211] The display 136 includes a display screen component available for presenting a picture, and a driving component for driving image display, a component available for receiving an image signal output from the processor, and performing display video content, image content, and a menu control interface, and a user control UI interface.

[0212] The display 136 can be a liquid crystal display, an organic light emitting diode (OLED) display, a projection display, and the like.

[0213] The communicator 132 is a component for communicating with external devices or servers according to various communication protocol types. For example, the communicator can include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module, and other network communication protocol chips or near field communication protocol chips, and an infrared receiver. The electronic device 1300 can perform transmission and reception of control signals and data signals with a control device or a server through the communicator 132.

[0214] The user interface can be used to receive control signals input by a user through a control device (e.g., an infrared remote controller, etc.) or touch or gestures, etc.

[0215] The detector 133 can be used to collect signals of an external environment or interaction with the outside. For example, the detector 133 includes a light receiver, which can be used to collect an ambient light intensity sensor; or the detector 133 includes an image collector, such as a camera, which can be used to collect an external environment scene, a user's attribute, or a user interaction gesture; or the detector 133 includes a sound collector, such as a microphone, etc., for receiving external sound.

[0216] The external device interface 134 can include, but is not limited to, any one or more of the following: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (Universal Serial Bus, USB), an RGB (Red, Green, Blue) port, and the like. It can also be a composite input / output interface formed by a plurality of the above interfaces.

[0217] The tuner demodulator 131 receives broadcast television signals through wired or wireless reception, and demodulates audio and video signals and EPG data signals from a plurality of wireless or wired broadcast television signals. In some embodiments, the processor 135 and the tuner demodulator 131 can be located in different split devices, i.e., the tuner demodulator 131 can also be in an external device of the main device where the processor 135 is located, such as an external set-top box, etc.

[0218] The processor 135 controls the operation of the display device and the response to the user's operation by storing various software control programs in the memory. The processor 135 controls the overall operation of the electronic device. For example, in response to receiving a user command for selecting a UI object displayed on the display 136, the processor 135 can perform an operation related to the object selected by the user command.

[0219] In some embodiments, the processor 135 includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), a RAM (Random Access Memory), a ROM (Read-Only Memory), a first to an n-th interface for input / output, a communication bus, and the like.

[0220] Some embodiments of the present application provide a computer readable storage medium having stored thereon a computer program, the computer program, when executed by a computing device, causing the computing device to implement the video encoding method of any of the above embodiments or the video decoding method of any of the above embodiments.

[0221] Some embodiments of the present application provide a chip, the chip including a processor and a memory, the memory being configured to store a program or an instruction executable on the processor, and the processor being configured to execute the program or the instruction, so that the video encoding method of any of the above embodiments or the video decoding method of any of the above embodiments is executed.

[0222] Some embodiments of the present application provide a computer program product, the computer program product, when executed on a computer, causing the computer to implement the video encoding method of any of the above embodiments or the video decoding method of any of the above embodiments.

[0223] Finally, it should be noted that the above embodiments are merely used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0224] For the sake of explanation, the foregoing descriptions have been presented in terms of specific embodiments. However, it is to be appreciated that specific embodiments described herein are not intended to limit the scope of the present application, which is defined with reference to the following claims. Various modifications and changes can be made thereto by those skilled in the art which fall within the scope of the present application as defined by the following claims. The embodiments were chosen and described in order to explain the principles of the application and the practical application and to enable others skilled in the art to understand for implementing various embodiments and with various modifications as are suited to the particular use contemplated.

Claims

1. A method of video coding, the method comprising: The method comprises the following steps: obtaining a student model; the student model is a model obtained by guiding the training of a first network model by a teacher model, the teacher model is a model obtained by training a second network model, and when the same video is encoded based on the first network model and the second network model respectively, the code rate of the encoding result of the first network model is smaller than the code rate of the encoding result of the second network model; obtaining an encoding result of a to-be-encoded video according to the student model.

2. The method of claim 1, wherein, The step of obtaining the encoding result of the to-be-encoded video according to the student model comprises the following steps: encoding the student model to obtain the encoding result of the to-be-encoded video.

3. The method of claim 1, wherein, The step of obtaining the encoding result of the to-be-encoded video according to the student model comprises the following steps: inputting the to-be-encoded video into the student model and obtaining representation information of each video frame of the to-be-encoded video output by the student model; encoding the representation information of each video frame of the to-be-encoded video to obtain the encoding result of the to-be-encoded video.

4. The method of claim 1, wherein, The step of guiding the training of the first network model by the teacher model comprises the following steps: obtaining a first feature set and a second feature set according to a sample video sequence; the first feature set is a set composed of features of each video frame of the sample video sequence output by a target layer structure of the first network model, and the second feature set is a set composed of features of each video frame of the sample video sequence output by a layer structure corresponding to the target layer structure in the teacher model; calculating a distillation loss corresponding to the target layer structure according to the first feature set, the second feature set and a distillation loss function; adjusting model parameters of the first network model based on the distillation loss corresponding to the target layer structure.

5. The method of claim 4, wherein, The distillation loss function is: wherein T is the number of video frames that make up the sample video sequence, a feature of a t-th video frame of the sample video sequence output for the target layer structure, L is a feature of a t-th video frame of the sample video sequence output by a layer structure in the teacher model corresponding to the target layer structure distill is a distillation loss corresponding to the target layer structure.

6. The method of claim 4, wherein, The step of adjusting the model parameters of the first network model based on the distillation loss corresponding to the target layer structure comprises the following steps: obtaining a reconstructed video sequence corresponding to the sample video sequence; the reconstructed video sequence is a video obtained by video reconstruction according to encoding data of the sample video sequence, the encoding data being obtained by encoding the sample video based on the first network model; obtaining a true value loss according to the sample video sequence, the reconstructed video sequence corresponding to the sample video sequence and a true value loss function; performing weighted summation on the distillation loss corresponding to the target layer structure and the true value loss to obtain a hybrid loss; adjusting the model parameters of the first network model according to the hybrid loss.

7. The method of claim 4, wherein, The step of adjusting the model parameters of the first network model based on the distillation loss corresponding to the target layer structure comprises the following steps: obtaining a reconstructed video sequence corresponding to the sample video sequence; the reconstructed video sequence is a video obtained by video reconstruction according to encoding data of the sample video sequence, the encoding data being obtained by encoding the sample video based on the first network model; obtaining a true value loss according to the sample video sequence, the reconstructed video sequence corresponding to the sample video sequence and a true value loss function; The distillation loss corresponding to the target layer structure, the distillation loss corresponding to at least one layer structure except the target layer structure in the first network model, and the true value loss are weighted and summed to obtain a mixed loss; Model parameters of the first network model are adjusted according to the mixed loss.

8. The method according to claim 6 or 7, characterized in that, The true value loss function is: where T is the number of video frames that make up the sample video sequence, f θ (t) is the reconstructed video frame corresponding to the t-th video frame in the sample video sequence, v t is the t-th video frame in the sample video sequence, a is a hyperparameter, and ||f θ (t) - v t ||1 is the L1 loss between f θ (t) and v t , SSIM(f θ (t), v t ) is the structural similarity index between f θ (t) and v t , and L org is the true value loss.

9. A method of video decoding, comprising: The method comprises the following steps: obtaining encoding data of a to-be-decoded video; constructing a student model according to the encoding data of the to-be-decoded video, the student model being a model obtained by training a first network model guided by a teacher model, the teacher model being a model obtained by training a second network model, and a code rate of an encoding result of the first network model being smaller than a code rate of an encoding result of the second network model when the same video is encoded based on the first network model and the second network model respectively; obtaining a reconstructed video corresponding to the to-be-decoded video according to the student model.

10. A method of video decoding, comprising: The method comprises the following steps: obtaining encoding data of a to-be-decoded video; obtaining representation information of each video frame of the to-be-decoded video according to the encoding data; inputting the representation information of each video frame of the to-be-decoded video into a student model to obtain a reconstructed video frame corresponding to each video frame of the to-be-decoded video output by the student model; wherein the student model is a model obtained by training a first network model guided by a teacher model, the teacher model is a model obtained by training a second network model, and a code rate of an encoding result of the first network model is smaller than a code rate of an encoding result of the second network model when the same video is encoded based on the first network model and the second network model respectively.

11. A video encoding apparatus, comprising: The method comprises the following steps: a model training unit is configured to obtain a student model; the student model is a model obtained by training a first network model guided by a teacher model, the teacher model is a model obtained by training a second network model, and a code rate of an encoding result of the first network model is smaller than a code rate of an encoding result of the second network model when the same video is encoded based on the first network model and the second network model respectively; a video encoding unit is configured to obtain an encoding result of a to-be-encoded video according to the student model.

12. A video decoding apparatus, comprising: The method comprises the following steps: an obtaining unit is configured to obtain encoding data of a to-be-decoded video; a constructing unit is configured to construct a student model according to the encoding data of the to-be-decoded video, the student model being a model obtained by training a first network model guided by a teacher model, the teacher model being a model obtained by training a second network model, and a code rate of an encoding result of the first network model being smaller than a code rate of an encoding result of the second network model when the same video is encoded based on the first network model and the second network model respectively; a reconstructing unit is configured to obtain a reconstructed video corresponding to the to-be-decoded video according to the student model.

13. A video decoding apparatus, comprising: The method comprises the following steps: an obtaining unit is configured to obtain encoding data of a to-be-decoded video; a processing unit is configured to obtain representation information of each video frame of the to-be-decoded video according to the encoding data. The reconstruction unit is configured to input the representation information of each video frame of the to-be-decoded video into the student model to obtain a reconstructed video frame corresponding to each video frame of the to-be-decoded video output by the student model. The student model is a model obtained by guiding training of a first network model by a teacher model, and the teacher model is a model obtained by training a second network model. When the same video is encoded based on the first network model and the second network model respectively, a code rate of an encoding result of the first network model is less than a code rate of an encoding result of the second network model.

14. An electronic device, comprising: The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1-10. The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1-10.

15. A computer-readable storage medium, characterized in that, The chip includes a processor and a memory. The memory is configured to store a program or an instruction executable on the processor. The processor is configured to execute the program or the instruction to cause the method according to any one of claims 1-10 to be executed.

16. A chip, characterized by ​

Images