Model training method, video encoding method, and video decoding method

Abstract

Embodiments of the present application provide a model training method, a video encoding method, and a decoding method. The model training method includes: obtaining a reference sample frame and a plurality of consecutive sample frames to be encoded; generating a reconstructed sample frame by transforming the reference sample frame through a generator in an initial generation model; inputting the reconstructed sample frame and the corresponding sample frame to be encoded into a first discriminator in the initial generation model to obtain a first discrimination result; combining the sample frames to be encoded in timestamp order to obtain a combined encoded sample frame; combining the reconstructed sample frames to obtain a combined reconstructed sample frame; inputting the combined encoded sample frame and the combined reconstructed sample frame into a second discriminator in the initial generation model to obtain a second discrimination result; obtaining an adversarial loss value based on the first discrimination result and the second discrimination result; and training the initial generation model based on the adversarial loss value. The present application enables the reconstructed video frame sequence and the video frame sequence to be encoded to maintain consistency in the time domain, thereby improving the reconstruction quality.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of computer technology, and more specifically, to a model training method, a video encoding method, and a decoding method.

Background Art

[0002] This application was filed with the China Patent Office on June 23, 2022, and claims the priority of Chinese Patent Application No. 202210716223.2, titled "Model Training Method, Video Encoding Method, and Video Decoding Method", the entire content of which is incorporated herein by reference.

[0003] Video encoding and decoding are the keys to realizing video conferencing, live video broadcasting, etc. With the continuous development of machine learning, codec methods based on deep video generation can be used to perform video (especially face video) encoding and decoding operations. This method mainly uses a generator and a neural network model in a generative model to deform a reference frame based on the movement of the frame to be encoded, and generate a reconstructed frame corresponding to the frame to be encoded.

[0004] During the model training stage, the above-mentioned generative model is usually an adversarial generative network including a generator and a discriminator. During training, the video frame to be encoded and the reconstructed video frame generated by the generator are input to a discriminator that performs authenticity discrimination and outputs a discrimination result. Then, a loss function is constructed based on the discrimination result to complete the model training.

[0005] However, in the related art, when the discriminator performs authenticity discrimination, only the similarity between the reconstructed video frame in the spatial domain and the video frame to be encoded is considered, that is, only the similarity between a single reconstructed video frame and the corresponding video frame to be encoded is compared. When reconstructing a video frame using the above-mentioned generation model, the resulting reconstructed video frame sequence (reconstructed video clip) usually has visual flicker and floating artifact phenomena, and the quality of video reconstruction is relatively low.

Summary of the Invention

[0006] Based on the above, the embodiments of the present application provide a model training method, a video encoding method, and a decoding method for at least partially solving the above problems.

[0007] According to a first aspect of the embodiments of the present specification, a model training method is provided, and the model training method includes: obtaining a reference sample frame and a plurality of consecutive sample frames to be encoded; using a generator in an initial generation model to deform the reference sample frame to generate a reconstructed sample frame corresponding to each of the sample frames to be encoded; inputting each reconstructed sample frame and the corresponding sample frame to be encoded into a first discriminator in the initial generation model to obtain a first discrimination result; according to the time stamp order, combining all the sample frames to be encoded to obtain a combined sample frame to be encoded, combining all the reconstructed sample frames to obtain a combined and reconstructed sample frame, and inputting the combined sample frame to be encoded and the combined and reconstructed sample frame into a second discriminator of the initial generation model to obtain a second discrimination result; Obtaining a adversarial loss value based on the first identification result and the second identification result, and training the initial generation model based on the adversarial loss value to obtain a trained generation model.

[0008] According to a second aspect of the embodiments of this specification, a video decoding method is provided. The video decoding method includes: Obtaining and decoding a video bitstream to obtain a reference video frame and features of an object to be encoded; Extracting features from the reference video frame to obtain reference features, and performing motion estimation based on the features of the object to be encoded and the reference features to obtain a motion estimation result; Generating a reconstructed video frame by transforming the reference video frame using a generator in a pre-trained generation model based on the motion estimation result. The generation model is obtained by using the model training method according to the first aspect.

[0009] According to a third aspect of the embodiments of the present application, a video decoding method applied to a conference terminal device is provided. This method includes: Obtaining and decoding a video bitstream to obtain a reference video frame and features of an object to be encoded, where a video clip captured by a video capture device is obtained, features of a video frame of the object to be encoded in the video clip are extracted to obtain features of the object to be encoded, and then the video bitstream is obtained by encoding the features of the object to be encoded and the reference video frame in the video clip; Extracting features from the reference video frame to obtain reference features, and performing motion estimation based on the features of the object to be encoded and the reference features to obtain a motion estimation result; Generating a reconstructed video frame by transforming the reference video frame using a generator in a pre-trained generation model based on the motion estimation result. displaying the reconstructed video frame on a display interface; The generation model is obtained by using the model training method according to the first aspect.

[0010] According to a fourth aspect of the embodiments of the present application, an electronic device is provided. The electronic device includes a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface communicate with each other via the communication bus. The memory is configured to store at least one executable instruction for causing the processor to execute an operation corresponding to the model training method according to the first aspect, or an operation corresponding to the video decoding method according to the second aspect or the third aspect.

[0011] According to a fifth aspect of the embodiments of the present application, a computer storage medium storing a computer program is provided. When the program is executed by a processor, it is possible to implement the model training method according to the first aspect, or the video decoding method according to the second aspect or the third aspect.

[0012] According to a sixth aspect of the present application, a computer program product including computer instructions is provided. The computer instructions instruct a computing device to perform an operation corresponding to the model training method according to the first aspect, or an operation corresponding to the video decoding method according to the second aspect or the third aspect.

[0013] In the embodiment of the present application, the model training method provided generates a reconstructed sample frame corresponding to a plurality of consecutive sample frames to be encoded through the generator in the initial generation model. While authenticity discrimination is performed on a single reconstructed sample frame and the corresponding sample frame to be encoded, another authenticity discrimination is also performed on the combined reconstructed sample frame collectively combined from all the reconstructed sample frames in timestamp order and the combined encoded sample frame collectively combined from all the sample frames to be encoded in timestamp order. Furthermore, the adversarial loss value is generated based on the discrimination result from a single sample frame (the first discrimination result) and the discrimination result from the combined sample frame (the second discrimination result), thereby completing the training of the initial generation model. That is, in the embodiment of the present application, when authenticity discrimination is performed, not only the similarity between the reconstructed sample frame and the sample frame to be encoded in the spatial domain but also the similarity between the reconstructed sample frame and the sample frame to be encoded in the time domain is considered. In other words, by comparing the similarity between the combined encoded sample frame and the combined reference sample frame, whether there is a continuous relationship between the consecutive reconstructed sample frames in the time domain existing in the consecutive sample frames to be encoded is considered. Therefore, by training the model based on the above discrimination result and reconstructing the video frame based on the trained generation model, the reconstructed video frame sequence maintains consistency with the video frame sequence to be encoded in the time domain, thereby improving the phenomena of flicker and floating artifacts and enhancing the video reconstruction quality.

Brief Description of the Drawings

[0014] To more clearly explain the technical solutions in the embodiments of the present application or the prior art, the following briefly describes the attached drawings necessary for explaining the embodiments or the prior art. Obviously, the attached drawings described below show only some of the embodiments of the present application, and those skilled in the art can derive other attached drawings therefrom.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Embodiments for Carrying Out the Invention

[0015] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some, but not all, of the embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application are within the protection scope of the present application.

[0016] The main principle of the encoding and decoding method based on deep video generation is to deform a reference frame based on the movement of the frame to be encoded by using a generator in a generation model in order to obtain a reconstructed frame corresponding to the frame to be encoded. Please refer to FIG. 1, which is a schematic diagram of the framework of the model training stage in the encoding and decoding method based on deep video generation. In the training stage, the generation model usually adopts an adversarial generation network composed of a generator and a spatial discriminator. After the generator obtains the reconstructed frame, the reconstructed frame and the frame to be encoded are input into the spatial discriminator, and the spatial discriminator performs authenticity identification and outputs a spatial identification result. Then, a spatial adversarial loss function is constructed based on the spatial identification result to complete the model training.

[0017] The basic framework of the training process will be described below in combination with FIG. 1. The first step, in the encoding stage, the encoder uses a feature extractor to extract the target keypoint information of the face, which is the single target, from the video frame to be encoded, and encodes the target keypoint information. At the same time, the reference face video frame is encoded using traditional image encoding methods (such as VVC, HEVC, etc.).

[0018] The second step is the decoding stage. The motion estimation model in the decoder extracts the reference keypoint information of the reference face video frame via the keypoint extractor, and performs dense motion estimation based on the reference keypoint information and the target keypoint information to obtain a dense motion estimation map and an occlusion map. The dense motion estimation map represents the relative motion relationship between the target face video frame and the reference face video frame within the feature domain represented by the keypoint information. The occlusion map represents the degree of occlusion of individual pixels within the target face video frame.

[0019] The third step is the decoding stage. The generator inside the generation model in the decoder deforms the reference face video frame based on the dense motion estimation map to obtain a deformation result, and then multiplies the deformation result by the occlusion map to output the reconstructed face video frame. At the same time, after the generator obtains the reconstructed frame, the reconstructed frame and the frame to be encoded are input into the spatial discriminator, and the spatial discriminator performs authenticity discrimination and outputs a spatial discrimination result.

[0020] The fourth step is the model training stage. Based on the spatial discrimination result and the target face video frame, a spatial adversarial loss value is generated. Then, the model is trained according to the above spatial adversarial loss value, and a trained feature extractor (feature extraction model), a trained motion estimation model, and a trained generation model are obtained.

[0021] In the training method shown in FIG. 1, when the spatial discriminator performs authenticity discrimination, only the similarity between a single reconstructed video frame in the spatial domain and its corresponding video frame to be encoded is considered, that is, only the similarity between a single reconstructed video frame and its corresponding video frame to be encoded is compared. When reconstructing a video frame using the above generative model, the resulting reconstructed video frame sequence (reconstructed video clip) is usually visually characterized by flicker and floating artifacts, and the video reconstruction quality is relatively low.

[0022] In an embodiment of the present application, a reconstructed sample frame corresponding to a plurality of consecutive sample frames to be encoded is generated through a generator of an initial generation model. The authenticity identification for a single reconstructed sample frame and the corresponding sample frame to be encoded is performed using a first identifier (spatial identifier), but another authenticity identification is also performed on a combined reconstructed sample frame that is collectively combined from all the reconstructed sample frames in timestamp order and a combined encoded sample frame that is collectively combined from all the sample frames to be encoded in timestamp order using a second identifier (temporal identifier). In this way, an adversarial loss value is generated based on the identification result from a single sample frame (the first identification result) and the identification result from the combined sample frames (the second identification result), and the training of the initial generation model can be completed. That is, in the embodiment of the present application, when authenticity identification is performed, not only the similarity between the reconstructed sample frame and the sample frame to be encoded in the spatial domain but also the similarity between the reconstructed sample frame and the sample frame to be encoded in the temporal domain is considered. In other words, by comparing the similarity between the combined encoded sample frame and the combined reference sample frame, it is considered whether there is a continuous relationship between the consecutive reconstructed sample frames in the temporal domain existing in the consecutive encoded sample frames. Therefore, by training the model based on the above identification results and reconstructing video frames based on the trained generation model, it is possible to maintain the consistency between the reconstructed video frame sequence and the video frame sequence to be encoded in the temporal domain, thereby improving the phenomena of flicker and floating artifacts and improving the video reconstruction quality.

[0023] The implementation of the embodiment of the present application is further illustrated with reference to the accompanying drawings of the embodiment of the present application.

[0024] Embodiment I Refer to FIG. 2, which is a flowchart showing the steps of the model training method according to Embodiment I of the present application. Specifically, the model training method provided in this embodiment includes the following steps.

[0025] S202: Obtain a reference sample frame and a plurality of consecutive sample frames to be encoded.

[0026] Specifically, the reference sample frame and the individual sample frames to be encoded in the embodiment of the present application can be video frames derived from the same video sample. Further, the reference sample frame and the individual sample frames to be encoded can be face video frames.

[0027] S204: Use the generator in the initial generation model to deform the reference sample frame and generate a reconstructed sample frame corresponding to each of the sample frames to be encoded.

[0028] Specifically, the reconstructed sample frame can be obtained by the following method.

[0029] To obtain the motion estimation result, for each sample frame to be encoded, perform motion estimation on the sample frame to be encoded based on the reference sample feature, and input the reference sample frame and the motion estimation result into the generator in the initial model to obtain a reconstructed sample frame corresponding to the sample frame to be encoded. The motion estimation result represents the relative motion relationship between the reference sample frame and the sample frame to be encoded within a preset feature domain.

[0030] Furthermore, it is possible to extract the reference sample features of the reference sample frame and the sample features to be encoded of each sample frame to be encoded. For each sample frame to be encoded, in order to obtain the motion estimation result, motion estimation is performed based on the reference sample features and the sample features to be encoded of the sample frame to be encoded. The reference sample frame and the motion estimation result are input to the initial generator, and a reconstructed sample frame corresponding to the sample frame to be encoded is obtained.

[0031] S206: Input the reconstructed individual sample frames and the corresponding sample frames to be encoded into the first discriminator in the initial generation model to obtain the first discrimination result.

[0032] The first discriminator in the embodiment of the present application may also be referred to as a spatial discriminator. Specifically, for a specific reconstructed sample frame and the corresponding sample frame to be encoded, after the two sample frames are input to the first discriminator, the features of the two sample frames are respectively extracted, whereby feature maps (reconstructed feature maps and feature maps to be encoded) of the two sample frames are obtained. Then, by comparing the distributions of the two feature maps in the spatial domain to confirm whether the two are similar, a first output result is obtained that characterizes whether the two sample frames are the same (or whether the two sample frames are sufficiently similar). For example, when the first output result is 1 (true), it indicates that the two sample frames are the same, and when the first output result is 0 (false), it indicates that the two sample frames are different sample frames. In the embodiment of the present application, the first discrimination result may include the feature map of the reconstructed sample frame extracted and reconstructed by the first discriminator (hereinafter referred to as the first discrimination result of the reconstructed sample frame), the feature map of the sample frame to be encoded extracted by the first discriminator (hereinafter referred to as the first discrimination result of the sample frame to be encoded), and the above-mentioned first output result.

[0033] S208: Combine the sample frames to be encoded according to the time stamp to obtain a combined encoded sample frame, and combine the reconstructed sample frames to obtain a combined reconstructed sample frame. Input the combined encoded sample frame and the combined reconstructed sample frame into the second discriminator of the initial generation model to obtain a second identification result.

[0034] The second discriminator in the embodiments of the present application may also be referred to as a time identifier. Similar to the first discriminator, for the combined reconstructed sample frame and the corresponding combined encoded sample frame, after the two sample frames are input into the second discriminator, the features of the two sample frames are respectively extracted, whereby feature maps of the two sample frames are obtained. Then, by comparing the distributions of the two feature maps in the spatial domain to confirm whether the two are similar, a second output result is obtained to characterize whether the two sample frames are the same (or whether the two sample frames are sufficiently similar). In the embodiments of the present application, the second identification result may include the feature map of the combined and reconstructed sample frame extracted by the second discriminator (hereinafter referred to as the second identification result of the combined reconstructed sample frame), the feature map of the combined sample frame to be encoded extracted by the second discriminator (hereinafter referred to as the second identification result of the combined encoded sample frame), and the above-mentioned second output result.

[0035] In the embodiments of the present application, the generation model includes a generator, a first discriminator, and a second discriminator. The first discriminator and the second discriminator are connected in parallel after the generator to perform authenticity identification based on the reconstructed sample frames output by the generator. Specifically, reference is made to FIG. 3, which is a schematic diagram showing the network architecture of the generation model according to the embodiment shown in FIG. 2.

[0036] The generator G includes an encoding part and a decoding part. The reference sample frame K, and the continuous sample frames I1, I2, ···, I to be encoded nA single sample frame I to be encoded within i The corresponding motion estimation result is input to the generator. Through the encoding part and the decoding part of the generator, the sample frame I^ i corresponding to the reconstructed sample frame I i is finally output, whereby the finally reconstructed sample frames: I^1, I^2, ···, I^ n are obtained. Here, i is a natural number greater than or equal to 1 and less than or equal to n.

[0037] The spatial discriminator (first discriminator) Ds located after the generator G is configured to perform authenticity discrimination on a single reconstructed sample frame I^ i and the corresponding sample frame I to be encoded. The temporal discriminator (second discriminator) Dt located after the generator and connected in parallel with the spatial discriminator Ds is configured to perform authenticity discrimination on the combined sample frames i i to be encoded and the combined and reconstructed sample frame I^ 1-n 1-n to output a second output result.

[0038] S210: Obtain an adversarial loss value based on the first discrimination result and the second discrimination result, and train the initial generation model based on the adversarial loss value to obtain a trained generation model.

[0039] Optionally, in some embodiments, the adversarial loss value may include a generative adversarial loss value, a spatial adversarial loss value, and a temporal adversarial loss value, and each of these may be obtained by the following method.

[0040] Obtain a generative adversarial loss value based on the first discrimination result of each reconstructed sample frame. Here, the larger the sum of the first discrimination results of each reconstructed sample frame, the smaller the generative adversarial loss value.

[0041] ​Based on the difference between the first identification result of each reconstructed sample frame and the first identification result of the corresponding sample frame to be encoded, a spatial adversarial loss value is obtained. Here, the smaller the difference between the first identification result of each reconstructed sample frame and the first identification result of the corresponding sample frame to be encoded, the smaller the spatial adversarial loss value.

[0042] Based on the difference between the second identification result of the combined encoded sample frame and the second identification result of the combined reconstructed sample frame, a temporal adversarial loss value is obtained. Here, the smaller the difference between the second identification result of the combined encoded sample frame and the second identification result of the combined reconstructed sample frame, the smaller the temporal adversarial loss value.

[0043] As described above, the first identification result of each reconstructed sample frame can be the feature map of each reconstructed sample frame extracted by the first identifier, the first identification result of the sample frame to be encoded can be the feature map of the sample frame to be encoded extracted by the first identifier, the second identification result of the combined encoded sample frame can be the feature map of the combined encoded sample frame extracted by the second identifier, and the second identification result of the combined reconstructed sample frame can be the feature map of the combined reconstructed sample frame extracted by the second identifier.

[0044] Furthermore, the generated adversarial loss value can be obtained as follows.

[0045] The probability distribution of the first identification result of each reconstructed sample frame is obtained as the first reconstruction probability distribution of each reconstructed sample frame, and based on the expected value of the first reconstruction probability distribution of each reconstructed sample frame, the generated adversarial loss value is obtained.

[0046] The spatial adversarial loss value can be obtained by the following method. Obtain the probability distribution of the first identification result for each sample frame to be encoded as the first encoding probability distribution for each sample frame to be encoded, and based on the expected difference between the expected value of the first reconstruction probability distribution for each individual sample frame to be encoded and the expected value of the first encoding probability distribution for each individual sample frame to be encoded, obtain the spatial adversarial loss value.

[0047] The temporal adversarial loss value can be obtained by the following method. Obtain the probability distribution of the second identification result of the combined reconstruction sample frame as the second reconstruction probability distribution, obtain the probability distribution of the second identification result of the combined encoded sample frame as the second encoding probability distribution, and based on the expected difference between the expected value of the second reconstruction probability distribution and the expected value of the second encoding probability distribution, obtain the temporal adversarial loss value.

[0048] Specifically, for the generated adversarial loss value, the sum of the expected values of the first reconstruction probability distributions of each of the reconstructed sample frames can be used as the generated adversarial loss value. The larger the sum of the above expected values, the smaller the generated adversarial loss value. Furthermore, the generated adversarial loss value can be expressed using the following formula.

[0049]

Equation

[0050] Here, L G represents the generated adversarial loss value, Ds(I^ i ) represents the first identification result of the reconstructed sample frame I^ i , Pg [Ds(I^ i )] represents the probability distribution of Ds(I^ i ), that is, the first reconstruction probability distribution of the reconstructed sample frame I^ i , and E I^i~Pg [Ds(I^ i )] represents the reconstructed sample frame I^ irepresents the expected value of the first reconstructed probability distribution, and n is the total number of reconstructed sample frames, i.e., the total number of sample frames to be encoded.

[0051] Furthermore, since the first discriminator and the second discriminator usually include a plurality of different operation layers, for each reconstructed sample frame, the expected value corresponding to the probability distribution of the discrimination results (feature maps extracted by the individual operation layers) output by the individual operation layers of the first discriminator can be calculated separately, and the expected values corresponding to the probability distributions of all the operation layers are summed up to obtain the expected value corresponding to the probability distribution of the first discrimination result of the reconstructed sample frame. This improves the realistic feeling of the reconstructed video frame.

[0052] Specifically, it can be expressed using the following formula.

[0053] [Number]

[0054] Here, Dsa(I^ i ) represents the discrimination result (extracted feature map) of the reconstructed sample frame I^ i output by the a-th operation layer of the first discriminator, Pg [Dsa(I^ i )] represents the probability distribution of Dsa(I^ i ), E I^i~Pg [Dsa(I^ i )] is Pg [Dsa(I^ i )] represents the expected value of, k is the total number of operation layers included in the first discriminator, and also the total number of operation layers included in the second discriminator.

[0055] Regarding the spatial adversarial loss value, the spatial adversarial loss value can be obtained based on the expected difference between the expected value of the first reconstruction probability distribution for each reconstructed sample frame and the expected value of the first encoding probability distribution for each sample frame to be encoded. The greater the expected difference, the greater the spatial adversarial loss value. Furthermore, the spatial adversarial loss value can be expressed using the following formula.

[0056] [Number]

[0057] Here, L Ds represents the spatial adversarial loss value, Ds(I i ) represents the first identification result of the sample frame I i to be encoded, Pr [Ds(I i )] represents the probability distribution of Ds(I i ), and E Ii~Pr [Ds(I i )] is Pr [Ds(I i )]'s expected value.

[0058] Similar to the generative adversarial loss value, furthermore, the spatial adversarial loss value can be obtained using the following formula.

[0059] [Number]

[0060] Here, Dsa(I i ) represents the identification result (extracted feature map) of the sample frame I i to be encoded output by the a-th operation layer of the first identifier, Pr [Dsa(I i )] represents the probability distribution of Dsa(I i ), and E Ii~Pr [Dsa(I i )] is Pr [Dsa(I i )]'s expected value, and Dsa(I^i ) represents I^ output by the a-th operation layer of the first discriminator i , the discrimination result (extracted feature map) of Pg [Dsa(I^ i )] represents the probability distribution of Dsa(I^ i ), and E I^i~Pg [Dsa(I^ i )] represents Pg [Dsa(I^ i )], the expected value of the probability distribution of

[0061] Regarding the temporal adversarial loss value, the greater the expected difference between the expected value of the second reconstruction probability distribution and the expected value of the second encoding probability distribution, the greater the temporal adversarial loss value. Furthermore, the above temporal adversarial loss value can be expressed using the following formula.

[0062]

Equation

[0063] Here, L Dt represents the temporal adversarial loss value, and Dt(I^ 1-n ) represents the second discrimination result of the combined reconstruction sample frame I^ 1-n ), Pg [Dt(I^ 1-n )] represents the probability distribution of Dt(I^ 1-n ), and E I^1-n~Pg [Dt(I^ 1-n )] represents Pg [Dt(I^ 1-n )], the expected value of the probability distribution of 1-n and Dt(I 1-n ) represents the second discrimination result of the combined encoding sample frame I Pr [Dt(I 1-n )] represents the probability distribution of Dt(I 1-n ), and E I1-n~Pr [Dt(I 1-n )] represents Pr [Dt(I 1-n )], the expected value of the probability distribution of

[0064] Similar to the spatial adversarial loss value, the temporal adversarial loss value can be obtained using the following formula.

[0065]

Equation

[0066] Here, Dta(I^ 1-n ) represents the discrimination result (extracted feature map) of the combined reconstruction sample frame I^ 1-n output by the a-th operation layer of the second discriminator, and E I^1-n~Pg [Dta(I^ 1-n )] represents the expected value of the probability distribution of Dta(I^ 1-n ), Dta(I 1-n ) represents the discrimination result (extracted feature map) of the combined sample frame I 1-n output by the a-th operation layer of the second discriminator, and E I1-n~Pr [Dta(I 1-n )] represents the expected value of the probability distribution of Dta(I 1-n ).

[0067] Optionally, in some embodiments, before step 210, the method may further include generating a perceptual loss value based on the reconstructed individual sample frames and the individual sample frames to be encoded, and correspondingly, step 210 may include training an initial generation model based on the adversarial loss value and the perceptual loss value to obtain a trained generation model.

[0068] Optionally, in some embodiments, step 204 performs motion estimation on the individual sample frames to be encoded based on a reference sample frame to obtain a motion estimation result for the individual sample frames to be encoded, and For each sample frame to be symbolized, input the motion estimation results of the reference sample frame and the sample frame to be symbolized into the generator of the initial generation model, and further deform the reference sample frame through the generator to generate a reconstructed sample frame corresponding to the sample frame to be symbolized.

[0069] Correspondingly, before step 210, the following steps may further be included. Input each sample frame to be symbolized into a pre-trained motion estimation module to obtain the actual motion result corresponding to each sample to be symbolized, and Generate an optical flow loss value based on the difference between the motion estimation result and the actual motion result of each sample frame to be symbolized.

[0070] Training the initial generation model based on the adversarial loss value and the perceptual loss value to obtain a trained generation model further includes training the initial generation model based on the adversarial loss value, the perceptual loss value, and the optical flow loss value to obtain a trained generation model.

[0071] Specifically, in the above-described embodiment, the motion estimation model may be a neural network model pre-trained to obtain the relative motion relationship (i.e., the actual motion result) between the input sample frame to be symbolized and the reference sample frame. In the embodiment of the present application, the estimation model is not limited to a specific structure. For example, the estimation model may be an end-to-end spatial pyramid network (SpyNet).

[0072] Regarding the optical flow loss value, the greater the difference between the motion estimation result and the actual motion result of each sample frame to be encoded, the greater the optical flow loss value. In other words, the optical flow loss value can characterize the accuracy of the motion estimation result. Therefore, in the model training process, the motion estimation process can be taught to improve the accuracy of the motion estimation process and further improve the quality of the reconstructed video frame by further considering the optical flow loss value in addition to the adversarial loss value and the perceptual loss value, and performing encoding and decoding processes based on the trained model to obtain the reconstructed video frame.

[0073] Optionally, in some embodiments, the process of generating an optical flow loss value based on the difference between the motion estimation result and the actual motion result of each sample frame to be encoded may for each sample frame to be encoded, calculate the difference between the motion estimation result and its actual motion result as the motion error corresponding to the sample frame to be encoded, and sum the individual motion errors as the optical flow loss value.

[0074] Specifically, the optical flow loss value can be calculated using the following formula.

[0075]

Equation

[0076] Here, L flow is the optical flow loss value, M i original is the actual motion result of the sample frame I to be encoded, M i is the motion estimation result of the sample frame I to be encoded, and n is the total number of sample frames to be encoded. i dense is the motion estimation result of the sample frame I to be encoded, and n is the total number of sample frames to be encoded. i is the motion estimation result of the sample frame I to be encoded, and n is the total number of sample frames to be encoded.

[0077] In the embodiment of the present application, the model training method provided generates a reconstructed sample frame corresponding to a plurality of consecutive sample frames to be encoded through a generator in an initial generation model. While authenticity discrimination is performed on a single reconstructed sample frame and the corresponding sample frame to be encoded, another authenticity discrimination is also performed on the reconstructed sample frames combined together from all the reconstructed sample frames in timestamp order and the sample frames to be encoded combined together from all the sample frames to be encoded in timestamp order. Therefore, the adversarial loss value is generated based on the discrimination result from a single sample frame (the first discrimination result) and the discrimination result from the combined sample frames (the second discrimination result), thereby completing the training of the initial generation model. That is, in the embodiment of the present application, when authenticity discrimination is performed, not only the similarity between the reconstructed sample frame and the sample frame to be encoded in the spatial domain but also the similarity between the reconstructed sample frame and the sample frame to be encoded in the temporal domain is considered. In other words, by comparing the similarity between the combined sample frames to be encoded and the combined reference sample frames, it is considered whether there is a continuous relationship between the consecutive reconstructed sample frames in the temporal domain existing in the consecutive sample frames to be encoded. Therefore, by training the model based on the above discrimination results and reconstructing video frames based on the trained generation model, it is possible to maintain the reconstructed video frame sequence to match the video frame sequence to be encoded in the temporal domain, thereby improving the phenomena of flicker and floating artifacts and improving the video reconstruction quality.

[0078] The model training method of the embodiment can be executed by any suitable electronic device having data capabilities, including but not limited to servers, PCs, etc.

[0079] Embodiment II Refer to FIG. 4, which is a flowchart showing the steps of the model training method according to Embodiment II of the present application. Specifically, the model training method provided in this embodiment includes the following steps.

[0080] S402: Obtain a reference sample frame and a plurality of consecutive sample frames to be encoded.

[0081] S404: Extract the reference sample features of the reference sample frame and the sample features to be encoded of each sample frame to be encoded through an initial feature extraction model.

[0082] S406: For each sample frame to be encoded, perform motion estimation based on the reference sample features and the sample features to be encoded of the sample frame to be encoded using an initial motion estimation model to obtain a motion estimation result. Input the reference sample frame and the motion estimation result into an initial generator to obtain a reconstructed sample frame corresponding to the sample frame to be encoded.

[0083] S408: Input the reconstructed individual sample frames and the corresponding sample frames to be encoded into a first discriminator in an initial generation model to obtain a first discrimination result.

[0084] S410: Combine the sample frames to be encoded according to the time stamp order to obtain a combined encoded sample frame, and combine the reconstructed sample frames to obtain a combined reconstructed sample frame. Input the combined encoded sample frame and the combined reconstructed sample frame into a second identifier of the initial generation model to obtain a second discrimination result.

[0085] S412: Train the initial feature extraction model, the initial motion estimation model, and the initial generation model based on the adversarial loss value to obtain a trained feature extraction model, a trained motion estimation model, and a trained generation model.

[0086] In the embodiments of the present application, the specific implementation methods of individual steps can be found in the corresponding steps of Embodiment II, and this will not be repeated herein.

[0087] Refer to FIG. 5, which is a schematic diagram of a scenario corresponding to Embodiment I of the present application. In the following, the embodiments of the present application will be described with reference to the schematic diagram shown in FIG. 5 using a specific scenario as an example.

[0088] A reference sample frame K and a plurality of consecutive sample frames I1, I2, ···, I n of the objects to be encoded are obtained. Through the initial feature extraction model, the reference sample features of the reference sample frame and the sample features of the objects to be encoded of each sample frame to be encoded are extracted. For each sample frame to be encoded, in order to obtain the motion estimation result, motion estimation is performed based on the reference sample features and the sample features of the objects to be encoded of the sample frame to be encoded using the initial motion estimation model. The reference sample frame K and the motion estimation result are input into the initial generator, and a reconstructed sample frame corresponding to the sample frame to be encoded is obtained. Therefore, the reconstructed sample frames: I^1, I^2, ···, I^ n are output through the generator. Each reconstructed sample frame and the corresponding sample frame to be encoded are input into the first discriminator of the initial generation model to obtain the first discrimination result. According to the time stamp order, all the sample frames to be encoded are combined to obtain a combined encoded sample frame, and all the reconstructed sample frames are combined to obtain a combined reconstructed sample frame. The combined encoded sample frame and the combined reconstructed sample frame are input into the second discriminator of the initial generation model to obtain the second discrimination result. Based on the adversarial loss value, the initial feature extraction model, the initial motion estimation model, and the initial generation model are trained to obtain a trained feature extraction model, a trained motion estimation model, and a trained generation model.

[0089] The model training method provided in the embodiments of the present application generates reconstructed sample frames corresponding to a plurality of consecutive sample frames to be encoded through the generator of the initial generation model. While authenticity discrimination is performed on a single reconstructed sample frame and the corresponding sample frame to be encoded, in timestamp order, a combined reconstructed sample frame combined from all the reconstructed sample frames, and in timestamp order, another authenticity discrimination is also performed on the combined encoded sample frame combined from all the sample frames to be encoded. Therefore, the adversarial loss value is generated based on the discrimination result from a single sample frame (the first discrimination result) and the discrimination result from the combined sample frames (the second discrimination result), thereby completing the training of the initial generation model. That is, in the embodiments of the present application, when authenticity discrimination is performed, not only the similarity between the reconstructed sample frame and the sample frame to be encoded in the spatial domain is considered, but also the similarity between the reconstructed sample frame and the sample frame to be encoded in the temporal domain is considered. In other words, by comparing the similarity between the combined sample frames to be encoded and the combined reference sample frames, it is considered whether there is a continuous relationship between the consecutive reconstructed sample frames in the temporal domain existing in the consecutive sample frames to be encoded. Therefore, by training the model based on the above discrimination results and reconstructing video frames based on the trained generation model, it is possible to maintain the reconstructed video frame sequence to match the video frame sequence to be encoded in the temporal domain, thereby improving the phenomena of flicker and floating artifacts and improving the video reconstruction quality.

[0090] The model training method of the embodiment can be executed by any suitable electronic device having data capabilities, including but not limited to servers, PCs, etc.

[0091] Embodiment III Refer to FIG. 6, which is a flowchart showing the steps of the video encoding method according to Embodiment III of the present application. Specifically, the video encoding provided in this embodiment includes the following steps.

[0092] S602: Obtain a reference video frame and a video frame to be encoded.

[0093] S604: Use a pre-trained feature extraction model to extract features from the video frame to be encoded to obtain the features to be encoded.

[0094] The feature extraction model is obtained through the model training method of Embodiment II.

[0095] S606: Encode the reference video frame and the features to be encoded respectively to obtain a bitstream.

[0096] The video encoding method of this embodiment can be executed by any suitable electronic device having data capabilities, including but not limited to servers, PCs, etc.

[0097] The video encoding method provided in Embodiment III of the present application can be executed by a video encoding end (encoder) to encode video files having different resolutions, particularly face video files, in order to compress the digital bandwidth of the video file. This method can be applied to various scenarios, examples of which include the storage and streaming transmission of conventional video games including faces with various resolutions. Specifically, the video frames of a video game can be encoded by the video encoding method provided by the embodiments of the present application to form corresponding video bitstreams for storage and transmission in a video streaming service or other similar applications. Other examples are low-latency scenarios such as video conferencing and live video broadcasting. Specifically, face video data with various resolutions collected by a video acquisition device can be encoded by the video encoding method provided in the embodiments of the present application to form corresponding video bitstreams, which are transmitted to a conference terminal, and the video bitstreams can be decoded by the conference terminal to obtain corresponding face video images. A further example is a virtual reality scenario, in which face video data with various resolutions collected by a video acquisition device can be encoded by the face video encoding method provided in the embodiments of the present application to form corresponding video bitstreams, the corresponding video bitstreams are transmitted to virtual reality-related devices (such as VR virtual glasses), decoded through the VR device to obtain corresponding face video images, and the corresponding VR functions are implemented based on the face video images.

[0098] Embodiment IV Refer to FIG. 7, which is a flowchart showing the steps of the video decoding method according to Embodiment IV of the present application. Specifically, the video decoding method provided in this embodiment includes the following steps.

[0099] S702: Obtain and decode a video bitstream to obtain a reference video frame and characteristics of the object to be encoded.

[0100] S704: Extract features from the reference video frame to obtain reference features, and perform motion estimation based on the features of the object to be encoded and the reference features to obtain a motion estimation result.

[0101] S706: Based on the motion estimation result, use the generator in the pre-trained generation model to deform the reference video frame to generate a reconstructed video frame.

[0102] Here, in this specification, the generation model is obtained by using the model training method according to the first aspect or the second aspect.

[0103] In the video decoding method provided by the embodiment of the present application, the generation model is trained and obtained by the following method. The reconstructed sample frames corresponding to a plurality of consecutive sample frames to be encoded are generated through the generator of the initial generation model. While authenticity discrimination is performed on a single reconstructed sample frame and the corresponding sample frame to be encoded, in timestamp order, the reconstructed sample frames combined and reconstructed together from all the reconstructed sample frames, and in timestamp order, for the sample frames combined together from all the sample frames to be encoded, another authenticity discrimination is also performed. Therefore, the adversarial loss value is generated based on the discrimination result from a single sample frame (the first discrimination result) and the discrimination result from the combined sample frame (the second discrimination result), and the training of the initial generation model can be completed. That is, when authenticity discrimination is performed, not only the similarity between the reconstructed sample frame and the sample frame to be encoded in the spatial domain is considered, but also the similarity between the reconstructed sample frame and the sample frame to be encoded in the temporal domain is considered. In other words, by comparing the similarity between the combined encoded sample frame and the combined reference sample frame, whether there is a continuous relationship between the consecutive reconstructed sample frames in the temporal domain existing in the consecutive sample frames to be encoded is considered. Therefore, by training the model based on the above discrimination result and decoding the video frame based on the trained generation model, it is possible to maintain the reconstructed video frame sequence to match the video frame sequence to be encoded in the temporal domain, thereby improving the phenomena of flicker and floating artifacts and improving the video reconstruction quality.

[0104] The video decoding method of this embodiment can be executed by any suitable electronic device having data capabilities, including but not limited to servers, PCs, etc.

[0105] Embodiment V Refer to FIG. 8, which is a flowchart showing the steps of the video decoding method according to Embodiment V of the present application. The application scenario of the video decoding method is as follows. After a video acquisition device acquires a conference video clip and an encoder extracts the features of the video frames to be encoded within the clip to obtain the features to be encoded, the features to be encoded within the video clip and a reference video frame are encoded to obtain a video bitstream to be transmitted to a conference terminal. The conference terminal decodes the video bitstream to obtain the corresponding conference video image and displays it.

[0106] Specifically, the video decoding method provided in this embodiment includes the following steps.

[0107] S802: Acquire and decode a video bitstream to obtain a reference video frame and the features to be encoded. After a video clip captured by a video capture device is acquired and the features of the video frames to be encoded within the video clip are extracted to obtain the features to be encoded, the video bitstream is obtained by encoding the features to be encoded within the video clip and the reference video frame.

[0108] S804: Extract features from the reference video frame to obtain reference features, and perform motion estimation based on the features to be encoded and the reference features to obtain a motion estimation result.

[0109] S806: Based on the motion estimation result, use a generator within a pre-trained generation model to deform the reference video frame to generate a reconstructed video frame.

[0110] Here, the generation model is obtained using a model training method according to the first aspect or the second aspect.

[0111] S808: Display the reconstructed video frame on a display interface.

[0112] In the video decoding method provided in the embodiments of the present application, the generation model is trained and obtained by the following method. The reconstructed sample frames corresponding to a plurality of consecutive sample frames to be encoded are generated through the generator of the initial generation model. While authenticity discrimination is performed on a single reconstructed sample frame and the corresponding sample frame to be encoded, for the reconstructed sample frames combined together from all the reconstructed sample frames in timestamp order, and for the sample frames combined together from all the sample frames to be encoded in timestamp order, another authenticity discrimination is also performed. Therefore, the adversarial loss value is generated based on the discrimination result from a single sample frame (the first discrimination result) and the discrimination result from the combined sample frames (the second discrimination result), and the training of the initial generation model can be completed. That is, in the embodiments of the present application, when authenticity discrimination is performed, not only the similarity between the reconstructed sample frame and the sample frame to be encoded in the spatial domain is considered, but also the similarity between the reconstructed sample frame and the sample frame to be encoded in the temporal domain is considered. In other words, by comparing the similarity between the combined encoded sample frames and the combined reference sample frames, it is considered whether there is a continuous relationship between the consecutive reconstructed sample frames in the temporal domain existing in the consecutive sample frames to be encoded. Therefore, by training the model based on the above discrimination results and reconstructing the video frames based on the trained generation model, it is possible to maintain the reconstructed video frame sequence to match the video frame sequence to be encoded in the temporal domain, thereby improving the phenomena of flicker and floating artifacts and improving the video reconstruction quality.

[0113] The video decoding method of this embodiment can be executed by any suitable electronic device with data capabilities, including but not limited to servers, PCs, etc.

[0114] Embodiment VI Refer to FIG. 9, which is a block diagram showing the configuration of the model training device according to Embodiment VI of the present application. Specifically, the model training device provided in this embodiment includes a reference sample frame, and a sample frame acquisition module 902 configured to acquire a plurality of consecutive sample frames to be encoded, a reconstructed sample frame generation module 904 configured to deform the reference sample frame by using the generator in the initial generation model to generate a reconstructed sample frame corresponding to each of the sample frames to be encoded, a first identification result acquisition module 906 configured to input the reconstructed individual sample frames and the corresponding sample frames to be encoded into the first identifier in the initial generation model to obtain a first identification result for obtaining a first identification result, a second identification result acquisition module 908 configured to combine all the sample frames to be encoded according to the time stamp order to obtain a combined encoded sample frame, and combine all the reconstructed sample frames to obtain a combined reconstructed sample frame, and input the combined encoded sample frame and the combined reconstructed sample frame into the second identifier of the initial generation model to obtain a second identification result, and a training module 910 configured to obtain an adversarial loss value based on the first identification result and the second identification result, and train the initial generation model based on the adversarial loss value to obtain a trained generation model.

[0115] Optionally, in some embodiments, the adversarial loss value includes a generative adversarial loss value, a spatial adversarial loss value, and a temporal adversarial loss value.

[0116] When the training module 910 executes the step of obtaining the adversarial loss value based on the first identification result and the second identification result, specifically, the training module 910 obtains a generative adversarial loss value based on the first identification result of the individual reconstructed sample frames, Based on the difference between the first identification result of each reconstructed sample frame and the first identification result of the corresponding sample frame to be encoded, obtain a spatial adversarial loss value, and configured to obtain a temporal adversarial loss value based on the difference between the second identification result of the combined encoded sample frame and the second identification result of the combined reconstructed sample frame.

[0117] Optionally, in some embodiments, when the training module 910 executes the step of obtaining a generative adversarial loss value based on the first identification result of the reconstructed individual sample frames, specifically, obtain the probability distribution of the first identification result of each reconstructed sample frame as the first reconstruction probability distribution of each reconstructed sample frame, and obtain a generative adversarial loss value based on the expected value of the first reconstruction probability distribution for each reconstructed sample frame.

[0118] When the training module 910 executes the step of obtaining a spatial adversarial loss value based on the difference between the first identification result of each reconstructed sample frame and the first identification result of the corresponding sample frame to be encoded, the training module 910 specifically obtain the probability distribution of the first identification result of each sample frame to be encoded as the first encoding probability distribution for each sample frame to be encoded, and obtain a spatial adversarial loss value based on the expected difference between the expected value of the first reconstruction probability distribution for each reconstructed sample frame and the expected value of the first encoding probability distribution for each sample frame to be encoded.

[0119] When the training module 910 executes the step of obtaining a temporal adversarial loss value based on the difference between the second identification result of the combined encoded sample frame and the second identification result of the combined reconstructed sample frame, the training module 910 specifically Obtain the probability distribution of the second identification result of the combined reconstruction sample as the second reconstruction probability distribution, obtain the probability distribution of the second identification result of the combined encoded sample frame as the second encoding probability distribution, and obtain a temporal adversarial loss value based on the expected difference between the expected value of the second reconstruction probability distribution and the expected value of the second encoding probability distribution.

[0120] Optionally, in some embodiments, the model training device Before training the initial generation model based on the adversarial loss value, obtain a trained generation model, and include a perceptual loss value acquisition module configured to generate a perceptual loss value based on individual reconstructed sample frames and individual sample frames to be encoded. When the training module 910 executes the step of training the initial generation model based on the adversarial loss value to obtain a trained generation model, specifically, The training module 910 is configured to train the initial generation model based on the adversarial loss value and the perceptual loss value to obtain a trained generation model.

[0121] Optionally, in some embodiments, the reconstructed sample frame generation module 904 Perform motion estimation on each individual sample frame to be encoded based on a reference sample frame to obtain a motion estimation result for each individual sample frame to be encoded. For each sample frame to be encoded, input the motion estimation results of the reference sample frame and the sample frame to be encoded into the generator of the initial generation model, and deform the reference sample frame through the generator to generate a reconstructed sample frame corresponding to the sample frame to be encoded.

[0122] The model training device Before training an initial generation model based on adversarial loss values and perceptual loss values to obtain a trained generation model, input the sample frames of each encoding target into a pre-trained motion estimation module in advance to obtain the actual motion results corresponding to each sample frame of the encoding target, and based on the difference between the motion estimation results and the actual motion results of each sample frame of the encoding target, an optical flow loss value generation module configured to generate an optical flow loss value is provided. When the training module 910 executes the step of training an initial generation model based on adversarial loss values and perceptual loss values to obtain a trained generation model, specifically, the training module 910 is configured to train an initial generation model based on adversarial loss values, perceptual loss values, and optical flow loss values to obtain a trained generation model.

[0123] Optionally, in some embodiments, when the optical flow loss value generation module executes the step of generating an optical flow loss value based on the difference between the motion estimation result and the actual motion result of each sample frame of the encoding target, specifically, the optical flow loss value generation module for each sample frame of the encoding target, calculates the difference between the motion estimation result and the actual motion result as the motion difference corresponding to the sample frame of the encoding target, is configured to sum up the individual motion differences as the total motion difference.

[0124] The ratio of the total motion difference to the total number of sample frames of the encoding target is calculated as the optical flow loss value.

[0125] Optionally, in some embodiments, specifically, the reconstructed sample frame generation module 904 extracts the reference sample features of the reference sample frame and the encoded sample features of each sample frame of the encoding target through the initial feature extraction model. To obtain the motion estimation result, a motion estimation model is used to perform motion estimation for each sample frame to be encoded based on the reference sample features and the encoded sample features of the sample frame to be encoded, and the reference sample frame and the motion estimation result are input into an initial generator to obtain a reconstructed sample frame corresponding to the sample frame to be encoded.

[0126] When the training module 910 executes the step of training the initial generation model based on the adversarial loss value to obtain the trained generation model, specifically, It is configured to train based on the adversarial loss value, the initial feature extraction model, the initial motion estimation model, and the initial generation model, and obtain the trained feature extraction model, the trained motion estimation model, and the trained generation model.

[0127] The model training apparatus in this embodiment is used to implement the corresponding model training methods in the above-described multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated again herein. Additionally, for the functional implementation of individual modules in the model training apparatus of this embodiment, reference can be made to the description of the corresponding parts in the foregoing method embodiments, which will not be repeated herein.

[0128] Embodiment VII Refer to FIG. 10, which is a block diagram showing the configuration of a video encoding apparatus according to Embodiment VII of the present application. Specifically, the video encoding apparatus provided in this embodiment includes A video frame acquisition module 1002 configured to acquire a reference video frame and a video frame to be encoded; An encoded feature acquisition module 1004 configured to extract features from the encoded video frame using a pre-trained feature extraction model to obtain the features to be encoded; An encoding module 1006 configured to encode a reference video frame and features of an object to be encoded respectively to obtain a bitstream.

[0129] The feature extraction model is obtained through the model training method of Embodiment II.

[0130] The video encoding device in this embodiment is used to implement the corresponding video encoding method in the above-described embodiments of the plurality of methods, has the beneficial effects of the corresponding method embodiments, which will not be elaborated again herein. Additionally, the functional implementation of each module in the video encoding device of this embodiment can refer to the description of the corresponding part in the above-described method embodiments and will not be repeated herein.

[0131] Embodiment VIII Refer to FIG. 11, which is a block diagram showing the configuration of a video decoding device according to Embodiment VIII of the present application. Specifically, the video decoding device provided in this embodiment includes A first decoding module 1102 configured to obtain and decode a video bitstream to obtain a reference video frame and features of an object to be encoded; A first motion estimation module 1104 configured to extract features from the reference video frame to obtain reference features, and perform motion estimation based on the features of the object to be encoded and the reference features to obtain a motion estimation result; A first reconstruction module 1106 configured to deform the reference video frame using a generator in a pre-trained generation model based on the motion estimation result to generate a reconstructed video frame.

[0132] Here, the generation model is obtained through the model training method of Embodiment I or Embodiment II described above.

[0133] The video decoding device in this embodiment is used to implement the corresponding video decoding method in the embodiments of the above-mentioned multiple methods, and has the beneficial effects of the embodiments of the corresponding methods, which will not be elaborated again in this specification. Additionally, for the functional implementation of individual modules in the video decoding device of this embodiment, reference can be made to the description of the corresponding parts in the embodiments of the foregoing methods, and will not be repeated in this specification.

[0134] Embodiment IX Refer to FIG. 12, which is a block diagram showing the configuration of the video decoding device according to Embodiment IX of the present application. Specifically, the video decoding device provided in this embodiment includes a second decoding module 1202 configured to obtain and decode a video bitstream in order to obtain a reference video frame and the characteristics of the encoding target, wherein a video clip captured by a video capture device is obtained, the characteristics of the video frames of the encoding target in the video clip are extracted, and after the characteristics of the encoding target are obtained, the video bitstream is obtained by encoding the characteristics of the encoding target and the reference video frame in the video clip; a second motion estimation module 1204 configured to extract features from the reference video frame to obtain reference features, and perform motion estimation based on the characteristics of the encoding target and the reference features to obtain a motion estimation result; a second reconstruction module 1206 configured to deform the reference video frame using a generator in a pre-trained generation model based on the motion estimation result to generate a reconstructed video frame; and a display module 1208 configured to display the reconstructed video frame on a display interface.

[0135] Here, the generation model is obtained through the model training method of Embodiment I or Embodiment II described above.

[0136] The video decoding device in this embodiment is used to implement the corresponding video decoding method in the embodiments of the above-mentioned multiple methods, and has the beneficial effects of the embodiments of the corresponding method, which will not be elaborated again in this specification. Additionally, for the functional implementation of individual modules in the video decoding device of this embodiment, reference can be made to the description of the corresponding parts in the embodiments of the foregoing methods, and it will not be repeated in this specification.

[0137] Embodiment X Refer to FIG. 13, which is a schematic diagram showing the configuration of an electronic device provided in Embodiment X of the present invention. The implementation method of the electronic device is not limited thereby.

[0138] As shown in FIG. 13, the conference terminal may include a processor 1302, a communication interface 1304, a memory 1306, and a communication bus 1308.

[0139] Here, the processor 1302, the communication interface 1304, and the memory 1306 communicate with each other via the communication bus 1308.

[0140] The communication interface 1304 is configured to communicate with other electronic devices or servers.

[0141] The processor 1302 is configured to execute a program 1310, and specifically, may execute related steps in the embodiments of the above-mentioned model training, video encoding, or video decoding methods.

[0142] For example, the program 1310 may include program code, and the program code may include computer operation instructions.

[0143] Processor 1302 can be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. One or more processors included in the smart device may be the same type of processors such as one or more CPUs, or may be different types of processors such as one or more CPUs and one or more ASICs.

[0144] Memory 1306 is configured to store program 1310. Memory 1306 may include high-speed random access memory (RAM), and may also include at least one non-volatile memory such as disk memory.

[0145] Program 1310 may be specifically configured to enable processor 1302 to perform the following operations. Obtain a reference sample frame and a plurality of consecutive sample frames to be encoded. Deform the reference sample frame through a generator in the initial generation model to generate a reconstructed sample frame corresponding to each of the sample frames to be encoded. Input an individual reconstructed sample frame and the corresponding sample frame to be encoded into a first discriminator in the initial generation model to obtain a first discrimination result. Combine the sample frames to be encoded in timestamp order to obtain a combined encoded sample frame, combine the reconstructed sample frames to obtain a combined reconstructed sample frame, input the combined encoded sample frame and the combined reconstructed sample frame into a second discriminator in the initial generation model to obtain a second discrimination result, obtain an adversarial loss value based on the first discrimination result and the second discrimination result, and train the initial generation model based on the adversarial loss value to obtain a trained generation model.

[0146] Alternatively, program 1310 may be specifically configured to enable processor 1302 to perform the following operations. Obtain a reference video frame and a video frame to be encoded, extract features from the video frame to be encoded using a pre-trained feature extraction model to obtain the features to be encoded, and encode the reference video frame and the features to be encoded respectively to obtain a bitstream. Here, the generation model is obtained through the model training method according to the second aspect described above.

[0147] Alternatively, program 1310 may be specifically configured to enable processor 1302 to perform the following operations. Obtain and decode a video bitstream to obtain a reference video frame and the features to be encoded, extract features from the reference video frame to obtain reference features, perform motion estimation based on the features to be encoded and the reference features to obtain a motion estimation result, and based on the motion estimation result, use a generator in a pre-trained generation model to deform the reference video frame and generate a reconstructed video frame. Here, the generation model is obtained through the model training method according to the first aspect or the second aspect described above.

[0148] Alternatively, program 1310 may be specifically configured to enable the processor 1302 to perform the following operations. Obtain and decode a video bitstream to obtain a reference video frame and features to be encoded, obtain a video clip captured by a video capture device, extract the features of the video frames to be encoded within the video clip to obtain the features to be encoded, and then obtain the video bitstream by encoding the features to be encoded and the reference video frame within the video clip. Extract features from the reference video frame to obtain reference features, perform motion estimation based on the features to be encoded and the reference features to obtain a motion estimation result, and based on the motion estimation result, use a generator within a pre-trained generation model to deform the reference video frame and generate a reconstructed video frame, and display the reconstructed video frame on a display interface. Here, the generation model is obtained through the model training method according to the above-described first aspect or second aspect.

[0149] For the implementation of the individual steps in program 1310, corresponding descriptions of the corresponding steps and units in the embodiments of the previously described model training, video encoding, or video decoding methods not detailed herein may be referred to. Those skilled in the art can clearly understand that for the specific working processes of the above-described devices and modules not detailed herein, for the sake of convenience and brevity of the description, they may refer to the descriptions of the corresponding processes in the embodiments of the previously described methods.

[0150] In the embodiments of the present application, the electronic device provided generates a reconstructed sample frame corresponding to a plurality of consecutive sample frames to be encoded through a generator of an initial generation model. While authenticity discrimination is performed on a single reconstructed sample frame and the corresponding sample frame to be encoded, in timestamp order, a combined reconstructed sample frame combined collectively from all the reconstructed sample frames, and in timestamp order, another authenticity discrimination is also performed on a combined encoded sample frame combined collectively from all the sample frames to be encoded. Therefore, the adversarial loss value is generated based on the discrimination result from a single sample frame (the first discrimination result) and the discrimination result from the combined sample frame (the second discrimination result), and the training of the initial generation model can be completed. That is, in the embodiments of the present application, when authenticity discrimination is performed, not only the similarity between the reconstructed sample frame and the sample frame to be encoded in the spatial domain is considered, but also the similarity between the reconstructed sample frame and the sample frame to be encoded in the time domain is considered. In other words, by comparing the similarity between the combined encoded sample frame and the combined reference sample frame, it is considered whether there is a continuous relationship between the consecutive reconstructed sample frames in the time domain existing in the consecutive encoded sample frames. Therefore, by training the model based on the above discrimination results and reconstructing video frames based on the trained generation model, it is possible to maintain the reconstructed video frame sequence to match the video frame sequence to be encoded in the time domain, thereby improving the phenomena of flicker and floating artifacts and improving the video reconstruction quality.

[0151] Further provided in the embodiments of the present application is a computer program product including computer instructions for instructing a computing device to perform operations corresponding to any of the above-described embodiments of the plurality of methods.

[0152] Depending on the implementation requirements, in the embodiments of the present application, the individual components / steps described in the embodiments may be divided into more components / steps, or two or more components / steps, or some operations of the components / steps may be combined into new components / steps to achieve the objectives of the embodiments of the present application.

[0153] The above-described method according to the embodiments of the present application can be implemented as hardware, firmware, or software or computer code stored in a recording medium (such as a CD ROM, RAM, floppy disk (registered trademark), hard disk, or magneto-optical disk, etc.), or downloaded over a network, originally stored in a remote recording medium or a non-transitory machine-readable medium, and implemented as computer code stored in a local recording medium. Therefore, the method described herein can be processed by such software using a general-purpose computer, a special-purpose processor, or programmable or dedicated hardware (such as an ASIC or FPGA, etc.). It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component (such as RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code that implements the model training method, video encoding method, or video decoding method described herein when accessed and executed by the computer, the processor, or the hardware. Further, when a general-purpose computer accesses the code for implementing the model training method, video encoding method, or video decoding method described herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the model training method, video encoding method, or video decoding described herein.

[0154] Those skilled in the art can understand that each example unit and method step described in conjunction with the embodiments disclosed in this specification can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether the function is implemented by hardware or software depends on the specific application and the design constraints of the technical solution. Those skilled in the art can implement the functions described using different methods for each application, but such implementations should not be regarded as exceeding the scope of the embodiments of this application.

[0155] The above implementation method is only used to illustrate the embodiments of this application, but not to limit the embodiments of this application. Those skilled in the art can also make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present invention, and the scope of patent protection of the embodiments of this application should be defined by the scope of the claims.

Claims

1. Obtaining a reference sample frame and multiple consecutive sample frames to be encoded, The process involves using a generator in the initial generative model to deform the reference sample frame and generate a reconstructed sample frame corresponding to each of the sample frames to be encoded, The reconstructed individual sample frames and the corresponding sample frames to be encoded are input to the first classifier in the initial generative model to obtain a first classification result, The sample frames to be encoded are combined in order of timestamp to obtain a combined encoded sample frame, and the reconstructed sample frames are combined to obtain a combined reconstructed sample frame. The combined coded sample frame and the combined reconstructed sample frame are input to the second classifier of the initial generation model to obtain a second classification result. Based on the first identification result and the second identification result, an adversarial loss value is obtained, A model training method comprising: training the initial generative model based on the adversarial loss value to obtain a trained generative model.

2. The adversarial loss value includes a generative adversarial loss value, a spatial adversarial loss value, and a temporal adversarial loss value. Obtaining the adversarial loss value based on the first identification result and the second identification result is: The generative adversarial loss value is obtained based on the first identification result of each reconstructed sample frame, The spatial adversarial loss value is obtained based on the difference between the first identification result of each reconstructed sample frame and the first identification result of the corresponding sample frame to be encoded. The method according to claim 1, comprising obtaining the temporal adversarial loss value based on the difference between the second identification result of the combined coded sample frame and the second identification result of the combined reconstructed sample frame.

3. Obtaining the generative adversarial loss value based on the first identification result of each reconstructed sample frame is: The probability distribution of the first identification result for each reconstructed sample frame is obtained as the first reconstruction probability distribution for each reconstructed sample frame, This includes obtaining the generative adversarial loss value based on the expected value of the first reconstruction probability distribution for each reconstructed sample frame, Obtaining the spatial adversarial loss value based on the difference between the first identification result of each reconstructed sample frame and the first identification result of the corresponding sample frame to be encoded is: The probability distribution of the first identification result for each sample frame to be encoded is obtained as the first coding probability distribution for each sample frame to be encoded, This includes obtaining the spatial adversarial loss value based on the expected difference between the expected value of the first reconstruction probability distribution for the reconstructed sample frame and the expected value of the first coding probability distribution for the sample frame to be coded, and Obtaining the temporal adversarial loss value based on the difference between the second identification result of the combined coded sample frame and the second identification result of the combined reconstructed sample frame is: The probability distribution of the second identification result of the combined reconstructed sample frame is obtained as the second reconstruction probability distribution, The probability distribution of the second identification result of the combined coded sample frame is obtained as the second coded probability distribution, The method according to claim 2, comprising obtaining the temporal adversarial loss value based on the expected difference between the expected value of the second reconstruction probability distribution and the expected value of the second coding probability distribution.

4. Before training the initial generative model based on the adversarial loss values ​​to obtain the trained generative model, the method further includes generating perceptual loss values ​​based on individual reconstructed sample frames and individual sample frames to be encoded. The method according to claim 1, wherein training the initial generative model based on the adversarial loss value to obtain the trained generative model includes training the initial generative model based on the adversarial loss value and the perceptual loss value to obtain the trained generative model.

5. By using the generator in the initial generation model, the reference sample frame is transformed to generate reconstructed sample frames corresponding to each of the sample frames to be encoded. Based on the aforementioned reference sample frame, motion estimation is performed for each sample frame to be encoded, and the motion estimation results for each sample frame to be encoded are obtained. The method includes inputting the reference sample frame and the motion estimation result of the sample frame to be encoded into the generator of the initial generative model for each sample frame to be encoded, and transforming the reference sample frame through the generator to generate the reconstructed sample frame corresponding to the sample frame to be encoded, Before training the initial generative model based on the adversarial loss value and the perceptual loss value to obtain the trained generative model, the method: The individual sample frames to be encoded are input into a pre-trained motion estimation module to obtain the actual motion result corresponding to each of the sample frames to be encoded. This includes generating an optical flow loss value based on the difference between the motion estimation result and the actual motion result of each of the sample frames to be encoded, To obtain the trained generative model by training the initial generative model based on the adversarial loss value and the perceptual loss value is to The method according to claim 4, comprising training the initial generative model based on the adversarial loss value, the perceptual loss value, and the optical flow loss value to obtain the trained generative model.

6. Generating the optical flow loss value based on the difference between the motion estimation result and the actual motion result of each of the sample frames to be encoded is: For each sample frame to be encoded, the difference between the motion estimation result and the actual motion result is calculated as the motion difference corresponding to each sample frame to be encoded. The method according to claim 5, comprising summing the individual motion differences as the optical flow loss value.

7. By using the generator in the initial generation model, the reference sample frame is deformed, and a reconstructed sample frame corresponding to each of the sample frames to be encoded is generated. Through an initial feature extraction model, the reference sample features of the reference sample frame and the sample features to be encoded for each of the sample frames to be encoded are extracted. For each sample frame to be encoded, motion estimation is performed using an initial motion estimation model, based on the reference sample features and the sample features of each sample frame to be encoded, to obtain the motion estimation result. The process includes inputting the reference sample frame and the motion estimation result into the generator to obtain the reconstructed sample frame corresponding to each of the sample frames to be encoded, The method according to claim 1, wherein training the initial generative model based on the adversarial loss value to obtain the trained generative model includes training the initial feature extraction model, the initial motion estimation model, and the initial generative model based on the adversarial loss value to obtain a trained feature extraction model, a trained motion estimation model, and a trained generative model.

8. A video encoding method, Obtaining the reference video frame and the video frame to be encoded, Using a pre-trained feature extraction model, features are extracted from the video frame to be encoded to obtain the features to be encoded. This includes encoding the reference video frame and the features to be encoded, respectively, to obtain a bitstream, A video encoding method wherein the feature extraction model is obtained using the model training method described in claim 7.

9. A video decoding method, The process involves acquiring and decoding a video bitstream to obtain a reference video frame and the features to be encoded, Extracting features from the aforementioned reference video frame to obtain reference features, Motion estimation is performed based on the features of the encoded target and the reference features to obtain the motion estimation result. The process includes, based on the motion estimation results, transforming the reference video frame using a generator in a pre-trained generative model to generate a reconstructed video frame, A video decoding method wherein the generative model is obtained using the model training method described in any one of claims 1 to 7.

10. A video decoding method applicable to conference terminal devices, The method involves acquiring and decoding a video bitstream to obtain a reference video frame and features to be encoded, wherein the video bitstream is obtained by acquiring a video clip captured by a video capture device, extracting features of the video frame to be encoded in the video clip to obtain the features to be encoded, and then encoding the features to be encoded and the reference video frame in the video clip, thereby obtaining the reference video frame and the features to be encoded. Features are extracted from the aforementioned reference video frame to obtain reference features, and motion estimation is performed based on the features to be encoded and the reference features to obtain motion estimation results. Based on the motion estimation results, the reference video frame is transformed using a generator in a pre-trained generative model to generate a reconstructed video frame. The reconstructed video frame is displayed on a display interface, and includes the following: A video decoding method wherein the generative model is obtained using the model training method described in any one of claims 1 to 7.

11. It is an electronic device, Processor and Memory and Communication interface, Equipped with a communication bus, The processor, the memory, and the communication interface communicate with each other via the communication bus. The memory is an electronic device configured to store at least one executable instruction that enables the processor to perform an operation corresponding to the model training method described in any one of claims 1 to 7, or an operation corresponding to the video encoding method described in claim 8.

12. An electronic device, Processor and Memory and Communication interface, Equipped with a communication bus, The processor, the memory, and the communication interface communicate with each other via the communication bus. The memory is an electronic device configured to store at least one executable instruction that enables the processor to perform an operation corresponding to the video decoding method described in claim 9.

13. An electronic device, Processor and Memory and Communication interface, Equipped with a communication bus, The processor, the memory, and the communication interface communicate with each other via the communication bus. The memory is an electronic device configured to store at least one executable instruction that enables the processor to perform an operation corresponding to the video decoding method described in claim 10.

14. A computer storage medium storing a computer program that, when executed by a processor, implements the model training method according to any one of claims 1 to 7, or the video encoding method according to claim 8.

15. A computer storage medium storing a computer program that, when executed by a processor, implements the video decoding method described in Claim 9.

16. A computer storage medium storing a computer program that, when executed by a processor, implements the video decoding method described in claim 10.

17. A computer program including a computer instruction, wherein the computer instruction instructs a computing device to perform an operation corresponding to the model training method described in any one of claims 1 to 7, or an operation corresponding to the video encoding method described in claim 8.

18. A computer program comprising a computer instruction, wherein the computer instruction instructs a computing device to perform an operation corresponding to the video decoding method described in Claim 9.

19. A computer program comprising a computer instruction, wherein the computer instruction instructs a computing device to perform an operation corresponding to the video decoding method described in Claim 10.

Images