Model training method, video encoding method and decoding method

Abstract

Embodiments of the present application provide a model training method, a video encoding method and a decoding method. The model training method comprises: obtaining a reference sample frame and a plurality of continuous to-be-encoded sample frames; performing morphing processing on the reference sample frame by a generator in an initial generation model to generate a reconstructed sample frame; inputting each reconstructed sample frame and the corresponding to-be-encoded sample frame into a first discriminator in the initial generation model to obtain a first discrimination result; splicing the to-be-encoded sample frames in chronological order to obtain a spliced to-be-encoded sample frame, and splicing the reconstructed sample frames to obtain a spliced reconstructed sample frame; inputting the spliced to-be-encoded sample frame and the spliced reconstructed sample frame into a second discriminator in the initial generation model to obtain a second discrimination result; obtaining an adversarial loss value based on the first discrimination result and the second discrimination result, and training the initial generation model based on the adversarial loss value. The present application maintains the consistency of the reconstructed video frame sequence and the to-be-encoded video frame sequence in the time domain, and improves the reconstruction quality.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer, and particularly relate to a model training method, a video encoding method and a decoding method. BACKGROUND

[0002] Video coding is the key to realizing video conferencing, video live broadcast and the like. With the continuous development of machine learning, a coding and decoding method based on deep video generation can be used for coding and decoding operation of video (especially facial video). The method mainly generates a reconstructed frame corresponding to a to-be-coded frame by deforming a reference frame based on the motion of the to-be-coded frame through a generator in a generation model, which is a neural network model.

[0003] In the model training stage, the above generation model is usually a generative adversarial network composed of a generator and a discriminator. During training, the to-be-coded video frame and the reconstructed video frame generated by the generator are input into the discriminator, the discriminator performs authenticity discrimination and outputs a discrimination result, and then a loss function is constructed based on the discrimination result to complete model training.

[0004] However, in the related art, when the discriminator performs authenticity discrimination, only the similarity between the reconstructed video frame and the to-be-coded video frame in the spatial domain is considered, that is, only the similarity between a single reconstructed video frame and the corresponding to-be-coded video frame is compared. When the above generation model is used for video frame reconstruction, the reconstructed video frame sequence (reconstructed video segment) obtained finally usually has visual phenomena such as flicker and floating artifacts, and the video reconstruction quality is poor. SUMMARY

[0005] Therefore, embodiments of the present application provide a model training method, a video encoding method and a decoding method to at least partially solve the above problems.

[0006] According to a first aspect of embodiments of the present application, a model training method is provided, comprising:

[0007] obtaining a reference sample frame and a plurality of continuous to-be-coded sample frames;

[0008] performing deformation processing on the reference sample frame through a generator in an initial generation model to generate a reconstructed sample frame corresponding to each to-be-coded sample frame;

[0009] inputting each reconstructed sample frame and the corresponding to-be-coded sample frame into a first discriminator in the initial generation model to obtain a first discrimination result;

[0010] splicing the to-be-coded sample frames in chronological order to obtain spliced to-be-coded sample frames, splicing the reconstructed sample frames in chronological order to obtain spliced reconstructed sample frames, and inputting the spliced to-be-coded sample frames and the spliced reconstructed sample frames into a second discriminator in the initial generation model to obtain a second discrimination result;

[0011] An adversarial loss value is obtained based on the first discrimination result and the second discrimination result, and the initial generation model is trained based on the adversarial loss value to obtain a trained generation model.

[0012] According to a second aspect of the embodiments of the present application, a video decoding method is provided, comprising:

[0013] A video bitstream is obtained and decoded to obtain a reference video frame and a to-be-encoded feature;

[0014] The reference video frame is subjected to feature extraction to obtain a reference feature, and motion estimation is performed based on the to-be-encoded feature and the reference feature to obtain a motion estimation result;

[0015] A generator in a pre-trained generation model is used to perform morphing processing on the reference video frame based on the motion estimation result to generate a reconstructed video frame;

[0016] The generation model is obtained by the model training method of the first aspect.

[0017] According to a third aspect of the embodiments of the present application, a video decoding method is provided, applied to a conference terminal device, comprising:

[0018] A video bitstream is obtained and decoded to obtain a reference video frame and a to-be-encoded feature; the video bitstream is obtained after a video segment collected by a video collection device is obtained, and a to-be-encoded feature is obtained by performing feature extraction on a to-be-encoded video frame in the video segment, and the to-be-encoded feature and a reference video frame in the video segment are encoded to obtain the video bitstream;

[0019] The reference video frame is subjected to feature extraction to obtain a reference feature, and motion estimation is performed based on the to-be-encoded feature and the reference feature to obtain a motion estimation result;

[0020] A generator in a pre-trained generation model is used to perform morphing processing on the reference video frame based on the motion estimation result to generate a reconstructed video frame;

[0021] The reconstructed video frame is displayed in a display interface;

[0022] The generation model is obtained by the model training method of the first aspect.

[0023] According to a fourth aspect of the embodiments of the present application, an electronic device is provided, comprising a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus; the memory is used to store at least one executable instruction, the executable instruction causes the processor to perform operations corresponding to the model training method according to the first aspect, or operations corresponding to the video decoding method according to the second aspect or the third aspect.

[0024] According to a fifth aspect of the embodiments of the present application, a computer storage medium is provided, and the computer storage medium stores a computer program, the program is executed by a processor to implement the model training method according to the first aspect, or the video decoding method according to the second aspect or the third aspect.

[0025] According to a sixth aspect of the embodiments of the present application, a computer program product is provided, comprising computer instructions, the computer instructions instruct a computing device to perform operations corresponding to the model training method according to the first aspect, or operations corresponding to the video decoding method according to the second aspect or the third aspect.

[0026] The model training method provided by the embodiments of the present application generates a plurality of continuous reconstructed sample frames corresponding to the to-be-encoded sample frames by the generator in the initial generation model, and simultaneously discriminates the authenticity of the single reconstructed sample frame and the corresponding to-be-encoded sample frame, and discriminates the authenticity of the spliced reconstructed sample frame spliced according to the timestamps of the reconstructed sample frames and the spliced to-be-encoded sample frame spliced according to the timestamps of the to-be-encoded sample frames, and then generates an adversarial loss value based on the discrimination results (first discrimination results) between the single sample frames and the discrimination results (second discrimination results) between the spliced sample frames, to complete the training of the initial generation model. That is, in the embodiments of the present application, when discriminating the authenticity, not only the similarity between the reconstructed sample frames and the to-be-encoded sample frames in the spatial domain is considered, but also the similarity between the reconstructed sample frames and the to-be-encoded sample frames in the time domain is considered, that is, by comparing the similarity between the spliced to-be-encoded sample frames and the spliced reference sample frames, whether the continuous relationship between the continuous to-be-encoded sample frames is considered in the time domain. Therefore, based on the above discrimination results for model training and based on the trained generation model for video frame reconstruction, the reconstructed video frame sequence and the to-be-encoded video frame sequence can be kept consistent in the time domain, the flicker and floating artifact phenomenon can be improved, and the quality of video reconstruction can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the accompanying drawings in the following description only represent some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained based on these drawings.

[0028] Figure 1 A schematic diagram of a framework of a coding method based on depth video generation;

[0029] Figure 2 A step flowchart of a model training method according to Embodiment One of the present application;

[0030] Figure 3 A schematic diagram of a network architecture of a model generated in the embodiment shown in Figure 2

[0031] Figure 4 A step flowchart of a model training method according to Embodiment Two of the present application;

[0032] Figure 5 A schematic diagram of a scene example in the embodiment shown in Figure 4

[0033] A step flowchart of a video encoding method according to Embodiment Three of the present application; Figure 6

[0034] A step flowchart of a video decoding method according to Embodiment Four of the present application; Figure 7

[0035] A step flowchart of a video decoding method according to Embodiment Five of the present application; Figure 8

[0036] A structural block diagram of a model training device according to Embodiment Six of the present application; Figure 9

[0037] A structural block diagram of a video encoding device according to Embodiment Seven of the present application; Figure 10

[0038] A structural block diagram of a video decoding device according to Embodiment Eight of the present application; Figure 11

[0039] A structural block diagram of a video decoding device according to Embodiment Nine of the present application; Figure 12

[0040] A structural schematic diagram of an electronic device according to Embodiment Ten of the present application. Figure 13 DETAILED DESCRIPTION ​​

[0041] In order for those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by those skilled in the art shall fall within the scope of protection of the embodiments of the present application.

[0042] The main principle of the coding method based on deep video generation is that the generator in the generation model performs morphing processing on the reference frame based on the motion of the frame to be encoded, to obtain the reconstructed frame corresponding to the frame to be encoded. Referring to Figure 1 , Figure 1 In the coding method based on deep video generation, the framework diagram of the model training stage is shown. In the training stage, the generation model usually adopts a generative adversarial network composed of a generator and a spatial discriminator. After the generator obtains the reconstructed frame, the reconstructed frame and the frame to be encoded are input into the spatial discriminator, the spatial discriminator performs authenticity discrimination and outputs a spatial discrimination result, and then a spatial adversarial loss function is constructed based on the spatial discrimination result to complete the model training.

[0043] The basic framework of the training process will be described below in combination with Figure 1

[0044] The first step is the encoding stage. The encoder uses a feature extractor to extract target key point information of a single target face video frame to be encoded, and encodes the target key point information. At the same time, a traditional image encoding method (such as VVC, HEVC, etc.) is used to encode the reference face video frame.

[0045] The second step is the decoding stage. The motion estimation model in the decoder extracts reference key point information of the reference face video frame through a key point extractor. Based on the reference key point information and the target key point information, a dense motion estimation is performed to obtain a dense motion estimation map and an occlusion map. The dense motion estimation map represents the relative motion relationship between the target face video frame and the reference face video frame in the feature domain represented by the key point information. The occlusion map represents the degree of occlusion of each pixel point in the target face video frame.

[0046] The third step is the decoding stage. The generator inside the generation model in the decoder performs morphing processing on the reference face video frame based on the dense motion estimation map to obtain a morphing processing result, and then multiplies the morphing processing result with the occlusion map to output a reconstructed face video frame. At the same time, after the generator obtains the reconstructed frame, the reconstructed frame and the frame to be encoded are input into the spatial discriminator, the spatial discriminator performs authenticity discrimination and outputs a spatial discrimination result.

[0047] ​In the fourth step, a model training phase, a spatial adversarial loss value is generated based on the spatial discrimination result and the target face video frame, and then the model is trained according to the spatial adversarial loss value to obtain a trained feature extractor (feature extraction model), a motion estimation model and a generation model.

[0048] Figure 1 In the training method, when the spatial discriminator performs authenticity discrimination, only the similarity in the spatial domain between the single reconstructed video frame and the corresponding to-be-encoded video frame is considered, that is, only the similarity between the single reconstructed video frame and the corresponding to-be-encoded video frame is compared. When the video frame is reconstructed by using the generation model, the reconstructed video frame sequence (reconstructed video segment) finally obtained usually has visual flickering and floating artifacts and the like, and the video reconstruction quality is poor.

[0049] In the embodiment of the present application, the generator in the generation model generates a plurality of reconstructed sample frames corresponding to the plurality of continuous to-be-encoded sample frames, and when the first discriminator (spatial discriminator) performs authenticity discrimination on the single reconstructed sample frame and the corresponding to-be-encoded sample frame, the second discriminator (temporal discriminator) also performs authenticity discrimination on the spliced reconstructed sample frame obtained by splicing the reconstructed sample frames in the time stamp order and the spliced to-be-encoded sample frame obtained by splicing the to-be-encoded sample frames in the time stamp order, and then the adversarial loss value is generated based on the discrimination result (first discrimination result) between the single sample frames and the discrimination result (second discrimination result) between the spliced sample frames to complete the training of the initial generation model. That is, in the embodiment of the present application, when performing authenticity discrimination, not only the similarity in the spatial domain between the reconstructed sample frame and the to-be-encoded sample frame is considered, but also the similarity in the time domain between the reconstructed sample frame and the to-be-encoded sample frame is considered, that is, by comparing the similarity between the spliced to-be-encoded sample frame and the spliced reference sample frame, whether the continuous relationship between the continuous to-be-encoded sample frames is considered in the time domain. Therefore, based on the above discrimination result, the model is trained, and based on the trained generation model, the video frame is reconstructed, so that the reconstructed video frame sequence and the to-be-encoded video frame sequence remain consistent in the time domain, the flickering and floating artifact phenomenon is improved, and the video reconstruction quality is improved.

[0050] The specific implementation of the embodiment of the present application will be further described below in combination with the drawings of the embodiment of the present application.

[0051] Embodiment one

[0052] Referring to Figure 2 , Figure 2 is a step flow chart of a model training method according to the embodiment one of the present application. Specifically, the model training method provided in the embodiment includes the following steps:

[0053] At step 202, a reference sample frame and a plurality of continuous to-be-encoded sample frames are obtained.

[0054] Specifically, the reference sample frame and each to-be-encoded sample frame in the embodiment of the present application can be a video frame derived from the same video sample, and further, the reference sample frame and each to-be-encoded sample frame can be a face video frame.

[0055] At step 204, the reference sample frame is morphed by a generator in the initial generation model to generate a reconstructed sample frame corresponding to each to-be-encoded sample frame.

[0056] Specifically, the reconstructed sample frame can be obtained in the following manner:

[0057] For each to-be-encoded sample frame, motion estimation is performed on the to-be-encoded sample frame based on the reference sample frame to obtain a motion estimation result; and the reference sample frame and the motion estimation result are input into the generator in the initial model to obtain a reconstructed sample frame corresponding to the to-be-encoded sample frame. The motion estimation result represents the relative motion relationship between the reference sample frame and the to-be-encoded sample frame in a preset feature domain.

[0058] Further, the reference sample feature of the reference sample frame and the to-be-encoded sample feature of each to-be-encoded sample frame can be extracted; for each to-be-encoded sample frame, motion estimation is performed based on the reference sample feature and the to-be-encoded sample feature of the to-be-encoded sample frame to obtain a motion estimation result; and the reference sample frame and the motion estimation result are input into the initial generator to obtain a reconstructed sample frame corresponding to the to-be-encoded sample frame.

[0059] At step 206, each reconstructed sample frame and the corresponding to-be-encoded sample frame are input into the first discriminator in the initial generation model to obtain a first discrimination result.

[0060] The first discriminator in the embodiments of the present application can also be referred to as a spatial discriminator. Specifically, for a certain reconstructed sample frame and a corresponding to-be-encoded sample frame, after the two sample frames are input into the first discriminator, the first discriminator extracts features of the two sample frames respectively to obtain feature maps of the two sample frames (reconstructed feature map and to-be-encoded feature map), and then obtains a first output result representing whether the two sample frames are the same sample frames (or whether the two sample frames are similar enough) by comparing whether the distribution of the two feature maps in the spatial domain is similar. For example, the first output result is 1 (true) indicating that the two sample frames are the same sample frames, and the first output result is 0 (false) indicating that the two sample frames are not the same sample frames. In the embodiments of the present application, the first discrimination result can include the feature map of the reconstructed sample frame extracted by the first discriminator (hereinafter referred to as the first discrimination result of the reconstructed sample frame), the feature map of the to-be-encoded sample frame extracted by the first discriminator (hereinafter referred to as the first discrimination result of the to-be-encoded sample frame), and the first output result.

[0061] In step 208, the to-be-encoded sample frames are spliced in chronological order to obtain a spliced to-be-encoded sample frame, and the reconstructed sample frames are spliced to obtain a spliced reconstructed sample frame. The spliced to-be-encoded sample frame and the spliced reconstructed sample frame are input into the second discriminator in the initial generation model to obtain a second discrimination result.

[0062] The second discriminator in the embodiments of the present application can also be referred to as a temporal discriminator. Similar to the first discriminator, for the spliced reconstructed sample frame and the corresponding spliced to-be-encoded sample frame, after the two sample frames are input into the second discriminator, the second discriminator extracts features of the two sample frames respectively to obtain feature maps of the two sample frames, and then obtains a second output result representing whether the two sample frames are the same sample frames (or whether the two sample frames are similar enough) by comparing whether the distribution of the two feature maps in the spatial domain is similar. In the embodiments of the present application, the second discrimination result can include the feature map of the spliced reconstructed sample frame extracted by the second discriminator (hereinafter referred to as the second discrimination result of the spliced reconstructed sample frame), the feature map of the spliced to-be-encoded sample frame extracted by the second discriminator (hereinafter referred to as the second discrimination result of the spliced to-be-encoded sample frame), and the second output result.

[0063] In the embodiments of the present application, the generation model includes a generator, a first discriminator and a second discriminator, wherein the first discriminator and the second discriminator are connected in parallel after the generator to respectively perform authenticity discrimination based on the reconstructed sample frame output by the generator. Specifically, referring to Figure 3 , Figure 3 For Figure 2 the network architecture schematic diagram of the generation model in the embodiments shown in FIG. 8:

[0064] Wherein, the generator G includes an encoding part and a decoding part, and the reference sample frame K and the continuous to-be-encoded sample frames I1, I2, …, In are input into the generator G through the encoding part and the decoding part of the generator G, and finally the to-be-encoded sample frame I n i The corresponding motion estimation result is input into the generator, and finally the to-be-encoded sample frame I i The corresponding reconstructed sample frame Thus, the reconstructed sample frame is finally obtained as follows: Wherein, i is a natural number greater than or equal to 1 and less than or equal to n.

[0065] The spatial discriminator (the first discriminator) D s is located behind the generator G, and is used for respectively performing authenticity discrimination on the single reconstructed sample frame and the corresponding to-be-encoded sample frame I i to output the first output result. The temporal discriminator (the second discriminator) D t is located behind the generator, and is connected in parallel with the above-mentioned spatial discriminator D s , and is used for performing authenticity discrimination on the spliced to-be-encoded sample frame I 1-n and the spliced reconstructed sample frame to output the second output result.

[0066] In step 210, the adversarial loss value is obtained based on the first discrimination result and the second discrimination result, and the initial generation model is trained based on the adversarial loss value to obtain the trained generation model.

[0067] Optionally, in some embodiments, the adversarial loss value can include a generation adversarial loss value, a spatial adversarial loss value and a temporal adversarial loss value; wherein each adversarial loss value can be obtained in the following manner:

[0068] The generation adversarial loss value is obtained based on the first discrimination result of each reconstructed sample frame; wherein the greater the sum of the first discrimination results of the reconstructed sample frames, the smaller the generation adversarial loss value;

[0069] The spatial adversarial loss value is obtained based on the difference value between the first discrimination result of each reconstructed sample frame and the first discrimination result of the corresponding to-be-encoded sample frame; wherein the smaller the difference value between the first discrimination result of each reconstructed sample frame and the first discrimination result of the corresponding to-be-encoded sample frame, the smaller the spatial adversarial loss value;

[0070] The temporal adversarial loss value is obtained based on the difference value between the second discrimination result of the spliced to-be-encoded sample frame and the second discrimination result of the spliced reconstructed sample frame; wherein the smaller the difference value between the second discrimination result of the spliced to-be-encoded sample frame and the second discrimination result of the spliced reconstructed sample frame, the smaller the temporal adversarial loss value; ​

[0071] As described above, the first discrimination result of each reconstructed sample frame can be the feature map of the reconstructed sample frame extracted by the first discriminator; the first discrimination result of the to-be-encoded sample frame can be the feature map of the to-be-encoded sample frame extracted by the first discriminator; the second discrimination result of the spliced to-be-encoded sample frame can be the feature map of the spliced to-be-encoded sample frame extracted by the second discriminator; and the second discrimination result of the spliced reconstructed sample frame can be the feature map of the spliced reconstructed sample frame extracted by the second discriminator.

[0072] Further, the generative adversarial loss value can be obtained in the following manner:

[0073] The probability distribution of the first discrimination result of each reconstructed sample frame is obtained as the first reconstruction probability distribution of each reconstructed sample frame; and based on the expected value of the first reconstruction probability distribution of each reconstructed sample frame, the generative adversarial loss value is obtained.

[0074] The spatial adversarial loss value is obtained in the following manner: the probability distribution of the first discrimination result of each to-be-encoded sample frame is obtained as the first to-be-encoded probability distribution of each to-be-encoded sample frame; and based on the expected difference between the expected value of the first reconstruction probability distribution of each to-be-encoded sample frame and the expected value of the first to-be-encoded probability distribution of each to-be-encoded sample frame, the spatial adversarial loss value is obtained.

[0075] The temporal adversarial loss value is obtained in the following manner: the probability distribution of the second discrimination result of the spliced reconstructed sample frame is obtained as the second reconstruction probability distribution; the probability distribution of the second discrimination result of the spliced to-be-encoded sample frame is obtained as the second to-be-encoded probability distribution; and based on the expected difference between the expected value of the second reconstruction probability distribution and the expected value of the second to-be-encoded probability distribution, the temporal adversarial loss value is obtained.

[0076] Specifically, for the generative adversarial loss value, the sum of the expected values of the first reconstruction probability distribution of each reconstructed sample frame can be taken as the generative adversarial loss value. The greater the sum of the expected values, the smaller the generative adversarial loss value. Further, the generative adversarial loss value can be represented by the following equation:

[0077]

[0078] wherein L G represents the generative adversarial loss value; represents the first discrimination result of the reconstructed sample frame ; represents the probability distribution of ; that is, the first reconstruction probability distribution of the reconstructed sample frame ; represents the first discrimination result of the to-be-encoded sample frame ; represents the probability distribution of ; that is, the first to-be-encoded probability distribution of the to-be-encoded sample frame an expectation value of the first reconstruction probability distribution of the reconstructed sample frame; n is a total number of the reconstructed sample frames, i.e., a total number of the sample frames to be encoded.

[0079] Further, since the first discriminator and the second discriminator each generally include a plurality of different operation layers, for each reconstructed sample frame, an expectation value corresponding to a probability distribution of a discrimination result (a feature map extracted by each operation layer) output by each operation layer of the first discriminator can be calculated respectively, and expectation values corresponding to probability distributions of all operation layers are summed to obtain an expectation value corresponding to a probability distribution of the first discrimination result of the reconstructed sample frame, so that the realism of the reconstructed video frame can be improved.

[0080] Specifically, the following equation can be used to represent:

[0081]

[0082] wherein, represents the reconstructed sample frame output by the a-th operation layer of the first discriminator; represents a probability distribution of represents an expectation value of k is a total number of operation layers included in the first discriminator, and is also a total number of operation layers included in the second discriminator.

[0083] For the spatial adversarial loss value, the spatial adversarial loss value can be obtained based on an expectation difference between an expectation value of the first reconstruction probability distribution of each reconstructed sample frame and an expectation value of the first encoding probability distribution of each sample frame to be encoded. The greater the expectation difference, the greater the spatial adversarial loss value. Further, the spatial adversarial loss value can be represented by the following equation:

[0084]

[0085] wherein, represents the spatial adversarial loss value; D s (I i ) represents the first discrimination result of the sample frame I i to be encoded; P r [D s (I i )] represents a probability distribution of D s (I i ); represents an expectation value of P r [D s (I i )].

[0086] Similar to the generative adversarial loss, the spatial adversarial loss can be further obtained through the following equation:

[0087]

[0088] Among them, D sa (I i ) represents the sample frame I to be encoded i The discrimination result (extracted feature map) output by the a-th operation layer of the first discriminator; P r [D sa (I i )] represents D sa (I i The probability distribution of ). P represents r [D sa (I i The expected value of )]; express The discrimination result (extracted feature map) output by the a-th operation layer of the first discriminator; express The probability distribution; express The expected value.

[0089] For the temporal adversarial loss value, the larger the expected difference between the expected value of the second reconstructed probability distribution and the expected value of the second probability distribution to be encoded, the larger the temporal adversarial loss value. Furthermore, the above-mentioned temporal adversarial loss value can be expressed by the following equation:

[0090]

[0091] in, This represents the time-resistance loss value; Indicates the splicing and reconstruction of sample frames The second identification result; express The probability distribution; express Expected value; D t (I 1-n ) indicates the coded sample frame I. 1-n The second identification result; P r [D t (I 1-n )] represents D t (I 1-n The probability distribution of ). P represents r [D t (I 1-n The expected value of ).

[0092] Further, the time adversarial loss value can be obtained by the following equation:

[0093]

[0094] wherein, denotes the spliced reconstructed sample frame the discrimination result (extracted feature map) output by the a th operation layer of the second discriminator; denotes the expectation value of the probability distribution of ta (I 1-n ) denotes the spliced to-be-encoded sample frame I 1-n the discrimination result (extracted feature map) output by the a th operation layer of the second discriminator; denotes the probability distribution of D ta (I 1-n ) the expectation value of the probability distribution.

[0095] Optionally, before step 210, the method can further include: generating a perceptual loss value based on the reconstructed sample frames and the to-be-encoded sample frames; correspondingly, step 210 can include: training the initial generation model based on the adversarial loss value and the perceptual loss value to obtain the trained generation model.

[0096] Optionally, in some embodiments, step 204 can include:

[0097] based on the reference sample frame, performing motion estimation on each to-be-encoded sample frame to obtain a motion estimation result of each to-be-encoded sample frame;

[0098] for each to-be-encoded sample frame, inputting the reference sample frame and the motion estimation result of the to-be-encoded sample frame into the generator in the initial generation model, and performing deformation processing on the reference sample frame by the generator to generate a reconstructed sample frame corresponding to the to-be-encoded sample frame;

[0099] correspondingly, before step 210, the method can further include:

[0100] inputting each to-be-encoded sample frame into a pre-trained motion prediction module to obtain a real motion result corresponding to each to-be-encoded sample;

[0101] generating an optical flow loss value based on a difference between the motion estimation result and the real motion result of each to-be-encoded sample frame;

[0102] training the initial generation model based on the adversarial loss value and the perceptual loss value to obtain the trained generation model, including:

[0103] ​The initial generation model is trained based on the adversarial loss value, the perception loss value, and the optical flow loss value to obtain a trained generation model.

[0104] Specifically, in the above embodiments, the motion prediction model can be a neural network model that is pre-trained to obtain the relative motion relationship (i.e., the real motion result) between the input sample frame to be encoded and the reference sample frame. In the embodiments of the present application, the specific structure of the prediction model is not limited, for example, it can be an end-to-end spatial pyramid network (SpyNet), etc.

[0105] As for the optical flow loss value, the greater the difference between the motion estimation result and the real motion result of each sample frame to be encoded, the greater the optical flow loss value. That is, the optical flow loss value can represent the accuracy of the motion estimation result. Therefore, in the model training process, on the basis of considering the adversarial loss value and the perception loss value, the optical flow loss value is further considered, which can supervise the motion estimation process, so that when the trained model is used for encoding and decoding operations, the accuracy of the motion estimation process can be improved, and the quality of the reconstructed video frame can be further improved.

[0106] Optionally, in some embodiments, the process of generating the optical flow loss value based on the difference between the motion estimation result and the real motion result of each sample frame to be encoded can include:

[0107] For each sample frame to be encoded, the difference between the motion estimation result and the real motion result of the sample frame to be encoded is calculated as the motion difference value corresponding to the sample frame to be encoded.

[0108] The sum of the motion difference values is calculated as the optical flow loss value.

[0109] Specifically, the optical flow loss value can be calculated by the following equation:

[0110]

[0111] wherein, L flow is the optical flow loss value; is the real motion result of the sample frame I i to be encoded; is the motion estimation result of the sample frame I i to be encoded; and n is the total number of sample frames to be encoded.

[0112] The model training method provided in the embodiments of the present application generates a plurality of continuous reconstructed sample frames corresponding to the to-be-encoded sample frames through the generator in the initial generation model, and simultaneously discriminates the authenticity of the single reconstructed sample frame and the corresponding to-be-encoded sample frame, and discriminates the authenticity of the spliced reconstructed sample frame spliced according to the timestamps of the reconstructed sample frames and the spliced to-be-encoded sample frame spliced according to the timestamps of the to-be-encoded sample frames, and then generates an adversarial loss value based on the discrimination results (first discrimination results) between the single sample frames and the discrimination results (second discrimination results) between the spliced sample frames, so as to complete the training of the initial generation model. That is, in the embodiments of the present application, when discriminating the authenticity, not only the similarity between the reconstructed sample frame and the to-be-encoded sample frame in the spatial domain is considered, but also the similarity between the reconstructed sample frame and the to-be-encoded sample frame in the time domain is considered, that is, by comparing the similarity between the spliced to-be-encoded sample frame and the spliced reference sample frame, whether the continuous relationship between the continuous to-be-encoded sample frames is considered between the continuous reconstructed sample frames in the time domain. Therefore, based on the above discrimination results for model training and based on the trained generation model for video frame reconstruction, the consistency of the reconstructed video frame sequence and the to-be-encoded video frame sequence in the time domain can be maintained, the flicker and floating artifact phenomenon can be improved, and the quality of video reconstruction can be improved.

[0113] The model training method of the embodiments can be executed by any appropriate electronic device with data capability, including but not limited to servers, PC machines, etc.

[0114] Embodiment Two

[0115] Reference Figure 4 , Figure 4 is a step flowchart of a model training method according to Embodiment Two of the present application. Specifically, the model training method provided in the embodiments includes the following steps:

[0116] Step 402, obtaining a reference sample frame and a plurality of continuous to-be-encoded sample frames.

[0117] Step 404, extracting reference sample features of the reference sample frame and to-be-encoded sample features of each to-be-encoded sample frame through an initial feature extraction model.

[0118] Step 406, for each to-be-encoded sample frame, performing motion estimation based on the reference sample features and the to-be-encoded sample features of the to-be-encoded sample frame through an initial motion estimation model to obtain a motion estimation result; inputting the reference sample frame and the motion estimation result into an initial generator to obtain a reconstructed sample frame corresponding to the to-be-encoded sample frame.

[0119] Step 408, input each reconstructed sample frame and the corresponding to-be-encoded sample frame into a first discriminator in the initial generation model to obtain a first discrimination result.

[0120] Step 410, splice each to-be-encoded sample frame to obtain a spliced to-be-encoded sample frame and splice each reconstructed sample frame to obtain a spliced reconstructed sample frame in chronological order; input the spliced to-be-encoded sample frame and the spliced reconstructed sample frame into a second discriminator in the initial generation model to obtain a second discrimination result.

[0121] Step 412, train the initial feature extraction model, the initial motion estimation model, and the initial generation model based on the adversarial loss value to obtain the trained feature extraction model, the motion estimation model, and the generation model.

[0122] In the embodiments of the present application, the specific implementation modes of the steps can be referred to the corresponding step parts in Embodiment Two, which will not be described here.

[0123] Referring to Figure 5 , Figure 5 is a scene diagram corresponding to Embodiment One of the present application. Hereinafter, with reference to the diagram shown in Figure 5 , the embodiments of the present application will be described with a specific scene example:

[0124] obtain a reference sample frame K and a plurality of continuous to-be-encoded sample frames I1, I2, …, In; n extract reference sample features of the reference sample frame and to-be-encoded sample features of each to-be-encoded sample frame through the initial feature extraction model; for each to-be-encoded sample frame, perform motion estimation based on the reference sample features and the to-be-encoded sample features of the to-be-encoded sample frame to obtain a motion estimation result through the initial motion estimation model; input the reference sample frame K and the motion estimation result into the initial generator to obtain a reconstructed sample frame corresponding to the to-be-encoded sample frame, and then output the reconstructed sample frame through the generator: input each reconstructed sample frame and the corresponding to-be-encoded sample frame into a first discriminator in the initial generation model to obtain a first discrimination result; splice each to-be-encoded sample frame to obtain a spliced to-be-encoded sample frame and splice each reconstructed sample frame to obtain a spliced reconstructed sample frame in chronological order; input the spliced to-be-encoded sample frame and the spliced reconstructed sample frame into a second discriminator in the initial generation model to obtain a second discrimination result; train the initial feature extraction model, the initial motion estimation model, and the initial generation model based on the adversarial loss value to obtain the trained feature extraction model, the motion estimation model, and the generation model.

[0125] The model training method provided in the embodiments of the present application generates a plurality of continuous reconstructed sample frames corresponding to the to-be-encoded sample frames through the generator in the initial generation model, and simultaneously discriminates the authenticity of the spliced reconstructed sample frames spliced according to the timestamps of the reconstructed sample frames and the spliced to-be-encoded sample frames spliced according to the timestamps of the to-be-encoded sample frames, and then generates an adversarial loss value based on the discrimination results (first discrimination results) between the single sample frames and the discrimination results (second discrimination results) between the spliced sample frames, so as to complete the training of the initial generation model. That is, in the embodiments of the present application, when discriminating the authenticity, not only the similarity between the reconstructed sample frames and the to-be-encoded sample frames in the spatial domain is considered, but also the similarity between the reconstructed sample frames and the to-be-encoded sample frames in the time domain is considered, that is, by comparing the similarity between the spliced to-be-encoded sample frames and the spliced reference sample frames, whether the continuous relationship between the continuous to-be-encoded sample frames is considered between the continuous reconstructed sample frames in the time domain. Therefore, based on the above discrimination results for model training and based on the trained generation model for video frame reconstruction, the consistency of the reconstructed video frame sequence and the to-be-encoded video frame sequence in the time domain can be maintained, the flicker and floating artifact phenomenon can be improved, and the quality of video reconstruction can be improved.

[0126] The model training method of the embodiments can be executed by any appropriate electronic device with data capability, including but not limited to servers, PC machines, and the like.

[0127] Embodiment Three

[0128] Reference Figure 6 , Figure 6 A step flowchart of a video encoding method according to Embodiment Three of the present application. Specifically, the video encoding method provided in the embodiments includes the following steps:

[0129] Step 602: obtaining reference video frames and to-be-encoded video frames.

[0130] Step 604: performing feature extraction on the to-be-encoded video frames through a pre-trained feature extraction model to obtain to-be-encoded features.

[0131] The feature extraction model is obtained through the model training method of Embodiment Two.

[0132] Step 606: encoding the reference video frames and the to-be-encoded features respectively to obtain a bitstream.

[0133] The video encoding method of the embodiments can be executed by any appropriate electronic device with data capability, including but not limited to servers, PC machines, and the like.

[0134] The video encoding method provided in the third embodiment of the present application can be executed by a video encoding end (encoder) to encode video files with different resolutions, especially face video files, to compress the digital bandwidth of the video files. It can be applied to various scenarios, such as the storage and streaming of regular face-related video games with various resolutions. Specifically, the game video frames can be encoded by the video encoding method provided in the present application to form corresponding video bitstreams for storage and transmission in video streaming services or other similar applications. For example, in low-latency scenarios such as video conferencing and video live streaming, the face video data with various resolutions collected by a video collection device can be encoded by the video encoding method provided in the present application to form corresponding video bitstreams, which are then sent to a conference terminal. The video bitstreams are decoded by the conference terminal to obtain corresponding face video frames. In virtual reality scenarios, the face video data with various resolutions collected by a video collection device can be encoded by the face video encoding method provided in the present application to form corresponding video bitstreams, which are then sent to virtual reality-related devices (such as VR virtual glasses). The video bitstreams are decoded by the VR devices to obtain corresponding face video frames, and the corresponding VR functions are realized based on the face video frames.

[0135] Embodiment Four

[0136] Referring to Figure 7 , Figure 7 The flowchart of the steps of a video decoding method according to the fourth embodiment of the present application is shown in FIG. 7. Specifically, the video decoding method provided in the present embodiment includes the following steps:

[0137] In step 702, a video bitstream is obtained and decoded to obtain a reference video frame and a to-be-encoded feature.

[0138] In step 704, a reference feature is obtained by feature extraction on the reference video frame, and a motion estimation result is obtained based on the to-be-encoded feature and the reference feature.

[0139] In step 706, the generator in the pre-trained generative model is used to deform the reference video frame based on the motion estimation result to generate a reconstructed video frame.

[0140] The generative model is obtained by the model training method of the first aspect or the second aspect.

[0141] The video decoding method provided in the embodiments of the present application is trained by the following method: the generator in the initial generation model generates a plurality of continuous reconstructed sample frames corresponding to the to-be-encoded sample frames, and, while performing authenticity discrimination on a single reconstructed sample frame and a corresponding to-be-encoded sample frame, authenticity discrimination is also performed on spliced reconstructed sample frames obtained by splicing the reconstructed sample frames in timestamp order and spliced to-be-encoded sample frames obtained by splicing the to-be-encoded sample frames in timestamp order, and then, based on the discrimination results (first discrimination results) between single sample frames and the discrimination results (second discrimination results) between spliced sample frames, an adversarial loss value is generated to complete the training of the initial generation model. That is, when performing authenticity discrimination, not only the similarity between the reconstructed sample frames and the to-be-encoded sample frames in the spatial domain is considered, but also the similarity between the reconstructed sample frames and the to-be-encoded sample frames in the time domain is considered, that is, by comparing the similarity between the spliced to-be-encoded sample frames and the spliced reference sample frames, whether the continuous relationship between the to-be-encoded sample frames is considered between the continuous reconstructed sample frames in the time domain. Therefore, based on the above discrimination results, the model is trained, and based on the trained generation model, video decoding is performed to obtain reconstructed video frames, which can make the reconstructed video frame sequence consistent with the to-be-encoded video frame sequence in the time domain, improve the flicker and floating artifact phenomenon, and improve the quality of video reconstruction.

[0142] The video decoding method of the embodiments can be executed by any appropriate electronic device with data capability, including but not limited to servers, PC machines, etc.

[0143] Embodiment Five

[0144] Reference Figure 8 , Figure 8 The flowchart of the steps of a video decoding method according to Embodiment Five of the present application. The application scenario of the video decoding method is that a video capture device captures a conference video segment; after extracting features of to-be-encoded video frames in the segment to obtain to-be-encoded features, the to-be-encoded features and reference video frames in the video segment are encoded to obtain a video bitstream and send it to a conference terminal; the video bitstream is decoded by the conference terminal to obtain corresponding conference video pictures and display them.

[0145] Specifically, the video decoding method provided in the embodiments includes the following steps:

[0146] Step 802, obtain a video bitstream and decode it to obtain reference video frames and to-be-encoded features; the video bitstream is obtained after a video capture device captures a video segment, extracts features of to-be-encoded video frames in the video segment to obtain to-be-encoded features, and encodes the to-be-encoded features and reference video frames in the video segment.

[0147] At step 804, feature extraction is performed on the reference video frame to obtain reference features; and motion estimation is performed based on the to-be-encoded features and the reference features to obtain a motion estimation result.

[0148] At step 806, the reference video frame is morphed based on the motion estimation result by using a generator in the pre-trained generative model to generate a reconstructed video frame.

[0149] The generative model is obtained by using the model training method in the first aspect or the second aspect.

[0150] At step 808, the reconstructed video frame is displayed in a display interface.

[0151] The video decoding method provided in the embodiments of the present application is trained by the following method: the generator in the initial generative model generates a plurality of reconstructed sample frames corresponding to a plurality of continuous to-be-encoded sample frames, and, while discriminating the authenticity of a single reconstructed sample frame and a corresponding to-be-encoded sample frame, the authenticity of a spliced reconstructed sample frame spliced by the reconstructed sample frames in chronological order and the authenticity of a spliced to-be-encoded sample frame spliced by the to-be-encoded sample frames in chronological order are also discriminated, and then, the adversarial loss value is generated based on the discrimination results between the single sample frames (first discrimination result) and the discrimination results between the spliced sample frames (second discrimination result) to complete the training of the initial generative model. That is, in the embodiments of the present application, when discriminating the authenticity, not only the similarity between the reconstructed sample frame and the to-be-encoded sample frame in the spatial domain is considered, but also the similarity between the reconstructed sample frame and the to-be-encoded sample frame in the time domain is considered, that is, by comparing the similarity between the spliced to-be-encoded sample frame and the spliced reconstructed sample frame, whether the continuous relationship between the to-be-encoded sample frames is considered in the time domain. Therefore, based on the above discrimination results, the model is trained, and based on the trained generative model, the video frame is reconstructed, which can make the reconstructed video frame sequence consistent with the to-be-encoded video frame sequence in the time domain, improve the flicker and floating artifact phenomenon, and improve the quality of video reconstruction.

[0152] The video decoding method of the embodiments can be executed by any appropriate electronic device with data processing capability, including but not limited to servers, PC machines, etc.

[0153] Embodiment Six

[0154] Reference Figure 9 , Figure 9 FIG. 1 is a structural block diagram of a model training device according to an embodiment of the present application. Specifically, the model training device provided in the embodiments of the present application comprises:

[0155] The sample frame obtaining module 902 is configured to obtain a reference sample frame and a plurality of continuous to-be-encoded sample frames;

[0156] The reconstructed sample frame generating module 904 is configured to perform morphing processing on the reference sample frame by using a generator in the initial generation model to generate a reconstructed sample frame corresponding to each to-be-encoded sample frame.

[0157] The first discrimination result obtaining module 906 is configured to input each reconstructed sample frame and the corresponding to-be-encoded sample frame into a first discriminator in the initial generation model to obtain a first discrimination result.

[0158] The second discrimination result obtaining module 908 is configured to splice the to-be-encoded sample frames in chronological order to obtain spliced to-be-encoded sample frames, and splice the reconstructed sample frames to obtain spliced reconstructed sample frames; and input the spliced to-be-encoded sample frames and the spliced reconstructed sample frames into a second discriminator in the initial generation model to obtain a second discrimination result.

[0159] The training module 910 is configured to obtain an adversarial loss value based on the first discrimination result and the second discrimination result, and train the initial generation model based on the adversarial loss value to obtain a trained generation model.

[0160] Optionally, in some embodiments, the adversarial loss value includes a generative adversarial loss value, a spatial adversarial loss value, and a temporal adversarial loss value.

[0161] When performing the step of obtaining the adversarial loss value based on the first discrimination result and the second discrimination result, the training module 910 is specifically configured to:

[0162] obtain the generative adversarial loss value based on the first discrimination result of each reconstructed sample frame.

[0163] obtain the spatial adversarial loss value based on a difference value between the first discrimination result of each reconstructed sample frame and the first discrimination result of the corresponding to-be-encoded sample frame.

[0164] obtain the temporal adversarial loss value based on a difference value between the second discrimination result of the spliced to-be-encoded sample frames and the second discrimination result of the spliced reconstructed sample frames.

[0165] Optionally, in some embodiments, when performing the step of obtaining the generative adversarial loss value based on the first discrimination result of each reconstructed sample frame, the training module 910 is specifically configured to:

[0166] obtain a probability distribution of the first discrimination result of each reconstructed sample frame as a first reconstruction probability distribution of each reconstructed sample frame; and obtain the generative adversarial loss value based on an expected value of the first reconstruction probability distribution of each reconstructed sample frame.

[0167] The training module 910, in the step of obtaining the spatial adversarial loss value based on the difference value between the first discrimination result of each reconstructed sample frame and the first discrimination result of the corresponding to-be-encoded sample frame, is specifically configured to:

[0168] obtain the probability distribution of the first discrimination result of each to-be-encoded sample frame as the first to-be-encoded probability distribution of each to-be-encoded sample frame; and obtain the spatial adversarial loss value based on the expected difference value between the expected value of the first reconstruction probability distribution of each reconstructed sample frame and the expected value of the first to-be-encoded probability distribution of each to-be-encoded sample frame.

[0169] The training module 910, in the step of obtaining the temporal adversarial loss value based on the difference value between the second discrimination result of the spliced to-be-encoded sample frame and the second discrimination result of the spliced reconstructed sample frame, is specifically configured to:

[0170] obtain the probability distribution of the second discrimination result of the spliced reconstructed sample frame as the second reconstruction probability distribution; obtain the probability distribution of the second discrimination result of the spliced to-be-encoded sample frame as the second to-be-encoded probability distribution; and obtain the temporal adversarial loss value based on the expected difference value between the expected value of the second reconstruction probability distribution and the expected value of the second to-be-encoded probability distribution.

[0171] Optionally, in some embodiments, the model training apparatus further comprises:

[0172] a perception loss value obtaining module configured to, before training the initial generative model based on the adversarial loss value to obtain the trained generative model, generate a perception loss value based on each reconstructed sample frame and each to-be-encoded sample frame.

[0173] The training module 910, in the step of training the initial generative model based on the adversarial loss value to obtain the trained generative model, is specifically configured to:

[0174] train the initial generative model based on the adversarial loss value and the perception loss value to obtain the trained generative model.

[0175] Optionally, in some embodiments, the reconstructed sample frame generating module 904 is specifically configured to:

[0176] perform motion estimation on each to-be-encoded sample frame based on the reference sample frame to obtain a motion estimation result of each to-be-encoded sample frame;

[0177] for each to-be-encoded sample frame, input the reference sample frame and the motion estimation result of the to-be-encoded sample frame into a generator in the initial generative model, and perform deformation processing on the reference sample frame by the generator to generate a reconstructed sample frame corresponding to the to-be-encoded sample frame;

[0178] The model training apparatus further comprises:

[0179] a light flow loss value generation module configured to, before training the initial generation model based on the adversarial loss value and the perceptual loss value to obtain the trained generation model, input each of the to-be-encoded sample frames into the pre-trained motion prediction module to obtain a real motion result corresponding to each of the to-be-encoded sample frames, and generate a light flow loss value based on a difference between the motion estimation result and the real motion result of each of the to-be-encoded sample frames.

[0180] The training module 910 is specifically configured to, when performing the step of training the initial generation model based on the adversarial loss value and the perceptual loss value to obtain the trained generation model:

[0181] training the initial generation model based on the adversarial loss value, the perceptual loss value, and the light flow loss value to obtain the trained generation model.

[0182] Optionally, in some embodiments, the light flow loss value generation module is specifically configured to, when performing the step of generating the light flow loss value based on the difference between the motion estimation result and the real motion result of each of the to-be-encoded sample frames:

[0183] calculating, for each of the to-be-encoded sample frames, a difference between the motion estimation result and the real motion result of the to-be-encoded sample frame as a motion difference value corresponding to the to-be-encoded sample frame;

[0184] calculating a sum of the motion difference values as a total motion difference value;

[0185] calculating a ratio of the total motion difference value to a total number of the to-be-encoded sample frames as the light flow loss value.

[0186] Optionally, in some embodiments, the reconstructed sample frame generation module 904 is specifically configured to:

[0187] extracting, by the initial feature extraction model, reference sample features of the reference sample frame and to-be-encoded sample features of each of the to-be-encoded sample frames;

[0188] for each of the to-be-encoded sample frames, performing motion estimation based on the reference sample features and the to-be-encoded sample features of the to-be-encoded sample frame by the initial motion estimation model to obtain a motion estimation result, and inputting the reference sample frame and the motion estimation result into the initial generator to obtain a reconstructed sample frame corresponding to the to-be-encoded sample frame;

[0189] The training module 910 is specifically configured to, when performing the step of training the initial generation model based on the adversarial loss value to obtain the trained generation model:

[0190] training the initial feature extraction model, the initial motion estimation model, and the initial generation model based on the adversarial loss value to obtain the trained feature extraction model, the trained motion estimation model, and the trained generation model.

[0191] The model training apparatus of the embodiment is used to implement the corresponding model training method in the foregoing method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described here. In addition, the function implementation of each module in the model training apparatus of the embodiment can be referred to the description of the corresponding part in the foregoing method embodiments, which will not be described here either.

[0192] Embodiment Seven

[0193] With reference to Figure 10 , Figure 10 is a structural block diagram of a video encoding apparatus according to Embodiment Seven of the present application. Specifically, the video encoding apparatus provided by the embodiment comprises:

[0194] A video frame acquisition module 1002 is configured to acquire a reference video frame and a to-be-encoded video frame.

[0195] A to-be-encoded feature obtaining module 1004 is configured to perform feature extraction on the to-be-encoded video frame by using a pre-trained feature extraction model to obtain to-be-encoded features.

[0196] An encoding module 1006 is configured to encode the reference video frame and the to-be-encoded features respectively to obtain a bitstream.

[0197] The feature extraction model is obtained by the model training method of Embodiment Two.

[0198] The video encoding apparatus of the embodiment is used to implement the corresponding video encoding method in the foregoing method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described here. In addition, the function implementation of each module in the video encoding apparatus of the embodiment can be referred to the description of the corresponding part in the foregoing method embodiments, which will not be described here either.

[0199] Embodiment Eight

[0200] With reference to Figure 11 , Figure 11 is a structural block diagram of a video decoding apparatus according to Embodiment Eight of the present application. Specifically, the video decoding apparatus provided by the embodiment comprises:

[0201] A first decoding module 1102 is configured to acquire a video bitstream and decode the video bitstream to obtain a reference video frame and to-be-encoded features.

[0202] A first motion estimation module 1104 is configured to perform feature extraction on the reference video frame to obtain reference features, and perform motion estimation based on the to-be-encoded features and the reference features to obtain a motion estimation result.

[0203] The first reconstruction module 1106 is configured to perform morphing processing on the reference video frame based on the motion estimation result by using a generator in a pre-trained generative model to generate a reconstructed video frame.

[0204] The generative model is obtained by using the model training method in the first embodiment or the second embodiment.

[0205] The video decoding apparatus in this embodiment is used to implement the corresponding video decoding method in the foregoing method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described here again. In addition, the functions of each module in the video decoding apparatus in this embodiment can be implemented by referring to the descriptions of the corresponding parts in the foregoing method embodiments, which will not be described here again either.

[0206] Embodiment Nine

[0207] With reference to Figure 12 , Figure 12 FIG. 1 is a structural block diagram of a video decoding apparatus according to an embodiment of the present application. Specifically, the video decoding apparatus provided in this embodiment comprises:

[0208] The second decoding module 1202 is configured to acquire a video bitstream and decode the video bitstream to obtain a reference video frame and a to-be-encoded feature. The video bitstream is obtained by acquiring a video clip collected by a video collection device, extracting a to-be-encoded feature from the video clip, and encoding the to-be-encoded feature and a reference video frame in the video clip.

[0209] The second motion estimation module 1204 is configured to extract a reference feature from the reference video frame, and perform motion estimation based on the to-be-encoded feature and the reference feature to obtain a motion estimation result.

[0210] The second reconstruction module 1206 is configured to perform morphing processing on the reference video frame based on the motion estimation result by using a generator in a pre-trained generative model to generate a reconstructed video frame.

[0211] The display module 1208 is configured to display the reconstructed video frame in a display interface.

[0212] The generative model is obtained by using the model training method in the first embodiment or the second embodiment.

[0213] The video decoding apparatus in this embodiment is used to implement the corresponding video decoding method in the foregoing method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described here again. In addition, the functions of each module in the video decoding apparatus in this embodiment can be implemented by referring to the descriptions of the corresponding parts in the foregoing method embodiments, which will not be described here again either.

[0214] Embodiment Ten

[0215] Referring to Figure 13 FIG. 10 shows a structural schematic diagram of an electronic device according to Embodiment Ten of the present application, and the specific implementation of the electronic device is not limited by the embodiments of the present application.

[0216] As shown in Figure 13 , the conference terminal can include a processor 1302, a communications interface 1304, a memory 1306, and a communications bus 1308.

[0217] Among them:

[0218] The processor 1302, the communications interface 1304, and the memory 1306 complete mutual communication through the communications bus 1308.

[0219] The communications interface 1304 is configured to communicate with other electronic devices or servers.

[0220] The processor 1302 is configured to execute the program 1310, and specifically can execute the related steps in the above-mentioned magic training method, video encoding method, or video decoding method embodiments.

[0221] Specifically, the program 1310 can include program code including computer operation instructions.

[0222] The processor 1302 can be a CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement one or more embodiments of the present application. One or more processors included in the smart device can be the same type of processor, such as one or more CPUs; or can be different types of processors, such as one or more CPUs and one or more ASICs.

[0223] The memory 1306 is configured to store the program 1310. The memory 1306 can include a high-speed RAM memory, and can also include a non-volatile memory such as at least one disk memory.

[0224] The program 1310 can be specifically used for causing the processor 1302 to perform the following operations: obtaining a reference sample frame and a plurality of continuous to-be-encoded sample frames; performing morphing processing on the reference sample frame by a generator in an initial generation model to generate a reconstructed sample frame corresponding to each to-be-encoded sample frame; inputting each reconstructed sample frame and the corresponding to-be-encoded sample frame into a first discriminator in the initial generation model to obtain a first discrimination result; splicing the to-be-encoded sample frames in a time stamp order to obtain a spliced to-be-encoded sample frame, and splicing the reconstructed sample frames to obtain a spliced reconstructed sample frame; inputting the spliced to-be-encoded sample frame and the spliced reconstructed sample frame into a second discriminator in the initial generation model to obtain a second discrimination result; obtaining an adversarial loss value based on the first discrimination result and the second discrimination result, and training the initial generation model based on the adversarial loss value to obtain a trained generation model.

[0225] Alternatively,

[0226] The program 1310 can be specifically used for causing the processor 1302 to perform the following operations: obtaining a reference video frame and a to-be-encoded video frame; performing feature extraction on the to-be-encoded video frame by a pre-trained feature extraction model to obtain to-be-encoded features; encoding the reference video frame and the to-be-encoded features respectively to obtain a bitstream; wherein the generation model is obtained by the model training method of the second aspect.

[0227] Alternatively,

[0228] The program 1310 can be specifically used for causing the processor 1302 to perform the following operations: obtaining a video bitstream and decoding to obtain a reference video frame and to-be-encoded features; performing feature extraction on the reference video frame to obtain reference features; and performing motion estimation based on the to-be-encoded features and the reference features to obtain a motion estimation result; performing morphing processing on the reference video frame based on the motion estimation result by a generator in a pre-trained generation model to generate a reconstructed video frame; wherein the generation model is obtained by the model training method of the first aspect or the second aspect.

[0229] Alternatively,

[0230] The program 1310 can be specifically used to enable the processor 1302 to perform the following operations: obtaining a video bitstream and decoding to obtain reference video frames and to-be-encoded features; the video bitstream is obtained after a video segment captured by a video capture device is obtained, and the to-be-encoded features are obtained after feature extraction is performed on to-be-encoded video frames in the video segment; the reference features are obtained by performing feature extraction on the reference video frames; and the motion estimation result is obtained by performing motion estimation based on the to-be-encoded features and the reference features; the generator in the pre-trained generative model is used to perform morphing processing on the reference video frames based on the motion estimation result to generate a reconstructed video frame; and the reconstructed video frame is displayed in a display interface; wherein the generative model is obtained by using the model training method in the first aspect or the second aspect.

[0231] The specific implementation of each step in the program 1310 can refer to the corresponding description in the corresponding steps and units in the foregoing model training method, video encoding method, or video decoding method embodiments, and will not be described herein again. It can be clearly understood by those skilled in the art that, for the convenience and brevity of description, the specific working processes of the devices and modules described above can refer to the corresponding process descriptions in the foregoing method embodiments, and will not be described herein again.

[0232] Through the electronic device in this embodiment, the generator in the initial generative model generates a plurality of reconstructed sample frames corresponding to a plurality of continuous to-be-encoded sample frames, and simultaneously performs authenticity identification on a single reconstructed sample frame and a corresponding to-be-encoded sample frame, and performs authenticity identification on a spliced reconstructed sample frame spliced according to the time stamp order of the reconstructed sample frames and a spliced to-be-encoded sample frame spliced according to the time stamp order of the to-be-encoded sample frames, and further generates an adversarial loss value based on the identification result (first identification result) between the single sample frames and the identification result (second identification result) between the spliced sample frames, to complete the training of the initial generative model. That is, in the embodiments of the present application, when performing authenticity identification, not only the similarity degree between the reconstructed sample frames and the to-be-encoded sample frames in the spatial domain is considered, but also the similarity degree between the reconstructed sample frames and the to-be-encoded sample frames in the time domain is considered, that is, by comparing the similarity degree between the spliced to-be-encoded sample frames and the spliced reference sample frames, whether the continuous relationship between the to-be-encoded sample frames is considered between the continuous reconstructed sample frames in the time domain. Therefore, based on the above identification results for model training and based on the video frame reconstruction performed by using the trained generative model, the consistency between the reconstructed video frame sequence and the to-be-encoded video frame sequence in the time domain can be maintained, the flicker and floating artifact phenomenon can be improved, and the quality of video reconstruction can be improved.

[0233] The embodiment of the present application further provides a computer program product comprising computer instructions for instructing a computing device to perform operations corresponding to any of the above-mentioned method embodiments.

[0234] It should be noted that, according to the needs of implementation, each component / step described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or parts of the operation of the components / steps can be combined into a new component / step, to achieve the purpose of the embodiments of the present application.

[0235] The above-mentioned method according to the embodiments of the present application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium such as a CD ROM, a RAM, a floppy disk, a hard disk or an optical disk, or computer code downloaded over a network from a remote recording medium or a non-transitory machine-readable medium and stored in a local recording medium, so that the method described herein can be processed by such software on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware such as an ASIC or an FPGA. It can be understood that the computer, the processor, the microprocessor controller or the programmable hardware includes a storage component (for example, RAM, ROM, flash memory, etc.) that can store or receive software or computer code, which, when accessed and executed by the computer, the processor or the hardware, implements the model training method, the video encoding method or the video decoding method described herein. In addition, when a general-purpose computer accesses the code for implementing the model training method, the video encoding method or the video decoding method shown herein, the execution of the code will convert the general-purpose computer into a special-purpose computer for executing the model training method, the video encoding method or the video decoding method shown herein.

[0236] Those of ordinary skill in the art can realize that the units and method steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of the present application.

[0237] The above embodiments are only used to illustrate the embodiments of the present application, but not to limit the embodiments of the present application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application, therefore all equivalent technical solutions belong to the scope of the embodiments of the present application, and the patent protection scope of the embodiments of the present application should be defined by the claims.

Claims

1. A model training method, comprising: Acquire a reference sample frame and multiple consecutive sample frames to be encoded; The generator in the initial generation model deforms the reference sample frame to generate the reconstructed sample frame corresponding to each sample frame to be encoded. Each reconstructed sample frame and its corresponding sample frame to be encoded are input into the first discriminator in the initial generation model to obtain the first discrimination result. According to the timestamp sequence, each sample frame to be encoded is spliced ​​to obtain a spliced ​​pre-encoded sample frame, and each reconstructed sample frame is spliced ​​to obtain a spliced ​​reconstructed sample frame; the spliced ​​pre-encoded sample frame and the spliced ​​reconstructed sample frame are input into the second discriminator in the initial generation model to obtain a second discrimination result; Based on the first identification result and the second identification result, an adversarial loss value is obtained, and the initial generative model is trained based on the adversarial loss value to obtain a trained generative model; The generative model is used to deform the reference video frame based on the motion estimation results of the reference video frame and the feature to be encoded, and generate a reconstructed video frame; the reference video frame and the feature to be encoded are obtained by decoding the video bitstream.

2. The method according to claim 1, wherein, The adversarial loss values ​​include: generation adversarial loss values, spatial adversarial loss values, and temporal adversarial loss values; The process of obtaining the adversarial loss value based on the first identification result and the second identification result includes: Based on the first discrimination result of each reconstructed sample frame, the generative adversarial loss value is obtained; Based on the difference between the first discrimination result of each reconstructed sample frame and the first discrimination result of the corresponding sample frame to be encoded, the spatial adversarial loss value is obtained. The temporal adversarial loss value is obtained based on the difference between the second discrimination result of the stitched reception coded sample frame and the second discrimination result of the stitched reconstructed sample frame.

3. The method according to claim 2, wherein, The generation adversarial loss value is obtained based on the first discrimination result of each reconstructed sample frame, including: Obtain the probability distribution of the first discrimination result of each reconstructed sample frame as the first reconstruction probability distribution of each reconstructed sample frame; and obtain the generative adversarial loss value based on the expected value of the first reconstruction probability distribution of each reconstructed sample frame. The spatial adversarial loss value is obtained based on the difference between the first discrimination result of each reconstructed sample frame and the first discrimination result of the corresponding sample frame to be encoded, including: Obtain the probability distribution of the first discrimination result of each sample frame to be encoded, as the first probability distribution to be encoded of each sample frame; and obtain the spatial adversarial loss value based on the expected difference between the expected value of the first reconstruction probability distribution of each reconstructed sample frame and the expected value of the first probability distribution to be encoded of each sample frame. The difference between the second discrimination result of the spliced ​​reception coded sample frame and the second discrimination result of the spliced ​​reconstructed sample frame is used to obtain the temporal adversarial loss value, including: Obtain the probability distribution of the second discrimination result of the spliced ​​and reconstructed sample frame as the second reconstruction probability distribution; obtain the probability distribution of the second discrimination result of the spliced ​​and unencoded sample frame as the second unencoded probability distribution; and obtain the temporal adversarial loss value based on the expected difference between the expected value of the second reconstruction probability distribution and the expected value of the second unencoded probability distribution.

4. The method according to claim 1, wherein, Before training the initial generative model based on the adversarial loss value to obtain the trained generative model, the method further includes: Based on each reconstructed sample frame and each sample frame to be encoded, a perceptual loss value is generated. The process of training the initial generative model based on the adversarial loss value to obtain the trained generative model includes: Based on the adversarial loss value and the perceptual loss value, the initial generative model is trained to obtain the trained generative model.

5. The method according to claim 4, wherein, The step of deforming the reference sample frame using the generator in the initial generation model to generate reconstructed sample frames corresponding to each sample frame to be encoded includes: Based on the reference sample frame, motion estimation is performed on each sample frame to be encoded to obtain the motion estimation result of each sample frame to be encoded. For each sample frame to be encoded, the motion estimation results of the reference sample frame and the sample frame to be encoded are input into the generator in the initial generation model. The generator performs deformation processing on the reference sample frame to generate the reconstructed sample frame corresponding to the sample frame to be encoded. Before training the initial generative model based on the adversarial loss value and the perceptual loss value to obtain the trained generative model, the method further includes: Each sample frame to be encoded is input into a pre-trained motion prediction module to obtain the real motion result corresponding to each sample to be encoded. Optical flow loss values ​​are generated based on the difference between the motion estimation results and the actual motion results of each sample frame to be encoded. The step of training the initial generative model based on the adversarial loss value and the perceptual loss value to obtain the trained generative model includes: Based on the adversarial loss value, the perceptual loss value, and the optical flow loss value, the initial generative model is trained to obtain the trained generative model.

6. The method according to claim 5, wherein, The optical flow loss value is generated based on the difference between the motion estimation result and the actual motion result of each sample frame to be encoded, including: For each sample frame to be encoded, the difference between the motion estimation result and the actual motion result of the sample frame to be encoded is calculated, and the difference is used as the motion difference corresponding to the sample frame to be encoded. The sum of the motion differences is calculated as the optical flow loss value.

7. The method according to claim 1, wherein, The step of deforming the reference sample frame using the generator in the initial generation model to generate reconstructed sample frames corresponding to each sample frame to be encoded includes: The reference sample features of the reference sample frame and the sample features to be encoded of each sample frame to be encoded are extracted using the initial feature extraction model. For each sample frame to be encoded, motion estimation is performed based on the reference sample features and the features of the sample frame to be encoded using an initial motion estimation model to obtain a motion estimation result; the reference sample frame and the motion estimation result are input into the initial generator to obtain the reconstructed sample frame corresponding to the sample frame to be encoded. The process of training the initial generative model based on the adversarial loss value to obtain the trained generative model includes: The initial feature extraction model, the initial motion estimation model, and the initial generation model are trained based on the adversarial loss value to obtain the trained feature extraction model, motion estimation model, and generation model.

8. A video encoding method, comprising: Obtain reference video frames and video frames to be encoded; The pre-trained feature extraction model is used to extract features from the video frames to be encoded, thereby obtaining the features to be encoded. The reference video frame and the feature to be encoded are encoded separately to obtain a bitstream; The feature extraction model is obtained through the model training method described in claim 7.

9. A video decoding method, comprising: The video bitstream is acquired and decoded to obtain reference video frames and features to be encoded; Feature extraction is performed on the reference video frame to obtain reference features; Based on the features to be encoded and the reference features, motion estimation is performed to obtain the motion estimation result; The generator in the pre-trained generative model performs deformation processing on the reference video frame based on the motion estimation results to generate a reconstructed video frame. The generative model is obtained by the model training method described in any one of claims 1-7.

10. A video decoding method, applied to a conferencing terminal device, comprising: The video bitstream is acquired and decoded to obtain reference video frames and features to be encoded; The video bitstream is obtained by acquiring a video segment captured by a video acquisition device, extracting features from the video frames to be encoded in the video segment to obtain the features to be encoded, and then encoding the features to be encoded and the reference video frames in the video segment. Feature extraction is performed on the reference video frame to obtain reference features; Based on the features to be encoded and the reference features, motion estimation is performed to obtain the motion estimation result; The generator in the pre-trained generative model performs deformation processing on the reference video frame based on the motion estimation results to generate a reconstructed video frame. The reconstructed video frames are displayed in the display interface; The generative model is obtained by the model training method described in any one of claims 1-7.

11. An electronic device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform an operation corresponding to the model training method as described in any one of claims 1-7, or an operation corresponding to the video encoding method as described in claim 8, or an operation corresponding to the video decoding method as described in claim 9 or 10.

12. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the model training method as described in any one of claims 1-7, or implements the video encoding method as described in claim 8, or the video decoding method as described in claim 9 or 10.

13. A computer program product comprising computer instructions that instruct a computing device to perform an operation corresponding to the model training method as described in any one of claims 1-7, or to perform an operation corresponding to the video encoding method as described in claim 8, or to perform an operation corresponding to the video decoding method as described in claim 9 or 10.

Images