Video encoding method, decoding method, electronic device and storage medium

Abstract

The embodiments of the present application provide a video encoding method, decoding method and device. The video encoding method includes: obtaining an original reference video frame and an original target video frame to be encoded; adjusting the resolution of the original target video frame to obtain an adjusted target video frame with a first preset resolution; and extracting features of the adjusted target video frame through a feature extraction network corresponding to the first preset resolution to obtain target features; respectively encoding the original reference video frame and the target features to obtain a video bitstream, so as to reconstruct the video frame based on the video bitstream and generate a reconstructed video frame with the same resolution as the original target video frame. In the embodiments of the present application, only one set of neural network models is needed to perform encoding and decoding operations for original target video frames of various resolutions, which has a wider scope of application and a simpler and more convenient operation process.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a video encoding method, a decoding method, an electronic device, and a storage medium. Background Art

[0002] With the continuous development of computer technology, people's lifestyles have also undergone tremendous changes. For example, in daily work and life, people's demand for video conferencing and video live broadcasting is increasing.

[0003] Video encoding and decoding is the key to realizing video conferencing and live video broadcasting. With the continuous development of machine learning, a codec method based on deep video generation can be used to encode and decode videos (especially facial videos). This method mainly uses a neural network model to deform the reference frame based on the motion of the frame to be encoded to obtain a reconstructed frame corresponding to the frame to be encoded. The above method performs encoding and decoding operations on video frames in an end-to-end manner and realizes the reconstruction of video frames.

[0004] The above-mentioned encoding and decoding method based on deep video generation, a complete set of trained neural network models, can usually only be used to reconstruct video frames with a fixed resolution for the video frames to be encoded, and is not compatible with multiple different resolutions. However, in actual application, due to factors such as network bandwidth, the resolution of the video frames to be encoded may be multiple, rather than a fixed resolution. At this time, only a corresponding neural network model can be trained for each resolution, and then in the application stage, the corresponding network model is loaded according to the actual resolution of the video frame to be encoded. Such operation is complicated and very inconvenient. Summary of the invention

[0005] In view of this, embodiments of the present application provide a video encoding method, a decoding method, an electronic device and a storage medium to at least partially solve the above problems.

[0006] According to a first aspect of an embodiment of the present application, a video encoding method is provided, including:

[0007] Obtaining an original reference video frame and an original target video frame to be encoded;

[0008] Adjusting the resolution of the original target video frame to obtain an adjusted target video frame with a first preset resolution; and extracting features from the adjusted target video frame through a feature extraction network corresponding to the first preset resolution to obtain target features;

[0009] The original reference video frame and the target feature are encoded respectively to obtain a video bitstream, so as to reconstruct the video frame based on the video bitstream and generate a reconstructed video frame with the same resolution as the original target video frame.

[0010] According to a second aspect of an embodiment of the present application, a video decoding method is provided, including:

[0011] Obtain the video bitstream and decode it to obtain the original reference video frame and target features;

[0012] Adjusting the resolution of the original reference video frame to obtain an adjusted reference video frame with a first preset resolution; and extracting features from the adjusted reference video frame through a feature extraction network to obtain reference features;

[0013] Performing motion estimation based on the reference feature and the target feature through a motion estimation network to obtain a motion estimation result;

[0014] A reconstructed video frame having the same resolution as the original reference video frame is generated by generating a network based on the motion estimation result and the original reference video frame.

[0015] According to a third aspect of an embodiment of the present application, a video encoding method is provided, including:

[0016] Obtaining an original reference video frame and an original target video frame to be encoded;

[0017] Adjusting the resolution of the original target video frame to obtain an adjusted target video frame with a first preset resolution; and extracting features from the adjusted target video frame through a feature extraction network corresponding to the first preset resolution to obtain target features;

[0018] Encode the original reference video frame and the target feature respectively to obtain a video bit stream, so that a decoding end device decodes the video bit stream to obtain the original reference video frame and the target feature; adjust the resolution of the original reference video frame to obtain an adjusted reference video frame with a first preset resolution; and extract features from the adjusted reference video frame through a feature extraction network to obtain a reference feature; perform motion estimation based on the reference feature and the target feature through a motion estimation network to obtain a motion estimation result; and generate a reconstructed video frame with the same resolution as the original reference video frame based on the motion estimation result and the original reference video frame through a generation network.

[0019] According to a fourth aspect of an embodiment of the present application, a video encoding method is provided, including:

[0020] Obtaining the original video clips captured by the video capture device;

[0021] Determining an original reference video frame and an original target video frame to be encoded from the original video clip;

[0022] Adjusting the resolution of the original target video frame to obtain an adjusted target video frame with a first preset resolution; and extracting features from the adjusted target video frame through a feature extraction network corresponding to the first preset resolution to obtain target features;

[0023] Encoding the original reference video frame and the target feature respectively to obtain a video bitstream;

[0024] The video bit stream is sent to a conference terminal device, so that the conference terminal device reconstructs a video frame based on the video bit stream, generates a reconstructed video frame with the same resolution as the original target video frame, and displays the reconstructed video frame.

[0025] According to the fifth aspect of the embodiments of the present application, there is provided an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the video encoding method described in the first aspect, the third aspect or the fourth aspect, or operations corresponding to the video decoding method described in the second aspect.

[0026] According to the sixth aspect of the embodiments of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the video encoding method as described in the first aspect, the third aspect or the fourth aspect is implemented, or the video decoding method as described in the second aspect is implemented.

[0027] According to the seventh aspect of an embodiment of the present application, a computer program product is provided, comprising computer instructions, wherein the computer instructions instruct a computing device to perform operations corresponding to the video encoding method described in the first aspect, the third aspect, or the fourth aspect, or operations corresponding to the video decoding method described in the second aspect.

[0028] According to the video encoding method and decoding method provided by the embodiment of the present application, in the encoding stage, after obtaining the original target video frame to be encoded, the original target video frame is uniformly resolved through the resolution adjustment operation, and the original target video frame is converted into an adjusted target video frame with a fixed resolution (first preset resolution), so that even if the original target video frame has a variety of resolutions, the video frame with a fixed resolution is still input to the feature extraction network. In this way, there is no need to train multiple feature extraction networks for different resolutions, and only one feature extraction network corresponding to the first preset resolution (a feature extraction network for extracting features from a video frame with a first preset resolution) is needed to realize the encoding of multiple original target video frames with different resolutions, which has a wider scope of application and higher versatility. At the same time, the operation is simpler and more convenient. In addition, correspondingly, in the decoding stage, the original reference video frame is also uniformly resolved, and the original reference video frame is converted into an adjusted reference video frame with a fixed resolution, thereby obtaining reference features and target features with a fixed resolution, so that the features with a fixed resolution are finally input to the motion estimation network and the generation network, and then only one motion estimation network and one generation network are needed to realize the decoding of target video frames with various resolutions. In summary, the embodiments of the present application only require one set of neural network models to perform encoding and decoding operations for original target video frames of various resolutions, which has a wider scope of application and a simpler and more convenient operation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0030] Figure 1 A schematic diagram of a framework of a coding and decoding method based on deep video generation;

[0031] Figure 2 is a flowchart of a video encoding method according to Embodiment 1 of the present application;

[0032] Figure 3 for Figure 2 A schematic diagram of an example scenario in the illustrated embodiment;

[0033] Figure 4 is a flowchart of a video decoding method according to Embodiment 2 of the present application;

[0034] Figure 5 is a flowchart of a video decoding method according to Embodiment 3 of the present application;

[0035] Figure 6 for Figure 5 A schematic diagram of an example scenario in the illustrated embodiment;

[0036] Figure 7 is a flowchart of a video decoding method according to Embodiment 4 of the present application;

[0037] Figure 8 for Figure 7 A schematic diagram of an example scenario in the illustrated embodiment;

[0038] Fig. 9 is a flowchart of a video decoding method according to Embodiment 5 of the present application;

[0039] Fig.10 for Fig. 9 A schematic diagram of an example scenario in the illustrated embodiment;

[0040] Fig.11 is a flowchart of a video decoding method according to Embodiment 6 of the present application;

[0041] Fig.12 for Fig.11 A schematic diagram of an example scenario in the illustrated embodiment;

[0042] Fig.13 is a flowchart of a video encoding method according to Embodiment 7 of the present application;

[0043] Fig.14 is a flowchart of a video encoding method according to Embodiment 8 of the present application;

[0044] Fig.15 is a structural block diagram of a video encoding device according to Embodiment 9 of the present application;

[0045] Fig.16 is a structural block diagram of a video decoding device according to Embodiment 10 of the present application;

[0046] Fig.17 This is a schematic diagram of the structure of an electronic device according to the eleventh embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0048] See also Figure 1 , Figure 1 Schematic diagram of the framework of the encoding and decoding method based on deep video generation. The main principle of this method is to deform the reference frame based on the motion of the frame to be encoded to obtain a reconstructed frame corresponding to the frame to be encoded. Figure 1 The basic framework of the encoding and decoding method based on deep video generation is explained:

[0049] In the first step, the encoding stage, the encoder uses a feature extractor to extract the target key point information of the target facial video frame to be encoded, and encodes the target key point information; at the same time, the traditional image coding method (such as VVC, HEVC, etc.) is used to encode the reference facial video frame.

[0050] The second step is the decoding stage. The motion estimation module in the decoder extracts the reference key point information of the reference facial video frame through the key point extractor; and performs dense motion estimation based on the reference key point information and the target key point information to obtain a dense motion estimation map and an occlusion map. The dense motion estimation map represents the relative motion relationship between the target facial video frame and the reference facial video frame in the feature domain represented by the key point information; the occlusion map represents the degree of occlusion of each pixel in the target facial video frame.

[0051] The third step is the decoding stage. The generation module in the decoder deforms the reference facial video frame based on the dense motion estimation map to obtain the deformation result, and then multiplies the deformation result with the occlusion map to output the reconstructed facial video frame.

[0052] Figure 1 The method shown is based on a neural network model composed of a feature extractor (feature extraction module), a motion estimation module and a generation module to perform encoding and decoding operations on video frames. After the training of each neural network module in the above model is completed, its internal parameters and input and output data resolution sizes remain fixed. Therefore, in the inference stage, a set of trained neural network models can only be used to reconstruct video frames to be encoded with a specific resolution, and is not compatible with multiple different resolutions.

[0053] However, in actual applications, due to factors such as network bandwidth, the resolution of the video frame to be encoded may be multiple, rather than a fixed resolution. At this time, the above-mentioned encoding and decoding method based on deep video generation can only train a corresponding neural network model for each resolution, and then load the corresponding model according to the actual resolution of the video frame to be encoded in the inference stage. Such operation is complicated and very inconvenient.

[0054] In the embodiment of the present application, the original target video frame is uniformly resolved through the resolution adjustment operation, and the original target video frame is converted into an adjusted target video frame with a specific resolution, and then subsequent operations such as feature extraction are performed to encode and decode the downsampled target video frame, and finally output a reconstructed video frame with the same resolution as the original target video frame. In this way, even if the original target video frame has a variety of resolutions, the data finally input to the feature extraction network, the motion estimation network, and the generation network still has a fixed resolution. Therefore, there is no need to train multiple neural networks for different resolutions, and only one neural network is needed to realize the encoding of multiple original target video frames with different resolutions, which has a wider scope of application and higher versatility. At the same time, the operation is simpler and more convenient.

[0055] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.

[0056] Embodiment 1

[0057] Reference Figure 2 , Figure 2 The following is a flowchart of a video encoding method according to Embodiment 1 of the present application. Specifically, the video encoding method provided in this embodiment includes the following steps:

[0058] Step 202: Acquire an original reference video frame and an original target video frame to be encoded.

[0059] Specifically, the original reference video frame and the original target video frame in the present application are video frames with the same resolution, and both the original reference video frame and the original target video frame can be facial video frames. In addition, in the embodiments of the present application, the resolution of the original reference video frame and the original target video frame is not limited.

[0060] Furthermore, in order to obtain higher quality reconstructed video frames when subsequently reconstructing the video frames, the original reference frames and the original target video frames to be encoded can be selected from the same video clip. That is to say, in this step, the original reference video frames and the original target video frames to be encoded originating from the same video clip can be obtained.

[0061] Step 204: Adjust the resolution of the original target video frame to obtain an adjusted target video frame with a first preset resolution.

[0062] Specifically, in the embodiment of the present application, there is no limitation on the specific value of the first preset resolution, and it can be customized according to the requirements of the computing power resources, network status, and encoding and decoding timeliness of the encoding and decoding device.

[0063] Specifically, in order to reduce the bit rate, the first preset resolution may be set to a lower value. Correspondingly, in this step, the original target video frame may be downsampled to obtain an adjusted target video frame with the first preset resolution.

[0064] Optionally, in some of the embodiments, the adjusted target video frame can be obtained in the following manner: determining a first target scaling factor based on the resolution of the original target video frame; scaling the original target video frame using the first target scaling factor to obtain an adjusted target video frame having a first preset resolution; wherein the first target scaling factor can be determined based on the relationship between the resolution of the original target video frame and the first preset resolution, specifically, the ratio of the resolution of the original target video frame to the first preset resolution can be determined as the first target scaling factor.

[0065] Furthermore, for the several resolutions that may appear in the original target video frame, the scaling factors corresponding to the resolutions can be calculated in advance and put into a first scaling factor sequence. Thereafter, after acquiring the original target video frame, the first target scaling factor corresponding to the resolution of the original target video frame can be determined from the above-mentioned preset first scaling factor sequence based on the preset correspondence between the resolution and the scaling factor.

[0066] Step 206: extract features from the adjusted target video frame using a feature extraction network corresponding to the first preset resolution to obtain target features.

[0067] In the embodiment of the present application, the adjusted target video frame can be feature extracted by means of a machine learning model (feature extraction network) to obtain target features. Specifically, the adjusted target video frame can be input into a pre-trained feature extraction network so that the feature extraction network outputs the target features of the adjusted target video frame.

[0068] The feature extraction network in the embodiment of the present application is a feature extraction network corresponding to the first preset resolution, that is, a network model for extracting features from video frames having the first preset resolution.

[0069] The target feature may be information used to characterize the features of the adjusted target video frame. For a facial video frame, the target feature may be a target key point feature or a target compact feature, wherein the target key point feature characterizes the feature information of a preset key point in the adjusted target video frame; the target compact feature characterizes key information such as the position information, posture information, and expression information of the facial features in the adjusted target video frame.

[0070] In the embodiment of the present application, there is no limitation on the structure and parameters of the feature extraction network, which can be set according to actual needs. For example, the feature extraction network can be a U-Net network composed of a convolutional layer and a generalized division normalization layer, and so on.

[0071] Step 208 , respectively encode the original reference video frame and the target feature to obtain a video bitstream, and reconstruct the video frame based on the video bitstream to generate a reconstructed video frame with the same resolution as the original target video frame.

[0072] Specifically, for the original reference video frame, relatively small quantization distortion can be used for encoding, and the encoding process retains the complete data of the original reference video frame. For example, the original reference video frame can be encoded by using the universal video coding (VVC) method. For the target features, quantization and entropy coding can be used for encoding.

[0073] Furthermore, in some embodiments of the present application, in order to further reduce the bit rate of video encoding, the resolution of the original reference video frame can also be adjusted to obtain an adjusted reference video frame, and feature extraction can be performed on the adjusted reference video frame to obtain a reference feature, and then a differential operation is performed on the target feature and the reference feature, and the difference obtained by the differential operation is encoded to form a video bit stream.

[0074] Compared with the method of directly encoding based on target features, in the above method, encoding is performed based on the difference between the target features and the reference features to obtain a video bit stream. Obviously, the data amount of the difference between the target features and the reference features is smaller than the data amount of the target features themselves. Therefore, encoding based on the difference between the target features and the reference features can effectively reduce the bit rate of video encoding.

[0075] See also Figure 3 , Figure 3 This is a schematic diagram of a scene corresponding to the first embodiment of the present application. Figure 3 The schematic diagram shown in the figure illustrates an embodiment of the present application by taking a specific scenario as an example:

[0076] An original reference video frame and an original target video frame to be encoded are obtained respectively, wherein the resolutions of the original reference video frame and the original target video frame are both: W×H (the number of pixels contained in a unit size in the width direction is W, and the number of pixels contained in a unit size in the height direction is H); from a first scaling factor sequence s={s1, s2, s3, ..., s n}, and then downsample the original target video frame based on the first target scaling factor to obtain an adjusted target video frame with W1×H1 (first preset resolution); and extract features from the adjusted target video frame through a feature extraction network to obtain target features; respectively encode the original reference video frame and the target features to obtain a video bitstream, wherein the target features are encoded by entropy coding and the original reference video frame is encoded by VVC.

[0077] In an embodiment of the present application, in the encoding stage, after the original target video frame to be encoded is obtained, the resolution of the original target video frame is unified through a resolution adjustment operation, and the original target video frame is converted into an adjusted target video frame with a fixed resolution (a first preset resolution), so that even if the original target video frame has a variety of resolutions, the video frame finally input into the feature extraction network is still a video frame with a fixed resolution. In this way, there is no need to train multiple feature extraction networks for different resolutions, but only one feature extraction network corresponding to the first preset resolution (a feature extraction network for extracting features from video frames with the first preset resolution) is required to realize the encoding of original target video frames with multiple different resolutions, which has a wider scope of application and higher versatility. At the same time, the operation is simpler and more convenient.

[0078] The video encoding method provided in the first embodiment of the present application can be executed by a video encoding end (encoder) to encode video files with different resolutions, especially facial video files, to achieve compression of the digital bandwidth of the video files. It can be applied to a variety of different scenarios, such as: conventional storage and streaming of video games involving faces and having various resolutions, specifically: the game video frames can be encoded by the video encoding method provided in the embodiment of the present application to form a corresponding video bit stream for storage and transmission in video streaming services or other similar applications; another example: low-latency scenarios such as video conferencing and live video broadcasting, specifically: the facial video data with various resolutions collected by the video acquisition device can be encoded by the video encoding method provided in the embodiment of the present application to form a corresponding video bit stream, and sent to the conference terminal, and the video bit stream is decoded by the conference terminal to obtain the corresponding facial video picture; another example: virtual reality scenario, the facial video data with various resolutions collected by the video acquisition device can be encoded by the facial video encoding method provided in the embodiment of the present application to form a corresponding video bit stream, and sent to virtual reality related equipment (such as VR virtual glasses, etc.), the video bit stream is decoded by the VR device to obtain the corresponding facial video picture, and the corresponding VR function is implemented based on the facial video picture, and so on.

[0079] Embodiment 2

[0080] Reference Figure 4 , Figure 4 1 is a flowchart of a video decoding method according to Embodiment 2 of the present application. Specifically, the video decoding method provided in this embodiment includes the following steps:

[0081] Step 402, obtain and decode the video bit stream to obtain the original reference video frame and target features.

[0082] The target feature is obtained by extracting features from an adjusted target video frame, wherein the adjusted target video frame is a video frame with a first preset resolution obtained by adjusting the resolution of an original target video frame.

[0083] Step 404: Adjust the resolution of the original reference video frame to obtain an adjusted reference video frame with a first preset resolution.

[0084] In this step, the specific manner of adjusting the resolution of the original reference video frame is the same as the specific manner of adjusting the resolution of the original target video frame in step 204 of embodiment 1. Specifically, the original reference video frame may be downsampled to obtain an adjusted reference video frame having a first preset resolution.

[0085] Step 406: extract features from the adjusted reference video frame through a feature extraction network to obtain reference features.

[0086] In this step, the specific method of obtaining the reference feature can refer to the specific method of obtaining the target feature in step 206 of the first embodiment, and will not be repeated here.

[0087] Step 408: Perform motion estimation based on the reference features and the target features through a motion estimation network to obtain a motion estimation result.

[0088] Specifically, in one method, sparse motion estimation can be performed based on the reference features and the target features to obtain a sparse motion estimation map, and the obtained sparse motion estimation map is directly used as the motion estimation result. The sparse motion estimation map represents the relative motion relationship between the original reference video frame corresponding to the reference feature and the original target video frame corresponding to the target feature in a preset sparse feature domain.

[0089] In another way, after obtaining the sparse motion estimation map, dense motion estimation can be performed again based on the sparse motion estimation map and the initial reconstructed video frame generated by the initial reference video frame, thereby obtaining a dense motion estimation map and an occlusion map as the final motion estimation result; wherein the dense motion estimation map represents the relative motion relationship between the original target video frame and the original reference video frame in a preset dense feature domain; and the occlusion map represents the degree to which each pixel in the original target video frame is occluded.

[0090] Compared with the above two methods, the first method has a simple calculation process, so the calculation efficiency is high and the motion estimation result can be obtained quickly; the second method obtains the relative motion relationship between the original target video frame and the original reference facial video frame in a denser feature domain. This relative motion relationship is more accurate than the relative motion relationship represented by the sparse motion estimation graph.

[0091] Step 410 , generating a reconstructed video frame having the same resolution as the original reference video frame based on the motion estimation result and the original reference video frame through a generation network.

[0092] Specifically, the generation network performs deformation processing on the original reference video frame based on the motion estimation result obtained in step 408 to obtain a deformation processing result, and outputs a reconstructed video frame based on the deformation processing result.

[0093] In the embodiment of the present application, during the decoding stage, the original reference video frame is also uniformed in resolution, and the original reference video frame is converted into an adjusted reference video frame with a fixed resolution, thereby obtaining reference features and target features with a fixed resolution, so that the features finally input to the motion estimation network and the generation network are also features with a fixed resolution, and thus only one motion estimation network and one generation network are required to achieve decoding in various target video frame scenarios with different resolutions. In the embodiment of the present application, only one set of neural network models is required to perform encoding and decoding operations for original target video frames with various resolutions, which has a wider scope of application and a simpler and more convenient operation process.

[0094] The video decoding method of this embodiment can be executed by any appropriate electronic device with data capabilities, including but not limited to: a server, a PC, etc.

[0095] Embodiment 3

[0096] Reference Figure 5 , Figure 5 1 is a flowchart of a video decoding method according to Embodiment 3 of the present application. Specifically, the video decoding method provided in this embodiment includes the following steps:

[0097] Step 502, obtain and decode the video bit stream to obtain the original reference video frame and target features.

[0098] Step 504, adjusting the resolution of the original reference video frame to obtain an adjusted reference video frame with a first preset resolution; and extracting features from the adjusted reference video frame through a feature extraction network to obtain reference features.

[0099] Specifically, for example, the original reference video frame may be downsampled to obtain an adjusted reference video frame having a first preset resolution; and features may be extracted from the adjusted reference video frame through a feature extraction network to obtain reference features.

[0100] Step 506: input the reference feature and the target feature into a motion estimation network, perform motion estimation through the motion estimation network, and obtain a first motion estimation result.

[0101] Step 508: Adjust the resolution of the original reference video frame to obtain an adjusted reference video frame with a second preset resolution.

[0102] In this step, the original reference video frame may be downsampled to obtain an adjusted reference video frame having a second preset resolution.

[0103] Optionally, in some of the embodiments, an adjusted reference video frame having a second preset resolution can be obtained in the following manner: determining a second target scaling factor based on the resolution of the original reference video frame; scaling the original reference video frame using the second target scaling factor to obtain an adjusted reference video frame having the second preset resolution; wherein the second target scaling factor can be determined based on the relationship between the resolution of the original reference video frame and the second preset resolution, specifically, the ratio of the resolution of the original reference video frame to the second preset resolution can be determined as the second target scaling factor.

[0104] Furthermore, for the several resolutions that may appear in the original reference video frame, the scaling factors corresponding to the resolutions can be calculated in advance and placed in a second scaling factor sequence. Thereafter, after acquiring the original reference video frame, the second target scaling factor corresponding to the resolution of the original reference video frame can be determined from the above-mentioned preset second scaling factor sequence based on the preset correspondence between the resolution and the scaling factor.

[0105] Step 510: Input the first motion estimation result and the adjusted reference video frame with the second preset resolution into a generation network, and perform deformation processing on the adjusted reference video frame with the second preset resolution through the generation network to generate a transition reconstructed video frame with the second preset resolution.

[0106] The second preset resolution in the above step 508 is set according to the first preset resolution and the structural parameters of the generated network.

[0107] Specifically, the first motion estimation result obtained in step 506 has a first preset resolution. In addition, the generation network generally includes a downsampling subnetwork, a deformation subnetwork, and an upsampling subnetwork. In this step, the specific operation performed by the generation network is: first, the downsampled reference video frame is downsampled by the internal downsampling subnetwork to obtain a second downsampled reference frame; then, the second downsampled reference frame is deformed by the deformation subnetwork with reference to the first motion estimation result to obtain a deformed reference frame, and then the deformed reference frame is upsampled by the upsampling subnetwork, thereby outputting the result. In order to smoothly perform the deformation process, the resolution of the second downsampled reference frame and the first motion estimation result should match each other, that is, the second downsampled reference frame and the first motion estimation result have the same resolution (both are the first preset resolution in the embodiment of the present application). Therefore, in the embodiment of the present application, when the second preset resolution is set, the second downsampled reference frame obtained after the downsampled reference video frame with the second preset resolution is downsampled by the downsampling subnetwork can match the resolution of the first motion estimation result, both of which are the first preset resolution.

[0108] Step 512: Adjust the resolution of the transition reconstructed video frame to obtain a reconstructed video frame having the same resolution as the original reference video frame.

[0109] The transition reconstructed video frame generated by the generation network has a second preset resolution. If the second preset resolution is obtained by downsampling the original reference video frame, then in order to obtain a reconstructed video frame with the same resolution as the original reference video frame, it is necessary to perform an upsampling operation on the transition reconstructed video frame in this step opposite to step 508.

[0110] Specifically, the upsampling in this step can be performed in the following manner: the inverse of the second target scaling factor in the above step 508 is determined as the third target scaling factor; the transition reconstructed video frame is downsampled using the third target scaling factor to obtain a reconstructed video frame with the same resolution as the original reference video frame.

[0111] See also Figure 6 , Figure 6 This is a schematic diagram of a scene corresponding to the third embodiment of the present application. Figure 6 The schematic diagram shown in the figure illustrates an embodiment of the present application by taking a specific scenario as an example:

[0112] The video bitstream is decoded to obtain an original reference video frame and a target feature with a resolution of W×H; the original reference video frame is downsampled to obtain an adjusted reference video frame with a resolution of W1×H1 (a first preset resolution); and the corresponding reference feature is obtained through a feature extraction network; the reference feature and the target feature are input into a motion estimation network to obtain a first motion estimation result; at the same time, a second scaling factor sequence x={x1, x2, x3, ..., x n} to downsample the original reference video frame to obtain an adjusted reference video frame with a second preset resolution (not shown in the figure); through a generation network, based on the adjusted reference video frame with the second preset resolution and the first motion estimation result, a transition reconstructed video frame with the second preset resolution is obtained; from a third scaling factor sequence 1 / x={1 / x1, 1 / x2, 1 / x3..., 1 / x n}A third target scaling factor is determined in the process to upsample the transition reconstructed video frame to obtain a reconstructed video frame with a resolution of W×H.

[0113] In the embodiment of the present application, during the decoding stage, the original reference video frame is also uniformed in resolution, and the original reference video frame is converted into an adjusted reference video frame with a fixed resolution, thereby obtaining reference features and target features with a fixed resolution, so that the features finally input to the motion estimation network and the generation network are also features with a fixed resolution, and thus only one motion estimation network and one generation network are required to achieve decoding in various target video frame scenarios with different resolutions. In the embodiment of the present application, only one set of neural network models is required to perform encoding and decoding operations for original target video frames with various resolutions, which has a wider scope of application and a simpler and more convenient operation process.

[0114] In addition, in the embodiments of the present application, the resolution adjustment processes (up-sampling process and down-sampling process) are all performed on video frames, that is, they are all performed in the image domain rather than the feature domain. Therefore, it is beneficial for each network in the neural network model to learn the correct motion information, etc., thereby improving the quality of video frame reconstruction.

[0115] The video decoding method of this embodiment can be executed by any appropriate electronic device with data capabilities, including but not limited to: a server, a PC, etc.

[0116] Embodiment 4

[0117] Reference Figure 7 , Figure 7 1 is a flowchart of a video decoding method according to Embodiment 4 of the present application. Specifically, the video decoding method provided in this embodiment includes the following steps:

[0118] Step 702, obtain and decode the video bit stream to obtain the original reference video frame and target features.

[0119] Step 704: adjust the resolution of the original reference video frame to obtain an adjusted reference video frame with a first preset resolution; and extract features from the adjusted reference video frame through a feature extraction network to obtain reference features.

[0120] Specifically, in this step, the original reference video frame may be downsampled to obtain an adjusted reference video frame having a first preset resolution.

[0121] Step 706: Adjust the resolution of the reference feature and the target feature to obtain an adjusted reference feature and an adjusted target feature.

[0122] If the original reference video frame is downsampled in step 704, then in this step, the reference features and the target features may be upsampled accordingly to obtain adjusted reference features and adjusted target features.

[0123] Step 708: Input the adjusted reference features and the adjusted target features into a motion estimation network, perform motion estimation through the motion estimation network, and obtain a second motion estimation result.

[0124] Step 710: Input the second motion estimation result and the original reference video frame into a generation network, and perform deformation processing on the original reference video frame through the generation network to generate a reconstructed video frame with the same resolution as the original reference video frame.

[0125] The scaling factor (sampling factor) used in the resolution adjustment in the above step 706 is set according to the resolution of the original reference video frame, the structural parameters of the motion estimation network, and the structural parameters of the generation network. Specifically, similar to step 510, in order to smoothly perform the deformation processing in the generation network, after the motion estimation is performed on the adjusted reference features and the adjusted target features obtained according to the above sampling factors, the second motion estimation result obtained can be the same as the resolution of the original reference video frame after the downsampling operation of the downsampling subnetwork.

[0126] See also Figure 8 , Figure 8 This is a schematic diagram of a scene corresponding to the fourth embodiment of the present application. Figure 8 The schematic diagram shown in the figure illustrates an embodiment of the present application by taking a specific scenario as an example:

[0127] Decode the video bitstream to obtain an original reference video frame and target features with a resolution of W×H; downsample the original reference video frame to obtain an adjusted reference video frame with a resolution of W1×H1 (a first preset resolution); and obtain corresponding reference features through a feature extraction network; from a scaling factor sequence x={x1, x2, x3, ..., x n} determines the target scaling factor to upsample the reference features and the target features to obtain adjusted reference features and adjusted target features; obtains a second motion estimation result through a motion estimation network; and then generates a reconstructed video frame with the same resolution as the original reference video frame based on the second motion estimation result and the original reference video frame through a generation network.

[0128] In the embodiment of the present application, during the decoding stage, the original reference video frame is also uniformed in resolution, and the original reference video frame is converted into an adjusted reference video frame with a fixed resolution, thereby obtaining reference features and target features with a fixed resolution, so that the features finally input to the motion estimation network and the generation network are also features with a fixed resolution, and thus only one motion estimation network and one generation network are required to achieve decoding in various target video frame scenarios with different resolutions. In the embodiment of the present application, only one set of neural network models is required to perform encoding and decoding operations for original target video frames with various resolutions, which has a wider scope of application and a simpler and more convenient operation process.

[0129] In addition, in the embodiment of the present application, when generating the network output result, the result is not upsampled or downsampled, so the visual artifacts in the final reconstructed video frame can be effectively avoided.

[0130] The video decoding method of this embodiment can be executed by any appropriate electronic device with data capabilities, including but not limited to: a server, a PC, etc.

[0131] Embodiment 5

[0132] Reference Fig. 9 , Fig. 9 1 is a flowchart of a video decoding method according to Embodiment 5 of the present application. Specifically, the video decoding method provided in this embodiment includes the following steps:

[0133] Step 902, obtain and decode the video bit stream to obtain the original reference video frame and target features.

[0134] Step 904: adjust the resolution of the original reference video frame to obtain an adjusted reference video frame with a first preset resolution; and extract features from the adjusted reference video frame through a feature extraction network to obtain reference features.

[0135] Specifically, in this step, the original reference video frame may be downsampled to obtain an adjusted reference video frame with a first preset resolution.

[0136] Step 906: input the reference feature and the target feature into a motion estimation network, perform motion estimation through the motion estimation network, and obtain a first motion estimation result.

[0137] Step 908: Adjust the resolution of the first motion estimation result to obtain a third motion estimation result.

[0138] If the original reference video frame is downsampled in step 904, then correspondingly, in this step, the first motion estimation result may be upsampled to obtain a third motion estimation result.

[0139] Step 910: Input the third motion estimation result and the original reference video frame into a generation network, and perform deformation processing on the original reference video frame through the generation network to generate a reconstructed video frame with the same resolution as the original reference video frame.

[0140] The sampling factor used in the resolution adjustment in the above step 908 is set according to the resolution of the original reference video frame and the structural parameters of the generation network. Specifically, in order to smoothly perform the deformation processing in the generation network, the resolution of the third motion estimation result can match (be equal to) the resolution of the original reference video frame after the downsampling operation of the downsampling sub-network.

[0141] See also Fig.10 , Fig.10 This is a schematic diagram of a scenario corresponding to the fifth embodiment of the present application. Fig.10 The schematic diagram shown in the figure illustrates an embodiment of the present application by taking a specific scenario as an example:

[0142] The video bitstream is decoded to obtain an original reference video frame and a target feature with a resolution of W×H; the original reference video frame is downsampled to obtain an adjusted reference video frame with a resolution of W1×H1 (a first preset resolution); and corresponding reference features are obtained through a feature extraction network; motion estimation is performed on the reference features and the target features to obtain a first motion estimation result with a first preset resolution; and a scaling factor sequence x={x1, x2, x3, ..., x n} determines the target scaling factor to upsample the first motion estimation result to obtain a third motion estimation result; and then generates a reconstructed video frame with the same resolution as the original reference video frame based on the third motion estimation result and the original reference video frame through a generation network.

[0143] In the embodiment of the present application, during the decoding stage, the original reference video frame is also uniformed in resolution, and the original reference video frame is converted into an adjusted reference video frame with a fixed resolution, thereby obtaining reference features and target features with a fixed resolution, so that the features finally input to the motion estimation network and the generation network are also features with a fixed resolution, and thus only one motion estimation network and one generation network are required to achieve decoding in various target video frame scenarios with different resolutions. In the embodiment of the present application, only one set of neural network models is required to perform encoding and decoding operations for original target video frames with various resolutions, which has a wider scope of application and a simpler and more convenient operation process.

[0144] In addition, in the embodiment of the present application, when generating the network output result, the resolution of the result is not adjusted (upsampling or downsampling operation), so the visual artifacts in the final reconstructed video frame can be effectively avoided.

[0145] The video decoding method of this embodiment can be executed by any appropriate electronic device with data capabilities, including but not limited to: a server, a PC, etc.

[0146] Embodiment 6

[0147] Reference Fig.11 , Fig.11 1 is a flowchart of a video decoding method according to Embodiment 6 of the present application. Specifically, the video decoding method provided in this embodiment includes the following steps:

[0148] Step 1102, obtain and decode the video bit stream to obtain the original reference video frame and target features.

[0149] Step 1104, adjusting the resolution of the original reference video frame to obtain an adjusted reference video frame with a first preset resolution; and extracting features from the downsampled reference video frame through a feature extraction network to obtain reference features.

[0150] Specifically, in this step, the original reference video frame may be downsampled to obtain an adjusted reference video frame with a first preset resolution.

[0151] Step 1106: input the reference feature and the target feature into a motion estimation network, perform motion estimation through the motion estimation network, and obtain a first motion estimation result.

[0152] Step 1108, input the original reference video frame and the first motion estimation result into the generation network, downsample the original reference video frame through the downsampling subnetwork to obtain a first downsampled reference frame; downsample the first downsampled reference frame through the downsampling layer to obtain a second downsampled reference frame; deform the second downsampled reference frame through the deformation subnetwork to obtain a deformed reference frame; upsample the deformed reference frame through the upsampling layer to obtain a first upsampled deformed frame; upsample the first upsampled deformed frame through the upsampling subnetwork to obtain a reconstructed video frame with the same resolution as the original reference video frame.

[0153] Specifically, the sampling factor used by the downsampling layer when downsampling the first downsampled reference frame is reciprocal of the sampling factor used by the upsampling layer when upsampling the first upsampled deformed frame. That is, assuming that the sampling factor used by the downsampling layer when downsampling the first downsampled reference frame is x1, the sampling factor used by the upsampling layer when upsampling the first upsampled deformed frame is 1 / x1.

[0154] The sampling factor used by the downsampling layer when downsampling the first downsampled reference frame is set according to the resolution of the original reference video frame, the first preset resolution, and the structural parameters of the generation network. Specifically, after the original reference video frame is input into the generation network, the resolution of the second downsampled reference frame finally output by the downsampling layer is the first preset resolution.

[0155] See also Fig.12 , Fig.12 This is a schematic diagram of a scene corresponding to the sixth embodiment of the present application. Fig.12 The schematic diagram shown in the figure illustrates an embodiment of the present application by taking a specific scenario as an example:

[0156] The video bitstream is decoded to obtain an original reference video frame and a target feature with a resolution of W×H; the original reference video frame is downsampled to obtain an adjusted reference video frame with a resolution of W1×H1 (a first preset resolution); and the corresponding reference feature is obtained through a feature extraction network; motion estimation is performed on the reference feature and the target feature to obtain a first motion estimation result with a first preset resolution; the original reference video frame and the first motion estimation result are input into a generation network, and the downsampling layer in the generation network is generated from a scaling factor sequence 1 / x={1 / x1, 1 / x2, 1 / x3..., 1 / x n}, the target scaling factor is determined in the downsampling subnetwork, and the first downsampled reference frame obtained after the downsampling subnetwork is downsampled again to obtain a second downsampled reference frame; the deformation subnetwork deforms the second downsampled reference frame to obtain a deformed reference frame; the upsampling layer is transformed from the scaling factor sequence x = {x1, x2, x3..., x n} determines the target scaling factor and upsamples the deformed reference frame to obtain a first upsampled deformed frame; the first upsampled deformed frame is upsampled again through the upsampling subnetwork, and finally a reconstructed video frame with the same resolution as the original reference video frame is obtained.

[0157] In the embodiment of the present application, during the decoding stage, the original reference video frame is also uniformed in resolution, and the original reference video frame is converted into an adjusted reference video frame with a fixed resolution, thereby obtaining reference features and target features with a fixed resolution, so that the features finally input to the motion estimation network and the generation network are also features with a fixed resolution, and thus only one motion estimation network and one generation network are required to achieve decoding in various target video frame scenarios with different resolutions. In the embodiment of the present application, only one set of neural network models is required to perform encoding and decoding operations for original target video frames with various resolutions, which has a wider scope of application and a simpler and more convenient operation process.

[0158] The video decoding method of this embodiment can be executed by any appropriate electronic device with data capabilities, including but not limited to: a server, a PC, etc.

[0159] Embodiment 7

[0160] Reference Fig.13 , Fig.13 1 is a flowchart of a video encoding method according to Embodiment 7 of the present application. Specifically, the video encoding method provided in this embodiment includes the following steps:

[0161] Step 1302: Acquire an original reference video frame and an original target video frame to be encoded.

[0162] Step 1304: adjust the resolution of the original target video frame to obtain an adjusted target video frame with a first preset resolution; and extract features from the adjusted target video frame through a feature extraction network corresponding to the first preset resolution to obtain target features.

[0163] Step 1306, respectively encode the original reference video frame and the target feature to obtain a video bitstream, so that the decoding end device decodes the video bitstream to obtain the original reference video frame and the target feature; adjust the resolution of the original reference video frame to obtain an adjusted reference video frame with a first preset resolution; and extract features from the adjusted reference video frame through a feature extraction network to obtain a reference feature; perform motion estimation based on the reference feature and the target feature through a motion estimation network to obtain a motion estimation result; and generate a reconstructed video frame with the same resolution as the original reference video frame based on the motion estimation result and the original reference video frame through a generation network.

[0164] In the video encoding method of this embodiment, the specific implementation of each step can refer to the corresponding steps in any of the above-mentioned embodiments 2 to 6, and will not be repeated here.

[0165] According to the video encoding method provided by the embodiment of the present application, in the encoding stage, after obtaining the original target video frame to be encoded, the original target video frame is uniformly resolved through the resolution adjustment operation, and the original target video frame is converted into an adjusted target video frame with a fixed resolution (first preset resolution), so that even if the original target video frame has a variety of resolutions, the video frame with a fixed resolution is still input to the feature extraction network. In this way, there is no need to train multiple feature extraction networks for different resolutions, and only one feature extraction network corresponding to the first preset resolution (a feature extraction network for extracting features from a video frame with a first preset resolution) is needed to realize the encoding of multiple original target video frames with different resolutions, which has a wider scope of application and higher versatility. At the same time, the operation is simpler and more convenient. In addition, correspondingly, in the decoding stage, the original reference video frame is also uniformly resolved, and the original reference video frame is converted into an adjusted reference video frame with a fixed resolution, thereby obtaining reference features and target features with a fixed resolution, so that the features with a fixed resolution are finally input to the motion estimation network and the generation network, and then only one motion estimation network and one generation network are needed to realize decoding in various scenes of target video frames with different resolutions. In summary, the embodiments of the present application only require one set of neural network models to perform encoding and decoding operations for original target video frames of various resolutions, which has a wider scope of application and a simpler and more convenient operation process.

[0166] Embodiment 8

[0167] Reference Fig.14 , Fig.14 This is a flowchart of the steps of a video encoding method according to Embodiment 8 of the present application. The application scenario of the video encoding method is: the video acquisition device acquires the conference video, and the video encoding is performed by the video encoding method provided in this embodiment to form a corresponding video bit stream, and the video bit stream is sent to the conference terminal, and the conference terminal decodes the video bit stream to obtain the corresponding conference video screen and displays it.

[0168] Specifically, the video encoding method provided in this embodiment includes the following steps:

[0169] Step 1402: Acquire the original video clip captured by the video capture device.

[0170] Step 1404 , determining an original reference video frame and an original target video frame to be encoded from the original video segment.

[0171] Step 1406, adjusting the resolution of the original target video frame to obtain an adjusted target video frame with a first preset resolution; and extracting features from the adjusted target video frame through a feature extraction network corresponding to the first preset resolution to obtain target features.

[0172] Step 1408, encode the original reference video frame and the target feature respectively to obtain a video bit stream.

[0173] Step 1410: Send the video bitstream to the conference terminal device, so that the conference terminal device reconstructs the video frame based on the video bitstream, generates and displays the reconstructed video frame with the same resolution as the original target video frame.

[0174] Embodiment 9

[0175] See also Fig.15 , Fig.15 1 is a structural block diagram of a video encoding device according to Embodiment 9 of the present application. The video encoding device provided in the embodiment of the present application includes:

[0176] The original video frame acquisition module 1502 is used to acquire the original reference video frame and the original target video frame to be encoded;

[0177] The target feature obtaining module 1504 is used to adjust the resolution of the original target video frame to obtain an adjusted target video frame with a first preset resolution; and extract features from the adjusted target video frame through a feature extraction network corresponding to the first preset resolution to obtain target features;

[0178] The bitstream obtaining module 1506 is used to encode the original reference video frame and the target feature respectively to obtain a video bitstream, so as to reconstruct the video frame based on the video bitstream and generate a reconstructed video frame with the same resolution as the original target video frame.

[0179] Optionally, in some of the embodiments, the target feature obtaining module 1504, when performing the step of adjusting the resolution of the original target video frame to obtain an adjusted target video frame having a first preset resolution, is specifically configured to:

[0180] determining a first target scaling factor based on a resolution of the original target video frame;

[0181] The original target video frame is scaled using a first target scaling factor to obtain an adjusted target video frame having a first preset resolution.

[0182] Optionally, in some of the embodiments, the target feature obtaining module 1504, when performing the step of determining the first target scaling factor based on the resolution of the original target video frame, is specifically configured to:

[0183] According to a preset correspondence relationship between resolution and scaling factor, a first target scaling factor corresponding to the resolution of the original target video frame is determined from a preset first scaling factor sequence.

[0184] The video encoding device of this embodiment is used to implement the corresponding video encoding methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here. In addition, the functional implementation of each module in the video encoding device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be described in detail here.

[0185] Embodiment 10

[0186] See also Fig.16 , Fig.16 1 is a structural block diagram of a video decoding device according to Embodiment 10 of the present application. The video decoding device provided in the embodiment of the present application includes:

[0187] A decoding module 1602 is used to obtain and decode a video bit stream to obtain an original reference video frame and target features;

[0188] The reference feature obtaining module 1604 is used to adjust the resolution of the original reference video frame to obtain an adjusted reference video frame with a first preset resolution; and extract features from the adjusted reference video frame through a feature extraction network to obtain reference features;

[0189] A motion estimation module 1606 is used to perform motion estimation based on reference features and target features through a motion estimation network to obtain a motion estimation result;

[0190] The generation module 1608 is used to generate a reconstructed video frame having the same resolution as the original reference video frame based on the motion estimation result and the original reference video frame through a generation network.

[0191] Optionally, in some embodiments, the motion estimation module 1606 is specifically configured to:

[0192] Inputting the reference feature and the target feature into a motion estimation network, performing motion estimation through the motion estimation network, and obtaining a first motion estimation result;

[0193] The generating module 1608 is specifically configured to: adjust the resolution of the original reference video frame to obtain an adjusted reference video frame having a second preset resolution;

[0194] Inputting the first motion estimation result and the adjusted reference video frame with a second preset resolution into a generation network, performing deformation processing on the adjusted reference video frame with the second preset resolution through the generation network to generate a transition reconstructed video frame with the second preset resolution;

[0195] The resolution of the transition reconstructed video frame is adjusted to obtain a reconstructed video frame with the same resolution as the target video frame.

[0196] Optionally, in some embodiments, the motion estimation module 1606 is specifically configured to:

[0197] Adjusting the resolution of the reference feature and the target feature to obtain an adjusted reference feature and an adjusted target feature;

[0198] Inputting the adjusted reference features and the adjusted target features into a motion estimation network, performing motion estimation through the motion estimation network, and obtaining a second motion estimation result;

[0199] The generation module 1608 is specifically used to: input the second motion estimation result and the original reference video frame into the generation network, and perform deformation processing on the original reference video frame through the generation network to generate a reconstructed video frame with the same resolution as the target video frame.

[0200] Optionally, in some embodiments, the motion estimation module 1606 is specifically configured to:

[0201] Inputting the reference feature and the target feature into a motion estimation network, performing motion estimation through the motion estimation network, and obtaining a first motion estimation result;

[0202] The generating module 1608 is specifically used for:

[0203] Adjusting the resolution of the first motion estimation result to obtain a third motion estimation result;

[0204] The third motion estimation result and the original reference video frame are input into a generation network, and the original reference video frame is deformed through the generation network to generate a reconstructed video frame with the same resolution as the target video frame.

[0205] Optionally, in some of the embodiments, the generation module includes: a downsampling subnetwork, a downsampling layer, a deformation subnetwork, an upsampling layer, and an upsampling subnetwork;

[0206] The motion estimation module 1606 is specifically used to: input the reference feature and the target feature into a motion estimation network, perform motion estimation through the motion estimation network, and obtain a first motion estimation result;

[0207] The generation module 1608 is specifically used to: input the original reference video frame and the first motion estimation result into the generation network, downsample the original reference video frame through the downsampling subnetwork to obtain a first downsampled reference frame; downsample the first downsampled reference frame through the downsampling layer to obtain a second downsampled reference frame; deform the second downsampled reference frame through the deformation subnetwork to obtain a deformed reference frame; upsample the deformed reference frame through the upsampling layer to obtain a first upsampled deformed frame; upsample the first upsampled deformed frame through the upsampling subnetwork to obtain a reconstructed video frame with the same resolution as the original reference video frame.

[0208] The video decoding device of this embodiment is used to implement the corresponding video decoding methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here. In addition, the functional implementation of each module in the video decoding device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be described in detail here.

[0209] Embodiment 11

[0210] Reference Fig.17 , shows a schematic diagram of the structure of an electronic device according to Embodiment 11 of the present application. The specific embodiments of the present application do not limit the specific implementation of the electronic device.

[0211] like Fig.17 As shown, the conference terminal may include: a processor (processor) 1702 , a communication interface (Communications Interface) 1704 , a memory (memory) 1706 , and a communication bus 1708 .

[0212] in:

[0213] The processor 1702 , the communication interface 1704 , and the memory 1706 communicate with each other via a communication bus 1708 .

[0214] The communication interface 1704 is used to communicate with other electronic devices or servers.

[0215] The processor 1702 is used to execute the program 1710, and specifically can execute the above-mentioned video encoding method, or the relevant steps in the embodiment of the video decoding method.

[0216] Specifically, the program 1710 may include program codes, which include computer operation instructions.

[0217] The processor 1702 may be a CPU, or an application specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0218] The memory 1706 is used to store the program 1710. The memory 1706 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0219] Program 1710 can be specifically used to enable processor 1702 to perform the following operations: obtain an original reference video frame and an original target video frame to be encoded; adjust the resolution of the original target video frame to obtain an adjusted target video frame having a first preset resolution; and extract features from the adjusted target video frame through a feature extraction network corresponding to the first preset resolution to obtain target features; encode the original reference video frame and the target features respectively to obtain a video bitstream, so as to reconstruct the video frame based on the video bitstream and generate a reconstructed video frame having the same resolution as the original target video frame.

[0220] or,

[0221] Program 1710 can be specifically used to enable processor 1702 to perform the following operations: obtain and decode a video bitstream to obtain an original reference video frame and a target feature; adjust the resolution of the original reference video frame to obtain an adjusted reference video frame having a first preset resolution; and extract features from the adjusted reference video frame through a feature extraction network to obtain reference features; perform motion estimation based on the reference features and the target features through a motion estimation network to obtain a motion estimation result; and generate a reconstructed video frame having the same resolution as the original reference video frame based on the motion estimation result and the original reference video frame through a generation network.

[0222] or,

[0223] Program 1710 can be specifically used to enable processor 1702 to perform the following operations: obtain an original reference video frame and an original target video frame to be encoded; adjust the resolution of the original target video frame to obtain an adjusted target video frame having a first preset resolution; and extract features of the adjusted target video frame through a feature extraction network corresponding to the first preset resolution to obtain target features; encode the original reference video frame and the target features respectively to obtain a video bitstream, so that a decoding end device decodes the video bitstream to obtain the original reference video frame and the target features; adjust the resolution of the original reference video frame to obtain an adjusted reference video frame having a first preset resolution; and extract features of the adjusted reference video frame through a feature extraction network to obtain reference features; perform motion estimation based on the reference features and the target features through a motion estimation network to obtain a motion estimation result; and generate a reconstructed video frame having the same resolution as the original reference video frame based on the motion estimation result and the original reference video frame through a generation network.

[0224] or,

[0225] Program 1710 can be specifically used to enable processor 1702 to perform the following operations: obtain an original video clip captured by a video capture device; determine an original reference video frame and an original target video frame to be encoded from the original video clip; adjust the resolution of the original target video frame to obtain an adjusted target video frame having a first preset resolution; and extract features from the adjusted target video frame through a feature extraction network corresponding to the first preset resolution to obtain target features; encode the original reference video frame and the target features respectively to obtain a video bitstream; and send the video bitstream to a conference terminal device so that the conference terminal device reconstructs the video frame based on the video bitstream, generates a reconstructed video frame having the same resolution as the original target video frame, and displays it.

[0226] The specific implementation of each step in program 1710 can refer to the corresponding description of the corresponding steps and units in the above-mentioned video encoding method or the video decoding method embodiment, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above-mentioned method embodiment, which will not be repeated here.

[0227] Through the electronic device of this embodiment,

[0228] In the encoding stage, after obtaining the original target video frame to be encoded, the original target video frame is uniformly resolved through the resolution adjustment operation, and the original target video frame is converted into an adjusted target video frame with a fixed resolution (first preset resolution), so that even if the original target video frame has a variety of resolutions, the video frame with a fixed resolution is still input to the feature extraction network. In this way, there is no need to train multiple feature extraction networks for different resolutions, and only one feature extraction network corresponding to the first preset resolution (a feature extraction network for extracting features from video frames with the first preset resolution) is needed to realize the encoding of original target video frames with multiple different resolutions, which has a wider scope of application and higher versatility. At the same time, the operation is simpler and more convenient. In addition, correspondingly, in the decoding stage, the original reference video frame is also uniformly resolved, and the original reference video frame is converted into an adjusted reference video frame with a fixed resolution, thereby obtaining reference features and target features with a fixed resolution, so that the features with a fixed resolution are finally input to the motion estimation network and the generation network, and then only one motion estimation network and one generation network are needed to realize the decoding of target video frames with various resolutions. In summary, the embodiments of the present application only require one set of neural network models to perform encoding and decoding operations for original target video frames of various resolutions, which has a wider scope of application and a simpler and more convenient operation process.

[0229] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to execute operations corresponding to any one of the above-mentioned multiple method embodiments.

[0230] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0231] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as a computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the video encoding method described herein is implemented, or the video decoding method. In addition, when a general-purpose computer accesses a code for implementing the video encoding method shown here, or the video decoding method, the execution of the code converts the general-purpose computer into a dedicated computer for executing the video encoding method shown here, or the video decoding method.

[0232] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0233] The above implementation methods are only used to illustrate the embodiments of the present application, and are not limitations on the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application. The scope of patent protection of the embodiments of the present application should be limited by the claims.

Claims

1. A video encoding method, comprising: Based on the same video clip, an original reference video frame and an original target video frame to be encoded are obtained; Adjusting the resolution of the original target video frame to obtain an adjusted target video frame with a first preset resolution; and extracting features from the adjusted target video frame through a feature extraction network corresponding to the first preset resolution to obtain target features; Encoding the original reference video frame and the target feature respectively to obtain a video bitstream, so as to reconstruct the video frame based on the video bitstream and generate a reconstructed video frame having the same resolution as the original target video frame; The step of adjusting the resolution of the original target video frame to obtain an adjusted target video frame having a first preset resolution includes: determining a first target scaling factor based on the resolution of the original target video frame; The original target video frame is scaled using the first target scaling factor to obtain an adjusted target video frame with a first preset resolution.

2. The method according to claim 1, wherein: The determining a first target scaling factor based on the resolution of the original target video frame comprises: According to a preset correspondence relationship between resolution and scaling factor, a first target scaling factor corresponding to the resolution of the original target video frame is determined from a preset first scaling factor sequence.

3. A video decoding method, comprising: Acquire and decode a video bitstream to obtain an original reference video frame and a target feature, wherein the original reference video frame and the original target video frame corresponding to the target feature are obtained based on the same video clip; Adjusting the resolution of the original reference video frame to obtain an adjusted reference video frame with a first preset resolution; and extracting features from the adjusted reference video frame through a feature extraction network to obtain reference features; Performing motion estimation based on the reference feature and the target feature through a motion estimation network to obtain a motion estimation result; Generate a reconstructed video frame having the same resolution as the original reference video frame based on the motion estimation result and the original reference video frame through a generation network; The step of adjusting the resolution of the original target video frame to obtain an adjusted target video frame having a first preset resolution includes: determining a first target scaling factor based on the resolution of the original target video frame; and scaling the original target video frame using the first target scaling factor to obtain an adjusted target video frame having the first preset resolution.

4. The method according to claim 3, wherein: The step of performing motion estimation based on the reference feature and the target feature through a motion estimation network to obtain a motion estimation result includes: Inputting the reference feature and the target feature into a motion estimation network, performing motion estimation through the motion estimation network, and obtaining a first motion estimation result; The generating network generates a reconstructed video frame having the same resolution as the target video frame based on the motion estimation result and the original reference video frame, comprising: Adjusting the resolution of the original reference video frame to obtain an adjusted reference video frame with a second preset resolution; Inputting the first motion estimation result and the adjusted reference video frame with a second preset resolution into a generation network, and performing deformation processing on the adjusted reference video frame with the second preset resolution through the generation network to generate a transition reconstructed video frame with the second preset resolution; The resolution of the transition reconstructed video frame is adjusted to obtain a reconstructed video frame having the same resolution as the target video frame.

5. The method according to claim 3, wherein: The step of performing motion estimation based on the reference feature and the target feature through a motion estimation network to obtain a motion estimation result includes: Adjusting the resolution of the reference feature and the target feature to obtain an adjusted reference feature and an adjusted target feature; Inputting the adjusted reference feature and the adjusted target feature into a motion estimation network, performing motion estimation through the motion estimation network, and obtaining a second motion estimation result; The generating network generates a reconstructed video frame having the same resolution as the target video frame based on the motion estimation result and the original reference video frame, comprising: The second motion estimation result and the original reference video frame are input into a generation network, and the original reference video frame is deformed through the generation network to generate a reconstructed video frame with the same resolution as the target video frame.

6. The method according to claim 3, wherein: The step of performing motion estimation based on the reference feature and the target feature through a motion estimation network to obtain a motion estimation result includes: Inputting the reference feature and the target feature into a motion estimation network, performing motion estimation through the motion estimation network, and obtaining a first motion estimation result; The generating network generates a reconstructed video frame having the same resolution as the target video frame based on the motion estimation result and the original reference video frame, comprising: Adjusting the resolution of the first motion estimation result to obtain a third motion estimation result; The third motion estimation result and the original reference video frame are input into a generation network, and the original reference video frame is deformed through the generation network to generate a reconstructed video frame with the same resolution as the target video frame.

7. The method according to claim 3, wherein: The generation network includes: a downsampling subnetwork, a downsampling layer, a deformation subnetwork, an upsampling layer and an upsampling subnetwork; The step of performing motion estimation based on the reference feature and the target feature through a motion estimation network to obtain a motion estimation result includes: Inputting the reference feature and the target feature into a motion estimation network, performing motion estimation through the motion estimation network, and obtaining a first motion estimation result; The generating network generates a reconstructed video frame having the same resolution as the target video frame based on the motion estimation result and the original reference video frame, comprising: The original reference video frame and the first motion estimation result are input into a generation network, the original reference video frame is downsampled through a downsampling subnetwork to obtain a first downsampled reference frame; the first downsampled reference frame is downsampled through a downsampling layer to obtain a second downsampled reference frame; the second downsampled reference frame is deformed through a deformation subnetwork to obtain a deformed reference frame; the deformed reference frame is upsampled through an upsampling layer to obtain a first upsampled deformed frame; the first upsampled deformed frame is upsampled through an upsampling subnetwork to obtain a reconstructed video frame with the same resolution as the target video frame.

8. A video encoding method, comprising: Based on the same video clip, an original reference video frame and an original target video frame to be encoded are obtained; Adjusting the resolution of the original target video frame to obtain an adjusted target video frame with a first preset resolution; and extracting features from the adjusted target video frame through a feature extraction network corresponding to the first preset resolution to obtain target features; Encoding the original reference video frame and the target feature respectively to obtain a video bit stream, so that a decoding end device decodes the video bit stream to obtain the original reference video frame and the target feature; Adjusting the resolution of the original reference video frame to obtain an adjusted reference video frame with a first preset resolution; and extracting features from the adjusted reference video frame through a feature extraction network to obtain reference features; Performing motion estimation based on the reference feature and the target feature through a motion estimation network to obtain a motion estimation result; generating a reconstructed video frame having the same resolution as the original reference video frame based on the motion estimation result and the original reference video frame through a generation network; The step of adjusting the resolution of the original target video frame to obtain an adjusted target video frame having a first preset resolution includes: determining a first target scaling factor based on the resolution of the original target video frame; The original target video frame is scaled using the first target scaling factor to obtain an adjusted target video frame with a first preset resolution.

9. A video encoding method, comprising: Obtaining the original video clips captured by the video capture device; Determining an original reference video frame and an original target video frame to be encoded from the original video clip; Adjusting the resolution of the original target video frame to obtain an adjusted target video frame with a first preset resolution; and extracting features from the adjusted target video frame through a feature extraction network corresponding to the first preset resolution to obtain target features; Encoding the original reference video frame and the target feature respectively to obtain a video bitstream; Sending the video bitstream to a conference terminal device, so that the conference terminal device reconstructs a video frame based on the video bitstream, generates a reconstructed video frame having the same resolution as the original target video frame, and displays the reconstructed video frame; The step of adjusting the resolution of the original target video frame to obtain an adjusted target video frame having a first preset resolution includes: determining a first target scaling factor based on the resolution of the original target video frame; The original target video frame is scaled using the first target scaling factor to obtain an adjusted target video frame with a first preset resolution.

10. An electronic device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the video encoding method as described in any one of claims 1-2 or claims 8 or 9, or an operation corresponding to the video decoding method as described in any one of claims 3-7.

11. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the program implements the video encoding method according to any one of claims 1 to 2 or according to claim 8 or 9, or implements the video decoding method according to any one of claims 3 to 7.

12. A computer program product, comprising computer instructions, wherein the computer instructions instruct a computing device to perform operations corresponding to the video encoding method as described in any one of claims 1-2 or claims 8 or 9, or to perform operations corresponding to the video decoding method as described in any one of claims 3-7.

Images