A video encoding method, a video decoding method and apparatus

Abstract

The application discloses a video encoding method, a video decoding method and a device, wherein the video encoding method comprises the following steps: acquiring a video frame to be encoded; using a resolution prediction network, which is used for modeling a video frame by using a space-time correlation of the video frame through a convolutional neural network and a recurrent neural network, and outputting a downsampling ratio corresponding to the video frame to be encoded; respectively encoding the video frame to be encoded according to the downsampling ratio and an original resolution; obtaining an optimal resolution according to a distortion value and an adjusted rate-distortion value; and adding the optimal resolution identifier to the encoded data. The video encoding and decoding method and device can adaptively select the downsampling information, improve the compression rate of the video encoding under the premise of ensuring the subjective quality, and solve the technical problem that the same resolution is used for video encoding and decoding in the related art, resulting in a large fluctuation of the peak signal-to-noise ratio.

Description

Technical Field

[0001] This invention relates to the field of video technology, and more specifically, to a video encoding method, a video decoding method, and an apparatus. Background Technology

[0002] With the development of digital media and computer technologies, video is applied in various fields, such as mobile communication, network monitoring, and IPTV. As hardware performance and screen resolution improve, users' demand for high-definition video is increasingly strong. Under limited bandwidth conditions, traditional encoders encode video frames indiscriminately, which may lead to poor video quality in certain scenarios. For example, at 750kbps, when all video frames are encoded indiscriminately, some frames exhibit poor quality. Analysis shows that encoders such as H.264, H.265, and iOS all suffer from similar problems.

[0003] In existing video encoding processes, such as Figure 1 As shown, if all frames in a video are encoded at high resolution, the transmission bandwidth will be relatively small (e.g., less than...). Figure 1 Under the bandwidth threshold Th shown, the peak signal-to-noise ratio (PSNR1) for encoding all frames in the video at high resolution is lower than the peak signal-to-noise ratio (PSNR2) for encoding some frames in the video at low resolution. In other words, when the transmission bandwidth is small, the peak signal-to-noise ratio (PSNR1) for encoding at high resolution is relatively small, and the distortion is relatively large.

[0004] Similarly, if all frames in a video are encoded at low resolution, then the transmission bandwidth will be relatively large (e.g., greater than...). Figure 1 Under the bandwidth threshold Th shown, the peak signal-to-noise ratio (PSNR) 3 corresponding to encoding all frames of the video at low resolution is lower than the peak signal-to-noise ratio (PSNR) 4 corresponding to encoding all frames of the video at high resolution. In other words, when the transmission bandwidth is large, the peak signal-to-noise ratio (PSNR) 3 is relatively small when encoding at low resolution, and the distortion is relatively large.

[0005] Furthermore, for different types of videos, or different frames within the same video, or different blocks within the same frame, such as Figure 1 The intersection point D shown in the diagram will shift, which increases the difficulty of choosing which resolution to encode frames in a video in the prior art. Summary of the Invention

[0006] The present application provides a video decoding method, a video encoding method and a device, which can adaptively select a down-sampling ratio and improve video quality under limited bandwidth, to solve the technical problem of large peak signal-to-noise ratio fluctuation caused by encoding and decoding videos with the same resolution.

[0007] The present application adopts the technical solution to overcome the technical problems:

[0008] The video encoding method comprises the following steps: obtaining a to-be-encoded video frame; obtaining a down-sampling ratio corresponding to the to-be-encoded video frame based on a resolution prediction network; encoding the to-be-encoded video frame according to the down-sampling ratio and an original resolution, to obtain first encoding data and second encoding data, and obtaining a first reconstructed video frame based on the down-sampling ratio and a second reconstructed video frame based on the original resolution; adjusting a rate-distortion value of the second reconstructed video frame based on the type of the first or second reconstructed video frame, to obtain a second rate-distortion value; comparing the distortion values of the first and second reconstructed video frames, and comparing the first rate-distortion value of the first reconstructed video frame with the second rate-distortion value of the second reconstructed video frame, to determine an optimal resolution; adding an identifier of the optimal resolution to the corresponding first encoding data or second encoding data, to obtain encoding data.

[0009] Further, the encoding of the to-be-encoded video frame according to the down-sampling ratio and the original resolution, to obtain the first encoding data and the second encoding data, and the obtaining of the first reconstructed video frame based on the down-sampling ratio and the second reconstructed video frame based on the original resolution, comprises the following steps: encoding the to-be-encoded video frame according to the original resolution, to obtain the second reconstructed video frame and corresponding second encoding data; performing down-sampling processing on the to-be-encoded video frame according to the down-sampling ratio, to obtain the first reconstructed video frame; obtaining a reference frame of the first reconstructed video frame or the to-be-encoded video frame; processing the reference frame according to the down-sampling ratio, to obtain a first reference frame; encoding the first reconstructed video according to the first reference frame, to obtain the first encoding data.

[0010] Further, the adjustment of the rate-distortion value of the second reconstructed video frame based on the type of the first or second reconstructed video frame, to obtain the second rate-distortion value, comprises the following steps: if the video type of the reconstructed video frame is an I frame or a P frame or a B frame that is referenced, reducing the rate-distortion value of the second reconstructed video frame by a preset first proportion value; if the video type of the reconstructed video frame is a B frame that is not referenced, increasing the rate-distortion value of the second reconstructed video frame by a preset second proportion value; and taking the adjusted rate-distortion value as the second rate-distortion value.

[0011] Further, based on the comparison result of the distortion values of the first reconstructed video frame and the second reconstructed video frame, and the comparison result of the first rate-distortion value of the first reconstructed video frame and the second rate-distortion value of the second reconstructed video frame, the optimal resolution is determined, including: if the distortion value of the first reconstructed video frame is less than the distortion value of the second reconstructed video frame, and the first rate-distortion value of the first reconstructed video frame is less than the second rate-distortion value of the second reconstructed video frame, the optimal resolution is the downsampling ratio, otherwise, the optimal resolution is the original resolution.

[0012] Further, the adding the optimal resolution identifier into the corresponding first encoding data or second encoding data to obtain the encoding data, including: if the optimal resolution is the downsampling ratio, the optimal resolution identifier is added into the first encoding data, and the first encoding data after adding the optimal resolution identifier is taken as the encoding data; if the optimal resolution is the original resolution, the optimal resolution identifier is added into the second encoding data, and the second encoding data after adding the optimal resolution identifier is taken as the encoding data.

[0013] Further, the resolution prediction network is used to utilize the space-time correlation of the to-be-encoded video frame, model the to-be-encoded video frame through a convolutional neural network and a recurrent neural network, and output the downsampling ratio corresponding to the to-be-encoded video frame.

[0014] In order to achieve the above-mentioned purpose, the application further provides a video decoding method, including: obtaining a to-be-decoded video frame including the encoding data according to any one of claims 1-6; obtaining an optimal resolution identifier corresponding to the to-be-decoded video frame from the encoding data, and obtaining a decoding resolution of the to-be-decoded video frame based on the optimal resolution identifier; and decoding the to-be-decoded video frame based on the decoding resolution to obtain a decoded video frame.

[0015] Further, the decoding the to-be-decoded video frame based on the decoding resolution to obtain a decoded video frame, including: obtaining a reference frame corresponding to the to-be-decoded video frame; processing the reference frame according to the decoding resolution to obtain a second reference frame; decoding the encoding data according to the decoding resolution and the second reference frame to obtain a reconstructed video frame; and processing the reconstructed video frame according to the original resolution to obtain the decoded video frame.

[0016] In order to achieve the above-mentioned purpose, the application further provides a video decoding device, the device including: a first obtaining module, used for obtaining encoding data corresponding to a to-be-decoded video frame; a second obtaining module, used as an optimal resolution obtaining module, for obtaining a decoding resolution corresponding to the to-be-decoded video frame from the encoding data; and a decoding module, used for decoding the encoding data according to the decoding resolution corresponding to the to-be-decoded video frame to obtain a corresponding decoded video frame.

[0017] To achieve the above object, the application further provides a video encoding device, which comprises: a third acquisition module, configured to acquire a video frame to be encoded; a first determination module, configured to determine a down-sampling ratio corresponding to the video frame to be encoded; an encoding module, configured to encode the video frame to be encoded according to the original resolution and the down-sampling ratio, to acquire encoding data of the video frame encoded using the original resolution and encoding data of the video frame encoded using the down-sampling ratio; an adding module, configured to compare the distortion value of the reconstructed video frame obtained by encoding using the down-sampling ratio and the reconstructed video frame obtained by encoding using the original resolution, and the rate-distortion value adjusted according to the type of the video frame, to determine an optimal resolution, and add the optimal resolution to the encoding data.

[0018] The application has the following beneficial effects:

[0019] The video encoding method, the video decoding method and the device provided by the application acquire an input video frame, use a resolution prediction network to output a corresponding down-sampling ratio according to the input video frame, then encode the input video frame according to the down-sampling ratio and the original resolution to obtain encoding data corresponding to the input video frame, compare the distortion value of the reconstructed video frame obtained by encoding using the down-sampling ratio and the reconstructed video frame obtained by encoding using the original resolution, and the rate-distortion value adjusted according to the type of the video frame, to determine an optimal resolution, and add the optimal resolution identifier to the encoding data. The encoding method adds the optimal resolution identifier to the encoding data, can flexibly and adaptively select a more suitable resolution for each input video frame in the encoding process, so that the corresponding peak signal-to-noise ratio is relatively large and the distortion is relatively small, whether the transmission bandwidth is small or large, thereby ensuring that the peak signal-to-noise ratio can change in a small range and the peak signal-to-noise ratio is relatively large, thereby achieving the technical effect of avoiding large fluctuation of the peak signal-to-noise ratio of the video encoding and decoding, and solving the technical problem of large fluctuation of the peak signal-to-noise ratio caused by encoding and decoding the video using the same resolution in the related art, which is beneficial to improving the video quality under limited bandwidth. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 A schematic diagram of the peak signal-to-noise ratio of the encoding and decoding mode in the related art;

[0021] Figure 2 An application environment diagram of the video encoding method in an embodiment of the application;

[0022] Figure 3 An encoding framework diagram corresponding to the video encoding method in an embodiment of the application;

[0023] Figure 4 A decoding framework diagram corresponding to the video decoding method in one embodiment of the present application;

[0024] Figure 5 A schematic diagram corresponding to the coding block in one embodiment of the present application;

[0025] Figure 6 A schematic diagram of an optional video coding method according to an embodiment of the present application;

[0026] Figure 7 A schematic diagram of an application environment of an optional video coding method according to an embodiment of the present application;

[0027] Figure 8 A schematic diagram of an optional video decoding method according to an embodiment of the present application;

[0028] Figure 9 A schematic diagram of an application environment of an optional video decoding method according to an embodiment of the present application;

[0029] Figure 10 A schematic diagram of an optional video decoding apparatus according to an embodiment of the present application;

[0030] Figure 11 A schematic diagram of an optional video coding apparatus according to an embodiment of the present application;

[0031] Figure 12 A schematic diagram of an application scenario of an optional video coding and decoding method according to an embodiment of the present application;

[0032] Figure 13 A schematic diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the technical personnel in the art better understand the present application scheme, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor should belong to the scope of protection of the present application.

[0034] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0035] Figure 2 This is a diagram illustrating the application environment of video encoding and decoding methods in one embodiment. For example... Figure 2 As shown, the application environment includes a terminal 210 and a server 220, which communicate via a network. The terminal 210 can be a desktop terminal or a mobile terminal; a mobile terminal can be at least one of a mobile phone, tablet, or laptop. The server 220 can be a standalone server or a server cluster consisting of multiple servers. Video encoding and decoding methods can be implemented in the terminal 210 or the server 220. The terminal 210 or the server 220 can perform video encoding using an encoder or video decoding using a decoder. Alternatively, the terminal 210 or the server 220 can run a video encoding program via a processor or a video decoding program via a processor. After receiving encoded data from the terminal 210 through an input interface, the server 220 can directly transmit it to the processor for decoding or store it in a database for subsequent decoding. After obtaining encoded data by encoding the original video frames through the processor, the server 220 can directly send it to the terminal 210 through an output interface or store the encoded data in a database for subsequent transmission.

[0036] Figure 3For the video encoding method provided in an embodiment, a corresponding encoding framework diagram is provided. The video encoding method provided in the embodiment can obtain each input video frame of an input video sequence for encoding to obtain a corresponding to-be-encoded video frame. In the processing mode decision unit 302, a processing mode decision can be performed on the to-be-encoded video to obtain a processing mode corresponding to the to-be-encoded video frame, that is, through a resolution prediction network, the space-time correlation of the to-be-encoded video frame is utilized, a convolutional neural network and a recurrent neural network are used, and a downsampling ratio of the to-be-encoded video frame is output. In the processing unit 304, the to-be-encoded video frame can be processed according to the processing mode, that is, the to-be-encoded video frame is encoded according to the downsampling ratio and the original resolution. In the first prediction unit 306, each coding block of the to-be-encoded video frame can be intra-predicted or inter-predicted according to the downsampling ratio and the original resolution, a prediction value and a corresponding motion vector are obtained according to the image value of a reference block corresponding to the coding block, the prediction residual is obtained by subtracting the actual value of the coding block from the prediction value, and the motion vector represents the displacement of the coding block relative to the reference block. In the transformation unit 308, the prediction residual in the spatial domain and the vector information are transformed into the frequency domain, and the transformed coefficients can be encoded. The transformation method can be discrete Fourier transform or discrete cosine transform, and the vector information can be an actual motion vector or a motion vector difference value, and the motion vector difference value is the difference between the actual motion vector and a predicted motion vector.

[0037] In the quantization unit 310, the transformed data is mapped into another numerical value, for example, a smaller value can be obtained by dividing the transformed data by a quantization step. The quantization parameter is the serial number corresponding to the quantization step, and the corresponding quantization step can be found according to the quantization parameter. If the quantization parameter is small, most of the details of the to-be-encoded video frame are retained, and the corresponding code rate is high. If the quantization parameter is large, the corresponding code rate is low, but the distortion of the to-be-encoded video frame is large and the quality is not high. For example, in some video encoding standards, for luminance encoding, the quantization step has 52 values, which are integers between 0 and 51, and for chrominance encoding, the quantization step has integer values between 0 and 39, and the quantization step increases with the increase of the quantization parameter. When the quantization parameter increases by 6, the quantization step doubles.

[0038] The first inverse quantization unit 312, the first inverse transformation unit 314, the first reconstruction unit 316 and the reference information adaptive unit 318 are units corresponding to the reconstruction path, and each unit of the reconstruction path is used to reconstruct the video frame coded using the down-sampling ratio to obtain a reconstructed video frame of the down-sampling ratio, and each unit of the reconstruction path is used to reconstruct the video frame coded using the original resolution to obtain a reconstructed video frame of the original resolution. The step performed by the first inverse quantization unit 312 is the inverse process of quantization, the step performed by the first inverse transformation unit 314 is the inverse process of the transformation performed by the transformation unit 310, and the first reconstruction unit 316 is used to add the prediction data to the residual data obtained by the inverse transformation to obtain the reconstructed video frame. The distortion values of the reconstructed video frame obtained by using the down-sampling ratio coding and the reconstructed video frame obtained by using the original resolution coding are compared with the rate-distortion values adjusted according to the video frame type, and the optimal resolution is determined according to the video frame type.

[0039] The entropy coding unit 320 is used to perform entropy coding. If the optimal resolution is the down-sampling ratio, the down-sampling ratio coding data is subjected to entropy coding, if the optimal resolution is the original resolution, the original resolution coding data is subjected to entropy coding, and the optimal resolution identifier is added to the coding data for entropy coding. Entropy coding is a data coding method that does not lose any information and is performed according to the entropy principle, and can express certain information by using smaller characters. The entropy coding method can be, for example, Shannon coding or Huffman coding. Finally, the coding data is stored or transmitted by the storage and transmission unit 322, or is stored and transmitted.

[0040] Figure 4 For the decoding framework diagram corresponding to the video decoding method provided in an embodiment, the video decoding method provided in the embodiment can obtain the coding data corresponding to each to-be-decoded video frame of the to-be-decoded video sequence through the coding data acquisition unit 400. After entropy decoding is performed by the entropy decoding unit 402, entropy decoding data is obtained. The second inverse quantization unit 404 performs inverse quantization on the entropy decoding data to obtain inverse quantization data. The second inverse transformation unit 406 performs inverse transformation on the inverse quantization data to obtain inverse transformed data. The inverse transformed data can be used to obtain the reconstructed video frame of the down-sampling ratio or the original resolution. Figure 3The data obtained after the inverse transformation by the first inverse transform unit 314 is consistent. The resolution information acquisition unit 408 is used to acquire the optimal resolution identifier corresponding to the video frame to be decoded. The second reference information adaptation unit 412 is used to acquire the current reference frame reconstructed by the second reconstruction unit, and to perform adaptive processing on at least one of the following reference information: the current reference frame, the position information corresponding to each block to be decoded in the video frame to be decoded, the position information corresponding to each reference block in the current reference frame, and motion vectors, based on the optimal resolution identifier of the video frame to be decoded, and to perform prediction based on the information after adaptive processing. The second prediction unit 414 acquires the reference block corresponding to the block to be decoded based on the adaptively obtained reference information, and obtains the image value of the reference block and the corresponding reference block. Figure 3 The second reconstruction unit 410 reconstructs the video frame based on the predicted value and the inverse transform data, i.e., the prediction residual. The second processing unit 416 processes the reconstructed video frame according to the optimal resolution identifier corresponding to the video frame to be decoded, and obtains the corresponding decoded video frame. The playback and storage unit 418 can play or store the decoded video frame, or play and store both.

[0041] It is understood that the above-described encoding and decoding framework diagrams are merely examples and do not constitute a limitation on the encoding method to which the scheme of this application is applied. Specific encoding and decoding framework diagrams may include more or fewer units than those shown in the diagrams, or may combine certain units, or may have different component units. For example, loop filtering can also be performed on the reconstructed video frames to reduce the blockiness of the video frames and improve video quality.

[0042] The video frame to be encoded can be divided into multiple coding blocks, and the size of the coding block can be set or calculated as needed. For example, the size of each coding block can be 8*8 pixels. Alternatively, the rate-distortion cost corresponding to various coding block division methods can be calculated, and the division method with the lower rate-distortion cost can be selected for coding block division. Figure 5 The diagram shows the division of a 64x64 pixel image block, where each square represents a coded block. Figure 5 It is known that the size of the encoded block can include 32*32 pixels, 16*16 pixels, 8*8 pixels, and 4*4 pixels. Of course, the size of the encoded block can also be other sizes, such as 32*16 pixels or 64*64 pixels. It can be understood that during decoding, since there is a one-to-one correspondence between the encoded block and the block to be decoded, the pixel size of the block to be decoded can also include 32*32 pixels, 16*16 pixels, 8*8 pixels, and 4*4 pixels, etc.

[0043] According to another aspect of the present invention, a video encoding method is provided, such as... Figure 6 As shown, the method includes:

[0044] S601, acquire the video frame to be encoded;

[0045] S602, the downsampling ratio corresponding to the video frame to be encoded is obtained based on the resolution prediction network;

[0046] S603, the video frame to be encoded is encoded according to the downsampling ratio and the original resolution respectively, thereby obtaining the first encoded data and the second encoded data, and obtaining the first reconstructed video frame encoded based on the downsampling ratio and the second reconstructed video frame encoded based on the original resolution respectively.

[0047] S604, adjust the rate-distortion value of the second reconstructed video frame based on the type of the first or second reconstructed video frame to obtain the second rate-distortion value;

[0048] S605. Based on the comparison results of the distortion values ​​of the first reconstructed video frame and the second reconstructed video frame, and the comparison results of the first rate distortion value of the first reconstructed video frame and the second rate distortion value of the second reconstructed video frame, the optimal resolution is determined.

[0049] S606. Add the optimal resolution identifier to the corresponding first or second encoded data to obtain the encoded data.

[0050] Optionally, in this embodiment, the above-described video encoding method can be applied to, for example... Figure 7 The hardware environment consisting of server 902, server 702, client 904, and client 704 is shown. Figure 9 As shown, server 902 acquires the video frames to be encoded collected by client 904. The downsampling ratio of these video frames is obtained through a resolution prediction network. The downsampling ratio and the original resolution are used to encode the video frames. The distortion values ​​of the reconstructed video frames encoded using the downsampling ratio and those encoded using the original resolution are compared, along with the rate-distortion value adjusted according to the video frame type, to determine the optimal resolution. An optimal resolution identifier is added to the encoded data. Server 902 sends the encoded video to server 702 for decoding. Server 702 then sends the decoded video to client 704 for playback.

[0051] It should be noted that, for a given video, which is to be encoded, compressed, and decoded to obtain a new video, a resolution needs to be set for the new video, that is, the resolution that the new video is expected to have. This resolution is the original resolution mentioned above.

[0052] In addition, it should be noted that after the decoding of the to-be-decoded video frame is completed, the to-be-decoded video frame is used as a reference frame of a subsequent to-be-decoded video frame. The to-be-decoded video frame records which frames in the front have been decoded.

[0053] Optionally, in the embodiment, the video encoding method can be applied in the scene of audio and video processing, but is not limited to this. For example, a client A and a client B perform a video call, and the client A side and the client B side respectively collect video pictures, encode the collected video pictures, and send the encoded video to the other side. The other side decodes the received video and plays the decoded video.

[0054] Optionally, in the embodiment, the video encoding method can also be applied in the scene of playing a video file, video live broadcast, and the like, but is not limited to this.

[0055] The client can be various types of applications, for example, an online education application, an instant messaging application, a community space application, a game application, a shopping application, a browser application, a financial application, a multimedia application, a live broadcast application, and the like. Specifically, the video encoding method can be applied in the scene of processing audio and video in the instant messaging application, or can also be applied in the scene of processing audio and video in the multimedia application, so as to avoid large fluctuation of the peak signal-to-noise ratio of video encoding and decoding.

[0056] It can be seen that, by using the above steps, different frames in the video are adaptively encoded by using corresponding resolutions. In this way, the corresponding peak signal-to-noise ratio is relatively large and the distortion is relatively small, whether the transmission bandwidth is small or large, so as to ensure that the peak signal-to-noise ratio can change in a small range and the peak signal-to-noise ratio is relatively large, thereby achieving the technical effect of avoiding large fluctuation of the peak signal-to-noise ratio of video encoding and decoding, and solving the technical problem of large fluctuation of the peak signal-to-noise ratio caused by using the same resolution to encode and decode the video in the related art.

[0057] As an optional solution, the method comprises:

[0058] S1, obtaining a downsampling ratio of the to-be-encoded video frame by using a resolution prediction network;

[0059] S2, encoding the to-be-encoded video frame according to the downsampling ratio and an original resolution;

[0060] S3, comparing the distortion value and the rate-distortion value adjusted according to the video frame type of the reconstructed video frame obtained by using the downsampling ratio and the reconstructed video frame obtained by using the original resolution, respectively, to determine the optimal resolution.

[0061] As an optional solution, the resolution prediction network is used to obtain the down-sampling ratio of the video frame to be encoded, including: using the space-time correlation of the video frame, modeling the video frame by a convolutional neural network and a recurrent neural network, and outputting the down-sampling ratio corresponding to the video frame to be encoded.

[0062] Optionally, in the embodiment, a multi-layer deep neural network is designed, and the filter size and number of each layer of the neural network are fully adjusted to maximize the performance. In the details of the specific implementation, the input data is normalized, and the activation function of the intermediate layer is selected according to the characteristics of the down-sampling ratio in the present application (such as Sigmoid, Tanh function), and the activation function of the remaining layer is still kept to improve the convergence speed and prevent overfitting (such as RELU function), and the gradient descent optimization algorithm is selected as an adaptive optimizer (such as Adam), and after sufficient verification, the above settings can maximize the network performance. The neural network sets different feature map numbers for each layer of the network according to the feature quantity (for example, a seven-layer fully connected network, the network nodes can be 192-300-192-(intermediate node)-192-300-192).

[0063] As an optional solution, the down-sampling method refers to the specific sampling method, and common sampling methods include: point extraction, direct averaging, filter, bilinear interpolation, bicubic interpolation, CNN (convolutional neural network) based algorithm, and statistical characteristic based method. Different sampling methods have different computational complexities. The down-sampling method can be adaptively selected according to the application environment.

[0064] Optionally, in the embodiment, the down-sampling method corresponding to the input video frame is related to the current environmental factor, such as the high and low of the real-time requirement of the current video. If the real-time requirement is relatively high, a sampling method with low computational complexity is generally used, and if the real-time requirement is not too high, a sampling method with high computational complexity is used. The higher the computational complexity of the down-sampling method, the better the video quality obtained. In an embodiment, the current environmental factor further includes at least one of the following: model, CPU occupancy, CPU core number, CPU memory, etc. Generally, the higher the computational complexity that can be processed by the current environmental factor, the higher the computational complexity of the down-sampling method selected, and vice versa. The lower the computational complexity of the down-sampling method selected, which is conducive to improving the encoding speed while ensuring the video quality.

[0065] As an optional solution, the video frame to be encoded is encoded according to the downsampling ratio and the original resolution respectively, so that first encoding data and second encoding data are obtained, and first reconstructed video frame based on the downsampling ratio and second reconstructed video frame based on the original resolution are obtained, including:

[0066] S31, encoding the video frame to be encoded according to the original resolution to obtain second reconstructed video frame and corresponding second encoding data;

[0067] S32, performing downsampling processing on the video frame to be encoded according to the downsampling ratio to obtain first reconstructed video frame;

[0068] S33, obtaining a reference frame of the first reconstructed frame or the video frame to be encoded;

[0069] S34, processing the reference frame according to the downsampling ratio to obtain a first reference frame;

[0070] S35, encoding the first reconstructed video frame based on the first reference frame to obtain first encoding data of the first reconstructed video frame.

[0071] Optionally, in the embodiment, the distortion value can be selected as any video quality evaluation standard such as PSNR, SSIM, VMAF, etc. depending on which aspect of the video image quality is desired to be improved. For example, if the subjective quality of the video is desired to be improved, VMAF can be used as the standard for selecting the optimal resolution.

[0072] Optionally, in the embodiment, the rate-distortion value is calculated using the formula J(s, c, m|QP, λ m ) = SSD(s, c, m|QP) + λ m R(s, c, m|QP), wherein QP is the quantization parameter of the macroblock, λm is the Lagrange multiplier, which is related to QP, SSD(.) is the sum of squares of differences between the original luminance block s and the reconstructed block c, and R(.) is the number of bits encoded using mode m.

[0073] Optionally, in the embodiment, the rate-distortion value can be obtained by adding the rate-distortion values of all encoded blocks, or the average of the rate-distortion values of all encoded blocks can be used instead of the rate-distortion value for comparison, or the rate-distortion value can be recalculated for the entire video image.

[0074] Optionally, in the embodiment, threshold values can be added in the comparison of distortion values and the comparison of rate-distortion values, to balance the effects of saving bit rate and improving video quality. For example, when the distortion value of a video image coded using a down-sampling ratio is less than 1.2 times the distortion value of a video image coded using the original resolution, the down-sampling resolution can be selected as the optimal resolution of the video image. In this embodiment, more down-sampling resolutions can be selected to code video frames, to save more bits while maintaining a relatively good video quality.

[0075] Optionally, in the embodiment, threshold values can be added in the comparison of rate-distortion values. For example, when the rate-distortion value of a video frame coded using a down-sampling ratio is less than 0.9 times the rate-distortion value of a video frame coded using the original resolution, the down-sampling resolution can be selected as the optimal resolution of the video image. This embodiment can achieve the beneficial result of saving more bits.

[0076] As an optional solution, the distortion values and the rate-distortion values adjusted according to the video frame types of the reconstructed video frames coded using a down-sampling ratio and the reconstructed video frames coded using the original resolution are compared respectively, to determine the optimal resolution, including:

[0077] S1, coding the video frame to be coded using the down-sampling ratio and the original resolution, to obtain coded data corresponding to the video frame to be coded;

[0078] S2, determining flag data corresponding to each video frame according to the distortion values and the rate-distortion values, wherein the flag data is used to identify the optimal resolution corresponding to each video frame;

[0079] S3, adding the optimal resolution flag data as a syntax element into the coded data corresponding to each video frame.

[0080] Optionally, in the embodiment, the syntax element used to indicate the resolution used to code each video frame can be a piece of data located at a fixed position of the video frame to be decoded, and different data values at the position represent different resolutions. The syntax element representing the resolution corresponding to the video frame can be added at the position.

[0081] Optionally, in the embodiment, different identification values can be directly used in the syntax element to represent different resolutions. For example, resolution 1 is represented by 00, resolution 2 is represented by 01, resolution 3 is represented by 10, and resolution 4 is represented by 11. It should be noted that the way of representing resolutions by identification values is not limited to this, and various identification value representation methods capable of distinguishing resolutions can be used to indicate different resolutions used in different regions.

[0082] Optionally, in the embodiment, the flag data can be determined in the following manner, but is not limited to:

[0083] S1, determining an identification value corresponding to each video frame according to a relationship between a resolution corresponding to the video frame and a resolution corresponding to a previous video frame of the video frame;

[0084] S2, determining the identification value corresponding to each video frame as the flag data.

[0085] Optionally, in the embodiment, the identification value corresponding to the current video frame can be used to indicate a relationship between a resolution of the current video frame and a resolution of a previous video frame of the current video frame. The identification value corresponding to each video frame is determined according to the relationship between the resolution of the current video frame and the resolution of the previous video frame.

[0086] As an optional solution, the identification value corresponding to each video frame is determined according to a relationship between a resolution corresponding to the video frame and a resolution corresponding to a previous video frame of the video frame, and the method comprises:

[0087] S1, in a case where the resolution corresponding to the video frame is same as the resolution corresponding to the previous video frame of the video frame, determining the identification value corresponding to the video frame as a first identification value;

[0088] S2, in a case where the resolution corresponding to the video frame is different from the resolution corresponding to the previous video frame of the video frame, determining the identification value corresponding to the video frame as a second identification value.

[0089] Optionally, in the embodiment, the relationship between the resolutions can include, but is not limited to, the same resolution or different resolution. The first identification value is used to represent the same resolution, and the second identification value is used to represent the different resolution. For example, 1 is used to identify the same resolution, and 0 is used to represent the different resolution, or 0 is used to represent the same resolution, and 1 is used to represent the different resolution.

[0090] Optionally, in the embodiment, for the first video frame in each frame, the identification value can be used to directly represent a resolution adopted by the video frame.

[0091] Optionally, in the embodiment, the flag data can also be determined for each video frame according to a relationship between a first resolution corresponding to the video frame and a second resolution corresponding to a reference video frame of the video frame to be encoded.

[0092] As an optional solution, the flag bit data corresponding to each video frame is determined according to a relationship between the first resolution corresponding to each video frame and the second resolution of the reference video frame of the video frame to be encoded, and the flag bit data comprises:

[0093] S1, determining an identification value corresponding to each video frame according to a relationship between the first resolution corresponding to each video frame and the second resolution of the reference video frame;

[0094] S2, determining the identification value corresponding to each video frame as the flag bit data.

[0095] Optionally, in the embodiment, the relationship between the resolutions can be represented by the identification value, which can represent whether the resolutions are the same, a gap between the levels corresponding to the resolutions, etc.

[0096] Optionally, in the embodiment, the identification value corresponding to each video frame can be added to a position of the flag bit data.

[0097] As an optional solution, the identification value corresponding to each video frame is determined according to a relationship between the first resolution corresponding to each video frame and the second resolution of the reference video frame, and the identification value comprises:

[0098] S1, determining the identification value corresponding to each video frame as a first identification value in a case where the first resolution corresponding to each video frame is the same as the second resolution corresponding to the reference region;

[0099] S2, determining the identification value corresponding to each video frame as a second identification value in a case where the first resolution corresponding to each video frame is different from the second resolution corresponding to the reference video frame.

[0100] Optionally, in the embodiment, if the resolution of each video frame is the same as the resolution of the reference video frame, the first identification value can be used to represent, and if the resolution of each video frame is different from the resolution of the reference video frame, the second identification value can be used to represent. For example, 1 represents that the two are the same, and 0 represents that the two are different. Alternatively, 1 represents that the two are different, and 0 represents that the two are the same.

[0101] According to an aspect of an embodiment of the present application, a video decoding method is provided, as shown in the following table: Figure 8 The method comprises:

[0102] S801, obtaining a video frame to be decoded comprising the encoding data;

[0103] S802, obtaining an optimal resolution identification corresponding to the video frame to be decoded from the encoding data, and obtaining a decoding resolution of the video frame to be decoded based on the optimal resolution identification;

[0104] S803, decode the video frame to be decoded based on the decoding resolution to obtain a decoded video frame.

[0105] Optionally, in this embodiment, the above-described video decoding method can be applied to, for example... Figure 9 The hardware environment consisting of server 702 and client 704 is shown. Figure 9 As shown, server 702 acquires the video frame to be decoded, retrieves the optimal resolution flag data corresponding to the video frame to be decoded from the encoded data, wherein the optimal resolution flag data is used to identify the decoding resolution corresponding to the video frame to be decoded, and decodes the video frame to be decoded using the decoding resolution. Server 702 sends the decoded video to client 704 for playback.

[0106] Optionally, in this embodiment, the above video decoding method can be applied to, but is not limited to, audio and video processing scenarios. For example, when client A and client B are having a video call, client A and client B respectively capture video frames, encode the captured video frames, and send the encoded video to the other party. The other party then decodes the received video and plays the decoded video.

[0107] Optionally, in this embodiment, the above video decoding method can also be applied to scenarios such as video file playback and live video streaming.

[0108] The aforementioned client can be, but is not limited to, various types of applications, such as online education applications, instant messaging applications, social networking applications, game applications, shopping applications, browser applications, financial applications, multimedia applications, live streaming applications, etc. Specifically, it can be applied, but is not limited to, scenarios involving audio and video processing in the aforementioned instant messaging applications, or scenarios involving audio and video processing in the aforementioned multimedia applications, to avoid large fluctuations in the peak signal-to-noise ratio during video encoding and decoding. The above is merely an example, and this embodiment does not impose any limitations.

[0109] As can be seen, by using the above steps to adaptively encode and decode different frames in the video using the corresponding resolution, the peak signal-to-noise ratio (PSNR) is relatively large and the distortion is relatively small, regardless of whether the transmission bandwidth is small or large. This ensures that the PSNR can vary within a small range and remains relatively large, thus achieving the technical effect of avoiding large fluctuations in PSNR when encoding and decoding the video. This solves the technical problem of large fluctuations in PSNR caused by using the same resolution for video encoding and decoding in related technologies.

[0110] As an optional solution, a corresponding resolution is determined for different frames in the video, including:

[0111] S1, obtaining the optimal resolution flag data corresponding to the to-be-decoded video frame from the syntax element of the to-be-decoded video frame, wherein the flag data is used to identify the resolution corresponding to the to-be-decoded video frame.

[0112] Optionally, in the embodiment, the resolution corresponding to each frame can be, but is not limited to, added by the encoding end in the syntax element and transmitted to the decoding end. The correspondence between the video frame and the resolution is indicated by the flag data in the syntax element, so that the decoding end decodes the to-be-decoded video frame using the resolution corresponding to the to-be-decoded video frame.

[0113] Optionally, in the embodiment, the syntax element used to indicate the resolution adopted by the encoding of each frame can be a piece of data located at a fixed position of the to-be-decoded video frame, and different data values at the position represent different resolutions. The syntax element representing the resolution corresponding to the to-be-decoded video frame can be added at the position.

[0114] Optionally, in the embodiment, after obtaining the reconstructed video frame of the to-be-decoded video frame;

[0115] adjusting the reconstructed video frame of the to-be-decoded video frame to a target resolution.

[0116] wherein the adjustment to the target resolution includes:

[0117] 1) in the case where the target resolution is equal to the resolution of the reconstructed video frame, no adjustment is made;

[0118] 2) in the case where the target resolution is greater than the resolution of the reconstructed video frame, the resolution of the reconstructed video frame is adjusted to the target resolution.

[0119] According to another aspect of the embodiment of the application, there is also provided a video decoding device for implementing the above-mentioned video decoding method, as shown in Figure 10 The device includes:

[0120] A first obtaining module 1002 is configured to obtain the encoding data corresponding to the to-be-decoded video frame.

[0121] A second obtaining module 1004, i.e., an optimal resolution obtaining module, is configured to obtain the decoding resolution corresponding to the to-be-decoded video frame from the encoding data.

[0122] A decoding module 1006 is configured to decode the encoding data according to the decoding resolution corresponding to the to-be-decoded video frame, to obtain the corresponding decoded video frame.

[0123] Optionally, the second obtaining module is configured to obtain the optimal resolution flag data corresponding to each video frame from syntax elements of the video frame to be decoded, wherein the flag data is used to identify the resolution corresponding to each video frame.

[0124] According to another aspect of the embodiments of the present application, a video encoding device for implementing the video encoding method is also provided, as shown in Figure 11 The device comprises:

[0125] The third obtaining module 1102 is configured to obtain an input video frame.

[0126] The first determining module 1104 is configured to determine a down-sampling ratio corresponding to the input video frame.

[0127] The encoding module 1106 is configured to encode the input video frame according to the down-sampling ratio and the original resolution respectively to obtain encoding data corresponding to the input video frame.

[0128] The adding module 1108 is configured to compare the distortion value and the rate-distortion value adjusted according to the video frame type of the reconstructed video frame obtained by using the down-sampling ratio and the reconstructed video frame obtained by using the original resolution respectively, determine an optimal resolution, and add the optimal resolution to the encoding data.

[0129] Optionally, the first determining module is configured to:

[0130] use a resolution prediction network, wherein the resolution prediction network is configured to utilize the space-time domain correlation of the video frame, model the video frame by using a convolutional neural network and a recurrent neural network, and output the down-sampling ratio corresponding to the video frame to be encoded.

[0131] Optionally, the encoding module comprises:

[0132] an encoding unit configured to encode the video frame by using the down-sampling ratio and the original resolution to obtain encoded data of each video frame;

[0133] an adding unit configured to compare the distortion value and the rate-distortion value adjusted according to the video frame type of the reconstructed video frame obtained by using the down-sampling ratio and the reconstructed video frame obtained by using the original resolution respectively, determine an optimal resolution, and add the flag data as syntax elements into the encoded data corresponding to each video frame.

[0134] As shown in Figure 12 In the video encoding process of the present application, different resolutions corresponding to different frames are adaptively used for encoding, so that no matter whether the bandwidth is small (for example, less than 1 Mbps) or large (for example, greater than 1 Mbps), the video can be encoded in a high quality and a low bit rate. Figure 12In the case that the bandwidth of the transmission is relatively small (for example, less than the bandwidth threshold Th shown in the figure), or in the case that the bandwidth of the transmission is relatively large (for example, greater than the bandwidth threshold Th shown in the figure) Figure 12 In the case that the bandwidth of the transmission is relatively small (for example, less than the bandwidth threshold Th shown in the figure), or in the case that the bandwidth of the transmission is relatively large (for example, greater than the bandwidth threshold Th shown in the figure)

[0135] In addition, since the corresponding resolution is adaptively used for different frames in the video, the corresponding resolution does not need to be selected according to the intersection (such as the intersection in the figure) corresponding to different types of videos or different frames of the same video when encoding the frames in the video, and the encoding complexity is reduced. Figure 1

[0136] According to still another aspect of the embodiments of the present application, an electronic device for implementing the above-mentioned video encoding method or video decoding method is further provided, as shown in the figure, the electronic device comprises one or more (only one is shown in the figure) processors 1302, a memory 1304, sensors 1306, an encoder 1308, and a transmission device 1310, the memory stores a computer program, and the processor is configured to execute the steps in any one of the method embodiments above by the computer program. Figure 13

[0137] Optionally, in the present embodiment, the above-mentioned electronic device can be located in at least one of the network devices in the computer network.

[0138] Optionally, in the present embodiment, the above-mentioned processor can be configured to execute the following steps by the computer program:

[0139] S1, obtaining the encoding data corresponding to the to-be-decoded video frame;

[0140] S2, obtaining the optimal resolution flag data corresponding to the to-be-decoded video frame from the encoding data, wherein the optimal resolution flag data is used to identify the decoding resolution corresponding to the to-be-decoded video frame;

[0141] S3, decoding the to-be-decoded video frame according to the decoding resolution to obtain the corresponding decoded video frame.

[0142] Optionally, in the present embodiment, the above-mentioned processor can be further configured to execute the following steps by the computer program:

[0143] S1, obtaining the to-be-encoded video frame;

[0144] S2, using the resolution prediction network to output the down-sampling ratio corresponding to the to-be-encoded video frame;

[0145] S3, encoding the to-be-encoded video frame according to the down-sampling ratio and the original resolution, respectively;

[0146] ​​S4, compare the distortion value of the reconstructed video frame obtained by using the down-sampling ratio coding and the reconstructed video frame obtained by using the original resolution coding, and the rate-distortion value adjusted according to the video frame type, respectively, to determine the optimal resolution;

[0147] S5, add the optimal resolution identifier to the encoded data.

[0148] Optionally, those skilled in the art can understand that, Figure 13 The structure shown is only schematic, and the electronic device can also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a Mobile Internet Device (MID), a PAD, or the like. Figure 13 It does not limit the structure of the electronic device described above. For example, the electronic device can further include more or less components (such as a network interface, a display device, etc.) than those shown in the figure, or have a different configuration from that shown in the figure. Figure 13 Figure 13

[0149] The memory 1304 can be used to store software programs and modules, such as program instructions / modules corresponding to the video decoding method and device in the embodiments of the present application. The processor 1302 executes various functions and data processing by running the software programs and modules stored in the memory 1304, that is, implements the control method of the target component described above. The memory 1304 can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 1304 can further include a memory remotely arranged with respect to the processor 1302, which can be connected to the terminal through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0150] The transmission device 1310 described above is used to receive or send data via a network. Specific examples of the above-mentioned network can include wired networks and wireless networks. In one example, the transmission device 1310 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers through a network cable to communicate with the Internet or a local area network. In one example, the transmission device 1310 is a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet in a wireless manner.

[0151] Specifically, the memory 1304 is used to store application programs.

[0152] ​​The embodiment of the present application also provides a storage medium, wherein the storage medium stores a computer program, and the computer program is configured to execute the steps in any of the method embodiments.

[0153] Optionally, in the embodiment, the storage medium is configured to store the computer program for executing the following steps.

[0154] S1, obtaining encoding data corresponding to a to-be-decoded video frame;

[0155] S2, obtaining optimal resolution flag data corresponding to the to-be-decoded video frame from the encoding data, wherein the optimal resolution flag data is used to identify a decoding resolution corresponding to the to-be-decoded video frame;

[0156] S3, decoding the to-be-decoded video frame according to the decoding resolution to obtain a corresponding decoded video frame.

[0157] Optionally, the storage medium is also configured to store the computer program for executing the following steps.

[0158] S1, obtaining a to-be-encoded video frame;

[0159] S2, using a resolution prediction network to output a down-sampling ratio corresponding to the to-be-encoded video frame;

[0160] S3, encoding the to-be-encoded video frame according to the down-sampling ratio and an original resolution respectively;

[0161] S4, respectively comparing distortion values and rate-distortion values adjusted according to video frame types of reconstructed video frames obtained by using the down-sampling ratio for encoding and reconstructed video frames obtained by using the original resolution for encoding, to determine an optimal resolution;

[0162] S5, adding the optimal resolution identification to the encoding data.

[0163] Optionally, the storage medium is also configured to store the computer program for executing the steps included in the method in the above embodiment, and details are not described herein.

[0164] Optionally, in the embodiment, a person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device by a program, and the program can be stored in a computer readable storage medium, and the storage medium can include a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0165] The above-mentioned embodiment numbers of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0166] The integrated units in the above-mentioned embodiments, if realized in the form of software function units and sold or used as independent products, can be stored in the above-mentioned computer-readable storage medium. Based on such understanding, the technical solutions of the present application or the whole or part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product stored in the storage medium includes a plurality of instructions for causing one or more computer devices (which can be personal computers, servers or network devices, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0167] In the above-mentioned embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0168] In several embodiments provided in the present application, it should be understood that the disclosed client can be implemented by other ways. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, units or modules, which can be electrical or other forms.

[0169] The units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0170] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or software function unit.

[0171] The above-mentioned is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be regarded as the protection scope of the present application.

Claims

1. A method of video coding, the method comprising: The method comprises the following steps: obtaining a to-be-encoded video frame; obtaining a down-sampling ratio corresponding to the to-be-encoded video frame based on a resolution prediction network; encoding the to-be-encoded video frame according to the down-sampling ratio and an original resolution respectively, thereby obtaining first encoding data and second encoding data corresponding respectively, and obtaining a first reconstructed video frame based on the down-sampling ratio and a second reconstructed video frame based on the original resolution; adjusting a rate-distortion value of the second reconstructed video frame based on a type of the first or second reconstructed video frame, thereby obtaining a second rate-distortion value; determining an optimal resolution based on a comparison result of distortion values of the first and second reconstructed video frames and a comparison result of the first rate-distortion value of the first reconstructed video frame and the second rate-distortion value of the second reconstructed video frame; adding the optimal resolution identifier into the corresponding first encoding data or second encoding data, thereby obtaining encoding data; encoding the to-be-encoded video frame according to the down-sampling ratio and an original resolution respectively, thereby obtaining first encoding data and second encoding data corresponding respectively, and obtaining a first reconstructed video frame based on the down-sampling ratio and a second reconstructed video frame based on the original resolution, which comprises: encoding the to-be-encoded video frame according to the original resolution, thereby obtaining the second reconstructed video frame and corresponding second encoding data; performing down-sampling processing on the to-be-encoded video frame according to the down-sampling ratio, thereby obtaining the first reconstructed video frame; obtaining a reference frame of the first reconstructed video frame or the to-be-encoded video frame; processing the reference frame according to the down-sampling ratio, thereby obtaining a first reference frame; encoding the first reconstructed video according to the first reference frame, thereby obtaining the first encoding data; adjusting a rate-distortion value of the second reconstructed video frame based on a type of the first or second reconstructed video frame, thereby obtaining a second rate-distortion value, which comprises: if the video type of the reconstructed video frame is an I frame or a P frame or a B frame which is referenced, reducing the rate-distortion value of the second reconstructed video frame by a preset first proportion value, if the type of the reconstructed video frame is a B frame which is not referenced, increasing the rate-distortion value of the second reconstructed video frame by a preset second proportion value, and the adjusted rate-distortion value is taken as the second rate-distortion value.

2. The video coding method of claim 1, wherein, determining an optimal resolution based on a comparison result of distortion values of the first and second reconstructed video frames and a comparison result of the first rate-distortion value of the first reconstructed video frame and the second rate-distortion value of the second reconstructed video frame, which comprises: if the distortion value of the first reconstructed video frame is smaller than the distortion value of the second reconstructed video frame, and the first rate-distortion value of the first reconstructed video frame is smaller than the second rate-distortion value of the second reconstructed video frame, the optimal resolution is the down-sampling ratio, otherwise, the optimal resolution is the original resolution.

3. The video coding method of claim 2, wherein, the adding of the optimal resolution identifier into the corresponding first encoding data or second encoding data, thereby obtaining encoding data, which comprises: if the optimal resolution is the down-sampling ratio, adding the optimal resolution identifier into the first encoding data, and taking the first encoding data with the added optimal resolution identifier as the encoding data. If the optimal resolution is the original resolution, an optimal resolution identifier is added to the second encoded data, and the second encoded data with the added optimal resolution identifier is used as the encoded data.

4. The video coding method of any of claims 1-3, wherein, The resolution prediction network is configured to utilize the spatial-temporal correlation of the video frame to be encoded, model the video frame to be encoded by a convolutional neural network and a recurrent neural network, and output a down-sampling ratio corresponding to the video frame to be encoded.

5. A method of video decoding, the method comprising: The method comprises: obtaining a video frame to be decoded including encoded data according to any one of claims 1-4; obtaining an optimal resolution identifier corresponding to the video frame to be decoded from the encoded data, and obtaining a decoding resolution of the video frame to be decoded based on the optimal resolution identifier; decoding the video frame to be decoded based on the decoding resolution to obtain a decoded video frame; The decoding of the video frame to be decoded based on the decoding resolution to obtain a decoded video frame comprises: obtaining a reference frame corresponding to the video frame to be decoded; processing the reference frame according to the decoding resolution to obtain a second reference frame; decoding the encoded data according to the decoding resolution and the second reference frame to obtain a reconstructed video frame; processing the reconstructed video frame according to the original resolution to obtain a decoded video frame.

6. A video decoding apparatus for performing the video decoding method of claim 5, wherein The apparatus comprises: a first obtaining module configured to obtain encoded data corresponding to a video frame to be decoded; a second obtaining module configured to obtain a decoding resolution corresponding to the video frame to be decoded from the encoded data; a decoding module configured to decode the encoded data according to the decoding resolution corresponding to the video frame to be decoded to obtain a corresponding decoded video frame.

7. A video encoding apparatus for performing the video encoding method of any one of claims 1-4. The apparatus comprises: a third obtaining module configured to obtain a video frame to be encoded; a first determining module configured to determine a down-sampling ratio corresponding to the video frame to be encoded; an encoding module configured to encode the video frame to be encoded according to the original resolution and the down-sampling ratio respectively, and obtain encoded data of the video frame encoded using the original resolution and encoded data of the video frame encoded using the down-sampling ratio; an adding module configured to compare distortion values of reconstructed video frames obtained by encoding using the down-sampling ratio and reconstructed video frames obtained by encoding using the original resolution respectively, and rate-distortion values adjusted according to a video frame type, determine an optimal resolution, and add the optimal resolution to the encoded data.

Images