Video encoding method, and training method, device and equipment of preprocessing network

Abstract

The application provides a video coding method and a training method, device and equipment of a preprocessing network. The video coding method comprises: preprocessing a first video frame by using a trained preprocessing network to obtain a second video frame; wherein the preprocessing comprises: extracting local features and global features of the first video frame, fusing the local features and the global features, and obtaining the second video frame based on a fusion result; and inputting the second video frame into a first video encoder to generate a code stream.

Description

Technical Field

[0001] This application relates to electronic technology, including but not limited to video encoding methods, apparatus, and equipment. Background Technology

[0002] Video preprocessing methods refer to preprocessing videos before encoding, such as denoising, brightness adjustment, super-resolution, image quality enhancement, and / or semantic coding, to improve the efficiency and quality of video encoding. This method can improve quality and encoding efficiency in various ways. For example, noise reduction, deblocking, and ring removal can reduce noise and distortion in the video, thereby improving visual effects. Furthermore, video content analysis algorithms can be used to apply specific processing methods to different scenarios, further optimizing encoding efficiency and output quality. Summary of the Invention

[0003] In a first aspect, embodiments of this application provide a video encoding method, the method comprising: preprocessing a first video frame using a trained preprocessing network to obtain a second video frame; wherein the preprocessing comprises: extracting local features and global features of the first video frame, fusing the local features and the global features, and obtaining the second video frame based on the fusion result; and inputting the second video frame into a first video encoder to generate a bitstream.

[0004] It is understood that in this embodiment of the application, the trained preprocessing network preprocesses the first video frame. The preprocessing includes extracting local features and global features of the first video frame, fusing the local features and global features, and obtaining the second video frame based on the fusion result. In this way, the features of the second video frame are richer (including both local features and global features of the first video frame), so as to achieve the purpose of image quality enhancement, thereby improving the quality of the reconstructed image of the video codec, and thus improving the performance of the video codec.

[0005] Secondly, embodiments of this application provide a training method for a preprocessing network. The method includes: preprocessing a third video frame using the preprocessing network to obtain a fourth video frame; determining the value of a first loss function based on the third video frame and the fourth video frame; inputting the fourth video frame into a second video encoder to obtain the bitrate of the fourth video frame, and using the bitrate of the fourth video frame as the value of the second loss function; adjusting the model parameters of the preprocessing network when the value of the first loss function is greater than or equal to a first threshold and / or the value of the second loss function is greater than or equal to a second threshold, or when the current training epoch is less than a third threshold, and iteratively executing the above steps until the value of the first loss function is less than the first threshold and the value of the second loss function is less than the second threshold, or until the current training epoch is the third threshold, thereby obtaining a trained preprocessing network.

[0006] It is understood that in the embodiments of this application, the loss function used to train the preprocessing network has two dimensions: one is the bitrate, and the other is the difference between the input and output of the preprocessing network. In this way, the video frames output by the trained preprocessing network can both help the video encoder improve compression efficiency and reduce bitrate, while ensuring the video quality of the reconstructed image.

[0007] Thirdly, embodiments of this application provide a video encoding apparatus, the apparatus including a first preprocessing module and an encoding module; wherein, the first preprocessing module is configured to preprocess a first video frame using a trained preprocessing network to obtain a second video frame; wherein, the preprocessing includes: extracting local features and global features of the first video frame, fusing the local features and the global features, and obtaining the second video frame based on the fusion result; the encoding module is configured to encode the second video frame to generate a bitstream.

[0008] Fourthly, embodiments of this application provide a training apparatus for a preprocessing network. The apparatus includes: a second preprocessing module configured to preprocess a third video frame using the preprocessing network to obtain a fourth video frame; a determination module configured to determine the value of a first loss function based on the third video frame and the fourth video frame; an input module configured to input the fourth video frame into a second video encoder to obtain the bitrate of the fourth video frame and use the bitrate of the fourth video frame as the value of the second loss function; and an adjustment module configured to adjust the model parameters of the preprocessing network when the value of the first loss function is greater than or equal to a first threshold and / or the value of the second loss function is greater than or equal to a second threshold, or when the current training epoch is less than a third threshold, and iteratively execute the functions of the second preprocessing module, the determination module, and the input module until the value of the first loss function is less than the first threshold and the value of the second loss function is less than the second threshold, or until the current training epoch is the third threshold, thereby obtaining a trained preprocessing network.

[0009] Fifthly, embodiments of this application provide an electronic device, the electronic device including a processor and a first video encoder; wherein: the processor is configured to preprocess a first video frame using a trained preprocessing network to obtain a second video frame; wherein the preprocessing includes: extracting local features and global features of the first video frame, fusing the local features and the global features, and obtaining the second video frame based on the fusion result; the first video encoder is configured to encode the second video frame to generate a bitstream.

[0010] In a sixth aspect, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor executes the program to implement the method described in the first or second aspect.

[0011] In a seventh aspect, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor or electronic device, implements the method described in the first or second aspect.

[0012] Eighthly, embodiments of this application provide a computer program product, including a computer program or instructions, which, when executed by a processor or electronic device, implement the method described in the first or second aspect of this application.

[0013] Ninthly, embodiments of this application provide a computer program that causes a processor or electronic device to perform the method described in the first or second aspect.

[0014] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application. Obviously, the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0016] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0017] Figure 1 A schematic diagram illustrating the implementation process of the video encoding method provided in this application embodiment;

[0018] Figure 2 A schematic diagram illustrating the implementation flow of the local and global feature extraction methods provided in the embodiments of this application;

[0019] Figure 3 This is a schematic diagram of the structure of the preprocessing network provided in the embodiments of this application;

[0020] Figure 4 A schematic diagram illustrating the implementation flow of the bitrate determination method provided in this application embodiment;

[0021] Figure 5 A schematic diagram illustrating the implementation flow of the training method for the preprocessing network provided in this application embodiment;

[0022] Figure 6 A schematic diagram of the training process of the preprocessing network provided in the embodiments of this application;

[0023] Figure 7 A schematic diagram illustrating the experimental comparison results of H.265 and H.264 using this solution and H.265 and H.264 without the preprocessing method in the same video, provided for embodiments of this application;

[0024] Figure 8 This is a schematic diagram of the structure of the video encoding device provided in the embodiments of this application;

[0025] Figure 9 A schematic diagram of the structure of the training device for the preprocessing network provided in the embodiments of this application;

[0026] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0029] In the following description, references to "some embodiments," "this embodiment," "this application embodiment," and examples, etc., describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subset of all possible embodiments and may be combined with each other without conflict.

[0030] The descriptions of "first," "second," and "third" appearing in the embodiments of this application are not in any particular order, nor do they indicate a special limitation on the number of devices in the embodiments of this application, and cannot constitute any limitation on the embodiments of this application.

[0031] With the rapid growth in demand for video content, network bandwidth and storage space are increasingly under pressure. To address these challenges, video coding technology is constantly evolving, aiming to reduce bitrate by improving compression efficiency while maintaining video quality.

[0032] Video preprocessing methods refer to preprocessing videos before encoding, such as denoising and brightness adjustment, to improve the efficiency and quality of video encoding. This method can enhance quality and encoding efficiency in various ways. For example, noise reduction, deblocking, and ring removal can reduce noise and distortion in the video, thereby improving visual effects. Furthermore, video content analysis algorithms can be used to apply specific processing methods to different scenarios, further optimizing encoding efficiency and output quality. With the development of artificial intelligence algorithms, especially deep learning, deep learning models can perform deeper analysis and processing of video content, achieving higher-level intelligent preprocessing functions such as super-resolution, image quality enhancement, and semantic coding to adapt to currently popular video encoding standards such as H.264 / H.265 and improve video encoding performance.

[0033] The inventors of this application, through research and analysis of relevant preprocessing methods, discovered that in the video feature extraction stage, these methods typically employ multi-layer convolution to globally model image features. However, these algorithms neglect the coordination relationship between local and global features. Although multi-layer convolution can capture image features, the result may lead to the loss of detailed features. This neglect is particularly evident in image detail restoration and reconstruction tasks, especially when processing high-resolution images or complex scenes. Simply relying on global feature modeling is insufficient to preserve image detail information, thus limiting the performance of these algorithms in practical applications. To improve the feature extraction capability of preprocessing methods, in some embodiments of this application, a spatial attention mechanism is introduced to combine global and local features, obtaining richer image features and thereby improving the effect of image detail restoration and reconstruction.

[0034] Furthermore, the inventors of this application, through research and analysis of relevant motion estimation methods, discovered that in the inter-frame motion estimation stage, related algorithms use block matching to complete motion estimation. This involves dividing the current frame into fixed-size macroblocks, typically 16x16 pixels. In a reference frame, a search is performed for each macroblock of the current frame to find the best matching block. Within the search window, the reference frame block that minimizes the error is selected as the best match. The displacement of the current macroblock relative to the best matching block, i.e., the motion vector, is calculated. Due to the macroblock division, motion information at block boundaries is prone to discontinuity and distortion, which negatively impacts the calculation of predicted frames. Therefore, in some embodiments of this application, during the training stage of the preprocessing network, optical flow is used to obtain the motion vector of each pixel. The resulting optical flow field is more accurate than block matching with a fixed block size, thereby increasing the accuracy of predicted frames, reducing the bitrate of residual frames, and ultimately enhancing the effect of indirectly reducing the bitrate through preprocessing methods.

[0035] The following is a video encoding method provided in an embodiment of this application.

[0036] Figure 1 This is a schematic diagram illustrating the implementation flow of the video encoding method provided in the embodiments of this application; as follows: Figure 1 As shown, the method may include the following steps 101 to 102:

[0037] Step 101: Preprocess the first video frame using the trained preprocessing network to obtain the second video frame; wherein, the preprocessing includes: extracting local features and global features of the first video frame, fusing the local features and the global features, and obtaining the second video frame based on the fusion result;

[0038] Step 102: Input the second video frame into the first video encoder to generate a bitstream.

[0039] It is understood that in this embodiment of the application, the trained preprocessing network preprocesses the first video frame. The preprocessing includes extracting local features and global features of the first video frame, fusing the local features and global features, and obtaining the second video frame based on the fusion result. In this way, the features of the second video frame are richer (including both local features and global features of the first video frame), so as to achieve the purpose of image quality enhancement, thereby improving the quality of the reconstructed image of the video codec, and thus improving the performance of the video codec.

[0040] The following sections will describe further optional implementation methods for each of the above steps, as well as related terms.

[0041] Step 101: Preprocess the first video frame using the trained preprocessing network to obtain the second video frame; wherein the preprocessing includes: extracting local features and global features of the first video frame, fusing the local features and the global features, and obtaining the second video frame based on the fusion result.

[0042] In some embodiments, the local features include edge information and / or texture information of the first video frame.

[0043] In some embodiments, the global features include semantic information / features of the first video frame.

[0044] In this embodiment of the application, the structure of the trained preprocessing network is not limited. The preprocessing network can be a variety of neural network models that can extract local features and global features of an image.

[0045] For example, in some embodiments, the trained preprocessing network includes convolutional layers and multiple deep convolutional layers; correspondingly, for the extraction of local and global features of the first video frame mentioned in step 101, such as Figure 2 As shown, it may further include the following steps 201 and 202:

[0046] Step 201: Extract local features of the first video frame using the convolutional layer to obtain multiple first feature maps, where each first feature map includes the local features. For example, the local features include edge information and / or texture information of the first video frame.

[0047] Step 202: Based on the multi-layer deep convolutional layers and the plurality of first feature maps, extract the global features of the first video frame to obtain a plurality of second feature maps, wherein the second feature maps include the global features. For example, the global features include the semantic information / features of the first video frame.

[0048] In one possible implementation, step 202 may further include: inputting the first feature map into the multi-layer deep convolutional layer after passing through the activation layer and pooling layer to obtain the second feature map.

[0049] Furthermore, in some embodiments, the trained preprocessing network further includes a residual network and a feature aggregation layer; correspondingly, the fusion of the local features and the global features mentioned in step 101 may further include: inputting multiple second feature maps corresponding to the first video frame and multiple second feature maps corresponding to the previous frame of the first video frame into the residual network to obtain multiple third feature maps; fusing the multiple third feature maps using the feature aggregation layer to obtain a fourth feature map; wherein the fourth feature map is the fusion result.

[0050] Furthermore, in some embodiments, the trained preprocessing network further includes a spatial attention mechanism and an upsampling layer; correspondingly, the step of obtaining the second video frame based on the fusion result mentioned in step 101 may further include: using the spatial attention mechanism to enhance the detailed features in the fourth feature map to obtain a fifth feature map; and using the upsampling layer to perform an upsampling operation on the fifth feature map to obtain the second video frame.

[0051] It is understood that in the video encoding method provided in this application embodiment, the spatial attention mechanism is used to enhance the detailed features in the fourth feature map, thereby obtaining richer features and thus improving the image quality of the second video frame. The higher quality second video frame input into the first video encoder helps the video decoder obtain a higher quality reconstructed image.

[0052] In one possible implementation, the preprocessing network is structured as follows: Figure 3 As shown in 301, the structure of the residual feature module is as follows: Figure 3 As shown in Figure 302, the structure of the spatial attention mechanism is as follows: Figure 3 303 shown.

[0053] Step 102: Input the second video frame into the first video encoder to generate a bitstream.

[0054] In this embodiment, the type of the first video encoder is not limited. For example, the first video encoder can be an encoder that supports video encoding standards such as H.264 or H.265.

[0055] The following describes the content related to training and preprocessing the network.

[0056] In some embodiments, the trained preprocessing network is obtained by using multiple third video frames as inputs to the preprocessing network and training with the goal of the value of the first loss function being less than a first threshold and the value of the second loss function being less than a second threshold, or with the training rounds being a third threshold.

[0057] The first loss function is used to indicate the difference between the fourth video frame output by the preprocessing network and the corresponding third video frame input to the preprocessing network;

[0058] The second loss function is the bitrate obtained when the fourth video frame is input to the second video encoder.

[0059] In this embodiment of the application, the plurality of third video frames can be understood as training samples or sample data.

[0060] It is understood that in the embodiments of this application, the loss function used to train the preprocessing network has two dimensions: one is the bitrate, and the other is the difference between the input and output of the preprocessing network. In this way, the video frames output by the trained preprocessing network can both help the video encoder improve compression efficiency and reduce bitrate, while ensuring the video quality of the reconstructed image.

[0061] In the embodiments of this application, the second video encoder may be the same encoder as the first video encoder, or the second video encoder may be a encoder with similar functions to the first video encoder, i.e., a proxy encoder.

[0062] In some embodiments, such as Figure 4 As shown, the second video encoder is used to perform the following steps 401 to 403:

[0063] Step 401: Determine the optical flow field between the fourth video frame and the first reference frame of the fourth video frame using an optical flow network; wherein, the first reference frame is obtained by the preprocessing network preprocessing the second reference frame of the fourth video frame.

[0064] In one possible implementation, the fourth video frame and the first reference frame of the fourth video frame are input into the optical flow network, thereby using the optical flow network to obtain the optical flow field (i.e., the motion vector of each pixel) between the two frames.

[0065] Step 402: Determine the residual frame of the fourth video frame based on the optical flow field and the first reference frame.

[0066] In some embodiments, step 402 may further include: performing a backward warping operation on the first reference frame using the optical flow field to obtain a first predicted frame; removing artifacts from the first predicted frame to obtain a second predicted frame; and determining the residual frame of the fourth video frame based on the second predicted frame and the fourth video frame.

[0067] In one possible implementation, a trained convolutional neural network (CNN) can be used to remove artifacts from the first predicted frame.

[0068] Step 403: Determine the bitrate of the fourth video frame based on the residual frame.

[0069] In some embodiments, step 403 may further include: performing a discrete cosine transform (DCT) on the residual frame to obtain first transform coefficients; quantizing the first transform coefficients to obtain second transform coefficients; inputting the second transform coefficients into an entropy model to obtain the probability distribution of the residual frame; and determining the bitrate of the fourth video frame based on the probability distribution of the residual frame.

[0070] It is understood that in this embodiment, the value of the second loss function (i.e., the bitrate) is determined by the second video encoder based on the optical flow network (as described in steps 401 to 403 above). In the second video encoder, since inter-frame prediction uses the optical flow field (i.e., the motion vector of each pixel) provided by the optical flow network, it is beneficial to improve the accuracy of predicted frames and reduce the redundancy information of residual frames, thereby helping to reduce the bitrate. The preprocessing network trained based on this is even more beneficial to the video encoder in improving compression efficiency and reducing bitrate, that is, enhancing the bitrate reduction effect / ability of the trained preprocessing network.

[0071] This application provides a method for training a preprocessing network. Figure 5 This is a schematic diagram illustrating the implementation flow of the training method for the preprocessing network provided in the embodiments of this application; as follows: Figure 5 As shown, the method includes the following steps 501 to 504:

[0072] Step 501: The third video frame is preprocessed using the preprocessing network to obtain the fourth video frame.

[0073] In this embodiment, further implementations of step 501 can be found in step 101 and its further implementations. That is, the preprocessing method of the preprocessing network for the third video frame is the same as the preprocessing method of the preprocessing network for the first video frame; therefore, it will not be described again here.

[0074] Step 502: Determine the value of the first loss function based on the third video frame and the fourth video frame.

[0075] That is, the value of the first loss function is used to indicate the difference between the output and input of the preprocessing network, namely the difference between the fourth video frame output by the preprocessing network and the corresponding third video frame input to the preprocessing network.

[0076] Step 503: Input the fourth video frame into the second video encoder to obtain the bitrate of the fourth video frame, and use the bitrate of the fourth video frame as the value of the second loss function.

[0077] The method for determining the bitrate of the fourth video frame has already been described above, so it will not be repeated here. For details on the method for determining the bitrate of the fourth video frame, please refer to steps 401 to 403 and further implementations of steps 401 to 403.

[0078] Step 504: If the value of the first loss function is greater than or equal to the first threshold, and / or the value of the second loss function is greater than or equal to the second threshold, adjust the model parameters of the preprocessing network, and iteratively execute steps 501 to 504 until the value of the first loss function is less than the first threshold and the value of the second loss function is less than the second threshold, thereby obtaining the trained preprocessing network.

[0079] In some other embodiments, step 504 can also be replaced by: adjusting the model parameters of the preprocessing network when the current training round is less than the third threshold, and iteratively executing steps 501 to 504 until the current training round is the third threshold, thereby obtaining the trained preprocessing network.

[0080] In some embodiments, adjusting the model parameters of the preprocessing network may further include: adjusting the model parameters of the preprocessing network based on the values ​​of the first loss function and the second loss function.

[0081] It is understood that in the embodiments of this application, the loss function used to train the preprocessing network has two dimensions: one is the bitrate, and the other is the difference between the input and output of the preprocessing network. In this way, the video frames output by the trained preprocessing network can both help the video encoder improve compression efficiency and reduce bitrate, while ensuring the video quality of the reconstructed image.

[0082] In the embodiments of this application, any video frame and reference frame mentioned above can be a video frame and reference frame of any component. For example, any video frame and reference frame mentioned above can be an image frame of the Y component, or any video frame and reference frame mentioned above can be an image frame of the R, G or B components, etc.

[0083] The following examples illustrate possible implementation schemes of the methods described in one or more of the above embodiments.

[0084] In this application embodiment, a bitrate-aware optimized intelligent preprocessing algorithm for video coding is provided, namely a training method for a preprocessing network or a video coding method.

[0085] (1) In this embodiment of the application, the global features obtained by the spatial attention mechanism and the local features obtained by the local information processing module are fused together so that the final image features can combine the advantages of both.

[0086] (2) In this embodiment of the application, by using an optical flow network to provide the motion vector of each pixel, the calculated motion field is continuous and there will be no distortion problem at the block boundary, which helps to improve the visual quality of the predicted frame.

[0087] In this embodiment, a proxy encoder (an example of a second video encoder, including inter-frame / intra-frame prediction, optical flow network motion estimation calculation, spatial frequency transformation and quantization, and entropy model, etc.) is constructed. This proxy encoder can simulate rate distortion behavior approximating that of a standard video encoder. A rate distortion loss formula is obtained in the preprocessing network, and perceptual and coding-oriented rate distortion optimization is performed to achieve end-to-end training. During inference / deployment, the proxy encoder is replaced with a standard encoder (an example of a first video encoder), ultimately achieving content-aware rate distortion preprocessing.

[0088] To achieve the above objectives, the specific implementation process includes the following steps (1) to (6):

[0089] (1) Preprocessing network

[0090] The role of this preprocessing network is to enhance the image quality of the video frame to be encoded (i.e., an example of a first or third video frame, such as a Y-component video frame) before inputting it to the video encoder. In some embodiments, the preprocessing network uses a convolutional neural network to perform multi-layer convolution operations on the Y-component video frame to extract low-level features and high-level semantic features, and then performs feature aggregation operations to provide high-quality feature representations for subsequent video processing.

[0091] In some embodiments, video frame 1 of the Y component is input into a preprocessing network to obtain video frame 2 of the Y component output by the preprocessing network; reference frame 1 of video frame 1 of the Y component is input into the preprocessing network to obtain reference frame 2 of the Y component output by the preprocessing network.

[0092] It should be noted that video frame 1 of the Y component can be understood as an example of the first or third video frame mentioned above, and video frame 2 of the Y component can be understood as an example of the second or fourth video frame mentioned above. Reference frame 1 of video frame 1 of the Y component can be understood as an example of the second reference frame mentioned above, and reference frame 2 of the Y component can be understood as an example of the first reference frame mentioned above.

[0093] (2) Motion estimation module

[0094] (2a) Input the video frame 2 of the Y component and the reference frame 2 of the Y component into the optical flow network.

[0095] (2b) Use an optical flow network to obtain the optical flow field (i.e. motion vector) between video frame 2 of the Y component and reference frame 2 of the Y component.

[0096] (3) Motion compensation module

[0097] (3a) The reference frame 2 of the Y component is back-distorted using the optical flow field to generate prediction frame 1 (i.e., an example of the first prediction frame mentioned above).

[0098] (3b) Use a CNN network to remove artifacts from the predicted frame to obtain predicted frame 2 (i.e., an example of the second predicted frame mentioned above).

[0099] As can be understood, the backward warp operation is achieved by applying the optical flow field in reverse, that is, mapping the pixels in the current frame to their corresponding positions in the reference frame according to the motion vectors in the optical flow field. Since the optical flow field describes the motion information of the pixels, the backward warp operation can be used to generate the prediction frame.

[0100] (4) Discrete Cosine Transform (DCT) and Quantization Module

[0101] (4a) The residual frame can be obtained by subtracting the predicted frame 2 obtained by the motion compensation module from the video frame 2 of the Y component. It can be understood that transmitting only the residual frame at the encoding end can significantly reduce the redundant information of the video.

[0102] (4b) Perform Discrete Cosine Transform (DCT) on the residual frame to transform it from the spatial domain to the frequency domain and obtain the discrete cosine transform coefficients.

[0103] (4c) Quantize the discrete cosine transform coefficients. In this way, by reducing and approximating the precision of the frequency domain coefficients, the amount of data representation can be further reduced, thereby achieving compression.

[0104] (5) Entropy Model

[0105] The quantized discrete cosine transform coefficients are input into the entropy model to obtain the probability distribution of the residual frame. Based on the probability distribution of the residual frame, the entropy of video frame 2 of the Y component is calculated, and the bitrate of video frame 2 of the Y component is determined according to the entropy of video frame 2 of the Y component.

[0106] (6) Model training for the preprocessed network

[0107] The loss functions used to train the preprocessing network include the loss function for video frame reconstruction (i.e., the first loss function) and the loss function for video frame bitrate (i.e., the second loss function).

[0108] The loss function for video frame reconstruction can be determined based on video frame 1 of the Y component input to the preprocessing network and video frame 2 of the Y component obtained after the preprocessing network preprocesses video frame 1 of the Y component.

[0109] The loss function for the video frame bitrate can be the bitrate of video frame 2 in the Y component. The calculation method for this bitrate is described in (2)-(5) above. That is, after preprocessing video frame 1 in the Y component and reference frame 1 in the Y component through the preprocessing network, the obtained video frame 2 in the Y component (i.e., the current frame) and the obtained reference frame 2 in the Y component are subjected to motion estimation and motion compensation to obtain the residual frame. The residual frame is subjected to discrete cosine transform and quantization to obtain the bitrate through the entropy model, which is used to calculate the bitrate loss.

[0110] Figure 6 This is a schematic diagram of the training process of the preprocessing network provided in the embodiments of this application; as shown below. Figure 6 As shown, the steps include 601 to 609 as follows:

[0111] Step 601: Reference frame 1 and current frame 1 are input into the preprocessing network to obtain reference frame 2 and current frame 2 respectively;

[0112] Among them, reference frame 1 can be reference frame 1 of the Y component, current frame 1 can be video frame 1 of the Y component, reference frame 2 can be reference frame 2 of the Y component, and current frame 2 can be video frame 2 of the Y component.

[0113] Step 602: Input both the current frame 2 and the reference frame 2 into the motion estimation module to obtain the optical flow field (i.e., motion vector) between the current frame 2 and the reference frame 2;

[0114] Step 603: Input the optical flow field and reference frame 2 into the motion compensation module to obtain the predicted frame 2 of the current frame 2;

[0115] Step 604: Based on the current frame 2 and the predicted frame 2, the residual frame can be determined;

[0116] Step 605: Input the residual frame into the DCT transform and quantization module to obtain the DCT transform coefficients;

[0117] Step 606: Input the DCT transform coefficients into the entropy model to obtain the probability distribution of the residual frame;

[0118] Step 607: Determine the bitrate of the residual frame based on the probability distribution of the residual frame, and use the bitrate as the value of the second loss function;

[0119] Step 608: Determine the value of the first loss function based on current frame 1 and current frame 2;

[0120] Step 609: Determine whether the value of the first loss function is less than the first threshold and whether the value of the second loss function is less than the second threshold; if so, end the training; otherwise, adjust the model parameters of the preprocessing network and return to step 601. At this time, the image input to the preprocessing network in step 601 is the new current frame 1 and the reference frame 1 of the new current frame 1.

[0121] Detailed implementation process:

[0122] The specific flow of the algorithm described in the embodiments of this application is described below.

[0123] 1. Preprocess the input video frames using a preprocessing network:

[0124] For a given video frame (i.e., an example of the first or third video frame), the preprocessing network completes the processing through a single forward pass. That is, after the video frame (i.e., an example of the first or third video frame) enters the preprocessing network, its convolutional layers extract low-level features, such as edge and texture information. Subsequently, the feature map output from the previous processing layer passes through multiple deep convolutional layers and residual blocks (i.e., residual networks) of the preprocessing network to further extract higher-level semantic features. These higher-level semantic features are integrated through the feature aggregation layer of the preprocessing network, and the spatial attention mechanism of the preprocessing network is used to enhance the representation of important features in the integrated feature map. Finally, the feature map output by the spatial attention mechanism is upsampled by the upsampling layer in the preprocessing network before being output; thus, the reconstruction / preprocessing of the input video frame is achieved.

[0125] 2. Calculate the entropy of the current frame.

[0126] a) Image pyramid construction and feature extraction

[0127] A multi-scale image pyramid representation is constructed for each of the two input image frames. This allows for the simultaneous processing of motion information at different scales. For each scale in the pyramid, a convolutional neural network is used to extract image features. These features include motion-related information such as texture and edges.

[0128] b) Optical flow motion estimation

[0129] An image pair processing approach is employed, progressively refining the optical flow motion estimation by estimating small optical flow updates at each pyramid level. At each pyramid level, the network receives the upsampled optical flow from the previous level and the current level's image pair as input to compute the optical flow update for that level. The final output is an optical flow field that has undergone multi-scale feature extraction and iterative updates.

[0130] c) Residual frame calculation

[0131] The optical flow field obtained from the current frame and the reference frame is used to generate a prediction frame through a backward warping operation. The prediction frame obtained through this warping operation contains artifacts, which can be removed using a CNN network through learning.

[0132] The residual frame can be obtained by subtracting the predicted frame obtained by the motion compensation module from the current frame. Transmitting only the residual frame can significantly reduce the redundancy of video information.

[0133] d) Discrete Cosine Transform and Quantization

[0134] The residual frame contains the pixel differences between the current frame and the predicted frame, typically exhibiting significant numerical variations. These differences manifest as high-frequency components in the frequency domain. A Discrete Cosine Transform (DCT) is applied to the residual frame, transforming it from the spatial domain to the frequency domain. The DCT allows image energy to be more concentrated in a smaller number of low-frequency coefficients, while the high-frequency coefficients represent image details and high-frequency variations. In this way, the DCT removes redundant information and spatial correlations from the image, enabling better quantization and encoding of high-frequency coefficients.

[0135] After discrete cosine transform, quantizing the frequency domain coefficients is a crucial step in video compression. By reducing and approximating the frequency domain coefficients with higher precision, the amount of data in video frames can be significantly reduced.

[0136] e) Entropy Model

[0137] Entropy coding leverages the statistical properties of data to achieve efficient data representation. By assigning codewords based on the probability distribution of symbols appearing in the data, common symbols are represented with shorter codewords, while less common symbols are represented with longer codewords. Entropy coding reduces the overall code length by representing high-frequency symbols with shorter codewords.

[0138] In one possible implementation, the probability distribution required for entropy encoding is provided by an entropy model. Indicates video data information, The true distribution is denoted as Entropy model An estimate is made of the probability distribution, denoted as . The former is The unknown distribution is denoted by , and the unknown distribution is an estimate of . Minimizing the cross-entropy between the two allows us to... Approaching When the two are equal, the cross-entropy reaches its minimum, at which point the theoretical minimum average coding length is reached.

[0139] probability distribution function From probability density function Points are earned as follows:

[0140]

[0141] Modeling the probability density function using a cumulative function:

[0142]

[0143] p = f K ′·f K ′ -1 …f1′

[0144] Among them, f K Let ′ be a three-dimensional vector function, where K = 4. The probability density function is modeled using four three-dimensional functions, and the parameters are optimized through training, with cross-entropy used as the loss function. This yields the entropy model.

[0145] f k (x)=g k (H (k) x+b (k) )

[0146] f K (x)=sigmoid(H (K) x+b (K) )

[0147] g k (x)=x+a (k) ⊙tanh (x)

[0148] The probability distribution of video information can be obtained using the entropy model above. Its minimum cross-entropy is close to the lower bound of the data entropy, which is the theoretical limit of the minimum average code length. The final cross-entropy is as follows:

[0149]

[0150] 3. Loss Function Design

[0151] The loss function of the preprocessing network consists of two parts: video frame reconstruction loss and... (i.e., the first loss function), the cross-entropy loss R is used as the bitrate loss. (i.e., the second loss function). The video frame reconstruction loss is composed of the L1 loss. and MS-SSIM loss constitute:

[0152]

[0153]

[0154] The final loss function is:

[0155]

[0156] Where λ and γ are hyperparameters used to adjust the weights of each loss.

[0157] 4. Model Training

[0158] Repeat steps 1-3 above for training, with 20 training epochs and an initial learning rate of 1e-5. During training, the bitrate-aware optimized preprocessing model performs inter-frame prediction optimization for the video frames.

[0159] It is understood that the above algorithm aims to overcome the problems of insufficient compression ratio and low image quality in video coding preprocessing methods in related technologies. Specifically, the algorithm employs an optical flow network, which improves the accuracy of residual frame calculation and reduces the bitrate of residual frames by calculating the motion vectors between pixels. Furthermore, the embodiments of this application also improve the preprocessing network, designing a local feature extraction layer and a spatial attention mechanism, ultimately obtaining image features that combine global and local features, thus improving the quality of feature extraction.

[0160] To verify the actual effectiveness of the above algorithm, the inventors of this application conducted experiments. Figure 7 The illustration shows a comparative experimental result of H.265 and H.264 using the proposed solution versus H.265 and H.264 without the preprocessing method in the same video, provided as an embodiment of this application. Figure 7 As shown, for the same video, the reconstructed frames obtained by H.265 and H.264 in this scheme have higher video quality scores.

[0161] It is understood that, in this embodiment of the application, considering the importance of local image features in the preprocessing method, a local feature extraction module is designed. Multiple residual networks are used as building blocks, and an additional layer (i.e., spatial attention mechanism) for learning local features is added to each building block. Finally, image features that combine global and local features are obtained, which improves the quality of image features extracted by the processing module. This is beneficial for the preprocessing network to output images of better quality, and thus beneficial for the decoding end to obtain higher quality reconstructed images.

[0162] In this embodiment, considering the low accuracy of motion estimation caused by block matching in related algorithms, an optical flow network is used for motion estimation. This network provides a motion vector for each pixel, which is more accurate than block matching with a fixed block size. It can capture complex non-rigid motions such as rotation and scaling, and avoids distortion at block boundaries. This improves the quality of predicted frames, reduces the bitrate of residual frames, and thus enhances the effect of indirectly reducing the bitrate through the preprocessing network.

[0163] It should be noted that although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps; or steps from different embodiments may be combined into a new technical solution.

[0164] Based on the foregoing embodiments, this application provides a video encoding device and a training device for a preprocessing network. These devices include various modules and units included in each module, which can be implemented by a processor; of course, they can also be implemented by specific logic circuits. In the implementation process, the processor can be an AI acceleration engine (such as an NPU), a GPU, a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA), etc.

[0165] Figure 8 This is a schematic diagram of the structure of the video encoding device provided in the embodiments of this application; as shown Figure 8 As shown, the video encoding device 800 includes:

[0166] The first preprocessing module 801 is configured to preprocess the first video frame using a trained preprocessing network to obtain the second video frame; wherein the preprocessing includes: extracting local features and global features of the first video frame, fusing the local features and the global features, and obtaining the second video frame based on the fusion result;

[0167] The encoding module 802 is configured to encode the second video frame to generate a bitstream.

[0168] In some embodiments, the trained preprocessing network is obtained by using multiple third video frames as inputs to the preprocessing network and training with the goal of the value of a first loss function being less than a first threshold and the value of a second loss function being less than a second threshold, or with the training epochs being a third threshold. The first loss function is used to indicate the difference between the fourth video frame output by the preprocessing network and the corresponding third video frame input to the preprocessing network. The second loss function is the bitrate obtained by inputting the fourth video frame into the second video encoder.

[0169] In some embodiments, the second video encoder is configured to: determine the optical flow field between the fourth video frame and a first reference frame of the fourth video frame using an optical flow network; wherein the first reference frame is obtained by the preprocessing network preprocessing a second reference frame of the fourth video frame; determine the residual frame of the fourth video frame based on the optical flow field and the first reference frame; and determine the bitrate of the fourth video frame based on the residual frame.

[0170] In some embodiments, the trained preprocessing network includes convolutional layers and multiple deep convolutional layers; the extraction of local and global features of the first video frame includes: using the convolutional layers to extract local features of the first video frame to obtain multiple first feature maps, the first feature maps including the local features; and extracting global features of the first video frame based on the multiple deep convolutional layers and the multiple first feature maps to obtain multiple second feature maps, the second feature maps including the global features.

[0171] Furthermore, in some embodiments, the trained preprocessing network further includes a residual network and a feature aggregation layer; the fusion of the local features and the global features includes: inputting multiple second feature maps corresponding to the first video frame and multiple second feature maps corresponding to the previous frame of the first video frame into the residual network to obtain multiple third feature maps; fusing the multiple third feature maps using the feature aggregation layer to obtain a fourth feature map; wherein the fourth feature map is the fusion result.

[0172] Furthermore, in some embodiments, the trained preprocessing network further includes a spatial attention mechanism and an upsampling layer; obtaining the second video frame based on the fusion result includes: using the spatial attention mechanism to enhance the detailed features in the fourth feature map to obtain a fifth feature map; and using the upsampling layer to perform an upsampling operation on the fifth feature map to obtain the second video frame.

[0173] Figure 9 A schematic diagram of the structure of the training device for the preprocessing network provided in the embodiments of this application; as shown Figure 9 As shown, the training device 900 for the preprocessing network includes:

[0174] The second preprocessing module 901 is configured to preprocess the third video frame using the preprocessing network to obtain the fourth video frame.

[0175] The determining module 902 is configured to determine the value of the first loss function based on the third video frame and the fourth video frame;

[0176] The input module 903 is configured to input the fourth video frame to the second video encoder to obtain the bitrate of the fourth video frame, and use the bitrate of the fourth video frame as the value of the second loss function;

[0177] The adjustment module 904 is configured to adjust the model parameters of the preprocessing network when the value of the first loss function is greater than or equal to the first threshold, and / or the value of the second loss function is greater than or equal to the second threshold, or when the current training epoch is less than the third threshold, and iteratively execute the functions of the second preprocessing module 901, the determination module 902, and the input module 903 until the value of the first loss function is less than the first threshold and the value of the second loss function is less than the second threshold, or until the current training epoch is the third threshold, thereby obtaining a trained preprocessing network.

[0178] In some embodiments, the second video encoder is configured to: determine the optical flow field between the fourth video frame and a first reference frame of the fourth video frame using an optical flow network; wherein the first reference frame is obtained by the preprocessing network preprocessing a second reference frame of the fourth video frame; determine the residual frame of the fourth video frame based on the optical flow field and the first reference frame; and determine the bitrate of the fourth video frame based on the residual frame.

[0179] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0180] It should be noted that the module division in the embodiments of this application is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, exist as separate physical units, or have two or more units integrated into one unit. The integrated units can be implemented in hardware, as software functional units, or a combination of software and hardware.

[0181] It should be noted that, in the embodiments of this application, if the above-described methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0182] This application provides an electronic device. Figure 10 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application; as shown below. Figure 10 As shown, the electronic device 100 includes a processor 1001 and a first video encoder 1002; wherein:

[0183] The processor 1001 is configured to preprocess a first video frame using a trained preprocessing network to obtain a second video frame; wherein the preprocessing includes: extracting local features and global features of the first video frame, fusing the local features and the global features, and obtaining the second video frame based on the fusion result;

[0184] The first video encoder 1002 is configured to encode the second video frame to generate a bitstream.

[0185] This application provides an electronic device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements the steps in the method provided in the above embodiments.

[0186] It should be noted that the memory is configured to store instructions and applications that can be executed by the processor, and can also cache data to be processed or already processed by various modules in the processor and electronic devices (e.g., image data, audio data, voice communication data and video communication data), which can be implemented through flash memory or random access memory (RAM).

[0187] In this application embodiment, the type of electronic device is not limited; the electronic device can be a variety of devices with information processing capabilities. For example, the electronic device can be a smartphone, laptop, tablet, drone, etc.

[0188] This application also provides a computer-readable storage medium for storing computer programs.

[0189] Optionally, the computer-readable storage medium can be applied to the electronic device in the embodiments of this application, and the computer program causes the processor or electronic device to perform the various methods of the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0190] This application also provides a computer program product, including computer program instructions.

[0191] Optionally, the computer program product can be applied to the electronic device in the embodiments of this application, and the computer program instructions cause the processor or electronic device to execute the various methods in the embodiments of this application. For the sake of brevity, they will not be described in detail here.

[0192] This application also provides a computer program.

[0193] Optionally, the computer program can be applied to the electronic device in the embodiments of this application. When the computer program runs on the processor or electronic device, it causes the processor or electronic device to execute the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0194] It should be noted that the descriptions of the electronic devices, storage media, computer program products, and computer program embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the electronic devices, storage media, computer program products, and computer program embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0195] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0196] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three kinds of relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0197] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0198] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or modules can be electrical, mechanical, or other forms.

[0199] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0200] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0201] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0202] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0203] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0204] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0205] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0206] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video encoding method, characterized in that, The method includes: The first video frame is preprocessed using a trained preprocessing network to obtain the second video frame; wherein, the preprocessing includes: extracting local features and global features of the first video frame, fusing the local features and the global features, and obtaining the second video frame based on the fusion result; The second video frame is input into the first video encoder to generate a bitstream.

2. The method according to claim 1, characterized in that, The trained preprocessing network is obtained by using multiple third video frames as inputs to the preprocessing network and training with the goal of the first loss function value being less than a first threshold and the second loss function value being less than a second threshold, or with the training rounds being a third threshold. The first loss function is used to indicate the difference between the fourth video frame output by the preprocessing network and the corresponding third video frame input to the preprocessing network; The second loss function is the bitrate obtained when the fourth video frame is input to the second video encoder.

3. The method according to claim 2, characterized in that, The second video encoder is used for: An optical flow network is used to determine the optical flow field between the fourth video frame and a first reference frame of the fourth video frame; wherein, the first reference frame is obtained by the preprocessing network preprocessing the second reference frame of the fourth video frame; The residual frame of the fourth video frame is determined based on the optical flow field and the first reference frame; The bitrate of the fourth video frame is determined based on the residual frame.

4. The method according to any one of claims 1-3, characterized in that, The trained preprocessing network includes convolutional layers and multiple deep convolutional layers; The extraction of local and global features from the first video frame includes: The convolutional layer is used to extract local features of the first video frame to obtain multiple first feature maps, and the first feature maps include the local features; Based on the multi-layer deep convolutional layers and the plurality of first feature maps, global features of the first video frame are extracted to obtain a plurality of second feature maps, wherein the second feature maps include the global features.

5. The method according to claim 4, characterized in that, The trained preprocessing network also includes a residual network and a feature aggregation layer; The fusion of the local features and the global features includes: Multiple second feature maps corresponding to the first video frame and multiple second feature maps corresponding to the previous frame of the first video frame are input into the residual network to obtain multiple third feature maps. The multiple third feature maps are fused using the feature aggregation layer to obtain a fourth feature map; wherein the fourth feature map is the fusion result.

6. The method according to claim 5, characterized in that, The trained preprocessing network also includes a spatial attention mechanism and an upsampling layer; Obtaining the second video frame based on the fusion result includes: The spatial attention mechanism is used to enhance the detailed features in the fourth feature map to obtain the fifth feature map. The second video frame is obtained by upsampling the fifth feature map using the upsampling layer.

7. A method for training a preprocessing network, characterized in that, The method includes: The third video frame is preprocessed using the preprocessing network to obtain the fourth video frame; The value of the first loss function is determined based on the third video frame and the fourth video frame; The fourth video frame is input to the second video encoder to obtain the bitrate of the fourth video frame, and the bitrate of the fourth video frame is used as the value of the second loss function; If the value of the first loss function is greater than or equal to the first threshold and / or the value of the second loss function is greater than or equal to the second threshold, or if the current training epoch is less than the third threshold, the model parameters of the preprocessing network are adjusted, and the above steps are iteratively executed until the value of the first loss function is less than the first threshold and the value of the second loss function is less than the second threshold, or until the current training epoch is the third threshold, and a trained preprocessing network is obtained.

8. The method according to claim 7, characterized in that, The second video encoder is used for: An optical flow network is used to determine the optical flow field between the fourth video frame and a first reference frame of the fourth video frame; wherein, the first reference frame is obtained by the preprocessing network preprocessing the second reference frame of the fourth video frame; The residual frame of the fourth video frame is determined based on the optical flow field and the first reference frame; The bitrate of the fourth video frame is determined based on the residual frame.

9. A video encoding device, characterized in that, The device includes a first preprocessing module and an encoding module; wherein... The first preprocessing module is configured to preprocess the first video frame using a trained preprocessing network to obtain the second video frame; wherein the preprocessing includes: extracting local features and global features of the first video frame, fusing the local features and the global features, and obtaining the second video frame based on the fusion result; The encoding module is configured to encode the second video frame to generate a bitstream.

10. An electronic device, characterized in that, The electronic device includes a processor and a first video encoder; wherein... The processor is configured to preprocess a first video frame using a trained preprocessing network to obtain a second video frame; wherein the preprocessing includes: extracting local features and global features of the first video frame, fusing the local features and the global features, and obtaining the second video frame based on the fusion result; The first video encoder is configured to encode the second video frame to generate a bitstream.

Images