Training method, device and computer equipment for video frame optimization model

Abstract

The present application relates to a training method, device, computer equipment and storage medium for a video frame optimization model. The method comprises: for each video frame in the first sample video frame stream in the training data, the features of the previous video frame of the video frame and the video frame are input into the feature extraction module in the video frame optimization model to be trained, and the target weighted features corresponding to the video frame are obtained; according to the video frame and the target weighted features corresponding to the video frame, the predicted video frame of the second resolution is determined; according to the second sample video frame stream and each predicted video frame stream, the training loss function is calculated; according to the training loss function, the network parameters of the video frame optimization model to be trained are updated, and the step of obtaining training data is returned until the training loss function meets the preset training completion condition, and the trained video frame optimization model is obtained. The present invention can speed up the training speed, reduce the computing power consumption of training, and improve the accuracy of model training.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a training method, apparatus and computer equipment for a video frame optimization model. Background Art

[0002] Super-resolution reconstruction technology is often used when the image acquisition equipment or environment is limited, resulting in poor image and video quality and low resolution. In the field of computer vision, super-resolution reconstruction technology is mainly used on images. In related technologies, the resolution of low-resolution images or videos is improved through reconstruction models. Specifically, the training process of the reconstruction model can be to perform pre-processing such as denoising on the input low-resolution video frames, then extract features from the low-resolution video frames, and then perform nonlinear mapping on the features to obtain a high-resolution feature map sequence, and finally restore the high-resolution feature map to a high-resolution video / frame.

[0003] However, the above training process involves multiple network structures, loss functions, model training and verification processes in various deep learning technologies. Therefore, the amount of computation required in the training process of the reconstruction model in the related technology is large. For the conference software terminal, the computing power requirements and configuration requirements are too high, making it difficult to achieve. Summary of the invention

[0004] Based on this, it is necessary to provide a training method, device and computer equipment for a video frame optimization model that can reduce the computing power consumption in the model training process and improve the accuracy of model training in order to address the above technical problems.

[0005] A method for training a video frame optimization model, the method comprising:

[0006] Acquire training data, where the training data includes a plurality of first sample video frame streams with a first resolution and a plurality of second sample video frame streams with a second resolution, where the first resolution is smaller than the second resolution;

[0007] For each video frame in the first sample video frame stream, inputting the features of the previous video frame of the video frame and the video frame into a feature extraction module in the video frame optimization model to be trained to obtain a target weighted feature corresponding to the video frame;

[0008] Determining a predicted video frame of a second resolution according to the video frame and a target weighted feature corresponding to the video frame;

[0009] Calculating a training loss function according to the second sample video frame stream and each of the predicted video frame streams;

[0010] The network parameters of the video frame optimization model to be trained are updated according to the training loss function, and the step of obtaining the training data is returned to be executed until the training loss function meets the preset training completion condition, thereby obtaining the trained video frame optimization model.

[0011] In one of the embodiments, determining the predicted video frame of the second resolution according to the video frame and the target weighted feature corresponding to the video frame includes:

[0012] Performing convolution calculation on the video frame to obtain an initial predicted video frame of a second resolution;

[0013] Performing convolution calculation on the target weighted features corresponding to the video frame to obtain the residual of the video frame;

[0014] The residual is superimposed on the initial predicted video frame of the second resolution to obtain a predicted video frame of the second resolution.

[0015] In one of the embodiments, the feature extraction module includes n feature extraction submodules;

[0016] For each video frame in the first sample video frame stream, the feature of the previous video frame of the video frame and the video frame are input into a feature extraction module in the video frame optimization model to be trained to obtain a target weighted feature corresponding to the video frame, including:

[0017] For each video frame in the first sample video frame stream, a first feature map of a previous video frame of the video frame and the video frame are input into a first feature extraction submodule to obtain a first initial weighted feature corresponding to the video frame, wherein the first feature map of the previous video frame of the video frame is obtained after the previous video frame is input into the first feature extraction submodule corresponding to the previous video frame;

[0018] Inputting the i-1th initial weighted feature of the video frame and the i-th feature map of the previous video frame of the video frame into the i-th feature extraction submodule to obtain the i-th initial weighted feature corresponding to the video frame, the i-th feature map of the previous video frame of the video frame is obtained after inputting the previous video frame into the i-th feature extraction submodule corresponding to the previous video frame, and the initial value of i is 2;

[0019] Add 1 to the value of i, and perform the step of inputting the i-1th initial weighted feature of the video frame and the i-th feature map of the previous video frame of the video frame into the i-th feature extraction submodule until the n-th initial weighted feature corresponding to the video frame is obtained;

[0020] The nth initial weighted feature corresponding to the video frame is used as the target weighted feature of the video frame.

[0021] In one of the embodiments, the feature extraction submodule includes a feature extraction unit and an attention mechanism unit;

[0022] The step of inputting the i-1th initial weighted feature of the video frame and the i-th feature map of the previous video frame of the video frame into the i-th feature extraction submodule to obtain the i-th initial weighted feature corresponding to the video frame, including:

[0023] Inputting the i-1th initial weighted feature of the video frame into the i-th feature extraction unit to obtain the i-th feature map corresponding to the video frame;

[0024] Inputting the i-th feature map corresponding to the video frame and the i-th feature map of the previous video frame into the attention mechanism unit to obtain the i-th initial feature weight of the video frame;

[0025] The i-th initial feature weight of the video frame and the i-th feature map corresponding to the video frame are weightedly calculated to obtain the i-th initial weighted feature corresponding to the video frame.

[0026] In one embodiment, the step of inputting the i-th feature map corresponding to the video frame and the i-th feature map of the previous video frame into the attention mechanism unit to obtain the i-th initial feature weight of the video frame includes:

[0027] Adding the i-th feature map corresponding to the video frame to the i-th feature map of the previous video frame to obtain the i-th target feature map of the video frame;

[0028] Performing a global average pooling operation on the i-th target feature map to obtain an i-th target feature vector of the video frame;

[0029] Performing a dimensionality reduction operation on the i-th target feature vector of the video frame to obtain the i-th target reduced-dimensionality feature vector of the video frame;

[0030] Performing a dimensionality-raising convolution operation on the i-th target dimensionality-reduced feature vector of the video frame to obtain the i-th target dimensionality-raising feature vector of the video frame;

[0031] Determine, according to a preset weight determination function, an i-th weight vector corresponding to the i-th target up-dimensional feature vector of the video frame;

[0032] The product of the i-th weight vector and the i-th target upscaled feature vector is used as the i-th initial feature weight of the video frame.

[0033] In one embodiment, the method further comprises:

[0034] The first resolution video frame stream to be reconstructed is input into the trained video frame optimization model to obtain a second resolution video frame stream corresponding to the first resolution video frame stream to be reconstructed.

[0035] In one of the embodiments, before the step of obtaining training data, the method further includes: obtaining a second resolution sample video frame stream;

[0036] The second-resolution sample video frame stream is reconstructed into a first-resolution sample video frame stream by using a preset video frame reconstruction algorithm.

[0037] A training device for a video frame optimization model, the device comprising:

[0038] A training data acquisition module, used to acquire training data, wherein the training data includes a plurality of first sample video frame streams with a first resolution and a plurality of second sample video frame streams with a second resolution, wherein the first resolution is smaller than the second resolution;

[0039] A target weighted feature determination module, for each video frame in the first sample video frame stream, inputs the features of the previous video frame of the video frame and the video frame into a feature extraction module in the video frame optimization model to be trained, so as to obtain a target weighted feature corresponding to the video frame;

[0040] A predicted video frame determination module, used to determine a predicted video frame of a second resolution according to the video frame and a target weighted feature corresponding to the video frame;

[0041] A training loss function determination module, used to calculate a training loss function according to the second sample video frame stream and each of the predicted video frame streams;

[0042] A training module is used to update the network parameters of the video frame optimization model to be trained according to the training loss function, and return to execute the step of obtaining training data until the training loss function meets the preset training completion condition to obtain a trained video frame optimization model.

[0043] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0044] Acquire training data, where the training data includes a plurality of first sample video frame streams with a first resolution and a plurality of second sample video frame streams with a second resolution, where the first resolution is smaller than the second resolution;

[0045] For each video frame in the first sample video frame stream, input the features of the previous video frame of the video frame and the video frame into a feature extraction module in the video frame optimization model to be trained to obtain a target weighted feature corresponding to the video frame;

[0046] Determining a predicted video frame of a second resolution according to the video frame and a target weighted feature corresponding to the video frame;

[0047] Calculating a training loss function according to the second sample video frame stream and each of the predicted video frame streams;

[0048] The network parameters of the video frame optimization model to be trained are updated according to the training loss function, and the step of obtaining the training data is returned to be executed until the training loss function satisfies the preset training completion condition, thereby obtaining the trained video frame optimization model.

[0049] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0050] Acquire training data, where the training data includes a plurality of first sample video frame streams with a first resolution and a plurality of second sample video frame streams with a second resolution, where the first resolution is smaller than the second resolution;

[0051] For each video frame in the first sample video frame stream, input the features of the previous video frame of the video frame and the video frame into a feature extraction module in the video frame optimization model to be trained to obtain a target weighted feature corresponding to the video frame;

[0052] Determining a predicted video frame of a second resolution according to the video frame and a target weighted feature corresponding to the video frame;

[0053] Calculating a training loss function according to the second sample video frame stream and each of the predicted video frame streams;

[0054] The network parameters of the video frame optimization model to be trained are updated according to the training loss function, and the step of obtaining the training data is returned to be executed until the training loss function satisfies the preset training completion condition, thereby obtaining the trained video frame optimization model.

[0055] The training method, device, computer equipment and storage medium of the video frame optimization model are as follows: for each video frame in the first sample video frame stream in the training data, the features of the previous video frame of the video frame and the video frame are input into the feature extraction module in the video frame optimization model to be trained to obtain the target weighted features corresponding to the video frame; the predicted video frame of the second resolution is determined according to the video frame and the target weighted features corresponding to the video frame; the training loss function is calculated according to the second sample video frame stream and each predicted video frame stream; the network parameters of the video frame optimization model to be trained are updated according to the training loss function, and the step of obtaining training data is returned to execute until the training loss function meets the preset training completion condition to obtain the trained video frame optimization model. The present invention performs sub-pixel convolution operations on low-resolution images and high-frequency residuals respectively and adds them as outputs, so that the feature extraction part in the trained video frame optimization model only learns the residual part between the high-resolution image and the low-resolution image. Since the residual image is relatively sparse, many pixel values ​​are very small or even zero, thereby accelerating the training speed, reducing the computing power consumption of the training, and improving the accuracy of the model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A schematic diagram of a flow chart of a method for training a video frame optimization model in one embodiment;

[0057] Figure 2 A schematic diagram of a flow chart of a step of determining a high-resolution predicted video frame in one embodiment;

[0058] Figure 3 A schematic diagram of a flow chart of a step of calculating target weighted features in one embodiment;

[0059] Figure 4 A schematic diagram of a flow chart of a step of calculating the i-th initial weighted feature in one embodiment;

[0060] Figure 5 A schematic diagram of a flow chart of a step of calculating the i-th initial weighted value in one embodiment;

[0061] Figure 6 is a schematic diagram of an attention mechanism unit in one embodiment;

[0062] Figure 7 A schematic diagram of a flow chart of a step of calculating and obtaining training data in one embodiment;

[0063] Figure 8 A schematic diagram of a training process of a video frame optimization model in another embodiment;

[0064] Fig. 9 A schematic diagram of a feature extraction unit of a video frame optimization model in one embodiment;

[0065] Fig.10 A structural block diagram of a training device for a video frame optimization model in one embodiment;

[0066] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0068] When the image acquisition device version is low or the acquisition environment is restricted, the quality of the acquired images and videos will be poor and the resolution will not be high. In this case, super-resolution reconstruction technology can be used to improve the resolution of poor quality images and videos. In recent years, with the introduction of convolutional neural networks, super-resolution has also experienced a transition from traditional algorithms to deep learning algorithms, and its accuracy and speed have also been greatly improved. In the field of computer vision, super-resolution reconstruction technology is mainly used in images, and video and real-time super-resolution reconstruction technology are relatively rare. At present, in addition to hardware design, the method of achieving real-time or video super-resolution mainly uses deep learning methods to extract features using convolutional neural networks to achieve super-resolution. Super-resolution reconstruction technology based on deep learning mainly includes the following steps: first, the input low-resolution video / video frame is pre-processed by denoising, then the low-resolution video / video frame is feature extracted, and then the feature is nonlinearly mapped to obtain a high-resolution feature map sequence, and finally the high-resolution feature map is restored to a high-resolution video / video frame. This involves various deep learning network structures, loss functions, model training and verification. The above process requires a large amount of computing power. For conference software terminals, the computing power requirements and configuration requirements are too high, making it difficult to achieve.

[0069] The training method of the video frame optimization model provided in the embodiment of the present disclosure aims to ensure the high resolution of the shared screen or the camera through the local device in the video conferencing scenario. In the related art, the clarity of real-time video conferencing needs to be achieved by improving the network bandwidth. Due to the limitations of the current technical level, the time cost and economic cost of increasing the corresponding resolution are relatively large. Therefore, the embodiment of the present disclosure is based on a deep learning algorithm and provides a super-resolution reconstruction algorithm based on the attention mechanism to extract time domain features, which can improve the resolution of the video / video frame in real time.

[0070] In one embodiment, Figure 1As shown, a training method for a video frame optimization model is provided. This embodiment uses the method applied to a training device as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a training device and a server, and is implemented through the interaction between the training device and the server. The above-mentioned training device can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server can be implemented as an independent server or a server cluster composed of multiple servers. In this embodiment, the training method for the video frame optimization model includes the following steps:

[0071] Step 101: Acquire training data, where the training data includes a plurality of first sample video frame streams at a first resolution and a plurality of second sample video frame streams at a second resolution, wherein the first resolution is smaller than the second resolution.

[0072] The training data is sample data of the training model, the first resolution may be a low resolution, the second resolution may be a high resolution, the first sample video frame stream is a low resolution sample video frame stream, and the second sample video frame stream is a high resolution sample video frame stream. For example, the low resolution may be 50×50, and the high resolution may be 100×100. In actual application scenarios, the ratio of the video resolution may be 16:9, so that the low resolution may be 640*360, and the high resolution may be 1280*720.

[0073] Specifically, the training device obtains a high-resolution sample video frame stream within a preset time period, and a low-resolution sample video frame stream corresponding to the high-resolution sample video frame stream. Each of the above sample video frame streams includes multiple sample video frames that are adjacent in the time domain. For example, the training device can obtain multiple high-resolution video streams, and downsample the multiple high-resolution sample video frame streams into low-resolution sample video frame streams according to a preset sampling algorithm and a preset sampling frequency to obtain training data. The training data includes multiple data pairs, and one data pair includes a high-resolution sample video frame stream and a low-resolution sample video frame stream obtained after downsampling the high-resolution sample video frame stream.

[0074] Step 102, for each video frame in the first sample video frame stream, input the features of the previous video frame and the video frame into a feature extraction module in the video frame optimization model to be trained to obtain the target weighted features corresponding to the video frame.

[0075] Specifically, the first sample video frame stream includes multiple video frames arranged in time sequence. For the first video frame arranged in time sequence in each first sample video frame stream, the training device inputs the first video frame into the feature extraction module in the video frame optimization model to be trained. The target feature of the first video frame is obtained. For each video frame except the first video frame in each segment of the first sample video frame stream, the training device inputs the feature of the previous video frame of the video frame, as well as the video frame, into the feature extraction module in the video frame optimization model to be trained. The output of the feature extraction model is the target weighted feature of the video frame. Among them, the target weighted feature is the feature of the video frame extracted based on the feature of the previous video frame of the video frame.

[0076] Step 103: Determine a predicted video frame stream of a second resolution according to the video frame and the target weighted features corresponding to the video frame.

[0077] Specifically, the training device performs convolution calculation on the video frame and the target weighted features corresponding to the video frame to obtain a high-resolution preset video frame corresponding to the video frame.

[0078] Step 104: Calculate a training loss function based on the second sample video frame stream and each predicted video frame stream.

[0079] Specifically, for each video frame in the first sample video frame stream, the training device can obtain a predicted video frame stream of the first sample video frame stream, that is, obtain a predicted high-resolution video frame stream output by the video frame optimization model, by executing the process from step 102 to step 103. The training device calculates a training loss function based on the second sample video frame stream and the output preset video frame stream.

[0080] For example, since the activation function in the feature extraction module can be a ReLU function, the initialization strategy of the training loss function can be MSRA initialization (also known as Kaiming initialization, He initialization) as an initialization strategy. In other words, the training device can train the network through L1 loss and ADAM optimizer, and the L1 loss can be calculated by the following formula:

[0081]

[0082] Among them, loss(x,y) is the calculated L1 loss, y i is the second sample video frame stream, f(x i ) is the corresponding predicted video frame stream.

[0083] Step 105, update the network parameters of the video frame optimization model to be trained according to the training loss function, and return to execute the step of obtaining training data until the training loss function meets the preset training completion condition, thereby obtaining the trained video frame optimization model.

[0084] The preset training completion condition may be that the loss function corresponding to the loss value has converged, or that the number of iterations of the training data has reached the target number, etc. For example, the target number may be 100 times, 300 times, etc., and the embodiment of the present invention does not specifically limit the target number.

[0085] Specifically, the training device calculates new network parameters of the video frame optimization model to be used according to the loss value, and then updates the video frame optimization model to be trained to obtain an updated video frame optimization model. In this way, the training device re-inputs the other first sample video frame streams and the corresponding second sample video frame streams in the training data into the updated video frame optimization model, and re-executes the steps of the above embodiment method until the calculated loss value meets the preset convergence condition, or the number of iterations of the training data has reached the target number, and the trained video frame optimization model is obtained.

[0086] In the training method of the above-mentioned video frame optimization model, for each video frame in the first sample video frame stream in the training data, the features of the previous video frame of the video frame and the video frame are input into the feature extraction module in the video frame optimization model to be trained to obtain the target weighted features corresponding to the video frame; the predicted video frame of the second resolution is determined according to the video frame and the target weighted features corresponding to the video frame; the training loss function is calculated according to the second sample video frame stream and each predicted video frame stream; the network parameters of the video frame optimization model to be trained are updated according to the training loss function, and the step of obtaining training data is returned to execute until the training loss function meets the preset training completion condition to obtain the trained video frame optimization model. The present invention performs sub-pixel convolution operations on low-resolution images and high-frequency residuals respectively, and adds them as outputs, so that the feature extraction part in the trained video frame optimization model only learns the residual part between the high-resolution image and the low-resolution image. Since the residual image is relatively sparse, many pixel values ​​are very small or even zero, thereby accelerating the training speed, reducing the computing power consumption of the training, and improving the accuracy of the model training.

[0087] In one embodiment, Figure 2 As shown, the specific execution process of step 103 "determining a predicted video frame of a second resolution according to the video frame and the target weighted features corresponding to the video frame" includes:

[0088] Step 201: Perform convolution calculation on the video frame to obtain an initial predicted video frame of a second resolution.

[0089] Specifically, the convolution calculation includes a first convolution calculation and a second sub-pixel convolution calculation. For each video frame in the first sample video frame stream, the training device performs a first convolution calculation and a second sub-pixel convolution calculation on the video frame to obtain an initial predicted video frame of the second resolution. In other words, the training device directly performs a first convolution calculation and a second sub-pixel convolution calculation on the video frame to obtain an initial predicted video frame of the second resolution with the resolution directly enlarged.

[0090] Step 202, performing convolution calculation on the target weighted features corresponding to the video frame to obtain the residual of the video frame.

[0091] Specifically, through the method of the above embodiment, the target weighted feature corresponding to the video frame is obtained. In fact, the target weighted feature is the calculated initial residual. In this way, the training device can perform a first convolution calculation and a second sub-pixel convolution calculation on the initial residual, that is, amplify the initial residual to obtain the amplified residual, which is the residual of the video frame.

[0092] Step 203: superimpose the residual with the initial predicted video frame of the second resolution to obtain a predicted video frame of the second resolution.

[0093] Specifically, the training device superimposes the residual of the video frame with the initial predicted video frame of the video frame to obtain a target predicted video frame corresponding to the video production, that is, a predicted video frame of the second resolution.

[0094] In this embodiment, one line first passes through the feature extraction module to obtain the feature map of the high-frequency residual, and then passes through the sub-pixel convolution to obtain the amplified residual, and the other line directly enlarges the low-resolution video frame to a high-resolution video frame through sub-pixel convolution. Finally, the outputs of these two lines are added together to obtain the amplified high-resolution video frame. The low-frequency information and high-frequency redundancy can be integrated to obtain a more accurate high-resolution predicted video frame.

[0095] In one embodiment, in order to further improve the accuracy of the feature extraction module in extracting features of each video frame, the feature extraction module may include n feature extraction submodules. Figure 3 As shown, the specific execution process of step 102 "for each video frame in the first sample video frame stream, inputting the features of the previous video frame of the video frame and the video frame into the feature extraction module in the video frame optimization model to be trained to obtain the target weighted features corresponding to the video frame" includes:

[0096] Step 301, for each video frame in the first sample video frame stream, the first feature map of the previous video frame of the video frame and the video frame are input into the first feature extraction submodule to obtain the first initial weighted feature corresponding to the video frame. The first feature map of the previous video frame of the video frame is obtained after the previous video frame is input into the first feature extraction submodule corresponding to the previous video frame.

[0097] Specifically, the first sample video frame stream includes multiple video frames arranged in time sequence. For each video frame except the first video frame arranged in time sequence in each first sample video frame stream, the training device inputs the adjacent previous video frame of the video frame and the video frame into the first feature extraction submodule of n feature extraction submodules included in the feature extraction module. The output of the first feature extraction submodule is the first initial weighted feature of the video frame.

[0098] Step 302, inputting the i-1th initial weighted feature of the video frame and the i-th feature map of the previous video frame of the video frame into the i-th feature extraction submodule to obtain the i-th initial weighted feature corresponding to the video frame, the i-th feature map of the previous video frame of the video frame is obtained after the previous video frame is input into the i-th feature extraction submodule corresponding to the previous video frame, and the initial value of i is 2;

[0099] Step 303, add 1 to the value of i, and execute the step of inputting the i-1th initial weighted feature of the video frame and the i-th feature map of the previous video frame of the video frame into the i-th feature extraction submodule until the n-th initial weighted feature corresponding to the video frame is obtained.

[0100] Step 304: Use the nth initial weighted feature corresponding to the video frame as the target weighted feature of the video frame.

[0101] In one embodiment, the feature extraction submodule includes a feature extraction unit and an attention mechanism unit; accordingly, Figure 4 As shown, the specific execution process of step 302 "inputting the i-1th initial weighted feature of the video frame and the i-th feature map of the previous video frame of the video frame into the i-th feature extraction submodule to obtain the i-th initial weighted feature corresponding to the video frame" includes:

[0102] Step 401: input the i-1th initial weighted feature of the video frame to the i-th feature extraction unit to obtain the i-th feature map corresponding to the video frame.

[0103] Specifically, the feature extraction unit can be a block unit. For the first feature extraction submodule, the training device inputs the video frame into the feature extraction unit, and the output of the feature extraction unit is the feature map of the video frame. For the i-th feature extraction submodule in the feature extraction module, the input is the output of the i-1-th feature extraction submodule, that is, the i-1-th initial weighted feature, and the output is the i-th feature map corresponding to the video frame, 2≤i≤n.

[0104] Step 402, input the i-th feature map corresponding to the video frame and the i-th feature map of the previous video frame into the attention mechanism unit to obtain the i-th initial feature weight of the video frame.

[0105] Specifically, the training device compares the i-th feature map corresponding to the video frame with the i-th feature map of the previous video frame. Figure 1 And input it into the attention mechanism unit to obtain the i-th initial feature weight of the video frame.

[0106] Step 403, performing weighted calculation on the i-th initial feature weight of the video frame and the i-th feature map corresponding to the video frame to obtain the i-th initial weighted feature corresponding to the video frame.

[0107] In this embodiment, the video frame stream is input into the model to be trained in time sequence, the low-resolution frame first passes through the feature extraction module, and the features of the high-resolution frame at the current moment are fused with the features of the video frame at the previous moment through the attention mechanism. Moreover, since the features of the video frame at the previous moment have been calculated before the previous moment, and the attention mechanism has a small amount of calculation, the amount of calculation in the model training process can be greatly reduced.

[0108] In one embodiment, Figure 5 As shown, the specific processing process of step 402 "inputting the i-th feature map corresponding to the video frame and the i-th feature map of the previous video frame into the attention mechanism unit to obtain the i-th initial feature weight of the video frame" includes:

[0109] Step 501, adding the i-th feature map corresponding to the video frame to the i-th feature map of the previous video frame to obtain the i-th target feature map of the video frame.

[0110] Step 502: Perform a global average pooling operation on the i-th target feature map to obtain the i-th target feature vector of the video frame.

[0111] Step 503 , performing a dimensionality reduction operation on the i th target feature vector of the video frame to obtain the i th target reduced-dimensionality feature vector of the video frame.

[0112] Step 504 , performing a dimensionality-increasing convolution operation on the i th target reduced-dimensionality feature vector of the video frame to obtain the i th target increased-dimensionality feature vector of the video frame.

[0113] Step 505: Determine the i-th weight vector corresponding to the i-th target up-dimensional feature vector of the video frame according to a preset weight determination function.

[0114] Step 506: The product of the i-th weight vector and the i-th target upscaled feature vector is used as the i-th initial feature weight of the video frame.

[0115] Specifically, the following is combined with a schematic diagram Figure 6 , describes the execution process of the attention mechanism unit in detail: the i-th feature map corresponding to the video frame is the feature map at time t, and the i-th feature map of the previous video frame is the feature map at time t-1. After superimposing the feature map at time t and the feature map at time t-1, they are input into the global average pooling layer, downsampling convolution layer, relu layer, upsampling convolution layer and sigmoid gating layer in sequence. The specific execution process may include:

[0116] After adding the feature map at time t-1 to the feature map at time t, the following operation is performed: the result of adding the feature map at time t-1 to the feature map at time t (target feature map) is input into the compression module, which includes a global average pooling layer. After the operation of the global average pooling layer, the size of the target feature map changes from W×H×C to 1×1×C (W, H are the width and height of the original feature map, respectively, and C is the number of channels), and the target feature map is compressed into a vector, that is, the target feature vector is obtained. Through global average pooling, the training device can add all the feature values ​​of each channel in the C channels and then calculate the mean.

[0117] Let X = [x1,…,x c ,…,x C ], that is, a feature map of size H×W and number of channels C, whose formula is as follows:

[0118]

[0119] where z c Indicates channel statistics The value of the cth element of GP (·) is the global average pooling function, x c (i,j) is the eigenvalue at position (i,j) on channel c.

[0120] The training device inputs the calculated target feature vector into the excitation module, which includes a downsampling convolution layer, a relu layer, an upsampling convolution layer, and a sigmoid gating layer. The training device obtains the interdependence between channels by performing downsampling convolution-ReLU-upsampling convolution-sigmoid operations. The vector of size 1×1×C output by the compression module is subjected to r times (for example, r=16) dimensionality reduction operation to obtain a vector of size 1×1×C / r, i.e., the target reduced dimensionality feature vector. Then, after passing through the nonlinear activation function ReLU, the vector is subjected to r times dimensionality increase convolution operation to obtain a vector of size 1×1×C, i.e., the target increased dimensionality feature vector. Finally, the excitation module uses the sigmoid activation function to implement the gating mechanism. Finally, a 1×1×C weight vector with each element taking a value in [0, 1] is output. The training device can calculate the weight vector by the following formula:

[0121] s=f(W U δ(W D z))

[0122] Among them, f(·) represents sigmoid gating, δ(·) represents the nonlinear activation function ReLU, and W D is the weight of the convolutional layer that downsamples the channel by a factor of r, W U is the weight of the convolutional layer that upsamples the channel by a factor of r.

[0123] The excitation module uses a bottleneck structure, that is, it uses a dimension reduction and then a dimension increase with a scaling ratio of r to reduce the number of parameters and reduce the difficulty of training. In addition, a ReLU layer is added between the two upsampling convolution layers and the downsampling convolution layer as an activation function to enhance the nonlinear ability during the dimensional change process.

[0124] The process of training the device to realize channel attention is as follows: the weight vector s obtained by the excitation module c With the convolutional layer feature map x c Multiply, thus finally realizing the channel attention mechanism. The final output feature map can be expressed as:

[0125]

[0126] In this embodiment, by introducing the attention mechanism, different weights are assigned to the features of different channels. After utilizing global information and learning, useful features are enhanced and useless information is suppressed. That is to say, by introducing the feature map information of the previous video frame of the video frame, effective information can be emphasized for the video frame, thereby reducing the computing power consumption in the model training process and ensuring the high accuracy of the model training.

[0127] In one embodiment, the training method of the video frame optimization model further includes:

[0128] The first resolution video frame stream to be reconstructed is input into the trained video frame optimization model to obtain a second resolution video frame stream corresponding to the first resolution video frame stream to be reconstructed.

[0129] In one embodiment, Figure 7 As shown, before the step of obtaining training data, the training method of the video frame optimization model also includes:

[0130] Step 601: Obtain a sample video frame stream of a second resolution.

[0131] Specifically, the second resolution sample video frame stream can be a high-resolution sample video frame stream. The training device can obtain multiple high-resolution video streams within a preset time range, and sample a target number of sample video frames in each high-resolution video stream through a preset sampling algorithm to obtain multiple high-resolution sample video frame streams.

[0132] Step 602: Reconstruct the second resolution sample video frame stream into the first resolution sample video frame stream by using a preset video frame reconstruction algorithm.

[0133] Specifically, the first resolution sample video frame stream may be a low-resolution sample video frame stream. The process of the training device generating a low-resolution sample video frame stream by a preset video frame reconstruction algorithm may be: first, data cleaning is performed on multiple segments of high-resolution sample video frame streams to obtain multiple cleaned high-resolution sample video frame streams. Then, the high-resolution video frame is reduced by 2 times to the corresponding low-resolution video frame through bicubic downsampling in MATLAB to obtain a 2X video frame super-resolution reconstruction data set, that is, multiple segments of second resolution sample video frame streams and corresponding multiple segments of first resolution sample video frame streams are obtained.

[0134] Optionally, the bicubic downsampling function provided in MATLAB is a bicubic function, which can arbitrarily enlarge or reduce the image resolution through an algorithm. For example, the training device can downsample a 100x100 high-resolution sample video frame stream to a 50x50 low-resolution sample video frame stream.

[0135] In one embodiment, before the step of acquiring training data, the method further includes:

[0136] Step 701: Obtain a sample video frame stream with a second resolution.

[0137] Step 702: Reconstruct the second resolution sample video frame stream into the first resolution sample video frame stream by using a preset video frame reconstruction algorithm.

[0138] The following is a schematic diagram Figure 8 , the process of model training by the training method of the video frame optimization model is described in detail: the first sample video frame stream may include multiple video frames adjacent in time domain, such as a video frame at time LRt-2, a video frame at time LRt-1, a video frame at time LRt, etc. For each video frame in the first sample video frame stream, the video frame may be a video frame at time LRt, then the previous video frame of the video frame is the video frame at time LRt-1, and the previous video frame of the previous video frame is the video frame at time LRt-2.

[0139] The feature extraction module in the video frame optimization model to be trained includes n feature extraction submodules, and each feature extraction submodule includes a block unit (feature extraction unit) and an attention mechanism unit. The training device inputs the video frame at time LRt into the feature extraction module. In fact, the video frame at time LRt is input into the first block unit to obtain the feature map of the video frame. The training device inputs the feature map of the video frame output by the first block unit into the attention mechanism unit in the feature extraction submodule and the attention mechanism unit in the first feature extraction submodule corresponding to the next video frame of the video frame.

[0140] In this way, for the video frame at time LRt, the input of the attention mechanism unit in the first feature extraction submodule is the feature map of the video frame and the feature map of the previous video frame output by the block unit in the first feature extraction submodule of the previous video frame. For the video frame at time LRt, the output of the attention mechanism unit in the first feature extraction submodule is the first initial feature weight of the video frame calculated based on the feature map of the video frame and the feature map of the previous video frame. The output of the first feature extraction submodule is the feature map of the video frame and the first initial feature weight of the video frame. After weighted calculation, the first initial weighted feature of the video frame is obtained.

[0141] Since the feature extraction module includes n feature extraction submodules, and the second feature extraction submodule includes a block unit and an attention mechanism unit, the training device inputs the first initial feature weight of the video frame output by the first feature extraction submodule into the block to obtain the second feature map of the video frame, and inputs the second feature map and the feature map of the video frame output by the block unit in the first feature extraction submodule into the attention mechanism unit to obtain the second initial feature weight of the video frame. The training device performs weighted calculation on the second initial feature weight and the second feature map of the video frame to obtain the second initial weighted feature of the video frame.

[0142] For the third feature extraction submodule to the nth feature extraction submodule, the training process is similar to the training process of the second feature extraction submodule, which will not be repeated here. The training device uses the nth initial weighted feature output by the nth feature extraction submodule as the target weighted feature of the video frame.

[0143] In this way, the training device performs convolution calculation and sub-pixel convolution calculation on the target weighted feature to obtain the enlarged residual. The training device directly performs convolution calculation and sub-pixel convolution calculation on the video frame at the LRt moment to obtain the enlarged predicted video frame. Finally, the training device superimposes the residual with the initial predicted video frame of the second resolution to obtain the predicted video frame of the second resolution.

[0144] The training method of the video frame optimization model provided in this embodiment helps the current video frame to perform pixel-level magnification by utilizing the information of the previous video frame of the video frame, and basically does not increase the amount of redundant calculations, while adding time domain features to the training network of the model. The feature extraction part of the model of the disclosed embodiment only learns the residual between the HR image (high-resolution image) and the LR image (low-resolution image), performs sub-pixel convolution operations on the low-resolution image and the high-frequency residual, and adds them as outputs. After applying the idea of ​​residuals, since the residual image is relatively sparse, many values ​​are small or even zero, thereby speeding up the training speed.

[0145] Optionally, the feature extraction unit may be a variant of a lightweight network (ShuffleNetV2) because the scenario of the present application is a real-time video conference and real-time performance must be considered, such as Fig. 9 As shown in the figure, the block in the feature extraction module is named ShuffleSR Block. According to the relevant characteristics of super-resolution, some operations are removed. The Block unit is a block in the existing ShuffleNetV2 network that is modified for super-resolution reconstruction. The BN operation is removed. BN, or BatchNorm, will stretch the contrast of the image and normalize the color distribution, which is not suitable for super-resolution reconstruction.

[0146] The specific execution process of the block can be to input a low-resolution video frame, or the initial weighted features corresponding to the video frame, into the block. The block unit will perform channel rearrangement (Channel split) on the video frame or the initial weighted features of the video frame, and then perform 1×1 convolution calculation (Conv), 3×3 depth convolution calculation (DWConv) and 1×1 convolution calculation again on the rearranged result. In this way, the block unit can connect (Concat) the result of the convolution calculation and the result of the rearrangement, and then perform a channel shuffle operation (Channel shuffl) to obtain the feature map of the video frame.

[0147] It should be understood that although Figure 1-9 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1-9 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0148] In one embodiment, Fig.10 As shown, a training device for a video frame optimization model is provided, and the training device includes:

[0149] The training data acquisition module 701 is used to acquire training data, where the training data includes a plurality of first sample video frame streams at a first resolution and a plurality of second sample video frame streams at a second resolution, where the first resolution is smaller than the second resolution.

[0150] The target weighted feature determination module 702 is used to input the features of the previous video frame and the video frame of each video frame in the first sample video frame stream into the feature extraction module in the video frame optimization model to be trained to obtain the target weighted features corresponding to the video frame.

[0151] The predicted video frame determination module 703 is used to determine a predicted video frame of a second resolution according to the video frame and the target weighted features corresponding to the video frame.

[0152] The training loss function determination module 704 is used to calculate the training loss function according to the second sample video frame stream and each predicted video frame stream.

[0153] The training module 705 is used to update the network parameters of the video frame optimization model to be trained according to the training loss function, and return to execute the step of obtaining training data until the training loss function meets the preset training completion condition to obtain the trained video frame optimization model.

[0154] In one embodiment, the predicted video frame determination module includes:

[0155] An initial predicted video frame obtaining unit, used for performing convolution calculation on the video frame to obtain an initial predicted video frame of a second resolution;

[0156] A convolution calculation unit, used to perform convolution calculation on the target weighted features corresponding to the video frame to obtain the residual of the video frame;

[0157] The superposition unit is used to superimpose the residual with the initial predicted video frame of the second resolution to obtain the predicted video frame of the second resolution.

[0158] In one of the embodiments, the feature extraction module includes n feature extraction submodules;

[0159] The target weighted feature determination module includes:

[0160] an initial training unit, for each video frame in the first sample video frame stream, inputting a first feature map of a previous video frame of the video frame and the video frame into a first feature extraction submodule to obtain a first initial weighted feature corresponding to the video frame, wherein the first feature map of the previous video frame of the video frame is obtained after the previous video frame is input into the first feature extraction submodule corresponding to the previous video frame;

[0161] A cyclic training unit, used for inputting the i-1th initial weighted feature of the video frame and the i-th feature map of the previous video frame of the video frame into the i-th feature extraction submodule to obtain the i-th initial weighted feature corresponding to the video frame, wherein the i-th feature map of the previous video frame of the video frame is obtained after the previous video frame is input into the i-th feature extraction submodule corresponding to the previous video frame, and the initial value of i is 2;

[0162] Add 1 to the value of i, and perform the step of inputting the i-1th initial weighted feature of the video frame and the i-th feature map of the previous video frame of the video frame into the i-th feature extraction submodule until the n-th initial weighted feature corresponding to the video frame is obtained;

[0163] The target weighted feature determination unit is used to use the nth initial weighted feature corresponding to the video frame as the target weighted feature of the video frame.

[0164] In one embodiment, the feature extraction submodule includes a feature extraction unit and an attention mechanism unit; and a loop training unit includes:

[0165] The feature map extraction subunit is used to input the i-1th initial weighted feature of the video frame into the i-th feature extraction unit to obtain the i-th feature map corresponding to the video frame;

[0166] The initial feature weight determination subunit is used to input the i-th feature map corresponding to the video frame and the i-th feature map of the previous video frame into the attention mechanism unit to obtain the i-th initial feature weight of the video frame;

[0167] The initial weighted feature determination subunit is used to perform weighted calculation on the i-th initial feature weight of the video frame and the i-th feature map corresponding to the video frame to obtain the i-th initial weighted feature corresponding to the video frame.

[0168] In one embodiment, the initial feature weight determination subunit is specifically used to:

[0169] Add the i-th feature map corresponding to the video frame to the i-th feature map of the previous video frame to obtain the i-th target feature map of the video frame;

[0170] Perform a global average pooling operation on the i-th target feature map to obtain the i-th target feature vector of the video frame;

[0171] Performing a dimensionality reduction operation on the i-th target feature vector of the video frame to obtain the i-th target reduced-dimensionality feature vector of the video frame;

[0172] Perform a dimensionality-increasing convolution operation on the i-th target dimension-reduced feature vector of the video frame to obtain the i-th target dimension-increasing feature vector of the video frame;

[0173] Determine the i-th weight vector corresponding to the i-th target dimension-raising feature vector of the video frame according to a preset weight determination function;

[0174] The product of the i-th weight vector and the i-th target upscaled feature vector is taken as the i-th initial feature weight of the video frame.

[0175] In one embodiment, the device further comprises:

[0176] A module is used to input the first resolution video frame stream to be reconstructed into the trained video frame optimization model to obtain a second resolution video frame stream corresponding to the first resolution video frame stream to be reconstructed.

[0177] In one embodiment, the device further comprises:

[0178] A sample acquisition module, used to acquire a sample video frame stream of a second resolution;

[0179] The reconstruction module is used to reconstruct the second resolution sample video frame stream into the first resolution sample video frame stream by using a preset video frame reconstruction algorithm.

[0180] For the specific definition of the training device of the video frame optimization model, please refer to the definition of the training method of the video frame optimization model above, which will not be repeated here. The various modules in the above-mentioned training device of the video frame optimization model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0181] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store relevant data of the optimization model of the video frame. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a training method for a video frame optimization model is implemented.

[0182] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0183] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0184] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0185] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0186] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0187] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A training method for a video frame optimization model, characterized in that: The method comprises: Acquire training data, where the training data includes a plurality of first sample video frame streams with a first resolution and a plurality of second sample video frame streams with a second resolution, where the first resolution is smaller than the second resolution; For each video frame in the first sample video frame stream, input the features of the previous video frame of the video frame and the video frame into a feature extraction module in the video frame optimization model to be trained to obtain a target weighted feature corresponding to the video frame; the feature extraction module includes n feature extraction submodules; Determining a predicted video frame of a second resolution according to the video frame and a target weighted feature corresponding to the video frame; Calculating a training loss function according to the second sample video frame stream and each of the predicted video frame streams; The network parameters of the video frame optimization model to be trained are updated according to the training loss function, and the step of obtaining the training data is returned to be executed until the training loss function satisfies the preset training completion condition, thereby obtaining the trained video frame optimization model; The step of determining a predicted video frame of a second resolution according to the video frame and a target weighted feature corresponding to the video frame includes: Performing a first convolution calculation and a second sub-pixel convolution calculation on the video frame to obtain an initial predicted video frame of a second resolution; performing a first convolution calculation and a second sub-pixel convolution calculation on the target weighted feature corresponding to the video frame to obtain a residual of the video frame; superimposing the residual with the initial predicted video frame of the second resolution to obtain a predicted video frame of the second resolution; For each video frame in the first sample video frame stream, the feature of the previous video frame of the video frame and the video frame are input into a feature extraction module in the video frame optimization model to be trained to obtain a target weighted feature corresponding to the video frame, including: For each video frame in the first sample video frame stream, a first feature map of a previous video frame of the video frame and the video frame are input into a first feature extraction submodule to obtain a first initial weighted feature corresponding to the video frame, wherein the first feature map of the previous video frame of the video frame is obtained after the previous video frame is input into the first feature extraction submodule corresponding to the previous video frame; Inputting the i-1th initial weighted feature of the video frame and the i-th feature map of the previous video frame of the video frame into the i-th feature extraction submodule to obtain the i-th initial weighted feature corresponding to the video frame, the i-th feature map of the previous video frame of the video frame is obtained after inputting the previous video frame into the i-th feature extraction submodule corresponding to the previous video frame, and the initial value of i is 2; Add 1 to the value of i, and perform the step of inputting the i-1th initial weighted feature of the video frame and the i-th feature map of the previous video frame of the video frame into the i-th feature extraction submodule until the n-th initial weighted feature corresponding to the video frame is obtained; The nth initial weighted feature corresponding to the video frame is used as the target weighted feature of the video frame.

2. The method according to claim 1, characterized in that The feature extraction submodule includes a feature extraction unit and an attention mechanism unit; The step of inputting the i-1th initial weighted feature of the video frame and the i-th feature map of the previous video frame of the video frame into the i-th feature extraction submodule to obtain the i-th initial weighted feature corresponding to the video frame, including: Inputting the i-1th initial weighted feature of the video frame into the i-th feature extraction unit to obtain the i-th feature map corresponding to the video frame; Inputting the i-th feature map corresponding to the video frame and the i-th feature map of the previous video frame into the attention mechanism unit to obtain the i-th initial feature weight of the video frame; The i-th initial feature weight of the video frame and the i-th feature map corresponding to the video frame are weightedly calculated to obtain the i-th initial weighted feature corresponding to the video frame.

3. The method according to claim 2, characterized in that The step of inputting the i-th feature map corresponding to the video frame and the i-th feature map of the previous video frame into the attention mechanism unit to obtain the i-th initial feature weight of the video frame includes: Adding the i-th feature map corresponding to the video frame to the i-th feature map of the previous video frame to obtain the i-th target feature map of the video frame; Performing a global average pooling operation on the i-th target feature map to obtain an i-th target feature vector of the video frame; Performing a dimensionality reduction operation on the i-th target feature vector of the video frame to obtain the i-th target reduced-dimensionality feature vector of the video frame; Performing a dimensionality-raising convolution operation on the i-th target dimensionality-reduced feature vector of the video frame to obtain the i-th target dimensionality-raising feature vector of the video frame; Determine, according to a preset weight determination function, an i-th weight vector corresponding to the i-th target up-dimensional feature vector of the video frame; The product of the i-th weight vector and the i-th target upscaled feature vector is used as the i-th initial feature weight of the video frame.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: The first resolution video frame stream to be reconstructed is input into the trained video frame optimization model to obtain a second resolution video frame stream corresponding to the first resolution video frame stream to be reconstructed.

5. The method according to claim 1, characterized in that Before the step of acquiring training data, the method further includes: Obtain a second resolution sample video frame stream; The second-resolution sample video frame stream is reconstructed into a first-resolution sample video frame stream by using a preset video frame reconstruction algorithm.

6. A training device for a video frame optimization model, characterized in that: The device comprises: A training data acquisition module, used to acquire training data, wherein the training data includes a plurality of first sample video frame streams with a first resolution and a plurality of second sample video frame streams with a second resolution, wherein the first resolution is smaller than the second resolution; A target weighted feature determination module is used to input the features of the previous video frame of each video frame in the first sample video frame stream and the video frame into a feature extraction module in the video frame optimization model to be trained to obtain the target weighted features corresponding to the video frame; the feature extraction module includes n feature extraction submodules; A predicted video frame determination module, used to determine a predicted video frame of a second resolution according to the video frame and a target weighted feature corresponding to the video frame; A training loss function determination module, used to calculate a training loss function according to the second sample video frame stream and each of the predicted video frame streams; A training module, used for updating the network parameters of the video frame optimization model to be trained according to the training loss function, and returning to execute the step of obtaining the training data until the training loss function satisfies a preset training completion condition, thereby obtaining a trained video frame optimization model; The predicted video frame determination module is specifically used to perform a first convolution calculation and a second sub-pixel convolution calculation on the video frame to obtain an initial predicted video frame of a second resolution; perform a first convolution calculation and a second sub-pixel convolution calculation on the target weighted feature corresponding to the video frame to obtain a residual of the video frame; and superimpose the residual with the initial predicted video frame of the second resolution to obtain a predicted video frame of the second resolution; The target weighted feature determination module is specifically used to input the first feature map of the previous video frame of each video frame in the first sample video frame stream and the video frame into the first feature extraction submodule to obtain the first initial weighted feature corresponding to the video frame, wherein the first feature map of the previous video frame of the video frame is obtained after the previous video frame is input into the first feature extraction submodule corresponding to the previous video frame; Inputting the i-1th initial weighted feature of the video frame and the i-th feature map of the previous video frame of the video frame into the i-th feature extraction submodule to obtain the i-th initial weighted feature corresponding to the video frame, the i-th feature map of the previous video frame of the video frame is obtained after inputting the previous video frame into the i-th feature extraction submodule corresponding to the previous video frame, and the initial value of i is 2; Add 1 to the value of i, and perform the step of inputting the i-1th initial weighted feature of the video frame and the i-th feature map of the previous video frame of the video frame into the i-th feature extraction submodule until the n-th initial weighted feature corresponding to the video frame is obtained; The nth initial weighted feature corresponding to the video frame is used as the target weighted feature of the video frame.

7. The device according to claim 6, characterized in that The feature extraction submodule includes a feature extraction unit and an attention mechanism unit; Circuit training unit, including: The feature map extraction subunit is used to input the i-1th initial weighted feature of the video frame into the i-th feature extraction unit to obtain the i-th feature map corresponding to the video frame; The initial feature weight determination subunit is used to input the i-th feature map corresponding to the video frame and the i-th feature map of the previous video frame into the attention mechanism unit to obtain the i-th initial feature weight of the video frame; The initial weighted feature determination subunit is used to perform weighted calculation on the i-th initial feature weight of the video frame and the i-th feature map corresponding to the video frame to obtain the i-th initial weighted feature corresponding to the video frame.

8. The device according to claim 7, characterized in that The initial feature weight determination subunit is specifically used for: Add the i-th feature map corresponding to the video frame to the i-th feature map of the previous video frame to obtain the i-th target feature map of the video frame; Perform a global average pooling operation on the i-th target feature map to obtain the i-th target feature vector of the video frame; Performing a dimensionality reduction operation on the i-th target feature vector of the video frame to obtain the i-th target reduced-dimensionality feature vector of the video frame; Performing a dimension-raising convolution operation on the i-th target dimension-reduced feature vector of the video frame to obtain the i-th target dimension-raising feature vector of the video frame; Determine the i-th weight vector corresponding to the i-th target up-dimensional feature vector of the video frame according to a preset weight determination function; The product of the i-th weight vector and the i-th target upscaled feature vector is taken as the i-th initial feature weight of the video frame.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images