Video processing method and device, equipment, medium and electronic product
Through the video processing model, the video is extracted and the motion scene category is determined, and the video alignment strategy is dynamically selected, which solves the problem of poor video super resolution recovery effect in the existing technology, and achieves more efficient video alignment and reconstruction effects.
Patent Information
- Application Number
- CN202411513321.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-28
- Publication Date
- 2025-02-07
AI Technical Summary
In the prior art, it is impossible to dynamically select video alignment strategies based on specific tasks and data set characteristics, resulting in poor video super-resolution recovery effect.
Through the video processing model, feature extraction is performed on the video to be processed, the motion scene category is determined, and the matching video alignment strategy is selected according to the category for alignment and reconstruction.
Dynamically selecting video alignment strategies improves the quality and efficiency of video alignment, and thus improves the video reconstruction effect.
Smart Images
Figure CN119402682A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of video super-resolution reconstruction, and in particular to a video processing method, device, equipment, medium and electronic product. Background Art
[0002] Video super-resolution (VSR) technology uses an algorithm model to utilize intra-frame information and inter-frame information of low-resolution videos to restore low-resolution video frames to high-resolution video frames, thereby improving the visual quality and viewing experience of the video. Feature alignment is an important step in the process of improving the resolution of video frames. Through feature alignment, it can help match and correspond the features in low-resolution video frames with the features in high-resolution reference frames, thereby restoring higher-resolution videos. In the prior art, the process of restoring low-resolution video frames to high-resolution video frames often adopts an alignment method, and does not dynamically select an alignment strategy based on the specific tasks and data set characteristics, making the alignment method relatively single, which makes the process of video super-resolution limited, resulting in poor video resolution restoration effect. Summary of the invention
[0003] The purpose of the embodiments of the present application is to provide a video processing method, apparatus, device, medium and electronic product to solve the problem in the related art that it is impossible to dynamically select an alignment strategy according to the specific tasks and data set characteristics, making the alignment method relatively single, resulting in poor video resolution recovery effect.
[0004] In order to solve the above technical problems, the embodiments of the present application are implemented as follows: In a first aspect, an embodiment of the present application provides a video processing method, including: Perform feature extraction processing on the video to be processed through the video processing model to obtain feature information of the video to be processed; Determining the motion scene category of the video to be processed by the video processing model according to the feature information of the video to be processed; Determining a video alignment strategy that matches the video to be processed according to the motion scene category of the video to be processed; The video processing model is used to align the video to be processed using the video alignment strategy, and the processed video to be processed is reconstructed to obtain a target video.
[0005] In a second aspect, an embodiment of the present application provides a video processing device, including: A feature extraction module is used to perform feature extraction processing on the video to be processed through a video processing model to obtain feature information of the video to be processed; A scene determination module, used to determine the motion scene category of the video to be processed according to the feature information of the video to be processed by using the video processing model; A strategy determination module, used to determine a video alignment strategy that matches the video to be processed according to the motion scene category of the video to be processed; The video processing module is used to align the video to be processed by using the video processing model and the video alignment strategy, and reconstruct the processed video to be processed to obtain a target video.
[0006] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus; wherein the processor, the communication interface and the memory communicate with each other through a bus; the memory is used to store computer programs; and the processor is used to execute the programs stored in the memory to implement the video processing method as described in the first aspect.
[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the video processing method described in the first aspect is implemented.
[0008] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the video processing method as described in the first aspect.
[0009] In an embodiment of the present application, first, a feature extraction process is performed on the video to be processed through a video processing model to obtain feature information of the video to be processed; then, the motion scene category of the video to be processed is determined according to the feature information of the video to be processed through the video processing model; according to the motion scene category of the video to be processed, a video alignment strategy matching the video to be processed is determined; then, the video to be processed is aligned through the video processing model using the video alignment strategy, and the processed video to be processed is reconstructed to obtain a target video. It can be seen that through the embodiment of the present application, the motion scene category of the video to be processed can be determined according to the feature information of the video to be processed, and the video alignment strategy matching the motion scene category of the video to be processed can be determined according to the motion scene category, and then the video to be processed can be aligned based on the matching video processing strategy, and finally the reconstructed target video can be obtained. The video alignment method is no longer single, but dynamically selects the video alignment strategy according to the motion scene characteristics of the specific video to be processed, which improves the quality and efficiency of video alignment, and thus improves the video reconstruction effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0011] Figure 1 A schematic diagram of a video processing method provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of the Transformer model provided in the embodiment of the present application; Figure 3 A schematic diagram of a VSR algorithm flow based on patch alignment provided in an embodiment of the present application; Figure 4 A schematic diagram of a VSR process based on a guided deformable attention algorithm provided in an embodiment of the present application; Figure 5 A schematic diagram of the GDA alignment algorithm flow provided in the embodiment of the present application; Figure 6 A schematic diagram of the structure of a super-resolution reconstruction module provided in an embodiment of the present application; Figure 7 A schematic diagram of an exemplary flow chart of a video processing method provided in an embodiment of the present application; Figure 8 A schematic diagram of the module composition of a video processing device provided in an embodiment of the present application; Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0012] Embodiments of the present application provide a video processing method, apparatus, device, medium and electronic product.
[0013] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.
[0014] Before introducing the embodiments of the present application, a video alignment method in the related art is first described.
[0015] Alignment methods in related technologies include optical flow alignment, deformable convolution (DC) alignment, deformable attention (DA) alignment, and sliding window-based video super-resolution (VSR).
[0016] Optical flow alignment can capture pixel-level motion information, but the accuracy of optical flow estimation is unstable. In some cases, optical flow estimation may be affected by, for example, illumination changes, occlusion, noise, etc., resulting in poor video alignment. In addition, optical flow alignment requires calculating the motion vector of each pixel in the video frame, which is computationally intensive. This may limit the speed of the algorithm when processing high-resolution videos or large-scale data sets. Optical flow alignment usually also requires moving pixels in adjacent video frames to corresponding positions in the target video frame. This process involves interpolation operations, which may introduce errors, resulting in a decrease in the quality of the aligned video frames.
[0017] DC alignment allows the network to adaptively adjust the sampling position during the convolution process to better capture the geometric transformation in the image. Because the sampling position needs to be dynamically calculated, the computational complexity of DC alignment is relatively high. Similar to optical flow alignment, DC alignment may also be affected by factors such as noise and occlusion, which may reduce its performance.
[0018] Similar to the DC alignment method, DA alignment allows the network to adaptively adjust the position and size of the receptive field in the attention mechanism. Combining the ideas of DC alignment and attention mechanism, DA alignment can better capture geometric transformations and local details in the image. By learning an offset for each position, the network can focus on a wider area related to the current position. Therefore, compared with other alignment methods, DA alignment can divide long sequence videos into multiple segments of the same length. DA alignment performs alignment between segments instead of just alignment between video frames. In addition, since it is necessary to learn an offset for each position and calculate the corresponding attention weight, the computational complexity of DA alignment is also relatively high.
[0019] Sliding window-based VSR also makes some contributions in alignment. It aligns small blocks or regions (patches) of the image instead of pixels. The main advantage of this method is that it can better capture local structure and detail information in the image while maintaining computational efficiency. However, this method also has some challenges and limitations. First, how to choose the appropriate patch size and number is a key issue. If the patch is too small, it may lead to insufficient alignment accuracy; if the patch is too large, it may increase computational complexity and sensitivity to noise. Secondly, this method will experience misalignment for fast motion and complex scenes, and the processing capability needs to be improved. In these cases, it may be necessary to combine other techniques to improve alignment accuracy and robustness.
[0020] In addition, in VSR tasks with low pixel motion amplitude, the attention mechanism can directly utilize multi-frame information from unaligned videos and is capable of handling misalignment within a certain range. At this time, using other alignment methods will increase the performance burden.
[0021] A video processing method, device, equipment, medium and electronic product provided in an embodiment of the present application are described in detail below in conjunction with the accompanying drawings.
[0022] Figure 1 A schematic diagram of a video processing method provided in an embodiment of the present application is provided. Figure 1 The execution subject of the method in the embodiment may be a server, wherein the server may be an independent server, a server cluster composed of multiple servers, or a cloud server for cloud computing processing. Moreover, the server may be a server capable of performing network operation processing. Figure 1 As shown, the method may specifically include the following steps: S102, performing feature extraction processing on the video to be processed through a video processing model to obtain feature information of the video to be processed.
[0023] In an embodiment of the present application, the video to be processed may be a continuous image sequence, including multiple frames of images, i.e., video frames. The video to be processed is a low-quality video with a low frame resolution. By aligning and super-resolution processing the video to be processed, the video frames of the video to be processed are restored from low-resolution video frames to high-resolution video frames. First, feature extraction processing can be performed on the video to be processed through a video processing model to obtain feature information of the video to be processed. The feature information of the video to be processed may include: motion features, color features, texture features, shape features, spatiotemporal features, etc.
[0024] In one embodiment, the video processing model includes a feature extraction network; through the video processing model, feature extraction processing is performed on the video to be processed to obtain feature information of the video to be processed, including: Through the feature extraction network, feature extraction is performed on the video to be processed to obtain a feature map of the video to be processed; Through the feature extraction network, the feature information of the video to be processed is determined according to the feature map of the video to be processed.
[0025] In an embodiment of the present application, the video processing model includes a feature extraction network, through which feature extraction processing is performed on the video to be processed to obtain a feature map of the video to be processed, and feature information of the video to be processed is determined based on the feature map of the video to be processed.
[0026] Feature Map is a high-dimensional representation extracted from raw video data through a series of algorithms or neural networks in video processing and analysis. Feature Map captures key information in the video, such as motion, color, texture, shape, and objects. Feature Maps include: spatial feature maps, temporal feature maps, and high-level semantic feature maps. Among them, spatial feature maps: mainly capture spatial information in video frames, such as color, texture, and shape, etc. Spatial feature maps are usually extracted from a single video frame. Temporal feature maps: capture motion information in the video, such as the movement trajectory and speed change of objects. High-level semantic feature maps: capture high-level semantic information in the video, such as object recognition, scene classification, behavior recognition, etc. The feature map of the video to be processed can be a feature map of each video frame in the video to be processed, or it can be a feature map extracted from multiple video frames.
[0027] In one implementation, the feature graph of the embodiment of the present application may be a time series feature graph. According to the time series feature graph, shallow features of the video to be processed are extracted to obtain feature information of the video to be processed, such as motion features.
[0028] In one implementation, the feature extraction network may include a convolutional neural network. When performing feature extraction on a video to be processed, first, a convolutional layer in the convolutional neural network may be used to perform feature extraction on the video to be processed to obtain a feature map of the video to be processed. The convolution calculation is as follows: ; (1) In formula (1), Y is the output feature map, W is the weight matrix of the convolution kernel, whose dimension depends on the size of the convolution kernel, the number of input channels and the number of output channels, and b is the bias term, which is the same as the number of output channels. represents the convolution operation, Represents an activation function, such as the ReLU activation function.
[0029] After obtaining the feature map of the video to be processed, the convolutional neural network can also be used to extract the feature information of the video frame of the video to be processed.
[0030] In another implementation, after obtaining the feature map of the video to be processed, the residual SwinTransformer block (RSTB) can also be used to extract shallow features of the video to be processed. In RSTB, each feature map is first divided into multiple non-overlapping windows, and then a self-attention operation is performed within each window, which allows the network to capture dependencies in local areas. The formula for the self-attention operation can be expressed as: ; (2) In formula (2), Q, K, and V are the query matrix, key matrix, and value matrix in the self-attention mechanism, which are used to calculate the attention score. is the dimension of the key, used to scale the dot product attention scores.
[0031] Through the embodiments of the present application, by performing feature extraction processing on the video to be processed, local and global information of the video frame can be captured to obtain feature information of the video frame of the video to be processed.
[0032] S104, determining the motion scene category of the video to be processed according to feature information of the video to be processed by using a video processing model.
[0033] In the embodiment of the present application, the motion scene category of the video to be processed is determined through a video processing model according to the feature information of the video frame to be processed extracted as above.
[0034] In one embodiment, the motion scene category of the video to be processed is determined by using a video processing model according to feature information of the video to be processed, including: Determine the motion information of the video to be processed according to the feature information of the video to be processed through the video processing model; The motion scene category of the video to be processed is determined according to the motion information of the video to be processed and a preset classification condition.
[0035] In an embodiment of the present application, first, through a video processing model, motion information of video frames of the video to be processed is determined according to feature information of the video frames of the video to be processed; then, motion scene information of the video to be processed is determined according to the motion information of the video frames of the video to be processed and preset classification conditions.
[0036] In one embodiment, the video processing model includes a selector network; determining motion information of the video to be processed according to feature information of the video to be processed through the video processing model includes: The feature information of the video to be processed is input into the selector network, and motion estimation processing is performed to obtain the motion information of the video to be processed.
[0037] In an embodiment of the present application, the video processing model may include a selector network, and the feature information of the video frame of the video to be processed can be input into the selector network for motion estimation processing to obtain the motion information of the video frame of the video to be processed.
[0038] In one implementation, the feature information of the video to be processed can be used to train the selector network. Through training, the selector network can learn the applicable conditions of various alignment strategies under different motion scenes. The selector network analyzes the changes in adjacent pixel values, local deformation degree and motion vector of the video frame to determine the motion scene characteristics of the input video to be processed. The selector network can perform general motion estimation: for example, based on the current video frame image f(x, y) at time t and the reference video frame f'(x, y) at time t, the corresponding motion field d(x; t, t+ t), the motion vector of the current video frame can be obtained.
[0039] In one embodiment, the motion information of the video to be processed includes the inter-frame motion intensity of the video to be processed; the preset classification condition includes a first motion intensity threshold and a second motion intensity threshold; According to the motion information of the video to be processed and the preset classification conditions, the motion scene category of the video to be processed is determined, including: When the inter-frame motion intensity is less than the first motion intensity threshold, determining that the motion scene type of the video to be processed is a static scene; When the inter-frame motion intensity is greater than the first motion intensity threshold and less than the second motion intensity threshold, determining that the motion scene type of the video to be processed is a light motion scene; When the inter-frame motion intensity is greater than the second motion intensity threshold, it is determined that the motion scene type of the video to be processed is a violent motion scene.
[0040] In the embodiment of the present application, the motion information of the video to be processed may be the inter-frame motion intensity of the video to be processed, and accordingly, the preset classification condition may be a preset motion intensity threshold, which may include a first motion intensity threshold and a second motion intensity threshold, wherein the first motion intensity threshold is less than the second motion intensity threshold.
[0041] The inter-frame motion intensity of a video is a quantitative value describing the intensity of image motion between consecutive video frames. The motion scene category of the video to be processed is determined by determining the intensity relationship between the inter-frame motion intensity of the video and the first motion intensity threshold and the second motion intensity threshold.
[0042] In one implementation, when the intensity relationship is that the inter-frame motion intensity is less than the first motion intensity threshold, the motion scene category of the video to be processed is determined to be a static scene. It is understandable that when the image motion intensity between consecutive video frames is small, the objects or scenes in the video content change relatively slowly in time, without significant rapid movement or mutation, and therefore, the motion scene category of the video to be processed can be determined to be a static scene. When the inter-frame motion intensity is greater than the first motion intensity threshold and less than the second motion intensity threshold, the motion scene category of the video to be processed is determined to be a mild motion scene. It is understandable that when the image motion intensity between consecutive video frames is large, there is a more significant motion change in the video content, such as the slow movement of an object or a person, or a scene switch in the video. When the inter-frame motion intensity is greater than the second motion intensity threshold, the motion scene type of the video to be processed is determined to be a violent motion scene. It is understandable that when the image motion between consecutive video frames is very intense, there are extremely significant motion changes in the video content. Such motion changes include, for example, the rapid movement of objects or people. The translation, rotation, zoom or shaking of the camera can also cause intense image motion between frames. It can also be the rapid switching of scenes in the video, such as a high-speed moving car or a flying bird.
[0043] In one example, the value of the first motion intensity threshold may be 0.1, and the value of the second motion intensity threshold may be 0.5. Assuming that the inter-frame motion intensity of the video to be processed output by the selector network is 0.2, then it can be determined that the motion scene category of the video to be processed is a mild motion scene. Similarly, when the inter-frame motion intensity of the video to be processed is less than 0.1, it can be determined that the motion scene category of the video to be processed is a static scene. When the inter-frame motion intensity of the video to be processed is greater than 0.5, it can be determined that the motion scene category of the video to be processed is a violent motion scene.
[0044] In the embodiment of the present application, a video processing model is used to determine the motion scene category of the video to be processed based on the extracted feature information of the video to be processed, and then a suitable video alignment strategy can be matched for the video to be processed based on the motion scene category of the video to be processed, thereby improving the video alignment effect.
[0045] S106: Determine a video alignment strategy that matches the video to be processed according to the motion scene category of the video to be processed.
[0046] In the embodiment of the present application, according to the motion scene category of the video to be processed determined as above, a video alignment strategy matching the video to be processed is determined, and then the video to be processed is aligned.
[0047] In one embodiment, determining a video alignment strategy that matches the video to be processed according to the motion scene category of the video to be processed includes: When the motion scene type of the video to be processed is a static scene, determining that the video alignment strategy matching the video to be processed is non-alignment; When the motion scene type of the video to be processed is a light motion scene, determining that the video alignment strategy matching the video to be processed is alignment between adjacent video frames; When the motion scene type of the video to be processed is a violent motion scene, the video alignment strategy that matches the video to be processed is determined to be alignment between adjacent segments; the segments are used to represent at least one motion scene of the video to be processed.
[0048] S108, using the video processing model and the video alignment strategy, aligning the video to be processed, and reconstructing the processed video to be processed to obtain a target video.
[0049] In the embodiment of the present application, after determining the video alignment strategy that matches the video to be processed, the video to be processed is aligned using the matched video alignment strategy. Then, the aligned video is reconstructed to obtain the target video.
[0050] In one implementation, the reconstruction process may be an upsampling process. Specifically, the upsampling process may be an upsampling based on interpolation or an upsampling based on learning.
[0051] In one implementation, the video processing model can be a Transformer-based processing model. When the motion scene type of the video to be processed is a static scene, the alignment strategy can be non-alignment, and the inter-frame alignment can be indirectly achieved directly through the self-attention mechanism of the Transformer model. Specifically, the self-attention mechanism of the Transformer model can calculate the similarity of pixels or feature vectors between different video frames, thereby finding their correspondence in space and time. This correspondence can be regarded as an implicit alignment method, which helps the model to utilize multi-frame information in the super-resolution process. At the same time, when calculating self-attention, the Transformer can process information at all positions at the same time, so the Transformer model has better parallel computing capabilities.
[0052] Figure 2 A schematic diagram of the structure of the Transformer model provided in the embodiment of the present application is shown in FIG. Figure 2As shown in the figure, the Transformer model includes an encoder and a decoder. The encoder is composed of N identical layers stacked together, each of which contains the following key sublayers: The Self-Attention Layer (not shown in the figure) is the core of the Transformer encoder. It allows the model to consider other positions in the sequence when processing each position of the input sequence, thereby capturing long-distance dependencies between elements. The Self-Attention Layer generates a new vector representation by calculating the attention score between three matrices: query, key, and value. These matrices are usually obtained from the input embedding vector through linear transformation. Multi-Head Self-Attention is an extension of the Transformer encoder that enhances the expressiveness of the model by performing self-attention operations in parallel on multiple different subspaces. Each head learns different aspects of the input, and then the information of these heads is merged. The Feed-Forward Neural Network Layer is a simple fully connected feedforward network that is used to further nonlinearly transform and map the output of each self-attention layer. It usually consists of two linear layers and an activation function (such as the ReLU activation function) to learn complex feature representations. In addition, each sublayer is followed by a residual connection, which helps alleviate the gradient vanishing or gradient exploding problem in deep networks and makes the model easier to train. The residual connection is followed by a layer normalization operation, which helps speed up the training process and improve the stability of the model.
[0053] The decoder is also composed of N identical layers stacked together. Compared with the encoder, it contains the following three key sub-layers: The Masked Self-Attention Layer (not shown in the figure) is similar to the self-attention layer in the encoder, but with the addition of a masking operation. Masking is used to ensure that when predicting the output of the current position, the model can only see the input information before the position (including the generated output sequence part), thereby simulating the process of sequence generation. The Multi-Head Self-Attention Layer allows the decoder to focus on the output representation of the encoder, thereby capturing the dependency between the input sequence and the output sequence. The query comes from the previous layer of the decoder, while the key and value come from the output of the encoder. The Feed-Forward Neural Network Layer is the same as the Feed-Forward Neural Network Layer in the encoder, which is used to further nonlinearly transform and map the outputs of the first two sub-layers. The sub-layers in the decoder also use residual connections and layer normalization to improve the training efficiency and stability of the model. Finally, a linear layer and a softmax activation function are used to obtain the final output.
[0054] When the motion scene type of the video to be processed is a light motion scene, the video alignment strategy that matches the video to be processed is determined to be alignment between adjacent video frames. In one implementation, the alignment between adjacent video frames can be performed based on a sliding window VSR Transformer. The video frame is divided into multiple non-overlapping patches according to the window size of the Transformer. Assuming that the size of each patch is H x W, the entire video frame can be divided into (image_height / / H) x (image_width / / W) patches.
[0055] The optical flow field between adjacent video frames is calculated by using an optical flow algorithm (such as the Farneback method, the Lucas-Kanade method, etc.). For each patch, the average or median of the optical flow vectors of all pixels inside it is calculated to obtain a vector representing the movement of the patch, namely the average optical flow vector. Assuming that the average optical flow vector can better represent the movement of the entire patch, each patch is moved to a new position as a whole according to the calculated average optical flow vector. During the movement process, the relative position and information between pixels inside the patch are not changed. If the position of the moved patch exceeds the boundary of the video frame image, appropriate cropping or padding can be performed. Afterwards, all aligned patches are reassembled into a new image. The VSR under this alignment method uses N multi-frame self-attention blocks (MFSAB) as the backbone network, which contains two layer normalizations (LayerNorm), a multi-frame self-attention layer, and a multi-layer perceptron layer.
[0056] Figure 3 Schematic diagram of the VSR algorithm flow based on patch alignment according to an embodiment of the present application. Figure 3 As shown, , , are three adjacent video frames, and t is the current time. The multi-frame self-attention block (MFSAB) contains two LayerNorm layers, a multi-frame self-attention layer, and a multi-layer perceptron layer. The three adjacent video frames are patch-aligned by N multi-frame self-attention blocks to obtain the aligned video. In each self-attention layer, the query (Query, Q), key (Key, K), and value (Value, V) are calculated as: , , ; (3) In formula (3), X is the feature of the patch, , , is the weight matrix.
[0057] When the motion scene type of the video to be processed is a violent motion scene, the video alignment strategy that matches the video to be processed is determined to be alignment between adjacent segments. In one implementation, the alignment between adjacent segments can be a GDA alignment method.
[0058] Figure 4 A schematic diagram of a VSR process based on a guided deformable attention algorithm provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, in an exemplary implementation, the video feature of the i-th layer is (T, H, W and C are the video length, height, width and channel respectively) The sequence is shortened and segmented into T / N segments, each segment includes N video frames, and each video feature is , based on optical flow (t is the current time), the fragment features at time t-1 and the fragment characteristics at time t Perform guided deformable attention (GDA) alignment, the alignment process is: ; (4) In formula (4), are the features of the aligned fragments.
[0059] The feature correction module , Fragment and snippets Perform feature correction to obtain hidden state The features of different video clips are propagated in multiple layers in a repeated manner, and all the clip features are aligned through propagation iterations.
[0060] The above VSR process based on the GDA algorithm can refer to the relevant technology, and this application will not go into details here.
[0061] Figure 5 The schematic diagram of the GDA alignment algorithm flow provided in the embodiment of the present application is as follows: Figure 5 As shown, in an exemplary implementation, the fragments are first pre-aligned, and the pre-alignment process is: ; (5) In formula (5), W is the warp operation. Represents pre-aligned fragments.
[0062] The optical flow prediction offset generated after pre-alignment is , update the optical flow according to the optical flow prediction offset, and finally perform self-attention operation to obtain: ; (6) ; (7) ; (8) Among them, Sampling in formula (7) and formula (8) represents DA alignment. , , represents a fully connected layer.
[0063] Finally, the aligned fragments are output .
[0064] Through this embodiment, a video alignment strategy that matches the motion scene category of the video to be processed is determined according to the motion scene category, and then the video to be processed can be aligned based on the matching video processing strategy. The video alignment method is no longer single, but the video alignment strategy is dynamically selected according to the specific motion scene characteristics, thereby improving the quality and efficiency of video alignment.
[0065] After the video to be processed is aligned, the aligned video is reconstructed to obtain the target video.
[0066] In one implementation, the video model to be processed may include a super-resolution reconstruction module. Figure 6 A schematic diagram of the structure of the super-resolution reconstruction module provided in the embodiment of the present application is shown in FIG. Figure 6 As shown in FIG. 1 , the module may be composed of 1 convolution layer, RSTB and sub-pixel convolution layer. The convolution layer, RSTB and sub-pixel convolution layer are used to upsample the aligned video to obtain the final target video. The resolution of the video frame of the final target video is higher than the resolution of the video frame before alignment and reconstruction.
[0067] The super-resolution reconstruction module uses the Charbonnier loss function, and the specific calculation method of the loss function is as follows: , ( ); (9) In formula (9), It is the HQ video reconstructed by the sub-pixel convolution layer, and the HQ video is the target video.
[0068] In an embodiment of the present application, first, a feature extraction process is performed on the video to be processed through a video processing model to obtain feature information of the video to be processed; then, the motion scene category of the video to be processed is determined according to the feature information of the video to be processed through the video processing model; according to the motion scene category of the video to be processed, a video alignment strategy matching the video to be processed is determined; then, the video to be processed is aligned through the video processing model using the video alignment strategy, and the processed video to be processed is reconstructed to obtain a target video. It can be seen that through the embodiment of the present application, the motion scene category of the video to be processed can be determined according to the feature information of the video to be processed, and the video alignment strategy matching the motion scene category of the video to be processed can be determined according to the motion scene category, and then the video to be processed can be aligned based on the matching video processing strategy, and finally the reconstructed target video can be obtained. The video alignment method is no longer single, but dynamically selects the video alignment strategy according to the motion scene characteristics of the specific video to be processed, which improves the quality and efficiency of video alignment, and thus improves the video reconstruction effect.
[0069] Figure 7 An exemplary flow chart of a video processing method provided in an embodiment of the present application is as follows: Figure 7 As shown, in an exemplary embodiment, the video LQ is a low-quality video with a low resolution of the video frame. The video LQ is restored to a high-resolution video HQ by a video processing method. First, the video LQ is input into the convolution layer and subjected to feature extraction processing to obtain a feature map of the video LQ. Then, the shallow features of the video LQ are extracted by the residual Swin Transformer block to obtain feature information of the video LQ. Next, the extracted feature information of the video LQ is input into the selector network. The video LQ is subjected to motion estimation processing by the selector network according to the feature information of the video LQ to obtain the inter-frame motion intensity T of the video LQ. Then, the motion scene of the video LQ is determined according to the intensity relationship between the inter-frame motion intensity and the preset motion intensity thresholds T1 and T2, where T1 is less than T2. When T is less than T1, the motion scene of the video LQ is a static scene; when T is greater than T1 and less than T2, the motion scene of the video LQ is a mild motion scene; when T is greater than T2, the motion scene of the video LQ is a violent motion scene. Next, the alignment method of video LQ is determined according to the motion scene category of video LQ. When the motion scene category of video LQ is a static scene, the motion scene category of video LQ is determined to be unaligned; when the motion scene category of video LQ is a light motion scene, the motion scene category of video LQ is determined to be patch alignment between adjacent video frames; when the motion scene category of video LQ is a violent motion scene, the motion scene category of video LQ is determined to be DA alignment. After determining the alignment method of video LQ, the video LQ is aligned using the corresponding alignment method, and then the aligned video LQ' is super-reconstructed to finally obtain video HQ.
[0070] Corresponding to the video processing method provided in the above embodiment, based on the same technical concept, the embodiment of the present application also provides a video processing device, Figure 8 A schematic diagram of the module composition of a video processing device provided in an embodiment of the present application, wherein the video processing device is used to execute Figure 1-Figure 6 Describe the video processing methods, such as Figure 8 As shown, the video processing device includes: The feature extraction module 801 is used to perform feature extraction processing on the video to be processed through the video processing model to obtain feature information of the video to be processed; A scene determination module 802, configured to determine the motion scene category of the video to be processed according to the feature information of the video to be processed by using the video processing model; A strategy determination module 803 is used to determine a video alignment strategy that matches the video to be processed according to the motion scene category of the video to be processed; The video processing module 804 is used to align the video to be processed by using the video processing model and the video alignment strategy, and reconstruct the processed video to be processed to obtain a target video.
[0071] Optionally, the video processing model includes a feature extraction network; the feature extraction module 801 is specifically used to perform feature extraction processing on the video to be processed through the feature extraction network to obtain a feature map of the video to be processed; The feature extraction network is used to determine feature information of the video to be processed according to the feature graph of the video to be processed.
[0072] Optionally, the scene determination module 802 is specifically configured to determine the motion information of the video to be processed according to the feature information of the video to be processed by using the video processing model; The motion scene category of the video to be processed is determined according to the motion information of the video to be processed and a preset classification condition.
[0073] Optionally, the video processing model includes a selector network; the scene determination module 802 is further specifically used to input feature information of the video to be processed into the selector network, perform motion estimation processing, and obtain motion information of the video to be processed.
[0074] Optionally, the motion information of the video to be processed includes the inter-frame motion intensity of the video to be processed; the preset classification condition includes a first motion intensity threshold and a second motion intensity threshold; the scene determination module 802 is further specifically used to determine that the motion scene type of the video to be processed is a static scene when the inter-frame motion intensity is less than the first motion intensity threshold; When the inter-frame motion intensity is greater than the first motion intensity threshold and less than the second motion intensity threshold, determining that the motion scene type of the to-be-processed video is a light motion scene; When the inter-frame motion intensity is greater than the second motion intensity threshold, it is determined that the motion scene type of the video to be processed is a violent motion scene.
[0075] Optionally, the strategy determination module 803 is specifically configured to determine that the video alignment strategy matching the video to be processed is non-alignment when the motion scene type of the video to be processed is a static scene; When the motion scene type of the video to be processed is a light motion scene, determining that the video alignment strategy matching the video to be processed is alignment between adjacent video frames; In the case where the motion scene type of the video to be processed is a violent motion scene, the video alignment strategy that matches the video to be processed is determined to be alignment between adjacent segments; the segments are used to represent at least one motion scene of the video to be processed.
[0076] In an embodiment of the present application, first, a feature extraction process is performed on the video to be processed through a video processing model to obtain feature information of the video to be processed; then, the motion scene category of the video to be processed is determined according to the feature information of the video to be processed through the video processing model; according to the motion scene category of the video to be processed, a video alignment strategy matching the video to be processed is determined; then, the video to be processed is aligned through the video processing model using the video alignment strategy, and the processed video to be processed is reconstructed to obtain a target video. It can be seen that through the embodiment of the present application, the motion scene category of the video to be processed can be determined according to the feature information of the video to be processed, and the video alignment strategy matching the motion scene category of the video to be processed can be determined according to the motion scene category, and then the video to be processed can be aligned based on the matching video processing strategy, and finally the reconstructed target video can be obtained. The video alignment method is no longer single, but dynamically selects the video alignment strategy according to the motion scene characteristics of the specific video to be processed, which improves the quality and efficiency of video alignment, and thus improves the video reconstruction effect.
[0077] The video processing device provided in the embodiment of the present application can implement each process in the embodiment corresponding to the above-mentioned video processing method. To avoid repetition, it will not be described again here.
[0078] It should be noted that the video processing device provided in the embodiment of the present application and the video processing method provided in the embodiment of the present application are based on the same inventive concept, so the specific implementation of this embodiment can refer to the implementation of the aforementioned video processing method, and the repeated parts will not be repeated.
[0079] Corresponding to the video processing method provided in the above embodiment, based on the same technical concept, the embodiment of the present application further provides an electronic device, which is used to execute the above video processing method. Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Fig. 9As shown, the electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors 901 and memory 902, and the memory 902 may store one or more storage applications or data. Among them, the memory 902 can be a temporary storage or a permanent storage. The application stored in the memory 902 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the electronic device. Furthermore, the processor 901 can be configured to communicate with the memory 902 to execute a series of computer executable instructions in the memory 902 on the electronic device. The electronic device may also include one or more power supplies 903, one or more wired or wireless network interfaces 904, one or more input and output interfaces 905, and one or more keyboards 906.
[0080] Specifically in this embodiment, the electronic device includes a processor, a communication interface, a memory and a communication bus; wherein the processor, the communication interface and the memory communicate with each other through the bus; the memory is used to store a computer program; the processor is used to execute the program stored in the memory to implement the following method steps: Perform feature extraction processing on the video to be processed through the video processing model to obtain feature information of the video to be processed; Determining the motion scene category of the video to be processed by the video processing model according to the feature information of the video to be processed; Determining a video alignment strategy that matches the video to be processed according to the motion scene category of the video to be processed; The video processing model is used to align the video to be processed using the video alignment strategy, and the processed video to be processed is reconstructed to obtain a target video.
[0081] In an embodiment of the present application, first, a feature extraction process is performed on the video to be processed through a video processing model to obtain feature information of the video to be processed; then, the motion scene category of the video to be processed is determined according to the feature information of the video to be processed through the video processing model; according to the motion scene category of the video to be processed, a video alignment strategy matching the video to be processed is determined; then, the video to be processed is aligned through the video processing model using the video alignment strategy, and the processed video to be processed is reconstructed to obtain a target video. It can be seen that through the embodiment of the present application, the motion scene category of the video to be processed can be determined according to the feature information of the video to be processed, and the video alignment strategy matching the motion scene category of the video to be processed can be determined according to the motion scene category, and then the video to be processed can be aligned based on the matching video processing strategy, and finally the reconstructed target video can be obtained. The video alignment method is no longer single, but dynamically selects the video alignment strategy according to the motion scene characteristics of the specific video to be processed, which improves the quality and efficiency of video alignment, and thus improves the video reconstruction effect.
[0082] The present application also provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the following method steps are implemented: Perform feature extraction processing on the video to be processed through the video processing model to obtain feature information of the video to be processed; Determining the motion scene category of the video to be processed by the video processing model according to the feature information of the video to be processed; Determining a video alignment strategy that matches the video to be processed according to the motion scene category of the video to be processed; The video processing model is used to align the video to be processed using the video alignment strategy, and the processed video to be processed is reconstructed to obtain a target video.
[0083] In an embodiment of the present application, first, a feature extraction process is performed on the video to be processed through a video processing model to obtain feature information of the video to be processed; then, the motion scene category of the video to be processed is determined according to the feature information of the video to be processed through the video processing model; according to the motion scene category of the video to be processed, a video alignment strategy matching the video to be processed is determined; then, the video to be processed is aligned through the video processing model using the video alignment strategy, and the processed video to be processed is reconstructed to obtain a target video. It can be seen that through the embodiment of the present application, the motion scene category of the video to be processed can be determined according to the feature information of the video to be processed, and the video alignment strategy matching the motion scene category of the video to be processed can be determined according to the motion scene category, and then the video to be processed can be aligned based on the matching video processing strategy, and finally the reconstructed target video can be obtained. The video alignment method is no longer single, but dynamically selects the video alignment strategy according to the motion scene characteristics of the specific video to be processed, which improves the quality and efficiency of video alignment, and thus improves the video reconstruction effect.
[0084] The embodiment of the present application also provides a computer program product. When the computer program is executed by a processor, each process of the above-mentioned video processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0085] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0086] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0087] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0088] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0089] In a typical configuration, an electronic device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0090] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.
[0091] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0092] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0093] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0094] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A video processing method, characterized in that: The method comprises: Perform feature extraction processing on the video to be processed through the video processing model to obtain feature information of the video to be processed; Determining the motion scene category of the video to be processed by the video processing model according to the feature information of the video to be processed; Determining a video alignment strategy that matches the video to be processed according to the motion scene category of the video to be processed; The video processing model is used to align the video to be processed using the video alignment strategy, and the processed video to be processed is reconstructed to obtain a target video.
2. The method according to claim 1, characterized in that The video processing model includes a feature extraction network; The video processing model is used to perform feature extraction processing on the video to be processed to obtain feature information of the video to be processed, including: Performing feature extraction processing on the video to be processed through the feature extraction network to obtain a feature map of the video to be processed; The feature extraction network is used to determine feature information of the video to be processed according to the feature graph of the video to be processed.
3. The method according to claim 1, characterized in that: Determining the motion scene category of the video to be processed by the video processing model according to the feature information of the video to be processed includes: Determining motion information of the video to be processed according to feature information of the video to be processed by the video processing model; The motion scene category of the video to be processed is determined according to the motion information of the video to be processed and a preset classification condition.
4. The method according to claim 3, characterized in that The video processing model includes a selector network; the step of determining the motion information of the video to be processed according to the feature information of the video to be processed by the video processing model includes: The feature information of the video to be processed is input into the selector network, and motion estimation processing is performed to obtain the motion information of the video to be processed.
5. The method according to claim 3, characterized in that: The motion information of the video to be processed includes the inter-frame motion intensity of the video to be processed; the preset classification condition includes a first motion intensity threshold and a second motion intensity threshold; The step of determining the motion scene category of the video to be processed according to the motion information of the video to be processed and a preset classification condition includes: When the inter-frame motion intensity is less than the first motion intensity threshold, determining that the motion scene type of the to-be-processed video is a static scene; When the inter-frame motion intensity is greater than the first motion intensity threshold and less than the second motion intensity threshold, determining that the motion scene type of the to-be-processed video is a light motion scene; When the inter-frame motion intensity is greater than the second motion intensity threshold, it is determined that the motion scene type of the video to be processed is a violent motion scene.
6. The method according to claim 1, characterized in that The step of determining a video alignment strategy that matches the video to be processed according to the motion scene category of the video to be processed includes: When the motion scene type of the video to be processed is a static scene, determining that the video alignment strategy matching the video to be processed is non-alignment; When the motion scene type of the video to be processed is a light motion scene, determining that the video alignment strategy matching the video to be processed is alignment between adjacent video frames; In the case where the motion scene type of the video to be processed is a violent motion scene, the video alignment strategy that matches the video to be processed is determined to be alignment between adjacent segments; the segments are used to represent at least one motion scene of the video to be processed.
7. A video processing device, characterized in that: The device comprises: A feature extraction module is used to perform feature extraction processing on the video to be processed through a video processing model to obtain feature information of the video to be processed; A scene determination module, used to determine the motion scene category of the video to be processed according to the feature information of the video to be processed by using the video processing model; A strategy determination module, used to determine a video alignment strategy that matches the video to be processed according to the motion scene category of the video to be processed; The video processing module is used to align the video to be processed by using the video processing model and the video alignment strategy, and reconstruct the processed video to be processed to obtain a target video.
8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus; wherein the processor, the communication interface and the memory communicate with each other through the bus; the memory is used to store computer programs; the processor is used to execute the programs stored in the memory to implement the steps of the video processing method as described in any one of claims 1-6.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video processing method according to any one of claims 1 to 6 are implemented.
10. A computer program product, characterized in that The method comprises a computer program, which, when executed by a processor, implements the steps of the video processing method according to any one of claims 1 to 6.