Video processing method and system

Abstract

The embodiment of the invention provides a video processing method and system. The method comprises the following steps: acquiring a to-be-enhanced video, and for each target video frame in the to-be-enhanced video, acquiring a plurality of continuous frames corresponding to the target video frame; performing feature extraction on the plurality of continuous frames to obtain an initial spatial-temporal feature corresponding to the target video frame, the initial spatial-temporal feature comprising a plurality of feature units; performing feature interaction on the plurality of feature units of the initial spatial-temporal feature to obtain a global spatial-temporal feature, the global spatial-temporal feature comprising a long-distance dependency relationship between the feature units; and generating an enhanced target video frame corresponding to each target video frame based on the initial spatial-temporal feature and the global spatial-temporal feature corresponding to each target video frame, so as to generate an enhanced target video based on each enhanced target video frame.

Description

Technical Field

[0001] This specification relates to the field of image processing technology, and in particular to a video processing method and system. Background Art

[0002] With the rapid development of Internet technology, the amount of online video data has shown explosive growth, and the demand for transmission bandwidth and storage resources for short videos, live broadcasts, high-definition movies and other content continues to rise.

[0003] In this context, video compression technology has become a key solution to the transmission and storage of massive amounts of data. By reducing redundant information in video data, it significantly reduces bandwidth usage. Related technologies utilize convolutional neural networks to enhance the quality of compressed video. For example, this approach enhances the quality of compressed video by stacking convolutional layers to gradually expand the receptive field. However, this approach is limited by its inherent local operations and struggles to efficiently model global correlations.

[0004] The content of the background technology section is merely information known to the inventor personally, and does not mean that the above information has entered the public domain before the application date of this disclosure, nor does it mean that it can become the prior art of the present disclosure. Summary of the Invention

[0005] This specification provides a video processing method and system that can efficiently establish long-distance dependencies between features without stacking a large number of levels to expand the receptive field, thereby achieving a global receptive field while maintaining low computational cost.

[0006] In a first aspect, the present specification provides a video processing method, wherein the method includes: obtaining a video to be enhanced, and for each target video frame in the video to be enhanced: obtaining multiple consecutive frames corresponding to the target video frame; performing feature extraction on the multiple consecutive frames to obtain initial spatiotemporal features corresponding to the target video frame, the initial spatiotemporal features including multiple feature units; performing feature interaction on the multiple feature units of the initial spatiotemporal features to obtain global spatiotemporal features, the global spatiotemporal features including long-distance dependencies between the feature units; and generating an enhanced target video frame corresponding to each target video frame based on the initial spatiotemporal features and the global spatiotemporal features, so as to generate an enhanced target video based on each enhanced target video frame.

[0007] In some embodiments, performing feature interaction on the multiple feature units of the initial spatiotemporal features includes: performing feature interaction on the multiple feature units of the initial spatiotemporal features through a preset feature arrangement strategy, wherein the preset feature arrangement strategy is used to transform the positions of pixels within the feature units.

[0008] In some embodiments, the feature interaction of the multiple feature units of the initial spatiotemporal feature through a preset feature arrangement strategy includes: for any current feature unit among the multiple feature units: based on the feature map size of the initial spatiotemporal feature and the first division size, determining the target feature unit corresponding to the current feature unit through a preset flow mapping method; and establishing the long-distance dependency relationship between the current feature unit and the target feature unit of the initial spatiotemporal feature through a multi-layer perceptron.

[0009] In some embodiments, the target feature unit corresponding to the current feature unit is determined by a preset stream mapping method based on the feature map size of the initial spatiotemporal feature and the first division size, including: based on the feature map size of the initial spatiotemporal feature and the first division size, a preset stream mapping method is used to perform feature transformation on multiple feature units in the initial spatiotemporal feature to obtain a transformed spatiotemporal feature; the transformed spatiotemporal feature includes multiple feature units; and for each current feature unit in the initial spatiotemporal feature: determining the corresponding target feature unit in the transformed spatiotemporal feature.

[0010] In some embodiments, based on the feature map size of the initial spatiotemporal features and the first division size, feature transformation is performed on multiple feature units in the initial spatiotemporal features through a preset stream mapping method to obtain the transformed spatiotemporal features, including: determining the target rows and target columns corresponding to each pixel point in the initial spatiotemporal features based on the feature map size of the initial spatiotemporal features and the first division size; and performing row transformation and column transformation on each pixel point in the initial spatiotemporal features, so as to transform each pixel point to the corresponding target row and target column through the row transformation and the column transformation to obtain the transformed spatiotemporal features.

[0011] In some embodiments, the continuous frames corresponding to the target video frame include: the target video frame, at least one forward video frame of the target video frame, and / or at least one backward video frame of the target video frame.

[0012] In some embodiments, the feature extraction of the multiple consecutive frames to obtain the initial spatiotemporal features corresponding to the target video frame includes: extracting features on the target video frame to obtain features of the target video frame; extracting features on at least one video frame forward of the target video frame in the multiple consecutive frames to obtain a first feature, and determining a forward alignment feature based on the first feature and the target video frame feature; extracting features on at least one video frame backward of the target video frame in the multiple consecutive frames to obtain a second feature, and determining a backward alignment feature based on the second feature and the target video frame feature; and obtaining the initial spatiotemporal features corresponding to the target video frame based on the forward alignment feature, the target video frame feature, and the backward alignment feature.

[0013] In some embodiments, determining the forward alignment feature based on the first feature and the target video frame feature includes: performing spatiotemporal alignment of the first feature with the target video frame feature to obtain the forward alignment feature; determining the backward alignment feature based on the second feature and the target video frame feature includes: performing spatiotemporal alignment of the second feature with the target video frame to obtain the backward alignment feature.

[0014] In some embodiments, obtaining the initial spatiotemporal features corresponding to the target video frame based on the forward alignment features, the target video frame features and the backward alignment features includes: determining the attention weights corresponding to the forward alignment features, the target video frame features and the backward alignment features; and performing weighted fusion of the forward alignment features, the target video frame features and the backward alignment features based on the attention weights to obtain the initial spatiotemporal features corresponding to the target video frame.

[0015] In some embodiments, determining the attention weights corresponding to the forward alignment feature, the target video frame feature, and the backward alignment feature includes: performing feature splicing on the forward alignment feature, the target video frame feature, and the backward alignment feature to obtain a spliced ​​feature; performing feature decomposition on the spliced ​​feature through a predetermined feature description operator to determine the attention weights corresponding to the forward alignment feature, the target video frame feature, and the backward alignment feature.

[0016] In some embodiments, the method further includes: dividing the initial spatiotemporal features based on a first division size to obtain a plurality of first feature units after division; each of the plurality of first feature units contains spatiotemporal features corresponding to a first preset number of pixels.

[0017] In some embodiments, the generating of an enhanced target video frame corresponding to each target video frame based on the initial spatiotemporal features and the global spatiotemporal features corresponding to each target video frame includes: extracting local spatiotemporal features of the initial spatiotemporal features corresponding to each target video frame; fusing the local spatiotemporal features and the global spatiotemporal features corresponding to each target video frame to obtain fused spatiotemporal features; and generating an enhanced target video frame corresponding to each target video frame based on the initial spatiotemporal features and the fused spatiotemporal features corresponding to each target video frame.

[0018] In some embodiments, the extracting of local spatiotemporal features of the initial spatiotemporal features corresponding to each of the target video frames includes: for each of the target video frames: dividing the initial spatiotemporal features corresponding to the target video frame based on a second division size to obtain a plurality of second feature units after division; each second feature unit contains spatiotemporal features corresponding to a second preset number of pixels; performing feature extraction on each of the second feature units to obtain the local spatiotemporal features of each of the second feature units, and the features of each of the second feature units constitute the local spatiotemporal features of the target video frame.

[0019] In some embodiments, the generating of an enhanced target video frame corresponding to each target video frame based on the initial spatiotemporal features and the fused spatiotemporal features corresponding to each target video frame includes: determining the spatiotemporal feature residuals based on the initial spatiotemporal features and the fused spatiotemporal features; summing the initial spatiotemporal features and the spatiotemporal feature residuals to determine the target spatiotemporal features; and generating an enhanced target video frame corresponding to each target video frame based on the target spatiotemporal features.

[0020] In a second aspect, the present specification also provides a video processing system comprising at least one storage medium and at least one processor, wherein the at least one storage medium stores at least one instruction set for performing video processing; the at least one processor is communicatively connected to the at least one storage medium, wherein the at least one processor reads the at least one instruction set during operation and executes any one of the methods described in the first aspect above according to the instructions of the at least one instruction set.

[0021] As can be seen from the above technical solutions, the video processing method and system provided in this specification, after the video processing system obtains the video to be enhanced, for each target video frame in the video to be enhanced: the video processing system obtains the corresponding multiple continuous frames to extract features from the multiple continuous frames to obtain the initial spatiotemporal features corresponding to the target video frame. This processing method processes each target video frame based on the multiple continuous frames corresponding to each target video frame. It can utilize the correlation between adjacent frames to better retain the original information of the video to be enhanced, avoid the loss of key information caused by processing only the target video frame, and lay the foundation for subsequent more accurate processing of the video to be enhanced. Subsequently, the video processing system performs feature interaction on multiple feature units in the initial spatiotemporal features to explore the long-distance dependency between feature units, thereby obtaining global spatiotemporal features. The above-mentioned method of performing feature interaction on multiple feature units corresponding to each target video frame enables effective information exchange and fusion between feature units at different positions, and better associates the feature information between different feature units. The acquisition of the above-mentioned global spatiotemporal features can enable the video processing system to better understand the overall content and structure of the video to be processed, help reduce the problem of blocking effects caused by local processing, and significantly improve the overall coherence and quality of the processed target video. Furthermore, the video processing system generates an enhanced target video frame corresponding to the target video frame based on the initial spatiotemporal features and the global spatiotemporal features corresponding to each target video frame, so as to generate an enhanced target video based on each enhanced target video frame. The above-mentioned generation of enhanced target video frames based on the initial spatiotemporal features and the global spatiotemporal features integrates local and global information and can enhance the video frames more comprehensively. The initial spatiotemporal features retain the basic features and details of the video frame, while the global spatiotemporal features provide a wider receptive field and deeper information fusion. The enhanced video frames generated in the above-mentioned manner can ensure the quality of the generated target video, thereby improving the user's visual experience.

[0022] Other functions of the video processing method and system provided in this specification will be partially listed in the following description. The creative aspects of the video processing method and system provided in this specification can be fully explained by practicing or using the methods, devices and combinations described in the following detailed examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1A schematic diagram of an application scenario of a video processing system provided according to an embodiment of this specification is shown;

[0025] Figure 2 A schematic diagram of the hardware structure of a computing device provided according to some embodiments of this specification is shown;

[0026] Figure 3 A schematic flow chart of a video processing method according to an embodiment of the present disclosure is shown;

[0027] Figure 4 A schematic diagram of a feature interaction process according to an embodiment of this specification is shown;

[0028] Figure 5 A schematic flow chart showing a video processing method according to another embodiment is shown; and

[0029] Figure 6 A schematic structural diagram of a spatiotemporal feature flow module provided in one embodiment of this specification is shown. DETAILED DESCRIPTION

[0030] The following description provides specific application scenarios and requirements for this specification, with the goal of enabling those skilled in the art to make and use the contents of this specification. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but is intended to be accorded the broadest scope consistent with the claims.

[0031] The terms used herein are for the purpose of describing specific example embodiments only and are not intended to be limiting. For example, as used herein, the singular forms "a," "an," and "the" may also include the plural forms unless the context clearly indicates otherwise. When used in this specification, the terms "comprise," "include," and / or "contain" are intended to refer to the presence of the associated integers, steps, operations, elements, and / or components, but do not preclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups or the addition of other features, integers, steps, operations, elements, components, and / or groups in the system / method.

[0032] These and other features of this specification, as well as the operation and function of the associated elements of the structure, and the economical assembly and manufacture of the components, can be significantly improved with consideration of the following description. Reference is made to the accompanying drawings, all of which form a part of this specification. However, it should be expressly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0033] The flowcharts used in this specification illustrate operations implemented by systems according to some embodiments of the present specification. It should be clearly understood that the operations of the flowcharts may not be implemented in sequence. Rather, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0034] In this specification, "X includes at least one of A, B, or C" means that X includes at least A, or X includes at least B, or X includes at least C. In other words, X may include only any one of A, B, and C, or any combination of A, B, and C, as well as other possible contents / elements. Any combination of A, B, and C may be A, B, C, AB, AC, BC, or ABC.

[0035] In this specification, unless otherwise specified, the association relationship between structures can be a direct association relationship or an indirect association relationship. For example, when describing "A is connected to B", unless it is clearly stated that A is directly connected to B, it should be understood that A can be directly connected to B or indirectly connected to B; for another example, when describing "A is above B", unless it is clearly stated that A is directly above B (AB are adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements and A is above B). And so on.

[0036] It should be noted that the user data obtained in this manual is authorized by the user and does not involve user privacy.

[0037] For the convenience of description, the terms that will appear in the following text of this specification are first explained.

[0038] Spatiotemporal alignment: A technique or method that coordinates and calibrates the correlation of data in spatial (location, layout) and temporal (dynamic change, sequence) dimensions to eliminate multi-dimensional inconsistencies and maintain structural continuity.

[0039] Dynamic-guided Deformable Alignment (DDA) module: A deep learning component for video processing. Its core uses a dynamically generated offset field to guide a deformable convolution operation, enabling adaptive alignment of features between adjacent frames. Compared to traditional optical flow alignment, the DDA module significantly improves motion compensation accuracy (especially for scenes with fast motion or motion blur). By using a dynamic guidance mechanism, it avoids the computational overhead of explicit motion estimation, providing efficient motion-aware feature alignment for tasks such as video super-resolution and frame interpolation.

[0040] The Spatial-Temporal Feature Flow (SFF) module is a neural network component that efficiently integrates spatiotemporal information by dynamically modeling the evolution of features across frames in video sequences. Its core concept is "feature flow," integrating motion compensation, feature alignment, and adaptive aggregation between consecutive frames into an end-to-end learnable process. SFF eliminates artifacts caused by motion shifts (such as ghosting and blurring) while preserving the spatiotemporal continuity of high-frequency details. Ultimately, it achieves the coordinated optimization of spatial detail enhancement and temporal consistency in tasks such as video super-resolution and denoising.

[0041] Benign Selective Compensation (BSC) module: An intelligent video enhancement component that uses a dual-channel mechanism to achieve precise feature restoration. This mechanism avoids over-processing of undamaged areas by traditional methods while also performing localized enhancements to repair compression artifacts. This improves video quality while maintaining the naturalness of the original content and significantly reduces the risk of secondary distortion introduced by restoration algorithms.

[0042] Multi-Layer Perceptron (MLP) is a feedforward neural network consisting of an input layer, at least one hidden layer, and an output layer. It learns complex data features through a fully connected structure and nonlinear activation functions (such as ReLU and Sigmoid) and can solve linearly inseparable problems (such as XOR classification).

[0043] Long-distance dependencies refer to the semantic associations or structural dependencies between distant pixel regions or features in image / video frames. Modeling such dependencies allows the algorithm to capture and utilize information across large spatial intervals to understand and optimize the global structure of the image, thereby maintaining or restoring the overall semantic rationality and visual coherence of the image during the enhancement process.

[0044] The method provided in this specification can be applied to scenarios where video optimization is required, such as enhancing videos compressed by traditional coding, enhancing historical images, enhancing security videos, etc. A specific application method is that when the video to be enhanced needs to be enhanced in the above scenarios, the video processing system can decode the video to be enhanced to obtain multiple video frames, and obtain the global spatiotemporal features (including the long-distance dependencies between each feature) and the initial spatiotemporal features corresponding to each video frame in the multiple video frames. Furthermore, the video processing system enhances each video frame based on the global spatiotemporal features and the initial spatiotemporal features of each video frame in the multiple video frames to obtain each enhanced video frame and form an enhanced target video and output it.

[0045] In the above process, video enhancement bears the heavy responsibility of video quality assurance. It can effectively extract the spatiotemporal information of each video frame in the enhanced video and optimize the feature space representation. It can achieve high-quality video reconstruction through collaborative processing between feature spaces, and can effectively solve the problem of inter-frame motion discontinuity caused by video compression, thereby improving the quality of the video viewed by the human eye and enhancing the user's visual experience of watching the video.

[0046] The video processing method provided in this specification is that after the video processing system obtains multiple video frames corresponding to the video to be enhanced, it can respectively obtain multiple continuous frames corresponding to each target video frame. For each target video frame: the video processing system obtains the initial spatiotemporal features corresponding to the target video frame based on the multiple continuous frames corresponding to the target video frame, and obtains the global spatiotemporal features corresponding to the target video frame by performing feature interaction on multiple feature units in the initial spatiotemporal features. Furthermore, the video processing system generates an enhanced target video frame based on the initial spatiotemporal features and the global spatiotemporal features corresponding to the target video frame, so as to obtain an enhanced target video based on the multiple enhanced target video frames. The above method enhances each video frame by reconstructing the spatiotemporal features, which solves the problem in the related art that the video quality of the compressed video cannot be guaranteed, thereby improving the quality of the video.

[0047] It should be noted that the above description of the application scenario is only one of the multiple usage scenarios provided in this specification. Those skilled in the art should understand that when the video processing method and system provided in this specification are applied to other usage scenarios, their implementation methods and technical effects are similar.

[0048] Figure 1 A schematic diagram of an application scenario 100 of a video processing system 130 provided according to an embodiment of this specification is shown.

[0049] like Figure 1 As shown, after the video processing system 130 obtains the video to be enhanced, it can use the video processing method provided in this specification to enhance each video frame in the video to be enhanced to obtain an enhanced target video corresponding to the video to be enhanced.

[0050] Specifically, after acquiring the video to be enhanced, the video processing system 130 can decode the video to be enhanced to obtain multiple video frames. The video processing system 130 uses the above-mentioned video processing method to sequentially acquire multiple consecutive frames corresponding to each target video frame in chronological order, and enhances each target video based on the multiple consecutive frames to obtain an enhanced target video frame, so as to generate an enhanced target video based on each enhanced target video frame. In the above-mentioned enhancement method, during the enhancement process of the target video frame, the video processing system 130 determines the initial spatiotemporal features corresponding to the target video based on the multiple consecutive frames corresponding to the target video frame, and performs feature interaction on multiple feature units in the initial spatiotemporal features to obtain global spatiotemporal features. Furthermore, the video processing system 130 generates an enhanced target video frame corresponding to each target video frame based on the initial spatiotemporal features and the global spatiotemporal features corresponding to each target video frame to obtain an enhanced target video. In the above-mentioned video processing method, since the target video frame is generated by the video processing system 130 based on the initial spatiotemporal features and the global spatiotemporal features, the global spatiotemporal features and local features of the target video frame are integrated, so that the video frame can be enhanced more comprehensively, thereby ensuring the video quality of the generated enhanced target video.

[0051] The video processing system 130 can be a computing system with certain computing capabilities. The video processing system 130 can execute the image processing method or video processing method provided in this specification. The video processing system 130 can store data or instructions for executing the image processing method or video processing method described in this specification, and can execute or be used to execute the data or instructions. The video processing system 130 can include hardware devices with data information processing functions and the necessary programs required to drive the operation of the hardware devices. The video processing system 130 can be a single computing device or a cluster system composed of multiple computing devices, which is not limited in this specification.

[0052] The video processing system 130 may be a single computing device or a cluster system composed of multiple computing devices, which is not limited in this specification.

[0053] It should be noted that the user data obtained in this manual has been authorized by the user and does not involve user privacy.

[0054] Figure 2 The hardware structure diagram of a computing device 200 provided according to some embodiments of this specification is shown. The computing device 200 can be used as Figure 1 In some embodiments, when the video processing system 130 adopts a device cluster, the computing device 200 can serve as any device in the video processing system 130.

[0055] like Figure 2 As shown, computing device 200 includes at least one storage medium 230 and at least one processor 220. In some embodiments, computing device 200 may further include an internal communication bus 210. In some embodiments, computing device 200 may further include a communication port 250. In some embodiments, computing device 200 may further include an I / O component 260.

[0056] Internal communication bus 210 may connect various system components, including storage media 230 and processor 220. I / O components 260 support input / output between computing device 200 and other components.

[0057] The communication port 250 is used for data communication between the computing device 200 and the outside world. For example, the computing device 200 can be connected to a network through the communication port 250.

[0058] The storage medium 230 may include a data storage device. The data storage device may be a non-transitory storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 232, a read-only storage medium (ROM) 234, or a random access storage medium (RAM) 236. The storage medium 230 also includes at least one instruction set stored in the data storage device. The instruction set is computer program code, which may include programs, routines, objects, components, data structures, processes, modules, etc. that execute the video processing method provided in this specification.

[0059] At least one processor 220 is communicatively coupled to at least one storage medium 230 via an internal communication bus 210. The at least one processor 220 is configured to execute the at least one instruction set described above. When the system 130 is running, the at least one processor 220 reads the at least one instruction set and executes the video processing method provided herein according to the instructions of the at least one instruction set.

[0060] The processor 220 can execute all steps included in the video processing method. The processor 220 can be in the form of one or more processors. The processor 220 can issue execution instructions. The processor 220 can include one or more hardware processors, such as a microcontroller, a microprocessor, a reduced instruction set computer (RISC), an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a microcontroller unit, a digital signal processor (DSP), a field programmable gate array (FPGA), an advanced RISC machine (ARM), a programmable logic device (PLD), any circuit or processor capable of performing one or more functions, etc., or any combination thereof.

[0061] For illustrative purposes only, only one processor 220 is shown in the computing device 200 in the drawings in this specification. However, it should be noted that the computing device 200 in this specification may also include multiple processors. Therefore, the operations and / or method steps disclosed in this specification may be performed by a single processor as described herein, or may be performed jointly by multiple processors. For example, if the processor 220 of the computing device 200 in this specification performs steps A and B, it should be understood that steps A and B may also be performed jointly or separately by two different processors 220 (e.g., the first processor performs step A, the second processor performs step B, or the first and second processors perform steps A and B together).

[0062] Figure 3 FIG. 1 is a flow chart of a video processing method according to an embodiment of the present specification; the video processing method P300 may be executed by the system 130. Figure 3 As shown, the method P300 provided in this specification may include S310-S370, wherein:

[0063] S310: Acquire a video to be enhanced, and for each target video frame in the video to be enhanced: acquire multiple consecutive frames corresponding to the target video frame.

[0064] In some embodiments, the continuous frames corresponding to the target video frame include: the target video frame, at least one forward video frame of the target video frame, and / or at least one backward video frame of the target video frame. From a temporal perspective, in a video sequence, the target video frame, the forward video frame of the target video frame, and the backward video frame represent states at different points in time. The target video frame is the frame of interest at the current moment, the forward video frame is the frame before the current moment, and the backward video frame is the frame after the current moment. The number of the at least one forward video frame is the same as the number of the at least one backward video frame. For example, the number of forward video frames and the number of backward video frames can both be three. That is, the continuous frames corresponding to the target video frame include seven video frames: three forward video frames, the target video frame, and three backward video frames. It should be understood that the above embodiment is merely illustrative, and the number of video frames included in the specific continuous video frame can be flexibly adjusted according to user needs and is not limited to that given in the above embodiment. It only needs to be a positive integer greater than or equal to 2.

[0065] When the target video frame is the first frame of the video to be enhanced, since there are no other forward video frames before the target video frame, the continuous frames corresponding to the target video frame can only include a preset number of backward video frames backward of the target video frame. Similarly, when the target video frame is the last frame of the video to be enhanced, since there are no other backward video frames after the target video frame, the continuous frames corresponding to the target video frame can only include a preset number of forward video frames forward of the target video frame. It should be understood that the above embodiment is merely illustrative, and the specific method for determining the continuous frames corresponding to the target video frame can be flexibly adjusted according to actual conditions and is not limited to the above embodiment.

[0066] S330: Perform feature extraction on multiple consecutive frames to obtain initial spatiotemporal features corresponding to the target video frame, where the initial spatiotemporal features include multiple feature units.

[0067] In some embodiments, the video processing system divides the initial spatiotemporal features based on a first division size to obtain a plurality of first feature units; each of the plurality of first feature units contains spatiotemporal features corresponding to a first preset number of pixels. The first division size is used to limit the number of pixels contained in each feature unit.

[0068] In some embodiments, the video processing system may perform feature extraction on the target video frame to obtain target video frame features. The video processing system may perform feature extraction on at least one video frame preceding the target video frame among multiple consecutive frames to obtain a first feature, and determine a forward alignment feature based on the first feature and the target video frame features. The video processing system may perform feature extraction on at least one video frame following the target video frame among multiple consecutive frames to obtain a second feature, and determine a backward alignment feature based on the second feature and the target video frame features. Furthermore, the video processing system may obtain initial spatiotemporal features corresponding to the target video frame based on the forward alignment feature, the target video frame features, and the backward alignment feature.

[0069] Among them, the video processing system effectively integrates the motion correlation in the temporal dimension and the consistency of spatial features by extracting the features of the target video frame and its forward and backward video frames and generating bidirectional alignment features (forward alignment features and backward alignment features).

[0070] The video processing system performs spatiotemporal alignment on the first feature to the target video frame feature to obtain a forward alignment feature, and performs spatiotemporal alignment on the second feature to the target video frame to obtain a backward alignment feature.

[0071] The video processing system first extracts the target video frame features as a benchmark, then extracts the first features from the forward video frame, and determines the forward alignment features based on the first features and the target video frame features. The second features are extracted from the backward video frame, and the backward alignment features are determined based on the second features and the target video frame features, so that the target video frame features can simultaneously capture the complementary information of the object displacement trajectory and the occlusion area between adjacent frames. The video processing system constructs an initial spatiotemporal feature containing the spatiotemporal correlation between multiple frames by three-dimensionally fusing the forward alignment features, the target video frame features, and the backward alignment features. The initial spatiotemporal feature not only retains the local details of the target video frame, but also embeds the motion compensation information of adjacent frames. Therefore, during the process of enhancing the target video frame by the video processing system, it can accurately correct artifacts such as motion blur and block dislocation caused by compression, and at the same time, use the redundancy of cross-frame features to restore the high-frequency texture lost in a single frame.

[0072] In some embodiments, the video processing system can determine the attention weights corresponding to the forward alignment features, target video frame features and backward alignment features, and then the video processing system performs weighted fusion on the forward alignment features, target video frame features and backward alignment features based on the attention weights to obtain the initial spatiotemporal features corresponding to the target video frame.

[0073] In the above embodiment, by introducing the attention weight mechanism to dynamically weight and fuse the forward alignment features, target video frame features and backward alignment features, the accuracy and adaptability of the initial spatiotemporal features corresponding to the target video frame are significantly improved. Compared with the traditional convolution operation that only relies on local information processing with a fixed receptive field, the above embodiment adaptively captures the correlation strength between cross-frame features through attention weights, so that the feature fusion process can focus on key areas in the video sequence that are highly correlated with the target video frame. This dynamic weight allocation mechanism not only effectively solves the technical defect that stacked convolution layers are difficult to model long-distance spatiotemporal dependencies, but also significantly improves the optimization capability of global motion consistency and detail recovery in the process of compressed video quality enhancement by strengthening the targeted interaction of cross-frame features, ultimately achieving a comprehensive improvement in the temporal coherence, spatial clarity and visual perception quality of the enhanced video.

[0074] The video processing system may perform feature concatenation on the forward alignment features, the target video frame features, and the backward alignment features to obtain concatenated features. Furthermore, the video processing system may perform feature decomposition on the concatenated features using a predetermined feature description operator to determine attention weights corresponding to the forward alignment features, the target video frame features, and the backward alignment features.

[0075] The video processing system combines feature splicing with predefined feature description operators to achieve refined modeling of cross-frame feature weights. Specifically, forward-aligned features, target video frame features, and backward-aligned features are spliced ​​together to construct a composite feature representation encompassing the spatiotemporal context of multiple frames. The spliced ​​features are then decomposed using predefined feature description operators to derive attention weights for each feature branch. This design not only fully leverages the information integrity of the joint distribution across multiple frames but also enhances the rationality of weight allocation through the mathematical interpretability of feature decomposition. Compared to fusion methods that directly employ fixed rules or simple averaging, this scheme dynamically adjusts the contribution of features from different frames to the target video frame based on video content. For example, it enhances the compensatory effect of adjacent frames in areas of intense motion while prioritizing the preservation of the target video frame's own features in static areas. This feature decomposition-based attention weight generation mechanism effectively overcomes the fusion bias caused by traditional methods that ignore nonlinear correlations between features. It significantly improves the ability to suppress spatiotemporal redundant information and the accuracy of detail reconstruction during compressed video enhancement, thereby reducing artifacts while ensuring the temporal smoothness and spatial structural integrity of the enhanced video.

[0076] Specifically, we use the target video frame feature A1, the forward-aligned feature A2, and the backward-aligned feature A3 as an example. When determining the initial spatiotemporal features based on A1, A2, and A3, A1, A2, and A3 are first concatenated along the channel dimension (total number of channels: 3n) to form a multi-frame joint feature representation, representing the potential correlation between cross-frame features. Subsequently, the video processing system uses a convolutional layer to reduce the number of channels from 3n to n, eliminating redundant information that may be introduced by concatenation while extracting more discriminative cross-frame feature representations, resulting in refined features. Furthermore, the video processing system spatially compresses the refined features to generate a channel-level statistic (cws, dimensionality 1×1×C). This statistic reflects the average response strength of each channel in the global spatiotemporal domain and implicitly captures the importance distribution of cross-frame features. The video processing system compresses cws into a low-dimensional vector z (dimension r, r = C / 8) through convolution, achieving dimensionality reduction and nonlinear abstraction of features while preserving the interaction information of key channels. The dimension of z is restored to the original number of channels (C) through three parallel convolutional upsampling layers, and three predetermined feature description operators v1, v2, and v3 are obtained. v1, v2, and v3 correspond to the potential weight templates of A1, A2, and A3, respectively. Subsequently, the video processing system normalizes v1, v2, and v3 to generate attention weight values ​​s1, s2, and s3 corresponding to A1, A2, and A3, respectively (the sum is 1), thereby achieving normalization and interpretability of the weight distribution. The final output is the initial spatiotemporal features of the target video frame: s1*A1+s2*A2+s3*A3. In the above method, through the closed-loop process of multi-frame feature splicing - channel compression - attention weight generation - dynamic weighted fusion, efficient suppression of spatiotemporal redundant information in the enhanced video and accurate restoration of key details are achieved, providing a scalable and lightweight solution for video quality enhancement.

[0077] S350: Perform feature interaction on multiple feature units of the initial spatiotemporal feature to obtain a global spatiotemporal feature, where the global spatiotemporal feature includes long-distance dependencies between the feature units.

[0078] In some embodiments, the video processing system may perform feature interaction on multiple feature units of the initial spatiotemporal features through a preset feature arrangement strategy, where the preset feature arrangement strategy is used to transform the positions of pixels within the feature units.

[0079] The preset feature arrangement strategy can be one or more. The video processing system dynamically transforms the pixel positions within multiple feature units in the initial spatiotemporal features using the preset feature arrangement strategy, thereby breaking the limitations of the fixed spatial arrangement on the range of feature interaction and indirectly achieving cross-regional expansion of the receptive field. Under the traditional fixed arrangement, the interaction between features is limited to the physical neighborhood relationship of pixels. By transforming the positions of pixels within the feature unit, originally scattered highly correlated pixels (such as continuous texture areas separated by compression artifacts) are reorganized into adjacent positions, allowing a single convolution or attention operation to capture a wider range of contextual dependencies. This implicit receptive field expansion mechanism allows the video processing system to fuse cross-regional semantic information (such as edge continuity features across blocking effects) within local feature units without increasing the network depth or convolution kernel size, thereby accurately reconstructing long-range spatial structures destroyed by compression (such as building outlines and linear textures in natural scenes), thereby significantly improving the consistency of repairing large-scale artifacts during video enhancement and improving the accuracy and visual quality of video enhancement.

[0080] In some embodiments, for any current feature unit among a plurality of feature units: the video processing system determines the target feature unit corresponding to the current feature unit through a preset stream mapping method based on the feature map size of the initial spatiotemporal feature and the first partition size. Furthermore, the video processing system establishes a long-range dependency relationship between the current feature unit and the target feature unit of the initial spatiotemporal feature through a multi-layer perceptron (MLP). In the embodiments of this specification, the MLP may select a spatially guided multi-layer perceptron (SG-MLP), which is more suitable for processing visual tasks.

[0081] In the above method, the video processing system dynamically calculates the flow mapping path based on the feature map size and the feature unit size (such as pairing feature units that are far apart according to motion trajectories or semantic associations), so that the video processing system can directly establish a directional connection between the current feature unit and the more distant target feature unit through a multi-layer perceptron. Compared with the cross-regional associations that require multi-layer convolution stacking to capture in related technologies, the above method of first transforming the position of the feature unit and then establishing long-distance dependencies through MLP can achieve efficient modeling in a single-layer operation. The above-mentioned method of constructing long-distance dependencies can not only avoid the computational redundancy of the global attention mechanism, but also achieve the coordinated optimization of local repair and global structural reconstruction, significantly improving the spatiotemporal coherence of video enhancement in complex motion scenes.

[0082] In some embodiments, the video processing system may perform feature transformation on multiple feature units in the initial spatiotemporal features using a preset stream mapping method based on the feature map size and the first partition size of the initial spatiotemporal features to obtain a transformed spatiotemporal feature; the transformed spatiotemporal feature includes multiple feature units. Furthermore, for each current feature unit in the initial spatiotemporal features, the video processing system determines a corresponding target feature unit in the transformed spatiotemporal features.

[0083] Among them, the position of the current feature unit in the initial spatiotemporal feature is the same as the position of the target feature unit in the transformed spatiotemporal feature, but the pixels contained in the current feature unit are different from the pixels contained in the target feature unit, and thus the spatiotemporal feature corresponding to the current feature unit is different from the spatiotemporal feature corresponding to the target feature unit.

[0084] In some embodiments, the video processing system determines the target row and target column corresponding to each pixel in the initial spatiotemporal features based on the feature map size of the initial spatiotemporal features and the first partition size. Furthermore, the video processing system performs a row transformation and a column transformation on each pixel in the initial spatiotemporal features to transform each pixel to the corresponding target row and target column through the row transformation and column transformation, thereby obtaining a transformed spatiotemporal feature.

[0085] For example, the video processing system determines the target row and target column corresponding to each pixel point respectively through a preset stream mapping method (m(i)=(p+n) / 2+(i-1)) to determine the target feature unit corresponding to each current feature unit. Wherein, i is the target column / row to be exchanged, n is the size of the feature map, and p is the block size when the feature map is blocked. The video processing system can be based on the preset stream mapping method. Subsequently, the video processing system establishes a direct connection between the current feature unit and the target feature unit based on a multi-layer perceptron, so that instant connection can be established between spatially separated features.

[0086] Figure 4 A schematic diagram of a feature interaction process according to an embodiment of this specification is shown. Figure 4As shown, after obtaining the feature map of the initial spatiotemporal features corresponding to the target video frame, the video processing system divides the initial spatiotemporal feature map based on the first division size (block size) to obtain an initial spatiotemporal feature map containing multiple feature units. Subsequently, the video processing system transforms the feature map of the initial spatiotemporal features by determining the target row and target column corresponding to each pixel point. The transformation method can be, for example: when it is currently necessary to find the exchange target column corresponding to the first column of pixels (i.e., i=1), the size of the feature map of the current initial spatiotemporal features is 6×6 (i.e., n=6), and the block size is 2×2 (i.e., p=2). Through the preset stream mapping method, it can be determined that the target column corresponding to the first column of pixels is the fourth column. Similarly, it can also be determined that the target column corresponding to the second column of pixels in the above case is the sixth column, and the target column corresponding to the third column of pixels is the sixth column. Similarly, through the preset stream mapping method, it can be determined that the target line corresponding to the first row of pixels is the fourth row. Similarly, it can also be determined that the target line corresponding to the second row of pixels in the above case is the sixth row, and the target line corresponding to the third row of pixels is the sixth row. Then, based on the above determination, the video processing system transforms the position of each pixel to obtain a transformed feature map. In the transformed feature map, for example, the pixels of four vertices may be transformed into the same feature unit (this is only an example, and the specific transformation of each pixel depends on the actual situation).

[0087] After obtaining the transformed feature map, the video processing system can establish a long-distance dependency relationship between different feature units (current feature unit and target feature unit) through a multi-layer perceptron, thereby significantly increasing the range of the receptive field. After establishing the long-distance dependency relationship between different feature units, the video processing system restores the transformed feature map and restores the position of each pixel to a position consistent with that in the feature map of the initial spatiotemporal feature. Furthermore, the video processing system merges the restored feature map with the feature map corresponding to the initial spatiotemporal feature to obtain a merged spatiotemporal feature. Each feature unit in the merged spatiotemporal feature contains the initial spatiotemporal feature and the global spatiotemporal feature corresponding to the feature unit.

[0088] The method provided by the above embodiment can enable pixels originally scattered in different feature units to be relocated to adjacent positions through row transformation and column transformation, and through the multi-layer perceptron, pixels with distant spatial distances can be directly connected without relying on multi-layer network stacking or global attention. By performing row transformation and column transformation on each pixel point in the initial spatiotemporal feature, the scattered pixels can be rearranged to the target rows and columns through a preset flow mapping method, and the association relationship between the units of multiple features before and after the transformation can be established through the multi-layer perceptron, so that separated features that have no adjacent or associated relationship in space can achieve efficient cross-regional association. Through the above-mentioned feature interaction method, long-distance information can be relocated to the local receptive field, so that the features corresponding to each pixel can be associated with the features corresponding to distant pixels, thereby expanding the actual receptive field.

[0089] S370: Based on the initial spatiotemporal features and the global spatiotemporal features corresponding to each target video frame, an enhanced target video frame corresponding to each target video frame is generated, so as to generate an enhanced target video based on each enhanced target video frame.

[0090] The video processing system combines initial spatiotemporal features containing local details with global spatiotemporal features, and can simultaneously impose local consistency constraints and global semantic guidance during pixel-level restoration. Local features provide the gradient information required for high-frequency detail reconstruction (such as hair texture and edge sharpening), while global spatiotemporal features ensure that the restored area conforms to the overall spatiotemporal logic of the video (such as the continuity of the path of moving objects). This dual-stream feature fusion mechanism can effectively suppress the global mismatch problem caused by local restoration in traditional methods, allowing the quality of the enhanced target video to be optimized, achieving a dual breakthrough in visual quality and temporal coherence.

[0091] In some embodiments, the video processing system extracts local spatiotemporal features from the initial spatiotemporal features corresponding to each target video frame. Subsequently, the video processing system fuses the local spatiotemporal features corresponding to each target video frame with the global spatiotemporal features to obtain fused spatiotemporal features. Furthermore, the video processing system generates an enhanced target video frame corresponding to each target video frame based on the initial spatiotemporal features and the fused spatiotemporal features corresponding to each target video frame.

[0092] Among them, the video processing system extracts local spatiotemporal features that focus on the motion details and texture changes in the vicinity of the target video frame from the initial spatiotemporal features, and fuses the local spatiotemporal features with the global spatiotemporal features to construct a bidirectional enhancement channel for fine-grained and coarse-grained features. Based on the local spatiotemporal features, it can accurately capture the restoration clues of high-frequency details (such as edge sharpness, small object structure, etc.), and based on the global spatiotemporal features, it can inject spatiotemporal consistency constraints into the restoration process (such as the coherence of the object's motion trajectory, the integrity of the background structure, etc.). Finally, based on the initial spatiotemporal features corresponding to each target video frame and the fused spatiotemporal features, the video processing system forms an enhanced target video frame that retains local restoration sensitivity while avoiding global information distortion. That is, the mechanism of fusing global spatiotemporal features with local spatiotemporal features in the above method can not only use global spatiotemporal features to guide local regions to perform semantically reasonable repair (such as reconstructing textures destroyed by blocking effects by referring to undamaged areas at the far end), but also enhance the authenticity of spatial details through local spatiotemporal features. Through the dynamic complementarity of local features and global spatiotemporal features, the inherent limitations of the processing process caused by single-scale features are effectively overcome, and the effect of video enhancement is improved.

[0093] The video processing system determines the spatiotemporal feature residuals based on the initial spatiotemporal features and the fused spatiotemporal features, and sums the initial spatiotemporal features and the spatiotemporal feature residuals to determine the target spatiotemporal features. Furthermore, the video processing system generates an enhanced target video frame corresponding to each target video frame based on the target spatiotemporal features.

[0094] In the above embodiment, the video processing system determines the spatiotemporal feature residual by calculating the difference between the fused feature and the original initial spatiotemporal feature, and enables the video processing system to focus the repair process on the compression artifacts and detail difference areas represented by the spatiotemporal feature residual while retaining the effective information of the original target video frame (such as intact texture and color). The target spatiotemporal feature is determined based on the sum of the initial spatiotemporal feature and the spatiotemporal feature residual, so that the video processing system not only utilizes the repair ability of the fused feature to perform targeted correction on the distorted area (such as eliminating block effects, restoring high-frequency edges, etc.), but also avoids information loss or over-smoothing problems caused by directly modifying the initial spatiotemporal feature through the "incremental correction" characteristic of the residual. Thus, in the process of video enhancement, the precise synergy between "local correction + global preservation" is achieved, which not only fully utilizes the repair ability of the fused feature, but also anchors the basic authenticity of the video content through the original initial spatiotemporal feature, and ultimately generates an enhanced target video frame with high visual quality and strong spatiotemporal consistency.

[0095] In some embodiments, the video processing system divides the initial spatiotemporal features corresponding to each target video frame based on a second division size to obtain a plurality of second feature units after division; each second feature unit contains spatiotemporal features corresponding to a second preset number of pixels; and performs feature extraction on each second feature unit to obtain local spatiotemporal features of each second feature unit, wherein the features of each second feature unit constitute the local spatiotemporal features of the target video frame.

[0096] In some embodiments, the first partition size and the second partition size can be set to the same or different, so that the global spatiotemporal features and the local spatiotemporal features can have cross-regional spatiotemporal context awareness through different scale block strategies. It should be understood that the above embodiment is only illustrative, and the specific settings of the first partition size and the second partition size can be flexibly adjusted according to user needs and are not limited to the above embodiment.

[0097] Figure 5 FIG. 1 shows a flow chart of a video processing method provided by another embodiment of the present specification, such as Figure 5As shown, for any target video frame in the enhanced video, the video processing system obtains multiple consecutive frames corresponding to the target video frame and inputs them into the Dynamically Guided Deformability Alignment (DDA) module to adaptively explore and align the features of the multiple consecutive frames. DDA establishes initial spatiotemporal features between multiple consecutive frames through dynamic displacement estimation, allowing the video processing system to adaptively align features across frames and explore their temporal relationships. These initial spatiotemporal features are then fed into the Spatiotemporal Feature Flow (SFF) module. SFF performs feature interactions on multiple feature units of the initial spatiotemporal features and extracts feature information from the feature units before and after the interaction using an MLP. This integrates feature information beyond the fixed receptive field and establishes strong long-range dependencies between feature units. This enables efficient global information exchange between distant features while ensuring computational efficiency. The initial spatiotemporal features are also fed into the Benign Selection Compensation (BSC) module, which focuses on improving the reconstruction of motion information in compressed video. Through a collaborative learning strategy, the BSC module combines deep feature extraction with fine local modeling, which can effectively alleviate the inter-frame motion discontinuity caused by compression. The BSC module divides the initial spatiotemporal features into multiple feature units based on the second division size, and refines the compensation features within these feature units to capture fine motion details, ultimately achieving accurate inter-frame motion compensation modeling. Subsequently, the global spatiotemporal features output by the SFF module and the local spatiotemporal features output by the BSC module will be fused, and the fused features will be input into the convolution layer to reduce the dimensionality of the fused features through the convolution layer, so that the dimension of the reduced spatiotemporal features is consistent with the dimension of the initial spatiotemporal features, thereby obtaining and outputting the fused spatiotemporal features. The video processing system determines the spatiotemporal feature residual based on the fused spatiotemporal features and the initial spatiotemporal features, and enhances the target video frame based on the spatiotemporal feature residual to generate an enhanced target video frame corresponding to the target video frame.

[0098] It should be understood that Figure 5For illustrative purposes only, the number of modules and the module contents included in the process of the specific video processing method can be flexibly adjusted according to user needs and are not limited to those given in the above embodiments. For example, a downsampling module can be added after the DDA module, and the initial spatiotemporal features processed by the downsampling module are input into the output SFF module and BSC module. At the same time, after obtaining the fused spatiotemporal features, the video processing system can input the fused spatiotemporal features into the upsampling module to process the fused spatiotemporal features through the upsampling module, so that the system generates an enhanced target video frame corresponding to the target video frame based on the processed features (fused spatiotemporal features and initial spatiotemporal features). Such a processing method can reduce the amount of computation and video memory occupancy by reducing the spatial resolution (downsampling) through upsampling and downsampling. Subsequently, the video processing system restores the features to their original size based on upsampling, thereby improving computational efficiency.

[0099] In some embodiments, the number of sub-modules (shown in the dotted box in the figure) composed of SFF, BSC and convolutional layers can be multiple, that is, the fused spatiotemporal features can be obtained after multiple fusion iterations, thereby ensuring the accuracy of the fused spatiotemporal features finally obtained and avoiding the interference of redundant features. Among them, the input of the first sub-module is the first fused feature. In addition, the input of the other sub-modules is the output of the previous sub-module until the last sub-module outputs the fused spatiotemporal features. It should be understood that the above embodiment is only an exemplary description, and the number of specific sub-modules can be flexibly adjusted according to user needs and is not limited to the above embodiment.

[0100] Figure 6 FIG. 1 shows a schematic diagram of the structure of a spatiotemporal feature flow module provided in an embodiment of the present specification. Figure 6 As shown in the figure, the spatiotemporal feature flow module includes two normalization layers (Layer Normalization, LN), multiple feature feedforward blocks (Feature Feedforward Block, FFB) and spatially guided multi-layer perceptron (Spatially Guided Multi-Layer Perceptron, SG-MLP). It should be understood that, Figure 6 This is only an example description, and the specific number and configuration of each module can be flexibly adjusted according to user needs and is not limited to the above embodiments.

[0101] Among them, the initial spatiotemporal features first enter the LN layer for normalization processing, adjusting the distribution of the initial spatiotemporal features so that the normalized initial spatiotemporal features have a stable mean and variance. Then, the normalized initial spatiotemporal features enter multiple FFB blocks respectively, and the multiple FFB blocks transform the normalized initial spatiotemporal features to build long-distance link relationships between feature units. Subsequently, the normalized initial spatiotemporal features are fused with the features output by the FFB to obtain fused spatiotemporal features, and the global spatiotemporal features are normalized based on the LN layer to adjust the distribution of the features again, and the adjusted fused spatiotemporal features are output. The adjusted features enter the SG-MLP, which uses a spatial guidance mechanism to process the features to capture the feature relationship between features from the spatial dimension, establish the association relationship between the feature units before and after the transformation processing, and obtain and output the processed global spatiotemporal features.

[0102] In summary, in the video processing method P300 and system 130 provided in this specification, after the video processing system obtains the video to be enhanced, for each target video frame in the video to be enhanced: the video processing system obtains the corresponding multiple continuous frames to perform feature extraction on the multiple continuous frames to obtain the initial spatiotemporal features corresponding to the target video frame. This processing method processes each target video frame based on the multiple continuous frames corresponding to each target video frame. It can utilize the correlation between adjacent frames to better retain the original information of the video to be enhanced, avoid the loss of key information caused by processing only the target video frame, and lay the foundation for subsequent more accurate processing of the video to be enhanced. Subsequently, the video processing system performs feature interaction on multiple feature units in the initial spatiotemporal features to explore the long-distance dependency between feature units, thereby obtaining global spatiotemporal features. The above-mentioned method of performing feature interaction on multiple feature units corresponding to each target video frame can enable effective information exchange and fusion between feature units at different positions, and better associate feature information between different feature units. The acquisition of the above-mentioned global spatiotemporal features can enable the video processing system to better understand the overall content and structure of the video to be processed, help reduce the problem of blocking effects caused by local processing, and significantly improve the overall coherence and quality of the processed target video. Furthermore, the video processing system generates an enhanced target video frame corresponding to the target video frame based on the initial spatiotemporal features and the global spatiotemporal features corresponding to each target video frame, so as to generate an enhanced target video based on each enhanced target video frame. The above-mentioned generation of enhanced target video frames based on the initial spatiotemporal features and the global spatiotemporal features integrates local and global information and can enhance the video frames more comprehensively. The initial spatiotemporal features retain the basic features and details of the video frame, while the global spatiotemporal features provide a wider receptive field and deeper information fusion. The enhanced video frames generated in the above-mentioned manner can ensure the quality of the generated target video, thereby improving the user's visual experience.

[0103] Another aspect of this specification provides a computer-readable, non-transitory storage medium storing at least one instruction set for performing video processing executable instructions. When executed by a processor, the at least one instruction set directs the processor to implement the steps of the video processing method P300 described herein. In some possible implementations, various aspects of this specification may also be implemented as a program product comprising program code. When the program product is executed on system 130, the program code causes system 130 to perform the steps of method P300 described herein. The program product for implementing the aforementioned method may comprise a portable compact disc read-only memory (CD-ROM) comprising the program code and may be executed on system 130. However, program products herein are not limited to such. In this specification, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system. The program product may utilize any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any combination thereof. More specific examples of computer-readable storage media include: an electrical connection having one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. A computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable storage medium may also be any readable medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing. Program code for performing the operations of this specification may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages.

[0104] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0105] In summary, after reading this detailed disclosure, those skilled in the art will appreciate that the foregoing detailed disclosure may be presented by way of example only and may not be limiting. Although not expressly stated herein, those skilled in the art will understand that this specification encompasses various reasonable changes, improvements, and modifications to the embodiments. Such changes, improvements, and modifications are intended to be suggested by this specification and are within the spirit and scope of the exemplary embodiments of this specification.

[0106] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, “one embodiment,” “an embodiment,” and / or “some embodiments” mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this specification. Therefore, it is emphasized and should be understood that two or more references to “an embodiment,” “one embodiment,” or “an alternative embodiment” in various parts of this specification do not necessarily refer to the same embodiment. Furthermore, particular features, structures, or characteristics may be appropriately combined in one or more embodiments of this specification.

[0107] It should be understood that in the foregoing descriptions of the embodiments of this specification, to facilitate understanding of a feature and to simplify this specification, various features are combined in a single embodiment, figure, or description thereof. However, this does not necessarily mean that these features are combined. When reading this specification, a person skilled in the art may label some of the devices as separate embodiments. In other words, the embodiments of this specification can also be understood as the integration of multiple sub-embodiments. The content of each sub-embodiment is also valid even when it includes fewer than all the features of a single previously disclosed embodiment.

[0108] Each patent, patent application, publication of a patent application, and other materials, such as articles, books, specifications, publications, documents, and the like, cited in this disclosure (excluding any historical review documents related thereto) is hereby incorporated by reference for all purposes relevant to this disclosure, such as within the specification and claims of this disclosure. However, if there is any inconsistency or conflict between the descriptions, definitions, and / or terminology of such materials and the descriptions, definitions, and / or terminology used in this disclosure, the descriptions, definitions, and / or terminology used in this disclosure shall control.

[0109] Finally, it should be understood that the embodiments of the application disclosed herein are illustrative of the principles of the embodiments of this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely examples and not limitations. Those skilled in the art can adopt alternative configurations based on the embodiments in this specification to implement the application in this specification. Therefore, the embodiments of this specification are not limited to the embodiments precisely described in the application.

Claims

1. A video processing method, wherein: The method comprises: Acquire a video to be enhanced, and for each target video frame in the video to be enhanced: acquire multiple consecutive frames corresponding to the target video frame; Performing feature extraction on the multiple consecutive frames to obtain initial spatiotemporal features corresponding to the target video frame, the initial spatiotemporal features including multiple feature units; Performing feature interaction on the multiple feature units of the initial spatiotemporal feature to obtain a global spatiotemporal feature, wherein the global spatiotemporal feature includes a long-range dependency relationship between the feature units; and Based on the initial spatiotemporal features and the global spatiotemporal features corresponding to each target video frame, an enhanced target video frame corresponding to each target video frame is generated, so as to generate an enhanced target video based on each enhanced target video frame.

2. The method according to claim 1, wherein The performing feature interaction on the multiple feature units of the initial spatiotemporal features includes: Feature interaction is performed on the multiple feature units of the initial spatiotemporal features using a preset feature arrangement strategy, where the preset feature arrangement strategy is used to transform the positions of pixels within the feature units.

3. The method according to claim 2, wherein: The performing feature interaction on the multiple feature units of the initial spatiotemporal features by using a preset feature arrangement strategy includes: For any current feature unit among the plurality of feature units: determining a target feature unit corresponding to the current feature unit by a preset stream mapping method based on the feature map size of the initial spatiotemporal feature and the first partition size; and The long-distance dependency relationship between the current feature unit and the target feature unit of the initial spatiotemporal feature is established through a multi-layer perceptron.

4. The method according to claim 3, wherein: The determining, based on the feature map size of the initial spatiotemporal features and the first division size, of a target feature unit corresponding to the current feature unit by a preset stream mapping method includes: Based on the feature map size of the initial spatiotemporal feature and the first division size, performing feature transformation on multiple feature units in the initial spatiotemporal feature by a preset stream mapping method to obtain a transformed spatiotemporal feature; the transformed spatiotemporal feature includes multiple feature units; and For each current feature unit in the initial spatiotemporal feature: determining a corresponding target feature unit in the transformed spatiotemporal feature.

5. The method according to claim 4, wherein: The step of performing feature transformation on a plurality of feature units in the initial spatiotemporal features by using a preset stream mapping method based on the feature map size of the initial spatiotemporal features and the first division size to obtain transformed spatiotemporal features includes: Determining a target row and a target column corresponding to each pixel point in the initial spatiotemporal feature based on a feature map size of the initial spatiotemporal feature and the first partition size; and Performing row transformation and column transformation on each pixel point in the initial spatiotemporal feature, so as to transform each pixel point to a corresponding target row and target column through the row transformation and the column transformation, thereby obtaining a transformed spatiotemporal feature.

6. The method of claim 1, wherein: The continuous frames corresponding to the target video frame include: the target video frame, at least one forward video frame of the target video frame, and / or at least one backward video frame of the target video frame.

7. The method according to claim 6, wherein: The extracting features from the plurality of consecutive frames to obtain initial spatiotemporal features corresponding to the target video frame includes: Extracting features of a target video frame to obtain features of the target video frame; Extracting features from at least one video frame preceding the target video frame in the plurality of consecutive frames to obtain a first feature, and determining a forward alignment feature based on the first feature and the target video frame feature; performing feature extraction on at least one video frame backward from the target video frame in the plurality of consecutive frames to obtain a second feature, and determining a backward alignment feature based on the second feature and a feature of the target video frame; and Based on the forward alignment features, the target video frame features and the backward alignment features, initial spatiotemporal features corresponding to the target video frame are obtained.

8. The method of claim 7, wherein: The determining of the forward alignment feature based on the first feature and the target video frame feature includes: Performing spatiotemporal alignment on the first feature with the target video frame feature to obtain the forward aligned feature; The determining of the backward alignment feature based on the second feature and the target video frame feature includes: The second feature is spatiotemporally aligned with the target video frame to obtain the backward aligned feature.

9. The method of claim 7, wherein: The obtaining, based on the forward alignment feature, the target video frame feature, and the backward alignment feature, an initial spatiotemporal feature corresponding to the target video frame includes: Determining attention weights corresponding to the forward alignment feature, the target video frame feature, and the backward alignment feature; and The forward alignment feature, the target video frame feature and the backward alignment feature are weightedly fused based on the attention weight to obtain an initial spatiotemporal feature corresponding to the target video frame.

10. The method of claim 9, wherein: The determining of attention weights corresponding to the forward alignment feature, the target video frame feature, and the backward alignment feature includes: Performing feature splicing on the forward alignment feature, the target video frame feature, and the backward alignment feature to obtain a spliced ​​feature; and The splicing features are feature decomposed by using a predetermined feature description operator to determine attention weights corresponding to the forward alignment features, the target video frame features, and the backward alignment features.

11. The method of claim 1, wherein: The method further comprises: The initial spatiotemporal features are divided based on a first division size to obtain a plurality of first feature units after division; each of the plurality of first feature units contains spatiotemporal features corresponding to a first preset number of pixels.

12. The method of claim 1, wherein: Generating an enhanced target video frame corresponding to each target video frame based on the initial spatiotemporal features and the global spatiotemporal features corresponding to each target video frame includes: Extracting local spatiotemporal features from the initial spatiotemporal features corresponding to each target video frame; fusing the local spatiotemporal features and the global spatiotemporal features corresponding to each target video frame to obtain fused spatiotemporal features; and Based on the initial spatiotemporal features and the fused spatiotemporal features corresponding to each target video frame, an enhanced target video frame corresponding to each target video frame is generated.

13. The method of claim 12, wherein: The extracting of local spatiotemporal features from the initial spatiotemporal features corresponding to each target video frame includes: For each of the target video frames: based on the second division size, the initial spatiotemporal features corresponding to the target video frame are divided to obtain a plurality of second feature units after division; each second feature unit contains spatiotemporal features corresponding to a second preset number of pixels; feature extraction is performed on each of the second feature units to obtain local spatiotemporal features of each second feature unit, and the features of each second feature unit constitute the local spatiotemporal features of the target video frame.

14. The method of claim 12, wherein: Generating an enhanced target video frame corresponding to each target video frame based on the initial spatiotemporal features and the fused spatiotemporal features corresponding to each target video frame includes: Determining a spatiotemporal feature residual based on the initial spatiotemporal feature and the fused spatiotemporal feature; Summing the initial spatiotemporal features and the spatiotemporal feature residuals to determine target spatiotemporal features; and Based on the target spatiotemporal features, an enhanced target video frame corresponding to each target video frame is generated.

15. A video processing system comprising: at least one storage medium storing at least one instruction set for performing video processing; as well as At least one processor is communicatively connected to the at least one storage medium, wherein the at least one processor reads the at least one instruction set when running and executes the method of any one of claims 1 to 14 according to the instructions of the at least one instruction set.

Images