A video super-resolution method and system based on deformable 3D convolutional network

Abstract

The present invention provides a video super-resolution method and system based on a deformable 3D convolutional network. The method comprises: inputting a low-resolution (LR) video reference frame and its adjacent supporting frames into the deformable 3D convolutional network; processing the deformable 3D convolutional network yields a high-resolution (HR) video frame; the deformable 3D convolutional network processing comprises: shallow feature extraction, implicit feature alignment, spatiotemporal feature fusion, and high-resolution (HR) video frame reconstruction. The present invention proposes a lightweight deformable network based on a priori frames for video super-resolution tasks, and designs an efficient hardware accelerator based on its data dependencies and computational characteristics. Compared to existing acceleration schemes, this method achieves higher video reconstruction quality, with throughput improvements of 2.75 times and energy efficiency improvements of 1.63 times.

Description

Technical Field

[0001] The present invention relates to a video super-resolution method and system based on a deformable 3D convolutional network. Background Art

[0002] As an underlying processing task in the field of computer vision, the basic idea of ​​video super-resolution (VSR) is to reconstruct high-resolution (HR) video images for low-resolution (LR) video images by collecting temporal and spatial information of adjacent frames. By restoring the high-frequency texture details in the video image, the blurred LR video stream is made clearer and can support higher-definition display devices.

[0003] In recent years, convolutional neural networks (CNNs) have achieved significant breakthroughs in the field of VSR compared to traditional interpolation methods. However, due to their high computational resource requirements, advanced CNN models typically require high-performance graphics processing unit clusters to execute, making them difficult to deploy on resource-constrained mobile devices. Therefore, developing high-speed, low-power VSR accelerators is crucial for improving reconstruction quality and user experience. Currently, the commonly used method is to perform single image super-resolution (SISR) frame by frame. Although this algorithm has low algorithmic complexity, it often ignores the continuity between video frames, resulting in reduced VSR performance.

[0004] As a new variant of the CNN model, the Deformable Convolutional Network (DCN) can effectively compensate for inter-frame continuity and improve VSR performance by enhancing the model's multi-frame alignment capabilities. This is because the offsets extracted by DCN have extremely similar patterns to the optical flow information implicit in the video. Compared with typical optical flow extraction methods, DCN can capture better alignment features and improve reconstruction quality by increasing the diversity of offsets in a compact feature space. However, this also places higher demands on deployment on resource-limited end-side devices.

[0005] Because DCNs differ significantly from typical CNNs, deconvolutional networks, and recurrent neural networks (RNNs), directly using existing accelerators to execute DCNs would severely reduce computational efficiency and even render the accelerator inoperable. This is due to the following reasons: First, the dynamically generated offsets enable DCNs to sample pixel features at arbitrary locations, rather than following the conventional sliding window order. This results in irregular memory access patterns. Furthermore, these irregular memory access patterns can cause serious access conflicts, jeopardizing parallel design and sacrificing opportunities for input reuse. Second, the flexible sampling locations may force processing modules to access pixel features not already loaded on-chip, increasing the memory burden. Finally, DCNs inevitably incur significant computational overhead due to the need for additional offset generation and interpolation of adjacent features. Although a large number of existing studies have made DCN hardware-friendly by constraining the size, shape, and diversity of offsets to limit irregular receptive fields, these oversimplifications can lead to inaccurate motion estimation, which seriously affects the inter-frame continuity of VSR. In addition, existing DCN-specific accelerators cannot solve the problem of real-time processing of VSR tasks while ensuring that the sampling range is flexible enough.

[0006] Random camera or object motion often leads to misalignment across multiple frames. To explore inter-frame correlation information, established VSR methods typically use explicit or implicit alignment mechanisms to mine complementary sub-pixel features. However, inaccurate motion estimation and compensation schemes can lead to blurry video quality. Compared to the explicit alignment mechanisms used in traditional optical flow estimation networks, DCN implicitly aligns adjacent frames at the feature level, thereby more accurately modeling motion information. D3DNet is a typical processing approach that adaptively perceives brightness changes and object movement by fusing spatiotemporal features with motion information. However, due to its high computational complexity (2.58M parameters, 408.82 GFLOPs of computation required to perform a 4x super-resolution of a single frame to a resolution of 1280×720, and an inference time of 45.2 seconds on an RTX2080Ti GPU), this advanced VSR method is difficult to effectively deploy on edge devices.

[0007] FPGA-based super-resolution methods are mainly divided into two categories: SISR and VSR. SISR uses spatial information to upsample LR video images frame by frame. However, due to the neglect of critical temporal information and the limited model fitting capabilities, although it can achieve relatively fast processing speeds, it sacrifices the consistency of the upsampled video stream, resulting in a loss of visual quality in the reconstructed video. VSR methods upsample the target frame by combining multiple adjacent LR frames and fusing spatial and temporal features. For example, ERVSR uses a lightweight RNN model to explore potential inter-frame correlations. However, RNN-based VSR methods require a large amount of memory resources to store hidden states for parallel processing. Moreover, due to the lack of an effective inter-frame alignment mechanism, this method is also difficult to adapt to rapidly changing motion. To date, few studies have attempted to incorporate inter-frame alignment mechanisms into VSR hardware accelerators. Summary of the Invention

[0008] Purpose of the invention: The technical problem to be solved by the present invention is to provide a video super-resolution method based on a deformable 3D convolutional network in response to the shortcomings of the existing technology, including: and its Niz-1 adjacent supporting frames Input the deformable 3D convolutional network, and after processing by the deformable 3D convolutional network, a high-resolution HR video frame is obtained. Among them, Nix, Niy, Nif are the height, width and number of channels of the low-resolution LR video frame image respectively, and s is the upsampling factor. Represents a low-resolution LR video reference frame Niz-1th adjacent supporting frame, represents the real number space of sNix×sNiy×Nif dimensions;

[0009] The processing of the deformable 3D convolutional network includes the following stages: shallow feature extraction, implicit feature alignment, spatiotemporal feature fusion and high-resolution HR video frame reconstruction.

[0010] The shallow feature extraction includes: extracting spatiotemporal information using a depth-separable 3D convolutional layer, wherein the depth-separable 3D convolutional layer includes a channel-by-channel 3D convolutional layer DwConv3D and a point-by-point 3D convolutional layer PwConv3D.

[0011] The implicit feature alignment process involves taking Niz frame features as input and performing implicit feature alignment based on a block-decoupled computational strategy (TDS). Specifically, the intermediate feature sequence obtained from the shallow feature extraction stage is partitioned, ensuring that all layers of the implicit feature alignment phase are executed at the block granularity rather than the entire intermediate feature sequence. The implicit feature alignment phase includes two or more Residual Deformable 3D Blocks (ResD3D). Each Residual Deformable 3D Block (ResD3D) is stacked based on a shifted Deformable 3D Conv3D. These blocks perform both spatial deformation and temporal alignment, using a residual learning strategy to improve the convergence of the model training process. The input and output blocks have the same size and the same number of input and output features. The input and output blocks follow the TDS strategy, and this phase is executed at the block granularity, so both the input and output features of ResD3D are blocks. The number of input and output features refers to the number of feature channels in the input block and the number of feature channels in the output block, corresponding to the Nif mentioned above. The Niz frame alignment feature is finally obtained through implicit feature alignment.

[0012] The block decoupling-based calculation strategy TDS includes: decomposing the Niz frame features into two or more overlapping blocks in the spatial dimension, and the implicit feature alignment stage is performed in blocks, and only one block is processed in each cycle.

[0013] The spatiotemporal feature fusion includes: further aggregating the obtained Niz frame alignment features in the time dimension through the temporal average pooling function TAP (Temporal Average Pooling), and then capturing the spatial features by the 2D convolution residual block ResC2D (Residual Convolutional 2D Block) built by the depth-separable 2D convolution layer.

[0014] The high-resolution HR video frame reconstruction includes: propagating spatial features to the upsampling module to reconstruct the video super-resolution VSR result, introducing a residual learning strategy based on the prior frame to replace the pixel-by-pixel interpolation in the traditional residual connection: copying the low-resolution LR video reference frame on the residual connection path based on the prior frame 2 times and concatenate into a sequence I z ' will become the priori block diagram of the final super-resolution SR video frame in the upsampling module, and the priori block diagram contains the low-frequency components in the low-resolution LR video reference frame. The final super-resolution SR video frame is generated by the entire network to estimate the aforementioned high-resolution HR video frame It is generated by the upsampling module and is also the final result of the entire ALDNet.

[0015] The shift-based deformable 3D convolution SDfConv3D includes a shifted point-by-point 3D convolution layer SPwConv3D, a shifted bilinear interpolation layer SBilinear and two or more ordinary separable 3D convolution layers.

[0016] The processing of the shift-based deformable 3D convolution SDfConv3D includes:

[0017] Step a1, offset generation: The shift-based deformable 3D convolution SDfConv3D is performed at a block granularity. For the i-th input block Get the corresponding predicted offset For pixels Offset generation process H Ψ Expressed as:

[0018]

[0019] in, It's a pixel In the spatial dimension coordinates, It's a pixel In the time dimension, It's a pixel The nth pair of coordinate offsets in the spatial dimension, represents a three-dimensional sampling grid of size Nkx×Nky×Nkz=3×3×3, and Nkx and Nky represent the width and height of the three-dimensional sampling grid in the spatial dimension, Nkz is the number of frames of the three-dimensional sampling grid in the time dimension, and the number of channels of the offset is Tix×Tiy represents the spatial size of each input block, Tix represents the width of the input block, and Tiy represents the height of the input block; the offset generation process H Ψ It consists of a series of cascaded ordinary separable 3D convolution layers and a shifted point-by-point 3D convolution layer SPwConv3D;

[0020] For the shifted point-by-point 3D convolution layer SPwConv3D, its weight Expressed as a power of 2:

[0021]

[0022] Where s and p represent sign flip and shift respectively, Nof represents the number of output feature maps, and sgn represents the sign function;

[0023] The complete computation process of the shifted point-by-point 3D convolution layer SPwConv3D is expressed as follows using shift and addition:

[0024]

[0025] in, and They represent the mth input feature map and the jth output feature map respectively, and * represents the convolution operation; represents the weight of the mth input channel and jth output channel of SPwConv3D; s j,m Represents the symbol value of the mth input channel and jth output channel;

[0026] Step a2, spatial deformation: The deformed receptive field is obtained by formula (1). The deformed receptive field includes the following parts:

[0027] Current pixel Coordinates

[0028] By 3D sampling grid get The current pixel The normal coordinate offset of the center

[0029] Indicates the current pixel The dynamic offset predicted in the spatial dimension is used to achieve spatial deformation and temporal alignment of SDfConv3D;

[0030] The coordinates of each element in the deformed receptive field Expressed as:

[0031]

[0032] in, is the current pixel The nth pair of coordinate values ​​in the spatial dimension; is the current pixel The nth coordinate value in the time dimension;

[0033] Represents a 3D sampling grid The coordinates of the elements in , Represents a 3D sampling grid The spatial dimension coordinate value of the nth grid; Represents a 3D sampling grid The time dimension coordinate value of the nth grid;

[0034] Each pair of coordinates corresponds to

[0035] and They are The decimal part and the decimal part of ;

[0036] Introducing the bilinear interpolation function G(·), G(·) is expressed as:

[0037]

[0038] Where T represents matrix transpose;

[0039] Convert the integer part and decimal part of the deformed coordinates into the interpolation coefficient vector U and sampling pixel vector V of each feature map respectively:

[0040]

[0041] in, and They are all shared parameters;

[0042] In order to reduce the number of multiplication operations in formula (5), each element u of U d It is expressed as L coefficients of powers of 2 using the following formula:

[0043]

[0044] in represents the codebook set, Represents the lth codebook set, each codebook contains M candidate elements, Ω d Represented as u d The searched power-of-two index, in the shifted bilinear interpolation layer SBilinear, G(·) with dense vector dot multiplication operations is simplified to only shift and add operations

[0045] Step a3, 3D feature extraction: based on F i and The shift-based deformable 3D convolution SDfConv3D is expressed as:

[0046]

[0047] in, and They are the weights of the channel-by-channel 3D convolution layer DwConv3D and the weights of the point-by-point 3D convolution layer PwConv3D, O i It is the output feature of the shift-based deformable 3D convolution SDfConv3D.

[0048] The present invention also provides a video super-resolution system based on a deformable 3D convolutional network, comprising a key processing core CPC, an auxiliary processing core APC, a global controller, a distribution module and an aggregation module;

[0049] The key processing core CPC performs shift-based deformable 3D convolution SDfConv3D in a fully pipelined manner; the key processing core CPC includes a class A standard 3D convolution module Conv3D-Module A, a class B standard 3D convolution module Conv3D-Module B, and a deformable 3D convolution module DfConv3D-Module;

[0050] The class A standard 3D convolution module Conv3D-Module A and the class B standard 3D convolution module Conv3D-ModuleB are used to perform offset generation;

[0051] The deformable 3D convolution module DfConv3D-Module is used to perform spatial deformation and 3D feature extraction;

[0052] The auxiliary processing core APC is used to support the remaining convolution operations in the deformable 3D convolution network except the shift-based deformable 3D convolution SDfConv3D, and the remaining convolution operations include depth-separable 3D convolution and depth-separable 2D convolution with only one input channel in shallow feature extraction;

[0053] In the global controller, the CPC controller and APC controller coordinate the key processing core (CPC) and auxiliary processing core (APC) to perform different calculations. Direct memory access (DMA) generates descriptors to manage data interaction between the key processing core (CPC) and auxiliary processing core (APC) and off-chip. The distribution module is responsible for allocating data to the key processing core (CPC) and auxiliary processing core (APC), and the aggregation module is used to rearrange output data and send it off-chip.

[0054] A block-level interleaved storage scheme is introduced in the deformable 3D convolution module DfConv3D-Module, specifically including:

[0055] Step b1, memory management: the deformable 3D convolution module DfConv3D-Module can call on-chip storage resources, and the on-chip storage resources include input buffer, offset buffer, DwConv3D weight buffer, PwConv3D weight buffer, and output buffer;

[0056] Input block of shift-based deformable 3D convolution SDfConv3D It is a four-dimensional tensor of size Tix×Tiy×Niz×Nif. The input buffer contains Piz input storage banks for storing Piz video frames. Similarly, the offset buffer also contains Piz storage banks, and each address contains all channel values ​​of each pixel. The weights are sent to the DwConv3D weight buffer and the PwConv3D weight buffer through the distribution module; finally, the output result is sent to the output buffer with the same arrangement pattern as the input buffer, waiting to be read;

[0057] Step b2, offset value decomposition: Based on the memory management of step b1, each cycle can access Noff×Piz offsets; expand the spatial dimension of the 3D convolution kernel, that is, the parallelism Pks of the 3D convolution kernel is Pks=Nkx×Nky; rearrange the offsets and divide them into Nkz groups, each group contains Pks×Niz coordinate offsets, in this way, a complete 3D convolution can be completed within Nkz cycles; then, according to the scale of fixed-point quantization, the offsets are divided into a group of integer values ​​P[i] and fractional values ​​Q[i], i=0,1,...,Pks×Piz, and then P[i] and Q[i] are assigned to the address converter and coefficient generator respectively. The address converter converts the integer value P[i] into an address index to sample the pixel vector, and the coefficient generator is used to convert Q[i] into the coefficient of the shifted bilinear interpolation layer SBilinear. Finally, a shift quantizer is constructed based on formula (7) to convert the shifted bilinear interpolation layer SBilinear coefficients into codebook indices;

[0058] Step b3: Design a block-level interleaved storage scheme as an additional memory hierarchy to cache sample values ​​and perform parallel shift interpolation with low memory resource overhead. The block-level interleaved storage scheme includes two register arrays RA, named RA0 and RA1, which are controlled by two signals respectively. To achieve high throughput, each register array RA can accommodate Nb×Nb×Pks data blocks and operate in ping-pong mode. 0 indicates that old data can be overwritten by new data from top to bottom, which is called the write state; 1 indicates that the full register array RA reads data from left to right, which is called the read state.

[0059] The write state and the read state are configured to run in orthogonal directions to execute the shifted bilinear interpolation layer SBilinear in parallel on more than two input channels at the same time; the calculation process of the block-level interleaved storage scheme is as follows:

[0060] Step b3-1, initial stage: according to the sampling index, each cycle retrieves Pks of input data of size Nif×Piz, each group of input data is divided into Nb data blocks of size Pif×Piz, and written into the register array RA1 by row, Represents the parallelism of the input feature channel dimension; after Nb cycles, the state signal of the register array RA1 is set to 1;

[0061] Step b3-2, Phase 0: Each cycle, new Pks×Nb data blocks will be loaded into register array RA0 and overwrite the old data in the register array by row. At the same time, Pks×Nb data blocks will be read from RA1 by column.

[0062] Step b3-3, Phase 1: After every Nb cycles, the operating states of register arrays RA0 and RA1 are swapped. The old data in register array RA1 is overwritten row by row every cycle, while register array RA0 pops out Pks × Nb data blocks column by column. After another Nb cycles, the entire process returns to Phase 0.

[0063] Step b3-4, final stage: Once all preloaded weights have been transferred and calculated, processing of the current input block stops; a new round of weights and input blocks are then transferred on-chip, and steps b3-1 to b3-4 are repeated; the weights are those sent to the DwConv3D weight buffer and the PwConv3D weight buffer by the distribution module in step b1;

[0064] Step b4: Calculation arrays at different layers:

[0065] The computational logic in the deformable 3D convolution module includes the computational arrays of the shifted bilinear interpolation layer SBilinear and the channel-by-channel 3D convolution layer DwConv3D and the point-by-point 3D convolution layer PwConv3D in 3D feature extraction. The computational arrays are designed with similar high-level connections, but differ in the microarchitecture of their computational units. The computational arrays include computational units and partial and memory. The computational units are interconnected and stacked to form a three-dimensional architecture. The partial and memory are used to temporarily store the results of the computational units. The input data is shared between the computational unit columns, and the weights are preloaded on different computational units. According to the data flow with fixed weights, after all calculations are completed, the results derived by the computational units are reduced in real time through the addition tree and transported to the partial and registers. The computational array is defined by the number of rows, columns, and clusters (Nr, Nc, Ng), where Nr, Nc, and Ng represent the number of computational units in the row, column, and cluster dimensions of the computational array, respectively.

[0066] The structural parameters of the shifted bilinear interpolation layer SBilinear calculation array are (Nr, Nc, Ng) = (Piz, Nb, Pks); each calculation unit includes 2Pif shifters and Pif adders, which process in parallel in the input channel dimension. At this time, the sample value and the shifted bilinear interpolation layer SBilinear coefficient are used as input and weight respectively.

[0067] The structural parameters of the channel-by-channel 3D convolution layer DwConv3D calculation array are (Nr, Nc, Ng) = (Pif, Pks, 1), and the structural parameters of the point-by-point 3D convolution layer PwConv3D calculation array are (Nr, Nc, Ng) = (Pof, Pif, 1); the calculation units of these two calculation arrays are the same, both containing Piz multipliers, so that parallel calculation is performed in the time dimension;

[0068] The class A standard 3D convolution module Conv3D-Module A and the class B standard 3D convolution module Conv3D-Module B are respectively used to calculate the standard separable 3D convolution in the offset generation stage and the special version with the SPwConv3D layer;

[0069] In the standard 3D convolution module Conv3D-Module A of type A and the standard 3D convolution module Conv3D-Module B of type B, the standard 3D convolution module Conv3D-Module A of type A calculates a standard separable 3D convolution (specifically, a channel-by-channel 3D layer DwConv3D and a point-by-point 3D convolution layer PwConv3D), and the standard 3D convolution module Conv3D-Module B of type B calculates a special version with an SPwConv3D layer (specifically, a channel-by-channel 3D layer DwConv3D and an SPwConv3D). These two convolution modules contain an input buffer, an output buffer, a DwConv3D weight buffer, a PwConv3D weight buffer, a DwConv3D processing element PE array, and a PwConv3D processing element PE array (for the standard 3D convolution module of type B, it is the SPwConv3D processing element PE array).

[0070] The difference between Conv3D-Module A and Conv3D-Module B is the PwConv3D processing element PE array and SPwConv3D processing element PE array.

[0071] The DwConv3D processing element PE array is composed of Pif stacked PE clusters, which are used to process input channel data in parallel. Each PE cluster contains Nkz×Piz-2 PEs, and each PE is responsible for performing Nky ​​multiplication and accumulation operations. The weights of DwConv3D are reused in each row of the PE array. In the time dimension, Piz input data are transmitted to each PE cluster at the same time and broadcast in the diagonal direction. Subsequently, Piz×Pif partial sums are accumulated along each column and directly transmitted to the subsequent cascaded PwConv3D or SPwConv3D layer; the high-level connection method of the processing element PE arrays of PwConv3D and SPwConv3D are consistent with the point-by-point 3D convolution layer PwConv3D calculation array of the deformable 3D convolution module. Their processing elements PE are also composed of Piz multipliers or shifters, thereby performing calculations in parallel in the time dimension.

[0072] The auxiliary processing core APC is reconfigured into two working modes through a switch flag and operates in a time-division multiplexing manner; the two working modes are respectively a depth-separable 3D convolution layer with only one input channel and a depth-separable 2D convolution layer;

[0073] In terms of storage of the auxiliary processing core APC, it includes an input buffer, a DwConv2D weight buffer, a PwConv2D weight buffer and an output buffer. A 3D or 4D input tensor is stored through an input buffer. For storing 3D input tensors, each address contains all input channels for each pixel; for storing 4D input tensors, since the number of input channels is one, the processing of the 3D convolution kernel in the time dimension is switched to the input channel dimension; the number of features in each pixel is flattened to Nkz×Niz; for the residual learning strategy based on the prior frame, when the auxiliary processing core APC executes the last layer, that is, when executing the last PwConv2D of the deformable 3D convolutional network, the pixels of the prior frame are preloaded to the corresponding address of the output buffer, and residual learning based on the prior frame is performed by combining the pixels of the prior frame and the convolution result; finally, the video super-resolution VSR result can be directly transferred to the same address of the output buffer;

[0074] In terms of the calculation of the auxiliary processing core APC, it includes a channel-by-channel convolution layer calculation array and a point-by-point convolution layer calculation array. The channel-by-channel convolution layer calculation array is used to calculate DwConv2D and DwConv3D with only one input channel, and the point-by-point convolution layer calculation array is used to calculate PwConv2D and PwConv3D. A row-fixed data flow is designed for the channel-by-channel convolution layer calculation array, and Pox row data is output in parallel channel by channel. The reconfigurable point-by-point convolution layer calculation array includes two calculation modes and is configured by a switch flag. One mode is to execute PwConv2D by expanding the input and output channel dimensions. The other mode is to execute the point-by-point 3D convolution layer PwConv3D. The point-by-point 3D convolution layer PwConv3D does not calculate the accumulation operation of the input channel. The partial sum psum of the 3D convolution at different positions is directly sent to the 3D-Psum register. Each calculation unit of the point-by-point convolution layer calculation array contains only one multiplier.

[0075] The computing array in APC for reconfigurable computation of point-by-point convolutional layers (including PwConv3D and PwConv2D) consists of computing units, partial sum registers, and 3D partial sum registers 3D-Psum.

[0076] In the prior art, the depthwise separable 3D convolutional layer includes a channel-by-channel 3D convolutional layer DwConv3D and a point-by-point 3D convolutional layer, and the depthwise separable 2D convolutional layer includes a channel-by-channel 2D convolutional layer DwConv2D and a point-by-point 2D convolutional layer DwConv2D.

[0077] Beneficial Effects: Deformable convolutional networks (ADCs) show great potential in video super-resolution tasks due to their powerful inter-frame feature alignment capabilities. However, their high computational complexity and irregular memory access patterns make their deployment on resource-constrained devices challenging. This solution addresses this issue by proposing an Anchor-Based Lightweight Deformable Network (ALDNet) for video super-resolution tasks. Furthermore, an efficient hardware accelerator is designed based on the network's data dependencies and computational characteristics. Compared to existing acceleration solutions, this approach achieves higher video reconstruction quality, with throughput improvements of 2.75 times and energy efficiency improvements of 1.63 times. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more apparent.

[0079] Figure 1a This is a principle block diagram of a deformable 3D convolutional network ALDNet for video super-resolution.

[0080] Figure 1b Schematic diagram of the ResD3D module for implicit feature alignment in ALDNet.

[0081] Figure 1c This is a schematic diagram of the ResC2D module for spatiotemporal feature fusion in ALDNet.

[0082] Figure 1d Schematic diagram of the upsampling module for high-resolution HR video frame reconstruction in ALDNet.

[0083] Figure 2 This is the principle block diagram of the shift-based deformable 3D convolution SDfConv3D.

[0084] Figure 3a This is a schematic diagram of traditional bilinear interpolation.

[0085] Figure 3b This is the calculation flow chart of the shifted bilinear interpolation layer SBilinear.

[0086] Figure 4 It is the overall architecture diagram of the system of the present invention.

[0087] Figure 5 This is the overall architecture diagram of the deformable 3D convolution module DfConv3D-Module.

[0088] Figure 6a This is a schematic diagram of the offset value rearrangement rules.

[0089] Figure 6b This is a schematic diagram of the address converter.

[0090] Figure 6c This is a schematic diagram of the coefficient converter.

[0091] Figure 7 This is a schematic diagram of the block-level interleaved storage scheme of the deformable 3D convolution module DfConv3D-Module.

[0092] Figure 8a This is the computational array structure diagram of the deformable 3D convolution module.

[0093] Figure 8b This is the microarchitecture diagram of the shifted bilinear interpolation layer SBlinear computing unit (SBCU).

[0094] Figure 9a This is the overall structure diagram of the PE array of the channel-by-channel 3D convolution layer DwConv3D processing element of the standard 3D convolution module.

[0095] Figure 9bIt is the micro-architecture diagram of the channel-by-channel 3D convolution layer DwConv3D processing element PE of the standard 3D convolution module.

[0096] Figure 10a This is a schematic diagram of the feature expansion of the auxiliary processing core APC.

[0097] Figure 10b This is a schematic diagram of the reconfigurable computing array of the auxiliary processing core APC for the point-by-point 3D convolution layer PwConv3D and the point-by-point 2D convolution layer PwConv2D. DETAILED DESCRIPTION

[0098] The present invention provides a video super-resolution method based on a deformable 3D convolutional network, wherein the deformable 3D convolutional network ALDNet includes the following contents:

[0099] 1.1 Overall Architecture of ALDNet

[0100] Considering the limited hardware resources of the end-side devices, this paper proposes a hardware-inspired deformable 3D-CNN model, namely ALDNet, which aims to extract the best 3D-CNN model from the low-resolution LR video reference frame. and its Niz-1 adjacent supporting frames Recover a high-resolution HR video frame Among them, Nix, Niy, Nif are the height, width and number of channels of the LR frame image respectively, and s is the upsampling factor. By continuously optimizing the ALDNet network, the predicted video super-resolution results are As close as possible like Figure 1a As shown in Figure 3, ALDNet mainly consists of four stages: shallow feature extraction, implicit feature alignment, spatiotemporal feature fusion, and HR frame reconstruction.

[0101] The shallow feature extraction stage uses depthwise separable 3D convolutional layers to extract spatiotemporal information, including channel-by-channel 3D convolutional layers (Depthwise 3D Convolutional Layer, DwConv3D) and pointwise 3D convolutional layers (PwConv3D).

[0102] In the implicit feature alignment stage, based on the tile decoupling computing strategy (TDS), the intermediate feature sequence is calculated at the tile granularity. Figure 1bAs shown, this stage consists of two or more ResD3D blocks, each of which is stacked with shift-based deformable 3D convolutions (SDfConv3D) to simultaneously perform spatial deformation and temporal alignment. A residual learning strategy is used to improve the convergence of the model training process, prevent gradient vanishing, and enhance learning efficiency. Furthermore, due to the use of a pure convolutional computational structure, the input and output blocks are of the same size, and the number of input and output features is kept consistent to facilitate subsequent hardware design.

[0103] In the spatiotemporal feature fusion stage, the obtained Niz frame alignment features are further aggregated in the temporal dimension through the temporal average pooling function (TAP), and then the 2D convolution residual block ResC2D built by the depth-separable 2D convolution layer is responsible for capturing the spatial features, such as Figure 1c shown.

[0104] The HR frame reconstruction stage propagates the spatial features to the upsampling module to reconstruct the VSR result, such as Figure 1d Considering the high structural similarity between the input and output sequences, the present invention introduces a residual learning strategy based on the prior frame to replace the pixel-by-pixel interpolation in the traditional residual connection. The basic principle is to copy the LR reference frame s on the residual connection path based on the prior frame. 2 times and concatenate into a sequence In this way, I z ′ will become the prior frame diagram of the final SR frame in the upsampling module. This prior frame diagram contains the low-frequency components in the original LR frame. Compared with the mainstream VSR method that pre-amplifies the LR reference frame through some basic interpolation, its main advantages are: 1) The residual learning strategy based on the prior frame can effectively avoid repeated and costly pixel-by-pixel interpolation; 2) The post-upsampling process can avoid performing a large number of convolutions in the HR feature frame. Based on these two advantages, the residual learning strategy based on the prior frame can ensure that the present invention effectively saves computing resources in hardware design without sacrificing reconstruction quality.

[0105] 1.2 Shift-based Deformable 3D Convolution (SDfConv3D)

[0106] The operators in conventional CNN are mainly multiplication operations. Although dense multiplication can give complex CNN models powerful fitting capabilities, it limits their deployment on resource-limited devices. In order to solve this problem, existing research uses low-cost shift and addition operations to replace heavy multiplication operations, and achieves a good trade-off between efficiency and accuracy. Based on this, the present invention proposes a hardware-friendly SDfConv3D, which contains a shift pointwise 3D convolutional layer (ShiftPointwise 3D Convolutional Layer, SPwConv3D), a shift bilinear interpolation layer (ShiftBilinearInterpolation Layer, SBilinear), and some lightweight ordinary separable 3D convolution layers to reduce the resource requirements of multiplication. Figure 2 As shown in Figure 3, SDfConv3D mainly consists of three stages: offset generation, spatial deformation, and feature extraction.

[0107] 1.2.1 Offset Generation

[0108] Since SDfConv3D is executed at block granularity, for the i-th input block The corresponding predicted offset can be obtained For example, for pixels Offset generation process H Ψ Expressed as:

[0109]

[0110] in, represents a three-dimensional sampling grid of size Nkx×Nky×Nkz=3×3×3, and The number of channels for the offset is Tix×Tiy represents the spatial size of each input block. Ψ The specific structure of Figure 2 As shown, it mainly consists of a series of cascaded ordinary separable 3D convolutional layers and an SPwConv3D.

[0111] For SPwConv3D, its weight Expressed as a power of 2:

[0112]

[0113] Where s and p represent sign flip and shift respectively, and Nof represents the number of output feature maps. Therefore, the complete calculation process of SPwConv3D can be expressed as shift and addition:

[0114]

[0115] in, and They represent the mth input feature map and the jth output feature map respectively, and * represents the convolution operation.

[0116] 1.2.2 Spatial deformation

[0117] To further save computational cost, SDfConv3D performs spatial deformation and maintains the structure of the original time series, so Δz = 0. Specifically, due to the spatially varying offset Ψ i May be a non-integer, so the coordinates of each element in the deformed receptive field Expressed as:

[0118]

[0119] in, Represents a 3D sampling grid The coordinates of the elements in , and They are and In order to synthesize accurate features based on the above coordinates, a bilinear interpolation function G(·) is usually introduced, such as Figure 3a As shown. Mathematically, G(·) is expressed as:

[0120]

[0121] The interpolation coefficient vector U and sampled pixel vector V of each feature map are expressed as:

[0122]

[0123] in, and All parameters are shared to avoid repeated calculations.

[0124] Considering that completing a 3D convolution operation usually requires Nkx×Nky×Nkz multiplications, if traditional bilinear interpolation is used, the number of multiplications will increase to 5 times the original number. A simple way to deal with it is to express U in a similar form as a power of 2, but the bilinear interpolation coefficient is essentially different from the randomly initialized weight, that is, it has a fixed value range [0,1], and calculating the power of 2 will cause the interpolation coefficient to have a large approximation error in the interval [0.5,1), while the approximation error in other intervals is smaller. In order to solve this problem, the present invention proposes SBilinear to replace the traditional vector dot multiplication, and effectively controls the approximation error by accumulating shift values ​​retrieved from multiple predefined codebooks. As Figure 3bAs shown, the address converter converts the integer part of the deformed coordinate into an available address index to sample the pixel vector V, and the coefficient generator is used to convert the decimal part of the non-integer coordinate into the coefficient U of SBilinear, and each element u of U d It is expressed as L coefficients of powers of 2 using the following formula:

[0125]

[0126] in Represents a codebook set, each codebook contains M candidate elements, Ω d Represented as u d The search is done on a power-of-two index, so in the shifted bilinear interpolation layer SBilinear, G(·) with dense vector dot multiplication operations is simplified to a shift and addition operation.

[0127] 1.2.3 Feature Extraction

[0128] Based on F i and The shift-based deformable 3D convolution SDfConv3D is expressed as:

[0129]

[0130] in, and They are the weights of the DwConv3D layer and the PwConv3D layer, O i It is the output feature of the shift-based deformable 3D convolution SDfConv3D.

[0131] 1.3 Block Decoupled Computing Strategy (TDS)

[0132] Considering that inter-frame alignment consumes a lot of hardware resources, the present invention proposes TDS to implement sub-region processing, thereby effectively removing memory limitations. Figure 1a As shown in the figure, TDS decomposes the input frame sequence into multiple overlapping blocks in the spatial dimension. The feature alignment stage is performed in blocks, and only one block is processed in each loop. The advantages of this approach are mainly reflected in two aspects. On the one hand, sub-region processing can effectively prevent deformable convolution from accessing pixels beyond the current block range. At the same time, the overlapping strategy can effectively maintain the visual consistency of the overall structure and the fidelity of details. On the other hand, in terms of hardware design, TDS provides a huge opportunity for designing partially fused data flows, so that computationally intensive ResD3D blocks can be integrated into the same computing core, and partially fused data flows can effectively reduce the huge energy consumption and off-chip memory bandwidth caused by large amounts of data movement.

[0133] The present invention also provides a video super-resolution system based on a deformable 3D convolutional network, wherein the system hardware architecture includes the following contents:

[0134] 2.1 Overall Hardware Architecture and Data Flow

[0135] The present invention develops a hardware architecture for ALDNet inference. Figure 4 The overall architecture diagram is described, which mainly includes the critical processing core (CPC), auxiliary processing core (APC) and global controller. Figure 4 The arrows with different number labels highlight different data types, such as activation values, weights, shift values, and control signals.

[0136] CPC specifically performs shift-based deformable 3D convolution SDfConv3D between layers in a fully pipelined manner. To avoid the additional memory overhead of the intermediate partial sum (psum), the present invention integrates the data flow of channel-by-channel convolution and point-by-point convolution, and develops some specialized processing modules to accelerate Figure 2 Specifically, the offset generation process is modularized into A-type Standard3D Convolution Module (Conv3D-Module A) and B-type Standard3D Convolution Module (Conv3D-Module B), while the Deformable 3D Convolution Module (DfConv3D-Module) is responsible for performing spatial deformation and 3D feature extraction.

[0137] To cooperate with CPC, APC is used to support the remaining convolution operations of ALDNet, including the depth-separable 3D convolution with only one input channel in the first layer and some depth-separable 2D convolutions. APC uses very little hardware resources and can be reconfigured into two working modes through a switch flag, working in a time-division multiplexing manner.

[0138] In the global controller, the CPC and APC controllers coordinate the execution of different computations by the two compute cores. Direct memory access (DMA) manages data exchange between on-chip and off-chip memory by generating descriptors. The distribution module allocates data to designated on-chip memory, while the aggregation module rearranges the output data for subsequent operations.

[0139] 2.2 Deformable 3D Convolution Module (DfConv3D-Module):

[0140] Deformable 3D convolution has two characteristics. On the one hand, the utilization of each input data is not uniform because the sampling position is random. On the other hand, the offset depends entirely on the input data and keeps changing. To solve this problem, the present invention introduces a block-level interleaved storage scheme in DfConv3D-Module to prevent irregular and dynamic memory access. The overall architecture of DfConv3D-Module is shown in the figure. Figure 5 shown.

[0141] 2.2.1 Memory Management

[0142] The input block is a four-dimensional tensor of size Tix×Tiy×Niz×Nif. In order to improve throughput, the present invention makes full use of the parallelism in the time and input channel dimensions. Therefore, the input buffer contains Piz input storage banks (banks) for storing multiple video frames, and the offset buffer also contains Piz storage banks, and each address contains Nif channel values ​​for each pixel. The weights are sent to the DwConv3D weight buffer and the PwConv3D weight buffer through the distribution module. Finally, the output result is sent to the output buffer with the same arrangement pattern as the input buffer, waiting to be read later. In order to perform residual learning, when the second SDfConv3D is executed, the output buffer usually pre-fetches the input of the identity map, so that the residual value generated by the DfConv3D-Module can be accurately fused with the corresponding identity map input value.

[0143] 2.2.2 Offset value decomposition

[0144] Based on the above memory management mechanism, Noff×Piz offsets can be accessed in each cycle. In order to ensure that SDfConv3D can quickly perform spatial deformation, the present invention chooses to expand the spatial dimension of the 3D convolution kernel, that is, the parallelism of the 3D convolution kernel is Pks=Nkx×Nky. Specifically, Figure 6aAs shown, since the original order of the offsets does not meet the design requirements, the present invention first rearranges the offsets and divides them into Nkz groups, each group containing Pks×Niz pairs of coordinate offsets. In this way, a complete 3D convolution can be completed within Nkz cycles. Then, according to the scale of fixed-point quantization, the offsets are divided into a set of integer values ​​P[i] and fractional values ​​Q[i], i=0,1,...,Pks×Piz, and then they are assigned to Figure 6b Address translator and Figure 6c For example, if Δx 00 and Δy 00 The integer part is assigned to P[i][0] and P[i][1] respectively, and the fractional part is assigned to Q[i][0] and Q[i][1] respectively. Then the address converter and coefficient generator can generate the corresponding sampling index A[i] and SBilinear coefficient C[i] respectively. Finally, a shift quantizer is constructed based on formula (7) to convert C[i] into a codebook index, reducing the use of multipliers through some low-cost shift and addition operations.

[0145] 2.2.3 Block-level interleaved storage solution

[0146] In order to handle irregular memory access patterns and improve input reuse of deformable convolution, a block-level interleaved storage scheme is designed as an additional memory hierarchy to cache sample values ​​and perform parallel shift interpolation with low memory resource overhead. Figure 7 As shown, the block-level interleaved storage scheme mainly includes two groups of register arrays (RA), named RA0 and RA1, which are controlled by two signals respectively. In order to obtain high throughput, each RA can accommodate Nb×Nb×Pks data blocks and work in ping-pong mode. "0" means that the old data can be overwritten by the new data from top to bottom, which is called the write state. "1" means that the filled RA sends the data from left to right to the subsequent computing module, which is called the read state. Since SBilinear usually uses four adjacent features to calculate the deformed features, that is, Nb=4, it takes Nb cycles to continuously retrieve adjacent features, which seriously affects the computing efficiency. Finally, the present invention configures the operation of the above two working states to be orthogonal directions to execute SBilinear in parallel. The calculation process of the block-level interleaved storage scheme is as follows:

[0147] 1) Initial stage: According to the sampling index, each cycle retrieves Pks of input data of size Nif×Piz. In order to expand the input channel dimension, each group of input data is divided into Nb data blocks of size Pif×Piz and written into RA1 row by row. Represents the parallelism of the input feature channel dimension. After Nb cycles, the state signal of RA1 is set to "1".

[0148] 2) Phase 0 (State 01): In each cycle, new Pks×Nb data blocks will be loaded into RA0 and overwrite the old data in the register array by row. At the same time, Pks×Nb data blocks will be sent from RA1 to the subsequent calculation array by column.

[0149] 3) Phase 1 (State 10): After every Nb cycles, the operating states of RA0 and RA1 are swapped. Similarly, RA1's old data is overwritten row by row every cycle, while RA0 pops Pks × Nb data blocks column by column. After another Nb cycles, the entire process returns to Phase 0.

[0150] 4) Final phase: Once all pre-loaded weights have been transferred and calculated, processing of these inputs stops. Afterwards, a new round of weights and input blocks are transferred on-chip and the above process is repeated.

[0151] 2.2.4 Computational Arrays at Different Layers

[0152] The computational logic in DfConv3D-Module mainly includes some computational arrays for SBilinear, DwConv3D, and PwConv3D operations. To simplify the design process, these arrays are designed to be similar in high-level connection mode, the difference being the micro-architecture of their computational units, such as Figure 8a As shown, computations at different levels can be implemented by simply instantiating this general structure and configuring different design parameters. Specifically, to enhance data reuse and reduce resource overhead, input data is shared between columns of computational units, and weights are preloaded on different computational units. Following a fixed-weight data flow, the results derived by the computational units are reduced in real time through an adder tree after all computations are completed and delivered to partial sum registers. For a clearer description, the present invention defines this computational array using the number of rows, columns, and clusters (Nr, Nc, Ng).

[0153] The structural parameters of the SBilinear calculation array are (Nr, Nc, Ng) = (Piz, Nb, Pks). Figure 8b As shown, each SBilinear computation unit (SBCU) includes 2Pif shifters and Pif adders for parallel processing in the input channel dimension. In this case, the sample values ​​and SBilinear coefficients serve as input and weights, respectively. The structural parameters of the DwConv3D computation array are (Nr, Nc, Ng) = (Pif, Pks, 1), and the structural parameters of the PwConv3D computation array are (Nr, Nc, Ng) = (Pof, Pif, 1). The computation units of these two computation arrays are the same, both containing Piz multipliers, thereby parallelizing computation in the time dimension.

[0154] 2.3 Standard 3D convolution calculation module (Conv3D-Module):

[0155] Conv3D-Module is responsible for performing the separable 3D convolution operation in the SDfConv3D offset generation stage. It has two main types, namely Conv3D-Module A and Conv3D-Module B, which are used to calculate the standard separable 3D convolution and the special version with SPwConv3D layer respectively. Figure 9a As shown in Figure 2, the DwConv3D processing element PE array helps reduce data movement, thereby avoiding the extra padding of a large number of zero elements in the time dimension. Each PE (Processing Element, PE) array contains Pif PE clusters for parallel processing of input channel data. Each PE cluster contains Nkz×Piz-2 PEs, and each PE is responsible for performing Nky ​​multiply-accumulate (MAC) operations, as shown in Figure 2. Figure 9b As shown. In addition, the weights are reused in each row of the PE array. In the time dimension, Piz input data are simultaneously transmitted to each PE cluster and broadcast in the diagonal direction. Subsequently, Piz×Pif partial sums are accumulated along each column and directly transmitted to the subsequent cascaded PwConv3D or SPwConv3D layer. The high-level connection method of the PE array of the processing elements of PwConv3D and SPwConv3D is the same as Figure 8a Consistent with the above, their computing units are also composed of Piz multipliers or shifters, thus performing calculations in parallel in the time dimension.

[0156] 2.4 Auxiliary Processing Core (APC):

[0157] APC is designed to perform other operations in addition to SDfConv3D in ALDNet, including depth-wise separable 3D convolution layers with only one input channel and some depth-wise separable 2D convolution layers. To save hardware resources and match the processing speed of CPC, it alternates between these two computation modes in a time-division multiplexing manner.

[0158] Regarding storage for the auxiliary processing core (APC), this invention uses an input buffer to store 3D or 4D input tensors. For 3D input tensors, each address contains all input channels for each pixel. For 4D input tensors, since the number of input channels is one, to unify the storage format of different convolution operations, the 3D convolution kernel processing in the time dimension is switched to the input channel dimension. Figure 10aAn example simplified to a one-dimensional convolution is provided, demonstrating the process of sliding a 3D convolution with only one input channel along the time dimension. Since the stride of the convolution kernel is one, the number of features in each pixel is flattened to Nkz×Niz. For residual learning based on the prior box, when the APC executes the last layer, the pixels of the prior box are preloaded to the corresponding address of the output buffer. The prior box-based residual learning is then performed by combining the pixels of the prior box with the convolution result. Finally, the video super-resolution (VSR) result can be directly transferred to the same address of the output buffer without any additional memory reshuffling.

[0159] In terms of the calculation of the auxiliary processing core APC, it includes a channel-by-channel convolution layer calculation array and a point-by-point convolution layer calculation array. The channel-by-channel convolution layer calculation array is used to calculate DwConv2D and DwConv3D with only one input channel, and the point-by-point convolution layer calculation array is used to calculate PwConv2D and PwConv3D. A fixed data stream is designed for the channel-by-channel convolution layer calculation array, and Pox row data is output in parallel channel by channel. To execute PWConv3D and PWConv2D, the present invention develops a reconfigurable calculation array, whose structure is as follows: Figure 10b The reconfigurable point-by-point convolutional layer computation array includes two computation modes, configured by a switch flag. One mode executes PwConv2D by expanding the input and output channel dimensions. The other mode executes PwConv3D, a point-by-point 3D convolution layer. PwConv3D does not perform accumulation operations on the input channels. The partial sums psum of the 3D convolutions at different positions are sent directly to the 3D-Psum register. To save hardware resources, each computation unit (PWCU) in the point-by-point convolutional layer computation array contains only one multiplier.

[0160] Example:

[0161] The video super-resolution method based on a deformable 3D convolutional network is evaluated: the present invention uses the Vimeo-90k dataset, the number of frames of the input video data is Niz=7, the number of input channels Nif of each convolutional layer, except for the first convolutional layer which is Nif=1, the remaining convolutional layers are all Nif=36; the number of output channels Nof of each convolutional layer, except for the last convolutional layer which is Nof=4, the remaining convolutional layers are all Nof=36. In addition, ALDNet is optimized based on quantization-aware training, and the weights and activation values ​​are fixed-point quantized to 8bit and 12bit. The algorithm performance of the present invention is then tested on some VSR and SISR datasets, and compared with the latest FPGA-based SR method. The experimental results are shown in Table 1. It can be seen that the present invention achieves optimal performance in both PSNR and SSIM indicators.

[0162] Table 1 Quantitative evaluation of the algorithm performance of the present invention under the PSNR / SSIM index

[0163]

[0164] The video super-resolution system based on the deformable 3D convolutional network is evaluated: different structural parameters (Nr, Nc, Ng) are set for different operators in the DfConv3D-Module. For SBilinear, its structural parameters are (Nr, Nc, Ng) = (Piz, Nb, Pks) = (7, 4, 9), and the number of shift registers and adders in its computing unit SBCU are 18 and 9 respectively; for DwConv3D, its structural parameters are (Nr, Nc, Ng) = (Pif, Pks, 1) = (9, 9, 1), and its computing unit contains Piz = 7 pixel-by-pixel multipliers; for PwConv3D, its structural parameters are (Nr, Nc, Ng) = (Pof, Pif, 1) = (12, 9, 1), and its computing unit contains Piz = 7 pixel-by-pixel multipliers. In Conv3D-Module A / B, the PE array of DwConv3D is as follows: Figure 9a As shown, the PE array is assumed to contain Pif = 9 PE clusters, each consisting of Nkz × Piz-2 = 3 × 7-2 = 19 PEs. In the A-Core, DwConv processing uses a fixed row data flow, outputting Pox = 4 rows of data in parallel per channel, and fully expanding the spatial dimensions of the 2D convolution kernel. For PwConv, the parallelism of both input and output channels is Pif = Pof = 12, and each computational unit contains one multiplier.

[0165] This invention was implemented on an Intel Stratix 10 GX FPGA board and compared with a GPU, FPGA-based DCN, and SR hardware accelerators. The experimental results are shown in Table 2. As can be seen, compared to the GPU, the invention achieves 11.67 times and 256.46 times improvements in throughput and energy efficiency, respectively. Compared to other FPGA-based hardware accelerators, the invention achieves 2.75 times and 1.63 times improvements in throughput and energy efficiency, respectively. Furthermore, it achieves a real-time processing speed of 32 FPS on the Vid4 dataset test.

[0166] Table 2 Comparison of performance and energy efficiency between the present invention and the prior art

[0167]

[0168] The present invention provides a method and system for video super-resolution based on a deformable 3D convolutional network. There are many methods and approaches to implement this technical solution. The above is only a preferred embodiment of the present invention. It should be noted that those skilled in the art may make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications should also be considered within the scope of protection of the present invention. Any components not specified in this embodiment may be implemented using existing technologies.

Claims

1. A video super-resolution method based on a deformable 3D convolutional network, characterized in that: include: The low-resolution LR video reference frame and its Niz-1 adjacent supporting frames Input the deformable 3D convolutional network, and after processing by the deformable 3D convolutional network, a high-resolution HR video frame is obtained. Among them, Nix, Niy, Nif are the height, width and number of channels of the low-resolution LR video frame image respectively, and s is the upsampling factor. Represents a low-resolution LR video reference frame Niz-1th adjacent supporting frame, represents the real number space of sNix×sNiy×Nif dimensions; The processing of the deformable 3D convolutional network includes the following stages: shallow feature extraction, implicit feature alignment, spatiotemporal feature fusion and high-resolution HR video frame reconstruction; The implicit feature alignment includes: inputting Niz frame features and performing implicit feature alignment according to a block decoupled computation strategy (TDS), specifically by partitioning the intermediate feature sequence obtained from the shallow feature extraction stage, thereby ensuring that all layers of the implicit feature alignment stage are performed at the granularity of blocks rather than the entire intermediate feature sequence; The implicit feature alignment stage includes two or more deformable 3D convolution residual blocks ResD3D. Each deformable 3D convolution residual block ResD3D is stacked based on a shifted deformable 3D convolution SDfConv3D, which is used to simultaneously perform spatial deformation and temporal alignment. The residual learning strategy is used to improve the convergence of the model training process. The input and output blocks have the same size and the number of input features and output features is consistent. The number of input features and output features refers to the number of feature channels of the input block and the number of feature channels of the output block. The Niz frame alignment feature is finally obtained through implicit feature alignment. The high-resolution HR video frame reconstruction includes: propagating spatial features to the upsampling module to reconstruct the video super-resolution VSR result, introducing a residual learning strategy based on the prior frame to replace the pixel-by-pixel interpolation in the traditional residual connection: copying the low-resolution LR video reference frame on the residual connection path based on the prior frame 2 times and concatenate into a sequence I z ’ will become the priori block diagram of the final super-resolution SR video frame in the upsampling module, and the priori block diagram contains the low-frequency components in the low-resolution LR video reference frame; The shift-based deformable 3D convolution SDfConv3D includes a shifted point-by-point 3D convolution layer SPwConv3D, a shifted bilinear interpolation layer SBilinear and two or more ordinary separable 3D convolution layers.

2. The method according to claim 1, characterized in that The shallow feature extraction includes: extracting spatiotemporal information using a depth-separable 3D convolutional layer, wherein the depth-separable 3D convolutional layer includes a channel-by-channel 3D convolutional layer DwConv3D and a point-by-point 3D convolutional layer PwConv3D.

3. The method according to claim 2, characterized in that The block decoupling-based calculation strategy TDS includes: decomposing the Niz frame features into two or more overlapping blocks in the spatial dimension, and the implicit feature alignment stage is performed in blocks, and only one block is processed in each cycle.

4. The method according to claim 3, characterized in that The spatiotemporal feature fusion includes: further aggregating the obtained Niz frame alignment features in the time dimension through the temporal average pooling function TAP, and then the ResC2D module built by the depth-separable 2D convolutional layer is responsible for capturing the spatial features.

5. The method according to claim 4, characterized in that The processing of the shift-based deformable 3D convolution SDfConv3D includes: Step a1, offset generation: The shift-based deformable 3D convolution SDfConv3D is performed at a block granularity. For the i-th input block Get the corresponding predicted offset For pixels Offset generation process H Ψ Expressed as: in, It's a pixel In the spatial dimension coordinates, It's a pixel In the time dimension, It's a pixel The nth pair of coordinate offsets in the spatial dimension, represents a three-dimensional sampling grid of size Nkx×Nky×Nkz=3×3×3, and Nkx and Nky represent the width and height of the three-dimensional sampling grid in the spatial dimension, Nkz is the number of frames of the three-dimensional sampling grid in the time dimension, and the number of channels of the offset is Tix×Tiy represents the spatial size of each input block, Tix represents the width of the input block, and Tiy represents the height of the input block; the offset generation process H Ψ It consists of a series of cascaded ordinary separable 3D convolution layers and a shifted point-by-point 3D convolution layer SPwConv3D; For the shifted point-by-point 3D convolution layer SPwConv3D, its weight Expressed as a power of 2: Where s and p represent sign flip and shift respectively, Nof represents the number of output feature maps, and sgn represents the sign function; The complete computation process of the shifted point-by-point 3D convolution layer SPwConv3D is expressed as follows using shift and addition: in, and They represent the mth input feature map and the jth output feature map respectively, and * represents the convolution operation; represents the weight of the mth input channel and jth output channel of SPwConv3D; s j,m Represents the symbol value of the mth input channel and jth output channel; Step a2, spatial deformation: The deformed receptive field is obtained by formula (1). The deformed receptive field includes the following parts: Current pixel Coordinates By 3D sampling grid get The current pixel The normal coordinate offset of the center Indicates the current pixel The dynamic offset predicted in the spatial dimension is used to achieve spatial deformation and temporal alignment of SDfConv3D; The coordinates of each element in the deformed receptive field Expressed as: in, is the current pixel The nth pair of coordinate values ​​in the spatial dimension; is the current pixel The nth coordinate value in the time dimension; Represents a 3D sampling grid The coordinates of the elements in , Represents a 3D sampling grid The spatial dimension coordinate value of the nth grid; Represents a 3D sampling grid The time dimension coordinate value of the nth grid; Each pair of coordinates corresponds to and They are The decimal part and the decimal part of ; Introducing the bilinear interpolation function G(·), G(·) is expressed as: Where T represents matrix transpose; Convert the integer part and decimal part of the deformed coordinates into the interpolation coefficient vector U and sampling pixel vector V of each feature map respectively: in, and They are all shared parameters; In order to reduce the number of multiplication operations in formula (5), each element u of U d It is expressed as L coefficients of powers of 2 using the following formula: in represents the codebook set, Represents the lth codebook set, each codebook contains M candidate elements, Ω d Represented as u d The searched power-of-two index, in the shifted bilinear interpolation layer SBilinear, G(·) with dense vector dot multiplication operations is simplified to only shift and add operations Step a3, 3D feature extraction: based on F i and The shift-based deformable 3D convolution SDfConv3D is expressed as: in, and They are the weights of the channel-by-channel 3D convolution layer DwConv3D and the weights of the point-by-point 3D convolution layer PwConv3D, O i It is the output feature of the shift-based deformable 3D convolution SDfConv3D.

6. A video super-resolution system based on a deformable 3D convolutional network, characterized in that: It includes the key processing core CPC, auxiliary processing core APC, global controller, distribution module and aggregation module; The key processing core CPC performs shift-based deformable 3D convolution SDfConv3D in a fully pipelined manner; the key processing core CPC includes a class A standard 3D convolution module Conv3D-Module A, a class B standard 3D convolution module Conv3D-ModuleB, and a deformable 3D convolution module DfConv3D-Module; The class A standard 3D convolution module Conv3D-Module A and the class B standard 3D convolution module Conv3D-Module B are used to perform offset generation; The deformable 3D convolution module DfConv3D-Module is used to perform spatial deformation and 3D feature extraction; The auxiliary processing core APC is used to support the remaining convolution operations in the deformable 3D convolution network except the shift-based deformable 3D convolution SDfConv3D, and the remaining convolution operations include depth-separable 3D convolution and depth-separable 2D convolution with only one input channel in shallow feature extraction; In the global controller, the CPC controller and APC controller coordinate the key processing core CPC and the auxiliary processing core APC to perform different calculations respectively; direct memory access (DMA) generates descriptors to manage the data interaction between the key processing core CPC and the auxiliary processing core APC and the off-chip; The distribution module is responsible for allocating data to the key processing core CPC and the auxiliary processing core APC, and the aggregation module is used to rearrange the output data and send it off-chip; A block-level interleaved storage scheme is introduced in the deformable 3D convolution module DfConv3D-Module, specifically including: Step b1, memory management: the deformable 3D convolution module DfConv3D-Module can call on-chip storage resources, and the on-chip storage resources include input buffer, offset buffer, DwConv3D weight buffer, PwConv3D weight buffer, and output buffer; Input block of shift-based deformable 3D convolution SDfConv3D It is a four-dimensional tensor of size Tix×Tiy×Niz×Nif. The input buffer contains Piz input storage banks for storing Piz video frames. The offset buffer also contains Piz storage banks, and each address contains all channel values ​​of each pixel. The weights are sent to the DwConv3D weight buffer and the PwConv3D weight buffer through the distribution module; finally, the output result is sent to the output buffer with the same arrangement pattern as the input buffer, waiting to be read; Step b2, offset value decomposition: Based on the memory management of step b1, each cycle can access Noff×Piz offsets; expand the spatial dimension of the 3D convolution kernel, that is, the parallelism Pks of the 3D convolution kernel is Pks=Nkx×Nky; rearrange the offsets and divide them into Nkz groups, each group contains Pks×Niz coordinate offsets, in this way, a complete 3D convolution can be completed within Nkz cycles; then, according to the scale of fixed-point quantization, the offsets are divided into a group of integer values ​​P[i] and fractional values ​​Q[i], i=0,1,...,Pks×Piz, and then P[i] and Q[i] are assigned to the address converter and coefficient generator respectively. The address converter converts the integer value P[i] into an address index to sample the pixel vector, and the coefficient generator is used to convert Q[i] into the coefficient of the shifted bilinear interpolation layer SBilinear. Finally, a shift quantizer is constructed based on formula (7) to convert the shifted bilinear interpolation layer SBilinear coefficients into codebook indices; Step b3: Design a block-level interleaved storage scheme as an additional memory hierarchy to cache sample values ​​and perform parallel shift interpolation with low memory resource overhead. The block-level interleaved storage scheme includes two register arrays RA, named RA0 and RA1, which are controlled by two signals respectively. To achieve high throughput, each register array RA can accommodate Nb×Nb×Pks data blocks and operate in ping-pong mode. 0 indicates that old data can be overwritten by new data from top to bottom, which is called the write state; 1 indicates that the full register array RA reads data from left to right, which is called the read state. The write state and the read state are configured to run in orthogonal directions to execute the shifted bilinear interpolation layer SBilinear in parallel on more than two input channels at the same time; the calculation process of the block-level interleaved storage scheme is as follows: Step b3-1, initial stage: according to the sampling index, each cycle retrieves Pks of input data of size Nif×Piz, each group of input data is divided into Nb data blocks of size Pif×Piz, and written into the register array RA1 by row, Represents the parallelism of the input feature channel dimension; after Nb cycles, the state signal of the register array RA1 is set to 1; Step b3-2, Phase 0: Each cycle, new Pks×Nb data blocks will be loaded into register array RA0 and overwrite the old data in the register array by row. At the same time, Pks×Nb data blocks will be read from RA1 by column. Step b3-3, Phase 1: After every Nb cycles, the operating states of register arrays RA0 and RA1 are swapped. The old data in register array RA1 is overwritten row by row every cycle, while register array RA0 pops out Pks × Nb data blocks column by column. After another Nb cycles, the entire process returns to Phase 0. Step b3-4, final stage: Once all preloaded weights have been transferred and calculated, processing of the current input block stops; a new round of weights and input blocks are then transferred on-chip, and steps b3-1 to b3-4 are repeated; the weights are those sent to the DwConv3D weight buffer and the PwConv3D weight buffer by the distribution module in step b1; Step b4: Calculation arrays at different layers: The computational logic in the deformable 3D convolution module includes the computational arrays of the shifted bilinear interpolation layer SBilinear and the channel-by-channel 3D convolution layer DwConv3D and the point-by-point 3D convolution layer PwConv3D in 3D feature extraction. The computational array includes computational units and partial and memory. The computational units are interconnected and stacked to form a three-dimensional architecture. The partial and memory are used to temporarily store the results of the computational units. The input data is shared between the computational unit columns, and the weights are preloaded on different computational units. According to the data flow with fixed weights, after all calculations are completed, the results derived by the computational units are reduced in real time through the addition tree and transported to the partial and registers. The computational array is defined by the number of rows, columns, and clusters (Nr, Nc, Ng), where Nr, Nc, and Ng represent the number of computational units in the row, column, and cluster dimensions of the computational array, respectively. The structural parameters of the shifted bilinear interpolation layer SBilinear calculation array are (Nr, Nc, Ng) = (Piz, Nb, Pks); each calculation unit includes 2Pif shifters and Pif adders, which process in parallel in the input channel dimension. At this time, the sample value and the shifted bilinear interpolation layer SBilinear coefficient are used as input and weight respectively. The structural parameters of the channel-by-channel 3D convolution layer DwConv3D calculation array are (Nr, Nc, Ng) = (Pif, Pks, 1), and the structural parameters of the point-by-point 3D convolution layer PwConv3D calculation array are (Nr, Nc, Ng) = (Pof, Pif, 1); the calculation units of these two calculation arrays are the same, both containing Piz multipliers, so that parallel calculation is performed in the time dimension; The class A standard 3D convolution module Conv3D-Module A and the class B standard 3D convolution module Conv3D-Module B are respectively used to calculate the standard separable 3D convolution in the offset generation stage and the special version with the SPwConv3D layer.

7. The system according to claim 6, characterized in that The auxiliary processing core APC is reconfigured into two working modes through a switch flag and operates in a time-division multiplexing manner; the two working modes are respectively a depth-separable 3D convolution layer with only one input channel and a depth-separable 2D convolution layer; In terms of storage of the auxiliary processing core APC, it includes an input buffer, a DwConv2D weight buffer, a PwConv2D weight buffer and an output buffer. A 3D or 4D input tensor is stored through an input buffer. For storing 3D input tensors, each address contains all input channels for each pixel; for storing 4D input tensors, since the number of input channels is one, the processing of the 3D convolution kernel in the time dimension is switched to the input channel dimension, and the number of features in each pixel is flattened to Nkz×Niz; for the residual learning strategy based on the prior frame, when the auxiliary processing core APC executes the last layer, that is, when executing the last PwConv2D of the deformable 3D convolutional network, the pixels of the prior frame are preloaded to the corresponding address of the output buffer, and residual learning based on the prior frame is performed by combining the pixels of the prior frame and the convolution result; finally, the video super-resolution VSR result can be directly transferred to the same address of the output buffer; In terms of the calculation of the auxiliary processing core APC, it includes a channel-by-channel convolution layer calculation array and a point-by-point convolution layer calculation array. The channel-by-channel convolution layer calculation array is used to calculate DwConv2D and DwConv3D with only one input channel, and the point-by-point convolution layer calculation array is used to calculate PwConv2D and PwConv3D. A row-fixed data flow is designed for the channel-by-channel convolution layer calculation array, and Pox row data is output in parallel channel by channel. The reconfigurable point-by-point convolution layer calculation array includes two calculation modes and is configured by a switch flag. One mode is to execute PwConv2D by expanding the input and output channel dimensions. The other mode is to execute the point-by-point 3D convolution layer PwConv3D. The point-by-point 3D convolution layer PwConv3D does not calculate the accumulation operation of the input channel. The partial sum psum of the 3D convolution at different positions is directly sent to the 3D-Psum register. Each calculation unit of the point-by-point convolution layer calculation array contains only one multiplier.

Images