Bidirectional optical flow guided high-quality integrated imaging video generation method

Abstract

The invention provides a bidirectional optical flow guided high-quality integrated imaging video generation method, which comprises five processes of low-quality micro-image array sequence acquisition, bidirectional optical flow estimation, optical flow guided feature alignment, space-time super-resolution reconstruction and high-quality three-dimensional video display. According to the method, a low-frame-rate and low-resolution micro-image array sequence is collected through a lens array to serve as input, a precise bidirectional optical flow field is obtained through bidirectional optical flow estimation based on micro-images, and feature self-adaptive alignment is achieved in combination with deformable convolution guided by optical flow. A grid-aligned Swin Transform network is adopted to perform texture and parallax collaborative reconstruction and improve the resolution, and finally, a dynamic three-dimensional video with a high frame rate and high resolution is reconstructed by means of an integrated imaging display system, so that the space-time quality and the three-dimensional display effect of the integrated imaging video are remarkably improved.

Description

A method for generating high-quality integrated imaging video guided by bidirectional optical flow Technical Field

[0001] This invention relates to the field of three-dimensional video display, and more specifically, to a method for generating high-quality integrated imaging video guided by bidirectional optical flow. Background Technology

[0002] With the development of glasses-free 3D display technology, integrated imaging has become an important method for realizing dynamic true 3D video display. This technology records scene light field information through a micro-image array. Each micro-image independently captures content from different perspectives, and through optical reconstruction, it can present a stereoscopic scene with continuous parallax. However, integrated imaging video faces a fundamental contradiction in practical applications: frame rate and resolution are mutually constrained. The total number of pixels in the micro-image array limits spatial resolution, while dynamic smoothness requires a high frame rate. Under current hardware conditions, improving spatial resolution often requires reducing the frame rate, leading to motion stuttering; pursuing a high frame rate requires sacrificing resolution, causing texture blurring and depth perception degradation. This contradiction severely restricts the practical generation and application of high-quality dynamic 3D video.

[0003] To improve the quality of integrated imaging videos, existing methods mainly focus on spatial resolution enhancement, texture restoration using optical field geometry, or improving single-frame sharpness based on deep learning. However, most of these methods neglect temporal correlation in video, making it difficult to effectively handle motion blur and visual discontinuity issues at low frame rates. Meanwhile, research on temporal frame rate enhancement is relatively limited. The challenge lies in the unique gridded optical structure of micro-image arrays, where viewpoint jumps and boundary discontinuities exist between adjacent micro-images. Traditional frame interpolation methods based on natural image assumptions are prone to producing boundary artifacts and parallax distortion, compromising the optical field consistency required for 3D reconstruction.

[0004] In summary, current technologies have not yet been able to synergistically improve spatial resolution and temporal frame rate while maintaining the optical structure and temporal coherence of micro-image arrays. Spatial super-resolution or temporal interpolation alone cannot simultaneously guarantee the sharpness and smoothness of dynamic scenes, resulting in limited 3D reconstruction quality. Therefore, there is an urgent need for a novel processing framework that can integrate the physical characteristics of imaging with the spatiotemporal evolution laws to achieve synergistic enhancement of resolution and frame rate, thereby promoting the practical development of high-quality dynamic 3D displays. Summary of the Invention

[0005] This invention proposes a high-quality integrated imaging video generation method guided by bidirectional optical flow. Its core lies in explicitly modeling the spatiotemporal motion information in a micro-image array sequence through bidirectional optical flow estimation and feature alignment mechanisms. Combined with a grid-aligned attention architecture, it collaboratively improves the spatial resolution and temporal frame rate of the video, thereby achieving high-quality dynamic 3D display while maintaining optical structural consistency. The method first acquires two adjacent low-quality micro-image arrays as input using a lens array. Then, based on the viewpoint continuity of the micro-images, bidirectional optical flow is calculated and inversely mapped to an optical flow field consistent with the micro-image array structure. Next, adaptive alignment between bidirectional frames is achieved in the feature space using optical flow-guided residual offset prediction and micro-image-aware deformable convolution. Finally, temporal fusion and spatial super-resolution reconstruction are performed using aligned features to output high-resolution intermediate frames. High-frame-rate, high-resolution micro-image array sequences are obtained through cyclic interpolation, ultimately achieving smooth and clear 3D dynamic display via an integrated imaging system.

[0006] The method comprises five processes: low-quality micro-image array sequence acquisition, bidirectional optical flow estimation, optical flow-guided feature alignment, spatiotemporal super-resolution reconstruction, and high-quality 3D video display. The specific process is shown in Figure 1.

[0007] The low-quality micro-image array sequence acquisition process involves optically acquiring a three-dimensional scene using a lens array to obtain a micro-image array sequence with limited frame rate and resolution. This sequence reflects the inherent constraints of existing acquisition equipment in the spatiotemporal dimensions and serves as the input data for the process of this invention.

[0008] The bidirectional optical flow estimation process takes two adjacent frames of low-resolution micro-image arrays as input. First, the micro-image array is uniformly divided into independent micro-image units based on the physical size of the micro-images. Then, for each micro-image pair corresponding to a spatial location, the forward and backward optical flows are calculated using a pre-trained optical flow network. Finally, the optical flow fields of all micro-images are reorganized according to the original arrangement rules to obtain a bidirectional optical flow field that strictly matches the structure of the micro-image array.

[0009] The optical flow-guided feature alignment process is shown in Figure 2. Taking the previous frame's micro-image array and its corresponding forward optical flow field as an example, the micro-image array is input into a feature extraction network consisting of five concatenated residual blocks. Each block contains two 3×3 convolutional layers and ReLU activation, outputting 256-dimensional depth features. Subsequently, this feature is concatenated with the forward optical flow field along the channel dimension and fed into a lightweight offset prediction network. This network consists of three 3×3 convolutional layers, outputting residual offsets and modulation masks used to correct the sampling coordinates. Finally, the feature, offset, and modulation mask are input into a deformable convolutional layer with a 3×3 kernel. The feature is resampled based on the offset, and the sampled values ​​are weighted using the modulation mask to obtain features forward aligned to the intermediate time step. The next frame uses the same mechanism for backward alignment, ultimately obtaining geometrically consistent bidirectional aligned features.

[0010] The spatiotemporal super-resolution reconstruction process is shown in Figure 3. First, the forward alignment features and backward alignment features are concatenated along the channel dimension and fused and dimensionality-reduced through a 1×1 convolutional layer to generate fused features at intermediate time points. Then, this fused feature is passed through a 3×3 convolutional layer for block embedding, mapping it to serialized high-dimensional features. This feature is fed into a backbone network consisting of six cascaded Swin Transformer blocks. Finally, the processed features are upsampled through a sub-pixel convolutional layer to reconstruct a high-resolution intermediate frame micro-image array. The structure of the Swin Transformer block is shown in Figure 4. First, the input features are layer-normalized. Then, a micro-image window attention mechanism is executed, where the attention window size is strictly set to the physical size of the micro-image to ensure alignment with the micro-image boundary. After computation, the window is added to the input through a residual connection. Next, the output is layer-normalized again, and then a two-layer multilayer perceptron is used for feature transformation, which is then added through a residual connection. Then, the output is layer-normalized again, and a cross-micro-image window attention mechanism is executed. This time, the window grid is shifted half a micro-image size to the lower right corner, allowing the attention window to span adjacent micro-image boundaries. After computation, the window is added through a residual connection. Finally, the multilayer perceptron and residual operations are repeated. After feature transformation of six blocks, the shallow embedded features are superimposed and output through a global residual connection.

[0011] The high-quality 3D video display process involves inputting the high-frame-rate, high-resolution micro-image array sequence reconstructed using the aforementioned method into an integrated imaging display system. This system utilizes the multi-view light field information recorded in the micro-image array to optically reconstruct a dynamic 3D scene in space with full parallax, continuous motion, and clear texture, achieving a high-quality naked-eye 3D video viewing experience.

[0012] This invention proposes a bidirectional optical flow-guided method for generating high-quality integrated imaging videos. By combining bidirectional optical flow estimation, optical flow-guided feature alignment, and a spatiotemporal super-resolution reconstruction architecture, it synergistically improves the spatial resolution and temporal frame rate of integrated imaging videos while maintaining the integrity of the micro-image array's optical structure and temporal coherence. This effectively overcomes the shortcomings of existing methods in terms of boundary artifacts, motion blur, and parallax distortion, providing reliable technical support for high-quality dynamic 3D display. Attached Figure Description

[0013] Figure 1 is a flowchart of a high-quality integrated imaging video generation method guided by bidirectional optical flow proposed in this invention.

[0014] Figure 2 is a flowchart of the optical flow-guided feature alignment method.

[0015] Figure 3 is a framework diagram of spatiotemporal super-resolution reconstruction.

[0016] Figure 4 is a schematic diagram of the Swin Transformer block.

[0017] It should be understood that the above figures are only schematic and are not drawn to scale. Detailed Implementation

[0018] The following detailed description of a typical embodiment of the bidirectional optical flow-guided high-quality integrated imaging video generation method proposed in this invention further illustrates the invention. It is necessary to point out that the following embodiments are only used for further illustrative purposes and should not be construed as limiting the scope of protection of this invention. Any non-essential improvements and adjustments made to this invention by those skilled in the art based on the above description still fall within the scope of protection of this invention.

[0019] This invention proposes a method for generating high-quality integrated imaging video guided by bidirectional optical flow, which includes five processes: low-quality micro-image array sequence acquisition, bidirectional optical flow estimation, optical flow-guided feature alignment, spatiotemporal super-resolution reconstruction, and high-quality three-dimensional video display. The specific process is shown in Figure 1.

[0020] The low-quality micro-image array sequence acquisition process utilizes a capture system composed of a lens array and an image sensor to optically record a dynamic 3D scene. This system acquires a low-resolution, low-frame-rate micro-image array sequence, which serves as the initial input data for all subsequent processing steps.

[0021] The bidirectional optical flow estimation process takes two temporally adjacent low-resolution micro-image arrays as input. First, based on the physical arrangement and imaging characteristics of the microlenses, the micro-image array is uniformly divided into multiple independent grid cells according to the standard size of each micro-image, achieving decoupling from the array to individual micro-images. Then, for each pair of micro-images corresponding to a spatial location, a pre-trained high-precision optical flow calculation network is invoked to calculate the forward optical flow from the previous frame to the next frame, and the backward optical flow from the next frame to the previous frame, in parallel. This step ensures that motion estimation is performed only within the continuous optical scene of each micro-image, effectively isolating interference caused by viewpoint jumps. Finally, all independently calculated micro-image optical flow fields are strictly spliced ​​and recombined according to their row and column indices and arrangement rules in the original array, thereby generating forward and backward optical flow fields that perfectly match the spatial dimensions of the input micro-image array.

[0022] The optical flow-guided feature alignment process is shown in Figure 2. Taking the previous frame of micro-image array and its corresponding forward optical flow field as an example, the frame is first input into a depth feature extraction network. This network consists of five sequentially connected residual blocks, each containing two 3×3 convolutional layers. Each layer is followed by a ReLU activation function, ultimately outputting a deep feature map with 256 channels. Subsequently, this depth feature map is concatenated with the forward optical flow field along the channel dimension to form a fusion tensor. This fusion tensor is fed into a lightweight offset prediction network, which consists of three 3×3 convolutional layers. Its output includes residual offsets for fine-tuning the sampling positions and a modulation mask tensor for weighted masking of invalid sampling points. Finally, the original depth feature map, the calculated final sampling offset, and the modulation mask are input together into a deformable convolutional layer. The deformable convolutional layer has a 3×3 kernel size. It dynamically adjusts the position of its sampling grid based on the provided offset and uses a modulation mask to weight the contribution of each sampling point, ultimately outputting forward features precisely aligned to the midpoint of the target time. For the subsequent frame's micro-image array and backward optical flow field, a completely symmetrical process is used to obtain backward aligned features.

[0023] The spatiotemporal super-resolution reconstruction process is shown in Figure 3. First, the forward alignment features and backward alignment features obtained in the previous steps are concatenated along the channel dimension and then fused and dimensionality reduced along the channel dimension through a 1×1 convolutional layer to generate an intermediate-time fused feature containing bidirectional spatiotemporal context. Subsequently, this fused feature is passed through a 3×3 convolutional layer for block embedding and is converted into a serialized high-dimensional feature representation. This feature sequence is input into a backbone network consisting of six identical Swin Transformer blocks cascaded together for deep reconstruction. The specific structure of each Swin Transformer block is shown in Figure 4. First, the input features are normalized, and then window attention calculation within the micro-image is performed. In this stage, the size of the attention window is strictly set to be the same as the physical size of a single micro-image, and the window's grid is precisely aligned with the physical boundary of the micro-image array. After this calculation is completed, it is added to the module input through residual connections. Next, the summed result is normalized again and nonlinearly transformed using a multilayer perceptron with two fully connected layers. The transformed result is then summed again via residual connections. Next, the current feature is normalized, and cross-microimage window attention computation is performed. Here, the attention window's grid is shifted half the size of a microimage in both the row and column directions, allowing the new computation window to span the boundaries of adjacent microimages and enabling information interaction between different perspectives. After computation, the feature is updated again via residual connections. Finally, the steps of layer normalization, multilayer perceptron transformation, and residual summation are repeated. After processing through six such SwinTransformer blocks, the shallow features initially obtained through block embedding are directly superimposed onto the deep features output by the network through a global residual connection. Finally, the enhanced features are upsampled through a subpixel convolutional layer to reconstruct a high-resolution array of intermediate frame microimages.

[0024] The high-quality 3D video display process involves inputting a sequence of high-resolution, high-frame-rate micro-image arrays, generated iteratively using the aforementioned method, into an integrated imaging 3D display system. This system typically includes a high-resolution display panel and a matching lens array. Utilizing the multi-view light field information recorded by each micro-image in the sequence, the system reconstructs a 3D dynamic scene with full parallax, continuous motion, and clear texture in the real space in front of the display panel through optical reconstruction principles, thereby providing users with a high-quality naked-eye 3D viewing experience.

[0025] This invention proposes a bidirectional optical flow-guided method for generating high-quality integrated imaging videos. This method addresses the unique gridded optical structure and spatiotemporal coupling issues of micro-image arrays in integrated imaging videos by designing a complete processing flow. The flow first estimates the optical flow of individual micro-images to obtain a precise motion field that conforms to physical constraints. Then, it utilizes optical flow-guided deformable convolution to achieve flexible and precise alignment of features between adjacent frames. Finally, a Swin Transformer reconstruction network, strictly aligned with the micro-image grid, collaboratively recovers the high-frequency spatial details and coherent temporal motion information of the video sequence. Therefore, the micro-image array sequence generated by this method achieves significant improvements in both spatial resolution and temporal frame rate. After optical reconstruction using an integrated imaging display device, it can present high-quality dynamic 3D videos with rich details and smooth motion.

Claims

1. A method for generating high-quality integrated imaging video guided by bidirectional optical flow, characterized in that, The method includes five processes: low-quality micro-image array sequence acquisition, bidirectional optical flow estimation, optical flow-guided feature alignment, spatiotemporal super-resolution reconstruction, and high-quality 3D video display. In the low-quality micro-image array sequence acquisition process, a lens array is used to optically acquire the 3D scene to obtain a low frame rate and low resolution micro-image array sequence. In the bidirectional optical flow estimation process, two adjacent low-resolution micro-image arrays are decoupled into independent micro-image units, and the forward and backward optical flows are calculated separately and reassembled into a bidirectional optical flow field that strictly matches the array structure. In the optical flow-guided feature alignment process, the depth features of the input frame are extracted, and the residual offset and modulation mask are predicted by combining the bidirectional optical flow field. The features are aligned to the intermediate time step through deformable convolution. In the spatiotemporal super-resolution reconstruction process, the aligned bidirectional features are fused in the temporal domain, and a backbone network based on SwinTransformer is used for texture and disparity co-reconstruction. The high-resolution intermediate frame is then output after upsampling. In the process of high-quality 3D video display, the reconstructed high frame rate and high resolution micro-image array sequence is optically reconstructed through an integrated imaging display system to present a dynamic 3D scene.

2. The method for generating high-quality integrated imaging video guided by bidirectional optical flow according to claim 1, characterized in that, In the acquisition of low-quality micro-image array sequences, a physical capture system consisting of a lens array and an image sensor is used to obtain low-frame-rate, low-resolution micro-image array sequences.

3. The method for generating high-quality integrated imaging video guided by bidirectional optical flow according to claim 1, characterized in that, In bidirectional optical flow estimation, the micro-image array is uniformly divided into multiple independent grid cells according to the physical size of each micro-image. For the two preceding and following micro-image pairs corresponding to each grid position, a pre-trained optical flow computing network is used to compute the forward optical flow from the previous frame to the next frame and the backward optical flow from the next frame to the previous frame in parallel. The optical flow fields of all micro-images are spliced ​​and recombined strictly according to their row and column indices and arrangement rules in the original array to generate forward and backward optical flow fields that perfectly match the spatial size of the input micro-image array.

4. The method for generating high-quality integrated imaging video guided by bidirectional optical flow according to claim 1, characterized in that, In optical flow-guided feature alignment, the input frame is fed into a deep feature extraction network consisting of five sequentially connected residual blocks. Each residual block contains two 3×3 convolutional layers, followed by a ReLU activation function, outputting a 256-channel deep feature map. This feature map is concatenated with the corresponding optical flow field along the channel dimension to form a fusion tensor. This fusion tensor is then fed into a lightweight offset prediction network consisting of three 3×3 convolutional layers, outputting a residual offset tensor for fine-tuning the sampling position and a modulation mask tensor for weighting and masking invalid sampling points. The original deep feature map, the calculated final sampling offset, and the modulation mask are then fed into a deformable convolutional layer with a kernel size of 3×3. The sampling grid position is dynamically adjusted based on the offset, and the contribution of each sampling point is weighted using the modulation mask, outputting features precisely aligned to the target intermediate time step.

5. The method for generating high-quality integrated imaging video guided by bidirectional optical flow according to claim 1, characterized in that, In spatiotemporal super-resolution reconstruction, the aligned forward and backward features are concatenated along the channel dimension and fused and channel-dimension-reduced through a 1×1 convolutional layer to generate intermediate-time fused features containing bidirectional spatiotemporal context. These fused features are then converted into serialized high-dimensional feature representations through a 3×3 convolutional layer for block embedding. This feature sequence is input into a backbone network consisting of six cascaded Swing Transformer blocks. Each Swing Transformer block performs layer normalization on the input features and then performs window attention calculation within micro-images. The attention window size is strictly set to be the same as the physical size of a single micro-image, and the window grid is precisely aligned with the physical boundaries of the micro-image array. The calculation result is added to the module input through residual connections. The added result is layer normalized again and then nonlinearly transformed through a multilayer perceptron containing two fully connected layers. The transformation result is added through residual connections. After layer normalization of the current feature, cross-micro-image window attention calculation is performed. At this time, the grid of the attention window is shifted to the lower right corner in both the row and column directions by half the size of a micro-image, so that the calculation window spans the boundaries of adjacent micro-images. The calculation result is updated again through residual connections. The steps of layer normalization, multilayer perceptron transformation, and residual addition are repeated once. After six Swing Transformer blocks... After processing by the Transformer block, the shallow features obtained from the initial block embedding are directly superimposed onto the deep features output by the network through a global residual connection. The enhanced features are then upsampled through a subpixel convolutional layer to reconstruct a high-resolution array of intermediate frame micro-images.

6. The method for generating high-quality integrated imaging video guided by bidirectional optical flow according to claim 1, characterized in that, In high-quality 3D video display, a high-resolution, high-frame-rate micro-image array sequence generated through iterative looping is input into an integrated imaging 3D display system containing a high-resolution display panel and a matching lens array. The system utilizes the multi-view light field information recorded by each micro-image in the sequence to reconstruct a 3D dynamic scene with full parallax, continuous motion, and clear texture in the real space in front of the display panel through the principle of optical reconstruction.

Images