Video super-resolution reconstruction method based on sparse camera array

Abstract

The invention discloses a video super-resolution reconstruction method based on a sparse camera array, and relates to the field of video image reconstruction, and the method comprises the steps: carrying out the high-resolution feature extraction of a video sequence shot by each camera in the sparse camera array; performing multi-view feature fusion on the extracted high-resolution features to obtain fused features; and performing up-sampling reconstruction processing on the fused features to obtain a high-resolution video. According to the method, video output with higher resolution can be provided for an existing aircraft pose dynamic vision measurement system when hardware bottleneck limitation of low detector resolution exists, so that high-precision and high-speed measurement of aircraft pose parameters is realized.

Description

Technical Field

[0001] This application relates to the field of video image reconstruction, and in particular to a video super-resolution reconstruction method based on a sparse camera array. Background Technology

[0002] Acquiring high-resolution and high-definition images is of great significance in scientific research. High-resolution images can provide more detailed information. However, due to issues such as the diffraction limit, improving image resolution by modifying the camera's optical structure or enhancing the performance of imaging elements is somewhat limited. Based on this, researchers have proposed super-resolution reconstruction techniques. Super-resolution reconstruction is a computational technique used to overcome the inherent resolution limitations of low-resolution imaging systems. This technique utilizes the pixel information of a given low-resolution image and, according to various prior conditions, generates a high-resolution image that meets the required specifications.

[0003] The simplest reconstruction problem is single-image super-resolution reconstruction, which uses a single image to reconstruct a high-resolution image that meets human visual requirements by finding a suitable functional matching relationship. However, in practical applications, the output image often deviates significantly from the target scene. This problem stems from the unstable nature of super-resolution; during signal blurring and degradation, details of objects and backgrounds are lost to varying degrees, resulting in multiple different high-resolution images corresponding to the same low-resolution image. A solution is to use more prior information and known conditions to constrain and adjust the degradation process and the spatial relationships of the original pixels. Based on this, multi-image super-resolution technology has emerged. Multi-image super-resolution uses multiple low-resolution images to reconstruct the original high-resolution image. By utilizing non-redundant complementary information between multiple cameras, the time efficiency and estimation accuracy of super-resolution are greatly improved. Therefore, using multiple cameras to capture images and creating a camera array capable of super-resolution reconstruction is a feasible solution. By simultaneously capturing multiple images, data utilization efficiency is improved, and resolution requirements can be met both spatially and temporally. Building upon the work described above, it is equally significant if a single camera can continuously capture multiple low-resolution images, i.e., use a single camera to capture a video and reconstruct a high-resolution video from the low-resolution video. This approach is not limited to information from different locations at the same time point, but cleverly utilizes redundant information from images at the same location at different time points for imaging, which may achieve more significant results. For example, the inventors' known paper "Caballero J, Ledig C, Aitken A, et al. Real-time video super-resolution with spatio-temporal networks and motion compensation[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition.2017:4778-4787." proposes a new joint motion compensation and video super-resolution algorithm, which is based on end-to-end trainable transformer modules and sub-pixel convolutions, and can improve the accuracy and temporal consistency of reconstruction. However, this method can only utilize partial information from the video sequence, resulting in significant information loss during reconstruction and making long-term super-resolution reconstruction impossible.In another known paper by the inventors, "Chan KCK, Wang X, Yu K, et al. Basicvsr: The search for essential components in video super-resolution and beyond [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021:4947-4956," a baseline for video super-resolution is proposed, including propagation, alignment, aggregation, and upsampling. This method can effectively utilize information in the overall sequence for video super-resolution reconstruction. However, it can only accept one video sequence as input, and has limitations in handling redundant information from multiple video inputs and multiple viewpoints. In yet another known paper by the inventors, "A New Dataset and Transformer for Stereoscopic Video Super-Resolution," a Transformer architecture including spatiotemporal convolutional self-attention and optical flow alignment feedforward layers is proposed. This architecture handles the temporal and disparity consistency of stereoscopic videos and deeply integrates stereo disparity attention (PAM) with the Transformer to achieve more accurate left and right view information interaction. However, this method cannot fully utilize the information in the overall sequence, resulting in the reconstructed video quality failing to meet requirements.

[0004] Based on the above description, due to the hardware bottleneck of low detector resolution, existing technologies are simply unable to provide higher resolution video output for aircraft attitude dynamic visual measurement systems, let alone achieve high-precision and high-speed measurement of aircraft attitude parameters. Summary of the Invention

[0005] The purpose of this application is to provide a video super-resolution reconstruction method based on a sparse camera array, which can provide higher resolution video output for existing aircraft pose dynamic vision measurement systems when there are hardware bottlenecks that limit the resolution of detectors, thereby achieving high-precision and high-speed measurement of aircraft pose parameters.

[0006] To achieve the above objectives, this application provides the following solution:

[0007] This application provides a video super-resolution reconstruction method based on a sparse camera array, including:

[0008] High-resolution feature extraction is performed on video sequences captured by each camera in a sparse camera array; high resolution refers to a resolution higher than a set value.

[0009] The extracted high-resolution features are fused from multiple perspectives to obtain fused features;

[0010] The fused features are upsampled and reconstructed to obtain a high-resolution video.

[0011] Optionally, high-resolution feature extraction is performed on the video captured by each camera in the sparse camera array, including:

[0012] Based on the video sequences captured by each camera in the sparse camera array, reference frames and support frames are obtained.

[0013] A high-resolution image feature extraction network is used to extract multi-dimensional features from the reference frame and the support frame respectively to obtain multi-dimensional features;

[0014] Determine the optical flow conversion matrix based on the reference frame;

[0015] The optical flow transformation matrix is ​​used to distort and align the multidimensional features.

[0016] Introduce forward cross-propagation and backward cross-propagation mechanisms;

[0017] Based on the aforementioned forward cross-propagation mechanism, in the forward channel of the high-resolution image feature extraction network, the multidimensional features output from the previous frame after distortion and alignment processing are iterated to the input of the next frame.

[0018] Based on the aforementioned backward cross-propagation mechanism, in the backward channel of the high-resolution image feature extraction network, the distorted and aligned multidimensional features output from the next frame are iterated to the input of the previous frame.

[0019] The features output from the forward channel and the features output from the backward channel are superimposed and fused to obtain the high-resolution features.

[0020] Optionally, the Spynet network is used to determine the optical flow conversion matrix based on the reference frame.

[0021] Optionally, the calculation principle of the Spynet network determining the optical flow transformation matrix based on the reference frame is expressed as follows:

[0022] V k =u(V k-1 )+v k ;

[0023] In the formula, V k Let u() be the optical flow at level k, and v be the upsampling function. k Let be the optical flow residual of the kth order.

[0024] Optionally, the extracted high-resolution features are fused using multi-view feature fusion to obtain fused features, including:

[0025] The extracted high-resolution features are registered and corrected using a pyramid multi-level network to obtain the fused features.

[0026] Optionally, the pyramid multi-level network learns residual flow and spatial warpage by using multi-level computation and cost aggregation to determine optical flow information between two images, and determines the sub-pixel displacement between the images based on the optical flow information; the sub-pixel displacement is used to determine the image transformation relationship between different cameras in the sparse camera array.

[0027] Optionally, during the registration and correction of the extracted high-resolution features using a pyramid multi-level network, the optical flow transformation matrix is ​​used to guide the alignment operation of the high-resolution features on the feature layer.

[0028] Optionally, the fused features are upsampled and reconstructed to obtain a high-resolution video, including:

[0029] The fused features are integrated and dimensionality reduced to obtain intermediate features;

[0030] The high-resolution video is obtained by shuffling and convolving each pixel in the intermediate features using PixelShuffle convolution.

[0031] Optionally, the process of integrating and reducing the dimensionality of the fused features to obtain intermediate features is represented as follows:

[0032]

[0033] In the formula, F re (θ) represents an intermediate feature, k i F represents the occupancy weight of the i-th camera. i (θ) represents the fusion feature of the i-th camera, i = 0, 1, 2, ..., n, and b represents the error.

[0034] Optionally, the process of shuffling and convolving each pixel in the intermediate features using PixelShuffle convolution can be represented as follows:

[0035] PS(T) x,y c = T x / r,y / r,c·r·mod(y ,r)+c·mod(x,r);

[0036] In the formula, PS() represents the PixelShuffle upsampling function, T represents the original feature layer, x, y, and c represent the three dimensions of the output feature layer, r represents the total number of channels in the original feature layer, and mod() represents the modulo function.

[0037] According to the specific embodiments provided in this application, this application has the following technical effects:

[0038] This application provides a video super-resolution reconstruction method based on a sparse camera array. By using multiple video sequences captured by a sparse camera array for super-resolution reconstruction, it can provide higher resolution video output for existing aircraft pose dynamic vision measurement systems when there are hardware bottlenecks that limit the resolution of detectors, thereby achieving high-precision and high-speed measurement of aircraft pose parameters. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 A flowchart illustrating a video super-resolution reconstruction method based on a sparse camera array, provided as an embodiment of this application;

[0041] Figure 2 A flowchart illustrating the implementation of a video super-resolution reconstruction method based on a sparse camera array, as provided in an embodiment of this application.

[0042] Figure 3 This is a schematic diagram of the structure of a high-resolution image feature extraction module provided in an embodiment of this application;

[0043] Figure 4 A schematic diagram of a cyclic iterative feature extraction process with propagation characteristics provided in an embodiment of this application;

[0044] Figure 5 An array diagram showing the effect obtained by applying the video super-resolution reconstruction method provided in this application to an aircraft landing video;

[0045] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0046] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0047] Existing research indicates that combining existing methods to construct an array-shaped image sequence (video) is of great significance for further research in the field of super-resolution imaging and reconstruction.

[0048] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] In one exemplary embodiment, this application provides a video super-resolution reconstruction method based on a sparse camera array. This method is executed by a computer device, specifically a terminal or server, or both. In this embodiment, the method is described using a server as an example. Figure 1 As shown, the method includes:

[0050] Step 100: Extract high-resolution features from the video sequences captured by each camera in the sparse camera array. High resolution refers to a resolution higher than a set value.

[0051] Step 101: Perform multi-view feature fusion on the extracted high-resolution features to obtain fused features.

[0052] Step 102: Upsample and reconstruct the fused features to obtain a high-resolution video.

[0053] By implementing steps 100 to 102 above, this application makes full use of the camera redundancy information from different perspectives to perform super-resolution reconstruction of low-resolution video, thereby obtaining a higher resolution reconstructed video.

[0054] In another exemplary embodiment of this application, in order to achieve long-term super-resolution reconstruction, overcome the shortcomings of processing redundant information from multiple video inputs and multiple viewpoints, and solve the problem of insufficient full utilization of information in the overall sequence, resulting in unsatisfactory reconstructed video quality, this embodiment introduces an iterative and cyclic propagation mechanism to fully utilize information in the overall image sequence and perform long-term feature extraction on low-resolution videos. Based on this, the implementation process of step 100 above can be replaced by steps 100-1 to 100-8 below.

[0055] Step 100-1: Based on the video sequences captured by each camera in the sparse camera array, obtain the reference frame and the support frame.

[0056] Step 100-2: A high-resolution image feature extraction network is used to extract multi-dimensional features from both the reference frame and the support frame, resulting in multi-dimensional features. The structure of the high-resolution image feature extraction network is as follows: Figure 3As shown. To enrich the feature information of each frame in the spatial domain and fully capture the details of the image under test, in this step, after multi-dimensional feature extraction is performed on the reference frame and the support frame respectively, the ResNet residual module (corresponding to...) is also used. Figure 3 The features extracted by refining the residual layer shown.

[0057] Step 100-3: Determine the optical flow transformation matrix based on the reference frame. In practical applications, the Spynet network can be used to calculate the optical flow transformation matrix. The main calculation principle of the Spynet network is as follows:

[0058]

[0059] V k =u(V k-1 )+v k .

[0060] In the formula, V k Let u() be the optical flow at level k, and v be the upsampling function. k Let G be the optical flow residual of the kth order, w() be the torsion function, and G be the optical flow residual of the kth order. k () represents the convolutional layer function, V k-1 For the (k-1)th order optical flow, I k Let be the input image for level k.

[0061] V obtained through multiple iterations k It is the optical flow transformation matrix. The specific form of the optical flow transformation matrix is ​​a tensor with the same height and width as the original image and a layer number of 2.

[0062] Step 100-4: Use an optical flow transformation matrix to distort and align the multidimensional features.

[0063] In practical applications, an optical flow transformation matrix is ​​used to distort and align the high-dimensional features of consecutive frames, and then re-input them into the input of the next frame. By introducing this iterative approach, key feature information from other frames can be obtained for each frame in the overall video sequence.

[0064] The optical flow transformation matrix can shift each pixel, a process known as feature warping, which is calculated through pixel-by-pixel multiplication. Alignment is the result of this warping; the warped features are aligned with other frames for better integration.

[0065] Step 100-5: Introduce forward cross-propagation mechanism and backward cross-propagation mechanism.

[0066] In order to enable the information from previous and subsequent frames to serve as a reference for the information in the current frame, a mechanism of forward and backward cross-propagation is introduced to make comprehensive use of the information in the overall sequence.

[0067] Step 100-6: Based on the forward cross-propagation mechanism, in the forward channel of the high-resolution image feature extraction network, the distorted and aligned multidimensional features output from the previous frame are iterated to the input of the next frame.

[0068] Step 100-7: Based on the backward cross-propagation mechanism, in the backward channel of the high-resolution image feature extraction network, the distorted and aligned multidimensional features output from the next frame are iterated to the input of the previous frame.

[0069] Step 100-8: Superimpose and fuse the features output from the forward channel and the features output from the backward channel to obtain high-resolution features.

[0070] Based on the descriptions in steps 100-6 to 100-8 above, two propagation channels, forward and backward, are designed, and the outputs of the two channels are superimposed and fused at the end of the network (i.e., the high-resolution image feature extraction network). For example... Figure 4 The forward channel shown is a cyclic iterative feature extraction process with propagation characteristics. This cyclic structure of the forward and backward channels makes full use of the redundant information in the long sequence, realizing feature extraction with long-term characteristics. Figure 4 In the middle, x i-1 and x i Let the features represent the input images of the (i-1)th frame and the i-th frame, respectively. i-1 and feature i These represent the features of the (i-1)th frame and the i-th frame, respectively.

[0071] In another exemplary embodiment of this application, in step 101, a pyramid multi-level network can be used to register and correct the extracted high-resolution features to obtain fused features. Specifically, for each time step, the pyramid multi-level network is used to register and correct the stereo images (i.e., feature maps) at the same time step. The pyramid structure employs multi-level computation and cost aggregation to learn residual flux and spatial warpage. Using a coarse-to-fine process, it calculates optical flow information between two similar images, thereby calculating the sub-pixel displacement between images based on the optical flow information to determine the image transformation relationship between different cameras in the sparse camera array. In this process, the optical flow (i.e., optical flow transformation matrix) calculated from the original low-resolution image guides the multi-camera video alignment operation on the feature layer.

[0072] In another exemplary embodiment of this application, based on the above description, this application designs an algorithm for super-resolution reconstruction using multiple video sequences captured by a sparse camera array. Utilizing an optical flow mechanism, it guides feature matching, alignment, and distortion between different cameras, fully leveraging redundant information from cameras at different viewpoints to perform super-resolution reconstruction of low-resolution videos. The implementation process of step 102 can be replaced by the following steps 102-1 to 102-2.

[0073] Step 102-1: Integrate and reduce the dimensionality of the fused features to obtain intermediate features. In video super-resolution tasks based on sparse camera arrays, the relationship between the reference frame and the support frame can be approximated as a linear function, as follows:

[0074]

[0075] In the formula, F re (θ) represents an intermediate feature, k i F represents the occupancy weight of the i-th camera. i (θ) represents the fused feature of the i-th camera (i.e., the high-dimensional feature extracted from the front end), i = 0, 1, 2, ..., n. b represents the error, which is mainly caused by parallax, blur, and camera shake. θ represents the high-resolution feature. Through this linear function relationship, the features of the reference frame and the supporting frame cameras are fused and input into the PixelShuffle convolution.

[0076] Step 102-2: Use PixelShuffle convolution to shuffle and convolve each pixel in the intermediate features to obtain a high-resolution video. The PixelShuffle convolution formula is expressed as:

[0077] PS(T) x,y,c =T x / r,y / r,c·r·mod(y,r) +c·mod(x,r).

[0078] In the formula, PS() represents the PixelShuffle upsampling function, T represents the original feature layer, x, y, and c represent the three dimensions of the output feature layer, r represents the total number of channels in the original feature layer, and mod() represents the modulo function.

[0079] In this method, the original high-dimensional features are output as larger-sized low-dimensional features, which facilitates upsampling and reconstruction of the output. A ReLU non-linear activation function is added after each PixelShuffle convolution to prevent gradient explosion during network training. Finally, each high-resolution image frame is stitched together to obtain the final high-resolution video output, completing the overall task.

[0080] Based on the above description, the overall implementation process of the video super-resolution reconstruction method based on sparse camera arrays provided in this application is as follows: Figure 2 As shown, this application fully utilizes the information in the overall image sequence by introducing an iterative and cyclic propagation mechanism to perform long-term feature extraction on low-resolution videos. Simultaneously, an algorithm for super-resolution reconstruction using multiple video sequences captured by a sparse camera array is designed. Utilizing an optical flow mechanism, it guides feature matching, alignment, and distortion between different cameras, fully leveraging redundant information from different camera perspectives to perform super-resolution reconstruction of low-resolution videos.

[0081] Furthermore, traditional video super-resolution algorithms have poor performance and cannot meet accuracy requirements. Existing learning-based algorithms cannot simultaneously utilize multi-view information in the spatial domain and sequential information in the temporal domain for high-resolution video reconstruction, resulting in poor performance for video super-resolution reconstruction of multiple sequences.

[0082] This application introduces a loop iteration and propagation mechanism, which has the characteristics of long duration and multiple perspectives. It can perform super-resolution reconstruction using multiple low-resolution videos captured by a sparse camera array, and can meet the video super-resolution reconstruction task based on a sparse camera array.

[0083] Peak signal-to-noise ratio (PSNR) reflects the quality of an image, while structural similarity (SSIM) comprehensively reflects the similarity between images in terms of brightness, contrast, and structure. PSNR and SSIM are important metrics for evaluating image super-resolution reconstruction. For high-resolution output video, the higher the average PSNR across all frames, the better the output video quality. For the original high-resolution video and the reconstructed output video, the higher the average structural similarity of corresponding frames, the closer the output video is to the true value.

[0084] The application effect of the method provided in this application is as follows: Figure 5 As shown in Table 1, experimental results evaluating the performance of video reconstruction algorithms using Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) are presented. Figure 5 As can be clearly seen from Table 1, the image quality in the output video of this application is higher than that of traditional reconstruction methods. Furthermore, the peak signal-to-noise ratio is improved to 32.6059 dB and the structural similarity is improved to 0.8548, both of which are superior to existing methods.

[0085] Table 1 Experimental Results

[0086] Evaluation parameters Nearest neighbor interpolation Bicubic interpolation This application PSNR (dB) 29.0190 31.7827 32.6059 SSIM 0.7804 0.8288 0.8548

[0087] The long-term and multi-perspective characteristics of the method provided in this application mainly stem from the following two aspects:

[0088] 1) Long duration: By introducing an iterative and cyclic propagation mechanism, each frame in the overall video sequence can obtain key feature information from other frames, making full use of the information in the overall image sequence.

[0089] 2) Multi-view: Using optical flow guidance, videos shot from different perspectives are matched and aligned. Redundant information from different perspectives is utilized to improve the resolution of the output video.

[0090] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 6 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores video super-resolution reconstruction data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When executed by the processor, the computer program implements a video super-resolution reconstruction method based on a sparse camera array.

[0091] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0092] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0093] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0094] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0095] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0096] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (RRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0097] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0098] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0099] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A video super-resolution reconstruction method based on a sparse camera array, characterized in that, include: High-resolution feature extraction is performed on video sequences captured by each camera in a sparse camera array. High resolution refers to a resolution higher than the set value; The extracted high-resolution features are fused from multiple perspectives to obtain fused features; The fused features are upsampled and reconstructed to obtain a high-resolution video.

2. The video super-resolution reconstruction method based on sparse camera array according to claim 1, characterized in that, High-resolution feature extraction is performed on the video captured by each camera in the sparse camera array, including: Based on the video sequences captured by each camera in the sparse camera array, reference frames and support frames are obtained. A high-resolution image feature extraction network is used to extract multi-dimensional features from the reference frame and the support frame respectively to obtain multi-dimensional features; Determine the optical flow conversion matrix based on the reference frame; The optical flow transformation matrix is ​​used to distort and align the multidimensional features. Introduce forward cross-propagation and backward cross-propagation mechanisms; Based on the aforementioned forward cross-propagation mechanism, in the forward channel of the high-resolution image feature extraction network, the multidimensional features output from the previous frame after distortion and alignment processing are iterated to the input of the next frame. Based on the aforementioned backward cross-propagation mechanism, in the backward channel of the high-resolution image feature extraction network, the distorted and aligned multidimensional features output from the next frame are iterated to the input of the previous frame. The features output from the forward channel and the features output from the backward channel are superimposed and fused to obtain the high-resolution features.

3. The video super-resolution reconstruction method based on a sparse camera array according to claim 2, characterized in that, The optical flow conversion matrix is ​​determined based on the reference frame using the Spynet network.

4. The video super-resolution reconstruction method based on a sparse camera array according to claim 3, characterized in that, The calculation principle of the Spynet network to determine the optical flow transformation matrix based on the reference frame is expressed as follows: V k =u(V k-1 )+v k ; In the formula, V k Let u() be the optical flow at level k, and v be the upsampling function. k Let be the optical flow residual of the kth order.

5. The video super-resolution reconstruction method based on a sparse camera array according to claim 2, characterized in that, The extracted high-resolution features are fused using multi-view feature fusion to obtain fused features, including: The extracted high-resolution features are registered and corrected using a pyramid multi-level network to obtain the fused features.

6. The video super-resolution reconstruction method based on a sparse camera array according to claim 5, characterized in that, The pyramid multi-level network uses multi-level computation and cost aggregation to learn residual flow and spatial warpage, determine optical flow information between two images, and determine sub-pixel displacement between images based on optical flow information; the sub-pixel displacement is used to determine the image transformation relationship between different cameras in a sparse camera array.

7. The video super-resolution reconstruction method based on a sparse camera array according to claim 5, characterized in that, During the registration and correction of the extracted high-resolution features using a pyramid multi-level network, the optical flow transformation matrix is ​​used to guide the alignment operation of the high-resolution features on the feature layer.

8. The video super-resolution reconstruction method based on a sparse camera array according to claim 1, characterized in that, The fused features are upsampled and reconstructed to obtain a high-resolution video, including: The fused features are integrated and dimensionality reduced to obtain intermediate features; The high-resolution video is obtained by shuffling and convolving each pixel in the intermediate features using PixelShuffle convolution.

9. The video super-resolution reconstruction method based on a sparse camera array according to claim 8, characterized in that, The process of integrating and reducing the dimensionality of the fused features to obtain intermediate features is represented as follows: In the formula, F re (θ) represents an intermediate feature, k i F represents the occupancy weight of the i-th camera. i (θ) represents the fusion feature of the i-th camera, i = 0, 1, 2, ..., n, and b represents the error.

10. The video super-resolution reconstruction method based on a sparse camera array according to claim 8, characterized in that, The process of shuffling and convolving each pixel in the intermediate features using PixelShuffle convolution is represented as follows: PS(T) x,y ,c=T x / r,y / r,c·r·mod(y ,r)+c·mod(x,r); In the formula, PS() represents the PixelShuffle upsampling function, T represents the original feature layer, x, y, and c represent the three dimensions of the output feature layer, r represents the total number of channels in the original feature layer, and mod() represents the modulo function.

Images