Multi-video super-resolution reconstruction method, camera array imaging device, equipment, medium and product

Abstract

The invention discloses a multi-video super-resolution reconstruction method, a camera array imaging device, equipment, a medium and a product, and relates to the field of computational imaging, and the method comprises the steps: constructing a multi-video super-resolution data set based on a historical video sequence collected by a camera array and a camera array imaging model; training a multi-video super-resolution reconstruction network by using the multi-video super-resolution data set, and enabling the multi-video super-resolution reconstruction network to learn a multi-video spatial-temporal feature extraction and fusion capability in a training stage; optimizing network parameters of the multi-video super-resolution reconstruction network, and constructing an optimized multi-video super-resolution reconstruction network; and inputting the to-be-detected multi-video sequence acquired by the camera array into the optimized multi-video super-resolution reconstruction network for super-resolution reconstruction, and determining the reconstructed super-resolution video sequence, thereby improving the super-resolution reconstruction effect and the reconstructed image quality.

Description

Technical Field

[0001] The present application relates to the field of computational imaging, and in particular to a multi-video super-resolution reconstruction method, a camera array imaging device, equipment, media and products. Background Art

[0002] Super-resolution reconstruction is a cutting-edge computational imaging technology that aims to overcome the resolution limitations of traditional imaging systems due to hardware constraints such as the optical diffraction limit and sensor sampling rate. In terms of application areas, super-resolution reconstruction technology has been widely used in scenarios such as high-precision satellite remote sensing imaging, industrial visual precision measurement, intelligent video surveillance systems, and consumer electronic devices, significantly improving the accuracy and reliability of downstream computer vision tasks such as object recognition, feature classification, and semantic segmentation.

[0003] Single-image super-resolution reconstruction techniques primarily rely on prior information from a single low-resolution image, generating a high-resolution image through interpolation and reconstruction or deep learning-based methods. However, due to the inherent ill-posedness of the low-resolution to high-resolution mapping, a single input may correspond to multiple reasonable super-resolution predictions, making reconstruction quality difficult to guarantee.

[0004] To address the above issues, existing technologies mainly adopt two solutions: first, multi-camera super-resolution technology uses a spatially distributed camera array to obtain multiple low-resolution images of the same scene, and uses the sub-pixel displacement information between them to improve the reconstruction quality; second, video super-resolution technology uses the temporal correlation between consecutive frames to enhance the reconstruction effect through motion compensation and time domain information fusion.

[0005] However, existing technologies still have limitations. First, traditional multi-camera super-resolution methods are primarily designed for static scenes. This means they only perform super-resolution reconstruction on multiple images taken at the same moment, making it difficult to address motion blur in dynamic scenes. Furthermore, relying solely on algorithms makes it difficult to achieve substantial improvements in image quality, resulting in poor super-resolution reconstruction. Furthermore, current algorithms lack the coordinated use of spatiotemporal information, resulting in insufficient continuity in the reconstruction results across both temporal and spatial dimensions, leading to low reconstructed image quality. Summary of the Invention

[0006] The purpose of this application is to provide a multi-video super-resolution reconstruction method, camera array imaging device, equipment, medium and product to solve the problems of low super-resolution reconstruction effect and reconstructed image quality.

[0007] To achieve the above objectives, this application provides the following solutions:

[0008] In a first aspect, the present application provides a multi-video super-resolution reconstruction method, comprising:

[0009] Based on the historical video sequences collected by the camera array and the camera array imaging model, a multi-video super-resolution dataset is constructed;

[0010] Using the multi-video super-resolution dataset to train a multi-video super-resolution reconstruction network, and enabling the multi-video super-resolution reconstruction network to learn multi-video spatiotemporal feature extraction and fusion capabilities during the training phase;

[0011] Optimizing network parameters of the multi-video super-resolution reconstruction network to construct an optimized multi-video super-resolution reconstruction network;

[0012] The multi-video sequences to be tested captured by the camera array are input into the optimized multi-video super-resolution reconstruction network for super-resolution reconstruction, and the reconstructed super-resolution video sequence is determined.

[0013] In a second aspect, the present application provides a camera array imaging device, comprising:

[0014] A camera array image acquisition module is configured to receive a global synchronization signal based on the synchronization control module to control the camera array to simultaneously acquire multiple video sequences within the field of view, and transmit the multiple video sequences to the multi-video processing module;

[0015] A synchronization control module, configured to generate a global synchronization signal and transmit the global synchronization signal to the camera array image acquisition module;

[0016] The multi-video processing module is used to perform super-resolution reconstruction on the multi-video sequences according to the multi-video super-resolution reconstruction method.

[0017] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any one of the multi-video super-resolution reconstruction methods described above.

[0018] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any one of the multi-video super-resolution reconstruction methods described above.

[0019] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements any one of the multi-video super-resolution reconstruction methods described above.

[0020] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0021] This application combines camera arrays with video super-resolution, and inputs multiple videos into a deep neural network for spatiotemporal super-resolution reconstruction. This deep neural network, namely the multi-video super-resolution reconstruction network, effectively solves the problems of dynamic consistency in camera array imaging systems and loss of structural details in video sequence reconstruction, and realizes the coordinated use of spatiotemporal information, thereby achieving high-precision and robust super-resolution reconstruction performance in complex scenes. Compared with traditional multi-camera super-resolution methods, this application obtains multi-dimensional information in time and space, enhances feature expression capabilities, and effectively restores texture information in blurred areas while maintaining structural details, achieving better super-resolution reconstruction effects. It can be seen that this application introduces the time dimension and uses the information of frames before and after different moments for reconstruction to solve problems such as motion blur in dynamic scenes, and from the perspective of information utilization, that is, to obtain more information, perform orderly fusion reconstruction, and improve image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 This is a diagram showing an application environment of a multi-video super-resolution reconstruction method in one embodiment of the present application;

[0024] Figure 2 A flowchart of a multi-video super-resolution reconstruction method provided in one embodiment of the present application;

[0025] Figure 3 This is a schematic diagram of the camera array placement;

[0026] Figure 4 Create a flow chart for the training set samples;

[0027] Figure 5 Schematic diagram of the multi-video super-resolution reconstruction network structure;

[0028] Figure 6 Schematic diagram of the detail-guided multi-layer feature extraction module;

[0029] Figure 7 Schematic diagram of feature registration module;

[0030] Figure 8 Schematic diagram of the spatial to temporal feature fusion module;

[0031] Figure 9 Schematic diagram of the camera array super-resolution reconstruction process;

[0032] Figure 10 Schematic diagram of a camera array imaging device;

[0033] Figure 11 This is a comparison chart of the effects of this application and different video super-resolution reconstruction methods; among them, Figure 11 (a) is the reconstruction effect diagram of the method of the present application and different video super-resolution reconstruction methods in the dataset REDS4 scene 1; Figure 11 (b) is the reconstruction effect diagram of the method of the present application and different video super-resolution reconstruction methods in the dataset REDS4 scene 2; Figure 11 (c) is the reconstruction effect diagram of the method of the present application and different video super-resolution reconstruction methods in the dataset REDS4 scene 3; Figure 11 (d) is a diagram showing the reconstruction effects of the present invention's method and different video super-resolution reconstruction methods in scene 4 of the Vid4 dataset;

[0034] Figure 12 This is a comparison chart of the effects of this application and different multi-image super-resolution reconstruction methods; among them, Figure 12 (a) is the reconstruction effect of the multi-image super-resolution method BSRT in the dataset REDS4; Figure 12 (b) is the reconstruction effect of the multi-image super-resolution method Rotation MFSR in the dataset REDS4; Figure 12 (c) is the rendering of the reconstruction in the dataset REDS4 by this application; Figure 12 (d) in the figure is the true value graph of the dataset;

[0035] Figure 13 This is the reconstruction effect diagram of this application when inputting in different dimensions; among them, Figure 13 (a) in the figure is the true value of the image; Figure 13 (b) in the figure is the blurred image; Figure 13 (c) is the reconstruction effect diagram when the number of input images in the time dimension is 1 and the number of input images in the spatial dimension is 1; Figure 13 (d) is the reconstruction effect when the number of input images in the time dimension is 1 and the number of input images in the spatial dimension is 5; Figure 13 (e) is the reconstruction effect diagram when the number of input images in the time dimension is 6 and the number of input images in the spatial dimension is 5; Figure 13 (f) is the reconstruction effect when the number of input images in the time dimension is 1 and the number of input images in the spatial dimension is 30; Figure 13 (g) is the reconstruction effect when the number of input images in the time dimension is 6 and the number of input images in the spatial dimension is 10;

[0036] Figure 14 This is a real scene experiment rendering; Figure 14(a) is the original image of the real scene 1 and its details; Figure 14 (b) is the real scene 1 and its detailed reconstruction effect; Figure 14 (c) is the original image of the real scene 2 and its details; Figure 14 (d) is the real scene 2 and its detailed reconstruction effect;

[0037] Figure 15 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0039] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0040] The present application embodiment provides a multi-video super-resolution reconstruction method, such as Figure 1 As shown, the method is executed by a computer device, specifically, it can be executed by a computer device such as a terminal or a server alone, or it can be executed by a terminal and a server together. In the embodiment of the present application, as shown in FIG. Figure 2 As shown, the method includes the following steps.

[0041] S1: Construct a multi-video super-resolution dataset based on historical video sequences collected by camera arrays and camera array imaging models.

[0042] S2: Using the multi-video super-resolution dataset to train a multi-video super-resolution reconstruction network, and enabling the multi-video super-resolution reconstruction network to learn multi-video spatiotemporal feature extraction and fusion capabilities during the training phase.

[0043] S3: Optimize the network parameters of the multi-video super-resolution reconstruction network and construct an optimized multi-video super-resolution reconstruction network.

[0044] S4: Inputting the multi-video sequence to be tested captured by the camera array into the optimized multi-video super-resolution reconstruction network for super-resolution reconstruction, and determining the reconstructed super-resolution video sequence.

[0045] In an exemplary embodiment, before S1, the method further includes: fixing multiple cameras on a rigid frame, with a spatial position relationship as follows: Figure 3As shown. Under the premise of ensuring that there is no physical interference, the spatial arrangement between each camera should meet the minimum spacing condition to optimize the field of view overlap. Multiple cameras in the camera array are started synchronously to shoot a video sequence of the same scene, thereby obtaining multi-video data containing time dimension and spatial dimension information. Among them, the spatial dimension information is provided by the imaging plane distribution of the camera array, reflecting the spatial complementary characteristics under different cameras; the time dimension information is provided by the video time series captured by each camera, reflecting the dynamic change characteristics of the scene. Through this synchronous acquisition method, rich spatiotemporal information can be obtained at the same time, providing a high-quality data foundation for subsequent multi-video super-resolution reconstruction.

[0046] In an exemplary embodiment, S1 may be replaced by the following steps.

[0047] S11: In combination with the camera array imaging model, for each high-resolution video frame in the historical video sequence, select an image in the horizontal direction as a reference image, and rotate the reference image based on a set increment to generate multiple additional images.

[0048] S12: Apply random translation and rotation transformations to the additional image to determine a transformed image.

[0049] S13: Processing all transformed images and the reference image using a bicubic downsampling method to simulate image degradation and generate corresponding multiple degraded images; the degraded images simulate low-resolution video frames simultaneously captured by the camera array. All moments constitute multiple low-resolution videos captured by the camera array.

[0050] In practical applications, optical blur, motion blur, and bicubic downsampling methods can also be used to process all transformed images and the reference image to simulate image degradation and generate corresponding multiple degraded images to generate different data sets, thereby further verifying the technical solution of the present application.

[0051] S14: All low-resolution video frames are regarded as low-resolution videos acquired by the camera array, and all high-resolution video frames are regarded as high-resolution videos acquired by the camera array.

[0052] S15: Using high-low resolution video pairs constructed from the plurality of low-resolution videos and the corresponding high-resolution videos as a multi-video super-resolution dataset.

[0053] In practical applications, training data is made as Figure 4 As shown in Figure 2, improvements are made using camera array imaging models based on standard video super-resolution datasets, including Realistic and Dynamic Scenes (REDS), Vimeo-90K, and Vid4:

[0054] (1) Multi-camera image generation: For each video frame, a horizontal image is selected as a reference, and five additional images are generated by rotating them in increments of 15°. To simulate the systematic calibration errors that may exist in a real camera array, these five rotated images are further subjected to random translation and rotation transformations, with a translation range of [-30, 30] pixels and a rotation range of [-3, 3] degrees.

[0055] (2) Image degradation simulation: Gaussian blur and random motion blur are optionally added to simulate the blurring effect of the optical system and the relative motion between the camera and the scene, respectively. All images are processed using bicubic downsampling with a scale factor of 4 to simulate the low-resolution degradation effect, resulting in six degraded images that simulate the low-resolution images captured simultaneously by the camera array.

[0056] (3) Construction of high-low resolution video pairs: Repeat the above operation for each frame of the video to generate six sub-videos. These sub-videos and the original high-resolution video together form a high-low resolution video pair. Among them, the low-resolution video with the same rotation angle as the high-resolution video is used as the reference video, and the remaining five videos are used as auxiliary videos to provide spatiotemporal complementary information.

[0057] In an exemplary embodiment, Python is used to build a Figure 5 The multi-video super-resolution reconstruction network shown in the figure specifically includes: a feature extraction module, a feature registration module, a space-to-time feature fusion module, a feature propagation module and an image reconstruction module.

[0058] Based on the feature extraction module, high-frequency detail information and image features are extracted, and a high-dimensional feature block is output; the feature extraction module is a detail-guided multi-layer feature extraction module, including an image branch, a detail branch, and an attention mechanism unit; the image branch is used to obtain image features through position encoding; the detail branch is used to use the LOG operator and the Sobel operator to extract high-frequency detail information of the image acquired by the camera array at the same time; the attention unit is used to enhance the image features using the high-frequency detail information through the attention mechanism and output a high-dimensional feature block.

[0059] In practical applications, such as Figure 6As shown in the figure, the detail-guided multi-layer feature extraction module includes an image branch and a detail branch. The detail branch uses the LOG (Laplacian of Gaussian) operator and the Sobel operator to extract the high-frequency detail information of the image obtained by the camera array at the same time. The image features obtained by the position encoding of the image branch are selectively enhanced through the attention mechanism. The module inputs the B*N*T*C*W*H image block and outputs a high-dimensional feature block; the high-dimensional feature block is Figure 6 The input features of Figure 7 In the box, there are reference features and auxiliary features, both of which are input features and serve as the input features of the network; where B is the number of samples processed simultaneously, N is the number of spatial dimension images, T is the number of time dimension images, C is the number of channels, W is the width of the image, and H is the height of the image.

[0060] Based on the feature registration module, the high-dimensional feature blocks are divided into reference features and auxiliary features. The auxiliary features are affine transformed in combination with the affine transformation matrix to obtain preliminary compensation features. Motion estimation is performed using a pre-trained optical flow estimator, and motion vectors are modeled using a coordinate network. The preliminary compensation features are aligned to the reference features through a cross-attention mechanism, and the registration features with the same dimension as the reference features are determined. The feature registration module is a spatial transformation consistency implicit alignment module.

[0061] In practical applications, such as Figure 7 As shown in the figure, the features extracted from the reference camera image are selected as reference features, that is, the reference camera image is taken by Figure 6 The features extracted by the feature extraction module and the features extracted by the remaining cameras in the camera array are used as auxiliary features, that is, the images taken by the remaining cameras in the camera array except the reference camera are Figure 6 The features extracted by the feature extraction module are Figure 5 Indicated in the reference image with a gray background. Figure 3 It was indicated in .

[0062] The spatial transformation consistency implicit alignment module uses a pre-calibrated imaging model to obtain an affine transformation matrix, which is then applied to the auxiliary features to obtain preliminary compensation features. Further motion estimation is performed using the pre-trained optical flow estimator SpyNet to obtain motion estimates. A coordinate network is used to model motion vectors. A cross-attention mechanism is used to align the preliminary compensation features to the reference features. This implicit resampling method achieves fine feature registration, accurately obtaining registration features with the same dimensions as the reference features. Under different sampling modes of the camera array, this two-stage, coarse-to-fine alignment method combines the global robustness of coarse alignment with the high precision of fine alignment, preserving the original clarity of features and avoiding the distortion commonly caused by traditional alignment methods.

[0063] Based on the space-to-time feature fusion module, a dynamic most adaptive feature fusion strategy is adopted to fuse the reference features and the registration features, and a multi-layer perceptron is used as the primary layer of global residual upsampling for image reconstruction; the primary layer is used to feed the fused features and the upsampled reference frame into the multi-layer perceptron and convert them into propagation input features in the time dimension; the propagation input features are time features.

[0064] In practical applications, such as Figure 15 As shown in Figure 2, the spatial-temporal feature fusion module adopts a dynamic adaptive feature fusion strategy to fuse the reference features from the camera array with the registration features, that is, non-local attention is to calculate the correlation of all positions of the reference features, and non-local cross attention is to calculate the correlation between the reference features and the registration features. The input to the multi-layer perceptron is the features output by the two parts of the attention mechanism, and finally the entire fusion process is realized. It is fed into the multi-layer perceptron with the upsampled reference frame and converted into a propagation input in the time dimension, that is, Figure 8 The multi-layer perceptron is also used as the primary layer of global residual upsampling for image reconstruction to achieve 2x super-resolution of features.

[0065] Based on the feature propagation module, a cyclic structure of second-order grid propagation is adopted to learn the multi-scale information between the temporal features, and the temporal features are gradually refined and enhanced according to the continuous extraction of inter-frame information.

[0066] In practical applications, the feature propagation module is based on a convolutional neural network and adopts a second-order grid propagation cyclic structure to learn multi-scale information between temporal features, and gradually refines and enhances temporal features by continuously extracting inter-frame information.

[0067] Based on the image reconstruction module, a super-resolution video is reconstructed by upsampling the high-resolution video through a secondary cascade global staggered approach according to the enhanced temporal features.

[0068] In practical applications, the final image reconstruction module reconstructs each frame of the high-resolution video through sub-cascade global residual upsampling, outputs a B*C*T*4W*4H video, and achieves 4x super-resolution reconstruction. Among them, the primary cascade layer and the secondary cascade layer are the two steps of upsampling. The primary cascade layer performs 2x super-resolution reconstruction, and the secondary cascade layer performs 2x super-resolution reconstruction. The connection relationship is as follows: Figure 5 As shown in the figure, the two upward arrows in the circles represent the image upsampling process, which are part of the input features of the two cascade layers; the primary cascade layer and the secondary cascade layer are Figure 8 Multilayer Perceptron and Figure 5 image reconstruction.

[0069] In an exemplary embodiment, the multi-video super-resolution reconstruction network adopts the Charbonnier loss function, and the loss function of the multi-video super-resolution reconstruction network is Loss Charbonnier for:

[0070]

[0071] Where T represents the number of frames contained in each input video during network training; represents the i-th high-resolution image output by the network using temporal and spatial information from multiple videos; represents the true value of the corresponding i-th high-resolution image; ∈ represents the bias constant.

[0072] In an exemplary embodiment, optimizing the network parameters of the multi-video super-resolution reconstruction network in S3 specifically includes:

[0073] The model weights obtained by training the multi-video super-resolution reconstruction network using the REDS dataset are used as pre-training weights.

[0074] The network parameters are initialized with the pre-trained weights, and the network parameters are iteratively trained on the Vimeo-90K dataset using a cosine learning rate decay strategy to determine the optimized network parameters.

[0075] In practical applications, the adaptive moment estimation method is used to update the network parameters of the model. Specifically, a dataset containing multiple sets of high-low resolution video pairs generated using standard datasets REDS, Vimeo-90K and Vid4 is trained end-to-end in parallel on two NVIDIA 3090 GPUs.

[0076] On the REDS dataset, a cosine learning rate decay strategy is adopted and 6×10 input frames are used for iterative training.

[0077] When training on the Vimeo-90K dataset, the network input is a 6×7 frame sequence. The model trained on the REDS dataset is used as the pre-training weight. The network parameters of the model are initialized according to the pre-training weight. The same cosine decay strategy as REDS training is used for iterative training to obtain two models and two optimized network parameters.

[0078] In an exemplary embodiment, S4 may be replaced by the following steps.

[0079] The multi-video sequences to be tested, captured by the camera array, are input into an optimized multi-video super-resolution reconstruction network. The multi-video super-resolution reconstruction network learns the multi-video spatiotemporal feature extraction and fusion capabilities during the training phase, and super-resolution reconstruction is performed on the input multiple multi-video sequences to be tested to determine the reconstructed super-resolution video sequence; the multi-video sequences to be tested are used as low-resolution video sequences.

[0080] In practical applications, the super-resolution reconstruction process is as follows Figure 9 As shown:

[0081] (1) Load model parameters.

[0082] The trained network model parameters and pre-calibrated imaging model parameters are loaded, and multiple videos captured in real time by the camera array imaging device are fed into the network. During the inference phase, the network model parameters are kept fixed, and the multi-video spatiotemporal feature extraction and fusion capabilities learned during the training phase are used to perform super-resolution reconstruction on multiple low-resolution input video sequences.

[0083] (2) Image super-resolution reconstruction.

[0084] Based on actual application requirements, the video captured by one of the cameras is selected as the reference video, and the remaining videos are used as auxiliary videos. The network input B*N*T*C*W*H image block dimensions are modified, and the last two dimensions are appropriately increased. This allows the network to process videos with resolutions larger than those used for training, and multiple videos are input to generate corresponding high-resolution videos. When the sequence length of the network's multiple video inputs is longer than the sequence length used for training, that is, the total data volume is larger than B*N*T*C*W*H, processing higher-resolution input videos requires only adjusting the input dimensions and replacing the graphics card with a higher memory capacity, without the need to retrain the network. This design significantly improves the network's flexibility and practicality, adapting to the needs of a variety of practical application scenarios.

[0085] Multi-video acquisition is achieved using a spatially distributed camera array. Specifically, in the spatial dimension, a precisely calibrated camera array synchronously captures scene images at the same moment. There is precisely controlled sub-pixel micro-displacement between each camera, forming a low-resolution image set with complementary spatial sampling. In the temporal dimension, each independent camera continuously captures scene images at a fixed frame rate, where objects move and change, forming a low-resolution video stream with multi-scale information in time.

[0086] The deep neural network part guided by clear dimensions realizes the efficient use of spatiotemporal information in multiple videos and completes the super-resolution reconstruction task, including sending the images taken by multiple cameras in the same spatial dimension at the same time into the detail-guided multi-layer feature extraction module in the deep neural network for feature extraction; among them, the spatial dimension at the same time is a set of low-resolution images with complementary spatial sampling at the same time, and the video stream refers to different moments. Specifically, the six images at each moment are fused into one feature, and the feature propagation is carried out along the time dimension. The features at different moments are cyclically fused. The entire framework is shown in FIG. Figure 5 shown.

[0087] The spatial transformation consistency implicit alignment module obtains the registered image features through a coarse-to-fine two-level registration method combined with a camera array model and an implicit resampling method. The image features are transferred from the spatial dimension to the temporal dimension through the time-to-spatial feature fusion module, and a bidirectional second-order propagation network is used to propagate the feature sequences at different moments in the temporal dimension. Finally, global residual cascade upsampling is used to obtain the reconstructed image frames and output high-resolution video.

[0088] Among them, the two-stage registration method uses the affine matrix representing the positional relationship between cameras for coarse registration, and then uses the implicit resampling method based on optical flow for fine registration.

[0089] The camera array model and implicit resampling method are specific imaging models that refer to the positional relationship between camera arrays. The implicit resampling method steps are: calculating the positional relationship between features through optical flow, using the coordinate network to model the motion vector, and aligning the preliminary compensation features to the reference features through the cross-attention mechanism.

[0090] The propagation of the time dimension is to further fuse the features fused in the spatial dimension along the time dimension, and use the information in the time dimension (such as the motion changes between the previous and next frames) to further improve the image resolution and quality.

[0091] This multi-video super-resolution reconstruction method, based on a deep neural network, employs an offline training strategy using semi-virtual high-resolution video and multiple low-resolution video pairs. Specifically, the method simulates the inverse camera imaging process by combining a real video dataset with the spatial arrangement characteristics of a camera array to generate multiple low-resolution videos, which form training samples for the high-resolution video-multiple low-resolution video pairs.

[0092] The "semi-virtual" in these semi-virtual high-resolution videos means that these low-resolution videos are not completely obtained from real data, but are instead synthesized or generated in some way. Specifically, the low-resolution videos are generated by combining a real video dataset (high-resolution video) with the spatial arrangement characteristics of the camera array, using an inverse camera imaging process to simulate the generation.

[0093] Multi-resolution video is the low-resolution image set and low-resolution video stream mentioned above. It is multiple low-resolution videos determined from two different dimensions. From the spatial dimension of the camera array, multiple low-resolution image sets are generated at the same moment (not considering the temporal dimension); from the temporal dimension, these low-resolution video streams are generated at different moments. A high-resolution video and multiple low-resolution videos form a high-low-resolution video pair.

[0094] Multiple low-resolution videos are generated by simulating the camera array imaging degradation process of high-resolution videos (real scenes).

[0095] During network training, the Charbonnier loss function is used to enhance the model's robustness to noise and outliers, and the adaptive moment estimation method (Adam) is combined to optimize network parameters to ensure the stability and efficiency of model convergence. The core innovation of this application lies in fully utilizing the complementary information of the temporal and spatial dimensions. Through a deep neural network architecture guided by clear dimensions, it effectively utilizes the position relationship of the camera array and the relationship between the previous and next frames in the time series to decode the information of different low-resolution images in multiple video sequences, thereby achieving high-fidelity super-resolution image reconstruction.

[0096] Among them, the decoding process is to use the position relationship of the camera array to fuse the features of the spatial position, and propagate the features along the time dimension, and utilize the image information at different positions.

[0097] Based on the same inventive concept, embodiments of the present application also provide a camera array imaging device for implementing the multi-video super-resolution reconstruction method described above. The solution provided by this device is similar to the solution described in the method described above. Therefore, the specific limitations of one or more camera array imaging device embodiments provided below can be found in the limitations of the multi-video super-resolution reconstruction method described above and will not be repeated here.

[0098] In an exemplary embodiment, Figure 10 As shown, a camera array imaging device is provided, comprising:

[0099] The camera array image acquisition module is used to receive the global synchronization signal based on the synchronization control module to control the camera array to simultaneously acquire multiple video sequences within the field of view, and transmit the multiple video sequences to the multi-video processing module.

[0100] In practical applications, the camera array image acquisition module comprises six industrial cameras, a rigid frame, and an image acquisition card. The six cameras are mounted on a designed rigid frame, and the actual positional relationships between the cameras are determined through calibration. This module primarily receives signals from the synchronization control module to control the multiple cameras in the camera array to simultaneously capture images within their field of view and store historical images. The captured images and the positional relationships between the cameras are transmitted to the image processing module via a high-speed transmission interface, providing high-quality input data for subsequent multi-video super-resolution reconstruction.

[0101] The synchronization control module is used to generate a global synchronization signal and transmit the global synchronization signal to the camera array image acquisition module.

[0102] In practical applications, the synchronization control module generates a global synchronization signal, precisely controls key parameters such as the exposure time and acquisition frame rate of each camera, ensures the time synchronization and parameter consistency of the camera array, and provides reliable guarantees for high-quality image acquisition.

[0103] The multi-video processing module is used to perform super-resolution reconstruction on the multi-video sequences according to the multi-video super-resolution reconstruction method.

[0104] In practical applications, the multi-video processing module builds a deep neural network based on the PyTorch deep learning framework and trains and fine-tunes the super-resolution reconstruction network using stored public datasets. Based on the trained model, the module reconstructs high-resolution video from multiple low-resolution segments of video captured by the camera array at the current and previous moments. Furthermore, the module supports real-time inference, efficiently processing the multi-video data captured by the camera array and outputting high-quality, high-resolution video sequences.

[0105] REDS, Vimeo-90K, and Vid4 are three important datasets for evaluating the performance of video super-resolution algorithms. The REDS dataset contains 500 high-quality, real-world long video clips, while Vimeo-90K provides 90,000 short video clips covering different scenes, lighting conditions, and complex dynamics, making it particularly suitable for evaluating the performance of video super-resolution algorithms in diverse scenarios. The Vid4 dataset contains four high-quality short videos, making it suitable for testing the performance of super-resolution algorithms in dynamic videos. When training deep neural networks for multi-video super-resolution reconstruction and evaluating the performance of super-resolution algorithms, the three datasets are inversely transformed into a camera array imaging model to generate a multi-video dataset as the algorithm input, and the original video is used as the ground truth to evaluate the performance of the super-resolution algorithm.

[0106] Super-resolution algorithms take a low-resolution image as input and generate a high-resolution image. The peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) calculated between the high-resolution image and the ground-truth image are used to evaluate algorithm performance. PSNR primarily assesses the overall pixel error between the reconstructed image and the ground-truth image; a higher PSNR indicates a smaller difference between the generated image and the ground-truth image. SSIM primarily assesses the perceptual quality and structural similarity of an image and is more consistent with the performance of the human visual system than PSNR. A SSIM closer to 1 indicates a smaller difference between the generated image and the ground-truth image. For the REDS4 dataset, PSNR and SSIM are calculated on the RGB channels, while for the Vimeo-90K-T and Vid4 datasets, they are calculated on the Y channel.

[0107] Compared with traditional video super-resolution and multi-image super-resolution reconstruction on the above improved test dataset, this application uses spatiotemporal information to achieve more accurate reconstruction of details. Figure 11-12 shown.

[0108] In order to further verify the advantages of this application in processing complex scene images by using multi-dimensional information, optical blur and random motion blur were introduced into the dataset, and the original network was retrained with different input dimensions. Figure 13 It can be seen that the reconstruction effect of this application when input is in different dimensions, where S represents the spatial dimension input and T represents the temporal dimension input. The inherent temporal correlation in the video sequence helps to restore motion blur more accurately, while the multi-camera spatial high-frequency information can more effectively restore optical blur, thereby restoring the details of the wheel pattern and achieving better reconstruction effect.

[0109] This application uses a camera array device to shoot real videos and uses a dimension-guided multi-video super-resolution reconstruction neural network to generate high-resolution videos. The performance test results are as follows: Figure 14 shown.

[0110] The results show that the details of the low-resolution image are obviously blurred, while the image reconstructed by this application shows clear details. The sources of high-quality reconstruction achieved by this application are mainly the following two aspects: (1) using a camera array imaging device to obtain multiple videos, and using the micro-displacement between cameras and the multi-scale changes in the video to obtain high-frequency information in multiple dimensions of time and space, thereby increasing the amount of information contained in the input; (2) using an image processing module with a clear physical imaging process to construct a deep neural network guided by clear dimensions, which can effectively decode the additional information of the time and space dimensions. The reconstructed image appears more realistic and detailed, with smoother edge transitions and less blur, ultimately improving the overall visual quality.

[0111] Experiments show that by utilizing diverse and information-rich spatiotemporal sampling, this application can more easily solve the challenges brought by the ill-posed mapping matrix in the reconstruction process, significantly improve the authenticity and detail restoration capabilities of the reconstructed image, and is superior to traditional video or multi-camera super-resolution reconstruction methods, completing super-resolution reconstruction tasks in complex scenes.

[0112] In an exemplary embodiment, a computer device is provided, which may be a server or a terminal. The computer device includes a processor, a memory, an input / output (I / O) interface, and a communication interface. The processor, memory, and I / O interface are connected via a system bus, and the communication interface is connected to the system bus via the I / O interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store multi-video super-resolution reconstruction data. The I / O interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a multi-video super-resolution reconstruction method is implemented.

[0113] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the above method when executing the computer program.

[0114] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the above method when executed by a processor.

[0115] In an exemplary embodiment, a computer program product is provided, including a computer program, which implements the above method when executed by a processor.

[0116] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0117] In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0118] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0119] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0120] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A multi-video super-resolution reconstruction method, characterized in that: include: Based on the historical video sequences collected by the camera array and the camera array imaging model, a multi-video super-resolution dataset is constructed; Using the multi-video super-resolution dataset to train a multi-video super-resolution reconstruction network, and enabling the multi-video super-resolution reconstruction network to learn multi-video spatiotemporal feature extraction and fusion capabilities during the training phase; Optimizing network parameters of the multi-video super-resolution reconstruction network to construct an optimized multi-video super-resolution reconstruction network; The multi-video sequences to be tested captured by the camera array are input into the optimized multi-video super-resolution reconstruction network for super-resolution reconstruction, and the reconstructed super-resolution video sequence is determined.

2. The multi-video super-resolution reconstruction method according to claim 1, characterized in that Based on the historical video sequences collected by the camera array and the camera array imaging model, a multi-video super-resolution dataset is constructed, specifically including: In combination with a camera array imaging model, for each high-resolution video frame in the historical video sequence, a horizontal image is selected as a reference image, and the reference image is rotated based on a set increment to generate a plurality of additional images; applying random translation and rotation transformations to the additional image to determine a transformed image; Processing all transformed images and the reference image using a bicubic downsampling method to simulate image degradation and generate degraded images; the degraded images simulate low-resolution video frames simultaneously captured by the camera array; All low-resolution video frames are regarded as low-resolution videos acquired by the camera array, and all high-resolution video frames are regarded as high-resolution videos acquired by the camera array; A high-low resolution video pair constructed from the plurality of low-resolution videos and the corresponding high-resolution videos is used as a multi-video super-resolution dataset.

3. The multi-video super-resolution reconstruction method according to claim 1, characterized in that The multi-video super-resolution reconstruction network specifically includes: a feature extraction module, a feature registration module, a space-to-time feature fusion module, a feature propagation module and an image reconstruction module; Based on the feature extraction module, high-frequency detail information and image features are extracted, and a high-dimensional feature block is output; the feature extraction module is a detail-guided multi-layer feature extraction module, including an image branch, a detail branch, and an attention mechanism unit; the image branch is used to obtain image features through position encoding; the detail branch is used to use the LOG operator and the Sobel operator to extract high-frequency detail information of the image acquired by the camera array at the same time; the attention unit is used to enhance the image features with the high-frequency detail information through the attention mechanism, and output a high-dimensional feature block; Based on the feature registration module, the high-dimensional feature block is divided into reference features and auxiliary features. The auxiliary features are affine transformed in combination with the affine transformation matrix to obtain preliminary compensation features. Motion estimation is performed using a pre-trained optical flow estimator, and motion vectors are modeled using a coordinate network. The preliminary compensation features are aligned to the reference features through a cross-attention mechanism, and registration features with the same dimension as the reference features are determined. The feature registration module is a spatial transformation consistency implicit alignment module. Based on the space-to-time feature fusion module, a dynamic most adaptive feature fusion strategy is adopted to fuse the reference features and the registration features, and the multi-layer perceptron is used as a primary layer of global residual upsampling for image reconstruction; the primary layer is used to feed the fused features and the upsampled reference frame into the multi-layer perceptron to convert them into propagation input features in the time dimension; the propagation input features are time features; Based on the feature propagation module, a second-order grid propagation cyclic structure is used to learn the multi-scale information between the temporal features, and the temporal features are gradually refined and enhanced according to the continuous extraction of inter-frame information; Based on the image reconstruction module, a super-resolution video is reconstructed by upsampling the high-resolution video through a secondary cascade global staggered approach according to the enhanced temporal features.

4. The multi-video super-resolution reconstruction method according to claim 1, characterized in that: The loss function of the multi-video super-resolution reconstruction network is Loss Charbonnier for: Where T represents the number of frames contained in each input video during network training; represents the i-th high-resolution image output by the network using temporal and spatial information from multiple videos; represents the true value of the corresponding i-th high-resolution image; ∈ represents the bias constant.

5. The multi-video super-resolution reconstruction method according to claim 1, characterized in that: Optimizing the network parameters of the multi-video super-resolution reconstruction network specifically includes: The model weights obtained by training the multi-video super-resolution reconstruction network using the REDS dataset are used as pre-training weights; The network parameters are initialized with the pre-trained weights, and the network parameters are iteratively trained on the Vimeo-90K dataset using a cosine learning rate decay strategy to determine the optimized network parameters.

6. The multi-video super-resolution reconstruction method according to claim 1, characterized in that: The multi-video sequence to be tested, which is captured by the camera array, is input into the optimized multi-video super-resolution reconstruction network for super-resolution reconstruction, and the reconstructed super-resolution video sequence is determined, specifically including: The multi-video sequences to be tested, captured by the camera array, are input into an optimized multi-video super-resolution reconstruction network. The multi-video super-resolution reconstruction network, which has learned the multi-video spatiotemporal feature extraction and fusion capabilities during the training phase, performs super-resolution reconstruction on the input multiple multi-video sequences to be tested, and determines the reconstructed super-resolution video sequences; the multi-video sequences to be tested are low-resolution video sequences.

7. A camera array imaging device, characterized in that: include: A camera array image acquisition module is configured to receive a global synchronization signal based on the synchronization control module to control the camera array to simultaneously acquire multiple video sequences within the field of view, and transmit the multiple video sequences to the multi-video processing module; A synchronization control module, configured to generate a global synchronization signal and transmit the global synchronization signal to the camera array image acquisition module; A multi-video processing module is used to perform super-resolution reconstruction on the multi-video sequences according to the multi-video super-resolution reconstruction method according to any one of claims 1 to 6.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-video super-resolution reconstruction method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-video super-resolution reconstruction method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-video super-resolution reconstruction method according to any one of claims 1 to 6 is implemented.

Images