Panoramic video stability enhancement method based on space-time consistency

Abstract

The invention relates to the technical field of panoramic video processing, in particular to a panoramic video stability enhancement method based on space-time consistency, which comprises the following steps: acquiring a panoramic video frame sequence; obtaining spherical projection coordinates of each frame in the panoramic video frame sequence; based on the spherical projection coordinates, a deep learning model is constructed, and the deep learning model comprises a feature extraction encoder and a motion prediction compensation decoder; performing feature extraction on the panoramic video frame sequence by using a feature extraction encoder; performing motion prediction and compensation by using a motion prediction compensation decoder based on a feature extraction result; constructing a space-time filter according to the motion prediction and compensation result, and smoothing the motion vector; based on the smoothed motion vector, performing multi-frame motion analysis by using a long time sequence network; according to a multi-frame motion analysis result, designing panoramic video optical flow motion estimation; performing stable enhancement processing on the panoramic video frame sequence based on an optical flow motion estimation result; and outputting the stably enhanced panoramic video.

Description

Technical Field

[0001] The present invention relates to the technical field of panoramic video processing, and more specifically, to a panoramic video stabilization enhancement method based on spatiotemporal consistency. Background Art

[0002] With the rapid development of virtual reality and augmented reality technologies, panoramic videos are increasingly used in various fields. However, due to the movement and vibration of the shooting equipment, the original panoramic videos are often unstable, which seriously affects the viewing experience and the effect of subsequent processing. Therefore, the stabilization enhancement technology of panoramic videos has become a hot topic in current research.

[0003] Traditional video stabilization methods are mainly based on feature point matching and motion estimation, which eliminate unnecessary camera motion by calculating inter-frame transformations. These methods perform well when processing ordinary videos, but encounter many challenges when facing panoramic videos. First, the spherical geometry of panoramic videos makes traditional planar transformation models no longer applicable, resulting in poor stabilization effects. Second, the large field of view and high resolution of panoramic videos bring a huge computational burden, making it difficult to achieve real-time processing. In addition, traditional methods often only consider the relationship between frames in a short period of time, and cannot effectively handle the accumulated errors in long time series, which easily leads to picture drift and deformation.

[0004] In recent years, deep learning-based video stabilization methods have made some progress. These methods directly learn the stabilization transformation by training neural networks, avoiding explicit feature matching and motion estimation steps. However, most of the existing deep learning methods are designed for general videos and fail to fully consider the particularity of panoramic videos. For example, some methods will produce severe distortion and discontinuity when processing the edge areas of panoramic videos. In addition, due to the lack of modeling of the long-term consistency of panoramic videos, these methods often experience jitter and unnatural motion when processing long sequence videos.

[0005] In addition, existing technologies still have many deficiencies when dealing with complex dynamic scenes. For example, when there are multiple moving objects in a scene, a simple global motion model is difficult to accurately describe the scene changes, which can easily cause object motion blur or ghosting. At the same time, for scenes with fast motion or drastic changes in lighting, existing methods often have difficulty maintaining consistency in stabilization effects.

[0006] In view of the above problems, there is an urgent need for a stable enhancement method that can fully utilize the characteristics of panoramic video while taking into account local details and global consistency. The panoramic video stable enhancement method based on spatiotemporal consistency proposed in the present invention is designed to solve these technical problems. Summary of the invention

[0007] In order to solve the above technical problems, the present invention provides a panoramic video stabilization enhancement method based on spatiotemporal consistency. The method of the present invention realizes comprehensive and in-depth panoramic video stabilization enhancement by innovatively combining spherical projection coordinate system, deep learning model, spatiotemporal filter, long time series network and optical flow motion estimation. This method can not only effectively process the spherical geometric characteristics of panoramic videos, but also capture complex motion patterns in long time series while maintaining local details and global consistency.

[0008] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0009] The panoramic video stabilization enhancement method based on spatiotemporal consistency includes:

[0010] The acquisition steps include:

[0011] Get panoramic video frame sequence;

[0012] Obtaining the spherical projection coordinates of each frame in the panoramic video frame sequence;

[0013] Processing steps include:

[0014] Based on the spherical projection coordinates, a deep learning model is constructed, wherein the deep learning model includes a feature extraction encoder and a motion prediction compensation decoder;

[0015] Using the feature extraction encoder, extracting features from the panoramic video frame sequence;

[0016] Based on the feature extraction result, using the motion prediction and compensation decoder to perform motion prediction and compensation;

[0017] According to the motion prediction and compensation results, a spatiotemporal filter is constructed to smooth the motion vector;

[0018] Based on the smoothed motion vector, a multi-frame motion analysis is performed using a long temporal network;

[0019] Designing panoramic video optical flow motion estimation based on the multi-frame motion analysis results;

[0020] Output steps include:

[0021] Based on the optical flow motion estimation result, performing stable enhancement processing on the panoramic video frame sequence;

[0022] Outputs a stable and enhanced panoramic video.

[0023] Preferably, the step of acquiring a panoramic video frame sequence specifically includes:

[0024] Use multiple cameras to capture the same scene at the same time to obtain a 360-degree panoramic image;

[0025] The panoramic image is converted into a spherical projection coordinate system.

[0026] Preferably, the step of constructing a deep learning model specifically includes:

[0027] Design a feature extraction encoder for feature extraction in spherical space;

[0028] Design a motion prediction and compensation decoder for motion prediction and compensation in spherical space;

[0029] A bidirectional mapping is established between the feature extraction encoder and the motion prediction compensation decoder.

[0030] Preferably, the step of constructing the space-time filter specifically includes:

[0031] Design multi-scale spatial pyramid structures;

[0032] At each scale of the spatial pyramid structure, a spatiotemporal filter is applied for motion vector smoothing.

[0033] Preferably, the step of performing multi-frame motion analysis using a long temporal network specifically includes:

[0034] Use LSTM encoder and decoder to build long time series network;

[0035] Taking the residual motion of the previous frame as the input of the long temporal network;

[0036] The long temporal network is used to generate a motion prediction result of the current frame.

[0037] Preferably, the step of designing panoramic video optical flow motion estimation specifically includes:

[0038] In the spherical projection coordinate system, the optical flow motion vector between adjacent frames is calculated;

[0039] The motion field in the panoramic video is estimated using the optical flow motion vector.

[0040] Preferably, the step of performing stable enhancement processing on the panoramic video frame sequence specifically includes:

[0041] Predicting and compensating motion residuals pixel by pixel in spherical space using the deep learning model;

[0042] reconstructing an enhanced panoramic video frame in a spherical space based on the motion residual;

[0043] The reconstructed panoramic video frame is projected back to a planar coordinate system.

[0044] Preferably, in the step of predicting and compensating motion residual pixel by pixel:

[0045] The encoder of the deep learning model performs feature extraction pixel by pixel;

[0046] The decoder of the deep learning model performs residual motion prediction and compensation pixel by pixel.

[0047] As a preference, it also includes:

[0048] Optimizing the spatiotemporal consistency of the stabilized and enhanced panoramic video;

[0049] The optimization results are further adjusted using the LSTM module.

[0050] Preferably, in the step of adjusting using the LSTM module:

[0051] The loss function that considers the cross-correlation between pixels is used for optimization;

[0052] The LSTM module is trained by an error back propagation algorithm.

[0053] The beneficial effects of the present invention are mainly reflected in the following aspects:

[0054] First, by introducing the spherical projection coordinate system and the corresponding deep learning model, this method effectively solves the geometric distortion problem in panoramic video processing. This design not only ensures the consistency of the stabilization effect within the panoramic range, but also greatly improves the accuracy of feature extraction and motion prediction.

[0055] Secondly, the application of multi-scale spatiotemporal filters enables this method to simultaneously consider motion information of different scales, effectively smooth motion vectors, and reduce jitter and discontinuity. This multi-scale processing strategy greatly improves the robustness of the stabilization effect and can cope with various complex camera motions and scene changes.

[0056] Furthermore, the introduction of long temporal networks enables this method to capture and utilize long-term temporal dependencies. This innovation not only effectively solves the problem of cumulative errors in long-sequence video processing, but also produces a more natural and coherent stabilization effect. In particular, this method shows obvious advantages when processing panoramic videos with long duration and complex motion.

[0057] In addition, motion estimation based on optical flow and optimization of spatiotemporal consistency further improve the accuracy and consistency of the stabilization effect. This refined processing can not only accurately capture local motion, but also maintain global spatiotemporal consistency, effectively avoiding the image distortion and unnatural motion common in traditional methods.

[0058] Finally, the modular design and end-to-end training strategy of this method ensure that the entire system works together, and each component can complement and enhance each other. This design not only improves the overall performance, but also gives the method good scalability and adaptability, and can flexibly respond to different application scenarios and requirements.

[0059] In general, the panoramic video stabilization and enhancement method based on spatiotemporal consistency provided by the present invention has achieved a qualitative leap by systematically solving the key technical problems in panoramic video stabilization. This method has not only achieved significant improvements in objective indicators such as peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), but also reached new heights in subjective visual effects. The stabilized panoramic video shows a smoother and more natural visual effect, which greatly improves the viewing experience. At the same time, due to the high fidelity of details, the stabilized video is more suitable for subsequent processing and analysis tasks, opening up new possibilities for the application of panoramic videos in virtual reality, augmented reality, video surveillance and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 The figure is an overall flow chart of the method of the present invention.

[0061] Figure 2 It is a logic block diagram of the long time series network processing of the present invention.

[0062] Figure 3 It is a logic block diagram of the pixel-by-pixel processing of the present invention. DETAILED DESCRIPTION

[0063] like Figure 1-3 As shown, the present invention provides a panoramic video stabilization enhancement method based on spatiotemporal consistency. The method effectively solves the key problems in panoramic video stabilization enhancement by innovatively combining spherical projection coordinate system and deep learning technology.

[0064] First, the method of the present invention includes an acquisition step. In this step, a panoramic video frame sequence and the spherical projection coordinates of each frame are obtained. Specifically, multiple cameras can be used to simultaneously perform 360-degree all-round acquisition of the same scene to obtain a complete panoramic image. Preferably, 6 fisheye cameras are arranged at 60-degree intervals to achieve seamless coverage. In one embodiment of the present invention, an equirectangular projection is used to convert the panoramic image into a spherical projection coordinate system. The equirectangular projection can be expressed by the following formula:

[0065] x=(λ-λ0)cosφ1

[0066] y=(φ-φ1)

[0067] Among them, x and y are the projection plane coordinates, λ and φ are longitude and latitude respectively, and λ0 and φ1 are the center longitude and latitude. This projection method can maintain the linearity of longitude and latitude, which is convenient for subsequent processing.

[0068] Next, the method of the present invention enters the processing step. Based on the acquired spherical projection coordinates, a deep learning model is constructed. The model includes a feature extraction encoder and a motion prediction compensation decoder. The feature extraction encoder adopts a structure similar to that of the VGG network, including multiple layers of convolution and pooling operations. Preferably, 5 layers of convolution layers are used, each followed by a maximum pooling layer, the convolution kernel size is 3x3, and the step size is 1. The motion prediction compensation decoder adopts a transposed convolution structure, which gradually upsamples to restore the original resolution. In one embodiment of the present invention, the decoder includes 4 layers of transposed convolution layers, each followed by a normal convolution layer for detail adjustment.

[0069] Using the feature extraction encoder, feature extraction is performed on the panoramic video frame sequence. This step can be expressed as:

[0070] F t =E(I t )

[0071] Among them, F t is the feature map of the tth frame, E() represents the encoder operation, I t is the t-th frame image of the input.

[0072] Based on the feature extraction results, the motion prediction and compensation decoder is used to perform motion prediction and compensation. This process can be expressed as:

[0073] M t =D(F t ,F t-1 )

[0074] Among them, M t is the predicted motion picture, D() represents the decoder operation, and F t and F t-1 are the feature maps of the current frame and the previous frame respectively.

[0075] According to the motion prediction and compensation results, the method constructs a spatiotemporal filter to smooth the motion vector. Preferably, a bilateral filter is used for smoothing, and its mathematical expression is:

[0076]

[0077] in, ) is the motion vector at point p after smoothing, W p is the normalization factor, S is the filter window, and are the spatial and range Gaussian kernels, σs and σ r is the corresponding standard deviation. In one embodiment of the present invention, σ s The value is 5, σ r The value is 0.1. These parameter values ​​have been experimentally verified to achieve a good balance between smoothing effect and detail preservation.

[0078] Based on the smoothed motion vector, this method uses a long temporal network to perform multi-frame motion analysis. Specifically, the LSTM (Long Short-Term Memory) network structure is used, and its core formula is as follows:

[0079] f=σ(W f ·[h t-1 ,x t ]+b f )

[0080] i=σ(W i ·[h t-1 ,x t ]+b i )

[0081]

[0082] o t =σ(W o ·[h t-1 ,x t ]+b o )

[0083] h t =o t *tanh(C t )

[0084] Among them, f t 、i t and t They are forget gate, input gate and output gate respectively, C t is the cell state, h t is the hidden state, W and b are the weight and bias parameters, σ is the sigmoid activation function, and tanh is the hyperbolic tangent function.

[0085] According to the results of multi-frame motion analysis, this method designs panoramic video optical flow motion estimation. Preferably, an improved Lucas-Kanade optical flow algorithm is used, and its objective function is:

[0086]

[0087] Wherein, I is the image intensity, (u, v) is the optical flow vector, and λ is the regularization parameter. In one embodiment of the present invention, λ is set to 0.01, which can achieve a good compromise between optical flow accuracy and smoothness.

[0088] Finally, the method of the present invention enters the output step. Based on the optical flow motion estimation result, the panoramic video frame sequence is stabilized and enhanced. Specifically, image stabilization is achieved through a backward warping operation:

[0089] I stable (x,y,t)=I(x+u,y+v,t)

[0090] Among them, I stable is the stabilized image, and (u,v) is the estimated optical flow vector.

[0091] The method of the present invention realizes efficient and stable enhancement of panoramic video through the above steps. Compared with the prior art, the present method has the following advantages: first, the use of spherical projection coordinate system effectively solves the problem of geometric distortion in panoramic video processing; second, the introduction of deep learning model greatly improves the accuracy of feature extraction and motion prediction; third, the application of spatiotemporal filter and long time series network ensures the spatiotemporal consistency of video stability; finally, motion estimation based on optical flow further optimizes the stabilization effect. Experiments show that the present method is superior to the existing methods in objective indicators such as PSNR (peak signal-to-noise ratio) and SSIM (structural similarity), and the subjective visual effect is also more outstanding.

[0092] It is worth noting that in practical applications, this method can also adjust parameters according to specific scenarios. For example, for fast-moving scenes, the spatial standard deviation σ of the spatiotemporal filter can be appropriately increased. s , to achieve a stronger smoothing effect; for scenes with rich details, σ can be reduced r value to retain more detailed information. In addition, the number of layers and hidden units of the LSTM network can also be adjusted according to the length and complexity of the video sequence to achieve a balance between computational efficiency and performance.

[0093] In summary, the panoramic video stabilization enhancement method based on spatiotemporal consistency provided by the present invention effectively solves the key problems in panoramic video stabilization enhancement by innovatively combining multiple advanced technologies, and provides new ideas and methods for the development of related fields. The method of the present invention further includes the step of constructing a spatiotemporal filter. This step specifically includes designing a multi-scale spatial pyramid structure and applying a spatiotemporal filter to smooth motion vectors at each scale of the structure.

[0094] Preferably, the method uses a Gaussian pyramid as a multi-scale spatial structure. Specifically, starting from the original image, each level of the pyramid is constructed through continuous downsampling operations. The image size of each layer is half of the previous layer, which can be expressed by the following formula:

[0095]

[0096] Among them, I l represents the image of the lth layer, and w(m,n) is the Gaussian weight kernel. In one embodiment of the present invention, a 5x5 Gaussian kernel is used, and the standard deviation σ is set to 1.0. This parameter setting can achieve a good balance between smoothing effect and computational efficiency.

[0097] At each scale of the pyramid, the method applies a spatiotemporal filter to smooth the motion vector. Preferably, a bilateral spatiotemporal filter is used, and its mathematical expression can be expanded to:

[0098]

[0099] Among them, among them, is the smoothed motion vector at layer l, time t, and position p, W p is the normalization factor, S is the spatial filter window, T is the temporal filter window radius, and are the space, time and range Gaussian kernels respectively. In the preferred embodiment of the present invention, σ s , σ t and σ r The values ​​are 5, 2 and 0.1 respectively, and the time window radius T is 3. This set of parameters has been verified by a large number of experiments and can achieve good smoothing effects in different scenarios.

[0100] Next, the method of the present invention uses a long time series network to perform multi-frame motion analysis. Specifically, an LSTM (Long Short-Term Memory) encoder and decoder are used to construct a long time series network. The core of LSTM lies in its special gating mechanism, which can effectively capture long-term dependencies.

[0101] In this method, the input of the LSTM encoder is the residual motion of the previous frame. Assume that at time t, the input sequence is x t =[r t-k ,...,r t-1 ], where r i represents the residual motion of the i-th frame, and k is the number of historical frames considered. The output of the encoder is a latent vector z t , which can be expressed as:

[0102] z t =LSTM enc (x t )

[0103] Among them, LSTM enc () indicates encoder operation.

[0104] The decoder uses the latent vector z tGenerate motion prediction results for the current frame

[0105]

[0106] Among them, LSTM dec () indicates decoder operation.

[0107] In one embodiment of the present invention, the LSTM network adopts a 2-layer structure, each layer contains 256 hidden units. This setting can achieve good performance in most scenarios while maintaining low computational complexity. Preferably, the Adam optimizer is used for model training, and the initial value of the learning rate is set to 0.001, and decays to 0.1 times the original value every 50 epochs.

[0108] After obtaining the output of the long time series network, this method further designs the panoramic video optical flow motion estimation. This step is performed in the spherical projection coordinate system to fully utilize the geometric characteristics of the panoramic video. Specifically, for a point on the sphere (θ, ), its optical flow can be expressed as:

[0109]

[0110] Here, u and v represent the movement in longitude and latitude respectively.

[0111] In order to solve the above optical flow equation, this method adopts the variational method framework. Preferably, the following energy function is defined:

[0112]

[0113] Among them, E D is the data item, E S is the smoothing term, and α is the smoothing coefficient. The data term and the smoothing term can be defined as:

[0114]

[0115] Among them, I t ,I θ and Represent the image at time, θ and Partial derivatives in direction.

[0116] In a preferred embodiment of the present invention, the alternating direction method of multipliers (ADMM) is used to solve the above variational problem. This method can effectively process the optical flow estimation on the spherical domain while maintaining high computational efficiency.

[0117] Finally, based on the optical flow motion estimation results, this method performs a stable enhancement process on the panoramic video frame sequence. Specifically, image stabilization is achieved through a backward warping operation:

[0118]

[0119] Among them, I stable () is the stabilized image, (u,v) is the estimated optical flow vector.

[0120] In order to deal with the hole problem that may be caused by the distortion operation, this method adopts the image restoration technology based on deep learning. Preferably, a partial convolution-based restoration network is used, which can effectively handle the missing areas of irregular shapes and maintain the overall structure and texture consistency of the image.

[0121] Through the above steps, the method of the present invention achieves efficient and stable enhancement of panoramic video. It is worth noting that the various modules of the present method can be flexibly adjusted according to specific application scenarios. For example, for scenarios with limited computing resources, the number of layers or hidden units of the LSTM network can be reduced; for applications with high real-time requirements, a lighter optical flow estimation algorithm can be used. This flexibility enables the present method to adapt to different application requirements and has a wide range of practical value.

[0122] In general, the panoramic video stabilization and enhancement method based on spatiotemporal consistency provided by the present invention effectively solves the key problems in panoramic video stabilization and enhancement by innovatively combining technologies such as multi-scale spatial pyramids, spatiotemporal filtering, long time series networks, and spherical optical flow estimation. This method can not only handle complex camera motions, but also maintain the global consistency and local details of the video, providing a new solution for the field of panoramic video processing. The method of the present invention adopts a carefully designed deep learning model in the steps of pixel-by-pixel prediction and compensation of motion residuals. The encoder and decoder of the model respectively undertake the important tasks of feature extraction and residual motion prediction compensation.

[0123] Specifically, the encoder of the deep learning model adopts a structure similar to U-Net to extract features pixel by pixel. Preferably, the encoder contains 5 downsampling blocks, each of which consists of two 3x3 convolutional layers and a maximum pooling layer. The number of channels in the convolutional layer starts from 64 and doubles with each downsampling, eventually reaching 1024 channels. This structural design can effectively capture the multi-scale features of the image and provide rich information for subsequent motion prediction.

[0124] The mathematical expression of the encoder can be summarized as follows:

[0125] F i =Ei (P i-1 (F i-1 )),i=1,2,…,5

[0126] Among them, F i represents the feature map of the i-th layer, E_i represents the operation of the i-th encoding block, P i-1 Represents a pooling operation.

[0127] In one embodiment of the present invention, in order to enhance the expressiveness of the model, an attention mechanism is introduced in each coding block. Specifically, a spatial attention module is adopted, and its mathematical expression is:

[0128]

[0129] in, represents a 7x7 convolution operation, σ is a sigmoid activation function, and \odot represents element-by-element multiplication. This attention mechanism can help the model focus on important spatial areas and improve the efficiency of feature extraction.

[0130] The decoder of the deep learning model is responsible for residual motion prediction and compensation pixel by pixel. The decoder adopts an upsampling structure symmetrical to the encoder, and introduces skip connections to retain detail information. Each upsampling block contains a transposed convolution layer and two ordinary convolution layers. The mathematical expression is as follows:

[0131] R i =D i (U i (R i+1 )+F i ),i=4,3,…,1

[0132] Among them, R i represents the residual prediction of the i-th layer, D i represents the operation of the i-th decoding block, U i Represents an upsampling operation.

[0133] In order to further improve the prediction accuracy, this method introduces a residual learning mechanism in the decoder. Specifically, the output of the last layer is regarded as a residual and added to the input image to obtain the final prediction result:

[0134] I pred =I input +R1

[0135] This residual learning mechanism can help the model learn small motion changes more easily and improve the stabilization effect.

[0136] The method of the present invention also includes a step of optimizing the temporal and spatial consistency of the panoramic video after stabilization enhancement, which is intended to eliminate possible jitter and discontinuity and further improve the visual quality of the video.

[0137] Preferably, the method adopts a spatiotemporal consistency optimization strategy based on optical flow. Specifically, firstly, the forward and backward optical flows between adjacent frames are calculated:

[0138] (u f ,v f )=OF(I t ,I t+1 )

[0139] (u b ,v b )=OF(I t+1 ,I t )

[0140] Among them, OF represents the optical flow estimation algorithm, (u f ,v f ) and (u b ,v b ) represent the forward and backward optical flows respectively.

[0141] Then, based on the calculated optical flow, the spatiotemporal consistency constraints are constructed:

[0142]

[0143] Among them, E temp represents the time consistency constraint, E spat represents the spatial consistency constraint, α t is a learnable occlusion mask.

[0144] In one embodiment of the present invention, the video sequence is optimized by minimizing the above constraints:

[0145]

[0146] Wherein, λ is a balancing factor, which is used to adjust the relative importance of temporal and spatial consistency. Preferably, λ is set to 0.5, which can achieve a good balancing effect in most scenarios.

[0147] Finally, the method of the present invention uses the LSTM module to further adjust the optimization results. The introduction of the LSTM module can capture long-term temporal dependencies and further improve the stability and coherence of the video.

[0148] Specifically, the LSTM module receives the output sequence of the previous step as input:

[0149] h t ,ct =LSTM(x t ,h t-1 ,c t-1 )

[0150] Among them, x t is the feature of the current frame, h t and c t are the hidden state and the cell state, respectively.

[0151] During the training process, this method uses a loss function that considers the cross-correlation between pixels for optimization. This loss function includes reconstruction loss and structural similarity loss:

[0152] L=L recon +βL ssim

[0153] Among them, L recon For the reconstruction loss, the L1 norm is used:

[0154]

[0155] L ssim is the structural similarity loss, defined as:

[0156]

[0157] Wherein, SSIM represents the structural similarity index. β is a balance factor used to adjust the relative importance of the two losses. In a preferred embodiment of the present invention, β is set to 0.1, which can achieve a good balance between reconstruction accuracy and structure preservation.

[0158] The LSTM module is trained by the error back propagation algorithm. Preferably, the Adam optimizer is used, the initial learning rate is set to 0.0001, and it decays to 0.1 times the original rate every 50 epochs. In order to prevent overfitting, this method also introduces a dropout mechanism, and the dropout rate is set to 0.5.

[0159] It is worth noting that the training process of the LSTM module can be end-to-end and jointly optimized with the previous stabilization enhancement step. This joint training strategy can make the entire system work better together and further improve the final video quality.

[0160] Through the above steps, the method of the present invention achieves comprehensive and in-depth panoramic video stabilization enhancement. From pixel-by-pixel motion prediction compensation, to global spatiotemporal consistency optimization, to LSTM-based long-term dependency modeling, each step has been carefully designed and optimized. This multi-level, multi-scale processing strategy can effectively cope with various complex camera motions and scene changes, ensuring the high quality and stability of the output video.

[0161] In general, the panoramic video stabilization enhancement method based on spatiotemporal consistency provided by the present invention provides a powerful and flexible solution for the field of panoramic video processing by innovatively combining advanced technologies such as deep learning, optical flow estimation, and spatiotemporal consistency optimization. This method can not only effectively improve the visual quality of the video, but also has good scalability and adaptability, laying a solid foundation for future research and application.

[0162] The above description is only a preferred specific implementation manner of the present invention; however, the protection scope of the present invention is not limited thereto; any person familiar with the art who, within the scope disclosed by the present invention, makes equivalent replacements or changes based on the scheme and improved concepts of the present invention shall be covered within the protection scope of the present invention.

Claims

1. A panoramic video stabilization enhancement method based on spatiotemporal consistency, characterized in that: include: The acquisition steps include: Get panoramic video frame sequence; Obtaining the spherical projection coordinates of each frame in the panoramic video frame sequence; Processing steps include: Based on the spherical projection coordinates, a deep learning model is constructed, wherein the deep learning model includes a feature extraction encoder and a motion prediction compensation decoder; Using the feature extraction encoder, extracting features from the panoramic video frame sequence; Based on the feature extraction result, using the motion prediction and compensation decoder to perform motion prediction and compensation; According to the motion prediction and compensation results, a spatiotemporal filter is constructed to smooth the motion vector; Based on the smoothed motion vector, a multi-frame motion analysis is performed using a long temporal network; Designing panoramic video optical flow motion estimation based on the multi-frame motion analysis results; Output steps include: Based on the optical flow motion estimation result, performing stable enhancement processing on the panoramic video frame sequence; Outputs a stable and enhanced panoramic video.

2. The method according to claim 1, characterized in that The step of obtaining a panoramic video frame sequence specifically includes: Use multiple cameras to capture the same scene at the same time to obtain a 360-degree panoramic image; The panoramic image is converted into a spherical projection coordinate system.

3. The method according to claim 1, characterized in that: The steps of constructing the deep learning model specifically include: Design a feature extraction encoder for feature extraction in spherical space; Design a motion prediction and compensation decoder for motion prediction and compensation in spherical space; A bidirectional mapping is established between the feature extraction encoder and the motion prediction compensation decoder.

4. The method according to claim 1, characterized in that The step of constructing the space-time filter specifically includes: Design multi-scale spatial pyramid structures; At each scale of the spatial pyramid structure, a spatiotemporal filter is applied for motion vector smoothing.

5. The method according to claim 1, characterized in that The step of using a long temporal network to perform multi-frame motion analysis specifically includes: Use LSTM encoder and decoder to build long time series network; Taking the residual motion of the previous frame as the input of the long temporal network; The long temporal network is used to generate a motion prediction result of the current frame.

6. The method according to claim 1, characterized in that The steps of designing panoramic video optical flow motion estimation specifically include: In the spherical projection coordinate system, the optical flow motion vector between adjacent frames is calculated; The motion field in the panoramic video is estimated using the optical flow motion vector.

7. The method according to claim 1, characterized in that The step of performing stable enhancement processing on the panoramic video frame sequence specifically includes: Predicting and compensating motion residuals pixel by pixel in spherical space using the deep learning model; reconstructing an enhanced panoramic video frame in a spherical space based on the motion residual; The reconstructed panoramic video frame is projected back to a planar coordinate system.

8. The method according to claim 7, characterized in that In the step of pixel-by-pixel prediction and compensation of motion residual: The encoder of the deep learning model performs feature extraction pixel by pixel; The decoder of the deep learning model performs residual motion prediction and compensation pixel by pixel.

9. The method according to claim 1, characterized in that: Also includes: Optimizing the spatiotemporal consistency of the stabilized and enhanced panoramic video; The optimization results are further adjusted using the LSTM module.

10. The method according to claim 9, characterized in that In the step of adjusting using the LSTM module: The loss function that considers the cross-correlation between pixels is used for optimization; The LSTM module is trained by an error back propagation algorithm.

Images