Unmanned aerial vehicle aerial photography VR video jitter correction method based on Gaussian mixture model

Abstract

The invention provides an unmanned aerial vehicle aerial photography VR video jitter correction method based on a Gaussian mixture model, and relates to the technical field of video processing. According to the embodiment, firstly, background edge information in a video sequence is extracted through edge detection, and a complex area and a simple area are distinguished; establishing a GMM (Gaussian Mixture Model) based on time-space domain self-adaption in a complex area near a background edge, adaptively fitting a change rule of a pixel value by utilizing a plurality of Gaussian distributions, and adopting a time average model with efficient calculation in a simple area to reduce the overall complexity; then, dynamically estimating the direction and amplitude of global jitter caused by wind power or mechanical vibration by analyzing a displacement vector of a background edge feature between continuous frames and combining with a probability density difference output by a Gaussian mixture model; and finally, performing reverse motion compensation on the video frame based on the jitter vector, optimizing compensation parameters through background edge consistency constraint, and eliminating false jitter artifacts, thereby improving the stability and immersion of the VR video while ensuring the real-time performance.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a method for correcting jitter in drone aerial VR videos based on a Gaussian mixture model. Background Technology

[0002] With the advancement of technology and the development of the times, drones, with their convenience, flexibility and low cost, are increasingly being used in video shooting scenarios. Drones can capture low-altitude footage that is difficult to capture with traditional equipment; the flying perspective in the sky has great visual tension and richness of content; however, traditional planar shooting methods cannot effectively preserve the sense of space in the sky, and their shooting perspective is limited to a specific range and cannot fully record video content from different directions in the sky.

[0003] To address this, existing technologies adapt VR photography modules to drones to shoot VR videos. VR videos refer to professional VR photography functions that realistically record the on-site environment, and then use computers for post-processing to create videos that can achieve three-dimensional spatial display. Drone aerial VR videos can effectively preserve video content from different directions, allowing viewers to immerse themselves in the experience and soar through the sky like birds, free from the limitations of physical space.

[0004] However, the powerful immersive experience of VR videos also makes viewers easily susceptible to environmental interference. VR videos require smooth transitions, stable playback, continuous scene changes, and clear content. However, due to the large and fast movements of drones, it is difficult to avoid shaking during flight. Directly output drone aerial VR videos can cause sensory discomfort for viewers, and existing drone post-processing software has not addressed this issue.

[0005] Therefore, we need to provide a method for jitter correction of drone aerial VR videos based on Gaussian mixture models to solve the technical problem that existing technologies do not correct jitter in drone aerial VR videos, thus causing sensory discomfort to viewers. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for correcting jitter in drone aerial VR videos based on Gaussian mixture models. This method aims to correct jitter in drone aerial VR videos, thereby alleviating the technical problem of sensory discomfort caused to viewers when watching VR videos.

[0007] To achieve the above objectives, this application proposes a method for jitter correction in UAV aerial VR video based on a Gaussian mixture model, which corrects jitter in UAV aerial VR video through the following steps: Step 1: Perform specific preprocessing on the VR video to divide the image into complex pixel regions and simple pixel regions; this includes frame rate synchronization processing, spherical mapping distortion correction, background edge detection, and image region division. Step 2: Perform adaptive blending background modeling for different image regions to obtain the corresponding pixel classification models; among them, establish a spatiotemporal adaptive GMM model for complex pixel regions and a time averaging model for simple pixel regions. Step 3: Classify pixels using the pixel classification model corresponding to each area of ​​the image to obtain a probability map of each pixel belonging to the background, and filter out reliable background areas using the probability map; Step 4: Project the reliable background area onto the spherical coordinates of the VR shot, extract spherical feature points in the reliable background area, and obtain matching point pairs by matching feature points in two consecutive frames; Step 5: Extract motion vectors based on matching point pairs and remove outlier vectors to obtain a set of motion vectors; fit the global motion model using the set of motion vectors to obtain the actual global motion vectors; Step 6: Calculate the theoretical global motion vector based on the UAV control data, and perform vector cancellation calculation by combining the actual global motion vector to obtain the global jitter vector; Step 7: Perform inverse motion compensation on the VR video to obtain the coordinates of each pixel in the 2D image of the VR video after inverse motion compensation; wherein, the global jitter vectors of each matching point pair are combined to form the corresponding global jitter transformation matrix, and the inverse matrix corresponding to the global jitter transformation matrix is ​​calculated, and the current frame is inversely transformed by the inverse matrix in spherical coordinates.

[0008] As a further solution, in step 1, the specific preprocessing includes: Frame rate unification and synchronization involves acquiring the display frame rate of the playback device and the video frame rate of the VR video and comparing them. If the video frame rate is lower than the display frame rate, the VR video is upscaled using a motion interpolation algorithm until the frame rates are unified. Spherical mapping and distortion correction maps each pixel of the VR video to spherical coordinates, establishes a distortion model to describe the distortion of each pixel after mapping, and performs distortion correction on each pixel based on the pixel distortion. Background edge detection and image region segmentation: Perform edge detection on VR videos and divide the image into pixel complex regions and pixel simple regions based on edge density and motion complexity.

[0009] As a further solution, in step 2, the spatiotemporal adaptive GMM model includes: The spatial adaptive mechanism independently models the color value of each pixel in the video sequence, and updates the parameters of each pixel independently of other pixels, thereby adapting to different lighting changes or background motion in complex areas of pixels. The time-adaptive mechanism matches the newly arrived pixel with all Gaussian distributions in the GMM of the current pixel. If a match is found, the parameters of the matched Gaussian distribution are updated according to the learning rate. If no match is found, a new Gaussian distribution is created or the distribution with the smallest weight is replaced.

[0010] As a further solution, the spatiotemporal adaptive GMM model uses a low learning rate near the background edge and in the video seam area, and before updating the GMM parameters, it first performs a reverse offset on the pixels using preliminary global motion estimates; wherein, the preliminary global motion estimates are derived from UAV motion sensing data.

[0011] As a further solution, the spatiotemporal adaptive GMM model performs parameter estimation using an online variant of the EM algorithm; wherein the parameter estimation steps include: In step E, calculate the posterior probability that the current pixel value belongs to each Gaussian component to obtain the responsibility value γ; In the M-step, based on the responsibility value γ calculated in the E-step, the weights, mean, and covariance matrix parameters of each Gaussian component are updated.

[0012] As a further solution, in step 2, the time-averaging model maintains a fixed-length time window by storing the pixel values ​​of the most recent N frames, and calculates the mean and standard deviation within the time window; wherein, if the judgment formula is true, it is determined to be foreground; otherwise, it is background; the judgment formula is: ; ; In the formula, Represents the frame coordinates at time t. The pixel value corresponding to the pixel. m The mean, For the effective standard deviation, For the minimum standard deviation, s This represents the actual standard deviation.

[0013] As a further solution, in step 5, the motion vector is extracted by fitting a global motion model; wherein, the three-dimensional coordinate components (ΔX, ΔY, ΔZ) and Euler angle components (Δθ, Δφ, Δψ) are extracted by a 6-DOF affine transformation or a spherical rotation model; wherein ΔX represents the X-axis coordinate component, ΔY represents the Y-axis coordinate component, ΔZ represents the Z-axis coordinate component, Δθ represents the yaw angle component, Δφ represents the pitch angle component, and Δψ represents the roll angle component.

[0014] As a further solution, in step 7, reverse motion compensation is performed through the following steps: The motion vector is represented as a global jitter transformation matrix using a homogeneous transformation matrix. ; Based on the global jitter transformation matrix Find the corresponding inverse matrix ; Convert the spherical coordinates of each pixel to 3D Cartesian coordinates, and then convert the 3D Cartesian coordinates to homogeneous coordinates. Where r is the unit radius of the sphere; Homogeneous coordinates of pixels and inverse matrix Multiplying these results in the 3D Cartesian coordinates of the pixel after inverse motion compensation. The 3D Cartesian coordinates of the transformed pixels are mapped back to the 2D image of the VR video to complete the inverse motion compensation.

[0015] As a further solution, the global jitter transformation matrix ; In the formula, R is the global rotation matrix.

[0016] As a further solution, background edge consistency constraint optimization is also performed on the VR video after inverse motion compensation; in this process, the edge overlap between the current frame after inverse motion compensation and the original frame in the reliable background area is calculated, and fine-tuning compensation is performed when the edge overlap is lower than a threshold.

[0017] Compared with related technologies, the UAV aerial VR video jitter correction method based on Gaussian mixture model provided by this invention has the following advantages: This embodiment first extracts background edge information from the video sequence through edge detection to distinguish complex regions (such as dynamic leaves and water ripples) from simple regions. In complex regions near the background edges, a Gaussian mixture model (based on spatiotemporal adaptive GMM modeling improvement) is established, and multiple Gaussian distributions are used to adaptively fit the variation law of pixel values ​​(such as light fluctuations and small movements). In simple regions, a computationally efficient time averaging model is used to reduce the overall complexity. Subsequently, by analyzing the displacement vectors of background edge features between consecutive frames (such as the motion vectors of keypoint pairs) and combining the probability density differences output by the Gaussian mixture model, the global jitter direction and amplitude caused by wind or mechanical vibration are dynamically estimated. Finally, inverse motion compensation is performed on the video frames based on the jitter vectors, and the compensation parameters are optimized by background edge consistency constraints to eliminate false jitter artifacts, thereby improving the stability and immersion of VR videos while ensuring real-time performance. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0020] Figure 1 This invention provides a schematic diagram of the steps involved in a method for correcting motion blur in VR aerial videos taken by drones based on a Gaussian mixture model.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0023] Example 1 Please see Figure 1 This application provides a method for correcting camera shake in drone aerial VR videos based on a Gaussian mixture model, which corrects shake in drone aerial VR videos through the following steps: Step 1: Perform specific preprocessing on the VR video to divide the image into complex pixel regions and simple pixel regions; this includes frame rate synchronization processing, spherical mapping distortion correction, background edge detection, and image region division. Step 2: Perform adaptive blending background modeling for different image regions to obtain the corresponding pixel classification models; among them, establish a spatiotemporal adaptive GMM model for complex pixel regions and a time averaging model for simple pixel regions. Step 3: Classify pixels using the pixel classification model corresponding to each area of ​​the image to obtain a probability map of each pixel belonging to the background, and filter out reliable background areas using the probability map; Step 4: Project the reliable background area onto the spherical coordinates of the VR shot, extract spherical feature points in the reliable background area, and obtain matching point pairs by matching feature points in two consecutive frames; Step 5: Extract motion vectors based on matching point pairs and remove outlier vectors to obtain a set of motion vectors; fit the global motion model using the set of motion vectors to obtain the actual global motion vectors; Step 6: Calculate the theoretical global motion vector based on the UAV control data, and perform vector cancellation calculation by combining the actual global motion vector to obtain the global jitter vector; Step 7: Perform inverse motion compensation on the VR video to obtain the coordinates of each pixel in the 2D image of the VR video after inverse motion compensation; wherein, the global jitter vectors of each matching point pair are combined to form the corresponding global jitter transformation matrix, and the inverse matrix corresponding to the global jitter transformation matrix is ​​calculated, and the current frame is inversely transformed by the inverse matrix in spherical coordinates.

[0024] It should be noted that: This embodiment first extracts background edge information from the video sequence through edge detection, distinguishing between complex regions (such as dynamic leaves and water ripples) and simple regions; in complex regions near the background edges, a Gaussian mixture model (based on spatiotemporal adaptive GMM modeling improvement) is established, using multiple Gaussian distributions to adaptively fit the variation law of pixel values ​​(such as light fluctuations and small movements), while in simple regions, a computationally efficient time averaging model is used to reduce overall complexity; subsequently, by analyzing the displacement vectors of background edge features between consecutive frames (such as motion vectors of keypoint pairs), combined with the probability density differences output by the Gaussian mixture model, the global jitter direction and amplitude caused by wind or mechanical vibration are dynamically estimated; finally, inverse motion compensation is performed on the video frames based on the jitter vector, and the compensation parameters are optimized through background edge consistency constraints to eliminate false jitter artifacts, thereby improving the stability and immersion of VR videos while ensuring real-time performance.

[0025] Specifically, in step 1, specific preprocessing and background edge information extraction are required for the VR video: First, read the drone aerial VR video sequence (usually in 360° or 180° rectangular projection format); perform grayscale processing on each frame to reduce computation, and apply Gaussian filtering or bilateral filtering for noise reduction while preserving edge information; Since VR videos typically require a frame rate of 60 FPS or higher (such as 90 FPS) to prevent motion sickness and provide a smooth experience, a low frame rate can cause screen jitter or ghosting, which is significantly amplified in immersive VR viewing. Therefore, we perform frame rate unification and synchronization by obtaining the display frame rate of the playback device and the video frame rate of the VR video and comparing them. If the video frame rate is lower than the display frame rate, the VR video is upscaled using a motion interpolation algorithm until the frame rates are unified. Secondly, VR videos captured by drones are typically in equirectangular projection. All processing should be performed in spherical coordinates or take into account their distortion characteristics to prevent polar stretching deformation; therefore, distortion correction is required for video frames to ensure the accuracy of subsequent edge detection and feature point extraction. Finally, Canny or Sobel edge detection is performed on the rectangular projection map. Based on edge density and motion complexity, the image is divided into pixel-complex regions (such as leaves and water surfaces) and pixel-simple regions (such as the sky and static buildings).

[0026] The improved Gaussian mixture model (GMM) is applied to complex regions: In this embodiment, a mixture model consisting of 3-5 Gaussian distributions is established for each pixel in a complex region. Each Gaussian distribution represents a possible pixel value (such as color or brightness) and has a weight ω, a mean μ, and a standard deviation σ.

[0027] This embodiment introduces a spatiotemporal adaptive learning rate mechanism; wherein... The spatial adaptive mechanism independently models the color value of each pixel in the video sequence, and updates the parameters of each pixel independently of other pixels, thereby adapting to different lighting changes or background motion in complex areas of pixels. The time-adaptive mechanism matches the newly arrived pixel with all Gaussian distributions in the GMM of the current pixel. If a match is found, the parameters of the matched Gaussian distribution are updated according to the learning rate. If no match is found, a new Gaussian distribution is created or the distribution with the smallest weight is replaced.

[0028] Specifically, for a new frame, the parameters (μ, σ) and all weights of the matching distribution are updated based on the matching status of the pixel value with the current GMM (e.g., whether it falls within 2.5σ of a certain distribution) (weight decay followed by renormalization). If there is no match, the distribution with the smallest weight is replaced with the new distribution. For pixels near the background edge, a lower learning rate (e.g., 0.001) is used to maintain model stability; for non-edge regions, a higher learning rate (e.g., 0.005) can be used to adapt to changes.

[0029] Furthermore, the spatiotemporal adaptive GMM model uses a low learning rate near the background edges and video seam areas, and before updating the GMM parameters, it first uses preliminary global motion estimates to reverse the pixels; the preliminary global motion estimates are derived from UAV motion sensing data, thus initially filtering out the effects of jitter based on sensor data.

[0030] Specifically, the spatiotemporally adaptive GMM uses an online variant of the EM (Expectation-Maximization) algorithm for parameter estimation; the parameter estimation steps include: In step E, calculate the posterior probability that the current pixel value belongs to each Gaussian component to obtain the responsibility value γ; In the M-step, based on the responsibility value γ calculated in the E-step, the weights, mean, and covariance matrix parameters of each Gaussian component are updated.

[0031] The EM algorithm uses the log-likelihood function as the loss function and optimizes the parameters by maximizing this function through iterative E-steps and M-steps.

[0032] Therefore, the spatiotemporally adaptive GMM in this embodiment can effectively handle complex scenes such as changes in lighting, slowly moving objects (such as swaying leaves), and slight camera shake. The model parameters are automatically adjusted, reducing the need for manual parameter tuning, and no offline training is required; it can be updated in real time.

[0033] The time-averaging model is applied to simple regions: This embodiment employs a lightweight, adaptive time averaging model specifically designed for handling simple regions (such as the sky or flat roads) to reduce overall computational complexity. The time averaging model maintains a fixed-length time window by storing the pixel values ​​of the most recent N frames and calculates the mean and standard deviation within that window. If the given formula is true, the element is identified as foreground; otherwise, it is identified as background. The formula is: ; ; In the formula, Represents the frame coordinates at time t. The pixel value corresponding to the pixel. m The mean, For the effective standard deviation, For the minimum standard deviation, s This represents the actual standard deviation.

[0034] Specifically, the parameter settings are shown in Table 1: Table 1 Parameter Reference Settings If a simple region suddenly becomes complex (e.g., edge density increases), the region can be dynamically switched to GMM processing. The foreground mask output by the time-averaged model is fused with the output of the GMM as the final background subtraction result. This design ensures efficient processing in simple regions while maintaining the real-time and low-latency characteristics required for VR videos.

[0035] Furthermore, in step 5, the motion vector is extracted by fitting a global motion model; wherein, the three-dimensional coordinate components (ΔX, ΔY, ΔZ) and Euler angle components (Δθ, Δφ, Δψ) are extracted by a 6-DOF affine transformation or a spherical rotation model; wherein ΔX represents the X-axis coordinate component, ΔY represents the Y-axis coordinate component, ΔZ represents the Z-axis coordinate component, Δθ represents the yaw angle component, Δφ represents the pitch angle component, and Δψ represents the roll angle component.

[0036] When conditions permit, data from airborne IMUs (inertial measurement units) (such as gyroscopes and accelerometers) can be combined for fusion estimation, which can improve the accuracy and robustness of the estimation. IMUs can directly provide angular velocity (which can be integrated to obtain Δθ, Δφ, Δψ) and acceleration (which can be integrated to obtain ΔX, ΔY, ΔZ) information, which complements the visual estimation results.

[0037] Furthermore, converting the global jitter vector of the matched point pairs into a transformation matrix and performing an inverse transformation is a crucial step in stabilizing VR video; the principles and operation methods of this process will be explained in detail below: The general form of a homogeneous transformation matrix is ​​as follows: ; Where R is a 3x3 matrix representing rotation and scaling, and t is a 3x1 vector representing translation, i.e.: The 1 in the bottom right corner is used to maintain the homogeneity of coordinates. Therefore, reverse motion compensation is performed through the following steps: The motion vector is represented as a global jitter transformation matrix using a homogeneous transformation matrix. ; To correct jitter, we need to apply a transformation to the current frame that is the opposite of the jitter. This "opposite" transformation is mathematically achieved through an inverse matrix. Therefore, according to the global jitter transformation matrix Find the corresponding inverse matrix ; Convert the spherical coordinates of each pixel to 3D Cartesian coordinates, and then convert the 3D Cartesian coordinates to homogeneous coordinates. Where r is the unit radius of the sphere; Homogeneous coordinates of pixels and inverse matrix Multiply to obtain the transformed new 3D points. ,Right now: The obtained 3D Cartesian coordinates of the pixels after inverse motion compensation are equivalent to rotating and translating the entire sphere in reverse in 3D space, which perfectly cancels out the shaking during shooting. The 3D Cartesian coordinates of the transformed pixels are mapped back to the 2D image of the VR video to complete the inverse motion compensation.

[0038] Wherein, the global jitter transformation matrix ; In the formula, R is the global rotation matrix.

[0039] In addition, this embodiment also optimizes the background edge consistency constraint of the VR video after inverse motion compensation; wherein, by calculating the edge overlap between the current frame after inverse motion compensation and the original frame in the reliable background area, and performing fine-tuning compensation when the edge overlap is lower than the threshold, this step helps to eliminate false jitter estimation caused by dynamic object interference or model error, and ensures the physical consistency of compensation.

[0040] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for correcting motion blur in UAV aerial VR videos based on Gaussian mixture models, characterized in that, The following steps are used to correct motion in drone aerial VR videos: Step 1: Perform specific preprocessing on the VR video to divide the image into complex pixel regions and simple pixel regions; this includes frame rate synchronization processing, spherical mapping distortion correction, background edge detection, and image region division. Step 2: Perform adaptive blending background modeling for different image regions to obtain the corresponding pixel classification models; among them, establish a spatiotemporal adaptive GMM model for complex pixel regions and a time averaging model for simple pixel regions. Step 3: Classify pixels using the pixel classification model corresponding to each area of ​​the image to obtain a probability map of each pixel belonging to the background, and filter out reliable background areas using the probability map; Step 4: Project the reliable background area onto the spherical coordinates of the VR shot, extract spherical feature points in the reliable background area, and obtain matching point pairs by matching feature points in two consecutive frames; Step 5: Extract motion vectors based on matching point pairs and remove outlier vectors to obtain a set of motion vectors; fit the global motion model using the set of motion vectors to obtain the actual global motion vectors; Step 6: Calculate the theoretical global motion vector based on the UAV control data, and perform vector cancellation calculation by combining the actual global motion vector to obtain the global jitter vector; Step 7: Perform inverse motion compensation on the VR video to obtain the coordinates of each pixel in the 2D image of the VR video after inverse motion compensation; wherein, the global jitter vectors of each matching point pair are combined to form the corresponding global jitter transformation matrix, and the inverse matrix corresponding to the global jitter transformation matrix is ​​calculated, and the current frame is inversely transformed by the inverse matrix in spherical coordinates.

2. The method for correcting camera shake in UAV aerial VR video based on Gaussian mixture model according to claim 1, characterized in that, In step 1, the specific preprocessing includes: Frame rate unification and synchronization involves acquiring the display frame rate of the playback device and the video frame rate of the VR video and comparing them. If the video frame rate is lower than the display frame rate, the VR video is upscaled using a motion interpolation algorithm until the frame rates are unified. Spherical mapping and distortion correction maps each pixel of the VR video to spherical coordinates, establishes a distortion model to describe the distortion of each pixel after mapping, and performs distortion correction on each pixel based on the pixel distortion. Background edge detection and image region segmentation: Perform edge detection on VR videos and divide the image into pixel complex regions and pixel simple regions based on edge density and motion complexity.

3. The method for correcting motion blur in UAV aerial VR video based on Gaussian mixture model according to claim 1, characterized in that, In step 2, the spatiotemporal adaptive GMM model includes: The spatial adaptive mechanism independently models the color value of each pixel in the video sequence, and updates the parameters of each pixel independently of other pixels, thereby adapting to different lighting changes or background motion in complex areas of pixels. The time-adaptive mechanism matches the newly arrived pixel with all Gaussian distributions in the GMM of the current pixel. If a match is found, the parameters of the matched Gaussian distribution are updated according to the learning rate. If no match is found, a new Gaussian distribution is created or the distribution with the smallest weight is replaced.

4. The method for correcting camera shake in UAV aerial VR video based on Gaussian mixture model according to claim 3, characterized in that, The spatiotemporal adaptive GMM model uses a low learning rate near the background edge and in the video seam area, and before updating the GMM parameters, it first uses a preliminary global motion estimate to offset the pixels in the reverse direction; wherein, the preliminary global motion estimate is derived from UAV motion sensing data.

5. The method for correcting motion blur in UAV aerial VR video based on Gaussian mixture model according to claim 3, characterized in that, The spatiotemporal adaptive GMM model is parameter estimated using an online variant of the EM algorithm; the parameter estimation steps include: In step E, calculate the posterior probability that the current pixel value belongs to each Gaussian component to obtain the responsibility value γ; In the M-step, based on the responsibility value γ calculated in the E-step, the weights, mean, and covariance matrix parameters of each Gaussian component are updated.

6. The method for correcting camera shake in UAV aerial VR video based on Gaussian mixture model according to claim 1, characterized in that, In step 2, the time-averaging model maintains a fixed-length time window by storing the pixel values ​​of the most recent N frames, and calculates the mean and standard deviation within the time window; wherein, if the judgment formula is true, it is determined to be foreground; otherwise, it is background; the judgment formula is: ； ； In the formula, Represents the frame coordinates at time t. The pixel value corresponding to the pixel. μ The mean, For the effective standard deviation, For the minimum standard deviation, σ This represents the actual standard deviation.

7. The method for correcting motion blur in UAV aerial VR video based on Gaussian mixture model according to claim 1, characterized in that, In step 5, the motion vector is extracted by fitting a global motion model; wherein, the three-dimensional coordinate components (ΔX, ΔY, ΔZ) and Euler angle components (Δθ, Δφ, Δψ) are extracted by a 6-DOF affine transformation or a spherical rotation model; wherein ΔX represents the X-axis coordinate component, ΔY represents the Y-axis coordinate component, ΔZ represents the Z-axis coordinate component, Δθ represents the yaw angle component, Δφ represents the pitch angle component, and Δψ represents the roll angle component.

8. The method for correcting motion blur in UAV aerial VR video based on Gaussian mixture model according to claim 7, characterized in that, In step 7, reverse motion compensation is performed through the following steps: The motion vector is represented as a global jitter transformation matrix using a homogeneous transformation matrix. ; Based on the global jitter transformation matrix Find the corresponding inverse matrix ; Convert the spherical coordinates of each pixel to 3D Cartesian coordinates, and then convert the 3D Cartesian coordinates to homogeneous coordinates. Where r is the unit radius of the sphere; Homogeneous coordinates of pixels and inverse matrix Multiplying these results in the 3D Cartesian coordinates of the pixel after inverse motion compensation. The 3D Cartesian coordinates of the transformed pixels are mapped back to the 2D image of the VR video to complete the inverse motion compensation.

9. A method for correcting motion blur in UAV aerial VR video based on a Gaussian mixture model according to claim 8, characterized in that, The global jitter transformation matrix ; In the formula, R is the global rotation matrix.

10. A method for correcting motion blur in UAV aerial VR video based on a Gaussian mixture model according to claim 1, characterized in that, Furthermore, background edge consistency constraint optimization is performed on the VR video after inverse motion compensation; specifically, the edge overlap between the current frame after inverse motion compensation and the original frame in the reliable background area is calculated, and fine-tuning compensation is performed when the edge overlap is lower than a threshold.

Images