Methods, systems, devices and equipment for UAV-borne multimodal fusion online image stabilization

Abstract

This application relates to an unmanned aerial vehicle (UAV)-borne multimodal fusion online image stabilization method, system, apparatus, and device. The method includes: acquiring preprocessed IMU sequence samples and first-level stabilized frame samples; inputting the IMU sequence samples and first-level stabilized frame samples into a multimodal fusion online image stabilization network; the first-level stabilized frame samples are obtained by mechanically stabilizing video frames; the multimodal fusion online image stabilization network includes an IMU feature extraction branch, an image feature extraction branch, and a multimodal feature fusion main branch; the multimodal feature fusion main branch includes a feature-level fusion module, an offset estimation module, and a reconstruction output module; training the multimodal fusion online image stabilization network based on a pre-constructed loss function, the preprocessed IMU sequence samples, and the first-level stabilized frame samples to obtain a trained multimodal fusion online image stabilization network; and performing online image stabilization using the trained multimodal fusion online image stabilization network. This method can significantly improve the visual effect and stability of stabilized frames.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an unmanned aerial vehicle (UAV)-borne multimodal fusion online image stabilization method, system, apparatus, and equipment. Background Technology

[0002] With the rapid development of drone technology and its widespread deployment in various application scenarios, such as environmental monitoring, agricultural remote sensing, urban planning, and film and television shooting, more stringent requirements have been placed on high-quality video acquisition. However, due to the unavoidable influence of various factors during drone flight, such as air turbulence, mechanical vibration, and control errors, the acquired video suffers from significant jitter. This jitter not only affects the visual quality of the video but may also negatively impact subsequent video analysis and processing tasks. Therefore, how to effectively suppress jitter in drone video and improve video quality has become one of the key problems that urgently need to be solved in the field of computer vision and image processing.

[0003] Existing image stabilization technologies are mainly divided into three types based on their implementation methods and principles: mechanical, optical, and electronic. Mechanical stabilization relies on servo systems and sensors to build a stable platform, achieving reverse motion compensation for the imaging device, and is suitable for scenarios with high-amplitude camera shake. However, its bulky structure, high power consumption, and cost limit its widespread application. Nevertheless, mechanical stabilization remains the mainstream for airborne platforms and can be integrated with other stabilization technologies to further improve performance. For example, a scheme combining gyroscopes with an improved ORB algorithm enhances system speed and accuracy. Optical stabilization achieves image stabilization by compensating for motion using optical elements. Its advantages include ease of operation, light weight, and small size, making it suitable for environments where the camera device does not experience significant vibration. However, its cost is high and its accuracy is limited. Electronic stabilization uses image processing technology to calculate inter-frame deviations and perform reverse compensation. It is widely used due to its ease of operation, small size, low power consumption, and strong real-time performance. Currently, research focuses on two main modules: motion estimation and motion filtering. The key to the motion estimation module is to accurately and quickly provide camera motion information. Common algorithms include block matching, optical flow, and feature matching, with the latter being widely used due to its scene independence and fewer constraints. In terms of filtering techniques, mean, Gaussian, and Kalman filtering are commonly used methods.

[0004] In the field of electronic image stabilization (EIS), deep learning methods have become a focal point of cutting-edge research. Compared to traditional EIS techniques, stabilization methods based on convolutional neural networks (CNNs) demonstrate superior robustness. Traditional techniques often face problems of insufficient feature extraction and inaccurate feature point matching when dealing with low-quality videos subjected to low light, blur, and noise interference. In contrast, CNNs, through their deep convolutional structures, can effectively extract high-dimensional abstract features, thereby avoiding dependence on explicit feature extraction and matching. In most cases, CNN-based stabilization models outperform traditional methods.

[0005] Nevertheless, deep learning-based electronic image stabilization methods alone still face performance bottlenecks when dealing with significant shaking. To comprehensively utilize the advantages of mechanical and electronic image stabilization, researchers have recently attempted to combine the two to achieve more comprehensive and efficient image stabilization results. Furthermore, the inertial measurement unit (IMU), as a crucial sensor for acquiring the motion state of UAVs, provides acceleration and angular velocity data that can strongly support image stabilization algorithms. However, how to effectively fuse image features with IMU sequence features remains a problem that urgently needs to be solved. Summary of the Invention

[0006] Therefore, it is necessary to provide an unmanned aerial vehicle (UAV) multimodal fusion online image stabilization method, system, device, and equipment to address the aforementioned technical problems.

[0007] A method for online image stabilization via multimodal fusion on an unmanned aerial vehicle (UAV), the method comprising:

[0008] Preprocessed IMU sequence samples and first-level stable frame samples are acquired and input into a multimodal fusion online image stabilization network. The first-level stable frame samples are obtained by mechanically stabilizing video frames. The multimodal fusion online image stabilization network includes an IMU feature extraction branch, an image feature extraction branch, and a multimodal feature fusion main branch. The multimodal feature fusion main branch includes a feature-level fusion module, an offset estimation module, and a reconstruction output module.

[0009] The IMU sequence samples and the first-level stable frame samples are respectively extracted through the IMU feature extraction branch and the image feature extraction branch, and the IMU sequence feature vector and the image feature vector are output to the feature-level fusion module.

[0010] The feature-level fusion module concatenates the IMU sequence feature vector and the image feature vector to output a joint feature vector. The offset estimation module maps the joint feature vector to an offset matrix and concatenates it with the first-level stable frame sample in the channel direction to obtain a multi-channel fused feature map. The reconstruction output module extracts features from the multi-channel fused feature map and outputs a second-level stable frame.

[0011] The multimodal fusion online image stabilization network is trained based on a pre-constructed loss function, pre-processed IMU sequence samples, and first-level stable frame samples to obtain a trained multimodal fusion online image stabilization network.

[0012] Online image stabilization is performed using a trained multimodal fusion online image stabilization network.

[0013] An unmanned aerial vehicle (UAV)-borne multimodal fusion online image stabilization system is disclosed, comprising a gimbal and a multimodal feature fusion subsystem; the gimbal is equipped with an UAV-borne camera and an IMU.

[0014] The UAV-borne camera is used to input the first-level stable frame obtained by gimbal mechanical image stabilization compensation into the multimodal feature fusion subsystem;

[0015] The IMU is used to input the preprocessed IMU sequence into the multimodal feature fusion subsystem;

[0016] The multimodal feature fusion subsystem acquires the preprocessed IMU sequence and the first-level stabilized frame after mechanical image stabilization compensation, and inputs the IMU sequence and the first-level stabilized frame into the trained multimodal fusion online image stabilization network for online image stabilization.

[0017] An unmanned aerial vehicle (UAV)-borne multimodal fusion online image stabilization device, the device comprising:

[0018] The sample acquisition module is used to acquire preprocessed IMU sequence samples and first-level stable frame samples, and input the IMU sequence samples and the first-level stable frame samples into the multimodal fusion online image stabilization network; the first-level stable frame samples are obtained by mechanically stabilizing the video frames; the multimodal fusion online image stabilization network includes an IMU feature extraction branch, an image feature extraction branch, and a multimodal feature fusion main branch; the multimodal feature fusion main branch includes a feature-level fusion module, an offset estimation module, and a reconstruction output module;

[0019] The feature extraction module is used to extract features from the IMU sequence samples and the first-level stable frame samples through the IMU feature extraction branch and the image feature extraction branch respectively, and output the IMU sequence feature vector and the image feature vector to the feature-level fusion module;

[0020] The result output module is used to concatenate the IMU sequence feature vector and the image feature vector through the feature-level fusion module to output a joint feature vector. The offset estimation module maps the joint feature vector to an offset matrix and then concatenates it with the first-level stable frame sample in the channel direction to obtain a multi-channel fused feature map. The reconstruction output module performs feature extraction on the multi-channel fused feature map and outputs a second-level stable frame.

[0021] The model training module is used to train the multimodal fusion online image stabilization network based on the pre-constructed loss function, pre-processed IMU sequence samples, and first-level stable frame samples, so as to obtain the trained multimodal fusion online image stabilization network.

[0022] The online image stabilization module is used to perform online image stabilization through a pre-trained multimodal fusion online image stabilization network.

[0023] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0024] Preprocessed IMU sequence samples and first-level stable frame samples are acquired and input into a multimodal fusion online image stabilization network. The first-level stable frame samples are obtained by mechanically stabilizing video frames. The multimodal fusion online image stabilization network includes an IMU feature extraction branch, an image feature extraction branch, and a multimodal feature fusion main branch. The multimodal feature fusion main branch includes a feature-level fusion module, an offset estimation module, and a reconstruction output module.

[0025] The IMU sequence samples and the first-level stable frame samples are respectively extracted through the IMU feature extraction branch and the image feature extraction branch, and the IMU sequence feature vector and the image feature vector are output to the feature-level fusion module.

[0026] The feature-level fusion module concatenates the IMU sequence feature vector and the image feature vector to output a joint feature vector. The offset estimation module maps the joint feature vector to an offset matrix and concatenates it with the first-level stable frame sample in the channel direction to obtain a multi-channel fused feature map. The reconstruction output module extracts features from the multi-channel fused feature map and outputs a second-level stable frame.

[0027] The multimodal fusion online image stabilization network is trained based on a pre-constructed loss function, pre-processed IMU sequence samples, and first-level stable frame samples to obtain a trained multimodal fusion online image stabilization network.

[0028] Online image stabilization is performed using a trained multimodal fusion online image stabilization network.

[0029] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0030] Preprocessed IMU sequence samples and first-level stable frame samples are acquired and input into a multimodal fusion online image stabilization network. The first-level stable frame samples are obtained by mechanically stabilizing video frames. The multimodal fusion online image stabilization network includes an IMU feature extraction branch, an image feature extraction branch, and a multimodal feature fusion main branch. The multimodal feature fusion main branch includes a feature-level fusion module, an offset estimation module, and a reconstruction output module.

[0031] The IMU sequence samples and the first-level stable frame samples are respectively extracted through the IMU feature extraction branch and the image feature extraction branch, and the IMU sequence feature vector and the image feature vector are output to the feature-level fusion module.

[0032] The feature-level fusion module concatenates the IMU sequence feature vector and the image feature vector to output a joint feature vector. The offset estimation module maps the joint feature vector to an offset matrix and concatenates it with the first-level stable frame sample in the channel direction to obtain a multi-channel fused feature map. The reconstruction output module extracts features from the multi-channel fused feature map and outputs a second-level stable frame.

[0033] The multimodal fusion online image stabilization network is trained based on a pre-constructed loss function, pre-processed IMU sequence samples, and first-level stable frame samples to obtain a trained multimodal fusion online image stabilization network.

[0034] Online image stabilization is performed using a trained multimodal fusion online image stabilization network.

[0035] The aforementioned UAV-borne multimodal fusion online image stabilization method, system, device, and equipment perform preliminary mechanical stabilization compensation on video frames to obtain a first-level stable frame. Multimodal data is obtained by preprocessing the IMU sequence output by the IMU unit and based on the preprocessed IMU sequence and the corresponding first-level stable frame. Training samples are constructed based on the multimodal data. The constructed multimodal fusion online image stabilization network performs feature extraction, feature-level fusion, offset estimation, and reconstruction on the input multimodal data, outputting a second-level stable frame. This achieves deep fusion of image features and motion sensor features, effectively improving the performance and robustness of UAV video stabilization. The multimodal fusion online image stabilization network is further trained using a loss function and training samples, and then used to perform online image stabilization on the multimodal data. Mechanical stabilization is used to achieve preliminary compensation for large-amplitude jitter, and then electronic stabilization is used to eliminate small-amplitude jitter, outputting a high-quality second-level stable frame. This invention significantly improves the stabilization effect and robustness of UAV-borne video frame processing, enhancing the visual effect and stability of the stabilized frame. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating an online image stabilization method for UAV-borne multimodal fusion in one embodiment.

[0037] Figure 2 This is a schematic diagram illustrating the online image stabilization principle of multimodal fusion in one embodiment;

[0038] Figure 3This is a schematic diagram of a multimodal fusion online image stabilization network structure in one embodiment;

[0039] Figure 4 This is a structural block diagram of an UAV-borne multimodal fusion online image stabilization system in one embodiment;

[0040] Figure 5 This is a structural block diagram of an UAV-borne multimodal fusion online image stabilization device in one embodiment;

[0041] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0043] In one embodiment, such as Figure 1 As shown, an online image stabilization method for UAV-borne multimodal fusion is provided, including the following steps:

[0044] Step 102: Obtain the preprocessed IMU sequence samples and first-level stabilized frame samples, and input the IMU sequence samples and first-level stabilized frame samples into the multimodal fusion online image stabilization network.

[0045] Level 1 stabilized frame samples are obtained through mechanical image stabilization compensation of video frames. During the video acquisition phase, the mechanical image stabilization module utilizes hardware devices such as a gimbal to significantly compensate for camera shake on the UAV, resulting in level 1 stabilized frames. This process primarily targets significant shake caused by the flight environment and mechanical vibrations, effectively reducing the adjustment required for subsequent processing and lowering the computational load. Simultaneously, the inertial measurement unit (IMU) acquires real-time acceleration and angular velocity data of the current frame and the previous N frames, providing accurate motion state information. This data undergoes preprocessing such as normalization and denoising to meet the input requirements of subsequent deep networks.

[0046] The multimodal fusion online image stabilization network includes an IMU feature extraction branch, an image feature extraction branch, and a multimodal feature fusion main branch; the multimodal feature fusion main branch includes a feature-level fusion module, an offset estimation module, and a reconstruction output module.

[0047] Step 104: Extract features from IMU sequence samples and first-level stable frame samples through IMU feature extraction branch and image feature extraction branch respectively, and output IMU sequence feature vector and image feature vector to feature-level fusion module.

[0048] like Figure 2The diagram illustrates the principle of multimodal fusion online image stabilization. In the IMU feature extraction branch, IMU sequence data undergoes feature extraction through a lightweight Transformer structure. Leveraging its multi-head attention mechanism, the Transformer effectively captures temporal dependencies and key motion features in the IMU data, generating high-dimensional motion feature representations. The IMU feature extraction branch demonstrates superior performance in processing dynamically changing sensor data, ensuring accurate extraction of motion features. In the image feature extraction branch, first-level stabilized frames serve as input, and a pre-trained CNN extracts high-level image features. This process captures spatial information and detailed features in the image, providing rich visual information support for subsequent image stabilization processes.

[0049] Step 106: The feature-level fusion module concatenates the IMU sequence feature vector and the image feature vector to output a joint feature vector. The offset estimation module maps the joint feature vector to an offset matrix and concatenates it with the first-level stable frame sample in the channel direction to obtain a multi-channel fused feature map. The reconstruction output module extracts features from the multi-channel fused feature map and outputs the second-level stable frame.

[0050] Image features from the image feature extraction branch and IMU motion features from the IMU feature extraction branch are fused at the feature level. The fusion method includes weighted combination of feature concatenation to fully utilize multi-source information and improve image stabilization. The fused feature vector is input to a fully connected layer to predict the offset information of the current frame. This offset reflects the specific parameters that need to be adjusted to achieve smooth transitions between frames and improve stability. After decoding, the predicted offset is skip-connected with a first-level stabilized frame to preserve the details of the original image and ensure high-quality image reconstruction. The skip-connected data is further processed by convolutional layers of equal size to generate a second-level stabilized video frame. This process ensures consistency in spatial resolution between the output and input frames while improving image stability and visual quality.

[0051] Step 108: Train the multimodal fusion online image stabilization network based on the pre-constructed loss function, pre-processed IMU sequence samples, and first-level stable frame samples to obtain the trained multimodal fusion online image stabilization network.

[0052] Step 110: Perform online image stabilization using the trained multimodal fusion online image stabilization network.

[0053] In the aforementioned UAV-borne multimodal fusion online image stabilization method, a first-level stable frame is obtained by performing preliminary mechanical stabilization compensation on video frames. The IMU sequence output by the IMU unit is preprocessed, and multimodal data is obtained based on the preprocessed IMU sequence and the corresponding first-level stable frame. Training samples are constructed based on the multimodal data. The constructed multimodal fusion online image stabilization network performs feature extraction, feature-level fusion, offset estimation, and reconstruction on the input multimodal data, outputting a second-level stable frame. This achieves deep fusion of image features and motion sensor features, effectively improving the performance and robustness of UAV video stabilization. The multimodal fusion online image stabilization network is further trained using a loss function and training samples, and then used to perform online image stabilization on the multimodal data. Mechanical stabilization is used to achieve preliminary compensation for large-amplitude jitter, and then electronic stabilization is used to eliminate small-amplitude jitter, outputting a high-quality second-level stable frame. This embodiment of the invention significantly improves the image stabilization effect and robustness of UAV-borne video frame processing, enhancing the visual effect and stability of the stabilized frame.

[0054] In one embodiment, the IMU feature extraction branch includes an embedding layer, a multi-head attention mechanism layer, a normalization layer, a fully connected layer, a ReLU activation function, and an IMU feature output layer; the output of the embedding layer and the output of the multi-head attention mechanism layer are fused in a skip connection manner; the output of the normalization layer and the output of the fully connected layer are fused in a skip connection manner.

[0055] In one embodiment, feature extraction of IMU sequence samples via an IMU feature extraction branch to output an IMU sequence feature vector includes: obtaining preprocessed IMU sequence samples through an embedding layer and encoding the IMU sequence into a high-dimensional vector; extracting key features from the high-dimensional vector through a multi-head attention mechanism layer; fusing the high-dimensional vector and key features to obtain a first fusion result; performing layer normalization on the first fusion result through a normalization layer and outputting it to a fully connected layer; performing a nonlinear transformation on the output of the normalization layer through a fully connected layer and a ReLU activation function to output frame offset features; fusing the output of the normalization layer and the frame offset features to obtain a second fusion result; and normalizing and max-pooling the second fusion result through an IMU feature output layer to obtain the IMU sequence feature vector.

[0056] In this embodiment, as Figure 3The diagram shows a multimodal fusion online image stabilization network structure. The IMU feature extraction branch adopts a Transformer architecture, specifically designed for processing IMU sequence data. The input IMU sequence contains motion state information such as acceleration and angular velocity of the current frame and the previous N frames. The IMU sequence is encoded through an embedding layer, mapping multidimensional motion data to a high-dimensional feature space, enhancing feature representation capabilities. Subsequently, a multi-head attention mechanism layer plays a crucial role in capturing temporal dependencies and dynamic motion patterns, effectively extracting key features from the IMU data. A skip connection is introduced between the output of the multi-head attention mechanism layer and the input of the embedding layer, mitigating the gradient vanishing problem in deep networks. After processing by a layer normalization (LN) layer, the stability of feature propagation and the efficiency of network training are ensured. Next, a nonlinear transformation using a fully connected layer and the ReLU activation function further enriches the feature representation. The aforementioned features are then fused with the output of the fully connected layer again through a skip connection, and finally, a 512-dimensional IMU sequence feature vector is generated through another normalization and max-pooling layer. The IMU feature extraction branch fully utilizes the advantages of the Transformer structure in sequence data processing. Considering the small amount of IMU sequence data, the Transformer network is designed to be lightweight to reduce computation and improve its practicality under airborne conditions.

[0057] In one embodiment, the image feature extraction branch includes a first feature extraction module, a second feature extraction module, a third feature extraction module, and an image feature output layer. The first feature extraction module convolves the first-level stabilized frame after mechanical image stabilization compensation to extract the spatial features of the first-level stabilized frame and output a first feature map. The second feature extraction module convolves the spatial feature map to extract the deep features of the spatial feature map and output a second feature map. The third feature extraction module convolves the second feature map to extract the image features of the second feature map and output a third feature map. The output layer performs max pooling on the third feature map to output an image feature vector.

[0058] In this embodiment, as Figure 3The diagram shows a multimodal fusion online image stabilization network. The image feature extraction branch uses a convolutional neural network (CNN) architecture specifically for extracting image features. The input is a first-level stabilized frame with a resolution of W×H×C, obtained after initial mechanical stabilization compensation. First, a 7×7×64 convolutional layer is applied, combined with batch normalization (BN) and ReLU activation functions, to extract low-level spatial features. Then, a 3×3×128 convolutional layer is used, again applying BN and ReLU, progressively increasing the depth of the feature map to capture more complex image patterns. Next, a 3×3×256 convolutional layer, combined with BN and ReLU, is used to further extract high-level image features. Finally, a max-pooling layer reduces the spatial resolution of the feature map while retaining the most salient features, generating a 256-dimensional image feature vector. This branch design, through layer-by-layer convolution and non-linear activation, effectively captures details and high-level semantic information in the image, providing rich visual feature support for the image stabilization process.

[0059] In one embodiment, the reconstructed output module includes a first convolutional submodule and a second convolutional submodule; the first convolutional submodule includes a convolutional layer, BN, and ReLU activation functions; the second convolutional submodule includes a convolutional layer, BN, and tanh activation functions.

[0060] In this embodiment, as Figure 3 The diagram shows the multimodal fusion online image stabilization network structure. The multimodal feature fusion branch includes a feature-level fusion module, an offset estimation module, and a reconstruction output module. In the feature-level fusion module, the 512-dimensional IMU feature vector from branch 1 and the 256-dimensional image feature vector from branch 2 are concatenated along the feature dimension to form a 768-dimensional joint feature vector. In the offset estimation module, a fully connected layer maps the joint feature vector to an offset matrix with the same size as the input first-level stabilized frame. This offset matrix reflects the subtle motion parameters that need to be adjusted between image frames, ensuring the smoothness and stability of the inter-frame transition. Next, the offset matrix is ​​concatenated with the original first-level stabilized frame along the channel direction to form a multi-channel fused feature map. In the reconstruction output module, a 3×3 convolutional layer is applied, combined with BN and ReLU activation functions, to further process spatial information and extract features from the fused feature map. Finally, another 3×3×C convolutional layer, combined with BN and tanh activation functions, outputs a second-level stabilized frame with the same size and channels as the input first-level stabilized frame. The entire generation process requires no additional post-processing steps, achieving an end-to-end image stabilization process, simplifying the system architecture and improving processing efficiency.

[0061] In one embodiment, the step of obtaining the loss function includes: constructing a loss function based on pixel difference loss, similarity loss, and optical flow consistency loss; the loss function is:

[0062] L=λ1L p +λ2L flow +λ3L SSIM

[0063] Where L is the total loss function, L p For pixel difference loss, L flow For optical flow uniformity loss, L SSIM The loss is used to measure similarity, and λ1, λ2, and λ3 are hyperparameters.

[0064] In this embodiment, to effectively train the proposed multimodal fusion deep network image stabilization model and ensure its optimal video stabilization performance in UAV-borne scenarios, the method of this invention maximizes the model's image stabilization effect and generalization ability through the construction of loss functions, the formulation of training strategies, and the selection of optimizers and hyperparameters. The loss functions include pixel difference loss, similarity loss, and optical flow consistency loss.

[0065] The pixel difference loss uses the Smooth L1 loss function, which measures the difference between the generated stable frame and the real stable frame. The specific definition is as follows:

[0066]

[0067] Where N is the number of training samples in the batch, the specific calculation method of smoothL1 is as follows:

[0068]

[0069] Here, β is a hyperparameter used to control the smoothness of the loss function at inflection points.

[0070] Calculating the Optical Flow Consistency Loss ensures motion consistency between the generated stable frames and the ground truth stable frames. Optical flow calculation is based on the estimation of the optical flow field, which describes the motion vector of each pixel between two consecutive frames. The fundamental assumption of optical flow is brightness consistency, meaning that the brightness value of the same object point remains unchanged in different frames. We utilize the pre-trained optical flow estimation network FlowNet to calculate the optical flow field between the generated stable frames and the ground truth stable frames. The specific definition of the optical flow consistency loss is as follows:

[0071]

[0072] in, This represents the optical flow field estimation between generated and ground truth frames by the pre-trained optical flow estimation network. By minimizing the optical flow consistency loss, the model can maintain motion consistency with ground truth stable frames when generating stable frames, thereby effectively improving the smoothness and stability of the video. N is the number of training samples in the batch.

[0073] Similarity loss measures the similarity between generated and real frames in terms of brightness, contrast, and structure, and is defined as:

[0074]

[0075] Where N is the number of training samples in the batch; SSIM(x,y) is calculated as follows:

[0076]

[0077] Where x and y are two images or image patches being compared, and μ x and μ y These are the average brightness of the image at x and y, respectively. and These are the brightness variances of the image x and y, respectively, σ xy Let x be the covariance of x and y, and C1 and C2 be stability constants used to prevent the denominator from being zero.

[0078] The specific training process is as follows:

[0079] Data preparation: Collect and preprocess video data and corresponding IMU sequences acquired by the UAV to generate training, validation, and test sets. Perform preliminary mechanical image stabilization compensation on the video frames and synchronize the corresponding IMU data.

[0080] Data augmentation: In each training iteration, random data augmentation is performed on the input data to increase sample diversity and improve the model's generalization ability.

[0081] Forward propagation: The jittery frames and IMU sequences are input into the network, and after feature extraction, fusion and offset prediction, stable frames are generated.

[0082] Loss calculation: Calculate the total loss function value based on the difference between the generated stable frame and the real stable frame.

[0083] Backpropagation and parameter update: The gradient is calculated using the backpropagation algorithm, and the model parameters are updated using the Adam optimizer.

[0084] Learning rate adjustment: Adjust the learning rate according to the cosine annealing strategy to promote stable convergence during the training process.

[0085] Model evaluation and early stopping: After each training round, the model performance is evaluated on the validation set. If the validation loss does not decrease significantly over several consecutive rounds, training is terminated early to save the best model.

[0086] Through the above training method, the proposed multimodal fusion deep network image stabilization model can not only effectively suppress large and small jitters in UAV videos, but also has good generalization ability and robustness, and can adapt to the video stabilization requirements of various complex flight environments.

[0087] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0088] In one embodiment, such as Figure 4 As shown, an unmanned aerial vehicle (UAV) multimodal fusion online image stabilization system is provided. The system includes:

[0089] The gimbal and multimodal feature fusion subsystem; the gimbal is equipped with an unmanned aerial vehicle (UAV) camera and IMU;

[0090] The UAV-borne camera is used to input the first-level stabilized frame obtained by gimbal mechanical image stabilization compensation into the multimodal feature fusion subsystem;

[0091] The IMU is used to input the preprocessed IMU sequence into the multimodal feature fusion subsystem;

[0092] The multimodal feature fusion subsystem acquires the preprocessed IMU sequence and the first-level stabilized frame after mechanical image stabilization compensation. The IMU sequence and the first-level stabilized frame are then input into the trained multimodal fusion online image stabilization network, and online image stabilization is performed through the trained multimodal fusion online image stabilization network.

[0093] The UAV-borne multimodal fusion online image stabilization system provided by this invention is an end-to-end online image stabilization system that combines the advantages of mechanical and electronic image stabilization. Mechanical image stabilization is used to achieve initial compensation for large-amplitude shaking, while electronic image stabilization eliminates small-amplitude shaking. This improves the overall efficiency and practicality of the system and lays the foundation for subsequent advanced vision tasks.

[0094] In one embodiment, such as Figure 5 As shown, an unmanned aerial vehicle (UAV)-borne multimodal fusion online image stabilization device is provided, comprising:

[0095] The sample acquisition module 502 is used to acquire preprocessed IMU sequence samples and first-level stable frame samples, and input the IMU sequence samples and first-level stable frame samples into the multimodal fusion online image stabilization network; the first-level stable frame samples are obtained by mechanically stabilizing the video frames; the multimodal fusion online image stabilization network includes an IMU feature extraction branch, an image feature extraction branch, and a multimodal feature fusion main branch; the multimodal feature fusion main branch includes a feature-level fusion module, an offset estimation module, and a reconstruction output module;

[0096] The feature extraction module 504 is used to extract features from IMU sequence samples and first-level stable frame samples through the IMU feature extraction branch and the image feature extraction branch, respectively, and output the IMU sequence feature vector and the image feature vector to the feature-level fusion module.

[0097] The result output module 506 is used to concatenate the IMU sequence feature vector and the image feature vector through the feature-level fusion module to output a joint feature vector. After the joint feature vector is mapped to the offset matrix through the offset estimation module, it is concatenated with the first-level stable frame sample in the channel direction to obtain a multi-channel fused feature map. The reconstruction output module performs feature extraction on the multi-channel fused feature map and outputs the second-level stable frame.

[0098] The model training module 508 is used to train the multimodal fusion online image stabilization network based on the pre-constructed loss function, pre-processed IMU sequence samples and first-level stable frame samples, so as to obtain the trained multimodal fusion online image stabilization network.

[0099] The online image stabilization module 510 is used for online image stabilization through a trained multimodal fusion online image stabilization network.

[0100] In one embodiment, the IMU feature extraction branch includes an embedding layer, a multi-head attention mechanism layer, a normalization layer, a fully connected layer, a ReLU activation function, and an IMU feature output layer; the output of the embedding layer and the output of the multi-head attention mechanism layer are fused in a skip connection manner; the output of the normalization layer and the output of the fully connected layer are fused in a skip connection manner.

[0101] In one embodiment, the method further includes obtaining preprocessed IMU sequence samples through an embedding layer, encoding the IMU sequence into a high-dimensional vector; extracting key features from the high-dimensional vector through a multi-head attention mechanism layer; fusing the high-dimensional vector and key features to obtain a first fusion result; performing layer normalization on the first fusion result through a normalization layer and outputting it to a fully connected layer; performing a nonlinear transformation on the output of the normalization layer through the fully connected layer and the ReLU activation function to output frame offset features; fusing the output of the normalization layer and the frame offset features to obtain a second fusion result; and performing normalization and max pooling on the second fusion result through an IMU feature output layer to obtain an IMU sequence feature vector.

[0102] In one embodiment, the image feature extraction branch includes a first feature extraction module, a second feature extraction module, a third feature extraction module, and an image feature output layer. The first feature extraction module convolves the first-level stable frame after mechanical image stabilization compensation to extract the spatial features of the first-level stable frame and output a first feature map. The second feature extraction module convolves the spatial feature map to extract the deep features of the spatial feature map and output a second feature map. The third feature extraction module convolves the second feature map to extract the image features of the second feature map and output a third feature map. The output layer performs max pooling on the third feature map to output an image feature vector.

[0103] In one embodiment, the reconstructed output module includes a first convolutional submodule and a second convolutional submodule; the first convolutional submodule includes a convolutional layer, BN, and ReLU activation functions; the second convolutional submodule includes a convolutional layer, BN, and tanh activation functions.

[0104] In one embodiment, a loss function is further constructed based on pixel difference loss, similarity loss, and optical flow consistency loss; the loss function is:

[0105] L=λ1L p +λ2L flow +λ3L SSIM

[0106] Where L is the total loss function, L p For pixel difference loss, L flow For optical flow uniformity loss, L SSIM The loss is used to measure similarity, and λ1, λ2, and λ3 are hyperparameters.

[0107] Specific limitations regarding the UAV-borne multimodal fusion online image stabilization device can be found in the limitations of the UAV-borne multimodal fusion online image stabilization method described above, and will not be repeated here. Each module in the aforementioned UAV-borne multimodal fusion online image stabilization device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0108] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an unmanned aerial vehicle (UAV) multimodal fusion online image stabilization method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0109] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0110] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0111] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0112] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0113] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0114] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for online image stabilization via multimodal fusion on an unmanned aerial vehicle (UAV), characterized in that, The method includes: Preprocessed IMU sequence samples and first-level stable frame samples are acquired and input into a multimodal fusion online image stabilization network. The first-level stable frame samples are obtained by mechanically stabilizing video frames. The multimodal fusion online image stabilization network includes an IMU feature extraction branch, an image feature extraction branch, and a multimodal feature fusion main branch. The multimodal feature fusion main branch includes a feature-level fusion module, an offset estimation module, and a reconstruction output module. The IMU sequence samples and the first-level stable frame samples are respectively extracted through the IMU feature extraction branch and the image feature extraction branch, and the IMU sequence feature vector and the image feature vector are output to the feature-level fusion module. The feature-level fusion module concatenates the IMU sequence feature vector and the image feature vector to output a joint feature vector. The offset estimation module maps the joint feature vector to an offset matrix and concatenates it with the first-level stable frame sample in the channel direction to obtain a multi-channel fused feature map. The reconstruction output module extracts features from the multi-channel fused feature map to output a second-level stable frame. The offset matrix has the same size as the first-level stable frame sample. The multimodal fusion online image stabilization network is trained based on a pre-constructed loss function, pre-processed IMU sequence samples, and first-level stable frame samples to obtain a trained multimodal fusion online image stabilization network. Online image stabilization is performed using a trained multimodal fusion online image stabilization network.

2. The method according to claim 1, characterized in that, The IMU feature extraction branch includes an embedding layer, a multi-head attention mechanism layer, a normalization layer, a fully connected layer, a ReLU activation function, and an IMU feature output layer; the output of the embedding layer and the output of the multi-head attention mechanism layer are fused in a skip connection manner; the output of the normalization layer and the output of the fully connected layer are fused in a skip connection manner.

3. The method according to claim 2, characterized in that, The IMU sequence samples are feature extracted using the IMU feature extraction branch, and the output IMU sequence feature vector includes: The preprocessed IMU sequence samples are obtained through the embedding layer, and the IMU sequences are encoded into high-dimensional vectors. Key features are extracted from the high-dimensional vector through the multi-head attention mechanism layer; The high-dimensional vector and the key features are fused to obtain a first fusion result. The first fusion result is then processed by the normalization layer and output to the fully connected layer. The output of the normalization layer is nonlinearly transformed by a fully connected layer and a ReLU activation function to output frame offset features. The output of the normalization layer and the frame offset features are then fused to obtain a second fusion result. The second fusion result is normalized and max-pooled by the IMU feature output layer to obtain the IMU sequence feature vector.

4. The method according to claim 1, characterized in that, The image feature extraction branch includes a first feature extraction module, a second feature extraction module, a third feature extraction module, and an image feature output layer; The first feature extraction module performs convolution on the first-level stable frame after mechanical stabilization compensation to extract the spatial features of the first-level stable frame and output the first feature map. The spatial feature map is convolved by the second feature extraction module to extract deep features from the spatial feature map and output a second feature map. The third feature extraction module performs convolution on the second feature map to extract image features from the second feature map and outputs the third feature map. The third feature layer is max-pooled through the output layer to output the image feature vector.

5. The method according to claim 1, characterized in that, The reconstructed output module includes a first convolutional submodule and a second convolutional submodule; the first convolutional submodule includes a convolutional layer, BN, and ReLU activation functions; the second convolutional submodule includes a convolutional layer, BN, and tanh activation functions.

6. The method according to claim 1, characterized in that, The steps to obtain the loss function include: A loss function is constructed based on pixel difference loss, similarity loss, and optical flow consistency loss; the loss function is: in, For the total loss function, For pixel difference loss, For optical flow consistency loss, To measure the loss for similarity, , , This is a hyperparameter.

7. An unmanned aerial vehicle (UAV) multimodal fusion online image stabilization system implemented according to any one of claims 1-6, characterized in that, The system includes a gimbal and a multimodal feature fusion subsystem; the gimbal is equipped with an unmanned aerial vehicle (UAV) camera and an IMU. The UAV-borne camera is used to input the first-level stable frame obtained by gimbal mechanical image stabilization compensation into the multimodal feature fusion subsystem; The IMU is used to input the preprocessed IMU sequence into the multimodal feature fusion subsystem; The multimodal feature fusion subsystem acquires the preprocessed IMU sequence and the first-level stabilized frame after mechanical image stabilization compensation, and inputs the IMU sequence and the first-level stabilized frame into the trained multimodal fusion online image stabilization network for online image stabilization.

8. An unmanned aerial vehicle (UAV)-borne multimodal fusion online image stabilization device, characterized in that, The device includes: The sample acquisition module is used to acquire preprocessed IMU sequence samples and first-level stable frame samples, and input the IMU sequence samples and the first-level stable frame samples into the multimodal fusion online image stabilization network; the first-level stable frame samples are obtained by mechanically stabilizing the video frames; the multimodal fusion online image stabilization network includes an IMU feature extraction branch, an image feature extraction branch, and a multimodal feature fusion main branch; the multimodal feature fusion main branch includes a feature-level fusion module, an offset estimation module, and a reconstruction output module; The feature extraction module is used to extract features from the IMU sequence samples and the first-level stable frame samples through the IMU feature extraction branch and the image feature extraction branch respectively, and output the IMU sequence feature vector and the image feature vector to the feature-level fusion module; The result output module is used to concatenate the IMU sequence feature vector and the image feature vector through the feature-level fusion module to output a joint feature vector. The offset estimation module maps the joint feature vector to an offset matrix and then concatenates it with the first-level stable frame sample in the channel direction to obtain a multi-channel fused feature map. The reconstruction output module extracts features from the multi-channel fused feature map and outputs a second-level stable frame. The offset matrix has the same size as the first-level stable frame sample. The model training module is used to train the multimodal fusion online image stabilization network based on a pre-constructed loss function, pre-processed IMU sequence samples, and first-level stable frame samples, so as to obtain a trained multimodal fusion online image stabilization network. The online image stabilization module is used to perform online image stabilization through a pre-trained multimodal fusion online image stabilization network.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images