Unmanned aerial vehicle-mounted multi-mode fusion online image stabilization method, system, device and equipment

Abstract

The invention relates to an unmanned aerial vehicle-mounted multi-mode fusion online image stabilization method, system, device and equipment. The method comprises the following steps: acquiring a preprocessed IMU sequence sample and a first-level stable frame sample, and inputting the IMU sequence sample and the first-level stable frame sample into a multi-modal fusion online image stabilization network; the first-level stable frame sample is obtained by performing mechanical image stabilization compensation on the video frame; the multi-modal fusion online image stabilization network comprises an IMU feature extraction branch, an image feature extraction branch and a multi-modal feature fusion master branch; the multi-modal feature fusion total branch comprises a feature level fusion module, an offset estimation module and a reconstruction output module; training the multi-modal fusion online image stabilization network according to a pre-constructed loss function, the preprocessed IMU sequence sample and the first-level stable frame sample to obtain a trained multi-modal fusion online image stabilization network; and carrying out on-line image stabilization through the trained multi-mode fusion on-line image stabilization network. By adopting the method, the visual effect and the stability of the image stabilization frame can be remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an unmanned aerial vehicle-mounted multi-modal fusion online image stabilization method, system, device and equipment. Background Art

[0002] With the rapid development of drone technology, drones are widely deployed in various application scenarios, such as environmental monitoring, agricultural remote sensing, urban planning, and film and television shooting, which puts forward more stringent requirements for high-quality video acquisition. However, since drones are inevitably affected by various factors during flight, such as air turbulence, mechanical vibration, and control errors, the collected videos have significant jitter problems. These jitters not only affect the visual quality of the video, but may also have a negative impact on subsequent video analysis and processing tasks. Therefore, how to effectively suppress the jitter in drone videos and improve video quality has become one of the key issues that need to be solved in the field of computer vision and image processing.

[0003] Existing image stabilization technologies are mainly divided into three types: mechanical, optical, and electronic image stabilization according to the implementation method and principle. Mechanical image stabilization relies on servo systems and sensors to build a stable platform to achieve reverse motion compensation for imaging devices, which is suitable for high-amplitude jitter scenes. However, its large structure, high power consumption and cost limit its wide application. However, mechanical image stabilization is still the mainstream of airborne platform image stabilization, and it can be integrated with other image stabilization technologies to further improve image stabilization performance. For example, the combination of gyroscope and improved ORB algorithm improves the speed and accuracy of the system. Optical image stabilization achieves image stabilization by compensating motion through optical elements. It is characterized by simple operation, light weight, and small size. It is suitable for environments where the camera device does not vibrate significantly, but its cost is high and its accuracy is limited. Electronic image stabilization uses image processing technology to calculate inter-frame deviation and perform reverse compensation. It is widely used due to its advantages such as easy operation, small size, low energy consumption, and strong real-time performance. At present, research focuses on two modules: motion estimation and motion filtering. The key to the motion estimation module is to provide camera motion information accurately and quickly. Common algorithms include block matching, optical flow and feature matching, the latter of which is widely used due to its scene independence and fewer conditional constraints. In terms of filtering technology, mean, Gaussian and Kalman filtering are commonly used methods.

[0004] In the field of electronic image stabilization technology, deep learning methods have become the focus of cutting-edge research. Compared with traditional electronic image stabilization technology, image stabilization methods based on convolutional neural networks (CNNs) show excellent robustness. Traditional technologies often face problems such as insufficient feature extraction and inaccurate feature point matching when processing low-quality videos with low light, blur and noise interference. In contrast, CNN can effectively extract high-dimensional abstract features through its deep convolutional structure, thereby avoiding dependence on explicit feature extraction and matching. In most cases, CNN-based image stabilization models are superior to traditional methods.

[0005] Despite this, the single electronic image stabilization method based on deep learning still faces performance bottlenecks when dealing with large-scale jitter. In order to comprehensively utilize the advantages of mechanical image stabilization and electronic image stabilization, researchers in recent years have tried to combine the two to achieve a more comprehensive and efficient image stabilization effect. In addition, the inertial measurement unit (IMU), as an important sensor for obtaining the motion state of the drone, provides acceleration and angular velocity data that can provide strong support for the image stabilization algorithm. However, how to effectively fuse image features with IMU sequence features is still a problem that needs to be solved. Summary of the invention

[0006] Based on this, it is necessary to provide an unmanned aerial vehicle-mounted multi-modal fusion online image stabilization method, system, device and equipment to address the above-mentioned technical problems.

[0007] An unmanned aerial vehicle multi-modal fusion online image stabilization method, the method comprising:

[0008] Obtain preprocessed IMU sequence samples and first-level stable frame samples, and input the IMU sequence samples and the first-level stable frame samples into a multimodal fusion online image stabilization network; the first-level stable frame samples are obtained by mechanically stabilizing the video frames; the multimodal fusion online image stabilization network includes an IMU feature extraction branch, an image feature extraction branch, and a multimodal feature fusion main branch; the multimodal feature fusion main branch includes a feature level fusion module, an offset estimation module, and a reconstruction output module;

[0009] Performing feature extraction on the IMU sequence samples and the first-level stable frame samples through the IMU feature extraction branch and the image feature extraction branch respectively, and outputting the IMU sequence feature vector and the image feature vector to the feature-level fusion module;

[0010] The IMU sequence feature vector and the image feature vector are concatenated by the feature-level fusion module to output a joint feature vector, the offset estimation module maps the joint feature vector to an offset matrix and then concatenates it with the first-level stable frame sample in the channel direction to obtain a multi-channel fusion feature map, and the reconstruction output module extracts features from the multi-channel fusion feature map to output a second-level stable frame;

[0011] The multimodal fusion online image stabilization network is trained according to a pre-constructed loss function, pre-processed IMU sequence samples and primary stable frame samples to obtain a trained multimodal fusion online image stabilization network;

[0012] Online image stabilization is performed through the trained multimodal fusion online image stabilization network.

[0013] A multi-modal fusion online image stabilization system carried by an unmanned aerial vehicle, the system comprising a gimbal and a multi-modal feature fusion subsystem; the gimbal is equipped with an unmanned aerial vehicle camera and an IMU;

[0014] The drone-mounted camera is used to input the primary stabilized frame obtained through the mechanical image stabilization compensation of the gimbal into the multi-modal feature fusion subsystem;

[0015] The IMU is used to input the preprocessed IMU sequence into the multimodal feature fusion subsystem;

[0016] The multimodal feature fusion subsystem obtains the preprocessed IMU sequence and the first-level stable frame after mechanical stabilization compensation, inputs the IMU sequence and the first-level stable frame into the trained multimodal fusion online stabilization network, and performs online stabilization through the trained multimodal fusion online stabilization network.

[0017] An unmanned aerial vehicle-mounted multi-modal fusion online image stabilization device, the device comprising:

[0018] A sample acquisition module is used to acquire preprocessed IMU sequence samples and first-level stable frame samples, and input the IMU sequence samples and the first-level stable frame samples into a multimodal fusion online image stabilization network; the first-level stable frame samples are obtained by mechanically stabilizing the video frames; the multimodal fusion online image stabilization network includes an IMU feature extraction branch, an image feature extraction branch, and a multimodal feature fusion main branch; the multimodal feature fusion main branch includes a feature level fusion module, an offset estimation module, and a reconstruction output module;

[0019] A feature extraction module, used to extract features from the IMU sequence samples and the first-level stable frame samples through the IMU feature extraction branch and the image feature extraction branch, respectively, and output an IMU sequence feature vector and an image feature vector to the feature-level fusion module;

[0020] A result output module is used to perform vector splicing on the IMU sequence feature vector and the image feature vector through the feature level fusion module, output a joint feature vector, map the joint feature vector to an offset matrix through the offset estimation module, and then splice it with the first-level stable frame sample in the channel direction to obtain a multi-channel fusion feature map, perform feature extraction on the multi-channel fusion feature map through the reconstruction output module, and output a second-level stable frame;

[0021] A model training module, used for training the multimodal fusion online image stabilization network according to a pre-constructed loss function, pre-processed IMU sequence samples and primary stable frame samples to obtain a trained multimodal fusion online image stabilization network;

[0022] The online image stabilization module is used to perform online image stabilization through the trained multimodal fusion online image stabilization network.

[0023] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0024] Obtain preprocessed IMU sequence samples and first-level stable frame samples, and input the IMU sequence samples and the first-level stable frame samples into a multimodal fusion online image stabilization network; the first-level stable frame samples are obtained by mechanically stabilizing the video frames; the multimodal fusion online image stabilization network includes an IMU feature extraction branch, an image feature extraction branch, and a multimodal feature fusion main branch; the multimodal feature fusion main branch includes a feature level fusion module, an offset estimation module, and a reconstruction output module;

[0025] Performing feature extraction on the IMU sequence samples and the first-level stable frame samples through the IMU feature extraction branch and the image feature extraction branch respectively, and outputting the IMU sequence feature vector and the image feature vector to the feature-level fusion module;

[0026] The IMU sequence feature vector and the image feature vector are concatenated by the feature-level fusion module to output a joint feature vector, the offset estimation module maps the joint feature vector to an offset matrix and then concatenates it with the first-level stable frame sample in the channel direction to obtain a multi-channel fusion feature map, and the reconstruction output module extracts features from the multi-channel fusion feature map to output a second-level stable frame;

[0027] The multimodal fusion online image stabilization network is trained according to a pre-constructed loss function, pre-processed IMU sequence samples and primary stable frame samples to obtain a trained multimodal fusion online image stabilization network;

[0028] Online image stabilization is performed through the trained multimodal fusion online image stabilization network.

[0029] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0030] Obtain preprocessed IMU sequence samples and first-level stable frame samples, and input the IMU sequence samples and the first-level stable frame samples into a multimodal fusion online image stabilization network; the first-level stable frame samples are obtained by mechanically stabilizing the video frames; the multimodal fusion online image stabilization network includes an IMU feature extraction branch, an image feature extraction branch, and a multimodal feature fusion main branch; the multimodal feature fusion main branch includes a feature level fusion module, an offset estimation module, and a reconstruction output module;

[0031] Performing feature extraction on the IMU sequence samples and the first-level stable frame samples through the IMU feature extraction branch and the image feature extraction branch respectively, and outputting the IMU sequence feature vector and the image feature vector to the feature-level fusion module;

[0032] The IMU sequence feature vector and the image feature vector are concatenated by the feature-level fusion module to output a joint feature vector, the offset estimation module maps the joint feature vector to an offset matrix and then concatenates it with the first-level stable frame sample in the channel direction to obtain a multi-channel fusion feature map, and the reconstruction output module extracts features from the multi-channel fusion feature map to output a second-level stable frame;

[0033] The multimodal fusion online image stabilization network is trained according to a pre-constructed loss function, pre-processed IMU sequence samples and primary stable frame samples to obtain a trained multimodal fusion online image stabilization network;

[0034] Online image stabilization is performed through the trained multimodal fusion online image stabilization network.

[0035] The above-mentioned unmanned aerial vehicle multimodal fusion online stabilization method, system, device and equipment obtains a first-level stable frame by performing preliminary mechanical stabilization compensation on the video frame, obtains multimodal data based on the preprocessed IMU sequence and the corresponding first-level stable frame by preprocessing the IMU sequence output by the IMU unit, constructs training samples based on the multimodal data, and performs feature extraction, feature-level fusion, offset estimation and reconstruction on the input multimodal data through the constructed multimodal fusion online stabilization network, outputs a second-level stable frame, realizes the deep fusion of image features and motion sensor features, effectively improves the performance and robustness of unmanned aerial vehicle video stabilization, continues to train the multimodal fusion online stabilization network using loss functions and training samples, and uses the trained multimodal fusion online stabilization network to perform online stabilization on the multimodal data. Preliminary compensation for large jitters is achieved by mechanical stabilization, and small jitters are eliminated by electronic stabilization, and high-quality second-level stable frames are output. The embodiments of the present invention significantly improve the stabilization effect and robustness of unmanned aerial vehicle video frame processing, and improve the visual effect and stability of the stabilized frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of a flow chart of a multi-modal fusion online image stabilization method for unmanned aerial vehicles in one embodiment;

[0037] Figure 2 A schematic diagram of the principle of multi-modal fusion online image stabilization in one embodiment;

[0038] Figure 3A schematic diagram of a multi-modal fusion online image stabilization network structure in one embodiment;

[0039] Figure 4 It is a structural block diagram of a multi-modal fusion online image stabilization system for unmanned aerial vehicles in one embodiment;

[0040] Figure 5 It is a structural block diagram of a multi-modal fusion online image stabilization device carried by an unmanned aerial vehicle in one embodiment;

[0041] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0043] In one embodiment, Figure 1 As shown, a method for online image stabilization of multi-modal fusion carried by an unmanned aerial vehicle is provided, comprising the following steps:

[0044] Step 102, obtaining the preprocessed IMU sequence samples and the first-level stable frame samples, and inputting the IMU sequence samples and the first-level stable frame samples into the multi-modal fusion online image stabilization network.

[0045] The first-level stable frame samples are obtained by mechanically stabilizing the video frames. During the video acquisition stage, the mechanical stabilization module uses hardware equipment such as the gimbal to physically compensate for the jitter of the drone's onboard camera to obtain the first-level stable frame. This process mainly targets the significant jitter caused by the flight environment and mechanical vibration, effectively reducing the adjustment range required for subsequent processing and reducing the amount of calculation. At the same time, the inertial measurement unit (IMU) collects data such as acceleration and angular velocity of the drone's current frame and the previous N frames in real time to provide accurate motion state information. These data are pre-processed by normalization and denoising to meet the input requirements of the subsequent deep network.

[0046] The multimodal fusion online image stabilization network includes an IMU feature extraction branch, an image feature extraction branch and a multimodal feature fusion main branch; the multimodal feature fusion main branch includes a feature-level fusion module, an offset estimation module and a reconstruction output module.

[0047] Step 104, feature extraction is performed on the IMU sequence samples and the first-level stable frame samples through the IMU feature extraction branch and the image feature extraction branch respectively, and the IMU sequence feature vector and the image feature vector are output to the feature-level fusion module.

[0048] like Figure 2The schematic diagram of the principle of multimodal fusion online stabilization is shown. In the IMU feature extraction branch, the IMU sequence data is subjected to feature extraction through a lightweight Transformer structure. With its multi-head attention mechanism, the Transformer can effectively capture the temporal dependency and key motion features in the IMU data and generate a high-dimensional motion feature representation. The IMU feature extraction branch shows superior performance in processing dynamically changing sensor data, ensuring the accurate extraction of motion features. In the image feature extraction branch, the first-level stabilized frame is used as input, and the pre-trained CNN extracts high-level image features. This process can capture the spatial information and detail features in the image, providing rich visual information support for the subsequent stabilization process.

[0049] Step 106, the IMU sequence feature vector and the image feature vector are concatenated through the feature-level fusion module to output a joint feature vector, the joint feature vector is mapped to an offset matrix through the offset estimation module and then concatenated with the first-level stable frame sample in the channel direction to obtain a multi-channel fusion feature map, the multi-channel fusion feature map is extracted through the reconstruction output module, and a second-level stable frame is output.

[0050] The image features from the image feature extraction branch are fused with the IMU motion features from the IMU feature extraction branch at the feature level. The fusion method includes feature concatenation for weighted combination to fully utilize multi-source information to improve image stabilization. The fused feature vector is input into the fully connected layer to predict the offset information of the current frame. This offset reflects the specific parameters that need to be adjusted to achieve smooth transition and stability between frames. After decoding, the predicted offset is jump-connected to the first-level stable frame to preserve the details of the original image and ensure high-quality image reconstruction. The jump-connected data is further processed through equal-sized convolutional layers to generate second-level stable video frames. This process ensures consistency in spatial resolution between the output frame and the input frame, while improving image stability and visual quality.

[0051] Step 108 , training the multimodal fusion online image stabilization network according to the pre-constructed loss function, the pre-processed IMU sequence samples and the first-level stable frame samples to obtain a trained multimodal fusion online image stabilization network.

[0052] Step 110: Perform online image stabilization using the trained multi-modal fusion online image stabilization network.

[0053] In the above-mentioned unmanned aerial vehicle multimodal fusion online stabilization method, a primary stable frame is obtained by performing preliminary mechanical stabilization compensation on the video frame, and a preprocessing is performed on the IMU sequence sequence output by the IMU unit, and multimodal data is obtained based on the preprocessed IMU sequence and the corresponding primary stable frame. A training sample is constructed based on the multimodal data, and a feature extraction, feature-level fusion, offset estimation and reconstruction are performed on the input multimodal data through the constructed multimodal fusion online stabilization network, and a secondary stable frame is output, so that the deep fusion of image features and motion sensor features is realized, and the performance and robustness of the unmanned aerial vehicle video stabilization are effectively improved. The multimodal fusion online stabilization network is further trained using the loss function and the training sample, and the multimodal fusion online stabilization network is used to perform online stabilization on the multimodal data. The mechanical stabilization is used to achieve preliminary compensation for large jitters, and then the electronic stabilization is used to eliminate small jitters, and a high-quality secondary stable frame is output. The embodiment of the present invention significantly improves the stabilization effect and robustness of the unmanned aerial vehicle video frame processing, and improves the visual effect and stability of the stabilized frame.

[0054] In one embodiment, the IMU feature extraction branch includes an embedding layer, a multi-head attention mechanism layer, a normalization layer, a fully connected layer, a ReLU activation function and an IMU feature output layer; the output of the embedding layer is fused with the output of the multi-head attention mechanism layer in a skip connection manner; the output of the normalization layer is fused with the output of the fully connected layer in a skip connection manner.

[0055] In one embodiment, feature extraction is performed on IMU sequence samples through an IMU feature extraction branch, and outputting an IMU sequence feature vector includes: obtaining preprocessed IMU sequence samples through an embedding layer, and encoding the IMU sequence into a high-dimensional vector; extracting key features in the high-dimensional vector through a multi-head attention mechanism layer; fusing the high-dimensional vector and the key features to obtain a first fusion result, performing layer normalization processing on the first fusion result through a normalization layer, and then outputting it to a fully connected layer; performing a nonlinear transformation on the output of the normalization layer through a fully connected layer and a ReLU activation function, outputting a frame offset feature, and fusing the output of the normalization layer and the frame offset feature to obtain a second fusion result; normalizing and maximum pooling the second fusion result through an IMU feature output layer to obtain an IMU sequence feature vector.

[0056] In this embodiment, if Figure 3The schematic diagram of the multimodal fusion online image stabilization network structure shown in the figure shows that the IMU feature extraction branch adopts the Transformer architecture, which is specifically used to process IMU sequence data. The input IMU sequence contains the motion state information such as acceleration and angular velocity of the current frame and the previous N frames. The IMU sequence is encoded through the embedding layer, and the multi-dimensional motion data is mapped to the high-dimensional feature space to enhance the feature representation capability. Subsequently, the multi-head attention mechanism layer plays a key role in capturing time dependencies and dynamic motion patterns, and can effectively extract key features from IMU data. A jump connection is introduced between the output of the multi-head attention mechanism layer and the input of the embedding layer, which can alleviate the gradient vanishing problem in the deep network, and is processed by the layer normalization (LN) layer to ensure the stability of feature propagation and the efficiency of network training. Then, after the nonlinear transformation of the fully connected layer and the ReLU activation function, the feature expression is further enriched. The aforementioned features are once again fused with the output of the fully connected layer through the jump connection, and another layer of normalization and maximum pooling layer is passed to finally generate a 512-dimensional IMU sequence feature vector. The IMU feature extraction branch fully utilizes the advantages of the Transformer structure in sequence data processing. Considering the small amount of IMU sequence data, the Transformer network is designed to be lightweight to reduce the amount of calculation and improve the practicality under airborne conditions.

[0057] In one embodiment, an image feature extraction branch includes a first feature extraction module, a second feature extraction module, a third feature extraction module and an image feature output layer; the first feature extraction module convolves the first-level stabilized frame that has been compensated for mechanical stabilization, extracts the spatial features of the first-level stabilized frame, and outputs a first feature map; the second feature extraction module convolves the spatial feature map, extracts the deep features of the spatial feature map, and outputs a second feature map; the third feature extraction module convolves the second feature map, extracts the image features of the second feature map, and outputs a third feature map; the output layer performs maximum pooling on the third feature layer, and outputs an image feature vector.

[0058] In this embodiment, if Figure 3The schematic diagram of the multimodal fusion online image stabilization network structure shown in Figure 1 shows that the image feature extraction branch uses a convolutional neural network (CNN) architecture specifically for extracting image features. The input is a first-level stabilized frame obtained after preliminary compensation of mechanical image stabilization, with a resolution of W×H×C. First, a 7×7×64 convolution layer is applied, combined with batch normalization (BN) and ReLU activation functions to extract low-level spatial features. Subsequently, after a 3×3×128 convolution layer, BN and ReLU are applied again to gradually increase the depth of the feature map to capture more complex image patterns. Next, a 3×3×256 convolution layer is used, combined with BN and ReLU, to further extract high-level image features. Finally, the spatial resolution of the feature map is reduced through a maximum pooling layer, while retaining the most significant features to generate a 256-dimensional image feature vector. This branch design can effectively capture the details and high-level semantic information in the image through layer-by-layer convolution and nonlinear activation, providing rich visual feature support for the image stabilization process.

[0059] In one embodiment, the reconstruction output module includes a first convolution submodule and a second convolution submodule; the first convolution submodule includes a convolution layer, BN and ReLU activation functions; the second convolution submodule includes a convolution layer, BN and tanh activation functions.

[0060] In this embodiment, if Figure 3 The schematic diagram of the multimodal fusion online image stabilization network structure shown in the figure shows that the multimodal feature fusion main branch includes a feature-level fusion module, an offset estimation module, and a reconstruction output module. In the feature-level fusion module, the 512-dimensional IMU feature vector from branch 1 and the 256-dimensional image feature vector from branch 2 are concatenated in the feature dimension to form a 768-dimensional joint feature vector. In the offset estimation module, a fully connected layer is used to map the joint feature vector to an offset matrix that is consistent with the input primary stable frame size. The offset matrix reflects the subtle motion parameters that need to be adjusted between image frames to ensure the smoothness and stability of the transition between frames. Next, the offset matrix is ​​concatenated with the original primary stable frame in the channel direction to form a multi-channel fusion feature map. In the reconstruction output module, a 3×3 convolutional layer is applied, combined with BN and ReLU activation functions, to further process the spatial information and extract features from the fused feature map. Finally, after another 3×3×C convolutional layer, combined with BN and tanh activation functions, a secondary stable frame output that is consistent with the input primary stable frame size and channel is achieved. The entire generation process does not require additional post-processing steps, realizing an end-to-end image stabilization process, simplifying the system architecture and improving processing efficiency.

[0061] In one embodiment, the step of obtaining the loss function includes: constructing a loss function according to pixel difference loss, similarity loss and optical flow consistency loss; the loss function is:

[0062] L=λ1L p +λ2L flow +λ3L SSIM

[0063] Among them, L is the total loss function, L p is the pixel difference loss, L flow is the optical flow consistency loss, L SSIM is the similarity metric loss, and λ1, λ2, and λ3 are hyperparameters.

[0064] In this embodiment, in order to effectively train the proposed multimodal fusion deep network image stabilization model and ensure that its video stabilization performance in drone-mounted scenarios is optimal, the method of the present invention maximizes the image stabilization effect and generalization ability of the model through the construction of loss functions, the formulation of training strategies, and the selection of optimizers and hyperparameters. The loss functions include pixel difference loss, similarity loss, and optical flow consistency loss.

[0065] The pixel difference loss uses the Smooth L1 loss function, which can measure the difference between the generated stable frame and the real stable frame. The specific definition is as follows:

[0066]

[0067] Among them, N is the number of batch training samples, and the specific calculation method of smoothL1 is as follows:

[0068]

[0069] Among them, β is a hyperparameter used to control the smoothness of the loss function at the turning point.

[0070] Calculating the optical flow consistency loss can ensure the motion consistency between the generated stable frame and the real stable frame. The optical flow calculation is based on the estimation of the optical flow field, which describes the motion vector of each pixel between two consecutive frames. The basic assumption of optical flow is brightness consistency, that is, the brightness value of the same object point in different frames remains unchanged. We use the pre-trained optical flow estimation network FlowNet to calculate the optical flow field between the generated stable frame and the real stable frame. The specific optical flow consistency loss is defined as follows:

[0071]

[0072] in, Represents the optical flow field estimation between the generated frame and the real frame by the pre-trained optical flow estimation network. By minimizing the optical flow consistency loss, the model can maintain the motion consistency between the generated stable frame and the real stable frame, thereby effectively improving the smoothness and stability of the video. N is the number of batch training samples.

[0073] The similarity loss measures the similarity between the generated frame and the real frame in terms of brightness, contrast, and structure, and is defined as:

[0074]

[0075] Where N is the number of batch training samples; SSIM(x,y) is calculated as:

[0076]

[0077] Where x and y are the two images or image patches to be compared, μ x and μ y are the average brightness of images x and y, respectively, and are the brightness variance of images x and y, σ xy is the covariance of x and y, C1 and C2 are stability constants used to prevent the denominator from being zero.

[0078] The specific training process is as follows:

[0079] Data preparation: Collect and preprocess the video data and corresponding IMU sequences collected by the drone to generate training, validation, and test sets. Perform preliminary mechanical stabilization compensation on the video frames and synchronize the corresponding IMU data.

[0080] Data augmentation: In each training iteration, random data augmentation is performed on the input data to increase sample diversity and improve the generalization ability of the model.

[0081] Forward propagation: The jittered frames and IMU sequences are input into the network, and stable frames are generated after feature extraction, fusion and offset prediction.

[0082] Loss calculation: Calculate the total loss function value based on the difference between the generated stabilized frame and the real stabilized frame.

[0083] Back propagation and parameter update: The gradient is calculated through the back propagation algorithm, and the Adam optimizer is used to update the model parameters.

[0084] Learning rate adjustment: The learning rate is adjusted according to the cosine annealing strategy to promote stable convergence of the training process.

[0085] Model evaluation and early stopping: After each training round, the model performance is evaluated on the validation set. If the validation loss does not decrease significantly in several consecutive rounds, the training is terminated early and the best model is saved.

[0086] Through the above training method, the proposed multimodal fusion deep network stabilization model can not only effectively suppress large and subtle jitters in drone videos, but also has good generalization ability and robustness, and can adapt to the video stabilization needs in various complex flight environments.

[0087] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0088] In one embodiment, Figure 4 As shown, a multi-modal fusion online image stabilization system for unmanned aerial vehicles is provided, and the system includes:

[0089] Gimbal and multi-modal feature fusion subsystem; the gimbal is equipped with a drone-mounted camera and IMU;

[0090] The drone-mounted camera is used to input the first-level stabilized frame obtained through the mechanical image stabilization compensation of the gimbal into the multi-modal feature fusion subsystem;

[0091] The IMU is used to input the preprocessed IMU sequence into the multi-modal feature fusion subsystem;

[0092] The multimodal feature fusion subsystem obtains the preprocessed IMU sequence and the first-level stabilized frame after mechanical stabilization compensation, inputs the IMU sequence and the first-level stabilized frame into the trained multimodal fusion online stabilization network, and performs online stabilization through the trained multimodal fusion online stabilization network.

[0093] The multi-modal fusion online stabilization system provided by the present invention is an end-to-end online stabilization system, which combines the advantages of mechanical stabilization and electronic stabilization, uses mechanical stabilization to achieve preliminary compensation for large jitters, and then eliminates small jitters through electronic stabilization. It improves the overall efficiency and practicality of the system and lays the foundation for subsequent advanced visual tasks.

[0094] In one embodiment, Figure 5 As shown, a multi-modal fusion online image stabilization device for unmanned aerial vehicles is provided, comprising:

[0095] The sample acquisition module 502 is used to obtain the pre-processed IMU sequence samples and the first-level stable frame samples, and input the IMU sequence samples and the first-level stable frame samples into the multi-modal fusion online image stabilization network; the first-level stable frame samples are obtained by mechanically stabilizing the video frames; the multi-modal fusion online image stabilization network includes an IMU feature extraction branch, an image feature extraction branch and a multi-modal feature fusion main branch; the multi-modal feature fusion main branch includes a feature level fusion module, an offset estimation module and a reconstruction output module;

[0096] A feature extraction module 504 is used to extract features from the IMU sequence samples and the first-level stable frame samples through an IMU feature extraction branch and an image feature extraction branch, respectively, and output an IMU sequence feature vector and an image feature vector to a feature-level fusion module;

[0097] The result output module 506 is used to perform vector splicing on the IMU sequence feature vector and the image feature vector through the feature level fusion module, output a joint feature vector, map the joint feature vector to the offset matrix through the offset estimation module, and then splice it with the first-level stable frame sample in the channel direction to obtain a multi-channel fusion feature map, perform feature extraction on the multi-channel fusion feature map through the reconstruction output module, and output a second-level stable frame;

[0098] A model training module 508 is used to train the multimodal fusion online image stabilization network according to a pre-constructed loss function, pre-processed IMU sequence samples and first-level stable frame samples to obtain a trained multimodal fusion online image stabilization network;

[0099] The online image stabilization module 510 is used to perform online image stabilization through a trained multi-modal fusion online image stabilization network.

[0100] In one embodiment, the IMU feature extraction branch includes an embedding layer, a multi-head attention mechanism layer, a normalization layer, a fully connected layer, a ReLU activation function and an IMU feature output layer; the output of the embedding layer is fused with the output of the multi-head attention mechanism layer in a skip connection manner; the output of the normalization layer is fused with the output of the fully connected layer in a skip connection manner.

[0101] In one of the embodiments, it is also used to obtain preprocessed IMU sequence samples through an embedding layer, and encode the IMU sequence into a high-dimensional vector; extract key features in the high-dimensional vector through a multi-head attention mechanism layer; fuse the high-dimensional vector and the key features to obtain a first fusion result, and perform layer normalization on the first fusion result through a normalization layer and then output it to a fully connected layer; perform nonlinear transformation on the output of the normalization layer through a fully connected layer and a ReLU activation function, output a frame offset feature, and fuse the output of the normalization layer and the frame offset feature to obtain a second fusion result; normalize and max-pool the second fusion result through an IMU feature output layer to obtain an IMU sequence feature vector.

[0102] In one of the embodiments, the image feature extraction branch includes a first feature extraction module, a second feature extraction module, a third feature extraction module and an image feature output layer; the first feature extraction module convolves the first-level stabilized frame that has been compensated for mechanical stabilization, extracts the spatial features of the first-level stabilized frame, and outputs a first feature map; the second feature extraction module convolves the spatial feature map, extracts the deep features of the spatial feature map, and outputs a second feature map; the third feature extraction module convolves the second feature map, extracts the image features of the second feature map, and outputs a third feature map; the output layer performs maximum pooling on the third feature layer, and outputs an image feature vector.

[0103] In one of the embodiments, the reconstructed output module includes a first convolution submodule and a second convolution submodule; the first convolution submodule includes a convolution layer, BN and ReLU activation functions; the second convolution submodule includes a convolution layer, BN and tanh activation functions.

[0104] In one of the embodiments, it is also used to construct a loss function based on pixel difference loss, similarity loss and optical flow consistency loss; the loss function is:

[0105] L=λ1L p +λ2L flow +λ3L SSIM

[0106] Among them, L is the total loss function, L p is the pixel difference loss, L flow is the optical flow consistency loss, L SSIM is the similarity metric loss, and λ1, λ2, and λ3 are hyperparameters.

[0107] For the specific definition of the unmanned aerial vehicle multimodal fusion online image stabilization device, please refer to the definition of the unmanned aerial vehicle multimodal fusion online image stabilization method mentioned above, which will not be repeated here. Each module in the above-mentioned unmanned aerial vehicle multimodal fusion online image stabilization device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0108] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for online stabilization of multi-modal fusion on a drone is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0109] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0110] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

[0111] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0112] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0113] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0114] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. An unmanned aerial vehicle multi-modal fusion online image stabilization method, characterized in that: The method comprises: Obtain preprocessed IMU sequence samples and first-level stable frame samples, and input the IMU sequence samples and the first-level stable frame samples into a multimodal fusion online image stabilization network; the first-level stable frame samples are obtained by mechanically stabilizing the video frames; the multimodal fusion online image stabilization network includes an IMU feature extraction branch, an image feature extraction branch, and a multimodal feature fusion main branch; the multimodal feature fusion main branch includes a feature level fusion module, an offset estimation module, and a reconstruction output module; Performing feature extraction on the IMU sequence samples and the first-level stable frame samples through the IMU feature extraction branch and the image feature extraction branch respectively, and outputting the IMU sequence feature vector and the image feature vector to the feature-level fusion module; The IMU sequence feature vector and the image feature vector are concatenated by the feature-level fusion module to output a joint feature vector, the offset estimation module maps the joint feature vector to an offset matrix and then concatenates it with the first-level stable frame sample in the channel direction to obtain a multi-channel fusion feature map, and the reconstruction output module extracts features from the multi-channel fusion feature map to output a second-level stable frame; The multimodal fusion online image stabilization network is trained according to a pre-constructed loss function, pre-processed IMU sequence samples and primary stable frame samples to obtain a trained multimodal fusion online image stabilization network; Online image stabilization is performed through the trained multimodal fusion online image stabilization network.

2. The method according to claim 1, characterized in that The IMU feature extraction branch includes an embedding layer, a multi-head attention mechanism layer, a normalization layer, a fully connected layer, a ReLU activation function and an IMU feature output layer; the output of the embedding layer is fused with the output of the multi-head attention mechanism layer in a jump connection manner; the output of the normalization layer is fused with the output of the fully connected layer in a jump connection manner.

3. The method according to claim 2, characterized in that The IMU sequence samples are subjected to feature extraction by the IMU feature extraction branch, and the output IMU sequence feature vector includes: Obtain the preprocessed IMU sequence samples through the embedding layer, and encode the IMU sequence into a high-dimensional vector; Extracting key features from the high-dimensional vector through the multi-head attention mechanism layer; fusing the high-dimensional vector and the key feature to obtain a first fusion result, performing layer normalization processing on the first fusion result through the normalization layer and then outputting the result to a fully connected layer; Performing a nonlinear transformation on the output of the normalization layer through a fully connected layer and a ReLU activation function, outputting a frame offset feature, and fusing the output of the normalization layer and the frame offset feature to obtain a second fusion result; The second fusion result is normalized and maximum pooled through the IMU feature output layer to obtain an IMU sequence feature vector.

4. The method according to claim 1, characterized in that: The image feature extraction branch includes a first feature extraction module, a second feature extraction module, a third feature extraction module and an image feature output layer; Convolving the first-level stabilized frame after mechanical image stabilization compensation through the first feature extraction module, extracting the spatial features of the first-level stabilized frame, and outputting a first feature map; Convolving the spatial feature map through the second feature extraction module to extract deep features of the spatial feature map, and outputting a second feature map; Convolving the second feature map through the third feature extraction module to extract image features of the second feature map, and outputting a third feature map; The third feature layer is subjected to maximum pooling through the output layer to output an image feature vector.

5. The method according to claim 1, characterized in that The reconstruction output module includes a first convolution submodule and a second convolution submodule; the first convolution submodule includes a convolution layer, BN and ReLU activation functions; the second convolution submodule includes a convolution layer, BN and tanh activation functions.

6. The method according to claim 1, characterized in that The step of obtaining the loss function comprises: A loss function is constructed based on pixel difference loss, similarity loss and optical flow consistency loss; the loss function is: L=λ1L p +λ2L flow +λ3L SSIM Among them, L is the total loss function, L p is the pixel difference loss, L flow is the optical flow consistency loss, L SSIM is the similarity metric loss, and λ1, λ2, and λ3 are hyperparameters.

7. An unmanned aerial vehicle multi-modal fusion online image stabilization system, characterized in that: The system includes a gimbal and a multi-modal feature fusion subsystem; the gimbal is equipped with a drone-mounted camera and an IMU; The drone-mounted camera is used to input the primary stabilized frame obtained through the mechanical image stabilization compensation of the gimbal into the multi-modal feature fusion subsystem; The IMU is used to input the preprocessed IMU sequence into the multimodal feature fusion subsystem; The multimodal feature fusion subsystem obtains the preprocessed IMU sequence and the first-level stable frame after mechanical stabilization compensation, inputs the IMU sequence and the first-level stable frame into the trained multimodal fusion online stabilization network, and performs online stabilization through the trained multimodal fusion online stabilization network.

8. An unmanned aerial vehicle-mounted multi-modal fusion online image stabilization device, characterized in that: The device comprises: A sample acquisition module is used to acquire preprocessed IMU sequence samples and first-level stable frame samples, and input the IMU sequence samples and the first-level stable frame samples into a multimodal fusion online image stabilization network; the first-level stable frame samples are obtained by mechanically stabilizing the video frames; the multimodal fusion online image stabilization network includes an IMU feature extraction branch, an image feature extraction branch, and a multimodal feature fusion main branch; the multimodal feature fusion main branch includes a feature level fusion module, an offset estimation module, and a reconstruction output module; A feature extraction module, used to extract features from the IMU sequence samples and the first-level stable frame samples through the IMU feature extraction branch and the image feature extraction branch, respectively, and output an IMU sequence feature vector and an image feature vector to the feature-level fusion module; A result output module is used to perform vector splicing on the IMU sequence feature vector and the image feature vector through the feature level fusion module, output a joint feature vector, map the joint feature vector to an offset matrix through the offset estimation module, and then splice it with the first-level stable frame sample in the channel direction to obtain a multi-channel fusion feature map, perform feature extraction on the multi-channel fusion feature map through the reconstruction output module, and output a second-level stable frame; A model training module, used for training the multimodal fusion online image stabilization network according to a pre-constructed loss function, pre-processed IMU sequence samples and primary stable frame samples to obtain a trained multimodal fusion online image stabilization network; The online image stabilization module is used to perform online image stabilization through the trained multimodal fusion online image stabilization network.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images