Unmanned aerial vehicle vision adaptive control method based on multi-scale spatio-temporal network

Abstract

The application provides a kind of unmanned aerial vehicle vision adaptive control method based on multi-scale space-time network. By constructing multi-branch hollow convolution feature extraction and adaptive scale re-labeling network, the cooperative perception of near and far target features and background noise suppression are realized. On this basis, the dynamic multi-head self-attention mechanism with fusion time decay penalty factor is introduced, and the physical evolution law constraint is carried out on the continuous image sequence, effectively filtering out the instantaneous motion blur and target occlusion interference. In the state estimation link, the system constructs a deep coupling residual injection mechanism of vision and dynamics, and uses high-dimensional space-time perception features to correct the prediction deviation of the rigid body model in real time. Finally, combined with the error adaptive model predictive control and nonlinear robust disturbance observer, the optimal closed-loop control command is generated and directly drives the underlying electromechanical actuator. The application fundamentally breaks through the technical barriers of the separation system architecture, significantly improves the high-frequency state perception accuracy, large maneuver trajectory tracking performance and all-weather operation system robustness of the aircraft in the face of nonlinear gust disturbance and satellite navigation limited extreme working conditions.

Description

Technical Field

[0001] This invention relates to the field of intelligent autonomous control technology for unmanned aerial vehicles (UAVs), specifically a visual perception and adaptive flight control method for UAVs based on a multi-scale attention spatiotemporal fusion network. Background Technology

[0002] Unmanned aerial vehicles (UAVs) have been widely used in complex operations such as power line inspection, geographic surveying, logistics transportation, and emergency rescue. Current mainstream UAV systems primarily rely on two core technologies for attitude estimation and navigation control. The first is a combined state estimation method based on the fusion of an airborne inertial measurement unit (IMU) and a global positioning system (GPS); the second is an assisted positioning method based on single-frame image visual feature point detection and matching.

[0003] However, as the complexity of actual working environments continues to increase, existing technologies have gradually revealed many insurmountable technical bottlenecks.

[0004] Traditional UAV system architectures typically design the environmental perception module and flight control module completely independently. This separate open-loop or semi-open-loop architecture means that unpredictable errors and computational delays generated at the perception end cannot be dynamically corrected in real time at the control end, resulting in lag in the overall system's maneuver response and a significant decrease in the accuracy of end-point trajectory tracking.

[0005] At the visual perception level, existing deep learning-assisted vision algorithms mostly employ single-scale convolutional neural network structures. When drones perform large-scale, high-speed, and highly maneuverable flight missions, the scale of targets within the onboard camera's field of view changes drastically. A single-layer scale structure struggles to simultaneously capture the overall contours of large targets at close range and accurately represent the local features of small targets at a distance. Furthermore, traditional single-frame visual processing lacks an effective time-series dynamic evolution modeling mechanism. When drones encounter image motion blur caused by high-speed movement, momentary physical occlusion of targets by obstacles, or instantaneous noise interference caused by drastic changes in ambient lighting, single-frame visual algorithms are highly prone to outputting erroneous observations, leading to drastic changes in the aircraft's state estimation and potentially causing flight loss of control.

[0006] At the underlying flight control level, the feedback gain parameters of traditional proportional-integral-derivative (PID) control algorithms are typically preset fixed values. This fixed-parameter mechanism is severely inadequate in its adaptive adjustment capability to complex nonlinear airflow disturbances and gust interference. Furthermore, existing control law designs rarely directly embed high-dimensional visual perception results into the cost function of the model prediction optimization mechanism, limiting the aircraft's dynamic disturbance rejection capability in the face of sudden disturbances and making it highly susceptible to control oscillations.

[0007] Therefore, the industry needs to develop a novel integrated control method that unifies multi-scale visual feature extraction, temporal dynamic evolution modeling, coupled state estimation of vision and flight dynamics, and adaptive predictive control within a single closed-loop optimization framework. By addressing the shortcomings of a single visual model in the temporal dimension and dynamic scenarios, and by directly constraining and compensating for the cumulative errors of the flight dynamics model using visual observation data, this approach fundamentally overcomes the limitations of traditional separation between perception and control, significantly improving the control stability, disturbance resistance, and all-weather operation capabilities of unmanned aerial vehicles in complex, dynamic, and unknown environments. Summary of the Invention

[0008] This invention aims to address the technical problems of existing unmanned aerial vehicles (UAVs) in complex dynamic environments, such as poor system coordination due to the independent operation of perception and control modules, the inability of a single visual scale to simultaneously consider the characteristics of near and far targets, weak motion trend perception due to the lack of temporal modeling, and insufficient adaptability of traditional control algorithms to nonlinear disturbances. To solve these problems, this invention proposes an integrated visual perception and control method for UAVs based on a multi-scale attention spatiotemporal fusion network and adaptive predictive control. This method achieves closed-loop optimization through deep coupling of perceived features and dynamic models.

[0009] The technical solution adopted by the present invention to achieve the above objectives is as follows:

[0010] The UAV vision-adaptive control method based on multi-scale spatiotemporal networks includes the following steps:

[0011] 1) Acquire real-time image sequences of the flight environment, perform distortion correction on them, and convert the corrected sequences into digital matrices;

[0012] 2) A high-order spatially perceptual feature matrix is ​​obtained by performing feature extraction and adaptive recalibration on the digital matrix through a multi-branch parallel convolutional network;

[0013] 3) Construct a sequence feature matrix from the high-order spatial perception feature matrix of multiple consecutive frames, and perform spatiotemporal fusion modeling through a dynamic multi-head self-attention mechanism that incorporates time decay penalty factors to generate high-dimensional spatiotemporal fusion features;

[0014] 4) The high-dimensional spatiotemporal fusion features are mapped to the evolutionary deviation compensation amount of the flight state through the residual regression network, and the compensation amount is injected into the prior state predicted based on the flight dynamics equation to obtain the posterior state estimate.

[0015] 5) Based on the posterior state estimate, the state weight matrix is ​​dynamically adjusted online according to the magnitude of the current tracking error using the error adaptive model predictive control algorithm, and the optimal nominal control sequence is solved under the condition of satisfying the physical constraints of the actuator.

[0016] 6) External disturbances and unmodeled dynamics are estimated in real time using a nonlinear robust disturbance observer to generate lumped disturbance estimates, and feedforward compensation is performed on the optimal nominal control sequence to generate the final closed-loop control command.

[0017] The multi-branch parallel convolutional network consists of parallel standard convolutional branches for acquiring local fine texture features and dilated convolutional branches for expanding the receptive field. The number of dilated convolutional branches is at least one, and when there are multiple dilated convolutional branches, the dilation rate of each branch is different.

[0018] The equivalent receptive field of a single dilated convolution branch for:

[0019]

[0020] in, The kernel size is the convolution kernel size. This represents the void ratio.

[0021] The adaptive recalibration includes the following steps:

[0022] The multi-scale feature maps output from all parallel branches are initially concatenated and stitched along the channel dimension to obtain the baseline fused feature tensor. ;

[0023] Perform a global average pooling operation on the spatial dimension of the baseline fused feature tensor to obtain a one-dimensional channel descriptor vector. ,Right now:

[0024]

[0025] in, For the channel descriptor vector, the first... Scalar values ​​for each channel, and These represent the spatial height and width of the feature map, respectively. The pixel activation values ​​of the baseline fused feature tensor in a specific spatial coordinate system;

[0026] A weight generation module consisting of a two-layer fully connected network and a non-linear activation function is constructed, and the scale weight vectors of features at each scale are calculated using channel descriptor vectors. :

[0027]

[0028] in, and This refers to the weight matrix that the network model can learn during training. It is a linear rectified activation function. The normalized activation function;

[0029] The generated scale weight vector is multiplied element-wise with the initial baseline fused feature tensor to obtain the final output high-order spatial perception feature matrix. :

[0030]

[0031] in, Element-wise multiplication operation representing the feature channel dimension.

[0032] The dynamic multi-head self-attention mechanism with fusion time decay penalty factor includes the following steps:

[0033] Let the total number of heads in the multi-head attention mechanism be . For the first Each attention point is used to construct a query matrix. Key matrix and value matrix Its linear mapping generation formula is:

[0034]

[0035]

[0036]

[0037] in, , and These are all linear mapping weight matrices that can be learned within the attention head. The input sequence feature matrix is ​​a combination of spatial features from multiple historical frames and temporal location coding.

[0038] Construct the time decay penalty matrix , The total number of historical image frames involved in time-series modeling, the nth frame in this matrix Line 1 Column element values for:

[0039]

[0040] Among them, variables and These are the time step indices in the sequence feature matrix. This is a non-negative time decay adjustment coefficient;

[0041] Attention distribution for calculating fusion time decay penalty factor :

[0042]

[0043] in, This is a matrix transpose operation. This refers to the feature dimension of the key matrix in a single-head attention mechanism.

[0044] All The outputs of each attention head are concatenated and spliced ​​along the feature dimension, and then aggregated through a fully connected feedforward network to obtain the final spatiotemporal fusion feature sequence matrix.

[0045] Step 4) includes the following steps:

[0046] 4.1) Construct a twelve-degree-of-freedom nonlinear state-space model of the unmanned aerial vehicle. Its discretized nominal dynamic model is as follows:

[0047]

[0048] in, Based on the posterior state of the previous time step With control input commands The predicted prior state vector at the current moment is composed of the UAV's three-axis position, three-axis axial velocity, three-axis Euler angles, and three-axis angular velocity, and the function... This is a nonlinear transfer function constructed based on rigid body dynamics.

[0049] 4.2) Construct a visual residual regression head based on deep neural networks to fuse high-dimensional spatiotemporal features. Mapped to residual estimation vector , Let the mapping function be denoted as;

[0050] 4.3) Obtaining posterior state estimates through residual injection mechanism .

[0051] The specific mapping process of the visual residual regression head is implemented through a multilayer perceptron structure, specifically as follows:

[0052] First, the spatiotemporal fusion feature matrix is ​​compressed in terms of feature dimension. Then, the compressed feature vector is sequentially input into a regression network containing at least two fully connected layers. Each fully connected layer is followed by a nonlinear activation function and a dropout layer. Finally, the fully connected layers output a residual estimation vector with the same dimension as the state vector.

[0053] The cost function of the error adaptive model predictive control algorithm is specifically as follows:

[0054] Using the posterior state estimate obtained through deep coupling of vision and dynamics as the initial state of the system at the current moment, and setting the prediction step size as... Control step size is Within each control cycle, the quadratic cost function is:

[0055]

[0056]

[0057] in, The total cost function at the current moment. For the state tracking error vector in the future prediction sequence, To control the input increment vector, For a fixed control increment weight penalty positive definite matrix, The terminal state penalty matrix, Here is the state error weight matrix. The initial state weight matrix, The adaptive adjustment coefficient is non-negative. This is the state tracking error vector at the current moment.

[0058] Step 6) includes the following steps:

[0059] 6.1) Calculate the posterior state estimate after visual-dynamic coupling. Compared with the nominal flight dynamics prediction value The difference between the state space vectors is used as the state prediction error vector. :

[0060]

[0061] 6.2) Construct a nonlinear robust perturbation observer with a forgetting memory mechanism, setting the lumped perturbation estimation vector at the current moment as... Its discretized recursive update formula is:

[0062]

[0063] in, This is the historical disturbance estimation vector from the previous control cycle. The forgetting factor is in diagonal matrix form. The linear observation gain matrix, It is a nonlinear robust gain matrix;

[0064] 6.3) Estimation vector based on lumped disturbance Feedforward compensation correction is performed at the output of predictive control, that is, the first term of the nominal optimal control sequence is set to... The final output closed-loop control command vector for:

[0065]

[0066] in, It is a disturbance input mapping matrix used to extract linear acceleration and angular acceleration disturbance components from high-dimensional state disturbances and transform them into an actuation mapping space of the same dimension as the flight control input.

[0067] The UAV visual adaptive control system based on multi-scale spatiotemporal networks includes a memory and a processor; the memory is used to store computer programs; the processor is used to implement the UAV visual adaptive control method based on multi-scale spatiotemporal networks when executing the computer programs.

[0068] The present invention has the following beneficial effects and advantages:

[0069] 1. This invention represents a substantial breakthrough from a perception-control separation architecture to a deeply integrated perception-control architecture. By constructing a vision-dynamics coupled state estimation module, the system directly utilizes the state residuals mapped from high-dimensional visual features to correct the prediction bias of the nominal dynamics model. This design fundamentally eliminates the command response lag problem caused by the separation of perception and control in traditional systems, greatly improving the trajectory tracking accuracy and closed-loop control frequency of unmanned aerial vehicles (UAVs) when performing high-maneuver missions.

[0070] 2. The multi-scale dilated feature recalibration mechanism significantly enhances the system's adaptability to complex dynamic environments. This invention does not employ a fixed single-scale convolutional structure, but instead uses a multi-branch parallel network with differentiated dilation rates to simultaneously capture local details and wide-area spatial structure. Combined with an adaptive scale recalibration network, the system can dynamically adjust the weights of each feature channel according to real-time changes in target distance, effectively suppressing interference from redundant background noise and solving the technical challenge of poor stability in recognizing small targets at long distances.

[0071] 3. The dynamic spatiotemporal self-attention mechanism incorporating a time decay penalty factor effectively solves the problem of state abrupt changes. Addressing extreme situations such as image motion blur and short-term physical occlusion of targets that are easily generated during high-speed flight of unmanned aerial vehicles (UAVs), this invention utilizes a temporal modeling mechanism to impose physical constraints on historical multi-frame perception information. This design ensures high consistency of the perception layer output in the temporal dimension, avoids severe attitude oscillations of the aircraft caused by instantaneous observation anomalies, and significantly improves the robustness of the system.

[0072] 4. The combination of error adaptive predictive control and a nonlinear robust disturbance observer significantly enhances the system's anti-interference level. This invention abandons the traditional control method with fixed parameters. By adjusting the error weight matrix online and compensating for the lumped disturbance estimate in real time, the aircraft can accurately counteract the negative effects of nonlinear gusts and unmodeled dynamic terms. This not only significantly reduces control energy consumption during steady-state flight but also provides a robust algorithmic guarantee for the safe operation of unmanned aerial vehicles in narrow or unknown complex spaces.

[0073] 5. Possesses strong industrial applicability and embedded deployment capabilities. The solution clearly defines the complete mapping link from image pixel tensors to airborne edge computing platform operations and then to the electronic speed controller's physical pulse width modulation signal. This end-to-end closed-loop design ensures that the algorithm can be directly transformed into actual physical electromechanical actuation, possessing extremely high industrial manufacturing value and technological feasibility. Attached Figure Description

[0074] Figure 1 A schematic diagram of the complete closed-loop execution flow of the integrated control algorithm proposed in this invention;

[0075] Figure 2 A schematic diagram of the hardware and software physical structure and heterogeneous module deployment architecture of the present invention;

[0076] Figure 3 A schematic diagram of the structural topology of the multi-scale hole feature extraction network and adaptive recalibration mechanism of this invention. Detailed Implementation

[0077] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0078] The core technical idea of ​​this invention lies in completely breaking down the technical barrier of the separation between the perception module and the control module in traditional unmanned aerial vehicle systems. Based on the theoretical design of the aforementioned core algorithm modules, this invention constructs a closed-loop joint optimization framework for directly constraining the generation of control laws from visual features. (See attached...) Figure 1 The process structure shown indicates that the overall adaptive closed-loop control method of the system specifically includes the following sequentially connected execution steps.

[0079] Step 1: Continuous image acquisition and alignment with time series reference.

[0080] The aircraft acquires continuous image sequences using an onboard global shutter vision sensor in complex operating environments. The system performs physical preprocessing, such as distortion correction, on the raw images, transforming them into standard tensors that can be resolved by a computer. Simultaneously, the system assigns an absolute timestamp to each frame of the image, strictly aligned with the main frequency of the flight control hardware, thereby constructing a strictly ordered continuous temporal feature source, laying the data foundation for subsequent motion blur filtering.

[0081] Step 2: Multi-scale hole feature extraction and adaptive recalibration.

[0082] The system inputs standard image tensors in parallel into a multi-branch network containing basic convolutional layers and dilated convolutional layers with different dilation rates. This network simultaneously captures the local texture and wide-area spatial structure of the target without increasing computational load. Subsequently, the system utilizes a channel weight generation mechanism to evaluate the response importance of features at each scale online, performs adaptive feature recalibration operations, and outputs a high-order spatial perception matrix that effectively suppresses redundant background noise.

[0083] Step 3: Dynamic spatiotemporal evolution modeling and self-attention fusion.

[0084] To overcome the perceptual abrupt changes that are easily caused by single-frame vision during violent UAV maneuvers or when the target encounters short-term occlusion, the system concatenates multi-frame spatial perception matrices in a temporal sequence. A dynamic multi-head self-attention mechanism with a fusion time decay penalty factor is introduced to force the network to forget long-standing features according to real physical laws and assign higher weights to adjacent temporal sequences, thereby extracting the target motion evolution trend with strong spatiotemporal consistency and generating high-dimensional spatiotemporal fusion features.

[0085] Step 4: Visual-dynamic coupling correction and state estimation.

[0086] The system constructs a multilayer perceptron network as the residual regression head, directly mapping the aforementioned high-dimensional spatiotemporal fusion features into the evolutionary deviation compensation amount of the UAV's twelve-DOF physical state. This is then deeply injected into the prior predicted state derived from rigid body mechanics mechanisms. The instantaneous observational nature of depth vision is used to correct the cumulative integral drift of the physical model, outputting a high-precision and high-confidence posterior state estimation benchmark.

[0087] Step 5: Error Adaptive Model Predictive Control Optimization.

[0088] The system sets the high-precision posterior state estimate as the initial state and constructs a multivariate rolling optimization cost function that includes state tracking error penalties and control increment penalties. The system dynamically adjusts the state weight matrix online based on the L2 norm of the current track deviation error, and solves for the optimal nominal control sequence that balances rapid response during large maneuvers and smooth hovering under the hard constraint of strictly satisfying the physical speed limits of the airborne actuators.

[0089] Step Six: Nonlinear disturbance observation compensation and underlying electromechanical closed-loop execution.

[0090] The system utilizes a nonlinear robust disturbance observer with a forgetting factor memory mechanism to capture external transient wind fields and unmodeled aerodynamic dynamics. The lumped disturbance estimate is feedforwarded to the first term of the nominal control sequence to generate the final closed-loop control command vector. A control distribution unit deployed within an airborne heterogeneous edge computing platform decouples this torque command into desired rotational speed signals for each rotor. These signals are then transmitted to the underlying electronic speed controller via pulse width modulation (PWM) protocol, precisely driving the brushless DC motor to generate corresponding aerodynamic thrust. Ultimately, this achieves holographic adaptive closed-loop flight from visual perception to physical electromechanical actuation.

[0091] The overall system structure of this invention aims to achieve a high degree of coupling and low-latency flow from front-end environmental perception to back-end physical and electromechanical actuation. (See attached diagram.) Figure 2 The module association logic shown indicates that the heterogeneous hardware and software architecture of this system specifically includes the following core modules.

[0092] Global shutter vision sensor: A physical hardware device deployed at the front or bottom of the unmanned aerial vehicle (UAV) to overcome the jelly effect caused by high-frequency vibrations and to acquire high-quality continuous images of the flight environment in real time.

[0093] Image sequence preprocessing module: Deployed at the front-end interface layer of the airborne edge computing node, it is used to perform camera intrinsic parameter calculation and distortion correction on the data input from the underlying image hardware, and to perform hardware-level high-precision timestamp alignment operation.

[0094] Multi-scale dilated feature extraction module: Deployed within the deep learning acceleration engine of the onboard graphics processing unit. This module includes standard convolution and parallel dilated convolution computation units with different dilation rates, and simultaneously extracts near-range details and far-range large-scale spatial environment information through an adaptive scale recalibration mechanism.

[0095] Dynamic Spatiotemporal Self-Attention Module: Synchronously deployed within the airborne graphics processing unit. This module, based on a multi-head self-attention network array with a time decay penalty factor, performs correlation operations on spatial features from multiple historical frames, filters out instantaneous perceptual abrupt changes caused by target physical occlusion, and outputs a high-dimensional spatiotemporal evolution feature matrix.

[0096] Visual-dynamic coupling estimation module: the core fusion hub bridging the visual perception layer and the physical dynamics layer. This module uses a multilayer perceptron network to transform high-dimensional visual features into corrected residuals of rigid body physical states, and dynamically injects them into a pre-set nonlinear flight dynamics model, outputting high-confidence posterior state pose estimates in real time.

[0097] Error Adaptive Predictive Control Module: Deployed within a high-frequency real-time operating system thread of the main control computing core. This module constructs a rolling optimization objective function based on the posterior state estimation benchmark and utilizes an online quadratic programming solver to calculate the nominal optimal control sequence with error adaptive adjustment capability within strict electromechanical hard constraints.

[0098] Nonlinear Robust Disturbance Observation Module: This module operates as a compensation bypass for the control system. It compares physical measurements with nominal model predictions and uses a nonlinear activation function with a forgetting factor to estimate external aerodynamic wind field disturbances in real time, then outputs a feedforward compensation signal to the predictive control unit.

[0099] The underlying electromechanical closed-loop execution module is the system's final physical actuation terminal. This module receives the compensated final closed-loop control torque command and decouples it into a pulse-width modulated digital signal with a specific duty cycle through the onboard control distribution unit. This signal is then transmitted at high frequency via a serial bus to the electronic speed controller hardware distributed in each rotor arm, precisely controlling the three-phase inverter current of the brushless DC motor stator coil, ultimately converting it into aerodynamic lift to drive the UAV to perform high-maneuver trajectory following or stable hovering. All modules are tightly connected sequentially via a high-speed system bus and a shared video memory interface, collectively forming an adaptive closed-loop system with both robust perception and high-maneuverability execution capabilities.

[0100] The multi-scale cavity feature extraction module of this invention adopts a highly generalized parallel multi-branch network topology. This structure abandons the limited configuration of the traditional single fixed receptive field and aims to solve the problem of missed detection caused by drastic changes in the scale of near and far targets during complex operations of unmanned aerial vehicles through collaborative perception of different spatial scales and adaptive feature recalibration mechanism.

[0101] Combined with appendix Figure 3 The network structure topology logic shown, input image tensor First, the data is synchronously distributed to multiple parallel convolutional processing branches. To maximize the scope of protection, the parallel multi-branch structure of this invention includes at least one basic standard convolutional branch and at least one convolutional branch with a specific dilatation rate.

[0102] The first processing layer on the left side of the structure diagram is the basic standard convolutional branch. This branch uses a pre-defined basic size convolutional kernel, and its core physical function is to accurately extract the local high-frequency texture and fine contour features of targets within the UAV's close-range field of view. After processing by this layer, the first feature matrix is ​​output. .

[0103] The processing links located on the sides and right of the structure diagram are additional sets of dilated convolutional branches. Each additional branch is configured with a different specific dilation rate to losslessly expand the equivalent receptive field. Based on the physical evolution of the network, let the dilation rate be... The kernel size is The equivalent receptive field calculation formula for any dilated convolution processing branch in the structure diagram is defined as follows:

[0104]

[0105] In a preferred embodiment, the system constructs two independent additional branches containing a first porosity and a second porosity. This differentiated porosity configuration allows the network to cover spatial structural information at medium and long distances, respectively, thereby simultaneously outputting a second feature matrix. With the third characteristic matrix .

[0106] After completing the independent feature extraction of each branch, such as Figure 3 As shown in the lower half of the structure, the system performs an initial feature concatenation operation. This operation concatenates the multi-scale feature maps output from all parallel branches along the channel dimension to construct a baseline fused feature tensor, the mathematical expression of which is:

[0107]

[0108] Next, the data flow enters the newly added core functional block in the structure diagram, namely the adaptive scale recalibration network. This network module first uses a spatial global average pooling mechanism to compress the complex two-dimensional spatial pixel distribution within each scale channel into a one-dimensional channel descriptor vector with a global perspective. The mathematical formula for this pooling compression process is defined as follows:

[0109]

[0110] Subsequently, the descriptor vector passes through a two-layer nonlinear fully connected weight generation module to calculate the optimal scale importance weight vector for the current dynamic changes in the environment. The formula for calculating the feedforward activation is defined as follows:

[0111]

[0112] Finally, element-wise multiplication is performed at the end of the structure graph, inversely activating the aforementioned adaptive weight vector into the initially concatenated baseline fusion feature tensor. The final output is a high-order spatially perceptual feature matrix. The calculation formula is defined as follows:

[0113]

[0114] This operation achieves feature amplification at the scale of high-value key targets and effective suppression of redundant environmental background scales at the physical level. The output high-order spatial perception feature matrix will directly flow into the subsequent dynamic spatiotemporal self-attention module, providing it with a highly robust spatial observation benchmark.

[0115] Example

[0116] A method for visual perception and adaptive flight control of unmanned aerial vehicles based on a multi-scale attention spatiotemporal fusion network includes the following steps:

[0117] S1. Continuous image acquisition. Real-time image sequences of the flight environment are acquired through airborne vision sensors and converted into a digital matrix that can be processed by a computer, serving as the input basis for subsequent perception networks.

[0118] S2. Perform multi-scale dilated convolution feature extraction. This scheme constructs a feature extraction network with multiple parallel convolutional branches, each containing at least one standard convolutional branch and at least one dilated convolutional branch. The standard convolutional branch uses standard convolutional layers to capture local fine texture features of the image. The dilated convolutional branch uses dilated convolutional layers with a specific dilation rate, effectively expanding the equivalent receptive field while maintaining a relatively constant computational parameter scale. When multiple dilated convolutional branches exist, each branch uses a different dilation rate. By concatenating and fusing the feature maps obtained from all parallel branches along the channel dimension, synchronous representation of flight target and environmental structure information at different spatial scales is achieved.

[0119] S3. Implement spatiotemporal attention fusion modeling. The extracted multi-frame feature sequences are mapped to a sequence feature matrix, and temporal position encoding is introduced to preserve the temporal order between image frames. By constructing a multi-head self-attention mechanism, the correlation between the query matrix, key matrix, and value matrix is ​​calculated, thereby dynamically extracting the temporal evolution patterns and motion trends in the image sequence. This module can effectively filter out transient perception noise and enhance the system's state preservation capability under short-term target occlusion.

[0120] S4. Conduct state estimation updates coupled with vision and flight dynamics. This scheme constructs a nonlinear state-space model of the unmanned aerial vehicle (UAV) and outputs residual estimates of the flight state vector in real time through a perception network. The residual terms obtained from visual observations are compensated into the prior states predicted based on the flight dynamics equations, thereby achieving fusion correction of state variables such as the UAV's attitude and velocity. This design utilizes visual information to offset the uncertainty errors generated during dynamic modeling in real time, ensuring the accuracy of state estimation.

[0121] S5. Perform adaptive predictive control optimization. Based on the fused state estimates, construct an objective function optimization problem in the finite time domain. Introduce an adaptive error weight adjustment factor into the predictive control framework to dynamically adjust the state weight matrix online according to the magnitude of the current tracking error. Obtain the optimal control sequence by solving a quadratic programming problem to ensure that the aircraft can respond quickly and smoothly to the preset trajectory commands.

[0122] S6. Implement disturbance observation compensation and closed-loop execution. The disturbance observer estimates external environmental wind field disturbances and unmodeled dynamic terms in real time online, and then compensates these disturbance estimates into the predictive control inputs to generate the final control command. This command drives the flight control execution module to adjust the power system speed via a data interface, ultimately forming a complete closed-loop control from environmental perception to precise execution.

[0123] A visual perception and adaptive flight control system for unmanned aerial vehicles based on a multi-scale attention spatiotemporal fusion network includes the following modules:

[0124] (a) Image Continuous Acquisition and Preprocessing Module

[0125] During the hardware data acquisition phase, in order to overcome the motion blur and rolling shutter effect of images caused by high-speed maneuvering of unmanned aerial vehicles or nonlinear wind disturbances, the system prioritizes the use of image acquisition equipment with a global shutter mechanism.

[0126] Before entering the feature extraction network, the acquired raw images need to undergo digital preprocessing. First, a camera intrinsic parameter matrix and distortion coefficient model are established to correct radial and tangential distortion in the acquired raw images. Then, the corrected continuous image is converted into a standard digital matrix that can be processed by a computer, with the input image tensor set to... Here Represents the set of real numbers, indicating that each pixel feature element in this tensor takes a real number value. The number of pixels representing the spatial height of the image. The number of pixels representing the spatial width of the image. The number of feature channels represents the image. In typical vision application implementations, the number of feature channels is usually set to three to represent the three primary colors: red, green, and blue.

[0127] Meanwhile, to provide a rigorous time reference for subsequent temporal dynamic modeling and state estimation fusion, the system assigns an absolute timestamp based on a high-precision hardware clock to each frame of the input image tensor. This timestamp is strictly aligned with the data sampling frequency of the airborne flight control system at the hardware level, thereby constructing a strictly ordered continuous time frame feature sequence.

[0128] (II) Multi-scale dilated convolution feature extraction module

[0129] The system introduces the preprocessed input image tensor into a feature extraction network with multiple parallel convolutional branches. To maximize the scope of rights protection and algorithm adaptability, the multi-branch parallel network structure includes at least one standard convolutional branch and at least one dilated convolutional branch.

[0130] In the specific implementation, the first branch is constructed as a standard convolutional layer, using a pre-defined base convolutional kernel, specifically designed to extract local fine texture features of the input image. Let the input image tensor be... The output feature map after processing by this standard convolutional branch is denoted as... .

[0131] To effectively expand the network's equivalent receptive field and avoid the spatial resolution degradation and loss of detail caused by traditional pooling operations, the system constructs additional branches containing one or more parallel dilated convolutional layers. In implementations with multiple dilated convolutional branches, each branch strictly employs a specific dilation rate. Let the dilation rate be... The kernel size is The formula for calculating the equivalent receptive field of a single dilated convolutional layer is defined as follows:

[0132]

[0133] By configuring differentiated dilatational parameters, the newly added parallel branches can respectively cover medium-scale and large-scale spatial structure information. Assuming the system includes several dilated convolutional branches in addition to standard convolution, their output feature maps can be denoted as follows: to , here the variable This represents the total number of convolutional branches in the parallel network. In a preferred embodiment, the base convolutional kernel size is set to three, and the system includes two additional dilated convolutional branches with dilation rates of two and four, respectively, thereby significantly expanding the equivalent receptive field of feature extraction while maintaining a substantially constant computational parameter size.

[0134] After obtaining the initial feature maps of each parallel branch, this module further introduces an adaptive scale feature recalibration mechanism to address the problem of fixed feature weights at different scales caused by the traditional direct splicing method.

[0135] First, the system performs initial concatenation and stitching of the multi-scale feature maps output by all parallel branches along the channel dimension to obtain the baseline fused feature tensor. Subsequently, to obtain the global response distribution of channels at each scale, the system performs a global average pooling operation on the spatial dimension of the baseline fused feature tensor, compressing the two-dimensional features containing rich spatial details into a one-dimensional channel descriptor vector. The mathematical formula for this compression process is defined as follows:

[0136]

[0137] In the formula The first one in the channel descriptor vector Scalar values ​​for each channel, and These represent the spatial height and width of the feature map, respectively. The pixel activation value represents the baseline fused feature tensor in a specific spatial coordinate system.

[0138] Next, the system constructs a weight generation module comprising a two-layer fully connected network and a nonlinear activation function. This module uses the acquired channel descriptor vectors to calculate the adaptive importance weights of features at each scale, defined by the following formula:

[0139]

[0140] In this formula, This represents the final scale weight vector. and This represents the weight matrix that the network model can learn during training. (Symbols) Represents the linear rectified activation function, used to enhance the nonlinear fitting ability of the network. (Symbol) This represents the normalized activation function, used to constrain the output weight scalar to the real number range of zero to one.

[0141] Finally, the system performs element-wise multiplication of the generated scale weight vector with the initial baseline fused feature tensor to complete the adaptive recalibration of multi-scale features. The final output feature tensor formula is defined as follows:

[0142]

[0143] Symbols in the formula This represents the element-wise multiplication operation of the feature channel dimension. Through this adaptive recalibration mechanism, the system can intelligently suppress background noise at redundant scales based on the dynamic changes in the current flight environment and target distance, and significantly enhance the feature representation capability at the key target scale, providing high-quality spatial observation data for subsequent temporal dynamic modeling.

[0144] (III) Spatiotemporal Self-Attention Fusion Module

[0145] The system first serializes the fused features from multiple consecutive time steps output by the multi-scale dilated convolution feature extraction module. Let the total number of historical image frames involved in the temporal modeling be... The one-dimensional feature dimension after flattening the single-frame feature tensor is: The constructed feature matrix of the input sequence is denoted as... Symbol here Represents the set of real numbers. Parameters The value is preset based on the video memory capacity of the airborne computing platform and the real-time requirements of the system tasks.

[0146] To enable the network model to recognize the logical order of image frames, the system superimposes a learnable temporal position encoding matrix onto the sequence feature matrix. Unlike basic sine and cosine fixed-position coding, the position coding matrix parameters in this scheme are updated with gradients through backpropagation during the offline training phase of the network to adapt to the specific flight kinematic evolution of the UAV.

[0147] To avoid the technical flaw of the basic self-attention mechanism being overly computationally uniform and neglecting the physical law of time decay, this scheme constructs a dynamic multi-head self-attention mechanism that incorporates a time decay penalty factor. Let the total number of heads in the multi-head attention mechanism be... For the first For each attention point, the system constructs a query matrix. Key matrix and value matrix Its linear mapping generation formula is defined as follows:

[0148]

[0149]

[0150]

[0151] In the formula , and All of these are linear mapping weight matrices that can be learned within the attention head.

[0152] Considering that in the actual flight dynamics of UAVs, the closer the historical state is to the current moment, the greater its reference value for the current estimate, the system innovatively introduces a time decay penalty matrix. The first in this matrix Line 1 The formula for calculating the element value of a column is defined as follows:

[0153]

[0154] In this formula, the variable and These represent the time step indices in the sequence feature matrix. Parameters This is a non-negative time decay adjustment coefficient. In a preferred embodiment of actual deployment, this adjustment coefficient... It can be set as a fixed constant based on the prior experience of flight control experts, or set as an adaptive scalar parameter that can be dynamically learned in the network model, specifically for controlling the rate at which the system forgets information from long ago.

[0155] Based on the above definition, the formula for calculating the attention distribution with the fusion time decay penalty factor is upgraded to:

[0156]

[0157] Symbols in the formula Represents the matrix transpose operation, parameter The key matrix feature dimension in the single-head attention mechanism is used to numerically scale the dot product result to prevent overflow. By superimposing the temporal penalty matrix onto the scaled correlation matrix, the system can enforce constraints that conform to the laws of time physics, assigning higher attention weights to nearby time frames, thereby effectively filtering out redundant features that are old and may contain severe pose noise.

[0158] Ultimately, the system will... The outputs of each attention head are concatenated along the feature dimension and then aggregated through a fully connected feedforward network to output the final spatiotemporal fusion feature sequence matrix. This feature matrix not only contains multi-scale spatial structure information but also deeply integrates the kinematic evolution trend of the target, providing high-precision and highly robust continuous observation data for the subsequent vision-flight dynamics coupled state estimation module.

[0159] (iv) Visual-dynamic coupled state estimation

[0160] The system first constructs a twelve-dimensional nonlinear state-space model of the unmanned aerial vehicle (UAV) to characterize its kinematic and dynamic evolution in three-dimensional space. The system state vector is set as follows: The specific components are defined as follows: The first three components represent the three-axis position coordinates of the aircraft in the world coordinate system; the next three components represent the three-axis velocities; the following three components represent the Euler angle attitude of the aircraft (roll angle, pitch angle, and yaw angle); and the last three components represent the three-axis angular velocities in the aircraft's body coordinate system. The state evolution process of the system is described by a system of first-order nonlinear differential equations, and its discretized nominal dynamic prediction model is expressed as:

[0161]

[0162] In this formula, the variable Represents the posterior state based on the previous time step. With control input commands The predicted prior state vector at the current time step. (Function) This represents a nonlinear transfer function constructed based on rigid body dynamics, which includes known physical constraints such as gravity compensation, rotor thrust mapping, and aerodynamic drag of the aircraft.

[0163] To address the technical challenges of accurately characterizing environmental wind disturbances, motor nonlinear dead zones, and changes in fuselage aerodynamic characteristics using nominal dynamic models, this module introduces a visual residual regression head based on deep neural networks. This regression head receives a high-dimensional spatiotemporal fusion feature matrix output from a spatiotemporal self-attention fusion module as input and predicts the evolutionary deviations of the flight state in real time.

[0164] The specific mapping process of the visual residual regression head is implemented through a multilayer perceptron structure. The system first compresses the feature dimension of the spatiotemporal fusion feature matrix. Then, the compressed feature vectors are sequentially input into a regression network containing at least two fully connected layers. Each fully connected layer is followed by a non-linear activation function and a dropout layer to improve the model's generalization ability and prediction stability. Assume that the output dimension of the final fully connected layer is consistent with the state vector dimension, i.e., the mapping function is denoted as... The residual estimation vector output by the visual network. The calculation formula is defined as follows:

[0165]

[0166] In the formula This represents the spatiotemporal fusion features output by the aforementioned modules. Physically, this residual estimation vector represents the amount of prediction error compensation caused by unmodeled dynamic terms and external environmental disturbances at the current moment.

[0167] Ultimately, the system achieves deep coupling updates of state estimation through a residual injection mechanism. The final posterior state estimate at the current time step is... The calculation formula is defined as follows:

[0168]

[0169] Through the above technical solution, the system achieves closed-loop joint optimization of perceived information and flight dynamics constraints. The innovation of this coupling mechanism lies in its approach: instead of simply weighted fusion of multi-sensor data, it utilizes visual features with temporal modeling capabilities as a nonlinear compensation term to dynamically correct the predicted trajectory of the physical model in real time. This design effectively eliminates the state estimation lag caused by the excessive reliance of traditional filtering algorithms on noise statistical characteristics, thus providing a high-frequency and high-confidence feedback reference signal for the subsequent adaptive predictive control module. In a practical industrial-grade implementation, this module runs on an onboard high-performance computing unit, ensuring that visual features directly constrain the physical pose evolution of the aircraft through end-to-end differential transmission.

[0170] (v) Adaptive Predictive Control Algorithm

[0171] The system uses the posterior state estimate obtained through deep coupling of vision and dynamics as the initial state of the system at the current moment. The prediction step size of the system is set to... Control step size is Within each control cycle, the system constructs a quadratic cost function that includes penalties for state tracking error, control input increment, and terminal state. The mathematical expression of this optimized cost function is defined as follows:

[0172]

[0173] In the formula This represents the total cost function value at the current moment. (Variable) This represents the state tracking error vector of the system in the future predicted sequence, i.e., the difference between the preset reference trajectory and the model predicted trajectory. (Variable) Representing the control input increment vector, the incremental penalty is introduced to limit high-frequency actions of the actuators, thereby smoothing the physical actions of the flight control system. Matrix The matrix represents a fixed control increment weight penalty positive definite matrix. This represents the terminal state penalty matrix, used to ensure the asymptotic stability of the closed-loop system.

[0174] To address the technical shortcomings of fixed weight matrices leading to sluggish system response or excessive overshoot during nonlinear high-maneuver flight, this solution innovatively introduces an error adaptive weight adjustment factor. The system adjusts the state error weight matrix... The design is an adaptive matrix that dynamically changes with the L2 norm of the current tracking error, and its update formula is defined as:

[0175]

[0176] The matrix here This represents the initial state weight matrix. This matrix is ​​a positive definite symmetric matrix, and during the initialization phase of system deployment, it is tuned offline based on the prior experience of flight control experts, maintaining fixed base values. Parameters This represents a non-negative adaptive adjustment coefficient, specifically used to control the sensitivity of the weights to changes in error. The specific value of this adjustment coefficient is determined through a comprehensive deduction of the maximum allowable overshoot of the unmanned aerial vehicle's dynamic system and the theoretical closed-loop stability margin.

[0177] The physical logic behind this adaptive mechanism is that when the unmanned aerial vehicle (UAV) deviates significantly from the target trajectory, causing a substantial increase in the error L2 norm, the system automatically amplifies the state penalty weight, forcing the UAV to converge quickly to the preset trajectory with more aggressive maneuvers. When the UAV approaches the target and hovers in a steady state, the tracking error decreases, and the state penalty weight automatically drops, effectively avoiding high-frequency oscillations in the underlying control commands.

[0178] In solving the aforementioned optimization problem, the system simultaneously introduces nonlinear hard constraints targeting the physical limits of the airborne actuators. The system sets limits on motor speeds and maximum Euler angles to ensure that the optimal control sequence is absolutely safe and executable in the real physical domain. Finally, the system utilizes an online quadratic programming solver deployed within the airborne computing platform to perform a rolling solution to the constrained optimization problem within each high-frequency sampling period. The first element of the optimal control sequence is extracted as the nominal control command for the current moment and passed to the subsequent disturbance observation and physical execution modules.

[0179] (vi) Disturbance Observation and Compensation

[0180] Traditional model predictive control relies heavily on the accuracy of the nominal dynamics model. When faced with unpredictable aerodynamic effects and sudden gusts in the real physical world, it is prone to steady-state control errors and even system divergence. While conventional linear disturbance observers can be used to estimate external disturbances, they are extremely sensitive to high-frequency noise in the perception layer. The inventiveness and non-obviousness of this invention lie in using the highly spatiotemporally consistent posterior state estimate output by the aforementioned spatiotemporal self-attention fusion module, which has filtered out instantaneous perceptual mutations, as the baseline input to the observer. This coupling mechanism, which binds the temporal filtering characteristics of depth visual features to the depth of physical disturbance observation, fundamentally prevents malfunctions in the control actuators caused by visual perception anomalies or short-term target occlusion.

[0181] The system first defines the state prediction error vector. Its physical meaning is the posterior state estimate after visual-dynamic coupling. Compared with the nominal flight dynamics prediction value The state-space vector difference between them. This error term represents the true deviation of the system from the influence of unmodeled dynamics and external wind disturbances, and its calculation formula is:

[0182]

[0183] To accurately capture high-frequency wind disturbances and suppress observer output chattering, this scheme constructs a nonlinear robust disturbance observer with a forgetting memory mechanism. The lumped disturbance estimation vector at the current moment is set as... Its discretized recursive update formula is defined as follows:

[0184]

[0185] The variables in the above formula This represents the historical disturbance estimation vector from the previous control cycle. Parameters The forgetting factor, in diagonal matrix form, is used to adjust the system's memory decay weight for historical perturbation states, thereby enabling the observer to quickly adapt to time-varying wind field environments. Parameters This is the linear observation gain matrix, used to ensure the basic dynamic response speed that guarantees the convergence of observation errors. Parameters For nonlinear robust gain matrix, the function The activation function is a hyperbolic tangent nonlinear function. By introducing this nonlinear constraint term, the system can provide strong compensation gain over a large error range and maintain an extremely smooth feedback curve in the steady-state range where the error is close to zero, thus effectively eliminating the high-frequency control chattering problem that is easily caused by traditional sign functions. To meet the requirement of full technical disclosure, the above three sets of diagonal gain matrices were all offline tuned and configured during the actual flight test and parameter tuning phase of the unmanned aerial vehicle, based on the distribution characteristics of the aircraft's overall mass inertia matrix and the desired wind resistance rating.

[0186] After obtaining a high-precision lumped disturbance estimation vector, the system performs feedforward compensation correction at the output of the predictive control. Let the first term of the nominal optimal control sequence output by the aforementioned adaptive predictive control module be... The final output closed-loop control command vector The calculation formula is defined as follows:

[0187]

[0188] This matrix The disturbance input mapping matrix is ​​used to selectively extract the disturbance components corresponding to linear acceleration and angular acceleration from the twelve-dimensional state disturbance of the lumped disturbance estimation vector, and accurately convert them to the actuation mapping space of the same dimension as the flight control input.

[0189] To effectively meet the requirements of patent law regarding the industrial applicability and manufacturability of technical solutions, this invention hereby discloses in detail specific underlying hardware application embodiments. All the aforementioned high-dimensional visual feature extraction, multi-level spatiotemporal attention inference matrix multiplication and addition operations, and large-scale nonlinear programming solution algorithms are uniformly deployed on an onboard heterogeneous edge computing platform equipped with a high-performance graphics processing unit, such as the NVIDIA Jason series embedded computing motherboard.

[0190] The final closed-loop control command vector generated after feedforward compensation correction typically includes the desired total lift and the desired roll, pitch, and yaw moments in the aircraft's body coordinate system. The low-level control allocation unit within the edge computing platform uses a preset pseudo-inverse control allocation matrix to linearly decouple and map this command vector into the square of the desired target rotational speed for each independent powered rotor. Subsequently, the low-level hardware input / output coprocessor converts the target rotational speed into a digital pulse-width modulation signal with a specific duty cycle or a digital control protocol data frame with a strict timing verification mechanism. This physical electrical signal is sent in real-time to the electronic speed controller hardware of each brushless DC motor via the onboard serial communication bus. Based on this command, the electronic speed controller precisely controls the frequency and amplitude of the three-phase inverter current in the stator coil, thereby driving the propeller to generate precise physical aerodynamic thrust. Thus, the system proposed in this invention completely completes the adaptive integrated closed-loop control data flow from multi-scale visual perception acquisition and dynamic feature recognition calculation to the underlying physical response of complex aircraft attitudes.

[0191] To further improve the system's operational efficiency and applicability, this invention has the following expandable technical evolution directions based on the existing technical framework.

[0192] Firstly, in terms of multi-source sensing fusion, this system can further integrate data from multimodal sensors such as inertial measurement units, barometers, and ultrasonic radars. By constructing a higher-dimensional heterogeneous data fusion filter, it can safely take over the system when visual sensors fail under extreme conditions such as direct sunlight or complete darkness, further improving the reliability of all-weather state estimation.

[0193] Secondly, in terms of intelligent optimization strategies, deep reinforcement learning algorithms can be introduced to jointly tune the weight matrix of the cost function of model predictive control online and offline. Utilizing interactive training in a large-scale simulation environment, the aircraft can adaptively learn the optimal control law for specific complex tasks, thereby achieving a better dynamic balance between optimal energy consumption and optimal accuracy.

[0194] Furthermore, in terms of task collaboration scale, the single-machine closed-loop control framework proposed in this invention can be extended as a basic node to a distributed collaborative control system for UAV swarms. By introducing inter-machine communication links, each flight node can share spatiotemporal attention characteristics, enabling spatiotemporal conflict avoidance and collaborative target search in large-scale swarm operations.

[0195] Finally, regarding platform adaptability and versatility, the integrated perception and control logic proposed in this invention is not limited to multi-rotor aircraft platforms, but can also be smoothly migrated to various unmanned system hardware such as fixed-wing UAVs, unmanned vehicles, and autonomous underwater vehicles. By adjusting the dynamic compensation terms for different physical configurations, high-precision autonomous perception and motion control can be achieved across platforms and media, demonstrating broad prospects for technological promotion.

Claims

1. A visual adaptive control method for unmanned aerial vehicles (UAVs) based on multi-scale spatiotemporal networks, characterized in that, Includes the following steps: 1) Acquire real-time image sequences of the flight environment, perform distortion correction on them, and convert the corrected sequences into digital matrices; 2) A high-order spatially perceptual feature matrix is ​​obtained by performing feature extraction and adaptive recalibration on the digital matrix through a multi-branch parallel convolutional network; 3) Construct a sequence feature matrix from the high-order spatial perception feature matrix of multiple consecutive frames, and perform spatiotemporal fusion modeling through a dynamic multi-head self-attention mechanism that incorporates time decay penalty factors to generate high-dimensional spatiotemporal fusion features; 4) The high-dimensional spatiotemporal fusion features are mapped to the evolutionary deviation compensation amount of the flight state through the residual regression network, and the compensation amount is injected into the prior state predicted based on the flight dynamics equation to obtain the posterior state estimate. 5) Based on the posterior state estimate, the state weight matrix is ​​dynamically adjusted online according to the magnitude of the current tracking error using the error adaptive model predictive control algorithm, and the optimal nominal control sequence is solved under the condition of satisfying the physical constraints of the actuator. 6) External disturbances and unmodeled dynamics are estimated in real time using a nonlinear robust disturbance observer to generate lumped disturbance estimates, and feedforward compensation is performed on the optimal nominal control sequence to generate the final closed-loop control command.

2. The UAV visual adaptive control method based on multi-scale spatiotemporal networks according to claim 1, characterized in that, The multi-branch parallel convolutional network consists of parallel standard convolutional branches for acquiring local fine texture features and dilated convolutional branches for expanding the receptive field. The number of dilated convolutional branches is at least one, and when there are multiple dilated convolutional branches, the dilation rate of each branch is different.

3. The UAV visual adaptive control method based on multi-scale spatiotemporal networks according to claim 2, characterized in that, The equivalent receptive field of a single dilated convolution branch for: in, The kernel size is the convolution kernel size. This represents the void ratio.

4. The UAV visual adaptive control method based on multi-scale spatiotemporal networks according to claim 1, characterized in that, The adaptive recalibration includes the following steps: The multi-scale feature maps output from all parallel branches are initially concatenated and stitched along the channel dimension to obtain the baseline fused feature tensor. ; Perform a global average pooling operation on the spatial dimension of the baseline fused feature tensor to obtain a one-dimensional channel descriptor vector. ,Right now: in, For the channel descriptor vector, the first... Scalar values ​​for each channel, and These represent the spatial height and width of the feature map, respectively. The pixel activation values ​​of the baseline fused feature tensor in a specific spatial coordinate system; A weight generation module consisting of a two-layer fully connected network and a non-linear activation function is constructed, and the scale weight vectors of features at each scale are calculated using channel descriptor vectors. : in, and This refers to the weight matrix that the network model can learn during training. It is a linear rectified activation function. The normalized activation function; The generated scale weight vector is multiplied element-wise with the initial baseline fused feature tensor to obtain the final output high-order spatial perception feature matrix. : in, Element-wise multiplication operation representing the feature channel dimension.

5. The UAV visual adaptive control method based on multi-scale spatiotemporal networks according to claim 1, characterized in that, The dynamic multi-head self-attention mechanism with fusion time decay penalty factor includes the following steps: Let the total number of heads in the multi-head attention mechanism be . For the first Each attention point is used to construct a query matrix. Key matrix and value matrix Its linear mapping generation formula is: in, , and These are all linear mapping weight matrices that can be learned within the attention head. The input sequence feature matrix is ​​a combination of spatial features from multiple historical frames and temporal location coding. Construct the time decay penalty matrix , The total number of historical image frames involved in time-series modeling, the nth frame in this matrix Line 1 Column element values for: Among them, variables and These are the time step indices in the sequence feature matrix. This is a non-negative time decay adjustment coefficient; Attention distribution for calculating fusion time decay penalty factor : in, This is a matrix transpose operation. This refers to the feature dimension of the key matrix in a single-head attention mechanism. All The outputs of each attention head are concatenated and spliced ​​along the feature dimension, and then aggregated through a fully connected feedforward network to obtain the final spatiotemporal fusion feature sequence matrix.

6. The UAV visual adaptive control method based on multi-scale spatiotemporal networks according to claim 1, characterized in that, Step 4) includes the following steps: 4.1) Construct a twelve-degree-of-freedom nonlinear state-space model of the unmanned aerial vehicle. Its discretized nominal dynamic model is as follows: in, Based on the posterior state of the previous time step With control input commands The predicted prior state vector at the current moment is composed of the UAV's three-axis position, three-axis axial velocity, three-axis Euler angles, and three-axis angular velocity, and the function... This is a nonlinear transfer function constructed based on rigid body dynamics. 4.2) Construct a visual residual regression head based on deep neural networks to fuse high-dimensional spatiotemporal features. Mapped to residual estimation vector , Let the mapping function be denoted as; 4.3) Obtaining posterior state estimates through residual injection mechanism .

7. The UAV visual adaptive control method based on multi-scale spatiotemporal networks according to claim 6, characterized in that, The specific mapping process of the visual residual regression head is implemented through a multilayer perceptron structure, specifically as follows: First, the spatiotemporal fusion feature matrix is ​​compressed in terms of feature dimension. Then, the compressed feature vector is sequentially input into a regression network containing at least two fully connected layers. Each fully connected layer is followed by a nonlinear activation function and a dropout layer. Finally, the fully connected layers output a residual estimation vector with the same dimension as the state vector.

8. The UAV visual adaptive control method based on multi-scale spatiotemporal networks according to claim 1, characterized in that, The cost function of the error adaptive model predictive control algorithm is specifically as follows: Using the posterior state estimate obtained through deep coupling of vision and dynamics as the initial state of the system at the current moment, and setting the prediction step size as... Control step size is Within each control cycle, the quadratic cost function is: in, The total cost function at the current moment. For the state tracking error vector in the future prediction sequence, To control the input increment vector, For a fixed control increment weight penalty positive definite matrix, The terminal state penalty matrix, Here is the state error weight matrix. The initial state weight matrix, The adaptive adjustment coefficient is non-negative. This is the state tracking error vector at the current moment.

9. The UAV visual adaptive control method based on multi-scale spatiotemporal networks according to claim 1, characterized in that, Step 6) includes the following steps: 6.1) Calculate the posterior state estimate after visual-dynamic coupling. Compared with the nominal flight dynamics prediction value The difference between the state space vectors is used as the state prediction error vector. : 6.2) Construct a nonlinear robust perturbation observer with a forgetting memory mechanism, setting the lumped perturbation estimation vector at the current moment as... Its discretized recursive update formula is: in, This is the historical disturbance estimation vector from the previous control cycle. The forgetting factor is in diagonal matrix form. The linear observation gain matrix, It is a nonlinear robust gain matrix; 6.3) Estimation vector based on lumped disturbance Feedforward compensation correction is performed at the output of predictive control, that is, the first term of the nominal optimal control sequence is set to... The final output closed-loop control command vector for: in, It is a disturbance input mapping matrix used to extract linear acceleration and angular acceleration disturbance components from high-dimensional state disturbances and transform them into an actuation mapping space of the same dimension as the flight control input.

10. A UAV visual adaptive control system based on multi-scale spatiotemporal networks, characterized in that, It includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement the UAV visual adaptive control method based on a multi-scale spatiotemporal network as described in any one of claims 1-9 when the computer program is executed.

Images