Multi-modal data fusion method for low-altitude flight risk early warning

Abstract

The invention belongs to the technical field of low-altitude flight safety early warning, and provides a multi-modal data fusion method for low-altitude flight risk early warning. Comprising the steps of flight state data and meteorological data alignment processing, missing data filling, spatial feature extraction, spatial feature and flight state data fusion and space-time convolutional neural network and space-time diagram convolutional network analysis. According to the invention, the flight state and the meteorological data are fused, so that the prediction accuracy and reliability are improved; through the space-time convolutional neural network and the space-time diagram convolutional network, the space and time dependency relationship in the flight data and the meteorological data can be captured at the same time; through spatio-temporal data fusion, missing data can be accurately aligned and complemented, and the data quality and the real-time performance of the system are improved.

Description

Technical Field

[0001] The present invention relates to the field of low-altitude flight safety warning technology, and in particular to a multimodal data fusion method, system, electronic equipment and storage medium for low-altitude flight risk warning. Background Art

[0002] Low-altitude flight safety warning uses technical means and management measures to conduct real-time monitoring, risk assessment, and early warning of flight activities within low-altitude airspace, aiming to prevent and reduce the occurrence of flight accidents. Low-altitude airspace generally refers to airspace below 1,000 meters above the ground. Flight activities within this area include general aviation, drones, helicopters, and other aircraft. The diverse types of aircraft, complex environments, and significant influences from terrain and weather are characteristic of this area. Low-altitude flight safety risk assessment and early warning, as key components of ensuring the safety of low-altitude flight activities, have become increasingly important with the widespread use of drones and other low-altitude aircraft. In recent years, scholars both domestically and internationally have conducted extensive research in the field of low-altitude flight safety warning, focusing primarily on core technologies, hazard warning technologies, and policies, regulations, and economics. Within these core technology areas, sensor technology, communication technology, and cross-modal information fusion constitute the cornerstones of low-altitude flight safety warning. Multimodal data fusion, in particular, integrates data from multiple sensors, including radar, ADS-B, and optical equipment, enabling comprehensive perception and precise analysis of the low-altitude flight environment. Low-altitude flight risk warning technology, supported by core technologies, effectively reduces the probability of flight accidents through real-time monitoring, risk assessment, and early warning. Furthermore, policies, regulations, and economic factors play a crucial guiding and supporting role in the development and application of low-altitude flight safety warning systems. Countries have promoted the development and promotion of these technologies through the formulation of relevant policies and increased economic investment. These three intertwined aspects have collectively driven the continuous improvement and development of the low-altitude flight safety warning system.

[0003] However, in traditional UAV flight risk assessment systems, flight risk prediction often relies solely on single flight data or meteorological data, ignoring the complex interaction between flight status and meteorological environment, resulting in low prediction accuracy, especially under complex environmental conditions, and unable to fully consider the spatiotemporal relationship of multi-source data. Summary of the Invention

[0004] In order to overcome the shortcomings of the existing technology, the purpose of the present invention is to provide a multimodal data fusion method for low-altitude flight risk warning, which solves the key problem of fusing flight data and meteorological data during UAV flight.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A multimodal data fusion method for low-altitude flight risk warning, including:

[0007] Use unified timestamps to align flight status data and meteorological data;

[0008] Filling missing data in the aligned flight status data and the meteorological data using an interpolation method;

[0009] Extracting local wind field changes and turbulence intensity from the interpolated meteorological data to obtain spatial features;

[0010] fusing the spatial features and the interpolated flight status data to obtain multimodal data;

[0011] The multimodal data is spatiotemporally modeled using a spatiotemporal convolutional neural network and a spatiotemporal graph convolutional network to obtain a multimodal spatiotemporal feature representation.

[0012] Preferably, the flight status data and meteorological data are aligned using a unified timestamp, including:

[0013] Collect angular velocity data, linear acceleration data, motor speed data, current data, and voltage data from the onboard IMU and ESC to obtain hardware data;

[0014] Use navigation GNSS to collect latitude and longitude data, altitude data, and heading data to obtain positioning data;

[0015] Integrating the hardware data and the positioning data to obtain the flight status data;

[0016] Collecting wind speed data, wind direction data, temperature data, humidity data, and air pressure data to obtain the meteorological data;

[0017] Establish an absolute time scale; the expression of the absolute time scale is: t UTC =t week ×604800000+t tow(ms) ; where t UTC is the Coordinated Universal Time; t week is the current week number; t tow(ms) is the number of seconds in a week;

[0018] The local clocks of the flight status data and the meteorological data are mapped to the absolute time scale using a mapping formula; the mapping formula is expressed as: UTC =αc local +β, Where α is the drift coefficient; β is the offset; c local is the local clock; c k Record the local count of the rising edge of PPS for the device; t k is UTC time; ck The mean of t k The mean of

[0019] Setting a standardized time base according to the absolute time scale to obtain a main time axis;

[0020] The flight status data and the meteorological data are attached to the main time axis in a left outer join manner for alignment processing.

[0021] Preferably, the interpolation method is used to fill missing data in the aligned flight status data and the meteorological data, including:

[0022] When the missing value interval is less than a preset critical interval, the flight status data and the meteorological data are filled with missing data using linear interpolation;

[0023] When the fluctuation amplitude or nonlinear trend of the aligned flight status data and the meteorological data exceeds a preset standard range, filling missing data of the flight status data and the meteorological data using cubic spline interpolation;

[0024] When the missing value is less than 3 frames, the flight status data and the meteorological data are filled with missing data by using forward filling;

[0025] When the missing value is greater than 3 frames and less than 1 second, Kalman smoothing is used to fill the missing data of the flight status data and the meteorological data.

[0026] Preferably, filling missing data on the aligned flight status data and the meteorological data using an interpolation method further comprises:

[0027] Normalizing the interpolated flight status data and the meteorological data using a Z-Score formula;

[0028] A time decay factor is integrated into the normalized flight status data and the meteorological data.

[0029] Preferably, extracting the local wind field changes and turbulence intensity of the interpolated meteorological data to obtain spatial features includes:

[0030] Converting the interpolated longitude and latitude of the meteorological data into an ENU coordinate system;

[0031] Sampling the wind field data of the meteorological data using a 3×3×3 neighborhood as a sliding window to obtain a four-dimensional tensor;

[0032] The spatial features are obtained by performing spatial differential operators, turbulence gust index calculation, body coordinate projection and cross-modal splicing on the four-dimensional tensor.

[0033] Preferably, performing spatial differential operators, turbulence gust index calculation, body coordinate projection, and cross-modal splicing on the four-dimensional tensor to obtain the spatial features includes:

[0034] Calculating a first-order central difference of the four-dimensional tensor;

[0035] calculating a shear modulus of the four-dimensional tensor;

[0036] Calculating the horizontal divergence of the four-dimensional tensor;

[0037] calculating the vertical vorticity of the four-dimensional tensor;

[0038] Calculate the wind direction angle of the four-dimensional tensor; the calculation formula of the wind direction angle is:

[0039]

[0040] Wherein, Δθ is the wind direction angle; For h u Wind speed vector at altitude level; For h u -The wind speed vector at the Δu altitude layer;

[0041] The first-order central difference, the shear modulus, the horizontal divergence, the vertical vorticity, and the wind direction angle are integrated to obtain the spatial differential operator.

[0042] Preferably, performing spatial differential operators, turbulence gust index calculation, body coordinate projection, and cross-modal splicing on the four-dimensional tensor to obtain the spatial features includes:

[0043] calculating the turbulence intensity of the four-dimensional tensor;

[0044] calculating a gust factor of the four-dimensional tensor;

[0045] calculating a volume turbulent dissipation rate of the four-dimensional tensor;

[0046] The turbulence intensity, the gust factor, and the volume turbulence dissipation rate are integrated to obtain the turbulence gust index.

[0047] Preferably, performing a spatial differential operator, turbulence gust index calculation, body coordinate projection, and cross-modal splicing on the four-dimensional tensor to obtain the spatial feature further includes:

[0048] The spatial differential operator and the turbulence gust index are integrated to obtain direct extraction features.

[0049] The wind vector of the four-dimensional tensor is transferred to the body axis by using an attitude rotation matrix, and angle of attack correction and sideslip angle wind-attitude coupling energy calculation are performed on each sliding window in the four-dimensional tensor to obtain a joint interaction feature;

[0050] The directly extracted features and the joint interactive features are fused to obtain the spatial features.

[0051] Preferably, the multimodal data is subjected to spatiotemporal modeling using a spatiotemporal convolutional neural network and a spatiotemporal graph convolutional network to obtain a multimodal spatiotemporal feature representation, including:

[0052] Performing feature extraction on the multimodal data using the spatial convolution layer and the first temporal convolution layer of the spatiotemporal convolutional neural network, and fusing outputs of the spatial convolution layer and the first temporal convolution layer using the spatiotemporal fusion layer of the spatiotemporal convolutional neural network to obtain a first feature representation;

[0053] Performing feature extraction on the multimodal data using the graph convolution layer and the second temporal convolution layer of the spatiotemporal graph convolutional network, and fusing outputs of the graph convolution layer and the second temporal convolution layer using the spatiotemporal fusion layer of the spatiotemporal graph convolutional network to obtain a second feature representation;

[0054] The first feature representation and the second feature representation are fused to obtain the spatiotemporal feature representation.

[0055] The present invention discloses the following technical effects:

[0056] The present invention provides a multimodal data fusion method for low-altitude flight risk warning. By fusing flight status and meteorological data, it solves the defect of traditional methods that often rely on a single data source and realizes the joint modeling of multimodal data. By adopting spatiotemporal convolutional neural networks and spatiotemporal graph convolutional networks, it solves the defect of existing technologies that only focus on time series or spatial information and realizes the simultaneous capture of spatial and temporal dependencies in flight data and meteorological data. Through timestamp alignment and interpolation completion, the problems caused by time asynchrony and data inconsistency or missing in traditional methods are solved, and precise alignment and completion of missing data are achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1A schematic diagram of the multimodal data fusion process for low-altitude flight risk warning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0060] The purpose of this invention is to provide a multimodal data fusion method for low-altitude flight risk warning, which solves the key problem of fusing flight data and meteorological data during UAV flight.

[0061] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] Figure 1 A schematic diagram of a multimodal data fusion process for low-altitude flight risk warning provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the present invention provides a multimodal data fusion method for low-altitude flight risk warning, comprising:

[0063] Step 100: Align the flight status data and meteorological data using a unified timestamp;

[0064] Step 200: Filling missing data in the aligned flight status data and the meteorological data using an interpolation method;

[0065] Step 300: extracting local wind field changes and turbulence intensity from the interpolated meteorological data to obtain spatial features;

[0066] Step 400: fusing the spatial features and the interpolated flight status data to obtain multimodal data;

[0067] Step 500: Use a spatiotemporal convolutional neural network and a spatiotemporal graph convolutional network to perform spatiotemporal modeling on the multimodal data to obtain a multimodal spatiotemporal feature representation.

[0068] Specifically, flight status data and meteorological data are aligned using a unified timestamp, including:

[0069] Collect angular velocity data, linear acceleration data, motor speed data, current data, and voltage data from the onboard IMU and ESC to obtain hardware data;

[0070] Use navigation GNSS to collect latitude and longitude data, altitude data, and heading data to obtain positioning data;

[0071] Integrating the hardware data and the positioning data to obtain the flight status data;

[0072] Collecting wind speed data, wind direction data, temperature data, humidity data, and air pressure data to obtain the meteorological data;

[0073] Establish an absolute time scale; the expression of the absolute time scale is: t UTC =t week ×604800000+t tow(ms) ; where t UTC is the Coordinated Universal Time; t week is the current week number; t tow(ms) is the number of seconds in a week;

[0074] The local clocks of the flight status data and the meteorological data are mapped to the absolute time scale using a mapping formula; the mapping formula is expressed as: UTC =αc local +β, Where α is the drift coefficient; β is the offset; c local is the local clock; c k Record the local count of the rising edge of PPS for the device; t k is UTC time; c k The mean of t k The mean of

[0075] Setting a standardized time base according to the absolute time scale to obtain a main time axis;

[0076] The flight status data and the meteorological data are attached to the main time axis in a left outer join manner for alignment processing.

[0077] Specifically, filling missing data on the aligned flight status data and the meteorological data using an interpolation method includes:

[0078] When the missing value interval is less than a preset critical interval, the flight status data and the meteorological data are filled with missing data using linear interpolation;

[0079] When the fluctuation amplitude or nonlinear trend of the aligned flight status data and the meteorological data exceeds a preset standard range, filling missing data of the flight status data and the meteorological data using cubic spline interpolation;

[0080] When the missing value is less than 3 frames, the flight status data and the meteorological data are filled with missing data by using forward filling;

[0081] When the missing value is greater than 3 frames and less than 1 second, Kalman smoothing is used to fill the missing data of the flight status data and the meteorological data.

[0082] Furthermore, filling missing data in the aligned flight status data and the meteorological data using an interpolation method further includes:

[0083] Normalizing the interpolated flight status data and the meteorological data using a Z-Score formula;

[0084] A time decay factor is integrated into the normalized flight status data and the meteorological data.

[0085] Specifically, the local wind field changes and turbulence intensity of the interpolated meteorological data are extracted to obtain spatial features, including:

[0086] Converting the interpolated longitude and latitude of the meteorological data into an ENU coordinate system;

[0087] Sampling the wind field data of the meteorological data using a 3×3×3 neighborhood as a sliding window to obtain a four-dimensional tensor;

[0088] The spatial features are obtained by performing spatial differential operators, turbulence gust index calculation, body coordinate projection and cross-modal splicing on the four-dimensional tensor.

[0089] Furthermore, the spatial differential operator, turbulence gust index calculation, body coordinate projection and cross-modal splicing are performed on the four-dimensional tensor to obtain the spatial features, including:

[0090] Calculating a first-order central difference of the four-dimensional tensor;

[0091] calculating a shear modulus of the four-dimensional tensor;

[0092] Calculating the horizontal divergence of the four-dimensional tensor;

[0093] calculating the vertical vorticity of the four-dimensional tensor;

[0094] Calculate the wind direction angle of the four-dimensional tensor; the calculation formula of the wind direction angle is:

[0095]

[0096] Wherein, Δθ is the wind direction angle; For h u Wind speed vector at altitude level; For hu -Δu wind speed vector at altitude level;

[0097] The first-order central difference, the shear modulus, the horizontal divergence, the vertical vorticity, and the wind direction angle are integrated to obtain the spatial differential operator.

[0098] Specifically, the spatial differential operator, turbulence gust index calculation, body coordinate projection and cross-modal splicing are performed on the four-dimensional tensor to obtain the spatial features, including:

[0099] calculating the turbulence intensity of the four-dimensional tensor;

[0100] calculating a gust factor of the four-dimensional tensor;

[0101] calculating a volume turbulent dissipation rate of the four-dimensional tensor;

[0102] The turbulence intensity, the gust factor, and the volume turbulence dissipation rate are integrated to obtain the turbulence gust index.

[0103] Furthermore, performing spatial differential operators, turbulence gust index calculation, body coordinate projection, and cross-modal splicing on the four-dimensional tensor to obtain the spatial features further includes:

[0104] The spatial differential operator and the turbulence gust index are integrated to obtain direct extraction features.

[0105] The wind vector of the four-dimensional tensor is transferred to the body axis by using an attitude rotation matrix, and angle of attack correction and sideslip angle wind-attitude coupling energy calculation are performed on each sliding window in the four-dimensional tensor to obtain a joint interaction feature;

[0106] The directly extracted features and the joint interactive features are fused to obtain the spatial features.

[0107] Preferably, the multimodal data is subjected to spatiotemporal modeling using a spatiotemporal convolutional neural network and a spatiotemporal graph convolutional network to obtain a multimodal spatiotemporal feature representation, including:

[0108] Performing feature extraction on the multimodal data using the spatial convolution layer and the first temporal convolution layer of the spatiotemporal convolutional neural network, and fusing outputs of the spatial convolution layer and the first temporal convolution layer using the spatiotemporal fusion layer of the spatiotemporal convolutional neural network to obtain a first feature representation;

[0109] Performing feature extraction on the multimodal data using the graph convolution layer and the second temporal convolution layer of the spatiotemporal graph convolutional network, and fusing outputs of the graph convolution layer and the second temporal convolution layer using the spatiotemporal fusion layer of the spatiotemporal graph convolutional network to obtain a second feature representation;

[0110] The first feature representation and the second feature representation are fused to obtain the spatiotemporal feature representation.

[0111] Specifically, the core method of this embodiment is based on a spatiotemporal multimodal data fusion architecture and is implemented through the following steps:

[0112] Data collection and preprocessing: Align drone hardware data and meteorological data using a unified timestamp, and use interpolation methods to fill in missing data;

[0113] Spatial feature extraction: Extract spatial features such as local wind field changes and turbulence intensity from meteorological data and fuse them with flight status data;

[0114] Spatiotemporal modeling: Spatiotemporal modeling of multimodal data is performed through spatiotemporal convolutional neural networks (ST-CNN) and spatiotemporal graph convolutional networks (ST-GCN) to extract the complex spatiotemporal relationships between flight status and meteorological data.

[0115] Furthermore, data collection and preprocessing are performed. The core goal of the cross-modal feature fusion network is to process and fuse data from different sources, including flight status, meteorological data, airspace information, etc. These data types vary and contain different spatiotemporal characteristics and structured information. Therefore, effective feature extraction and fusion must be used to construct a multi-dimensional unified feature space to enhance the accuracy and real-time performance of the early warning system. This section involves:

[0116] Data collection

[0117] 1) Onboard IMU and ESC: angular velocity, linear acceleration, motor speed, current, and voltage. Sampling frequency: 50Hz to 200Hz. Hardware clock source: MCU clock and PPS synchronization.

[0118] 2) Navigation GNSS: latitude, longitude, altitude, and heading. Sampling frequency: 10 Hz. Hardware clock source: GNSS pulse-per-second.

[0119] 3) Meteorological data: wind speed, wind direction, temperature, humidity, and air pressure. Sampling frequency: 1Hz. Hardware clock sources: RTC and PPS.

[0120] Clock unification and drift correction

[0121] The counter of the drone's onboard IMU, the MCU in the ESC, and the RTC of the weather probe each have their own beats. To make their times consistent, this embodiment connects each device to GPS PPS (Pulse-Per-Second). When the rising edge of PPS arrives, each device will record the local count value c at that moment. k and GPSUTC time t kThis example uses linear fitting to map local counts to UTC:

[0122] 1) Establish an absolute time scale:

[0123] t UTC =t week ×604800000+t tow(ms)

[0124] t UTC is the Coordinated Universal Time (UTC) time, t week Represents the current week, t tow The number of seconds in the week.

[0125] 2) Local clock to UTC mapping:

[0126] For each device, record the local count c at the rising edge of PPS k Corresponding to UTCt k , perform linear fitting to obtain the drift coefficient α and offset β:

[0127]

[0128] Constructing the main timeline

[0129] Assume that the flight record starts from t0 to t N Finally, the fastest sampling rate is 50Hz (20ms period), which provides a standardized time base for subsequent data fusion and analysis. that is:

[0130]

[0131] Among them, t0 is the start time; t N is the end time; Δt is the time step.

[0132] Attach each data stream (IMU, ESC, wind speed, air pressure, etc.) to this main axis by left outer connection: if a data is missing at a certain moment, leave NaN in the table and record mask = 0 in the mask matrix.

[0133] Specifically, resampling and interpolation are used to complete the data. When aligning low-frequency data (such as 1Hz meteorological data) to the high-frequency main time axis (such as 50Hz), there will inevitably be missing data at many time points. In order to ensure that the data has a value at each time step, resampling and interpolation are required:

[0134] Low-frequency to high-frequency numerical interpolation

[0135] 1) Use linear interpolation for data with short time intervals

[0136]

[0137] y(t k ) is at time t k Interpolation, y(t i ) is time t i The known value of .

[0138] 2) Use cubic spline interpolation for data with large fluctuations or nonlinear trends

[0139]

[0140] Get the second-order derivative m i , where the segment function S i :

[0141] S i (t) = a i +b i (tt i )+c i (tt i ) 2 +d i (tt i ) 3

[0142] Among them, a i 、b i 、c i d i are all spline function coefficients.

[0143] Multi-step deletion repair

[0144] 1) Forward filling is used for short deletions (less than 3 frames)

[0145] y(t k )=y(t k-1 )

[0146] 2) Long deletions use Kalman smoothing (greater than 3 frames and less than 1 second)

[0147]

[0148] where z k is the observation vector, H is the observation matrix, represents the state estimate of all observations, P k|k-1 is the covariance matrix of state prediction, and R is the covariance matrix of observation noise.

[0149] 3) Unrecoverable (longer than 1s)

[0150] If the missing segment is >1s, set mask=0; ignore it according to the mask during model training.

[0151] Preferably, normalization and time decay weights are used. Data from different sources (such as temperature, wind speed, and motor speed) may have very different dimensions and numerical ranges. For example, the motor speed may be in the thousands or tens of thousands, while the temperature may be only in the tens. If these raw values ​​are directly input into the neural network, features with larger values ​​may dominate the learning process of the model, while features with smaller values ​​may be ignored. Z-Score normalization converts all features to a distribution with a mean of 0 and a standard deviation of 1, eliminating the dimensionality effect and making different features comparable in model training:

[0152] 1) Use Z-Score to unify the scale

[0153]

[0154] z i is the standardized value, μ is the mean, σ is the standard deviation, x i is the original value, and M is the number of data points.

[0155] 2) Time decay factor

[0156]

[0157] Input w into the model as an additional feature, t lastValid is the time of the last valid data point, and τ is the decay constant.

[0158] 3) Tensor output

[0159] Finally, we get a structured tensor:

[0160] M=[m tf ]∈{0,1} T×F

[0161] Where F is the number of features and M is the mask matrix.

[0162] After the above processing, cross-modal inputs are aligned at the same moment and frequency, laying a reliable data foundation for subsequent spatiotemporal convolution and graph network modeling.

[0163] Specifically, spatial feature extraction. This embodiment processes the three-dimensional meteorological field into spatial feature vectors that can be input into the deep learning model and couples them with the drone's body state at the same moment. The process is divided into six steps: coordinate unification, local wind field sampling, spatial differential operator, turbulence / gust index, body coordinate projection, and cross-modal splicing.

[0164] Furthermore, coordinates are unified. UAV flight trajectories and numerical weather forecast data are usually located in different coordinate systems. In order to be able to integrate and analyze this information, they must be converted into a unified coordinate system. Latitude and longitude are converted to the Earth Center Coordinate System (ECEF):

[0165]

[0166] a=6378137m,e 2 =0.006694

[0167] ECEF to ENU:

[0168] p enu =R ecef→enu (p ecef -p 0,ecef )

[0169]

[0170] Among them, p enu is the position vector in the ENU coordinate system, R ecef→enu is the rotation matrix from ECEF to ENU, p ecef is the position vector in the ECEF coordinate system.

[0171] After completion, the flight coordinates p u (t) = (e, n, u), the wind field grid and the terrain grid are all located on the same ENU cylinder, and the wind field, terrain, and UAV positions all fall in the same three-dimensional Cartesian coordinate system.

[0172] Specifically, local wind field sampling (3-DPatch construction). Single-point wind speed is difficult to reflect local phenomena such as wind shear and vortex. Therefore, this embodiment focuses on the position P of the drone at time t. u (t), take its 3×3×3 neighborhood in the three-dimensional numerical wind field, a total of 27 grid points, denoted as Forecast or measured wind field: U(e,n,u,t)=(u,v,w)(eastward, northward, vertical component). Assume that the UAV position P u (t), take 27 neighborhood (3×3×3) grid points, index (i,j,k). Use KD-Tree nearest neighbor search to ensure (Δs = grid spacing). Output 4D tensor

[0173]

[0174] In the case of uneven or discontinuous vertical distribution of wind speed data, the logarithmic wind speed profile is interpolated to the UAV height along the vertical direction.

[0175] Furthermore, spatial differential operators (shear, divergence, vorticity). While the direct wind speed and direction are important, the spatial variation of the wind field (i.e., differential characteristics) often reveals potential flight risks, such as wind shear and turbulence. First-order central difference. This embodiment uses the second-order central difference to calculate the first-order partial derivative of the central grid point (0, 0, 0):

[0176]

[0177] Δx=Δn=Δu=Δs

[0178] By calculating the wind speed difference between the center point and its adjacent points in the local wind field sampling, the rate of change of wind speed in each direction can be estimated.

[0179] Shear modulus:

[0180]

[0181] It measures the difference in wind speed between different altitudes and is a key indicator for determining vertical wind shear. Strong wind shear can have a drastic impact on the drone's attitude and lift.

[0182] Horizontal divergence and vertical vorticity:

[0183]

[0184] Horizontal divergence describes the degree to which air converges or diverges on a horizontal plane. If the divergence is greater than 0 (divergence), it may cause updrafts; if it is less than 0 (convergence), it may cause downdrafts. These vertical air currents have a significant impact on drone flight stability. Vertical vorticity, on the other hand, describes the degree of rotation of the air on a horizontal plane. Areas of high vorticity are often associated with small-scale eddies and turbulence risks.

[0185] Wind direction angle:

[0186]

[0187] The above formula calculates the difference in wind direction between different altitude levels. Drastic changes in wind direction with altitude (wind shear) are also a threat to flight safety.

[0188] Turbulence and gust indicators. In addition to the average state and spatial variation of wind, the pulsation characteristics of wind (i.e. turbulence and gusts) are also important factors affecting flight safety:

[0189] Turbulence intensity (TI) is used to describe the fluctuation amplitude of wind speed in a small window. In the sliding window W:

[0190]

[0191] Gust factor (GF), which is defined as the maximum wind speed within a small time window W:

[0192]

[0193] A GF greater than 1.4 generally indicates moderate to strong gusty winds.

[0194] Volume turbulent dissipation rate (ε), using Kolmogorov2 / 3 law:

[0195]

[0196] C≈0.5,L(=Δs)

[0197] Where L (=Δs) is the grid size, which is the minimum spatial scale for calculation; σ u is the velocity standard deviation, which reflects the turbulence intensity; ε is the volume turbulence dissipation rate, which indicates the dissipation rate of fluid energy per unit volume.

[0198] Body coordinate projection and cross-modal stitching: The wind felt by the drone is relative to its own motion and attitude. Therefore, it is necessary to transform the ambient wind vector into the drone's body coordinate system to more accurately analyze the impact of wind on the drone's attitude and control:

[0199] Wind vector to body axis (BodyFrame):

[0200] Posture rotation matrix R nb (φ,θ,ψ):

[0201] V body =R nb (U center '-V uav )

[0202] Among them, V uav is the UAV airspeed vector.

[0203] Joint interaction features:

[0204] Angle of attack correction:

[0205] Sideslip angle:

[0206] Wind-attitude coupling energy:

[0207] Fusion vector construction, the time sequence flight state: s(t) = [φ,θ,ψ,RPM 1..4 ,a x ,a y ,a z ,…] and spatial features: Concatenate by columns:

[0208]

[0209] Right now:

[0210]

[0211] Synchronous mask, confidence w(t). Finally, we get the spatiotemporal cross-modal input tensor

[0212]

[0213] This is used for subsequent spatiotemporal convolution / graph attention network modeling of flight risks.

[0214] Specifically, spatiotemporal modeling. Spatiotemporal modeling is a core step in multimodal data fusion, especially in drone flight risk assessment, where there is a strong spatiotemporal dependency between flight data and meteorological data. To fully capture these complex spatiotemporal interactions, this example proposes a modeling method based on spatiotemporal convolutional neural networks (ST-CNN) and spatiotemporal graph convolutional networks (ST-GCN). By simultaneously modeling the temporal and spatial dimensions, it extracts the potential relationships in multimodal data.

[0215] Furthermore, the spatiotemporal convolutional neural network architecture (ST-CNN) is a neural network architecture that combines spatial convolution and temporal convolution to process spatiotemporal data. It can extract features from both spatial and temporal dimensions and is particularly suitable for processing data with spatiotemporal dependencies such as wind speed, flight speed, and acceleration. It mainly consists of the following components:

[0216] 1) Spatial Convolution Layer: This layer is used to extract spatial features from meteorological data. For local wind field sampling data, the spatial convolution layer can identify wind speed variations in different geographic locations.

[0217] 2) Temporal Convolution Layer: This layer is used to extract features from time series data. Flight data has strong temporal dependencies, and the temporal convolution layer can effectively capture the dynamic changes in flight status.

[0218] 3) Spatio-Temporal Fusion Layer: In this layer, the features extracted by spatial convolution and temporal convolution are fused to form a spatio-temporal feature representation. This layer uses a specific fusion mechanism to integrate spatial and temporal features.

[0219] Specifically, the spatiotemporal convolution formula:

[0220] 1) Spatial convolution (2D): In the spatial convolution layer, the input wind field feature map undergoes a 2D convolution operation, that is, for the feature map X at position (i, j) (spatial) , the convolution operation can be expressed as:

[0221]

[0222] Among them, W k,l is the convolution kernel, and m is the size of the convolution kernel.

[0223] 2) Temporal convolution (one-dimensional): In the temporal convolution layer, the input flight data feature map undergoes a one-dimensional convolution operation, that is, for the feature map X at time step t (temporal) , the convolution operation can be expressed as:

[0224]

[0225] Among them, W k is the temporal convolution kernel, and n is the size of the convolution window.

[0226] 3) Spatiotemporal fusion: In the spatiotemporal fusion layer, the output features of spatial convolution and temporal convolution will be fused. Assume that the feature after spatial convolution is X (spatial) , the feature after time convolution is X (temporal) , then the space-time fusion can be expressed as:

[0227] X (spatio-temporal) =f(X (spatial) ,X (temporal) )

[0228] Where f is the fusion operation.

[0229] 4) Activation function and pooling: After the convolution operation, ReLU activation function and maximum pooling are used to increase nonlinearity and reduce the dimension of features:

[0230] X ReLU =ReLU(X (spatial) ),X pool =MaxPooling(X ReLU )

[0231] Preferably, ST-CNN is optimized. ST-CNN uses backpropagation and the Adam optimizer, with the mean squared error (MSE) loss function. During training, ST-CNN learns features in the spatial and temporal dimensions layer by layer, and optimizes network parameters through multiple iterations, ultimately outputting a spatiotemporal representation of flight risk.

[0232] Furthermore, when processing spatiotemporal data, in order to effectively capture the spatial and temporal dependencies, the data is converted into a graph structure G = (V, E), where V is a set of nodes and E is a set of edges. Each node v∈V represents a feature point in the spatiotemporal data. The features of these nodes include spatial location and time series data related to the location. The connection relationship between nodes is represented by the edge E, which represents the similarity between nodes.

[0233] Specifically, the spatiotemporal graph convolutional network structure (ST-GCN) is on the graph G = (V, E), and the node feature x j After normalized graph convolution

[0234]

[0235] The global feature g is formed by temporal convolution t Graph attention assigns a learnable weight to each edge, increasing attention to key nodes. t With g t Fusion in GatedResidualNetwork:

[0236] h t =GRN([c t ||g t ])

[0237] Output unified spatiotemporal representation for risk assessment.

[0238] As an optional implementation, this embodiment further provides a multimodal data fusion system for low-altitude flight risk warning, including:

[0239] Data alignment module, used to align flight status data and meteorological data using a unified timestamp;

[0240] a missing value filling module, configured to fill missing data in the aligned flight status data and the meteorological data using an interpolation method;

[0241] A feature extraction module, configured to extract local wind field changes and turbulence intensity from the interpolated meteorological data to obtain spatial features;

[0242] a data fusion module, configured to fuse the spatial features and the interpolated flight status data to obtain multimodal data;

[0243] The neural network analysis module is used to perform spatiotemporal modeling on the multimodal data using a spatiotemporal convolutional neural network and a spatiotemporal graph convolutional network to obtain a multimodal spatiotemporal feature representation.

[0244] The beneficial effects of the present invention are as follows:

[0245] (1) Multi-source data fusion: Traditional methods often rely on a single data source, while the present invention achieves joint modeling of multi-modal data by fusing flight status and meteorological data, thereby improving the accuracy and reliability of predictions.

[0246] (2) Spatiotemporal relationship modeling: The present invention adopts two network structures: spatiotemporal convolutional neural network and spatiotemporal graph convolutional network, which can simultaneously capture the spatial and temporal dependencies in flight data and meteorological data.

[0247] (3) Efficient data processing: Through spatiotemporal data fusion, the present invention can accurately align and complete missing data, thereby improving data quality and the real-time performance of the system.

[0248] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0249] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A multimodal data fusion method for low-altitude flight risk warning, characterized by: include: Use unified timestamps to align flight status data and meteorological data; Filling missing data in the aligned flight status data and the meteorological data using an interpolation method; Extracting local wind field changes and turbulence intensity from the interpolated meteorological data to obtain spatial features; fusing the spatial features and the interpolated flight status data to obtain multimodal data; The multimodal data is spatiotemporally modeled using a spatiotemporal convolutional neural network and a spatiotemporal graph convolutional network to obtain a multimodal spatiotemporal feature representation.

2. The multimodal data fusion method for low-altitude flight risk warning according to claim 1 is characterized in that: Use unified timestamps to align flight status data and meteorological data, including: Collect angular velocity data, linear acceleration data, motor speed data, current data, and voltage data from the onboard IMU and ESC to obtain hardware data; Use navigation GNSS to collect latitude and longitude data, altitude data, and heading data to obtain positioning data; Integrating the hardware data and the positioning data to obtain the flight status data; Collecting wind speed data, wind direction data, temperature data, humidity data, and air pressure data to obtain the meteorological data; Establish an absolute time scale; the expression of the absolute time scale is: t UTC =t week ×604800000+t tow(ms) ; where t UTC is the Coordinated Universal Time; t week is the current week number; t tow(ms) is the number of seconds in a week; The local clocks of the flight status data and the meteorological data are mapped to the absolute time scale using a mapping formula; the mapping formula is expressed as: UTC =αc local +β, Where α is the drift coefficient; β is the offset; c local is the local clock; c k Record the local count of the rising edge of PPS for the device; t k is UTC time; c k The mean of t k The mean of Setting a standardized time base according to the absolute time scale to obtain a main time axis; The flight status data and the meteorological data are attached to the main time axis in a left outer join manner for alignment processing.

3. The multimodal data fusion method for low-altitude flight risk warning according to claim 2 is characterized in that: Filling missing data in the aligned flight status data and the meteorological data using an interpolation method, including: When the missing value interval is less than a preset critical interval, the flight status data and the meteorological data are filled with missing data using linear interpolation; When the fluctuation amplitude or nonlinear trend of the aligned flight status data and the meteorological data exceeds a preset standard range, filling missing data of the flight status data and the meteorological data using cubic spline interpolation; When the missing value is less than 3 frames, the flight status data and the meteorological data are filled with missing data by using forward filling; When the missing value is greater than 3 frames and less than 1 second, Kalman smoothing is used to fill the missing data of the flight status data and the meteorological data.

4. The multimodal data fusion method for low-altitude flight risk warning according to claim 3 is characterized in that: Filling missing data in the aligned flight status data and the meteorological data using an interpolation method further includes: Normalizing the interpolated flight status data and the meteorological data using a Z-Score formula; A time decay factor is integrated into the normalized flight status data and the meteorological data.

5. The multimodal data fusion method for low-altitude flight risk warning according to claim 4 is characterized in that: Extract the local wind field changes and turbulence intensity of the interpolated meteorological data to obtain spatial characteristics, including: Converting the interpolated longitude and latitude of the meteorological data into an ENU coordinate system; Sampling the wind field data of the meteorological data using a 3×3×3 neighborhood as a sliding window to obtain a four-dimensional tensor; The spatial features are obtained by performing spatial differential operators, turbulence gust index calculation, body coordinate projection and cross-modal splicing on the four-dimensional tensor.

6. The multimodal data fusion method for low-altitude flight risk warning according to claim 5, characterized in that: The spatial features are obtained by performing spatial differential operators, turbulence gust index calculation, body coordinate projection, and cross-modal splicing on the four-dimensional tensor, including: Calculating a first-order central difference of the four-dimensional tensor; calculating a shear modulus of the four-dimensional tensor; Calculating the horizontal divergence of the four-dimensional tensor; calculating the vertical vorticity of the four-dimensional tensor; Calculate the wind direction angle of the four-dimensional tensor; the calculation formula of the wind direction angle is: Wherein, Δθ is the wind direction angle; For h u Wind speed vector at altitude level; For h u -Δu wind speed vector at altitude level; The first-order central difference, the shear modulus, the horizontal divergence, the vertical vorticity, and the wind direction angle are integrated to obtain the spatial differential operator.

7. The multimodal data fusion method for low-altitude flight risk warning according to claim 6, characterized in that: The spatial features are obtained by performing spatial differential operators, turbulence gust index calculation, body coordinate projection, and cross-modal splicing on the four-dimensional tensor, including: calculating the turbulence intensity of the four-dimensional tensor; calculating a gust factor of the four-dimensional tensor; calculating a volume turbulent dissipation rate of the four-dimensional tensor; The turbulence intensity, the gust factor, and the volume turbulence dissipation rate are integrated to obtain the turbulence gust index.

8. The multimodal data fusion method for low-altitude flight risk warning according to claim 7, characterized in that: Performing spatial differential operators, turbulence gust index calculation, body coordinate projection, and cross-modal splicing on the four-dimensional tensor to obtain the spatial features further includes: The spatial differential operator and the turbulence gust index are integrated to obtain direct extraction features. The wind vector of the four-dimensional tensor is transferred to the body axis by using an attitude rotation matrix, and angle of attack correction and sideslip angle wind-attitude coupling energy calculation are performed on each sliding window in the four-dimensional tensor to obtain a joint interaction feature; The directly extracted features and the joint interactive features are fused to obtain the spatial features.

9. The multimodal data fusion method for low-altitude flight risk warning according to claim 8, characterized in that: The multimodal data is subjected to spatiotemporal modeling using a spatiotemporal convolutional neural network and a spatiotemporal graph convolutional network to obtain a multimodal spatiotemporal feature representation, including: Performing feature extraction on the multimodal data using the spatial convolution layer and the first temporal convolution layer of the spatiotemporal convolutional neural network, and fusing outputs of the spatial convolution layer and the first temporal convolution layer using the spatiotemporal fusion layer of the spatiotemporal convolutional neural network to obtain a first feature representation; Performing feature extraction on the multimodal data using the graph convolution layer and the second temporal convolution layer of the spatiotemporal graph convolutional network, and fusing outputs of the graph convolution layer and the second temporal convolution layer using the spatiotemporal fusion layer of the spatiotemporal graph convolutional network to obtain a second feature representation; The first feature representation and the second feature representation are fused to obtain the spatiotemporal feature representation.

Images