Unmanned aerial vehicle monitoring method and system based on multi-modal machine learning

Abstract

The invention relates to the technical field of unmanned aerial vehicle multi-mode monitoring, and discloses an unmanned aerial vehicle monitoring method and system based on multi-mode machine learning, and the method comprises the steps: obtaining a historical data sequence of each monitoring terminal, and generating a spatio-temporal data set; generating a data comparison sequence of each processing time point based on the effective data sequence; calculating a single-point risk index and a comprehensive risk index, and evaluating a terminal fault risk; performing fault isolation or data fusion processing according to the comprehensive risk index, and outputting a monitoring result; according to the method, the problem of geometric constraint damage caused by manifold structure differences of different sensors is solved, the problem of high-fidelity repair of multi-source heterogeneous data in a non-Euclidean space is solved, the method adapts to changes of optical characteristics of the surface of a novel storage tank material, and intelligent completion of complex geometric monitoring data is achieved.

Description

Technical Field

[0001] This invention relates to the field of multimodal monitoring technology for unmanned aerial vehicles (UAVs), and more specifically, to a UAV monitoring method and system based on multimodal machine learning. Background Technology

[0002] While existing UAV multimodal monitoring technologies enhance coverage and collaborative monitoring capabilities by adding multiple identical monitoring terminals, this approach introduces new technical bottlenecks. As the number of terminals increases, the probability of a single terminal failing due to calibration drift, physical damage, or signal interference multiplies. Since all terminals participate in data fusion synchronously, interference data from any abnormal terminal can spread to the fusion results. Specifically, non-physical anomalies from faulty terminals are misjudged as real threats, triggering invalid alarms; if abnormal data coincidentally overlaps with normal ranges, it can mask real anomalies in the coverage area, creating monitoring blind spots; and existing methods lack a dynamic mechanism for distinguishing between persistent faults and transient interference. This results in the advantages of multi-terminal collaboration being negated by data contamination, ultimately leading to a decrease in monitoring accuracy instead of an increase. Summary of the Invention

[0003] This invention provides a multimodal machine learning-based method and system for monitoring unmanned aerial vehicles (UAVs), which solves the technical problem of geometric constraint violation caused by differences in the manifold structure of different sensors in related technologies.

[0004] This invention provides a multimodal machine learning-based method for monitoring unmanned aerial vehicles (UAVs), comprising the following steps:

[0005] Obtain a preset time period including multiple processing time points, and obtain the data sequence of the first monitoring terminal to the nth monitoring terminal on the UAV within the preset time period before the current time.

[0006] Multiple valid data sequences are aligned in the time dimension. For each processing time point, the monitoring data corresponding to the time is extracted from each valid data sequence and arranged in the order of the spatial location of the monitoring terminal to form the data comparison sequence corresponding to that processing time point.

[0007] Based on the data comparison sequence corresponding to each processing time point, the difference between adjacent monitoring data is calculated to generate a single-point risk indicator;

[0008] Calculate the comprehensive risk index based on the single-point risk indicators at all processing time points;

[0009] Determine whether the comprehensive risk index exceeds the preset threshold. If it does, obtain the monitoring data of each monitoring terminal at the current moment, process the data of each terminal separately, and output independent monitoring results.

[0010] If the preset threshold is not exceeded, the monitoring data of each monitoring terminal at the current moment is obtained, and the multimodal machine learning model is used to fuse multiple monitoring data to generate a comprehensive monitoring result.

[0011] The single-point risk index is obtained by calculating the average of the absolute values ​​of the differences between all adjacent monitoring data in the data comparison sequence, and the comprehensive risk index is obtained by time-weighted summation of the single-point risk indices at each processing time point.

[0012] Furthermore, after obtaining the data sequence, a data validity check step is also included:

[0013] Identify and remove invalid data sequences from n data sequences, and treat the remaining data sequences as multiple valid data sequences;

[0014] The criteria for determining invalid data sequences include: measurements at consecutive time points exceeding the normal fluctuation range, the presence of a large number of invalid filler values, or a data missing ratio exceeding the preset tolerance.

[0015] Furthermore, the calculation of the single-point risk indicator includes:

[0016] For the i-th processing time point, calculate the absolute value of the difference between the j-th monitoring data and the (j+1)-th monitoring data in its data comparison sequence. Sum the absolute values ​​of the differences of all adjacent data pairs and divide by the number of adjacent data pairs to obtain the single-point risk index for that processing time point.

[0017] Furthermore, the calculation of the comprehensive risk index includes:

[0018] A time weight is assigned to each processing time point. The time weight is set according to the time distance between the processing time point and the current time. The processing time point that is closer to the current time has a higher weight.

[0019] The comprehensive risk index is obtained by multiplying the individual risk indicators at each processing time point with their corresponding time weights and then summing the results.

[0020] Furthermore, the multimodal machine learning model employs a convolutional neural network, a recurrent neural network, or a graph neural network;

[0021] When using a convolutional neural network, the input layer receives a multidimensional tensor containing the number of effective monitoring terminals, the height, width, and number of monitoring data channels, and the output layer is a fully connected layer that outputs the anomaly probability or feature vector of the monitoring area.

[0022] When using a recurrent neural network, the input layer receives a sequence tensor containing the time step, the number of effective monitoring terminals, and the feature dimension, and the output layer maps the hidden state of the last time step to a monitoring result vector through a linear mapping layer.

[0023] When using a graph neural network, the input layer receives the node feature matrix and the adjacency matrix constructed based on the spatial distance between terminals, and the output layer is a graph-level pooling layer followed by a fully connected layer to generate a global monitoring state vector.

[0024] Furthermore, the data fusion step also includes spatial-feature joint modeling:

[0025] Spatially align and stitch data from adjacent terminals to generate a global monitoring map of the target area;

[0026] Extract spatial gradient and distribution features as joint features; perform anomaly detection and localization based on joint features.

[0027] Furthermore, the single-point risk indicators are normalized by using the maximum-minimum normalization or z-score standardization methods to eliminate the impact of differences in the units of different types of monitoring data.

[0028] When using max-min normalization, subtract the minimum value of the sequence from each value in the original single-point risk index sequence and then divide by the sequence range;

[0029] When using z-score standardization, each value in the original single-point risk index sequence is subtracted from the sequence mean and then divided by the sequence standard deviation.

[0030] Furthermore, the monitoring terminal includes at least one of a thermal imager, an optical camera, a lidar, and an infrared sensor;

[0031] The monitoring terminals are arranged at equal intervals along the horizontal axis of the UAV to ensure spatial coverage of a wide cross-section of the flight path.

[0032] Furthermore, the preset time period is set according to actual monitoring needs, and the value ranges from 5 seconds to 30 seconds;

[0033] The processing time points are discrete sampling points within a preset time period, and sampling is performed at fixed time intervals;

[0034] The preset threshold is determined based on statistical analysis of historical monitoring data and is set as the 95th percentile of the distribution of comprehensive risk indicators under normal conditions.

[0035] This invention provides a multimodal machine learning-based drone monitoring system for performing the aforementioned multimodal machine learning-based drone monitoring method, comprising:

[0036] The data acquisition module is used to acquire historical data sequences from multiple monitoring terminals on the UAV;

[0037] The data comparison module is used to generate data comparison sequences for each processing time point;

[0038] The risk calculation module is used to calculate single-point risk indicators and comprehensive risk indicators;

[0039] The decision execution module is used to determine whether to perform fault isolation or data fusion based on comprehensive risk indicators;

[0040] The results output module is used to output independent monitoring results or comprehensive monitoring results.

[0041] The beneficial effects of this invention are: overcoming the problem of geometric constraint destruction caused by differences in the manifold structure of different sensors, solving the problem of high-fidelity repair of multi-source heterogeneous data in non-Euclidean space, adapting to the changes in the surface optical properties of new storage tank materials, and realizing intelligent completion of complex geometric monitoring data. Attached Figure Description

[0042] Figure 1 This is a flowchart of the multimodal machine learning-based drone monitoring method of the present invention. Detailed Implementation

[0043] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0044] Example 1

[0045] This embodiment provides a multimodal machine learning-based method for monitoring unmanned aerial vehicles (UAVs), such as... Figure 1 As shown, the method includes the following steps:

[0046] Step 1: Obtain historical data sequences from each monitoring terminal and generate a spatiotemporal dataset;

[0047] A preset time period including multiple processing time points is obtained, and data sequences from the first monitoring terminal to the nth monitoring terminal on the UAV within the preset time period before the current time are obtained respectively. Each data sequence contains the monitoring data of the corresponding monitoring terminal at each processing time point within the preset time period.

[0048] It should be noted that the preset time period can be set according to actual monitoring needs, such as the past 5 seconds, 20 seconds, or 30 frames. The processing time points are discrete sampling points within the preset time period, and sampling is performed at fixed time intervals (such as per second or per frame).

[0049] In this embodiment of the application, to improve data quality, a data validity check step is included after acquiring the data sequence: invalid data sequences are identified and removed from the n data sequences, and the remaining data sequences are used as multiple valid data sequences. The criteria for determining invalid data sequences include: the measured values ​​at consecutive time points exceed the normal fluctuation range, there are a large number of invalid filler values, or the proportion of missing data exceeds a preset tolerance.

[0050] Step 2: Generate a data comparison sequence for each processing time point based on the valid data sequence;

[0051] Multiple valid data sequences are aligned along the time dimension. For each processing time point, the monitoring data corresponding to that time moment is extracted from each valid data sequence and arranged in order of the spatial location of the monitoring terminal to form a data comparison sequence corresponding to that processing time point.

[0052] It should be noted that the data comparison sequence reflects the synchronous monitoring status of different positions in the spatial lateral section of the monitoring area below the UAV at a specific moment. If the data sequence of a monitoring terminal is determined to be invalid, that terminal will not participate in the composition of the data comparison sequence.

[0053] Step 3: Calculate single-point risk indicators and comprehensive risk indicators to assess terminal failure risk;

[0054] Based on the data comparison sequence corresponding to each processing time point, the difference between adjacent monitoring data is calculated to generate a single-point risk indicator. The formula for calculating the single-point risk indicator is:

[0055]

[0056] in, For the first Single-point risk indicators at each processing time point For the first The number of monitored data points in the data comparison sequence at each processing time point For the first In the data comparison sequence of the processing time point, the first... One monitoring data point, For the first In the data comparison sequence of the processing time point, the first... One monitoring data point.

[0057] Calculate the comprehensive risk index based on the single-point risk indicators at all processing time points:

[0058]

[0059] in, As a comprehensive risk indicator, To process the number of time points, For the first The time weight of each processing time point.

[0060] It should be noted that time weighting Typically set to 1, indicating that all time points have equal importance. In specific applications, decreasing weights can be set based on the distance of a time point from the current time, giving more recent data a higher impact.

[0061] The aforementioned single-point risk index algorithm quantifies the discontinuity of spatial data by calculating the average absolute difference between data from adjacent monitoring terminals.

[0062] The algorithm is based on the assumption of spatial continuity, which means that under normal working conditions, the outputs of adjacent monitoring terminals should show a gradual change with small differences.

[0063] When a terminal malfunctions, the data difference between it and its neighboring terminals will increase significantly.

[0064] The algorithm obtains a quantitative index reflecting the degree of spatial data anomaly at a given time point by traversing all adjacent data pairs in the data comparison sequence, accumulating their absolute differences, and averaging them.

[0065] The aforementioned comprehensive risk index algorithm achieves risk aggregation in the time domain by weighted summation of single-point risk indicators at multiple time points. The key to this algorithm lies in introducing time weights. It can adjust the importance of different time points according to the application scenario.

[0066] When all weights are equal, the algorithm considers the risk status at each point in history equally.

[0067] As the weights decrease over time, the algorithm places greater emphasis on recent risk changes, thus improving its sensitivity to real-time faults.

[0068] In this embodiment, to improve the accuracy of fault detection, the single-point risk indicators can be normalized to eliminate the impact of differences in the units of measurement of different types of monitoring data. Normalization methods include max-min normalization or z-score standardization. When max-min normalization is used, the input is the original single-point risk indicator sequence. The output is a normalized index sequence. ,in:

[0069]

[0070] and These are the minimum and maximum values ​​in the sequence, respectively.

[0071] When z-score standardization is used, the input is the original single-point risk indicator sequence, and the output is the standardized indicator sequence, where:

[0072]

[0073] The mean of the sequence. is the standard deviation of the sequence.

[0074] Step 4: Based on the comprehensive risk indicators, perform fault isolation or data fusion processing and output the monitoring results;

[0075] Determine whether the comprehensive risk indicators exceed the preset threshold.

[0076] If the preset threshold is exceeded, the monitoring data of each monitoring terminal at the current moment is obtained, the data of each terminal is processed separately and independent monitoring results are output to avoid data contamination due to faults.

[0077] If the preset threshold is not exceeded, the monitoring data of each monitoring terminal at the current moment is obtained, and the multimodal machine learning model is used to fuse multiple monitoring data to generate a comprehensive monitoring result.

[0078] It should be noted that when the comprehensive risk index exceeds the preset threshold, it indicates a persistent hardware failure in the monitoring terminal network. In this case, by comparing abrupt changes in data from adjacent terminals, the faulty terminal can be accurately located. Data from normal terminals can still be used to output valid monitoring results independently.

[0079] It should be noted that multimodal machine learning models can employ conventional models such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), or Graph Neural Networks (GNNs). Specifically, when using a convolutional neural network, the input layer's receiving dimension is... The tensor, in which To effectively monitor the number of terminals, and For monitoring the height and width of data for a single terminal, The number of channels; the output layer is a fully connected layer, and the output dimension is:

[0080]

[0081] in The number of grids in the monitoring area, The anomaly probability or feature vector dimension for each grid.

[0082] When using a recurrent neural network, the input layer receives data in dimensions of 10 ... The sequence tensor, where For time step, To effectively monitor the number of terminals, For a single terminal's feature dimension;

[0083] The output layer uses a linear mapping layer to hide the state at the last time step. Mapping to Dimensions The monitoring result vector, where Output the dimension for the number of anomaly categories or continuous values.

[0084] When using a graph neural network, the input layer receives the node feature matrix:

[0085]

[0086] Adjacency matrix:

[0087]

[0088] in To monitor the number of terminal nodes, For node feature dimensions, the adjacency matrix is ​​constructed based on the spatial distance between terminals;

[0089] The output layer is a graph-level pooling layer followed by a fully connected layer, with a generation dimension of... The global monitoring state vector.

[0090] The training methods for the aforementioned multimodal machine learning models include:

[0091] The training mode employs supervised learning, using labeled historical monitoring data as the training set;

[0092] The optimization strategy uses the Adam optimizer, with an initial learning rate of 0.001 and a batch size of 32.

[0093] The loss function is selected based on the task type; for anomaly detection tasks, binary cross-entropy loss is used.

[0094]

[0095] For regression tasks such as temperature distribution prediction, mean squared error loss is used:

[0096]

[0097] in For real labels, These are the model's predicted values. This represents the number of samples.

[0098] During training, the hidden states of the aforementioned recurrent neural network are decoded through a linear mapping layer, and the weight matrix of the linear mapping layer... Hidden state dimension Mapping to output dimension The final output is generated through an activation function (softmax for classification tasks and linear activation for regression tasks).

[0099] In this embodiment of the application, to further improve monitoring accuracy, spatial-feature joint modeling is also included in the data fusion step:

[0100] Spatially align and stitch data from adjacent terminals to generate a global monitoring map of the target area;

[0101] Extract joint features such as spatial gradient and distribution characteristics; perform anomaly detection and localization based on joint features.

[0102] Example 2

[0103] The multimodal machine learning-based UAV monitoring method provided in this embodiment, in a UAV hardware environment equipped with a spherical panoramic camera and a curved surface LiDAR, further performs the following data processing steps based on the basic process of embodiment 1:

[0104] Step 5: Acquire spherical camera data and curved radar data, and calculate the manifold geometry descriptor;

[0105] Step 5 specifically includes: acquiring spherical image data from a spherical panoramic camera, acquiring curved point cloud data from a curved surface LiDAR, and calculating the Riemann metric tensor for each data point. The local curvature of the data points is calculated based on this metric tensor. and geodesic distance from neighboring points Generate a manifold geometry descriptor that includes geometric properties.

[0106] It should be noted that the Riemannian metric tensor The calculation is performed by defining an inner product structure in the tangent space of each data point, where:

[0107]

[0108] and Represents local coordinates on the manifold.

[0109] Furthermore, the specific methods for calculating the Riemannian metric tensor include:

[0110] For spherical image data, spherical parameterization is used. The metric tensor is:

[0111]

[0112]

[0113]

[0114] in Polar angle, It is the azimuth angle;

[0115] For curved point cloud data, the tangent plane is estimated through local principal component analysis, and the covariance matrix is ​​calculated:

[0116]

[0117] in for Nearest neighbor set, pair The principal direction is obtained by performing eigenvalue decomposition. metric tensor In a locally orthogonal coordinate system;

[0118] Calculate partial derivatives using the finite difference method:

[0119]

[0120] in Step size, For the first Unit vectors for each coordinate axis;

[0121] The positive definiteness of the metric tensor is verified by numerical integration, and the eigenvalues ​​are calculated. Ensure the measurement is reasonable.

[0122] It should be noted that the geodesic distance The calculation is obtained by solving the geodesic equation:

[0123]

[0124] in For Christofel's symbol.

[0125] Furthermore, the specific calculation method for the Christofel symbol includes:

[0126] Defined using the second type of Christofel notation:

[0127]

[0128] in The inverse matrix elements of the metric tensor;

[0129] Calculate the partial derivative of the metric tensor using finite differences:

[0130]

[0131] in This is the numerical differentiation step size;

[0132] For spherical surfaces, we can directly use analytical expressions:

[0133]

[0134]

[0135] The remaining components are zero;

[0136] For general curved surfaces, the calculation is performed by matrix inversion. Cholesky decomposition is used to improve numerical stability;

[0137] Verify the symmetry of the Christofer symbol And covariance properties ensure geometric consistency of calculation results.

[0138] It should be noted that, due to the differences in numerical range and dimensions between spherical image data (pixel values ​​ranging from 0 to 255) and curved point cloud data (3D coordinate values, in meters), the original data are preprocessed by normalization before calculating the manifold geometry descriptor: spherical image data is normalized to the range of [0,1] by dividing by 255, and curved point cloud data is normalized to the range of [-1,1] by subtracting the centroid and dividing by the length of the bounding box diagonal.

[0139] Step 6: Map the manifold data to the tangent space to generate the tangent vector representation;

[0140] Step 6 specifically includes: selecting a reference point on the manifold, and applying the logarithmic mapping algorithm to the spherical image data and the surface point cloud data respectively. Points on the manifold Mapped to reference point tangent space In this process, a unified tangent vector representation is output, achieving linearization of non-Euclidean data.

[0141] Furthermore, specific methods for selecting reference points include:

[0142] For spherical image data, the North Pole of the sphere is selected as the reference point, with coordinates as follows: This point has the smallest geodesic distortion and the largest tangential space coverage;

[0143] For curved surface point cloud data, a reference point is determined using the geometric center method, and the centroid of all data points is calculated:

[0144]

[0145] Then project it onto the nearest manifold surface point:

[0146]

[0147] Using curvature-based optimization, the point with the minimum local curvature in the candidate point set is selected as the reference point. The calculation criterion is as follows:

[0148]

[0149] in For the candidate point set, and Points The mean curvature and Gaussian curvature;

[0150] Verify the validity of the reference point, ensuring that the geodesic distance from the reference point to all data points is less than the cut point distance of the manifold. ,when Accept this reference point at that time.

[0151] It should be noted that the logarithmic mapping algorithm is applicable to the present invention when the input is two points (reference points) on the manifold. and target point and the Riemannian metric of manifolds The output is the reference point. tangent space tangent vector in ;

[0152] Logarithmic mapping The calculation is performed by solving from arrive Find the shortest geodesic and return the geodesic in The initial velocity vector of the point.

[0153] Furthermore, the specific calculation methods for logarithmic mappings include:

[0154] Solving the geodesic boundary value problem using the Newton-Raphson iterative method, with an initial guessed tangent vector. Objective function:

[0155]

[0156] Calculate the Jacobian matrix:

[0157]

[0158] Approximated by finite difference:

[0159]

[0160] in ;

[0161] Iterative updates:

[0162]

[0163] Set convergence criteria:

[0164]

[0165] Or maximum number of iterations:

[0166]

[0167] For the spherical case, use the analytical solution:

[0168]

[0169] in for exist The tangent space projection;

[0170] Verify the consistency of the results, check Whether it is valid or not.

[0171] Step 7: Calculate feature correlation based on cross-modal attention mechanism to generate cross-modal association weights;

[0172] Step 7 specifically includes: passing the spherical image tangent vector and the curved point cloud tangent vector obtained in step 6 through a feature extraction network to obtain image features. and point cloud features The weight matrix is ​​calculated using a cross-modal attention mechanism, with the input being the feature matrices of the two modalities. and The output is the attention weight matrix. ;

[0173] The specific calculation is as follows:

[0174]

[0175] Based on this weight matrix, complementary information between the two modes is mined in the tangent space, and cross-modal correlation weights are output.

[0176] Furthermore, the specific computational process of the cross-modal attention mechanism includes:

[0177] Image features and point cloud features Perform L2 normalization preprocessing:

[0178]

[0179]

[0180] Calculate the similarity matrix:

[0181]

[0182] in Indicates the first The image features and the first Cosine similarity of point cloud features;

[0183] Temperature scaling of the similarity matrix To prevent softmax saturation;

[0184] The attention weights are calculated using the softmax function along the row direction:

[0185]

[0186] Calculate the weighted feature representation:

[0187]

[0188] in The fused cross-modal features;

[0189] The final output is obtained through residual connections and layer normalization:

[0190]

[0191] It should be noted that the feature extraction network uses a multilayer perceptron (MLP), and its input layer has a receiving dimension of [missing information]. The tangent vector representation, where the tangent vector of the spherical image is:

[0192] ( , , (Height, width, and number of channels, respectively).

[0193] Point cloud tangent vectors:

[0194] ( (in points)

[0195] It contains three fully connected layers. The first layer maps the input to a 256-dimensional hidden space, the second layer maintains the 256-dimensionality, and the third layer serves as the output layer, mapping to... 3D feature space;

[0196] The first two layers use the ReLU activation function, while the output layer does not use an activation function.

[0197] The aforementioned feature extraction network adopts a supervised learning training mode, updates parameters through the Adam optimizer, and sets the learning rate to 0.001;

[0198] Use the contrastive loss function during training:

[0199]

[0200] in The cosine similarity of the feature vectors. For temperature parameters, For batch size, For indicator functions, when the condition The value is 1 if true, and 0 otherwise.

[0201] Network output The 3D feature vectors are directly used for subsequent cross-modal attention calculations.

[0202] Furthermore, the specific definitions and calculation methods of each parameter in the comparison loss function are as follows:

[0203] Cosine similarity:

[0204]

[0205] in For the first and the The feature vector of each sample, with values ​​ranging from... ;

[0206] Batch size The range of values ​​is Dynamically adjusted based on GPU memory limitations, with a computational complexity of O(n). ;

[0207] Indicator Function Ensure that the similarity between a sample and itself is not calculated to avoid trivial solutions to the loss function;

[0208] Summation term in denominator It contains positive and negative sample pairs, with the number of positive sample pairs being [number missing]. The number of negative sample pairs is This ensures a balance in comparative learning.

[0209] Furthermore, temperature parameters The value range is set to The specific values ​​are obtained through grid search. Choose from, when When the size is small, it enhances the discrimination of difficult samples. Larger values ​​improve the robustness of the model.

[0210] It should be noted that, due to the tangent vector of the spherical image (dimension...) ) and surface point cloud tangent vector (dimension) The numerical range and distribution of the two cut vectors differ significantly. Before inputting them into the feature extraction network, the two cut vectors undergo standardization preprocessing using the Z-score standardization method:

[0211]

[0212] in and These are the mean and standard deviation of the tangent vector, respectively, ensuring that the input features have zero mean and unit variance.

[0213] Step 8: Input the neighborhood features of the missing region into the manifold perception generator to generate repair data;

[0214] Step 8 specifically includes: identifying missing regions in spherical images and curved point clouds, extracting geometric features and cross-modal correlation features of the neighborhood of the missing regions, inputting these features into a manifold perception generator, generating preliminary repair values ​​through geodesic convolution operations, preserving the local geometric structure of the data, and outputting the repaired monitoring data.

[0215] Furthermore, specific methods for identifying missing regions include:

[0216] For spherical image data, threshold segmentation is used to detect pixel values ​​that are 0 or close to 0. For a connected region of ), use morphological opening operations to remove noise, retaining an area greater than 1. The region of pixels is considered as the missing region;

[0217] For curved surface point cloud data, calculate the value of each point. Nearest neighbor density:

[0218]

[0219] in For the first The distance of a nearest neighbor, when When it is determined to be a sparse region, among which The global average density;

[0220] Local mean curvature is calculated using curvature-based geometric anomaly detection:

[0221]

[0222] when The time marker is used to define the geometric anomaly region, where The global average curvature, The standard deviation of curvature;

[0223] By merging adjacent missing regions through connectivity analysis and using a region growing algorithm to expand the boundaries of the missing regions, the final repair range is determined.

[0224] It should be noted that the manifold-aware generator consists of two parts: an encoder and a decoder. The encoder consists of three geodesic convolutional layers, with the first layer receiving the input features. ( For the number of data points, (For the number of input channels), perform a geodesic convolution operation:

[0225]

[0226] Mapped to a 64-dimensional feature space;

[0227] Second layer:

[0228]

[0229] Further extract 128-dimensional deep features;

[0230] Third layer

[0231]

[0232] Output 256-dimensional encoded features.

[0233] The decoder consists of three transposed geodesic convolutional layers, in the following order:

[0234]

[0235]

[0236]

[0237] in This represents the number of output channels.

[0238] Furthermore, the number of input channels Compared with the feature dimensions in step 7 Maintain consistency, that is Number of output channels Determined based on the type of data being recovered: For spherical image data (Number of image channels), for curved point cloud data (3D coordinates) ensure that the generator's input and output dimensions are consistent with the feature representations of the preceding steps.

[0239] It should be noted that the geodesic convolution operation is defined as follows:

[0240]

[0241] in For point The geodesic neighborhood, This is a weighting function based on geodesic distance;

[0242] The data transfer relationship in the encoder is as follows:

[0243]

[0244]

[0245]

[0246] The transitive relationship in the decoder is as follows:

[0247]

[0248]

[0249]

[0250] in This is the ReLU activation function.

[0251] The aforementioned geodesic convolution operation extends convolution in traditional Euclidean space to manifold space. Its core improvement lies in using geodesic distance. Use Euclidean distance instead of Euclidean distance to calculate weights;

[0252] The specific implementation steps include:

[0253] For each center point Calculate its using Dijkstra's algorithm Distance between nearest geodesic lines;

[0254] Calculate the Gaussian weights based on the geodesic distance:

[0255]

[0256] in The bandwidth parameter of the Gaussian kernel;

[0257] The convolutional output is obtained by weighting and aggregating the feature values ​​within the neighborhood.

[0258] This operation preserves the inherent geometry of the manifold, avoiding the distortion problem of Euclidean convolution on curved data.

[0259] Furthermore, the specific implementation of Dijkstra's algorithm for calculating geodesic distances on a manifold includes:

[0260] Construct a weighted undirected graph from the manifold data points:

[0261]

[0262] in Given a set of data points, the edge weights are:

[0263]

[0264] This represents the Euclidean distance between adjacent points;

[0265] To ensure graph connectivity, for each point Connect its Add the nearest neighbor at the same time. - All points within the neighborhood are considered neighbors;

[0266] Initialize distance array (Except for the source point) ), priority queue Store the point pairs to be processed ;

[0267] Main loop: From Find the point with the smallest distance from the given information. to all his neighbors Perform relaxation operation if:

[0268]

[0269] Then update:

[0270]

[0271] And Add to queue;

[0272] Repeat step 4 until the queue is empty or the target point is found, then output the result. The array represents the approximate geodesic distances from the source point to all other points.

[0273] To improve accuracy, local geodesic correction is adopted, and Newton's method is used to solve the geodesic equations for point pairs with large distances.

[0274] Furthermore, the number of nearest neighbors The value range is set to The specific value is adaptively determined based on the data point density:

[0275]

[0276] in This represents the total number of data points within the local area. To ensure that the neighborhood size matches the data distribution density during the floor operation.

[0277] Furthermore, bandwidth parameters The value range is set to the scale of local features of the manifold. The specific value is:

[0278]

[0279] in Scaling factor The mean of the geodesic distances within a local neighborhood, when When smaller, it highlights local details and features. When the size is large, the global geometry is preserved.

[0280] The aforementioned manifold-aware generator employs a self-supervised training model, using the AdamW optimizer for parameter updates, with an initial learning rate of 0.0001 and a weight decay coefficient of 0.01; during training, a time-aware combined loss function is used.

[0281]

[0282] The reconstruction loss is:

[0283]

[0284] Geometric consistency loss is used to ensure data fidelity.

[0285]

[0286] Time consistency loss is used to maintain the manifold structure.

[0287]

[0288] Used to maintain temporal continuity, where Here are the time decay parameters and time weighting coefficients:

[0289]

[0290]

[0291]

[0292] The generator takes feature vectors from the neighborhood of the missing region as input and outputs the repaired, complete data.

[0293] Furthermore, the specific manifestations and constraints of the time dimension are as follows:

[0294] Time Index:

[0295]

[0296] in To monitor the total length of the sequence, constraints are set. Ensure the stability of the time series;

[0297] The time weighting coefficient adopts an adaptive adjustment strategy:

[0298]

[0299]

[0300] in For the fitness constant;

[0301] Temporal consistency constraints ensure the smoothness of repair results between adjacent time steps. The constraints are as follows:

[0302]

[0303] Historical information weight decay parameter The range of values ​​is ,when When the data is smaller, more recent data is considered. When the value is large, consider longer historical information.

[0304] It should be noted that the output of the manifold sensing generator... It needs to be decoded into specific, usable monitoring data. The decoding process includes:

[0305] For spherical image data, the feature vectors are recombined into The dimensions are then restored to the original pixel value range through an inverse normalization operation (multiplied by 255);

[0306] For curved point cloud data, the feature vectors are recombined into The dimensions are determined and restored to the original coordinate system through an anti-normalization operation (multiplying by the length of the bounding box diagonal and adding the centroid);

[0307] The geometric consistency and physical rationality of the repaired data are verified through the quality inspection module.

[0308] Furthermore, the specific inspection methods for the quality inspection module include:

[0309] Geometric consistency test: Calculate the curvature continuity index between the repaired region and its neighboring regions.

[0310]

[0311] in For the number of boundary points, when It passed the inspection in time;

[0312] Physical rationality verification verifies whether the repaired pixel values ​​are within a reasonable range. Does the point cloud coordinate satisfy the geometric constraints of the storage tank?

[0313]

[0314] in Where is the radius of the storage tank;

[0315] Statistical consistency testing was performed to calculate the statistical distribution difference between the repaired area and the global data, using the Kolmogorov-Smirnov test statistic.

[0316]

[0317] when The distribution is considered to be consistent at that time;

[0318] Visual quality assessment, calculating the structural similarity index of the repaired area:

[0319]

[0320] when The visual quality was deemed acceptable at that time.

[0321] For repair results that do not meet quality requirements, iterative optimization is performed by increasing regularization weights or adjusting generator parameters.

[0322] In this embodiment of the application, in order to adapt to the changes in the properties of the new storage tank material, the following steps are also included in step 6:

[0323] Step 9: Use a domain discriminator network to identify differences in material data distribution and generate a domain offset metric;

[0324] Step 9 specifically includes: inputting the tangent vector representations of the new material data and historical material data into the domain discriminator network, identifying the distribution differences between the two types of data through adversarial training, and calculating the Wasserstein distance in the manifold space.

[0325]

[0326] Generate a domain offset metric, where and These represent the probability distributions of the new and old material data on the manifold, respectively. The marginal distribution is represented as and The coupled set of all probability measures.

[0327] Furthermore, numerical methods for calculating the Wasserstein distance include:

[0328] Discretized probability distribution from new materials dataset and historical data datasets Sample from each sample and construct an empirical distribution:

[0329]

[0330]

[0331] Calculate the geodesic distance matrix ,in:

[0332]

[0333] Construct a linear programming problem:

[0334]

[0335] The constraints are:

[0336]

[0337]

[0338]

[0339] The Sinkhorn algorithm is used to approximate the solution, and the process is iteratively updated.

[0340]

[0341] in:

[0342]

[0343] For regularization parameters;

[0344] Calculate the final Wasserstein distance:

[0345]

[0346] in This is the optimal transfer matrix after convergence.

[0347] It should be noted that the domain discriminator network consists of four fully connected layers. The first layer receives the input tangent vector. Through linear transformation Mapped to 512-dimensional space;

[0348] Second floor ;

[0349] Third layer Stepwise dimensionality reduction to extract discriminative features;

[0350] Fourth floor Output domain discrimination score.

[0351] The data transfer relationship is as follows:

[0352]

[0353]

[0354]

[0355]

[0356] The negative slope of LeakyReLU is 0.2.

[0357] Furthermore, the negative slope parameter of the LeakyReLU activation function The value range is set to The specific value is 0.2, chosen within this range to balance gradient propagation and feature representation capability. When the expression is smaller, it enhances nonlinear expression. Larger gradients improve liquidity.

[0358] The aforementioned domain discriminator network adopts an adversarial training mode, uses the RMSprop optimizer, and has a learning rate of 0.0001;

[0359] During training, time-aware Wasserstein distance is used as the loss function.

[0360]

[0361] in For time-dependent discriminator functions, The time regularization coefficient;

[0362] To satisfy the 1-Lipschitz constraint, spectral normalization is applied to the network weights; the network input consists of the tangent vector representation of the old and new material data and timestamps, and the output is a scalar domain discrimination score, which is used to calculate the domain offset metric.

[0363] Furthermore, the specific implementation of the time-aware discriminator includes:

[0364] The time encoding layer indexes the time. Converted into a temporal feature vector through positional encoding:

[0365]

[0366] in ;

[0367] Multi-scale temporal convolutional layers using different kernel sizes Capture short-term, medium-term, and long-term time dependencies;

[0368] Temporal attention mechanism, calculating temporal weights:

[0369]

[0370] in ;

[0371] The discriminator loss function uses a time-weighted adversarial loss:

[0372]

[0373] in For the discriminator output, and The actual and generated times are respectively. data;

[0374] To ensure training stability, the discriminator parameter update frequency is set to twice that of the generator, and the learning rate is set to 0.5 times that of the generator.

[0375] Furthermore, the parameters and constraints of the time-aware discriminator are defined as follows:

[0376] probability distribution and Representing time respectively Distribution of new and historical material data, with constraints:

[0377]

[0378] Ensure the normalization of the probability distribution;

[0379] Time regularization term:

[0380]

[0381] Constraints used to control the smoothness of the discriminator over time:

[0382]

[0383] Discriminator function:

[0384]

[0385] Where the input feature dimension Time dimension , This is the maximum monitoring duration;

[0386] Expectation calculation is based on Monte Carlo sampling, and the number of samples... The sampling strategy is importance sampling to improve estimation accuracy.

[0387] Furthermore, specific implementation methods for spectral normalization include:

[0388] For each weight matrix The maximum singular value is calculated using the power iteration method. Initialize random vector , ;

[0389] Iterative updates:

[0390]

[0391]

[0392] repeat The next iteration;

[0393] Calculate the maximum singular value:

[0394]

[0395] Perform spectral normalization on the weight matrix:

[0396]

[0397] make sure ;

[0398] Use normalized weights during each forward propagation. Update the original weights during backpropagation. The gradient is calculated using the chain rule:

[0399]

[0400] Every The singular vector is updated once every training step. and In order to maintain computational efficiency.

[0401] Furthermore, the specific steps for implementing adversarial training include:

[0402] Initialize the discriminator and generator Network parameters, setting the number of training rounds and batch size ;

[0403] In each training iteration, the generator parameters are fixed, and the discriminator is trained using historical material data (labeled 1) and new material data (labeled 0). The discriminator is then optimized using gradient ascent. ;

[0404] Then, with the discriminator parameters fixed, train the generator so that its generated data can "fool" the discriminator, and optimize it using gradient descent:

[0405]

[0406] A gradient penalty update is performed every 5 iterations:

[0407]

[0408] in:

[0409]

[0410]

[0411]

[0412] Ensure Lipschitz constraints are met;

[0413] Repeat steps 2-4 until convergence or the maximum number of iterations is reached.

[0414] It should be noted that the scalar domain discrimination score output by the domain discriminator network needs to be interpreted and applied to specific domain adaptation strategies. Interpretation methods include:

[0415] When the score is greater than the threshold When the data is identified as new material data, the domain adaptation mechanism is activated.

[0416] When the score is less than the threshold When the data is classified as historical material, the existing model is used.

[0417] When the score falls between the two thresholds, a weighted fusion strategy is employed, with the weights as follows:

[0418]

[0419] The prediction results of the new and old models are weighted and combined.

[0420] Furthermore, the specific implementation steps of the domain adaptation mechanism include:

[0421] An adaptive dataset was constructed based on new materials data, and historical data with the distribution most similar to that of the new materials data were selected. One sample is used as the source domain data;

[0422] Using the maximum mean difference (MMD) loss:

[0423]

[0424] Perform feature alignment, where For feature mapping function, and These are samples from the source domain and the target domain, respectively.

[0425] The model parameters are updated using a gradual fine-tuning strategy, with the initial learning rate set to the original learning rate. ,Every epoch decay times;

[0426] A pseudo-labeling strategy is used to label data with a confidence level greater than 100%. The new material prediction results are used to generate pseudo-labels to expand the training set;

[0427] Using combined loss:

[0428]

[0429] Joint optimization is performed, among which , .

[0430] Furthermore, threshold and The selection was determined based on ROC curve analysis on the training set. Set as the score value corresponding to achieving 95% accuracy on the validation set. The score value corresponding to a 95% recall rate on the validation set is set as the interval between the two thresholds:

[0431]

[0432] To ensure the stability of the decision boundary.

[0433] In this embodiment of the application, in order to improve the geometric consistency of the repaired data, the following steps are also included in step 8:

[0434] Step 10: Optimize the repair content using cyclic consistency constraints to generate geometrically consistent repair data;

[0435] Step 10 specifically includes: applying the repair results from step 8 through forward mapping. Mapping from the source manifold to the target manifold, and then through the inverse mapping Map back to the source manifold and calculate the cycle consistency loss:

[0436]

[0437] Based on this loss, the repair content is optimized to ensure the invertibility of the mapping and generate geometrically consistent repair data.

[0438] Furthermore, specific implementation methods for circular consistency mapping include:

[0439] Forward mapping A three-layer fully connected neural network is used. The first layer is:

[0440]

[0441] Second layer:

[0442]

[0443] Third layer:

[0444]

[0445] in and These are the embedding dimensions of the source manifold and the target manifold, respectively;

[0446] Inverse mapping Using the same network structure but with independent parameters:

[0447]

[0448]

[0449]

[0450] Using an activation function with manifold constraints, normalization is applied to spherical data:

[0451]

[0452] For general manifolds, the projection operator is used:

[0453]

[0454] in It is the normal vector;

[0455] During training, an alternating optimization strategy is used, first fixing... optimization Then fix optimization Each iteration step;

[0456] Add identity mapping regularization:

[0457]

[0458] Weighting coefficient To prevent mapping degradation.

[0459] In this embodiment of the application, in order to further improve the repair accuracy, the following steps are also included in addition to step 8:

[0460] Step 11: Adjust the Riemannian metric based on adaptive curvature estimation and project the tangent vector back into the manifold space;

[0461] Step 11 specifically includes:

[0462] Based on the local geometric characteristics of the repaired area, the average curvature of the area is calculated using an adaptive curvature estimation method. and Gaussian curvature Based on the rate of change of curvature and Dynamically Adjusting the Riemannian Metric Tensor The value;

[0463] The exponential mapping algorithm is applied, with the reference point as the input. tangent vector and the updated Riemann metric The output is points on the manifold. ;

[0464] Exponential mapping The repaired tangent vector is projected back into the original manifold space to obtain repaired data points that preserve geometric properties.

[0465] Furthermore, specific implementation methods of the exponential mapping algorithm include:

[0466] Solving the geodesic initial value problem using the fourth-order Runge-Kutta method:

[0467]

[0468] Initial conditions are , ;

[0469] Set the integration step size and integration interval Transform the second-order differential equation into a system of first-order equations:

[0470]

[0471] in ;

[0472] Calculate the RK4 coefficients:

[0473]

[0474]

[0475]

[0476]

[0477] Update status:

[0478]

[0479] Repeat until Output As ;

[0480] For the spherical case, use the analytical solution:

[0481]

[0482] in Let be the magnitude of the tangent vector;

[0483] Verify that the output point lies on the manifold and check the constraints. (For a unit sphere) or other corresponding manifold constraints.

[0484] The aforementioned adaptive curvature estimation method includes the following steps:

[0485] Sampling at the boundary of the repair area For each sampling point, calculate the first basic form:

[0486]

[0487] Second basic form:

[0488]

[0489] pass:

[0490]

[0491] Calculate the mean curvature;

[0492] pass:

[0493]

[0494] Calculate Gaussian curvature;

[0495] in The trace operation represents the sum of the elements on the main diagonal of a matrix.

[0496] The curvature field was fitted using the moving least squares method to obtain the rate of change of curvature. and ;

[0497] To eliminate the dimensional mismatch problem, the rate of change of curvature is made dimensionless:

[0498]

[0499]

[0500] in The feature length scale;

[0501] The metric tensor is adjusted based on the dimensionless rate of change of curvature, using the following formula:

[0502]

[0503] in This is the second fundamental form of dimensionless designation. These are adaptive coefficients.

[0504] Furthermore, the specific implementation of the moving least squares method for fitting curvature fields includes:

[0505] For each query point Define the neighborhood radius:

[0506]

[0507] in To determine the average grid spacing, select sampling points within the neighborhood. ;

[0508] Construct the weight function:

[0509]

[0510] in For bandwidth parameters;

[0511] Establish a weighted least squares problem:

[0512]

[0513] in Let be the basis functions of a quadratic polynomial, and the coefficients to be determined are:

[0514]

[0515] Solve for the coefficients using the normal equations:

[0516]

[0517] in For polynomial basis matrices, This is a weighted diagonal matrix. It is a curvature vector;

[0518] Calculate the gradient:

[0519]

[0520] For Gaussian curvature The same process was performed to obtain .

[0521] Furthermore, the number of boundary sampling points The value range is set to the ratio of the repair region boundary length to the average grid spacing. times, that is:

[0522]

[0523] in For sampling density factor, The average grid spacing, To perform the floor function, the sampling density is adapted to the local geometric complexity.

[0524] Furthermore, adaptive coefficients The value range is set to The specific value is determined through:

[0525]

[0526] Adaptive determination, where As a baseline coefficient, when the rate of change of curvature is large. Automatic reduction to maintain numerical stability when the rate of curvature change is small. Increase the size appropriately to enhance the adjustment effect.

[0527] Step 12: Perform multi-scale geodesic smoothing and output complete monitoring data;

[0528] Step 12 specifically includes: applying a multi-scale geodesic smoothing filter to the repair boundary region at different scales. Calculate the geodesic Gaussian kernel below:

[0529]

[0530] By optimizing the transition of the repair boundary through weighted averaging, the output is complete monitoring data that maintains the non-Euclidean geometric properties.

[0531] The aforementioned multi-scale geodesic smoothing filter achieves boundary smoothing by performing multi-resolution filtering in the manifold space; the specific steps include:

[0532] Define the scale sequence:

[0533]

[0534] in , Basic scale;

[0535] For each scale Calculate the repair boundary points geodesic neighborhood:

[0536]

[0537] Filtering using the geodesic Gaussian kernel:

[0538]

[0539] The filtering results at different scales are fused using the Laplacian pyramid:

[0540]

[0541] Wherein the weights are:

[0542]

[0543] This method preserves the geodesic structure of the manifold and avoids the geometric distortion caused by traditional Euclidean smoothing.

[0544] Furthermore, basic scale The value is set to the length of the repair area boundary. arrive ,Right now:

[0545]

[0546] in The scale factor. To repair the total geodesic length of the regional boundary, the number of layers in the scale sequence. Set as arrive Layers are used to balance computational efficiency and smoothing effects.

[0547] It should be noted that the final output of complete monitoring data includes the repaired spherical image data and curved surface point cloud data, which need to be converted into a format that the monitoring system can directly use:

[0548] Spherical image data is saved in standard image formats (such as JPEG and PNG), containing complete pixel information and metadata;

[0549] The curved surface point cloud data is saved in a standard point cloud format (such as PCD, PLY), which includes three-dimensional coordinate information and intensity information;

[0550] Simultaneously, a monitoring report is generated, marking the location, confidence level, and geometric characteristic parameters of the repaired area.

[0551] It is understood that data preprocessing methods known to those skilled in the art include data cleaning, data transformation, and data reduction. Data transformation includes type conversion and normalization and standardization. Although the dimensions and types of data were omitted in the description of the preceding embodiments, data preprocessing is a technical knowledge known to those skilled in the art and a prerequisite step in data processing. Therefore, the previously described well-known data preprocessing steps were not described independently.

[0552] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A multimodal machine learning-based method for monitoring unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: Obtain a preset time period including multiple processing time points, and obtain the data sequence of the first monitoring terminal to the nth monitoring terminal on the UAV within the preset time period before the current time. Multiple valid data sequences are aligned in the time dimension. For each processing time point, the monitoring data corresponding to the time is extracted from each valid data sequence and arranged in the order of the spatial location of the monitoring terminal to form the data comparison sequence corresponding to that processing time point. Based on the data comparison sequence corresponding to each processing time point, the difference between adjacent monitoring data is calculated to generate a single-point risk indicator; Calculate the comprehensive risk index based on the single-point risk indicators at all processing time points; Determine whether the comprehensive risk index exceeds the preset threshold. If it does, obtain the monitoring data of each monitoring terminal at the current moment, process the data of each terminal separately, and output independent monitoring results. If the preset threshold is not exceeded, the monitoring data of each monitoring terminal at the current moment is obtained, and the multimodal machine learning model is used to fuse multiple monitoring data to generate a comprehensive monitoring result. The single-point risk index is obtained by calculating the average of the absolute values ​​of the differences between all adjacent monitoring data in the data comparison sequence, and the comprehensive risk index is obtained by time-weighted summation of the single-point risk indices at each processing time point.

2. The UAV monitoring method based on multimodal machine learning according to claim 1, characterized in that, After obtaining the data sequence, a data validity check step is also included: Identify and remove invalid data sequences from n data sequences, and treat the remaining data sequences as multiple valid data sequences; The criteria for determining invalid data sequences include: measurements at consecutive time points exceeding the normal fluctuation range, the presence of a large number of invalid filler values, or a data missing ratio exceeding the preset tolerance.

3. The UAV monitoring method based on multimodal machine learning according to claim 1, characterized in that, The calculation of the single-point risk indicator includes: For the i-th processing time point, calculate the absolute value of the difference between the j-th monitoring data and the (j+1)-th monitoring data in its data comparison sequence. Sum the absolute values ​​of the differences of all adjacent data pairs and divide by the number of adjacent data pairs to obtain the single-point risk index for that processing time point.

4. The UAV monitoring method based on multimodal machine learning according to claim 1, characterized in that, The calculation of the comprehensive risk index includes: A time weight is assigned to each processing time point. The time weight is set according to the time distance between the processing time point and the current time. The processing time point that is closer to the current time has a higher weight. The comprehensive risk index is obtained by multiplying the individual risk indicators at each processing time point with their corresponding time weights and then summing the results.

5. The UAV monitoring method based on multimodal machine learning according to claim 1, characterized in that, The multimodal machine learning model employs convolutional neural networks, recurrent neural networks, or graph neural networks; When using a convolutional neural network, the input layer receives a multidimensional tensor containing the number of effective monitoring terminals, the height, width, and number of monitoring data channels, and the output layer is a fully connected layer that outputs the anomaly probability or feature vector of the monitoring area. When using a recurrent neural network, the input layer receives a sequence tensor containing the time step, the number of effective monitoring terminals, and the feature dimension, and the output layer maps the hidden state of the last time step to a monitoring result vector through a linear mapping layer. When using a graph neural network, the input layer receives the node feature matrix and the adjacency matrix constructed based on the spatial distance between terminals, and the output layer is a graph-level pooling layer followed by a fully connected layer to generate a global monitoring state vector.

6. The UAV monitoring method based on multimodal machine learning according to claim 1, characterized in that, The data fusion step also includes spatial-feature joint modeling: Spatially align and stitch data from adjacent terminals to generate a global monitoring map of the target area; Extract spatial gradient and distribution features as joint features; perform anomaly detection and localization based on joint features.

7. The UAV monitoring method based on multimodal machine learning according to claim 1, characterized in that, Single-point risk indicators are normalized by using maximum-minimum normalization or z-score standardization methods to eliminate the impact of differences in the units of different types of monitoring data. When using max-min normalization, subtract the minimum value of the sequence from each value in the original single-point risk index sequence and then divide by the sequence range; When using z-score standardization, each value in the original single-point risk index sequence is subtracted from the sequence mean and then divided by the sequence standard deviation.

8. The UAV monitoring method based on multimodal machine learning according to claim 1, characterized in that, The monitoring terminal includes at least one of a thermal imager, an optical camera, a lidar, and an infrared sensor; The monitoring terminals are arranged at equal intervals along the horizontal axis of the UAV to ensure spatial coverage of a wide cross-section of the flight path.

9. The UAV monitoring method based on multimodal machine learning according to any one of claims 1 to 8, characterized in that, The preset time period is set according to actual monitoring needs, and the value ranges from 5 seconds to 30 seconds; The processing time points are discrete sampling points within a preset time period, and sampling is performed at fixed time intervals; The preset threshold is determined based on statistical analysis of historical monitoring data and is set as the 95th percentile of the distribution of comprehensive risk indicators under normal conditions.

10. A multimodal machine learning-based drone monitoring system, used to execute the multimodal machine learning-based drone monitoring method according to any one of claims 1-9, characterized in that, include: The data acquisition module is used to acquire historical data sequences from multiple monitoring terminals on the UAV; The data comparison module is used to generate data comparison sequences for each processing time point; The risk calculation module is used to calculate single-point risk indicators and comprehensive risk indicators; The decision execution module is used to determine whether to perform fault isolation or data fusion based on comprehensive risk indicators; The results output module is used to output independent monitoring results or comprehensive monitoring results.

Images