A UAV multimodal data fusion method based on transition graph neural network

Abstract

The present application relates to a method for multimodal data fusion of an unmanned aerial vehicle based on a conversion graph neural network, comprising: constructing a graph structure based on a first global feature, a second global feature, and a third global feature; performing a first round of node updates on the nodes in the graph structure; determining the aggregation effect of the node updates through quantitative analysis and qualitative analysis, and the aggregation effect includes good or poor; when the aggregation effect is good, performing a second round of node updates on the nodes in the graph structure, and determining the aggregation effect of the second round of node updates; when the aggregation effect is poor, converting the graph structure after the first round of node updates, performing a second round of node updates on the nodes in the converted graph structure, and determining the aggregation effect of the second round of node updates; determining the node update method based on the aggregation effect of the second round of node updates to perform a third round of node updates, and after the three rounds of node updates are completed, obtaining the multimodal fusion features of the unmanned aerial vehicle.

Description

Technical Field

[0001] The present application relates to the technical field of data fusion, and in particular to a method for multimodal data fusion of unmanned aerial vehicles based on a transformation graph neural network. Background Art

[0002] With the rapid development of drone technology, multimodal data processing has become a key area, especially in scenarios with low computing power but huge data volumes. When performing tasks, drones need to simultaneously process data from multiple sensors, such as high-resolution images, videos, radar signals, etc. The fusion of these data is crucial to improving the accuracy and efficiency of mission execution. However, due to the limited computing resources of drone platforms, how to achieve efficient multimodal data fusion under limited computing power has become a technical challenge. Summary of the invention

[0003] Based on this, it is necessary to provide a UAV multimodal data fusion method based on conversion graph neural network, which includes:

[0004] S1: Extract features from the RGB images, lidar point cloud images, and infrared images collected by the drone to obtain the first global feature, the second global feature, and the third global feature respectively;

[0005] S2: constructing a graph structure based on the first global feature, the second global feature, and the third global feature;

[0006] S3: Perform the first round of node updates on the nodes in the graph structure;

[0007] S4: Determine the aggregation effect of node updates through quantitative analysis and qualitative analysis, and the aggregation effect includes good or poor;

[0008] When the aggregation effect is good, a second round of node updates is performed on the nodes in the graph structure, and the aggregation effect of the second round of node updates is determined;

[0009] When the aggregation effect is poor, the graph structure after the first round of node update is converted, the nodes in the converted graph structure are updated in the second round, and the aggregation effect of the second round of node update is determined;

[0010] S5: Based on the aggregation effect of the second round of node update, the node update method is determined to perform the third round of node update. After the three rounds of node update are completed, the multimodal fusion features of the drone are obtained.

[0011] Beneficial effects: This method fuses the features of three different modalities collected by drones based on the transformation of graph structure, providing an effective solution for drones to fuse large amounts of multimodal data in resource-constrained environments, and is expected to promote the further development and application of drone technology in various fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0013] Figure 1 This is a flowchart of a method for fusion of multimodal data of unmanned aerial vehicles based on a transformation graph neural network according to an embodiment of the present application. DETAILED DESCRIPTION

[0014] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.

[0015] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0016] Due to its advantages in processing graph structured data, graph neural networks (GNNs) have become an effective tool for solving the technical problem of how to achieve efficient multimodal data fusion under limited computing power, especially the Transformation Graph Neural Network (T-GNN), which combines the graph structure processing capabilities of GNNs and the self-attention mechanism of converters to more deeply understand and process multimodal data. The design of T-GNN takes into account the low computing power characteristics of drones. By optimizing the algorithm and model structure, it can operate efficiently in a resource-constrained environment while maintaining the ability to deeply analyze multimodal data. In addition, the real-time and near real-time processing capabilities of T-GNN meet the needs of drone missions for rapid response.

[0017] like Figure 1 As shown, this embodiment provides a method for fusion of multimodal data of unmanned aerial vehicles based on a transformation graph neural network, the method comprising:

[0018] S1: Perform feature extraction on the RGB images, lidar point cloud images, and infrared images collected by the drone to obtain the first global feature, the second global feature, and the third global feature, respectively.

[0019] Specifically,

[0020] Use a convolutional neural network to extract features from the RGB image and obtain the first global feature;

[0021] This step includes:

[0022] Step 1: Input image; suppose the input image is a three-dimensional tensor I, whose dimension is , where H is the image height, W is the image width, and C = 3 is the number of color channels.

[0023] Step 2: Convolutional layer; the convolutional layer consists of multiple convolution kernels Each convolution kernel slides on the input image to calculate local features. The size of is k×k, and the convolution operation can be expressed as follows:

[0024] ;

[0025] in, is the i-th channel of the output feature map at position ( i , j ), The input image is j Channels at position ( m+ 1, n+ 1) pixel value, It is i The convolution kernels are at positions ( m+ 1, n+ 1)’s pixel value.

[0026] Step 3: Activation function; apply the nonlinear activation function ReLU (Rectified Linear Unit) to the output of each convolutional layer:

[0027] ;

[0028] Step 4: Pooling layer; The pooling layer (usually maximum pooling or average pooling) is used to reduce the spatial dimension of the feature map, reduce the number of parameters and the amount of calculation. The maximum pooling operation can be expressed as:

[0029] ;

[0030] in, It is the feature map after pooling at position ( i , j ), is the output feature map k Channels at position ( k , l )’s pixel value.

[0031] Step 5: Fully connected layer; after multiple convolution and pooling layers, the feature map will be flattened and passed to the fully connected layer. Let the weight matrix of the fully connected layer be , the bias vector is , then the full connection operation is:

[0032] ;

[0033] in, It is the output of the pooling layer after the activation function.

[0034] Step 6: Global Average Pooling; In order to extract global features, global average pooling (GAP) is used after the fully connected layer:

[0035] ;

[0036] in, is the feature map size after convolution and pooling operations, H' is the image height after convolution and pooling operations, W' is the image width after convolution and pooling operations, and C' is the number of color channels after convolution and pooling operations.

[0037] Step 7: Global Features It is used as input for subsequent tasks and the first global features are transmitted back to the UAV.

[0038] PointNet is used to extract features from the lidar point cloud image to obtain the second global feature.

[0039] PointNet is a deep learning network for processing point cloud data. It can extract global features from the original point cloud regardless of the order of the points. The following are the steps to extract the second global feature using PointNet, including:

[0040] Step 1: Define the input point cloud; let the input point cloud It consists of N points, each of which Represents a point in three-dimensional space;

[0041] .

[0042] Step 2: Feature extraction; for each point Apply a learnable multi-layer perceptron (MLP) to extract the second local features;

[0043] ;

[0044] in, Represents the first i points.

[0045] Step 3: Feature transformation: Use the Transformation Network to learn a differentiable transformation to eliminate the effect of the arrangement of points in the point cloud.

[0046] ;

[0047] in, Represents a transformation network; and is a network parameter.

[0048] Step 4: Feature aggregation; transform the transformed features Aggregate through a symmetric function g, and PointNet uses maximum pooling as the symmetric function;

[0049] ;

[0050] Step 5: Global feature extraction: The aggregated features are passed through an MLP again to extract the second global feature , transmitting the second global feature back to the UAV;

[0051] .

[0052] ResNet is used to extract features from infrared images and obtain the third global feature.

[0053] This step includes:

[0054] Step 1: Input image; suppose the input infrared image I' is a three-dimensional tensor with dimension ,in is the image height, is the image width, is the number of channels (for infrared images, C'' = 1).

[0055] Step 2: Convolutional layer; in the first convolutional layer, multiple convolution kernels are used Perform convolution operation on the input image to extract the third local feature;

[0056] ;

[0057] in, O is the third local feature, ' is the bias term.

[0058] Step 3: Activation function: Apply a nonlinear activation function to the output of the convolutional layer, using the ReLU function;

[0059] .

[0060] Step 4: Residual block; The residual block is the core component of ResNet, which contains two convolutional layers, where the output of the second convolutional layer is added to the input of the first convolutional layer.

[0061] Suppose the output of the first convolutional layer is , the output of the second convolutional layer is , the output of the residual block for:

[0062] ;

[0063] in, is the bias term within the residual block.

[0064] Step 5: Pooling layer: Use a pooling layer (such as max pooling) to reduce the spatial dimension of the feature map.

[0065] ;

[0066] Step 6: Global average pooling: At the end of the network, global average pooling is used to aggregate all features and extract a one-dimensional feature vector G ;

[0067] ;

[0068] in, '' is the depth of the feature map after multiple convolution and pooling layers; Represents the value of the element at position (i, j, k) in the feature map.

[0069] Step 7: Fully connected layer: The one-dimensional feature vector after global average pooling is passed through a fully connected layer for classification or other tasks.

[0070] ;

[0071] in, and are the weights and biases of the second fully connected layer.

[0072] Step 8: Output of the output layer It can be used for classification or other tasks; the third global feature is transmitted back to the drone.

[0073] S2: Construct a graph structure based on the first global feature, the second global feature, and the third global feature.

[0074] Specifically, the step includes:

[0075] S2.1: Integrate the first global feature, the second global feature, and the third global feature into a feature matrix; the feature matrix is ​​expressed as:

[0076] ;

[0077] Where X represents the feature matrix; Represents the feature vector of the first node in the first global feature; Represents the feature vector of the second node in the first global feature; Indicates the first global feature N The feature vector of each node; Represents the feature vector of the first node in the second global feature; Represents the feature vector of the second node in the second global feature; Indicates the second global feature M The feature vector of each node; Represents the feature vector of the first node in the third global feature; Represents the feature vector of the second node in the third global feature; Indicates the third global feature O The feature vector of each node;

[0078] S2.2: Construct an adjacency matrix based on the connection relationship between each node; when two nodes are directly connected, the weight of the edge is 1; when two nodes are not directly connected, the weight of the edge is 0; the adjacency matrix is ​​expressed as:

[0079] ;

[0080] Where A represents the adjacency matrix; Represents the weight of the edge between the first node and the first node; Represents the weight of the edge between the first node and the second node; Represents the weight of the edge between the 1st node and the Vth node; Represents the weight of the edge between the second node and the first node; Represents the weight of the edge between the second node and the second node; Represents the weight of the edge between the 2nd node and the Vth node; Represents the weight of the edge between the Vth node and the 1st node; Represents the weight of the edge between the Vth node and the second node; represents the weight of the edge between the Vth node and the Vth node; V is the total number of nodes;

[0081] S2.3: Construct a degree matrix based on the sum of the weights of the edges connecting each node, where the weight of the edge is the connection relationship between two nodes; the degree matrix is ​​expressed as:

[0082] ;

[0083] ;

[0084] Where D represents the degree matrix; represents the degree of the first node; represents the degree of the second node; represents the degree of the Vth node; Indicates i The degree of each node; Indicates i The node and j The weight of the edge between nodes;

[0085] S2.4: normalizing the adjacency matrix based on the degree matrix to obtain a normalized adjacency matrix; the calculation formula is:

[0086] ;

[0087] in, represents the normalized adjacency matrix;

[0088] In this embodiment, S2.4 also includes:

[0089] When i When the degree of a node is 0, it is a node i Add self-loops to update the degree matrix; the update formula is:

[0090] ;

[0091] in, Represents the updated degree matrix i The degree of a node; max(·) means taking the maximum value; Indicates i The degree of each node;

[0092] Normalizing the adjacency matrix based on the updated degree matrix to obtain a second normalized adjacency matrix;

[0093] The second normalized connectivity matrix, the updated degree matrix, and the feature matrix are combined to obtain the graph structure.

[0094] S2.5: Combine the normalized adjacency matrix, the degree matrix and the feature matrix to obtain the graph structure.

[0095] S3: Perform the first round of node updates on the nodes in the graph structure.

[0096] Specifically, the step includes:

[0097] S3.1: Based on the normalized adjacency matrix, the initial feature matrix is ​​aggregated to obtain an aggregated feature matrix; the calculation formula is:

[0098] ;

[0099] in, The feature matrix representing the aggregation; represents the normalized adjacency matrix; represents the initial feature matrix;

[0100] S3.2: The aggregated feature matrix is ​​transformed through a graph convolution kernel and a bias is added to obtain a transformed feature matrix; the calculation formula is:

[0101] ;

[0102] in, The feature matrix representing the transformation; Represents the graph convolution kernel; Indicates bias;

[0103] S3.3: Perform nonlinear activation on the transformed feature matrix to obtain an activated feature matrix; the calculation formula is:

[0104] ;

[0105] in, represents the activated feature matrix; represents a nonlinear activation function;

[0106] S3.4: learning the importance weight vector of each node in the activated feature matrix; updating the feature vector of the node in the feature matrix based on the feature vector and the importance weight vector of the node in the activated feature matrix; the calculation formula is:

[0107] ;

[0108] ;

[0109] in, Represents the updated feature matrix i The feature vector of each node; Represents the activated feature matrix i The importance weight vector of each node; Represents the activated feature matrix i The feature vector of each node; represents the Hadamard product;

[0110] S3.5: Integrate all updated eigenvectors to obtain an updated eigenmatrix; the updated eigenmatrix is ​​expressed as:

[0111] ;

[0112] in, represents the updated feature matrix; Represents the eigenvector of the first node in the updated feature matrix; Represents the eigenvector of the second node in the updated feature matrix; Represents the updated feature matrix N The feature vector of each node.

[0113] S4: Determine the aggregation effect of node updates through quantitative analysis and qualitative analysis, and the aggregation effect includes good or poor;

[0114] When the aggregation effect is good, a second round of node updates is performed on the nodes in the graph structure, and the aggregation effect of the second round of node updates is determined;

[0115] When the aggregation effect is poor, the graph structure after the first round of node update is converted, the nodes in the converted graph structure are updated in the second round, and the aggregation effect of the second round of node update is determined.

[0116] Specifically, determining the aggregation effect of node update through quantitative analysis and qualitative analysis includes:

[0117] Step 1: For any node in the updated feature matrix, calculate the standard deviation of its updated feature vector; the calculation formula is:

[0118] ;

[0119] ;

[0120] in, Indicated in t After the round update i The standard deviation of the feature vector of each node; Indicated in t After the round update i The mean of the feature vectors of nodes; N Indicates the number of nodes; Indicated in t After the round of update, the feature matrix j The feature vector of each node;

[0121] Step 2: Calculate the aggregation coefficient based on the standard deviation of the updated feature vector; the calculation formula is:

[0122] ;

[0123] in, Indicated in t After the round update i The aggregation coefficient of nodes; Indicates that all nodes in t The minimum value of the standard deviation after the round node is updated;

[0124] Calculate the cosine similarity between the updated node and the previous round of updates; the calculation formula is:

[0125] ;

[0126] in, Indicates i Nodes in t After the round update t - Cosine similarity after 1 round of updates; Indicated in t After the round update i The feature vector of each node; Indicated in t- After 1 round of update i The feature vector of each node; represents the dot product; represents the Euclidean norm;

[0127] Calculate the information entropy of the node; the calculation formula is:

[0128] ;

[0129] ;

[0130] in, Indicated in t After the round update i The information entropy of nodes; F represents the dimension of the feature vector; Indicated in i The node f The probability of taking a specific value on a feature dimension; Indicated in t After the round update i The feature vector of the node j Elements on the feature dimension; Indicated in t After the round update i The feature vector of the node jThe elements on the feature dimension are f The number of elements in each feature dimension is equal;

[0131] Step 3: Calculate a weighted evaluation index based on the aggregation coefficient, the cosine similarity and the information entropy; the calculation formula is:

[0132] ;

[0133] in, Indicates i Weighted evaluation index of each node; represents the first weight; represents the second weight; represents the third weight; Indicates i The aggregation coefficient of nodes; Indicates i The cosine similarity of nodes before and after update; Indicates i The information entropy of each node;

[0134] Step 4: normalizing the weighted evaluation index;

[0135] ;

[0136] in, After normalization, i Weighted evaluation index of each node; Represents the weighted evaluation index of all nodes;

[0137] Step 5: Set a judgment threshold, compare the normalized weighted judgment index with the judgment threshold, and determine the aggregation effect; the calculation formula is:

[0138] ;

[0139] in, Indicates i The aggregation effect of nodes is When When , the polymerization effect is poor; Indicates the threshold value.

[0140] Furthermore, the converting of the graph structure after the first round of node update includes:

[0141] Step 1: Transform and update the adjacency matrix in the graph structure after the first round of node update to obtain an updated adjacency matrix;

[0142] Step 2: Introduce a virtual node for each node in the adjacency matrix after the first round of node update;

[0143] Step 3: Calculate a new degree matrix based on the sum of the connection relationships in the updated adjacency matrix;

[0144] Step 4: Use the new degree matrix to normalize the updated adjacency matrix. The calculation formula is:

[0145] ;

[0146] in, represents the updated adjacency matrix after normalization; represents the new degree matrix; represents the updated adjacency matrix;

[0147] Step 5: Create an extended adjacency matrix based on the connection relationships between nodes and the connection relationships between virtual nodes, and calculate a second degree matrix based on the sum of all the connection relationships in the extended adjacency matrix;

[0148] Step 6: Combine the normalized updated adjacency matrix, the new degree matrix, the expanded adjacency matrix, the second degree matrix, and the updated feature matrix to obtain the transformed graph structure.

[0149] Furthermore, the step of converting and updating the adjacency matrix in the graph structure after the first round of node update to obtain an updated adjacency matrix includes:

[0150] Step 1: Convert all weights of the adjacency matrix in the graph structure after the first round of node update to obtain the converted adjacency matrix; the calculation formula is:

[0151] ;

[0152] ;

[0153] in, The adjacency matrix of the transformation i The node and j The weight of the edge between nodes; Represents the first round of node update in the adjacency matrix i The node and j The weight of the edge between nodes; represents the conversion function; w Represents the connection relationship in the new adjacency matrix; Indicates a positive number; represents the first adjustment parameter; represents the second adjustment parameter; represents the third adjustment parameter;

[0154] Step 2: directly connecting the nodes in the converted adjacency matrix to form an indirect connection by connecting the corresponding one or more virtual nodes, and assigning a fourth weight; converting the indirect connection between the nodes in the converted adjacency matrix to a direct connection, and assigning a fifth weight;

[0155] In this embodiment, the weights of the original edges in the adjacency matrix are sorted from large to small and normalized to the interval [0,1]; in the interval [0,1], the original edge weight value corresponding to 0.6 is taken as the fourth weight, and the original edge weight value corresponding to 0.3 is taken as the fifth weight; if 0.6 and 0.3 do not have an exact corresponding original edge weight value, then the weight value is selected one position backward according to the weight sorting; for example, there are 12 weight values, sorted as 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1; at this time, 0.6 has no corresponding weight value after normalization, among which 7 corresponds to 0.5455 after normalization; 8 is normalized to 0.6364, at this time 0.6 has no corresponding weight value, but is between 7 and 8, at this time, according to the sorting, 8 is selected as the weight value, that is, it is postponed one position; 0.3 is the same as above.

[0156] Step 3: Repeat step 2 until all connection relationships in the adjacency matrix are updated to obtain an updated adjacency matrix.

[0157] S5: Based on the aggregation effect of the second round of node update, the node update method is determined to perform the third round of node update. After the three rounds of node update are completed, the multimodal fusion features of the drone are obtained.

[0158] Specifically, when the aggregation effect of the second round of node update is good, a third round of node update is performed on the nodes in the graph structure after the second round of node update;

[0159] When the aggregation effect of the second round of node update is not good, the graph structure after the second round of node update is converted twice, and the nodes in the graph structure after the second conversion are updated for the third round;

[0160] After three rounds of node updates are completed, the multimodal fusion features of the UAV are obtained.

[0161] The UAV multimodal data fusion method provided in this embodiment fuses the features of three different modalities collected by the UAV based on the transformation of the graph structure, providing an effective solution for UAVs to fuse a large amount of multimodal data in resource-constrained environments, and is expected to promote the further development and application of UAV technology in various fields.

[0162] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0163] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A UAV multimodal data fusion method based on conversion graph neural network, characterized in that: include: S1: Extract features from the RGB images, lidar point cloud images, and infrared images collected by the drone to obtain the first global feature, the second global feature, and the third global feature respectively; S2: constructing a graph structure based on the first global feature, the second global feature, and the third global feature; S2.1: Integrate the first global feature, the second global feature, and the third global feature into a feature matrix; the feature matrix is ​​expressed as: ； Where X represents the feature matrix; Represents the feature vector of the first node in the first global feature; Represents the feature vector of the second node in the first global feature; Indicates the first global feature N The feature vector of each node; Represents the feature vector of the first node in the second global feature; Represents the feature vector of the second node in the second global feature; Indicates the second global feature M The feature vector of each node; Represents the feature vector of the first node in the third global feature; Represents the feature vector of the second node in the third global feature; Indicates the third global feature O The feature vector of each node; S2.2: Construct an adjacency matrix based on the connection relationship between each node; when two nodes are directly connected, the weight of the edge is 1; when two nodes are not directly connected, the weight of the edge is 0; the adjacency matrix is ​​expressed as: ； Where A represents the adjacency matrix; Represents the weight of the edge between the first node and the first node; Represents the weight of the edge between the first node and the second node; Represents the weight of the edge between the 1st node and the Vth node; Represents the weight of the edge between the second node and the first node; Represents the weight of the edge between the second node and the second node; Represents the weight of the edge between the 2nd node and the Vth node; Represents the weight of the edge between the Vth node and the 1st node; Represents the weight of the edge between the Vth node and the second node; represents the weight of the edge between the Vth node and the Vth node; V is the total number of nodes; S2.3: Construct a degree matrix based on the sum of the weights of the edges connecting each node; the degree matrix is ​​expressed as: ； ； Where D represents the degree matrix; Indicates the degree of the first node; represents the degree of the second node; represents the degree of the Vth node; Indicates i The degree of each node; Indicates i The node and j The weight of the edge between nodes; S2.4: normalizing the adjacency matrix based on the degree matrix to obtain a normalized adjacency matrix; the calculation formula is: ； in, represents the normalized adjacency matrix; S2.5: combining the normalized adjacency matrix, the degree matrix and the feature matrix to obtain the graph structure; S3: Perform the first round of node updates on the nodes in the graph structure; S3.1: Based on the normalized adjacency matrix, the initial feature matrix is ​​aggregated to obtain an aggregated feature matrix; the calculation formula is: ； in, The feature matrix representing the aggregation; represents the normalized adjacency matrix; represents the initial feature matrix; S3.2: The aggregated feature matrix is ​​transformed through a graph convolution kernel and a bias is added to obtain a transformed feature matrix; the calculation formula is: ； in, The feature matrix representing the transformation; Represents the graph convolution kernel; Indicates bias; S3.3: Perform nonlinear activation on the transformed feature matrix to obtain an activated feature matrix; the calculation formula is: ； in, represents the activated feature matrix; represents a nonlinear activation function; S3.4: learning the importance weight vector of each node in the activated feature matrix; updating the feature vector of the node in the feature matrix based on the feature vector and the importance weight vector of the node in the activated feature matrix; the calculation formula is: ； ； in, Represents the updated feature matrix i The feature vector of each node; Represents the activated feature matrix i The importance weight vector of each node; Represents the activated feature matrix i The feature vector of each node; represents the Hadamard product; S3.5: Integrate all updated eigenvectors to obtain an updated eigenmatrix; the updated eigenmatrix is ​​expressed as: ； in, represents the updated feature matrix; Represents the eigenvector of the first node in the updated feature matrix; Represents the eigenvector of the second node in the updated feature matrix; Represents the updated feature matrix N The feature vector of each node; S4: Determine the aggregation effect of node updates through quantitative analysis and qualitative analysis, and the aggregation effect includes good or poor; When the aggregation effect is good, a second round of node updates is performed on the nodes in the graph structure, and the aggregation effect of the second round of node updates is determined; When the aggregation effect is poor, the graph structure after the first round of node update is converted, the nodes in the converted graph structure are updated in the second round, and the aggregation effect of the second round of node update is determined; S5: Based on the aggregation effect of the second round of node update, the node update method is determined to perform the third round of node update. After the three rounds of node update are completed, the multimodal fusion features of the drone are obtained.

2. The method for multimodal data fusion of unmanned aerial vehicles based on conversion graph neural network according to claim 1 is characterized in that: S1 includes: Use a convolutional neural network to extract features from the RGB image and obtain the first global feature; Use PointNet to extract features from the lidar point cloud image and obtain the second global feature; ResNet is used to extract features from infrared images and obtain the third global feature.

3. The method for multimodal data fusion of unmanned aerial vehicles based on conversion graph neural network according to claim 1 is characterized in that: S2.4 also includes: When i When the degree of a node is 0, it is a node i Add self-loops to update the degree matrix; the update formula is: ； in, Represents the updated degree matrix i The degree of a node; max(·) means taking the maximum value; Indicates i The degree of each node; Normalizing the adjacency matrix based on the updated degree matrix to obtain a second normalized adjacency matrix; The second normalized connectivity matrix, the updated degree matrix, and the feature matrix are combined to obtain the graph structure.

4. The method for multimodal data fusion of unmanned aerial vehicles based on conversion graph neural network according to claim 1 is characterized in that: In S4, determining the aggregation effect of node update through quantitative analysis and qualitative analysis includes: Step 1: For any node in the updated feature matrix, calculate the standard deviation of its updated feature vector; the calculation formula is: ； ； in, Indicated in t After the round update i The standard deviation of the feature vector of each node; Indicated in t After the round update i The mean of the feature vectors of nodes; N Indicates the number of nodes; Indicated in t After the round of update, the feature matrix j The feature vector of each node; Step 2: Calculate the aggregation coefficient based on the standard deviation of the updated feature vector; the calculation formula is: ； in, Indicated in t After the round update i The aggregation coefficient of nodes; Indicates that all nodes in t The minimum value of the standard deviation after the round node is updated; Calculate the cosine similarity between the updated node and the previous round of updates; the calculation formula is: ； in, Indicates i Nodes in t After the round update t - Cosine similarity after 1 round of updates; Indicated in t After the round update i The feature vector of each node; Indicated in t- After 1 round of update i The feature vector of each node; represents the Euclidean norm; Calculate the information entropy of the node; the calculation formula is: ； ； in, Indicated in t After the round update i The information entropy of nodes; F represents the dimension of the feature vector; Indicated in i The node f The probability of taking a specific value on a feature dimension; Indicated in t After the round update i The feature vector of the node j Elements on the feature dimension; Indicated in t After the round update i The feature vector of the node j The elements on the feature dimension are f The number of elements in each feature dimension is equal; Step 3: Calculate a weighted evaluation index based on the aggregation coefficient, the cosine similarity and the information entropy; the calculation formula is: ； in, Indicates i Weighted evaluation index of each node; represents the first weight; represents the second weight; represents the third weight; Indicates i The aggregation coefficient of nodes; Indicates i The cosine similarity of nodes before and after update; Indicates i The information entropy of each node; Step 4: normalizing the weighted evaluation index; ； in, After normalization, i Weighted evaluation index of each node; Represents the weighted evaluation index of all nodes; Step 5: Set a judgment threshold, compare the normalized weighted judgment index with the judgment threshold, and determine the aggregation effect; the calculation formula is: ； in, Indicates i The aggregation effect of nodes is When When , the polymerization effect is poor; Indicates the threshold value.

5. The method for multimodal data fusion of unmanned aerial vehicles based on conversion graph neural network according to claim 1 is characterized in that: In S4, converting the graph structure after the first round of node update includes: Step 1: Transform and update the adjacency matrix in the graph structure after the first round of node update to obtain an updated adjacency matrix; Step 2: Introduce a virtual node for each node in the adjacency matrix after the first round of node update; Step 3: Calculate a new degree matrix based on the sum of the connection relationships in the updated adjacency matrix; Step 4: Use the new degree matrix to normalize the updated adjacency matrix; Step 5: Create an extended adjacency matrix based on the connection relationships between nodes and the connection relationships between virtual nodes, and calculate a second degree matrix based on the sum of all the connection relationships in the extended adjacency matrix; Step 6: Combine the normalized updated adjacency matrix, the new degree matrix, the expanded adjacency matrix, the second degree matrix, and the updated feature matrix to obtain the transformed graph structure.

6. The method for multimodal data fusion of unmanned aerial vehicles based on conversion graph neural network according to claim 5 is characterized in that: The step of converting and updating the adjacency matrix in the graph structure after the first round of node update to obtain an updated adjacency matrix includes: Step 1: Convert all weights of the adjacency matrix in the graph structure after the first round of node update to obtain the converted adjacency matrix; the calculation formula is: ； ； in, The adjacency matrix of the transformation i The node and j The weight of the edge between nodes; Represents the first round of node update in the adjacency matrix i The node and j The weight of the edge between nodes; represents the conversion function; w Represents the connection relationship in the new adjacency matrix; Indicates a positive number; represents the first adjustment parameter; represents the second adjustment parameter; represents the third adjustment parameter; Step 2: directly connecting the nodes in the converted adjacency matrix to form an indirect connection by connecting the corresponding one or more virtual nodes, and assigning a fourth weight; converting the indirect connection between the nodes in the converted adjacency matrix to a direct connection, and assigning a fifth weight; Step 3: Repeat step 2 until all connection relationships in the adjacency matrix are updated to obtain an updated adjacency matrix.

7. The method for multimodal data fusion of unmanned aerial vehicles based on conversion graph neural network according to claim 6 is characterized in that: The calculation formula of the adjacency matrix updated by normalization with the new degree matrix is: ； in, represents the updated adjacency matrix after normalization; represents the new degree matrix; represents the updated adjacency matrix.

8. The method for multimodal data fusion of unmanned aerial vehicles based on conversion graph neural network according to claim 1, characterized in that S5 include: When the aggregation effect of the second round of node updates is good, a third round of node updates is performed on the nodes in the graph structure after the second round of node updates; When the aggregation effect of the second round of node update is not good, the graph structure after the second round of node update is converted twice, and the nodes in the graph structure after the second conversion are updated for the third round; After three rounds of node updates are completed, the multimodal fusion features of the UAV are obtained.

Images