Unmanned aerial vehicle sensing data processing method, device and equipment and readable storage medium

Abstract

The invention discloses an unmanned aerial vehicle sensing data processing method, device and equipment and a readable storage medium, and belongs to the field of data processing. The method comprises the steps of obtaining sensing data of different sensors of an unmanned aerial vehicle; constructing a local feature map of each sensor based on the sensing data of each sensor of the unmanned aerial vehicle, wherein the local feature map comprises feature points of the sensing data and feature indexes for identifying the positions of the feature points; according to the matching feature points between the different local feature maps and the feature indexes of the positions corresponding to the matching feature points, determining a spatial transformation matrix between the perception data corresponding to the different local feature maps through minimizing a re-projection error; spatial registration is carried out based on the spatial transformation matrix between the sensing data to obtain the multi-modal data, the feature points in the multi-modal data comprise the sensing data of different sensors at the same position so as to determine the target decision according to the multi-modal data, and the reliability of the processing decision obtained based on semantic information can be improved.

Description

Technical Field

[0001] The present application belongs to the field of data processing, and in particular, relates to a method, device, equipment, computer-readable storage medium and computer program product for processing drone perception data. Background Art

[0002] With the rapid development of drone technology, drones are increasingly used in military and civilian fields. Drones are usually equipped with various types of sensors, such as visible light cameras, infrared cameras, radars, etc., to achieve environmental perception and information collection. However, the massive heterogeneous drone perception data poses a huge challenge to the real-time processing and intelligent analysis of data.

[0003] In the existing UAV perception data processing method, multiple analysis results are obtained by performing data analysis on heterogeneous perception data of different modalities, and processing decisions are made based on the multiple analysis results. However, the correlation between the multiple analysis results is poor, so that the effectiveness and reliability of the final processing decision are low. Summary of the invention

[0004] The embodiments of the present application provide a method, apparatus, device, computer-readable storage medium, and computer program product for processing drone perception data, which can improve the effectiveness and reliability of processing decisions.

[0005] In a first aspect, an embodiment of the present application provides a method for processing drone perception data, the method comprising:

[0006] Obtain perception data from different sensors of the drone;

[0007] Based on the perception data of each sensor of the drone, a local feature map of each sensor is constructed, where the local feature map includes feature points of the perception data and feature indexes identifying the locations of the feature points;

[0008] According to the matching feature points between different local feature maps and the feature indexes of the positions corresponding to the matching feature points, the spatial transformation matrix between the perception data corresponding to the different local feature maps is determined by minimizing the reprojection error;

[0009] Multimodal data is obtained by spatial registration based on the spatial transformation matrix between the perception data. The feature points in the multimodal data include the perception data of different sensors at the same position, so as to determine the target decision according to the multimodal data.

[0010] In an achievable implementation, after obtaining multimodal data by spatial registration based on a spatial transformation matrix between the perception data, wherein the feature points in the multimodal data include perception data of different sensors at the same position, the method further includes:

[0011] Based on the Kalman filter, the perception data of the feature points of the multimodal data are fused to obtain the local environment representation data corresponding to the multimodal data, so as to determine the target decision according to the multimodal data and the local environment representation corresponding to the multimodal data.

[0012] In a feasible implementation, the feature points of the perception data include at least one of feature information of the collected object and information of the sensor.

[0013] In an achievable implementation, after obtaining multimodal data by spatial registration based on a spatial transformation matrix between the perception data, wherein the feature points in the multimodal data include perception data of different sensors at the same position, the method further includes:

[0014] The graph neural network model is used to perform semantic extraction on multimodal data to obtain semantic information, which is used to determine target decisions based on the semantic information.

[0015] In an achievable implementation, after obtaining multimodal data by spatial registration based on a spatial transformation matrix between the perception data, wherein the feature points in the multimodal data include perception data of different sensors at the same position, the method further includes:

[0016] The multimodal data is sent to the target processing terminal so that the target processing terminal can use the graph neural network model to perform semantic extraction on the multimodal data to obtain semantic information and determine the target decision based on the semantic information.

[0017] In one practicable implementation, sending multimodal data to a target processing terminal includes:

[0018] The multimodal data are sent to the target processing terminal according to the target data transmission rate and priority, and the target data transmission rate and priority are determined according to the importance information of each multimodal data and the network load information of the target processing terminal.

[0019] In an achievable implementation, the method is applied to a local processing unit; before sending the multimodal data to a target processing terminal, the method further includes:

[0020] Obtain computing resource information of a local processing unit;

[0021] According to the computing resource information of the local processing unit and the data computing amount of the multimodal data, it is determined whether to send the multimodal data to the target processing terminal.

[0022] In a second aspect, an embodiment of the present application provides a drone perception data processing device, the device comprising:

[0023] The acquisition module is used to obtain the perception data of different sensors of the drone;

[0024] A construction module, used to construct a local feature map of each sensor based on the perception data of each sensor of the drone, wherein the local feature map includes feature points of the perception data and feature indexes identifying positions of the feature points;

[0025] A determination module, for determining a spatial transformation matrix between perception data corresponding to different local feature maps by minimizing a reprojection error based on matching feature points between different local feature maps and feature indexes of positions corresponding to the matching feature points;

[0026] The registration module is used to obtain multimodal data by performing spatial registration based on the spatial transformation matrix between the perception data. The feature points in the multimodal data include the perception data of different sensors at the same position, so as to determine the target decision based on the multimodal data.

[0027] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement a method for processing drone perception data as in any one of the embodiments of the first aspect.

[0028] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, a method for processing drone perception data as in any one of the embodiments of the first aspect is implemented.

[0029] In a fifth aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes a drone perception data processing method as in any one of the embodiments of the first aspect.

[0030] The present invention discloses a method, device, equipment, computer-readable storage medium and computer program product for processing perception data of a drone in an embodiment of the present invention. The method first obtains the perception data of different sensors of the drone, and constructs a local feature map of each sensor based on the perception data of each sensor of the drone. The local feature map includes feature points of the perception data and feature indexes of the positions of the feature points. Then, according to the matching feature points between different local feature maps and the feature indexes of the positions corresponding to the matching feature points, the spatial transformation matrix between the perception data corresponding to different local feature maps is determined by minimizing the reprojection error. Finally, spatial registration is performed based on the spatial transformation matrix between the perception data to obtain multimodal data. The feature points in the multimodal data include the perception data of different sensors at the same position, so as to determine the target decision according to the multimodal data. By constructing the local feature maps of each sensor of the drone, and determining the spatial transformation matrix between the data of different sensors by matching feature points and minimizing the reprojection error, the spatial registration of the multimodal data is realized, and the perception data from different sensors can be effectively integrated to form a unified multimodal data set. The environmental information can be comprehensively understood from multiple perspectives, so as to be used for determining the final processing decision based on the processing analysis of the registered multimodal data, thereby improving the effectiveness and reliability of the target decision made based on the multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0032] Figure 1 It is a flowchart of a method for processing drone perception data provided by an embodiment of the present application;

[0033] Figure 2 This is a schematic diagram of a multi-sensor perception data fusion process provided by an embodiment of the present application;

[0034] Figure 3 It is a flowchart of a semantic extraction method based on multimodal data provided by an embodiment of the present application;

[0035] Figure 4 It is a flowchart of a cloud-edge-device collaborative processing mechanism provided by an embodiment of the present application;

[0036] Figure 5 This is a system architecture diagram of a method for processing drone perception data provided by an embodiment of the present application;

[0037] Figure 6 is a flowchart of a method for processing drone perception data provided by another embodiment of the present application;

[0038] Figure 7 is a structural block diagram of a drone perception data processing device provided by an embodiment of the present application;

[0039] Figure 8 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0040] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0041] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0042] With the rapid development of drone technology, drones are increasingly used in military and civilian fields. Drones are usually equipped with various types of sensors, such as visible light cameras, infrared cameras, radars, etc., to achieve environmental perception and information collection. However, the massive heterogeneous drone perception data poses a huge challenge to the real-time processing and intelligent analysis of data. For example, drones have become an important tool for post-disaster rescue operations due to their flexibility and efficient perception capabilities. In order to achieve accurate search and rescue work, drones need to obtain multi-modal perception data of the environment in real time, such as visible light images, infrared thermal imaging data, and three-dimensional maps generated by LiDAR. These data not only need to be processed quickly, but also must be fused between different sensors to form a complete environmental cognition.

[0043] In the existing UAV perception data processing methods, multiple analysis results are obtained by performing data analysis on heterogeneous perception data of different modalities, and processing decisions are made based on the multiple analysis results. However, the fusion processing of UAV multi-sensor data is not considered, and it is difficult to fully explore the complementarity and correlation of data of different modalities. The ability to extract semantic information is limited, and the correlation between multiple analysis results is poor, so that the effectiveness and reliability of the final processing decision are low.

[0044] In order to solve the problems of the prior art, the embodiments of the present application provide a method, device, equipment, computer storage medium and computer program product for processing drone perception data. The following first introduces the drone perception data processing method provided in the embodiments of the present application.

[0045] The advanced version of the fifth generation mobile communication technology (Fifth-generation Advanced, 5G-A) is an evolutionary stage of 5G technology. It enhances the existing 5G network performance and service quality by introducing new functions and technical improvements. 5G-A focuses on providing higher data transmission rates, larger system capacity, lower latency, and more reliable connectivity. It supports more application scenarios, such as Massive IoT, Ultra-Reliable Low Latency Communications (URLLC), etc.

[0046] Figure 1 FIG. 1 is a flow chart of a method for processing drone perception data provided by an embodiment of the present application. Figure 1 As shown, the method includes steps S110 to S140.

[0047] S110: Obtain the perception data of different sensors of the drone.

[0048] Obtain environmental perception data collected in real time by the drone's onboard multimodal sensors according to the sensor's configuration parameters. The sensors may include visible light cameras, infrared cameras, radars, LiDAR, etc. The collected perception data may include visible light images, infrared thermal imaging data, and three-dimensional maps generated by LiDAR.

[0049] In one embodiment, environmental perception data collected in real time by a multimodal sensor onboard a drone according to the configuration parameters of the sensor is obtained, and the perception data is preprocessed. The preprocessing may be processing the noise data captured by the sensor through a filtering algorithm. Exemplarily, filtering is used to remove noise points in a visible light image to improve data quality.

[0050] S120: constructing a local feature map of each sensor based on the perception data of each sensor of the drone, where the local feature map includes feature points of the perception data and feature indexes identifying positions of the feature points.

[0051] The local feature map of each sensor is constructed based on the data features of the perception data of each sensor of the drone. Data features refer to information extracted directly from the original perception data that can represent certain attributes or characteristics of the data. Data features can include color, texture, edges, etc., which are used to describe the basic properties of the data, help identify objects, classify, and conduct deeper data analysis. Texture refers to a pattern on the surface of an image, which reflects the spatial distribution characteristics of color or intensity on the surface of the material. Texture features can help distinguish different materials or backgrounds.

[0052] The local feature map includes feature points of the perceived data and feature indexes that identify the locations of the feature points. Feature points are key points, key locations or areas with specific meanings and their basic attribute information that are further screened out based on the data features. They are usually points that remain relatively stable under different viewing angles, such as corner points, mutation points on edges, etc. Each feature point corresponds to a feature index, which is the unique identifier of this feature point and can be used to quickly locate the feature point when processing a large amount of data. At the same time, the feature index is also associated with the location information of the feature point. For example, for an image, it may be pixel coordinates, and for a point cloud in three-dimensional space, it may be spatial coordinates. Each feature point also includes a feature descriptor, which is a quantitative representation of the characteristics of the local area around the feature point, which enables the feature point to be uniquely identified and matched in different images or data sets.

[0053] For example, for each sensor i of the drone, the data features of the sensor data collected are extracted and a local feature map is constructed. The extracted features can be expressed as Where j is the index of the feature point, the local feature map can be expressed as: Among them, N i is the number of feature points of sensor i.

[0054] S130: Determine the spatial transformation matrix between the perception data corresponding to different local feature maps by minimizing the reprojection error according to the matching feature points between different local feature maps and the feature indexes of the positions corresponding to the matching feature points.

[0055] The matching feature points between different local feature maps are determined based on the similarity between the feature descriptors of the feature points in the local feature maps corresponding to the perception data of different sensors and the feature descriptors of the feature points in other local feature maps.

[0056] Based on the matched feature points in the two local feature maps and the feature indexes corresponding to the matched feature points that identify the spatial positions, the spatial transformation matrix between the two local feature maps is determined by minimizing the reprojection error. The specific implementation method may include the following methods:

[0057] Using the graph optimization algorithm, the spatial transformation relationship between different sensors is determined by feature matching. Assume that the spatial transformation between sensors i and j is T ij ∈SE(3), then the graph optimization goal is to minimize the reprojection error:

[0058]

[0059] Among them, N ij is the number of feature point matches between sensors i and j, ρ(·) is the robust kernel function, To measure covariance, T ij is the spatial transformation matrix between sensors i and j, and are the feature matching points between sensors i and j respectively. By calculating the minimum reprojection error between sensors i and j, the spatial transformation matrix T between sensors i and j is determined ij .

[0060] S140: Perform spatial registration based on the spatial transformation matrix between the perception data to obtain multimodal data, wherein the feature points in the multimodal data include perception data of different sensors at the same position, so as to determine the target decision according to the multimodal data.

[0061] Feature points are matched based on the feature descriptors of the feature points in the local feature map corresponding to the perception data of each sensor with the feature points in other local feature maps, and spatial alignment is performed based on the matched feature points in the two local feature maps and the feature indexes corresponding to the matched feature points that identify the spatial positions. The spatial positions identified by the feature indexes corresponding to the matched feature points in the two local feature maps are uniformly calibrated. The perception data of all sensors are converted into a spatial matrix through the above method. The perception data associated with the feature points corresponding to each feature index after spatial alignment includes the perception data associated with the feature points corresponding to the feature index that identifies the same spatial position in each local feature map. By further processing and analyzing the multimodal data, the final target decision can be determined based on the multimodal data.

[0062] For example, the image size captured by the visible light camera is 1920x1080 pixels, the thermal imaging data captured by the infrared camera is 640x480 pixels, and the point cloud data generated by the LiDAR contains 100,000 points. After spatial calibration, all perception data are uniformly mapped to the same spatial coordinate system for subsequent processing.

[0063] This embodiment first obtains the perception data of different sensors of the drone, and constructs a local feature map of each sensor based on the perception data of each sensor of the drone, wherein the local feature map includes feature points of the perception data and feature indexes of the positions of the identification feature points; then, according to the matching feature points between different local feature maps and the feature indexes of the positions corresponding to the matching feature points, the spatial transformation matrix between the perception data corresponding to different local feature maps is determined by minimizing the reprojection error; finally, spatial registration is performed based on the spatial transformation matrix between the perception data to obtain multimodal data, wherein the feature points in the multimodal data include the perception data of different sensors at the same position, so as to determine the target decision according to the multimodal data. By constructing the local feature maps of each sensor of the drone, and determining the spatial transformation matrix between different sensor data by matching feature points and minimizing the reprojection error, the spatial registration of multimodal data is realized, and the perception data from different sensors can be effectively integrated to form a unified multimodal data set, so as to comprehensively understand the environmental information from multiple perspectives, so as to determine the final processing decision based on the processing analysis of the registered multimodal data, and improve the effectiveness and reliability of the target decision made based on the multimodal data.

[0064] As another implementation of this application, Figure 2 FIG. 1 is a schematic diagram of a multi-sensor sensing data fusion process provided by an embodiment of the present application. Figure 2 As shown, in step S140: after performing spatial registration based on the spatial transformation matrix between the perception data to obtain multimodal data, and the feature points in the multimodal data include perception data of different sensors at the same position, the method further includes:

[0065] Based on the feature points in the multimodal data, including the time deviation between the time of the perception data of different sensors at the same position and the real time, the perception data corresponding to each feature index after spatial registration is time synchronized to obtain the multimodal data after time registration. The feature points in the multimodal data after spatial and temporal registration include the perception data of different sensors at the same position at different times.

[0066] Sensor time offset refers to the time difference between different sensors in a multi-sensor system due to inconsistent sampling time or data processing delay. The time offset of different sensors can be estimated through a distributed clock synchronization protocol to achieve time synchronization.

[0067] Based on the time deviation between the time when each sensor collects the perception data and the real time, the clock drift and offset of each sensor are determined, and the time of the perception data collected by different sensors is synchronized. Then, the perception data corresponding to each feature index after registration, that is, the perception data associated with the feature points corresponding to the feature index identifying the same spatial position in each local feature map, can be time synchronized to obtain multimodal data. The feature points in the multimodal data include the perception data of different sensors at the same spatial position and different times.

[0068] For example, the multimodal data after spatial registration and time synchronization fusion of the perception data of different sensors may contain a three-dimensional environmental model, in which the temperature value of each spatial point in the model comes from the infrared camera, the position coordinates come from the LiDAR, and the color information comes from the visible light camera. Time synchronization is achieved by estimating the time offset of different sensors through the distributed clock synchronization protocol. The specific implementation method can be: Assume that the clock model of sensor i is: t i =α i t+β i , where t is the real time, α i and β i They are clock drift and offset respectively. Using maximum likelihood estimation, we can find the optimal clock parameters:

[0069]

[0070] in, is the set of neighbor sensors of sensor i, i.e., sensors other than sensor i carried on the same UAV; t ij k N is the timestamp measurement value of the kth observation of the same event by sensors i and j; ij The number of times the same event is observed between sensors i and j; is the measurement covariance between sensors i and j; α i and β i The clock parameters to be solved represent clock drift and offset respectively.

[0071] Using maximum likelihood estimation, the numerical optimization method is used to iteratively optimize formula (1) until convergence to obtain the optimal clock parameter α i and β i , the clock parameter α is obtained i and β i Applied to all sensors to achieve global time synchronization.

[0072] This embodiment obtains multimodal data after spatial and temporal registration by performing time synchronization on the spatially registered data, thereby effectively integrating the perception data of different sensors at different time points and improving the accuracy of decisions made based on multimodal data.

[0073] As another implementation of this application, Figure 2 As shown, in step S140: after performing spatial registration based on the spatial transformation matrix between the perception data to obtain multimodal data, and the feature points in the multimodal data include perception data of different sensors at the same position, the method further includes:

[0074] Based on the Kalman filter, the perception data of the feature points of the multimodal data are fused to obtain the local environment representation data corresponding to the multimodal data, so as to determine the target decision according to the multimodal data and the local environment representation corresponding to the multimodal data.

[0075] The multimodal data obtained by spatially registering the perception data of multiple sensors on the drone, or the multimodal data obtained by preliminary fusion of spatial registration and time synchronization, are denoised by using Kalman filtering on the perception data of the feature points of the multimodal data, and further data fusion is performed to obtain the multimodal fusion data and the local environment representation corresponding to the multimodal data. By further processing and analyzing the multimodal fusion data and the local environment representation corresponding to the multimodal data, the final target decision can be determined based on the multimodal fusion data and the local environment representation corresponding to the multimodal data.

[0076] The specific implementation method of using Kalman filtering to reduce the noise of the perception data of the feature points of multimodal data and further perform data fusion to obtain multimodal fusion data and the local environment representation corresponding to the multimodal data is as follows: Kalman filtering mainly includes two steps: prediction and update. The prediction process is shown in formula (3), and the update process is shown in formula (4).

[0077] predict:

[0078] renew:

[0079] Among them, x is the state quantity, which represents the system state estimation, and may include information such as the target position observed by the system; z i is the observed quantity, which can form a consistent observation value on a time and space basis for the data from each sensor after spatial registration and time synchronization; F is the state transfer matrix, which describes the relationship between the system state and time, such as predicting the state at the next moment based on the physical motion model; Q is the process noise covariance, which characterizes the uncertainty of the system dynamic model; H i is the observation matrix of sensor i, which represents the mapping of state space to observation space and is used to calculate the predicted observation value; R iis the observation noise covariance of sensor i, which represents the measurement noise or sensor error.

[0080] In the prediction phase, based on the current state estimate x, the state transition matrix F is used to predict the state x at the next moment. - In the update phase, the latest observation data z is used i Update the state estimate x to obtain a more accurate state estimate x. In this process, the Kalman gain K adjusts the weight between the prediction and the actual observation.

[0081] The Kalman filter algorithm is used to fuse the perception data from various sensors to form perception data on a consistent time and space basis after spatial registration and time synchronization, generating a more accurate and reliable local environment representation and obtaining multimodal fusion data.

[0082] The computational process of obtaining multimodal fusion data from multi-sensor perception data in drone scenarios through spatial registration, time synchronization, and Kalman filtering can be considered as being completed through a distributed multi-sensor fusion algorithm. The distributed multi-sensor fusion algorithm fully considers the heterogeneity and distribution of multi-sensor data in drone scenarios, achieves robust spatial registration through graph optimization, achieves accurate time synchronization through distributed clock synchronization, and achieves effective noise reduction of fusion data through Kalman filtering.

[0083] Table 1 gives the key parameters of the distributed multi-sensor fusion algorithm.

[0084]

[0085]

[0086] Table 1

[0087] This embodiment further uses Kalman filtering to perform noise reduction after completing time synchronization based on the time deviation between the time of the perception data and the real time and obtaining multimodal data, thereby reducing data fluctuations caused by factors such as sensor noise and environmental interference, thereby obtaining more stable and reliable perception data. It can better cope with various uncertain factors, making the semantic information finally generated more accurate and credible, and enhancing the robustness of the system.

[0088] In a feasible implementation, the feature points of the perception data include at least one of feature information of the collected object and information of the sensor.

[0089] The local feature map includes feature points of the perception data and may include at least one of feature information of the collected object and information of the sensor.

[0090] The feature information of the collected object may include information such as color, texture, edge, etc. used to describe the basic attributes of the data. The local feature map includes the feature points of the perceived data and may include the basic attribute information of key points, key positions or areas with specific meanings determined in the feature information of the collected object. The information of the sensor may include parameter information such as sensor type, model, quantity, resolution, frame rate, measurement range and installation location. The local feature map includes the feature points of the perceived data and may also include parameter information and parameter values ​​determined from the sensor parameter information.

[0091] Exemplarily, the configuration parameters of the sensor are shown in Table 2, and the sensor configuration parameters include sensor type, model, quantity, resolution, frame rate, measurement range and installation location, etc.

[0092]

[0093]

[0094] Table 2

[0095] As another implementation of this application, Figure 3 A flowchart of a semantic extraction method based on multimodal data provided by an embodiment of the present application is shown. In step S140: after spatial registration is performed based on the spatial transformation matrix between the perception data to obtain multimodal data, and the feature points in the multimodal data include perception data of different sensors at the same position, the method further includes: using a graph neural network model to perform semantic extraction on the multimodal data to obtain semantic information, so as to determine the target decision based on the semantic information.

[0096] like Figure 3 As shown, semantic information is obtained by semantically extracting multimodal data using a graph neural network model, including steps S310 to S350.

[0097] S310: Organizing the fused multimodal fusion data into a graph structure, where nodes represent objects or regions, and edges represent relationships between nodes.

[0098] Organize the fused multimodal fusion data into a graph structure The node set represents an object or region, and the edge set ε represents the relationship between nodes. With characteristic vector Each edge (v i ,v j )∈ε has a characteristic vector Table 3 shows the graph representation construction method of multimodal fusion data and perception data.

[0099]

[0100] Table 3

[0101] S320: Extract feature representations of nodes and edges in the graph using operations such as graph convolution.

[0102] The mathematical expression of the l-th layer of graph convolution is:

[0103]

[0104] in, is the feature vector of node i in layer l, is the set of neighbor nodes of node i, is the learnable weight matrix of the lth layer, and σ(·) is the activation function.

[0105] S330: Aggregate the features of nodes and edges through operations such as graph pooling to generate graph-level feature representation.

[0106] Through the graph pooling operation, the updated node features are aggregated into graph-level features. Common graph pooling methods include maximum pooling, average pooling, etc. Taking maximum pooling as an example, graph-level features The calculation formula is:

[0107]

[0108] Among them, L is the number of graph convolution layers.

[0109] S340: Use technologies such as attention mechanism to learn the importance of different nodes and edges and enhance the expressiveness of semantic features.

[0110] Using the attention mechanism, we learn the importance weights of different nodes and edges. The attention weight α between nodes i and j is ij The calculation formula is:

[0111]

[0112] Among them, a, W4, and W5 are learnable attention parameters.

[0113] Node feature x after weighted aggregation i′ for:

[0114]

[0115] Among them, W6 is a learnable weight matrix.

[0116] S350: Combine downstream tasks and fine-tune the graph neural network model to achieve accurate semantic information extraction.

[0117] The graph neural network is used to extract basic semantic information based on the feature representation of nodes and edges and the feature representation of the graph level. Combined with specific target tasks, such as target detection and semantic segmentation, the detection head network is connected after the graph neural network, and the loss function is used to fine-tune the graph neural network model. Taking the target detection task as an example, the detection head network can be connected after the graph neural network, and the multi-task loss function can be used for end-to-end training. The multi-task loss function is as follows:

[0118]

[0119] in, They are classification loss, bounding box regression loss and mask prediction loss respectively, and λ1 and λ2 are balancing factors.

[0120] A head network for target detection is added after the graph neural network, which can include a series of convolutional layers, fully connected layers, and final classification and regression layers. The head network is responsible for converting the high-level semantic features extracted by the graph neural network into the output format required for target detection, such as bounding box coordinates and category probabilities. The entire system including the graph neural network and the subsequent detection head network is trained end-to-end to optimize the multi-task loss function defined above. During the training process, the parameters of the graph neural network are continuously adjusted to better meet the needs of the target detection task, while ensuring that the graph neural network can effectively capture the semantic information in the environment. Table 4 shows an example of the semantic feature extraction results of the graph neural network.

[0121]

[0122]

[0123] Table 4

[0124] This embodiment uses a graph neural network model to extract semantics from multimodal data. By organizing data into a graph structure, using graph convolution and other technologies to extract features, using graph pooling to aggregate information, introducing an attention mechanism to emphasize key nodes, and fine-tuning the model for specific tasks, it can effectively capture the relationship between objects, enhance feature expression capabilities, and achieve accurate semantic understanding. At the same time, through end-to-end feature learning, it automatically mines the semantic information contained in the perception data, greatly improving the intelligence level and accuracy of semantic extraction.

[0125] As another implementation of the present application, in step S140: after spatial registration is performed based on the spatial transformation matrix between the perception data to obtain multimodal data, and the feature points in the multimodal data include perception data of different sensors at the same position, the method further includes:

[0126] The multimodal data is sent to the target processing terminal so that the target processing terminal can use the graph neural network model to perform semantic extraction on the multimodal data to obtain semantic information and determine the target decision based on the semantic information.

[0127] like Figure 1 The UAV perception data processing method shown in Figure 3 The process of the multimodal data semantic extraction method shown can be completed in the local processing unit of the drone, or it can be completed in the local processing unit of the drone as shown in FIG. Figure 1 After the drone perceives the data processing method shown, the obtained multimodal data is sent to the target processing terminal, so that the target processing terminal can use the graph neural network model to perform semantic extraction on the multimodal data to obtain semantic information and determine the target decision based on the semantic information. Figure 3 The process of the multimodal data semantic extraction method shown is the same and will not be repeated here.

[0128] The target processing terminal can include 5G edge nodes and central cloud platforms, etc. The target processing terminal can be used to complete semantic extraction of multimodal data to obtain semantic information and determine target decisions based on the semantic information. The target processing terminal globally aggregates semantic information from different regions to form an overall environmental understanding, and performs task planning and decision optimization based on global semantic information.

[0129] In one embodiment, the extracted multiple semantic information can be locally aggregated through the 5G edge node, and then the aggregated semantic information can be sent to the central cloud platform through the 5G-A network, so that the central cloud platform can globally aggregate the semantic information from different areas to form an overall environmental understanding, and perform task planning, decision optimization, etc. based on the global semantic information.

[0130] In one embodiment, the central cloud platform can send a decision instruction determined based on the globally aggregated semantic information to the local processing unit of the drone, and the decision instruction can be used to guide the drone to perform a task.

[0131] As another implementation of this application, Figure 4 FIG. 1 is a flow chart of a cloud-edge-device collaborative processing mechanism provided by an embodiment of the present application. Figure 4 As shown, sending multimodal data to a target processing terminal includes: sending multimodal data to the target processing terminal according to a target data transmission rate and priority, wherein the target data transmission rate and priority are determined based on importance information of each multimodal data and network load information of the target processing terminal.

[0132] The perception data of different sensors of multiple drones can be processed by multiple local processing units. Each drone corresponds to a local processing unit, and the target processing terminal corresponds to at least one local processing unit. The target processing terminal can be a 5G-A edge computing node, that is, each edge node corresponds to at least one drone. The local processing unit of the drone can send the multimodal data obtained by fusing multiple sensors to the nearest 5G-A edge computing node through the 5G-A network.

[0133] Through edge nodes, the target transmission rate is determined based on the data importance information of multimodal data, the network bandwidth corresponding to the edge nodes, and the number of drones. The data transmission rate between the local processing unit of the drone and the edge node, and between the edge node and the cloud platform is optimized. According to the load of the 5G network and the importance of the data, the data transmission rate and priority between the drone and the edge node, and between the edge node and the cloud platform are adaptively adjusted to reduce bandwidth usage while ensuring the real-time performance of key data. The specific implementation method can be:

[0134] Assume that the data transmission rate of the i-th UAV local processing unit transmitting multimodal data to the edge node through the 5G-A network is R i , the multi-mode is w i , the bandwidth of 5G-A network is B, then the data transmission optimization problem can be expressed as:

[0135]

[0136] Include constraints: Where N is the number of drones, n i is the noise power of the ith UAV, and are the lower and upper limits of the transmission rate, respectively. The objective function is used to calculate the sum of the logarithm of the transmission rate of all drone local processing units and the data importance of the corresponding data, and the data transmission rate of each drone local processing unit sending multimodal data to the edge node through the 5G-A network is determined to ensure the fairness and efficiency of transmission.

[0137] This embodiment can optimize bandwidth usage, ensure real-time transmission of critical data, and reduce bandwidth usage of non-critical data by adaptively adjusting the data transmission rate and priority between the local processing unit of the drone and the edge node, and between the edge node and the cloud platform according to the load of the 5G network and the importance of the data. This strategy not only improves the real-time performance and response speed of the system, but also enhances the robustness of the system, saves energy, and improves the overall user experience, supports large-scale deployment, and improves security.

[0138] As another implementation of this application, Figure 4 As shown, the method is applied to a local processing unit; before sending multimodal data to a target processing terminal, the method also includes: obtaining computing resource information of the local processing unit; and judging whether to send multimodal data to the target processing terminal based on the computing resource information of the local processing unit and the data calculation amount of the multimodal data.

[0139] The task offloading decision is made based on the computing resource status of the drone's local processing unit and the complexity of the task to be processed, that is, the amount of data calculation of multimodal data. Based on the computing resource status of the drone and the complexity of the task, a decision is made whether to offload the task to the target processing device for processing. The target processing device can be an edge node. When the drone load is too high or the task exceeds the local processing capacity, the task is offloaded to the edge node to utilize its computing acceleration capability. The specific implementation method can be:

[0140] Assume that the computing power of the local processing unit of the i-th drone is c i , the computational effort of task j is w j , the execution time of the local processing unit of the drone is The time for offloading to the edge node is Then the task offloading decision process can be expressed as:

[0141]

[0142] Include constraints: in, The computing capacity constraint for the local processing unit is that the computing capacity of each UAV i’s local processing unit must not exceed its own computing resources c i ; The computing power constraint implemented for edge nodes means that the computing power of edge nodes shall not exceed their own computing resources C; The selection constraint for task execution indicates that each task j is either executed on the local processing unit or on the edge node; is a binary variable constraint, representing the decision variable It is a binary variable, that is, it can only take the value 0 or 1.

[0143] The task offloading decision can determine whether the target task is executed in the local processing unit of the drone or in the edge node. The target task can be a data processing task of fusing data from multiple sensors to obtain multimodal fusion data in the embodiment of the present application, or a data processing task of extracting semantics from multimodal fusion data using a graph neural network model.

[0144] This embodiment makes intelligent task offloading decisions based on the computing resource status and task complexity of the drone's local processing unit. When local computing resources are insufficient or the task is too complex, the task is offloaded to the edge node for processing, thereby improving processing efficiency, optimizing resource utilization, enhancing system response speed, extending the drone's flight time, and supporting more complex application scenarios, while improving the system's robustness and adaptability to dynamic environments.

[0145] As another implementation of this application, Figure 4 As shown, the method may also include: dynamically adjusting the deployment strategy of the graph neural network model according to the computing power and storage capacity of the edge nodes, making task offloading decisions and performing hierarchical deployment of the model.

[0146] As the target processing terminal, the edge node can use the graph neural network model to extract semantic information from multimodal data and its local environment representation. The deployment strategy of the graph neural network model can be dynamically adjusted according to the computing power and storage capacity of the edge node. For nodes with sufficient resources, a complete semantic extraction model is deployed; for nodes with limited resources, a lightweight model after quantization and compression is deployed. The specific implementation method can be:

[0147] Assume that the computational complexity of model k is O k , the storage requirement is S k , the computing power of edge node n is C n , the storage capacity is M n , then the model hierarchical deployment problem can be expressed as:

[0148]

[0149] Include constraints: The constraints ensure that the deployment of the model meets the computational and storage capacity constraints of the nodes. nk is the performance improvement coefficient of model k on edge node n, x nk is a binary decision variable, indicating whether model k is deployed on edge node n. The objective function is used to calculate the sum of the performance improvements of models deployed on all edge nodes. The objective function is used to determine the configuration specifications of the model deployed on each edge node, where the model can be a graph neural network or other model.

[0150] This embodiment dynamically adjusts the deployment strategy of the graph neural network model according to the computing power and storage capacity of the edge nodes, that is, deploys the complete model on nodes with sufficient resources and deploys the lightweight model on nodes with limited resources. This method can optimize resource utilization, improve processing efficiency and response speed, reduce energy consumption, and ensure efficient semantic extraction capabilities under different hardware conditions.

[0151] As another implementation of this application, Figure 4 As shown, the method may also include: the target processing terminal includes multiple edge nodes and a cloud platform at the same time. Obtain resource allocation ratios and computing capabilities of multiple edge nodes and global task computing requirement information; based on the resource allocation ratios and computing capabilities of the multiple edge nodes and the global task computing requirement information, determine the target resource allocation ratio of each edge node.

[0152] The cloud platform can uniformly schedule the computing resources of edge nodes according to the global mission requirements and the status information of each drone's local processing unit, dynamically balance the load and perform global scheduling optimization. The specific implementation methods can be:

[0153] Assume that the computing resource allocation ratio of edge nodes is λ l , the computational requirement of the global task is C global , the computing power of the edge node is f l , then the global scheduling optimization problem can be expressed as:

[0154]

[0155] Include constraints: The constraints ensure the rationality of resource allocation and the satisfaction of task requirements. The maximum load ratio of all edge nodes is minimized through objective function calculation, and the target resource allocation ratio of each edge node is determined to achieve load balancing.

[0156] By obtaining the resource allocation ratios and computing capabilities of multiple edge nodes and global task computing demand information, the cloud platform can dynamically adjust the target resource allocation ratio of each edge node, uniformly schedule computing resources and balance the load, thereby improving resource utilization, reducing processing delays, enhancing system flexibility, and ensuring efficient completion of tasks.

[0157] As another implementation of this application, Figure 5 A system architecture diagram of a drone perception data processing method provided by an embodiment of the present application is shown.

[0158] like Figure 5 As shown, the system includes drone sensors, local processing units, edge nodes and cloud platform devices.

[0159] Among them, drone sensors are used to collect environmental perception data. Drone sensors can include multimodal sensors such as visible light cameras, infrared cameras, radars, LiDAR, etc. Drones can collect environmental perception data in real time through onboard multimodal sensors.

[0160] The local processing unit can be a local processing unit of the drone, which is used to obtain perception data and sensor time data, and perform real-time preprocessing and preliminary fusion of the perception data, and then transmit the fused multimodal data to the nearest edge node through the 5G-A network.

[0161] The edge node can be a 5G-A edge computing node. The 5G-A edge computing node is used to extract multimodal data using AI algorithms such as graph neural networks, that is, to achieve intelligent understanding of the semantic information in the fused perception data, and to aggregate the extracted semantic information locally and further upload it to the cloud platform device through the 5G-A network.

[0162] The cloud platform device can be a central cloud platform, which is used to globally aggregate semantic information from different regions to form an overall environmental understanding, and perform task planning and decision optimization based on global semantic information. The central cloud platform is also used to send decision instructions to 5G-A edge nodes and drone local processing units to guide drones to perform tasks.

[0163] As another implementation of this application, Figure 6 A flowchart of a method for processing drone perception data provided by another embodiment of the present application is shown. The method includes steps S610 to S670.

[0164] S610: The drone collects environmental perception data in real time through onboard multi-modal sensors (such as visible light cameras, infrared cameras, radars, LiDAR, etc.).

[0165] S620: The local processing unit of the UAV performs pre-processing on the collected raw perception data, such as denoising, calibration, spatiotemporal synchronization, etc., and uses a distributed multi-sensor fusion algorithm to achieve preliminary data fusion.

[0166] S630: The pre-processed and initially integrated perception data is transmitted to the nearest 5G-A edge computing node via the 5G-A network.

[0167] S640: 5G-A edge computing nodes use AI algorithms such as graph neural networks to extract semantic information from perception data, achieve intelligent understanding, and aggregate the extracted semantic information locally.

[0168] S650: The aggregated regional semantic information is further uploaded to the central cloud platform through the 5G-A network.

[0169] S660: The central cloud platform globally aggregates semantic information from different regions to form an overall environmental understanding, and performs task planning, decision optimization, etc. based on the global semantic information.

[0170] S670: The central cloud platform sends decision instructions to the 5G-A edge nodes and drone local processing units to guide the drones to perform tasks.

[0171] Based on the same concept, the present application embodiment provides a drone perception data processing device, as shown below: Figure 7 The drone perception data processing device provided in the embodiment of the present application is described in detail.

[0172] Figure 7 1 is a block diagram of a drone perception data processing device shown in an embodiment of the present application. Figure 7 As shown, the drone perception data processing device may include:

[0173] An acquisition module 710 is used to acquire perception data of different sensors of the drone;

[0174] A construction module 720 is used to construct a local feature map of each sensor based on the perception data of each sensor of the drone, where the local feature map includes feature points of the perception data and feature indexes identifying positions of the feature points;

[0175] A determination module 730 is used to determine a spatial transformation matrix between perception data corresponding to different local feature maps by minimizing a reprojection error based on matching feature points between different local feature maps and feature indexes of positions corresponding to the matching feature points;

[0176] The registration module 740 is used to perform spatial registration based on the spatial transformation matrix between the perception data to obtain multimodal data. The feature points in the multimodal data include the perception data of different sensors at the same position, so as to determine the target decision based on the multimodal data.

[0177] In one embodiment, after multimodal data is obtained by spatial registration based on the spatial transformation matrix between the perception data, and the feature points in the multimodal data include perception data of different sensors at the same position, the registration module 740 is also used to fuse the perception data of the feature points of the multimodal data based on Kalman filtering to obtain local environment representation data corresponding to the multimodal data, so as to determine the target decision based on the multimodal data and the local environment representation corresponding to the multimodal data.

[0178] In one embodiment, the feature points of the perception data include at least one of feature information of the collected object and information of the sensor.

[0179] In one embodiment, after multimodal data is obtained by spatial registration based on the spatial transformation matrix between the perception data, and the feature points in the multimodal data include the perception data of different sensors at the same position, the registration module 740 is also used to use the graph neural network model to perform semantic extraction on the multimodal data to obtain semantic information for determining target decisions based on the semantic information.

[0180] In one embodiment, after multimodal data is obtained by spatial registration based on the spatial transformation matrix between the perception data, and the feature points in the multimodal data include the perception data of different sensors at the same position, the registration module 740 is also used to send the multimodal data to the target processing terminal, so that the target processing terminal can use the graph neural network model to perform semantic extraction on the multimodal data to obtain semantic information, and determine the target decision based on the semantic information.

[0181] In one embodiment, the registration module 740 is specifically used to send multimodal data to the target processing terminal according to the target data transmission rate and priority, and the target data transmission rate and priority are determined according to the importance information of each multimodal data and the network load information of the target processing terminal.

[0182] In one embodiment, the method is applied to a local processing unit; before sending multimodal data to a target processing terminal, the acquisition module 710 is also used to obtain computing resource information of the local processing unit; based on the computing resource information of the local processing unit and the data calculation amount of the multimodal data, it is determined whether to send the multimodal data to the target processing terminal.

[0183] Figure 7 Each module in the device shown has the function of realizing Figure 1 , Figure 3 and Figure 6 The functions of each step in the process can achieve the corresponding technical effects, which will not be described in detail here for the sake of brevity.

[0184] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application is shown.

[0185] The electronic device may include a processor 810 and a memory 820 storing computer program instructions.

[0186] Specifically, the processor 810 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0187] The memory 820 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 820 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 820 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 820 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 820 is a non-volatile solid-state memory.

[0188] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect of the present disclosure.

[0189] The processor 810 implements any one of the drone perception data processing methods in the above embodiments by reading and executing computer program instructions stored in the memory 820.

[0190] In one example, the electronic device may further include a communication interface 830 and a bus 840. Figure 8 As shown, the processor 810, the memory 820, and the communication interface 830 are connected via a bus 840 and communicate with each other.

[0191] The communication interface 830 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0192] Bus 840 includes hardware, software or both, and the parts of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 840 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0193] The electronic device can execute the drone perception data processing method in the embodiment of the present application, thereby realizing the combination Figure 1 , Figure 3 and Figure 6 Described is the UAV perception data processing method.

[0194] In addition, in combination with the drone perception data processing method in the above embodiment, the embodiment of the present application can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the drone perception data processing methods in the above embodiment is implemented.

[0195] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, implements any one of the drone perception data processing methods in the above embodiments.

[0196] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0197] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0198] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0199] The above reference is according to the method of the embodiment of the present application, the flow chart of the device (system) and the computer program product and / or the block diagram described various aspects of the present application.It should be understood that each square box in the flow chart and / or the block diagram and the combination of each square box in the flow chart and / or the block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the realization of the function / action specified in one or more square boxes of the flow chart and / or the block diagram.Such a processor can be but is not limited to a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit.It can also be understood that each square box in the block diagram and / or the flow chart and the combination of the square boxes in the block diagram and / or the flow chart can also be realized by the dedicated hardware that performs the specified function or action, or can be realized by the combination of dedicated hardware and computer instructions.

[0200] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A method for processing drone perception data, characterized in that The method comprises: Obtain perception data from different sensors of the drone; Based on the perception data of each sensor of the drone, a local feature map of each sensor is constructed, wherein the local feature map includes feature points of the perception data and feature indexes identifying positions of the feature points; According to the matching feature points between different local feature maps and the feature indexes of the positions corresponding to the matching feature points, determining the spatial transformation matrix between the perception data corresponding to the different local feature maps by minimizing the reprojection error; Multimodal data is obtained by performing spatial registration based on a spatial transformation matrix between the perception data. Feature points in the multimodal data include perception data of different sensors at the same position, so as to determine a target decision based on the multimodal data.

2. The method according to claim 1, characterized in that After obtaining multimodal data by performing spatial registration based on the spatial transformation matrix between the perception data, wherein the feature points in the multimodal data include perception data of different sensors at the same position, the method further includes: Based on Kalman filtering, the perception data of the feature points of the multimodal data are fused to obtain local environment representation data corresponding to the multimodal data, so as to determine the target decision according to the multimodal data and the local environment representation corresponding to the multimodal data.

3. The method according to claim 1, characterized in that The feature points of the perception data include at least one of feature information of the collected object and information of the sensor.

4. The method according to claim 1, characterized in that After obtaining multimodal data by performing spatial registration based on the spatial transformation matrix between the perception data, wherein the feature points in the multimodal data include perception data of different sensors at the same position, the method further includes: The multimodal data is semantically extracted using a graph neural network model to obtain semantic information for determining target decisions based on the semantic information.

5. The method according to claim 1, characterized in that After obtaining multimodal data by performing spatial registration based on the spatial transformation matrix between the perception data, wherein the feature points in the multimodal data include perception data of different sensors at the same position, the method further includes: The multimodal data is sent to a target processing terminal so that the target processing terminal uses a graph neural network model to perform semantic extraction on the multimodal data to obtain semantic information, and determines a target decision based on the semantic information.

6. The method according to claim 5, characterized in that The sending the multimodal data to the target processing terminal includes: The multimodal data are sent to the target processing terminal according to a target data transmission rate and priority, wherein the target data transmission rate and priority are determined according to importance information of each multimodal data and network load information of the target processing terminal.

7. The method according to claim 5, characterized in that The method is applied to a local processing unit; Before sending the multimodal data to the target processing terminal, the method further includes: Obtaining computing resource information of the local processing unit; According to the computing resource information of the local processing unit and the data computing amount of the multimodal data, it is determined whether to send the multimodal data to the target processing terminal.

8. A drone perception data processing device, characterized in that: The device comprises: The acquisition module is used to obtain the perception data of different sensors of the drone; A construction module, configured to construct a local feature map of each sensor based on the perception data of each sensor of the drone, wherein the local feature map includes feature points of the perception data and feature indexes identifying positions of the feature points; A determination module, configured to determine a spatial transformation matrix between perception data corresponding to different local feature maps by minimizing a reprojection error based on matching feature points between different local feature maps and feature indexes of positions corresponding to the matching feature points; A registration module is used to perform spatial registration based on the spatial transformation matrix between the perception data to obtain multimodal data, wherein the feature points in the multimodal data include the perception data of different sensors at the same position, so as to determine the target decision according to the multimodal data.

9. An electronic device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the drone perception data processing method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by the processor, the method for processing drone perception data according to any one of claims 1 to 7 is implemented.

11. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the drone perception data processing method as described in any one of claims 1 to 7.

Images