A deep learning-based multi-modal perception fusion unmanned aerial vehicle cluster control method

Abstract

The present application relates to the technical field of unmanned aerial vehicle cluster control, in particular to a multi-modal perception fusion unmanned aerial vehicle cluster control method and device based on deep learning, equipment and storage medium. The present application collects gesture, voice, visual and touch data in parallel through a multi-modal perception access layer, and pre-processes and synchronizes the data. A special deep network is used to extract high-level features rich in cluster semantics. Through cross-modal semantic alignment, graph neural network correlation learning, and multi-dimensional conflict detection and intelligent resolution mechanism, the present application realizes the deep fusion of the four modes in the cluster control semantic space and the accurate analysis of the intention. The present application uses hierarchical task decomposition, virtual structure formation control and multi-objective optimization algorithm to efficiently convert high-level instructions into single-machine executable tasks and coordination strategies. The present application also performs real-time state feedback and adaptive parameter optimization, improving the accuracy, reliability, environmental adaptability and human-computer interaction efficiency of large-scale unmanned aerial vehicle cluster control.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) swarm control technology, and in particular to a method, apparatus, device, and storage medium for UAV swarm control based on deep learning and multimodal perception fusion. Background Technology

[0002] With the rapid development of UAV technology, UAV swarm control has become a hot research topic. Traditional swarm control systems mainly rely on complex ground control stations and specialized operators, resulting in high learning costs and operational complexity, making it difficult to meet the needs of large-scale swarm collaborative operations. In recent years, researchers have proposed various swarm control methods based on natural interaction, but all of them suffer from the following key technical shortcomings:

[0003] (1) Single-modal interaction has limitations

[0004] Each single-modal interaction method has its limitations in terms of expressive range: gesture recognition struggles to accurately represent complex three-dimensional spatial relationships; speech recognition is inefficient when expressing precise numerical values ​​and spatial coordinates; and touch interaction has difficulties expressing dynamically changing formation commands. These limitations severely restrict the complexity and accuracy of swarm control commands. Each single-modal perception method also has specific applicable scenarios and failure conditions. Gesture recognition performance degrades in strong or low light environments; speech recognition lacks reliability in noisy multi-drone operating environments; and visual perception performs poorly in adverse weather conditions. This dependence on specific environmental conditions limits the practical application scope of swarm systems. Single-modal interaction is difficult to effectively manage large-scale drone swarms. Operators cannot simultaneously monitor and control the status of multiple drones using a single interaction method, leading to omissions and errors, thus limiting the scalability of the swarm.

[0005] (2) Existing multimodal fusion technology has shortcomings

[0006] Most existing systems employ simple decision-level fusion or early feature stitching, failing to delve into the semantic relationships between different modalities. For example, gestures and speech have natural semantic complementarity in expressing swarm maneuver commands, but existing systems cannot effectively utilize this relationship. Existing systems lack a deep understanding of swarm control semantics. They cannot convert multimodal operator input into control commands with swarm coordination significance, such as complex swarm coordination commands like "spread out while maintaining formation" or "group to perform different tasks." Swarm control involves complex spatiotemporal relationships, with significant differences in spatiotemporal information features across different modalities. Gestures possess continuous spatiotemporal trajectory information, speech possesses temporal semantic information, and vision possesses spatial geometric information; existing methods struggle to effectively fuse these within a unified spatiotemporal framework.

[0007] (3) Lack of intelligent cluster coordination mechanism

[0008] Existing systems typically employ preset control and formation modes, failing to dynamically adjust cluster coordination strategies based on task requirements and environmental changes. When faced with complex and ever-changing task scenarios, these systems lack adaptive cluster coordination capabilities. In multimodal input scenarios, different modalities may issue conflicting cluster control commands. Existing systems lack effective conflict detection and resolution mechanisms; simple voting or priority strategies can lead to chaotic cluster coordination. Existing systems primarily focus on the precision of individual machine control, lacking design for the overall intelligent behavior of the cluster. They cannot achieve collaborative perception, collaborative decision-making, and collaborative execution within the cluster, and the overall performance of the cluster is difficult to surpass the simple sum of individual machine performance.

[0009] In summary, existing UAV swarm control methods suffer from limitations in single-modal interaction, poor effectiveness of multimodal fusion, and poor accuracy and intelligence in swarm coordination mechanisms. Summary of the Invention

[0010] Therefore, the technical problem to be solved by the present invention is to overcome the limitations of single-modal interaction, poor effectiveness of multimodal fusion, and poor accuracy and intelligence of the swarm coordination mechanism in the existing UAV swarm control methods.

[0011] To address the aforementioned technical problems, this invention provides a deep learning-based multimodal perception fusion unmanned aerial vehicle (UAV) swarm control method, comprising:

[0012] Acquire multimodal operator input, including cluster gesture commands, cluster voice commands, cluster visual monitoring, and cluster touch planning, and perform data preprocessing and data synchronization;

[0013] The cluster control features of each modality are extracted by using a pre-designed deep neural network, mapped to a unified cluster control semantic space, and cross-modal semantic association learning is performed to obtain deep fusion features of each modality. Multimodal conflict resolution is performed on the deep fusion features of each modality to obtain a unified cluster control intent.

[0014] Cluster control instructions are generated based on a unified cluster control intent. The central coordinator decomposes the cluster control instructions into individual machine execution tasks and performs formation control, path planning, and global optimization.

[0015] The system monitors the overall execution status of the cluster in real time, detects and handles anomalies, and adaptively optimizes feature fusion and the network parameters of the central coordinator and the control parameters of the cluster coordination control based on performance feedback to form a closed loop.

[0016] Preferably, the step of acquiring operator multimodal input, including cluster gesture commands, cluster voice commands, cluster visual monitoring, and cluster touch planning, and performing data preprocessing and data synchronization includes:

[0017] Depth RGB images of operator gestures are acquired using a depth camera array, transformed into a unified three-dimensional coordinate system, and the cluster gesture commands are temporally segmented using the sliding window method to segment out complete gesture segments that express the control intent. Key point data is then extracted to obtain preprocessed cluster gesture commands.

[0018] The system uses a microphone array to collect operator voice commands on the console, uses an endpoint detection algorithm to locate the start and end points of the voice commands, filters out silent segments, performs beamforming voice enhancement, echo cancellation and noise suppression, and extracts mel spectrum features to obtain preprocessed cluster voice commands.

[0019] Wide-angle camera arrays are used to collect visual scenes of the cluster's flight airspace. Eye-tracking devices are used to collect operator eye movements and extract the coordinates of the operator's visual focus. These coordinates are then calibrated and mapped onto the visual scene. Target detection is performed on the visual scene to obtain image data, which is the pre-processed cluster visual monitoring.

[0020] The operator's touch point trajectory coordinate sequence is collected using a multi-touch display screen. The trajectory of each touch point is grouped using a clustering algorithm, and the operation semantics are identified to obtain the preprocessed cluster touch planning.

[0021] The preprocessed multimodal inputs are timestamped, formatted uniformly, and their integrity is checked.

[0022] Preferably, the step of extracting cluster control features of each modality using a pre-designed deep neural network and mapping them to a unified cluster control semantic space includes:

[0023] An improved YOLO-v8 architecture is used to detect clustered control actions on keypoint data. The HRNet architecture is used to optimize the modeling of spatial relationships between keypoints. The Transformer encoder and multi-head attention mechanism are used to learn long-distance temporal dependencies in gesture sequences to obtain clustered gesture features.

[0024] The Conformer encoder is used to encode the mel spectral features, and the CRF layer is used to perform semantic parsing of the cluster entities, actions and parameters on its output to obtain the cluster speech features;

[0025] The EfficientDet architecture is used to perform target detection and tracking on image data. The ResNet50+FPN architecture is used to perform multi-scale fusion on its output. The Attention mechanism is used to enhance the features of the specific area that the operator is focused on based on the operator's visual focus coordinates, so as to obtain the cluster visual features.

[0026] A bidirectional long short-term memory network is used to perform temporal trajectory analysis on each touch point to obtain temporal features. A geometric deep learning method is used to identify the overall shape formed by the clustered touch point set to determine what kind of formation or flight path it represents, thus obtaining spatial geometric features. Combining the temporal and spatial geometric features of the trajectory, a classification network is used to identify the operator's cluster planning intention to obtain cluster touch features.

[0027] Layer Normalization is applied to the cluster control features of each modality, and dynamic time warping algorithm is used to align the cluster control features of each modality.

[0028] Define a unified cluster control semantic space that includes formation mode, maneuver type, coordination method, mission mode, and parameters;

[0029] By mapping the cluster control features of each modality to a unified cluster control semantic space through multiple MLP networks and introducing contextual information, the target cluster control features of each modality are obtained.

[0030] Preferably, the step of performing cross-modal semantic association learning to obtain deep fusion features of each modality includes:

[0031] Construct a fully connected graph where the node set represents each modality and the edge set represents the semantic relationships between modalities;

[0032] The correlation strength between each pair of nodes is calculated using a multilayer perceptron.

[0033] Multi-layer graph convolution is performed on the fully connected graph. Based on the target cluster control features of each node's neighboring nodes and the association strength between each node and its neighboring nodes, the features of each node are updated to obtain deep fusion features of each modality.

[0034] Preferably, the step of resolving multimodal conflicts in the deep fusion features of each modality to obtain a unified cluster control intent includes:

[0035] Calculate the cosine similarity of the deep fusion features between any two modalities. If the cosine similarity is not greater than the first empirical value, it is determined that there is a semantic conflict between the two modalities.

[0036] Calculate the relative error of numerical parameters between any two modes. If the error is not less than the second empirical value, it is determined that there is a parameter conflict between the two modes.

[0037] If the relative error between timestamps of any two modes is not less than a third empirical value, it is determined that there is a timing conflict between the two modes.

[0038] Detect whether any two modalities with the same priority have issued instructions with different content at the same time. If so, determine that the two modalities have a priority conflict.

[0039] The severity of conflicts across all modalities is comprehensively assessed. In cases of minor conflicts, the deep fusion features of each modality are weighted by confidence to obtain a unified cluster control intent. In cases of moderate conflicts, the deep fusion feature of the highest priority modality is determined as the unified cluster control intent. In cases of severe conflicts, the deep fusion feature of the operator-selected modality is determined as the unified cluster control intent.

[0040] Preferably, the step of generating cluster control commands based on a unified cluster control intent, decomposing the cluster control commands into individual machine execution tasks through a central coordinator, and performing formation control, path planning, and global optimization includes:

[0041] The Transformer decoder architecture is adopted, with a unified cluster control intent as memory. The structured cluster control instructions are generated by decoding through a cross-attention mechanism, and the specific numerical parameters in the cluster control instructions are determined by a parameter regression network.

[0042] The core objective of the cluster control command is parsed by the central coordinator to obtain the cluster-level task. The cluster-level task is then decomposed into multiple sub-cluster-level tasks. Specific tasks are assigned to each sub-cluster-level task to obtain multiple single-machine-level tasks. An improved Hungarian algorithm is then used to solve the task allocation problem to obtain the task allocation list for each single-machine-level task.

[0043] A motion trajectory is defined for each formation to represent the virtual leader of the entire formation. The desired position of each UAV in the formation is determined based on the position of the virtual leader, the formation attitude, the preset inter-UAV offset and the formation scaling scale. PD is used to control the UAV to track the desired position. When the cluster control command requires the formation to change, polynomial interpolation is used to plan a smooth trajectory for the virtual leader.

[0044] Establish a global path optimization objective for the entire cluster, including total task completion time, total energy consumption, overall risk assessment, and coordination cost, and use optimization algorithms to find the Pareto optimal solution;

[0045] Continuously monitor the task load and remaining power of each drone, dynamically adjust task allocation, and continuously monitor the drone status. If a drone malfunctions, immediately initiate dynamic reallocation.

[0046] Preferably, the real-time monitoring of the overall execution status of the cluster, detecting and handling anomalies, and adaptively optimizing feature fusion and the network parameters of the central coordinator and the control parameters of the cluster coordination control based on performance feedback to form a closed loop includes:

[0047] The system receives the status data of each drone in real time through the communication link, merges the status data of all drones to generate an overall situational map of the cluster, and calculates performance indicators including formation quality, mission progress and system efficiency.

[0048] Based on machine learning models, the system analyzes the overall situation of the cluster in real time, identifies abnormal patterns, and provides abnormal information alarms, fault isolation, and emergency plan triggering in the event of a failure.

[0049] The control parameters of cluster coordination control are fine-tuned in real time based on performance indicators.

[0050] By employing a reinforcement learning framework, the cluster control process is treated as a Markov decision process, with performance metrics as rewards, to continuously optimize the network parameters of feature fusion and the central coordinator.

[0051] The present invention also provides a deep learning-based multimodal perception fusion unmanned aerial vehicle (UAV) swarm control device, comprising:

[0052] The data acquisition and preprocessing module is used to acquire multimodal operator input, including cluster gesture commands, cluster voice commands, cluster visual monitoring, and cluster touch planning, and to perform data preprocessing and data synchronization.

[0053] The unified cluster control intent acquisition module is used to extract cluster control features of each modality using a pre-designed deep neural network, map them to a unified cluster control semantic space, perform cross-modal semantic association learning to obtain deep fusion features of each modality, and perform multimodal conflict resolution on the deep fusion features of each modality to obtain the unified cluster control intent.

[0054] The cluster control and coordination module is used to generate cluster control commands based on a unified cluster control intent. The central coordinator decomposes the cluster control commands into individual machine execution tasks and performs formation control, path planning, and global optimization.

[0055] The adaptive update module is used to monitor the overall execution status of the cluster in real time, detect and handle anomalies, and adaptively optimize the feature fusion and network parameters of the central coordinator and the control parameters of the cluster coordination control based on performance feedback to form a closed loop.

[0056] The present invention also provides a deep learning-based multimodal perception fusion unmanned aerial vehicle (UAV) swarm control device, comprising:

[0057] Memory, used to store computer programs;

[0058] A processor is used to execute the computer program to implement the steps of the above-described deep learning-based multimodal perception fusion unmanned aerial vehicle swarm control method.

[0059] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described deep learning-based multimodal perception fusion UAV swarm control method.

[0060] The technical solution of the present invention has the following advantages compared with the prior art:

[0061] The deep learning-based multimodal perception fusion UAV swarm control method described in this invention acquires gesture, voice, visual, and touch data in parallel through a multimodal perception access layer, followed by professional preprocessing and strict synchronization. It utilizes a dedicated deep network to extract high-level features rich in swarm semantics. Through cross-modal semantic alignment, graph neural network association learning, and multi-dimensional conflict detection and intelligent resolution mechanisms, it achieves deep fusion and accurate intent parsing of the four modalities in the swarm control semantic space. Employing hierarchical task decomposition, virtual structure formation control, and multi-objective optimization algorithms, it efficiently transforms high-level instructions into single-machine executable tasks and coordination strategies. Real-time state feedback and adaptive parameter optimization are performed before final execution, significantly improving the accuracy, reliability, environmental adaptability, and human-machine interaction efficiency of large-scale UAV swarm control. Attached Figure Description

[0062] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:

[0063] Figure 1 This is a flowchart illustrating the implementation of a deep learning-based multimodal perception fusion unmanned aerial vehicle (UAV) swarm control method provided by the present invention.

[0064] Figure 2 This is a diagram illustrating the overall architecture of a deep learning-based multimodal perception fusion UAV swarm control method provided by the present invention.

[0065] Figure 3 This is a detailed structural diagram of the multimodal sensing access layer;

[0066] Figure 4 Diagram of the network architecture for multimodal feature extraction;

[0067] Figure 5 This is a diagram of the deep semantic fusion layer structure.

[0068] Figure 6 Here is a flowchart for multimodal conflict detection and resolution;

[0069] Figure 7 This is a diagram of the cluster intelligent coordination layer architecture.

[0070] Figure 8Diagram illustrating task allocation within the cluster;

[0071] Figure 9 This is a schematic diagram of cluster formation control.

[0072] Figure 10 This is a flowchart illustrating the implementation of a deep learning-based multimodal perception fusion unmanned aerial vehicle (UAV) swarm control method, as provided in an embodiment of the present invention. Detailed Implementation

[0073] The core of this invention is to provide a method, device, equipment, and computer storage medium for controlling a large-scale UAV swarm based on deep learning and multimodal perception fusion, which effectively improves the accuracy, reliability, environmental adaptability, and human-computer interaction efficiency of large-scale UAV swarm control.

[0074] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] Please refer to Figure 1 and Figure 2 , Figure 1 This is a flowchart illustrating the implementation of a deep learning-based multimodal perception fusion unmanned aerial vehicle (UAV) swarm control method provided by the present invention. Figure 2 The overall architecture diagram for the implementation of a deep learning-based multimodal perception fusion UAV swarm control method provided by this invention is shown below; the specific operation steps are as follows:

[0076] S101: Acquire multimodal operator input, including cluster gesture commands, cluster voice commands, cluster visual monitoring, and cluster touch planning, and perform data preprocessing and data synchronization;

[0077] S102: Extract cluster control features of each modality using a pre-designed deep neural network, map them to a unified cluster control semantic space, perform cross-modal semantic association learning to obtain deep fusion features of each modality, and resolve multimodal conflicts of each deep fusion feature to obtain a unified cluster control intent.

[0078] S103: Generates cluster control instructions based on unified cluster control intent, decomposes cluster control instructions into single-machine execution tasks through a central coordinator, and performs formation control, path planning, and global optimization;

[0079] S104: Monitors the overall execution status of the cluster in real time, detects and handles anomalies, and adaptively optimizes feature fusion and the network parameters of the central coordinator and the control parameters of the cluster coordination control based on performance feedback to form a closed loop.

[0080] Based on the above embodiments, this embodiment will provide a detailed description of step S101:

[0081] like Figure 3 In some embodiments, acquiring operator multimodal input, including cluster gesture commands, cluster voice commands, cluster visual monitoring, and cluster touch planning, and performing data preprocessing and synchronization, includes:

[0082] • Use a depth camera array to acquire depth RGB images of operator gestures, transform them into a unified three-dimensional coordinate system, use the sliding window method to perform temporal segmentation of cluster gesture commands, segment out complete gesture segments that express control intentions, and extract key point data to obtain preprocessed cluster gesture commands.

[0083] It should be noted that, in one specific embodiment, the depth camera array is an RGB-D depth camera array, which forms a 120-degree panoramic coverage with a calibrated layout. The RGB resolution is 1920×1080 @ 60fps, and the depth map resolution is 848×480. Four cameras are simultaneously triggered to acquire RGB images and depth information of the operator's gestures.

[0084] It should be noted that, in one specific embodiment, based on the multi-camera calibration parameters, the 2D image coordinates are transformed to a unified three-dimensional gesture space coordinate system, such as a three-dimensional gesture space coordinate system with a size of 3m×3m×2m.

[0085] It should be noted that, in one specific embodiment, the gestures for swarm semantics include: formation change gestures: hands together (swarm aggregation), hands spread out (swarm dispersion); direction indication gestures: a single finger pointing (indicating the direction of swarm maneuvering); and size adjustment gestures: the number of extended fingers (indicating the number of drones participating in the mission).

[0086] It should be noted that, in one specific embodiment, a sliding window method is used to segment the continuous data stream. The window length is fixed at 200 frames (approximately 6.7 seconds) and the overlap rate is 50% to segment out gesture segments that express complete control intentions. This step outputs a 200-frame length coordinate sequence of 42 key points of both hands in a unified three-dimensional space.

[0087] • The system uses a microphone array to collect operator voice commands on the console, uses an endpoint detection algorithm to locate the start and end points of the voice commands, filters out silent segments, performs beamforming voice enhancement, echo cancellation and noise suppression, and extracts mel spectrum features to obtain preprocessed cluster voice commands.

[0088] It should be noted that, in one specific embodiment, the microphone array is an 8-channel circular microphone array with a microphone spacing of 8cm, a sampling rate of 48kHz, a quantization bit depth of 24bit, and includes a DSP processing chip.

[0089] It should be noted that, in one specific embodiment, beamforming is performed in real time by a DSP chip to enhance the operator's voice, and echo cancellation (AEC) and noise suppression (NS) are performed, specifically optimized to suppress multi-aircraft propeller noise; the VAD algorithm is used to accurately locate the start and end points of the voice commands, filter out silent segments, and separate complete command statements, such as "First squadron, fly north; second squadron, maintain position"; 80-dimensional Log Mel-filterbank features are calculated for each frame of audio, and prosodic features such as tone and intonation are additionally extracted. This step outputs an 80×T-dimensional Mel spectral feature sequence (T is the time frame number) and its prosodic features.

[0090] • Use a wide-angle camera array to collect visual scenes of the cluster's flight airspace, use eye-tracking devices to collect operator eye movements, extract the operator's visual focus coordinates, calibrate and map them onto the visual scene, perform target detection on the visual scene, and obtain image data, i.e., pre-processed cluster visual monitoring.

[0091] It should be noted that, in one specific embodiment, the main monitoring camera uses a 4K resolution ultra-wide-angle lens, covering a 120-degree field of view to monitor the entire cluster; the auxiliary camera provides cluster views from different angles; and the eye-tracking device uses a sampling rate of 1000Hz and an accuracy of 0.5 degrees to accurately track the operator's eye movements.

[0092] It should be noted that, in one specific embodiment, the main camera captures a global 4K video stream, and the eye-tracking device outputs the screen coordinates (or spatial coordinates) of the operator's gaze point in real time, which are then calibrated and mapped to a specific area of ​​the monitoring video; the video stream is then preliminarily processed to identify targets such as drones and obstacles in the image;

[0093] It should be noted that, in one specific embodiment, this step outputs raw image data of 1920×1080×3, the coordinates (x, y) of the operator's visual focus point, and preliminary target detection results (such as drones).

[0094] • Collect the trajectory coordinate sequence of each touch point of the operator using a multi-touch display screen, group the trajectory of each touch point using a clustering algorithm, and identify the operation semantics to obtain the preprocessed cluster touch planning;

[0095] It should be noted that, in one specific embodiment, the multi-touch display screen adopts a 55-inch 4K ultra-high-definition multi-touch display screen, supports 40 simultaneous touch points, has a touch sampling rate of 240Hz, a pressure sensing accuracy of 4096 levels, and has an anti-glare coating on the surface.

[0096] It should be noted that, in one specific embodiment, the continuous coordinate sequence of each touch point is recorded at high speed, and a clustering algorithm (such as DBSCAN) is used to group the touch point trajectories that occur simultaneously, identify which trajectories belong to "simultaneous drawing of multi-machine paths" and which belong to "direct design of formation shape", and preliminarily identify the touch action as "task area division", "no-fly zone marking" or other predefined cluster planning instructions.

[0097] It should be noted that, in one specific embodiment, this step outputs a sequence of trajectory coordinates (X, Y, Pressure, Timestamp) for N touch points, along with their clustering labels and operation intent labels.

[0098] • Perform timestamp alignment, data format unification, and integrity checks on the preprocessed multimodal inputs.

[0099] It should be noted that, in one specific embodiment, all sensors are connected to a unified time system and stamped with precise hardware timestamps to ensure that the time deviation of the data collected at any time is less than 1 millisecond (ms).

[0100] like Figure 4 Based on the above embodiments, this embodiment will provide a detailed description of step S102:

[0101] In some embodiments, extracting cluster control features for each modality using a pre-designed deep neural network and mapping them to a unified cluster control semantic space includes:

[0102] • An improved YOLO-v8 architecture is used to detect clustered control actions on keypoint data. The HRNet architecture is used to optimize the modeling of spatial relationships between keypoints. The Transformer encoder and multi-head attention mechanism are used to learn long-distance temporal dependencies in gesture sequences to obtain clustered gesture features.

[0103] It should be noted that, in one specific embodiment, the input for this step is a 3D coordinate sequence of 42 key points × 200 frames;

[0104] It should be noted that, in one specific embodiment, an improved YOLO-v8 architecture is adopted, and a spatial relationship detection head is added to detect complex cluster control actions such as "two-handed coordinated gestures" and "multi-finger combined gestures".

[0105] It should be noted that, in one specific embodiment, the HRNet architecture is adopted to maintain high-resolution representation throughout the process, further refine the coordinates of 42 hand key points, and in particular optimize the spatial relationship modeling between hand key points.

[0106] It should be noted that, in one specific embodiment, the keypoint sequence is input into a Transformer encoder. A multi-head self-attention mechanism is used to learn long-range temporal dependencies in the gesture sequence, capturing dynamic semantics such as "first clenching a fist, then opening it."

[0107] It should be noted that, in one specific embodiment, this step outputs a 1024-dimensional floating-point vector, which is the cluster gesture feature.

[0108] • The Conformer encoder is used to encode the mel spectral features, and the CRF layer is used to perform semantic parsing of the cluster entities, actions and parameters on its output to obtain cluster speech features;

[0109] It should be noted that, in one specific embodiment, the input for this step is an 80×T dimensional Mel feature sequence;

[0110] It should be noted that, in one specific embodiment, a 16-layer Conformer block is used for feature encoding. Each Conformer block integrates a multi-head self-attention mechanism to capture global dependencies between speech frames; a convolutional module to extract local speech features; and a feedforward network for nonlinear transformation.

[0111] It should be noted that, in one specific embodiment, a Conditional Random Field (CRF) layer is added on top of the Conformer output for sequence labeling to accurately identify semantic elements in the instructions, such as cluster entities (e.g., "first squad"), actions (e.g., "search", "ascend"), and parameters (e.g., "50 meters", "north").

[0112] It should be noted that, in one specific embodiment, this step outputs a 1024-dimensional floating-point vector, which is the cluster speech feature.

[0113] • The EfficientDet architecture is used to perform target detection and tracking on image data. The ResNet50+FPN architecture is used to perform multi-scale fusion on its output. The Attention mechanism is used to enhance the features of the specific area that the operator is focused on based on the operator's visual focus coordinates, so as to obtain the cluster visual features.

[0114] It should be noted that, in one specific embodiment, the input for this step is image data, eye-tracking focus, and preliminary detection results;

[0115] It should be noted that, in one specific embodiment, the EfficientDet architecture is used to perform target detection and tracking on the input image, and outputs the bounding box, ID, and position confidence of each drone in real time, thereby estimating the spatial distribution, flight status and formation integrity of the cluster;

[0116] It should be noted that, in one specific embodiment, based on the eye-tracking focus coordinates, the Attention mechanism is used to enhance the features of a specific area (such as a drone or an obstacle) that the operator is focused on, and to extract the detailed visual features of that area.

[0117] It should be noted that, in one specific embodiment, this step outputs a 1024-dimensional floating-point vector, which is the cluster visual feature.

[0118] • A bidirectional long short-term memory network is used to perform temporal trajectory analysis on each touch point to obtain temporal features. A geometric deep learning method is used to identify the overall shape formed by the clustered touch point set to determine what kind of formation or flight path it represents, thus obtaining spatial geometric features. Combining the temporal and spatial geometric features of the trajectory, a classification network is used to identify the operator's cluster planning intention to obtain cluster touch features.

[0119] It should be noted that, in one specific embodiment, this step inputs a trajectory sequence of N touch points and their labels;

[0120] It should be noted that, in one specific embodiment, a recurrent neural network (RNN), particularly a bidirectional long short-term memory network (Bi-LSTM), is used to process the temporal trajectory of each touch point and learn its motion pattern.

[0121] It should be noted that, in one specific embodiment, a geometric deep learning method is used to identify the overall shape (such as an arrow, circle, or polygon) formed by the clustered set of touch points, and to determine what kind of formation shape or flight path it represents.

[0122] It should be noted that, in one specific embodiment, the temporal and spatial geometric features of the integrated trajectory are combined to understand the operator's cluster planning intent (such as area patrol, encirclement, and delivery route planning) through a classification network.

[0123] It should be noted that, in one specific embodiment, this step outputs a 1024-dimensional floating-point vector, which is the cluster touch feature.

[0124] Based on the above embodiments, we can standardize and align the above features:

[0125] • Layer Normalization is performed on the cluster control features of each modality, and dynamic time warping algorithm is used to align the cluster control features of each modality;

[0126] It should be noted that, in one specific embodiment, LayerNormalization is performed on each feature vector to standardize its numerical distribution, which is beneficial for subsequent model training and fusion.

[0127] It should be noted that, in one specific embodiment, although the data is synchronized at the hardware layer, to eliminate the slight delays in different modal processing, the Dynamic Time Warping (DTW) algorithm is used to perform final, fine-grained alignment of the four feature sequences at the software level. The DTW distance formula is:

[0128]

[0129] X, Y: Two temporal feature sequences to be aligned.

[0130] : sequence point and Euclidean distance between

[0131] Weight of the regularized path.

[0132] like Figure 5 Based on the above embodiments, we map the features of different modalities to a unified cluster control semantic space:

[0133] • Define a unified cluster control semantic space that includes formation mode, maneuver type, coordination method, mission mode, and parameters;

[0134] It should be noted that, in one specific embodiment:

[0135] Formation modes: straight, V-shape, circular, free, etc.; Maneuver types: forward, backward, ascend, descend, turn, hover, etc.; Coordination methods: unified action, group action, independent action; Mission modes: cruise, search, surveillance, formation performance, etc.; Parameters: speed, altitude, direction, scale, etc.

[0136] • By using multiple MLP networks, the cluster control features of each modality are mapped to a unified cluster control semantic space, and contextual information is introduced to obtain the target cluster control features of each modality.

[0137] It should be noted that, in one specific embodiment:

[0138] Through four independent, learnable 3-layer MLP networks Each modality feature is mapped to this semantic space, and the mapping process incorporates contextual information:

[0139]

[0140]

[0141]

[0142]

[0143] in, Learnable scalar weight parameters It is a contextual information vector provided by other modal features.

[0144] Based on the above embodiments, in some embodiments, cross-modal semantic association learning is performed to obtain deep fusion features of each modality, including:

[0145] • Construct a fully connected graph G = (V, E) with a set of nodes representing each modality V = {gesture, voice, visual, touch} and an edge set E representing the semantic associations between modalities;

[0146] • The association strength (edge ​​weight) between each pair of nodes is calculated using a multilayer perceptron. Specifically:

[0147]

[0148] concat: concatenate vectors.

[0149] The absolute difference between vectors represents the difference.

[0150] Element-wise multiplication between vectors represents similarity.

[0151] softmax: Normalizes the weights.

[0152] • Perform multi-layer graph convolution operations on the fully connected graph. Update the features of each node based on the target cluster control features of its neighboring nodes and the strength of the association with its neighboring nodes, obtaining deep fusion features for each modality. Specifically:

[0153]

[0154] : The set of neighbors of node i.

[0155] : The learnable weight matrix of the l-th layer.

[0156] Activation function.

[0157] like Figure 6 In some embodiments, multimodal conflict resolution is performed on the deep fusion features of each modality to obtain a unified cluster control intent, including:

[0158] • Calculate the cosine similarity of the deep fusion features between any two modalities. If the cosine similarity is not greater than a first empirical value (which can be set to -0.5), it is determined that there is a semantic conflict between the two modalities.

[0159] • Calculate the relative error of numerical parameters (such as velocity and altitude) between any two modes. If the error is not less than a second empirical value (e.g., 0.5), it is determined that there is a parameter conflict between the two modes.

[0160] • Detect the relative error of timestamps between any two modes. If the error is not less than a third empirical value (set according to system response requirements, such as 100ms), it is determined that there is a timing conflict between the two modes.

[0161] • Detect whether any two modalities with the same priority have issued instructions with different content at the same time. If so, determine that the two modalities have a priority conflict.

[0162] • Comprehensively assess the severity of conflicts across all modalities. In cases of minor conflicts, perform confidence-weighted fusion of the deep fusion features of each modality to obtain a unified cluster control intent. In cases of moderate conflicts, determine the deep fusion feature of the highest priority modality as the unified cluster control intent. In cases of severe conflicts, determine the deep fusion feature of the operator-selected modality as the unified cluster control intent.

[0163] It should be noted that, in one specific embodiment, this applies to all strategies. In any conflict, if safety is involved, the safer option is automatically selected (e.g., a lower conflict speed, a higher conflict altitude, and a looser conflict formation).

[0164] It should be noted that, in one specific embodiment, the conflict severity assessment evaluates the number of conflicts occurring across four conflict dimensions (semantics, parameters, timing, and priority), classifying conflicts into three levels: minor (1), moderate (2), and severe (≥3).

[0165]

[0166] Conflict severity level. This is an integer ranging from [0, 4], with higher values ​​indicating more severe conflicts.

[0167] i: Conflict dimension index. i = 1 represents semantic conflict, i = 2 represents temporal conflict, i = 3 represents parameter conflict, and i = 4 represents priority conflict.

[0168] The indicator function is a binary function. When a conflict is detected in the i-th dimension, = 0, when no conflict is detected in the i-th dimension.

[0169] The formula means that the sum of the four conflict dimensions is calculated, and 1 is added for each conflict in each dimension, so that the total score of the conflict severity is obtained.

[0170] It should be noted that, in one specific embodiment, confidence-weighted fusion is used:

[0171]

[0172] : Normalized weights for the i-th mode. A weighted fusion strategy used in conflict resolution.

[0173] : The confidence score of the i-th modality. is a non-negative real number, usually output by the feature extraction network for each modality, and its value range is generally [0, 1]. The larger the value, the more reliable the recognition result of that modality.

[0174] N: Total number of modalities. In this embodiment, N=4, representing the four modalities of gesture, voice, vision, and touch.

[0175] j: Modal index variable. j = 1, 2, ..., N, used for summation operations in the denominator.

[0176] like Figure 7 and Figure 8 Based on the above embodiments, this embodiment will provide a detailed description of step S103:

[0177] In some embodiments, cluster control instructions are generated based on a unified cluster control intent. A central coordinator then decomposes these instructions into individual machine execution tasks and performs formation control, path planning, and global optimization, including:

[0178] • It adopts a Transformer decoder architecture, uses a unified cluster control intent as memory, decodes and generates structured cluster control instructions through a cross-attention mechanism, and uses a parameter regression network to determine the specific numerical parameters in the cluster control instructions.

[0179] In some specific embodiments, a 6-layer Transformer decoder architecture is employed. Using the fused semantic features as memory, a cross-attention mechanism is used to decode and generate a structured sequence of control commands. A parameter regression network is run in parallel to predict the specific numerical parameters required in the commands, such as precise flight speed (m / s), target altitude (m), formation size (m), heading angle (°), etc.

[0180] It should be noted that, in another specific embodiment, a real-time rule base is integrated during the instruction generation process to check whether the generated instructions comply with preset flight safety specifications (such as maximum speed, minimum obstacle avoidance distance, and no-fly zone restrictions).

[0181] It should be noted that, in one specific embodiment, the system predefines a series of cluster control instruction templates (JSON Schema). Based on the decoded instruction type and predicted parameters, the templates are filled to generate the final, parsable instructions.

[0182] • The core objective of the cluster control command is parsed by the central coordinator to obtain the cluster-level task. The cluster-level task is decomposed into multiple sub-cluster-level tasks. Specific tasks are assigned to each sub-cluster-level task to obtain multiple single-machine-level tasks. An improved Hungarian algorithm is used to solve the task allocation problem to obtain the task allocation list for each single-machine-level task.

[0183] It should be noted that cluster-level tasks: parsing the core objective of instructions, such as "searching an area of ​​25 square kilometers"; sub-cluster-level tasks: breaking down large tasks. For example, dividing the search area into a north area and a south area, and assigning them to human-machine groups A and B; designating group C as the communication relay; and single-machine-level tasks: assigning specific tasks to each UAV within the group, such as "UAV A1 flies along path P1, with sensor mode set to visible light scanning".

[0184] It should be noted that, in one specific embodiment, an improved Hungarian algorithm or a similar optimization algorithm is used to solve the task allocation problem, and the objective function is:

[0185]

[0186] min J: Minimize the total cost J. This is the objective of the task allocation optimization problem.

[0187] N: Number of drones.

[0188] M: Number of tasks.

[0189] i: Drone index. i = 1, 2, ..., N.

[0190] j: Task index. j = 1, 2, ..., M.

[0191] Let $\frac{i}{j}$ be the cost of the $i$-th drone performing the $j$-th task. $\frac{i}{j}$ is a non-negative real number that takes into account factors such as time, energy consumption, risk, and capability matching.

[0192] : Assignment decision variable. It is a binary variable (0 or 1). This means assigning the j-th task to the i-th drone. This indicates that no allocation will be made.

[0193]

[0194] J: Total cost function. This is the objective function that needs to be minimized, representing the overall cost of the entire drone swarm performing the task.

[0195] : The time it takes for the i-th drone to complete its mission. The unit is usually seconds (s).

[0196] Energy consumption of the i-th drone. The unit is usually watt-hours (Wh).

[0197] : Risk cost of the i-th drone. This is a dimensionless risk score; a higher value indicates a higher risk. w1, w2, w3: Weighting coefficients. These represent the weights of time, energy consumption, and risk in the total cost, respectively. They are manually set hyperparameters that satisfy the normalization constraint w1 + w2 + w3 = 1.

[0198] • Define a motion trajectory for each formation to represent the virtual leader of the entire formation. Determine the desired position of each UAV in the formation based on the position of the virtual leader, formation attitude, preset inter-UAV offset and formation scaling scale, and use PD to control the UAV to track the desired position. When the cluster control command requires the formation to change, use polynomial interpolation to plan a smooth trajectory for the virtual leader.

[0199] It should be noted that, in one embodiment, the control law is designed to allow the drone to track the desired position, typically using PD control:

[0200]

[0201] : The first derivative of the position of the i-th UAV (i.e., the velocity vector). The unit is meters per second (m / s). This is the output of the controller.

[0202] : The current position vector of the i-th UAV. is a three-dimensional coordinate vector. The unit is meters (m).

[0203] : The desired position vector of the i-th UAV. is a three-dimensional coordinate vector. The unit is meters (m).

[0204] : The desired velocity vector of the i-th UAV. is a three-dimensional velocity vector, with units of meters per second (m / s).

[0205] Position proportional gain. It can be a positive definite matrix or a positive scalar.

[0206] : Velocity proportional gain. It can be a positive definite matrix or a positive scalar.

[0207] like Figure 9 It should be noted that:

[0208] Formation modes include: straight formation, horizontal line arrangement (20-meter spacing, suitable for long-distance cruise), V-formation: goose-shaped V-formation (60° angle, suitable for drag reduction flight), circular formation: circular ring arrangement (30-meter radius, suitable for area encirclement), cube formation: 3D three-dimensional arrangement (40-meter side length, suitable for three-dimensional monitoring).

[0209] Formation parameter configuration includes: setting formation size, flight altitude (50-200 meters), speed (5-15 m / s), and safety distance constraint (5-10 meters);

[0210] The transformation planning algorithm includes: calculating the transformation trajectory using optimal transmission theory, using fifth-order polynomial interpolation to ensure a smooth transition, and completing the formation transformation within 15 seconds;

[0211] Obstacle avoidance strategy settings include: local path adjustment, formation deformation, group detour, real-time monitoring, and dynamic adjustment.

[0212] It should be noted that, in one embodiment, when the instruction requires a formation change, a fifth-order polynomial interpolation is used to plan a smooth trajectory for the virtual leader:

[0213]

[0214] V: Total potential energy of the formation. It is a non-negative real number, expressed in meters squared (m²). The smaller this value, the smaller the overall deviation between the current position and the desired position of the UAV, and the better the formation is maintained.

[0215] N: Number of drones in the formation.

[0216] i: Drone index. i = 1, 2, ..., N.

[0217] : The actual position vector of the i-th UAV .

[0218] : The expected position vector of the i-th UAV .

[0219] : The L2 norm of a vector (i.e., the Euclidean distance).

[0220] The square of the Euclidean distance. .

[0221]

[0222] q(t): The position of the trajectory at time t. It can be a scalar (such as a one-dimensional position) or a vector (such as a point in three-dimensional space), and the unit is meters (m).

[0223] t: Time variable. It is a non-negative real number, and the unit is seconds (s). Usually t ∈ [0, T_total], where T_total is the total time of the trajectory.

[0224] : Polynomial coefficients, a_0: determines the initial position of the trajectory, a_1: determines the initial velocity of the trajectory, a_2: determines half of the initial acceleration of the trajectory, a_3, a_4, a_5: higher-order polynomial coefficients, jointly determined by the endpoint boundary conditions of the trajectory (such as the position, velocity, and acceleration of the endpoint).

[0225] • Establish a global path optimization objective for the entire cluster, including total task completion time, total energy consumption, overall risk assessment, and coordination cost, and use optimization algorithms to find the Pareto optimal solution;

[0226] It should be noted that planning the globally optimal path for the entire cluster is modeled as a multi-objective optimization problem:

[0227]

[0228] f1(x): Total task completion time.

[0229] f2(x): Total energy consumption.

[0230] f3(x): Overall risk assessment (proximity to obstacles, severe weather, etc.).

[0231] f4(x): Coordination cost (such as communication load, control complexity).

[0232] Optimization algorithms (such as genetic algorithms and particle swarm optimization) are used to find Pareto optimal solutions.

[0233] • Continuously monitor the task load and remaining power of each drone, dynamically adjust task allocation, and continuously monitor the drone status. If a drone malfunctions, immediately initiate dynamic reallocation.

[0234] It should be noted that the task load and remaining battery power of each drone are monitored, and task allocation is dynamically adjusted to prevent some drones from being overloaded or running out of power prematurely. The drone status is continuously monitored. When a drone malfunctions (communication interruption, engine failure), dynamic reallocation is immediately initiated, assigning its tasks to other drones and adjusting the formation.

[0235] It should be noted that the task allocation matching degree is as follows:

[0236]

[0237] : The matching degree between the i-th drone and the j-th task. It is a real number with a value in the range [0, 1]. The larger the value, the higher the matching degree.

[0238] capability_i: The set of capabilities of the i-th drone. It is a set containing capability parameters such as the type of sensors carried by the drone, maximum payload, endurance, and maximum speed.

[0239] requirement_j: The set of requirements for the j-th task. It is a set containing requirements such as the type of sensor, minimum payload, maximum completion time, and special flight capabilities required to successfully execute the task.

[0240] ∩: Set intersection operation. Used to find the common elements of two sets, capability_i and requirement_j (i.e., the items for which the drone's capabilities meet the mission requirements).

[0241] | · |: Radix operation for sets. Used to count the number of elements in a set.

[0242] Based on the above embodiments, this embodiment will provide a detailed description of step S104:

[0243] In some embodiments, real-time monitoring of the overall cluster execution status, detection and handling of anomalies, and adaptive optimization of feature fusion and network parameters of the central coordinator and control parameters of cluster coordination control based on performance feedback to form a closed loop include:

[0244] • Receive real-time status data of each UAV (such as latitude and longitude, altitude, speed, attitude, battery voltage, sensor status, mission progress) through the communication link, merge the status data of all UAVs to generate an overall situational map of the cluster, and calculate performance indicators including formation quality, mission progress and system efficiency.

[0245] It should be noted that the performance metrics are calculated as follows:

[0246] Formation quality: Calculate formation holding accuracy (meters) and cohesion C ( Coordination .

[0247] Task progress: Real-time calculation of the overall task completion percentage and the completion status of each subtask.

[0248] System performance: Estimate total energy consumption and average task time.

[0249] • Based on machine learning models, the system analyzes the overall situation of the cluster in real time, identifies abnormal patterns, and provides abnormal information alarms, fault isolation, and emergency plan triggering in the event of a failure.

[0250] It should be noted that the system analyzes data in real time based on machine learning models (such as Isolation Forest and Autoencoder) to identify abnormal patterns: flight anomalies (severe shaking, deviation from the flight path), coordination anomalies (disorganized formation), communication anomalies (sudden increase in packet loss rate), and equipment anomalies (sudden drop in battery power, sensor failure).

[0251] Once a malfunctioning drone is identified, it should be immediately isolated, all new commands should be stopped, and nearby drones should be notified to maintain a safe distance from it.

[0252] The corresponding contingency plan is triggered according to the preset rule base, such as: emergency landing / return to home for command failure, activation of fault tolerance mechanism to reallocate tasks, emergency hovering of command cluster, etc.

[0253] • Fine-tune the control parameters of cluster coordination control in real time based on performance indicators;

[0254] It should be noted that control parameters are fine-tuned in real time based on performance feedback. For example, the PID gain of formation control can be adjusted: in, The learning rate;

[0255] • Employing a reinforcement learning framework, the cluster control process is viewed as a Markov decision process, with performance metrics as rewards, to continuously optimize the network parameters of feature fusion and the central coordinator.

[0256] It should be noted that a reinforcement learning framework is used, treating the cluster control process as a Markov decision process (MDP), and using performance metrics as rewards to continuously optimize the policy network parameters of the fusion network and the coordinator.

[0257]

[0258] The updated model parameter vector at time t+1.

[0259] : The model parameter vector at time t. It contains all the parameters that need to be optimized, such as the weights and biases of each layer of the neural network.

[0260] η: Learning rate. It is a positive real scalar used to control the step size of each parameter update, usually η ∈ (0,1).

[0261] : The gradient vector of the loss function L at the parameter θ_t. The gradient direction indicates the direction in which the loss function grows the fastest; therefore, a negative sign indicates updating in the direction that reduces the loss. Its dimension is the same as the parameter vector θ_t.

[0262] Loss function. It is a scalar function used to measure the difference between the model's predicted output and the true value (or the desired target) at the current parameters θ_t.

[0263] like Figure 10 , Figure 10 This is a flowchart illustrating the implementation of a deep learning-based multimodal perception fusion unmanned aerial vehicle (UAV) swarm control method, as provided in an embodiment of the present invention.

[0264] This invention also provides a deep learning-based multimodal perception fusion unmanned aerial vehicle (UAV) swarm control device; the specific device may include:

[0265] The data acquisition and preprocessing module is used to acquire multimodal operator input, including cluster gesture commands, cluster voice commands, cluster visual monitoring, and cluster touch planning, and to perform data preprocessing and data synchronization.

[0266] The unified cluster control intent acquisition module is used to extract cluster control features of each modality using a pre-designed deep neural network, map them to a unified cluster control semantic space, perform cross-modal semantic association learning to obtain deep fusion features of each modality, and perform multimodal conflict resolution on the deep fusion features of each modality to obtain the unified cluster control intent.

[0267] The cluster control and coordination module is used to generate cluster control commands based on a unified cluster control intent. The central coordinator decomposes the cluster control commands into individual machine execution tasks and performs formation control, path planning, and global optimization.

[0268] The adaptive update module is used to monitor the overall execution status of the cluster in real time, detect and handle anomalies, and adaptively optimize the feature fusion and network parameters of the central coordinator and the control parameters of the cluster coordination control based on performance feedback to form a closed loop.

[0269] The deep learning-based multimodal perception fusion UAV swarm control device of this embodiment is used to implement the aforementioned deep learning-based multimodal perception fusion UAV swarm control method. Therefore, the specific implementation of the deep learning-based multimodal perception fusion UAV swarm control device can be found in the previous embodiment section of the deep learning-based multimodal perception fusion UAV swarm control method. For example, the data acquisition and preprocessing module, the unified swarm control intent acquisition module, the swarm control coordination module, and the adaptive update module are respectively used to implement steps S101, S102, S103, and S104 in the above-mentioned deep learning-based multimodal perception fusion UAV swarm control method. Therefore, its specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0270] A specific embodiment of the present invention also provides a deep learning-based multimodal perception fusion UAV swarm control device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the aforementioned deep learning-based multimodal perception fusion UAV swarm control method.

[0271] A specific embodiment of the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described deep learning-based multimodal perception fusion UAV swarm control method.

[0272] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0273] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0274] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0275] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0276] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for controlling a multimodal perception fusion unmanned aerial vehicle (UAV) swarm based on deep learning, characterized in that, include: Acquire multimodal operator input, including cluster gesture commands, cluster voice commands, cluster visual monitoring, and cluster touch planning, and perform data preprocessing and data synchronization; The cluster control features of each modality are extracted by using a pre-designed deep neural network, mapped to a unified cluster control semantic space, and cross-modal semantic association learning is performed to obtain deep fusion features of each modality. Multimodal conflict resolution is performed on the deep fusion features of each modality to obtain a unified cluster control intent. Cluster control instructions are generated based on a unified cluster control intent. The central coordinator decomposes the cluster control instructions into individual machine execution tasks and performs formation control, path planning, and global optimization. The system monitors the overall execution status of the cluster in real time, detects and handles anomalies, and adaptively optimizes feature fusion and the network parameters of the central coordinator and the control parameters of the cluster coordination control based on performance feedback to form a closed loop.

2. The deep learning-based multimodal perception fusion UAV swarm control method according to claim 1, characterized in that, The process of acquiring operator multimodal input, including cluster gesture commands, cluster voice commands, cluster visual monitoring, and cluster touch planning, and performing data preprocessing and synchronization, includes: Depth RGB images of operator gestures are acquired using a depth camera array, transformed into a unified three-dimensional coordinate system, and the cluster gesture commands are temporally segmented using the sliding window method to segment out complete gesture segments that express the control intent. Key point data is then extracted to obtain preprocessed cluster gesture commands. The system uses a microphone array to collect operator voice commands on the console, uses an endpoint detection algorithm to locate the start and end points of the voice commands, filters out silent segments, performs beamforming voice enhancement, echo cancellation and noise suppression, and extracts mel spectrum features to obtain preprocessed cluster voice commands. Wide-angle camera arrays are used to collect visual scenes of the cluster's flight airspace. Eye-tracking devices are used to collect operator eye movements and extract the coordinates of the operator's visual focus. These coordinates are then calibrated and mapped onto the visual scene. Target detection is performed on the visual scene to obtain image data, which is the pre-processed cluster visual monitoring. The operator's touch point trajectory coordinate sequence is collected using a multi-touch display screen. The trajectory of each touch point is grouped using a clustering algorithm, and the operation semantics are identified to obtain the preprocessed cluster touch planning. The preprocessed multimodal inputs are timestamped, formatted uniformly, and their integrity is checked.

3. The deep learning-based multimodal perception fusion UAV swarm control method according to claim 2, characterized in that, The step of extracting cluster control features for each modality using a pre-designed deep neural network and mapping them to a unified cluster control semantic space includes: An improved YOLO-v8 architecture is used to detect clustered control actions on keypoint data. The HRNet architecture is used to optimize the modeling of spatial relationships between keypoints. The Transformer encoder and multi-head attention mechanism are used to learn long-distance temporal dependencies in gesture sequences to obtain clustered gesture features. The Conformer encoder is used to encode the mel spectral features, and the CRF layer is used to perform semantic parsing of the cluster entities, actions and parameters on its output to obtain the cluster speech features; The EfficientDet architecture is used to perform target detection and tracking on image data. The ResNet50+FPN architecture is used to perform multi-scale fusion on its output. The Attention mechanism is used to enhance the features of the specific area that the operator is focused on based on the operator's visual focus coordinates, so as to obtain the cluster visual features. A bidirectional long short-term memory network is used to perform temporal trajectory analysis on each touch point to obtain temporal features. A geometric deep learning method is used to identify the overall shape formed by the clustered touch point set to determine what kind of formation or flight path it represents, thus obtaining spatial geometric features. Combining the temporal and spatial geometric features of the trajectory, a classification network is used to identify the operator's cluster planning intention to obtain cluster touch features. Layer Normalization is applied to the cluster control features of each modality, and dynamic time warping algorithm is used to align the cluster control features of each modality. Define a unified cluster control semantic space that includes formation mode, maneuver type, coordination method, mission mode, and parameters; By mapping the cluster control features of each modality to a unified cluster control semantic space through multiple MLP networks and introducing contextual information, the target cluster control features of each modality are obtained.

4. The deep learning-based multimodal perception fusion UAV swarm control method according to claim 3, characterized in that, The cross-modal semantic association learning to obtain deep fusion features for each modality includes: Construct a fully connected graph where the node set represents each modality and the edge set represents the semantic relationships between modalities; The correlation strength between each pair of nodes is calculated using a multilayer perceptron. Multi-layer graph convolution is performed on the fully connected graph. Based on the target cluster control features of each node's neighboring nodes and the association strength between each node and its neighboring nodes, the features of each node are updated to obtain deep fusion features of each modality.

5. The deep learning-based multimodal perception fusion UAV swarm control method according to claim 4, characterized in that, The process of resolving multimodal conflicts by performing deep fusion features of each modality to obtain a unified cluster control intent includes: Calculate the cosine similarity of the deep fusion features between any two modalities. If the cosine similarity is not greater than the first empirical value, it is determined that there is a semantic conflict between the two modalities. Calculate the relative error of numerical parameters between any two modes. If the error is not less than the second empirical value, it is determined that there is a parameter conflict between the two modes. If the relative error between timestamps of any two modes is not less than a third empirical value, it is determined that there is a timing conflict between the two modes. Detect whether any two modalities with the same priority have issued instructions with different content at the same time. If so, determine that the two modalities have a priority conflict. The severity of conflicts across all modalities is comprehensively assessed. In cases of minor conflicts, the deep fusion features of each modality are weighted by confidence to obtain a unified cluster control intent. In cases of moderate conflicts, the deep fusion feature of the highest priority modality is determined as the unified cluster control intent. In cases of severe conflicts, the deep fusion feature of the operator-selected modality is determined as the unified cluster control intent.

6. The deep learning-based multimodal perception fusion UAV swarm control method according to claim 1 or 5, characterized in that, The process of generating cluster control commands based on a unified cluster control intent, decomposing these commands into individual machine execution tasks through a central coordinator, and performing formation control, path planning, and global optimization includes: The Transformer decoder architecture is adopted, with a unified cluster control intent as memory. The structured cluster control instructions are generated by decoding through a cross-attention mechanism, and the specific numerical parameters in the cluster control instructions are determined by a parameter regression network. The core objective of the cluster control command is parsed by the central coordinator to obtain the cluster-level task. The cluster-level task is then decomposed into multiple sub-cluster-level tasks. Specific tasks are assigned to each sub-cluster-level task to obtain multiple single-machine-level tasks. An improved Hungarian algorithm is then used to solve the task allocation problem to obtain the task allocation list for each single-machine-level task. A motion trajectory is defined for each formation to represent the virtual leader of the entire formation. The desired position of each UAV in the formation is determined based on the position of the virtual leader, the formation attitude, the preset inter-UAV offset and the formation scaling scale. PD is used to control the UAV to track the desired position. When the cluster control command requires the formation to change, polynomial interpolation is used to plan a smooth trajectory for the virtual leader. Establish a global path optimization objective for the entire cluster, including total task completion time, total energy consumption, overall risk assessment, and coordination cost, and use optimization algorithms to find the Pareto optimal solution; Continuously monitor the task load and remaining power of each drone, dynamically adjust task allocation, and continuously monitor the drone status. If a drone malfunctions, immediately initiate dynamic reallocation.

7. The deep learning-based multimodal perception fusion UAV swarm control method according to claim 1, characterized in that, The real-time monitoring of the overall execution status of the cluster, detection and handling of anomalies, and adaptive optimization of feature fusion and the network parameters of the central coordinator and the control parameters of the cluster coordination control based on performance feedback to form a closed loop include: The system receives the status data of each drone in real time through the communication link, merges the status data of all drones to generate an overall situational map of the cluster, and calculates performance indicators including formation quality, mission progress and system efficiency. Based on machine learning models, the system analyzes the overall situation of the cluster in real time, identifies abnormal patterns, and provides abnormal information alarms, fault isolation, and emergency plan triggering in the event of a failure. The control parameters of cluster coordination control are fine-tuned in real time based on performance indicators. By employing a reinforcement learning framework, the cluster control process is treated as a Markov decision process, with performance metrics as rewards, to continuously optimize the network parameters of feature fusion and the central coordinator.

8. A deep learning-based multimodal perception fusion unmanned aerial vehicle (UAV) swarm control device, characterized in that, include: The data acquisition and preprocessing module is used to acquire multimodal operator input, including cluster gesture commands, cluster voice commands, cluster visual monitoring, and cluster touch planning, and to perform data preprocessing and data synchronization. The unified cluster control intent acquisition module is used to extract cluster control features of each modality using a pre-designed deep neural network, map them to a unified cluster control semantic space, perform cross-modal semantic association learning to obtain deep fusion features of each modality, and perform multimodal conflict resolution on the deep fusion features of each modality to obtain the unified cluster control intent. The cluster control and coordination module is used to generate cluster control commands based on a unified cluster control intent. The central coordinator decomposes the cluster control commands into individual machine execution tasks and performs formation control, path planning, and global optimization. The adaptive update module is used to monitor the overall execution status of the cluster in real time, detect and handle anomalies, and adaptively optimize the feature fusion and network parameters of the central coordinator and the control parameters of the cluster coordination control based on performance feedback to form a closed loop.

9. A deep learning-based multimodal perception fusion unmanned aerial vehicle (UAV) swarm control device, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the deep learning-based multimodal perception fusion unmanned aerial vehicle swarm control method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the deep learning-based multimodal perception fusion unmanned aerial vehicle swarm control method as described in any one of claims 1 to 7.

Images