A multi-modal human-computer interaction intention recognition method for unmanned clusters

Abstract

The present application relates to a kind of multi-modal human-computer interaction intention recognition method for unmanned cluster, belong to cluster intelligence control and human-computer interaction technical field, solve the problem that multi-modal interaction in prior art is difficult to realize signal quality self-adaptive fusion and intention accurate analysis under complex environment.The method comprises: obtaining the multi-modal interaction signal of unmanned cluster controller, and pre-processing and time alignment, obtain the time alignment after each modality signal segment;After extracting the feature of each modality signal segment respectively, cross-modal semantic alignment is carried out, and the aligned each modality feature vector is obtained;Based on the quality evaluation result of each modality signal segment, the feature interaction of each modality feature vector is carried out, and the joint semantic embedding vector is obtained;Based on joint semantic embedding vector and pre-constructed knowledge base, the interaction intention of controller is analyzed and is decomposed into instruction sequence with time sequence logic;The instruction sequence is converted into the bottom layer control instruction of unmanned cluster and sent to unmanned cluster.

Description

Technical Field

[0001] This invention relates to the field of cluster intelligent control and human-computer interaction technology, and in particular to a multimodal human-computer interaction intent recognition method for unmanned clusters. Background Technology

[0002] With the development of unmanned systems technology, drone swarms are increasingly being used in search and rescue, logistics and transportation, and factory operations, effectively improving the efficiency of human-machine collaborative work. However, in existing interaction systems, the control architecture still generally follows a command-driven control mode. Limited by the traditional ground station design concept, operators mainly operate through terminal touch screens, supplemented by keyboard input or joystick control, resulting in a single operation method and high cognitive load.

[0003] Although technologies such as voice, motion sensing, and eye tracking have been gradually introduced in recent years to improve the naturalness of interaction, they are mostly limited to simple action mapping. Multimodal interaction is mostly limited to the level of simple action mapping, lacking a deep understanding of complex task intentions and semantic alignment capabilities. It is difficult to achieve efficient conversion from macro-level control intentions to low-level instructions. At the same time, in terms of multimodal fusion, existing technologies mostly adopt parallel recognition or channel switching mechanisms, lacking dynamic evaluation and adaptive fusion capabilities for modal signal quality. In complex environments such as strong noise and strong light, single-modal failures are prone to occur, affecting the robustness of the system. Summary of the Invention

[0004] Based on the above analysis, the embodiments of the present invention aim to provide a multimodal human-computer interaction intent recognition method for unmanned swarms, in order to solve the problem that existing multimodal interactions are difficult to achieve adaptive fusion of signal quality and accurate interpretation of intent in complex environments.

[0005] The objective of this invention is mainly achieved through the following technical solutions: On the one hand, the present invention provides a multimodal human-computer interaction intent recognition method for unmanned swarms, comprising the following steps: The multimodal interaction signals of the unmanned swarm operator are acquired, preprocessed, and time-aligned to obtain time-aligned signal segments of each modality; wherein, the multimodality includes speech, electromyography, and inertia; After extracting features from each modal signal segment, cross-modal semantic alignment is performed to obtain aligned feature vectors for each modality. Based on the quality assessment results of each modal signal segment, the feature vectors of each modality are interacted through a cross-modal attention mechanism to obtain a joint semantic embedding vector; Based on the joint semantic embedding vector and a pre-built knowledge base including action instruction mapping, the operator's interaction intention is parsed and decomposed into an instruction sequence with temporal logic; The instruction sequence is converted into the underlying control instructions of the unmanned cluster and sent to the unmanned cluster.

[0006] Furthermore, the step of extracting features from each modal signal segment in the multimodal signal segment and then performing cross-modal semantic alignment to obtain an aligned multimodal feature vector set includes: inputting each of the modal signal segments into a trained cross-modal alignment model to obtain the aligned multimodal feature vector set; wherein, The cross-modal alignment model includes a speech feature encoder, an electromyography feature encoder, and an inertial feature encoder arranged in parallel, as well as a linear projection layer connected to the output layer of each feature encoder. The speech feature encoder is used to extract feature vectors from speech signal segments; The electromyography feature encoder is used to extract feature vectors from electromyography signal segments; The inertial feature encoder is used to extract feature vectors from inertial signal segments; Each of the linear projection layers is used to map the feature vectors output by each feature encoder to a semantic space of the same dimension, so as to obtain aligned speech feature vectors, electromyographic feature vectors and inertial feature vectors.

[0007] Furthermore, the cross-modal alignment model is trained using the following method: Acquire multimodal signal segments under multiple different control commands. Each control command corresponds to a set of synchronously acquired speech signal segments, electromyographic signal segments, and inertial signal segments, and label the corresponding intention category. Based on each of the multimodal signal segments and the corresponding intent categories, a training dataset for the cross-modal alignment model is constructed; wherein, for each group of synchronously acquired multimodal signal segments, a signal segment of any one modality is used as an anchor point, a signal segment of another modality in the same group is used as a positive sample, and a signal segment of the same modality as the anchor point in other groups is used as a negative sample, thereby obtaining training data including anchor points, positive samples, negative samples, and intent category labels; The training dataset is loaded into the cross-modal alignment model. The contrast loss is calculated for each pairwise combination of modalities. The average of the contrast losses of each modal combination is used as the total loss to train the cross-modal alignment model, thus obtaining the trained cross-modal alignment model.

[0008] Furthermore, a quality assessment is performed on each modal signal segment, including: Based on each of the modal signal segments, the signal-to-noise ratio of each modal signal is obtained; Based on each modal signal segment, the feature completeness of each modal signal is obtained; wherein, the feature completeness is the proportion of non-zero frames in the modal signal segment; Based on the feature vectors output by the feature encoders of each modality, the confidence score of each modality signal segment is obtained; The signal-to-noise ratio, feature completeness, and confidence score of each modality are concatenated sequentially to obtain the quality vector of each modal signal segment.

[0009] Furthermore, based on the quality assessment results, the modal feature vectors are interacted through a cross-modal attention mechanism to obtain a joint semantic embedding vector, including: The quality vectors of each modal signal segment are input into the trained gating network to obtain the fusion weights of each modal feature vector; The corresponding modal feature vectors are weighted based on the fusion weights to obtain the weighted modal feature vectors. The weighted feature vectors of each modality are input into the trained cross-modal attention module for feature interaction to obtain a joint semantic embedding vector.

[0010] Furthermore, the gating network is trained using the following method: Construct a training dataset for the gated network; wherein the training dataset includes several sets of sample data, and each set of sample data includes the feature vector of each modality and its corresponding original quality vector; For each set of sample data, the original mass vector of one or more modes is randomly masked by a modal masking mechanism to obtain the masked mass vector of each mode. The masked quality vectors of each modality are input into the gating network to obtain the prediction weights of each modality. Based on the prediction weights, the feature vectors of each modality are weighted and fused to obtain the prediction fusion features; The original mass vectors of each modality are input into the gating network to obtain the target weights of each modality; Based on the target weights, the feature vectors of each modality are weighted and fused to obtain the target fusion features; Based on the predicted fusion features and the target fusion features, the reconstruction loss function is calculated; Based on the predicted fusion features, calculate the KL divergence loss between them and the uniform distribution; The gated network is trained using the sum of reconstruction loss and KL divergence loss as the total loss. When the training converges, the trained gated network is obtained.

[0011] Furthermore, the step of inputting the weighted modal feature vectors into the trained cross-modal attention module for feature interaction to obtain a joint semantic embedding vector includes: The weighted modal feature vectors are concatenated to obtain a feature vector matrix; where each row of the feature vector matrix corresponds to a feature vector of a modality. The feature vector matrix is ​​linearly mapped into a query matrix, a key matrix, and a value matrix, respectively. The query matrix, key matrix, and value matrix are input into the trained cross-modal attention module, and the attention weight matrix is ​​calculated by multiple parallel attention heads respectively. Based on the attention weight matrix of each attention head, the value matrix corresponding to each attention head is weighted and summed to obtain the updated feature matrix output by each attention head; The updated feature matrices output by each attention head are concatenated and aggregated to obtain a joint semantic embedding vector.

[0012] Furthermore, the cross-modal attention module is a multi-head attention module; The cross-modal attention module was trained using the following method: Construct a training dataset for the cross-modal attention module; wherein the training dataset includes several sets of sample data, and each set of sample data includes the weighted feature vectors of each modality output by the gating network; The training dataset is input into the cross-modal attention module, and the LoRA fine-tuning method is used to train the cross-modal attention module to obtain the trained cross-modal attention module.

[0013] Furthermore, the process of parsing the operator's interaction intent and decomposing it into a sequence of instructions with temporal logic based on the joint semantic embedding vector and a pre-built knowledge base including action instruction mapping includes: The joint semantic embedding vector is combined with the action instruction mapping rules in a pre-built knowledge base including action instruction mapping and preset prompt word templates. Through a large language model, an instruction sequence with clear semantic boundaries and temporal logic is obtained. The prompt word template includes role definition, action instruction mapping list and output format constraints.

[0014] Furthermore, it also includes controlling the unmanned platforms in the unmanned cluster based on the control commands, and dynamically correcting the underlying control commands based on feedback from the unmanned cluster; wherein, the dynamic correction of the control commands based on feedback from the unmanned cluster includes: The status information of each unmanned platform in the unmanned cluster is received in real time, and the deviation value between each status information and the preset state is calculated; wherein, the deviation value includes position deviation value, velocity deviation value and formation deviation value; When the deviation value exceeds the preset threshold, the control layer compensation command is activated to correct the flight trajectory and / or speed command of each unmanned platform. When the deviation value persists and cannot converge, the deviation value is converted into a vector value, concatenated with the quality vector of each modality, and then passed through the gating network and the cross-modal attention mechanism in sequence to obtain the updated joint semantic embedding vector. Based on the updated joint semantic embedding vector and the knowledge base, the operator's interaction intent is re-parsed and decomposed into an updated instruction sequence; The updated instruction sequence is converted into the underlying control instructions for the unmanned swarm, controlling the movement of each unmanned platform.

[0015] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: 1. This invention targets multi-source heterogeneous interactive signals such as speech, electromyography, and inertia. It adopts differentiated preprocessing and a unified time window mechanism to achieve temporal alignment, and uses a dedicated feature encoder to map different modal signals to a unified semantic space, thereby achieving the alignment of cross-modal interactive features at the temporal and semantic levels.

[0016] 2. This invention uses a dynamic modal gating network to adaptively allocate fusion weights based on the quality of each modal signal, and combines a cross-modal attention mechanism to achieve semantic interaction and information complementarity between different modal features. This improves the stability and reliability of multimodal intent representation in complex interference environments and effectively avoids the interruption problem caused by single-modal failure in traditional methods.

[0017] 3. This invention combines the fused multimodal features with a knowledge base including action command mapping. It uses a large language model to parse and decompose the high-level interactive intent of the operator and maps the generated task sequence into cluster-executable instructions. At the same time, it performs consistency detection on the instruction sequence based on the "If-Then" safety rule base, intercepts conflicting instructions in real time, and makes dynamic corrections based on the execution feedback of the UAV cluster. This forms a closed-loop control mechanism of intent understanding, instruction execution, and feedback adjustment, which improves the security and operational stability of the system in dynamic task environments.

[0018] 4. By constructing a knowledge base including platform configuration, task payload and cooperative rules, this invention formally models the dynamic constraints and task logic of heterogeneous UAVs, realizing automatic mapping from macroscopic abstract intentions to low-level specific instructions. Compared with existing methods that can only recognize simple actions such as "take-off" and "turn", this invention can handle macroscopic tasks with complex semantics and temporal logic, improving the accuracy and intelligence level of task execution.

[0019] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0020] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0021] Figure 1 This is a flowchart illustrating a multimodal human-computer interaction intent recognition method for unmanned swarms according to an embodiment of the present invention. Detailed Implementation

[0022] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0023] A specific embodiment of the present invention discloses a multimodal human-computer interaction intent recognition method for unmanned swarms, such as... Figure 1 As shown, it includes the following steps S1-S5: Step S1: Acquire the multimodal interaction signals of the unmanned swarm operator, and perform preprocessing and time alignment to obtain time-aligned signal segments of each modality; wherein, the multimodality includes speech, electromyography and inertia.

[0024] Specifically, raw interactive data streams are collected in real time through sensor arrays worn by unmanned swarm operators. ;in, The audio signal stream is captured via a headset microphone, with a sampling rate that can be set to 16kHz to capture the operator's voice commands. The electromyographic signal flow is represented by an array of electromyographic sensors worn on the operator's forearm, with a sampling rate set to 450Hz, used to capture the electrical activity of muscles triggered by hand gestures, in order to represent hand gestures. The inertial signal stream is acquired through an array of IMU (Inertial Measurement Units) worn on the operator's back and limbs, with a sampling rate set to 200Hz, to capture changes in body posture. It should be noted that, to ensure time consistency in subsequent multi-source data fusion, all three signals are timestamped with a unified system hardware clock, serving as a common time reference for synchronization and alignment of the multi-source signals.

[0025] Furthermore, for each modal interaction signal, differentiated preprocessing algorithms are used for noise reduction and feature enhancement; among them, For the preprocessing of the speech signal, this embodiment uses an adaptive filtering method for noise reduction, and its time-domain discrete output signal... The calculation equation is:

[0026] in, This represents the noise-reduced speech signal, which is a discrete point; Indicates the order of the filter; Indicates the sampling sequence number in the discrete time domain; This represents the standardized sample sequence index that is uniformly marked on the global timeline. These represent the adaptively updated filter coefficients used to filter out ambient background noise. This represents the original speech signal.

[0027] For electromyography (EMG) signal preprocessing, this embodiment uses wavelet transform to perform time-frequency analysis on the original EMG signal, filtering out small-amplitude jitter and power frequency interference. The transform formula is as follows:

[0028] in, The extracted time-frequency domain gesture features are represented by a coefficient matrix, which indicates the energy distribution of the signal at different time-frequency scales. Indicates the scaling factor; Indicates the time point at which the control signal is observed; Indicates the selected wavelet basis function; Represents a time variable.

[0029] For the coefficient matrix Soft threshold denoising is performed to filter out coefficients corresponding to small noise, and then the time domain signal is reconstructed by wavelet inverse transform to obtain the denoised electromyographic signal.

[0030] For the preprocessing of inertial signals, this embodiment uses the extended Kalman filter algorithm to perform attitude calculation and drift elimination on the original inertial signal, fuses angular velocity and gravity vector data, and calculates the original signal into a standardized quaternion attitude representation through prediction and update iteration, and outputs the state vector q as the preprocessed inertial signal.

[0031] It should be noted that, due to the different sampling rates and physical characteristics of the three signals, this embodiment uses a sliding window to generate multimodal time window signals of equal duration to ensure consistency on the time axis. It can be expressed using the following formula:

[0032] in, This represents the nth time-aligned multimodal time window signal, including all aligned multimodal signals within the window time; n represents the window index, n=1,2,...,N, where N represents the total number of windows in the current acquisition session; Indicates the time step of each window movement; Indicates the fixed duration of window coverage; This represents the set of preprocessed signals for the j-th time. , This represents the j-th speech signal after preprocessing. This represents the j-th electromyographic signal after preprocessing. This represents the j-th inertial signal after preprocessing; The time stamp representing the j-th preprocessed signal set is generated by a unified hardware clock during acquisition. Indicates the start time of the nth window; This represents the end time of the nth window.

[0033] Understandably, the above sliding window mechanism is used to align and extract asynchronous multi-source signal streams with equal durations, generating a series of continuous time-aligned windows. Each window includes preprocessed signals from three sources: speech, electromyography, and inertial signals within the same time period. It should be noted that, considering the possibility of a signal temporarily stopping or being delayed in the actual acquisition environment, such as when the operator does not output any instructions, missing modal sensor data is filled with all-zero vectors at the corresponding positions.

[0034] Furthermore, since the operator's voice, gestures, and body movements occur continuously on the timeline, and the start and end times of a single control command vary depending on modal characteristics and command type, single task boundary detection is performed on each preprocessed modal signal. The start and end time boundaries of a complete control command are determined by combining voice endpoint detection and motion stillness judgment.

[0035] Specifically, the preprocessed speech signal stream is processed using a speech endpoint detection method, which uses short-time energy threshold and zero-crossing rate as criteria to detect the start and end of speech activity, thereby obtaining the timestamp interval of the speech command. .

[0036] The action-resting state judgment method was applied to both electromyographic (EMG) signal streams and inertial signal streams. The mean signal energy was calculated using a sliding window, and the duration interval exceeding the resting state threshold was used to obtain the EMG command timestamp interval. and inertial command timestamp interval .

[0037] Understandably, due to the different physical characteristics of the three signals, their detected start and end times may differ. To ensure the integrity of this command and avoid loss of command information due to delay or premature termination of one mode, this embodiment uses the earliest start timestamp among the three signals as the command start boundary and the latest end timestamp as the command end boundary, thus obtaining the time boundary range of the complete control command for this instruction. ,in, .

[0038] Furthermore, from the aforementioned multimodal time window signals, the time tag located within the time boundary interval of the current instruction is extracted. All time windows within the time frame are aggregated in chronological order to form the multimodal signal segment corresponding to the current instruction; wherein, the multimodal signal segment includes all aligned window frames of the three signals within the time boundary interval, to ensure that the multimodal semantic information of the same control instruction is not broken due to window cutting.

[0039] Step S2: Extract features from each modal signal segment and perform cross-modal semantic alignment to obtain aligned modal feature vectors, including: inputting each modal signal segment into a trained cross-modal alignment model to obtain an aligned set of multimodal feature vectors.

[0040] Specifically, the cross-modal alignment model includes a speech feature encoder, an electromyography feature encoder, and an inertial feature encoder arranged in parallel, as well as a linear projection layer connected to the output layer of each feature encoder.

[0041] The speech feature encoder is used to extract feature vectors from speech signal segments.

[0042] The electromyography feature encoder is used to extract feature vectors from electromyography signal segments.

[0043] The inertial feature encoder is used to extract feature vectors from inertial signal segments.

[0044] Each of the linear projection layers is used to map the feature vectors output by each feature encoder to a semantic space of the same dimension, so as to obtain aligned speech feature vectors, electromyographic feature vectors and inertial feature vectors.

[0045] Specifically, the speech feature encoder can be a temporal convolutional network or a Transformer structure, used to extract deep features rich in semantic information from the speech signal segment. The electromyography feature encoder can be a temporal convolutional network structure, used to extract fine-grained temporal features of hand gestures from electromyography signal segments.

[0046] The inertial feature encoder can be a temporal convolutional network or a bidirectional LSTM structure, used to extract spatiotemporal features of body posture from inertial signal segments.

[0047] More specifically, since the speech feature encoder, the electromyography feature encoder and the inertial feature encoder have different structures, their output feature vector dimensions are usually inconsistent, making it impossible to directly perform cross-modal semantic comparison. Therefore, a linear projection layer is connected after each feature encoder to map the feature vectors of each modality to a common semantic space of the same dimension. The projected feature vectors are then L2 normalized to eliminate the influence of modulus length differences on subsequent similarity calculations.

[0048] Furthermore, in order to enable the cross-modal alignment model to map features from different modalities to a unified semantic space, this embodiment uses a contrastive learning method to train the cross-modal alignment model, including: Acquire multimodal signal segments under multiple different control commands. Each control command corresponds to a set of synchronously acquired speech signal segments, electromyographic signal segments, and inertial signal segments, and the corresponding intention category is labeled.

[0049] Specifically, several operators are organized to execute multiple sets of predefined standard control commands, such as "area patrol", "takeoff", "turn left", "hover", "return", "turn left first and then ascend", etc. When each set of commands is executed, the operator's voice, electromyography and inertial signals are collected simultaneously and preprocessed as in step S1 to obtain preprocessed multimodal signal segments. At the same time, each command is labeled with the corresponding intent category.

[0050] Based on each of the multimodal signal segments and the corresponding intent categories, a training dataset for the cross-modal alignment model is constructed. For each group of synchronously acquired multimodal signal segments, a signal segment of any one modality is used as an anchor point, a signal segment of another modality in the same group is used as a positive sample, and a signal segment of the same modality as the anchor point in other groups is used as a negative sample, resulting in training data including anchor points, positive samples, negative samples, and intent category labels.

[0051] Specifically, contrastive learning aims to bring features from different modalities under the same command closer together in the semantic space, while keeping features from different commands further apart. For each command, the synchronously acquired speech, electromyography (EMG), and inertial signal segments are treated as a group of multimodal signals with the same semantic label. For example, the speech signal segment acquired under the "takeoff" command is used as the anchor point, the EMG signal segment acquired under the "takeoff" command is used as its positive sample, and the speech signal segment acquired under the "landing" command is used as the negative sample, thus constructing the training data.

[0052] The training dataset is loaded into the cross-modal alignment model. The contrast loss is calculated for each pairwise combination of modalities. The average of the contrast losses of each modal combination is used as the total loss to train the cross-modal alignment model, thus obtaining the trained cross-modal alignment model.

[0053] Specifically, the contrastive loss function is expressed as:

[0054] in, This represents the value of the contrast loss function; Represents the anchor point modal eigenvector; Represents the feature vector of a positive sample; This represents the feature vector of the comparison sample; m represents the traversal index of the comparison sample. This represents the cosine similarity function, used to measure the degree of similarity between vectors; The hyperparameter representing the similarity sensitivity is set to 0.07 in this embodiment; This indicates the total number of samples in the current batch.

[0055] More specifically, the above-mentioned loss is constructed for each pairwise direction of the three modalities and the average value is taken as the total alignment loss. This loss is then jointly optimized with the parameters of each modal encoder and projection layer. After training convergence, the three-way alignment feature vector with uniform dimension is obtained through the trained cross-modal alignment model.

[0056] Step S3: Based on the quality assessment results of each modal signal segment, the feature vectors of each modality are interacted through a cross-modal attention mechanism to obtain a joint semantic embedding vector.

[0057] Specifically, in real-world unmanned swarm control environments, the acquisition quality of multimodal signals is affected by various factors. For example, speech signals may be interfered with by background noise such as drone rotor noise and wind noise; electromyographic signals may be weakened due to poor sensor-body contact or sweat; and inertial signals may accumulate errors due to vigorous movement. If the quality of each modality is ignored during fusion and fusion is performed with the same weight, then when a modality is severely interfered with, the noise or erroneous information it contains will contaminate the fusion result, leading to deviations or even complete errors in the intended understanding. Therefore, this embodiment evaluates the quality of each modality signal segment and adjusts the fusion weight of each modality based on the evaluation results. High-quality modalities are given higher weights, while low-quality modalities are given lower weights or even suppressed, thereby ensuring the robustness and reliability of the fusion result.

[0058] Furthermore, the quality of each modal signal segment is evaluated, including the following steps S301-S304: Step S301: Based on each modal signal segment, obtain the signal-to-noise ratio of each modal signal.

[0059] Specifically, the signal-to-noise ratio reflects the degree to which the signal is contaminated by environmental noise, and this embodiment uses the energy ratio to quantify it.

[0060] For voice signals, the timestamp interval of the voice command will be used. The active speech segment is used as the signal segment, and the 0.5-second silence segment before the start of the instruction is used as the noise segment. The short-time energy averages of the two signal segments are obtained respectively. Based on the short-time energy averages of the two signal segments, the signal-to-noise ratio of the speech signal is obtained using the following formula:

[0061] in, The signal energy of the speech signal is obtained by averaging the squares of all speech sampling points within the time period. The noise energy of the speech signal is obtained by averaging the squares of all sampling points within the noise time period. For electromyographic signals, this embodiment uses frequency domain filtering to filter the timestamp intervals of electromyographic commands. The electromyographic signals within the band are bandpass filtered from 20 to 450 Hz. The filtered signal energy is taken as the effective signal energy, and the out-of-band energy is taken as the noise energy. The energy values ​​are obtained by averaging the squares of the signal sampling points. The signal-to-noise ratio is calculated using the following formula:

[0062] in, This represents the energy of the filtered signal; It represents external energy.

[0063] For inertial signals, this embodiment uses a static calibration method. Inertial data is collected during system initialization in a 2-second static state. The variance of each axis's data is calculated as the noise basis, within the inertial command timestamp interval. The variance of the data for each axis is used as the signal variance, and the average value of the calculated signal-to-noise ratio for each axis is used as the signal-to-noise ratio of the inertial signal.

[0064] Step S302: Based on each modal signal segment, obtain the feature completeness of each modal signal; wherein, the feature completeness is the proportion of non-zero frames in the modal signal segment.

[0065] Specifically, the feature completeness is used to reflect the degree of data completeness of each modal signal within the time range of the current instruction.

[0066] For the feature completeness of the speech signal, the timestamp interval of the speech command is... The proportion of non-silent frames within the frame.

[0067] For the feature completeness of electromyographic signals, the electromyographic command timestamp interval is defined. The proportion of valid data frames within the frame.

[0068] For the feature integrity of inertial signals, the inertial command timestamp interval is defined. The proportion of valid data frames within the frame.

[0069] Step S303: Based on the feature vectors output by the feature encoders of each modality, obtain the confidence score of each modality signal segment.

[0070] Specifically, the confidence score reflects the degree of confidence that the feature encoder has in each of the current input modal signals.

[0071] In this embodiment, the mean and variance of the L2 norm of all sample data in the training dataset of the cross-modal alignment model are pre-calculated; for the feature vectors output by the feature encoder of each modality, their L2 norms are calculated respectively; based on the mean and variance of the training data, the L2 norms of the feature vectors of each modality are normalized, and the sigmoid function is used to map the confidence of each modality to the (0,1) interval to obtain the confidence score of each modality signal segment.

[0072] Step S304: The signal-to-noise ratio, feature completeness, and confidence score of each modality are concatenated sequentially to obtain the quality vector of each modality signal segment.

[0073] Specifically, due to the differences in physical characteristics of different modes, the numerical distribution range of quality evaluation items for each mode is significantly different. Therefore, in this embodiment, each quality evaluation item for each mode is normalized separately, and then the evaluation values ​​of the normalized quality evaluation items are concatenated in order to obtain the quality vector of each mode signal segment.

[0074] It should be noted that there is no strict execution order between steps S301 and S303. Those skilled in the art should understand that the processing of the above two branches is independent in logic and time, and their execution order should not constitute a limitation on the scope of protection of this invention.

[0075] Furthermore, based on the quality assessment results, the feature vectors of each modality are interacted through a cross-modal attention mechanism to obtain a joint semantic embedding vector, including steps S311-S313: Step S311: Input the quality vector of each modal signal segment into the trained gating network to obtain the fusion weight of each modal feature vector.

[0076] Specifically, the gated network adopts a lightweight multilayer perceptron structure, which processes each modality independently but shares network parameters. The input is the quality vector of each modality signal segment. After being mapped by the multilayer perceptron, the original weights corresponding to each modality are output and Softmax normalized to obtain the fusion weights where the sum of the modality weights is 1.

[0077] Furthermore, the gating network is trained using a modal masking mechanism, including the following steps S3111-S3119: Step S3111: Construct the training dataset of the gated network; wherein the training dataset includes several sets of sample data, and each set of sample data includes the feature vector of each modality and its corresponding original quality vector.

[0078] Specifically, each modal signal segment in the training dataset of the cross-modal alignment model is used, and the aligned modal feature vectors are obtained by passing them through the trained cross-modal alignment model. The quality vectors of each modal signal segment are obtained by performing quality evaluation as described in steps S301-S304. The above set of feature vectors and corresponding quality vectors are used as sample data for a set of gating networks.

[0079] Step S3112: For each set of sample data, the original quality vector of one or more modes is randomly masked by a modal masking mechanism to obtain the masked quality vector of each mode.

[0080] Specifically, during the training phase of the gated network, the original quality vectors of one or more modalities are randomly masked to simulate different input conditions such as single-modal, dual-modal, or full-modal modes. This drives the gated network to learn and identify the missing and available states of each modality, enabling the gated network to automatically adjust the fusion weights of each modality according to the actual input conditions. When a modality is affected by environmental interference or is missing, its variance will abnormally increase or its confidence will decrease, and the weight of that modality will be automatically reduced. For example, in a high-noise environment, the weight of the speech signal will be automatically reduced due to environmental interference, while the weight of the electromyography modality, which has physical noise immunity, will be increased to ensure that the fusion process is not contaminated by noise.

[0081] More specifically, the masking operation sets the mass vector of a randomly selected mode to a vector of all zeros.

[0082] Step S3113: Input the masked quality vector of each mode into the gating network to obtain the prediction weight of each mode.

[0083] Specifically, the mass vectors of each masked mode are input into the current gating network, which then outputs the prediction weights for each mode after forward computation. In essence, in scenarios where the mode mask simulates signal loss, the gating network obtains the prediction weights for each mode based on the currently available, albeit incomplete, mass information.

[0084] Step S3114: Based on the prediction weights, the feature vectors of each modality are weighted and fused to obtain the prediction fusion features.

[0085] Step S3115: Input the original mass vector of each mode into the gating network to obtain the target weight of each mode.

[0086] Specifically, the original quality vectors in the unmasked sample data are passed through the gating network with identical parameters to obtain the target weights for each modality. Understandably, in scenarios with missing signals, the gating network first outputs a weight allocation scheme, which is then compared with the target weights under complete information to train the network to make reasonable weight allocations even with incomplete information.

[0087] Step S3116: Based on the target weights, the feature vectors of each modality are weighted and fused to obtain the target fused features.

[0088] Step S3117: Calculate the reconstruction loss function based on the predicted fusion features and the target fusion features.

[0089] Specifically, the reconstruction loss is used to measure the consistency between the predicted fusion features output by the gating network and the target fusion features obtained using complete quality information in the case of modality masking. The reconstruction loss function can be calculated using mean squared error. Understandably, the reconstruction loss forces the gating network to recover, as far as possible, fusion features consistent with those obtained with complete information, using the quality information of the remaining modalities, even when some modal information is missing.

[0090] Step S3118: Based on the predicted fusion features, calculate the KL divergence loss between them and the uniform distribution.

[0091] Specifically, in order to prevent the gated network from over-relying on a single mode, this embodiment calculates the KL divergence loss between it and a uniform distribution. Through the KL divergence loss function, the gated network encourages the fusion weight distribution to be as uniform as possible when the quality of each mode is comparable. When the quality of a certain mode is significantly better than that of other modes, weight concentration is allowed, but premature concentration on a certain mode is avoided without sufficient reason.

[0092] Step S3119: Train the gated network using the sum of reconstruction loss and KL divergence loss as the total loss. When the training converges, the trained gated network is obtained.

[0093] Specifically, with the goal of minimizing the total loss, the Adam optimizer is used to optimize the parameters of the gated network. In each training batch, different modal mask patterns are randomly applied to allow the network to fully learn reasonable weight allocation strategies under various signal absence conditions. Training stops when the loss tends to stabilize, resulting in a well-trained gated network.

[0094] Understandably, the gating network receives the quality vectors of each modality as input and outputs a weight for each modality through forward computation. This weight reflects the relative importance of the modality in the current instruction. That is, high-quality modalities are given larger weights so that they dominate in subsequent fusion, while low-quality modalities are given smaller weights to reduce their negative impact on the fusion result.

[0095] Through this dynamic weight allocation mechanism, the system can automatically adjust the fusion strategy under different environments and task scenarios. For example, in a noisy environment, the weight of speech is reduced and the weight of electromyography is increased; when light interference makes gesture recognition unreliable, the weight of electromyography is reduced and the weight of inertia is increased. This ensures that the fused features are always based on the most reliable modal information at present, thereby improving the robustness and environmental adaptability of the overall system.

[0096] Step S312: Weight the corresponding modal feature vectors based on the fusion weights to obtain the weighted modal feature vectors.

[0097] Step S313: Input the weighted modal feature vectors of each modality into the trained cross-modal attention module for feature interaction to obtain a joint semantic embedding vector.

[0098] Specifically, after completing the modality weight allocation, in order to further explore the deep semantic relationships between different modalities, this embodiment inputs the weighted multimodal feature vectors into the cross-modal attention module, and realizes information complementarity and semantic interaction between modalities through the attention mechanism.

[0099] Furthermore, the step of inputting the weighted modal feature vectors into the trained cross-modal attention module for feature interaction to obtain a joint semantic embedding vector includes: The weighted modal feature vectors are concatenated to obtain a feature vector matrix; where each row of the feature vector matrix corresponds to a feature vector of a modality.

[0100] Specifically, the weighted feature vectors of the three modalities—speech, electromyography, and inertia—are concatenated to obtain a feature vector matrix.

[0101] The feature vector matrix is ​​linearly mapped to a query matrix, a key matrix, and a value matrix, respectively.

[0102] Specifically, the concatenated feature vector matrix is ​​transformed using three different linear projection matrices to obtain the query matrix, key matrix, and value matrix.

[0103] The query matrix, key matrix, and value matrix are input into the trained cross-modal attention module, and the attention weight matrix is ​​calculated by multiple parallel attention heads.

[0104] Specifically, the cross-modal attention module is a multi-head attention module, which uses multiple parallel attention heads to perform calculations separately, enabling it to focus on the relationship between modalities from different perspectives.

[0105] Based on the attention weight matrix of each attention head, the value matrix corresponding to each attention head is weighted and summed to obtain the updated feature matrix output by each attention head.

[0106] Specifically, each row of the attention weight matrix is ​​used as a weight, and the corresponding rows of the value matrix are weighted and combined to obtain the updated feature matrix of each attention head output.

[0107] The updated feature matrices output by each attention head are concatenated and aggregated to obtain a joint semantic embedding vector.

[0108] Specifically, the updated feature matrices output by multiple attention heads are concatenated in the column direction. For each row of the concatenated feature matrix, an aggregation operation is performed. In this embodiment, the features of each row are averaged to compress the information of the three modalities into one vector, resulting in the final joint semantic embedding vector.

[0109] Understandably, the cross-modal attention module projects features from different modalities onto the same semantic space through linear mapping and uses attention scores to measure cross-modal semantic relevance in order to achieve semantic alignment. This allows manipulation intentions that are difficult to express in a single modality to be supplemented by multimodal joint representations. Finally, the cross-modal attention module outputs a joint semantic embedding vector of a unified dimension.

[0110] Furthermore, the cross-modal attention module is trained using the following method: Construct a training dataset for the cross-modal attention module; wherein the training dataset includes several sets of sample data, and each set of sample data includes the weighted feature vectors of each modality output by the gating network.

[0111] Specifically, using the sample data in the training dataset of the gated network, the fusion weights of each modal feature vector in the sample data of the gated network are obtained through the trained gated network; based on the fusion weights, the weighted modal feature vectors in the sample data of the gated network are weighted and calculated to obtain the weighted modal feature vectors as the sample data of the cross-modal attention module.

[0112] The training dataset is input into the cross-modal attention module, and the LoRA fine-tuning method is used to train the cross-modal attention module to obtain the trained cross-modal attention module.

[0113] Specifically, LoRA (low-rank adaptation) fine-tuning is a parameter-efficient fine-tuning method that adds trainable low-rank decomposition matrices to specific layers of the model while keeping the original parameters of the pre-trained model frozen, and trains only these newly added parameters.

[0114] The cross-modal attention module described in this embodiment is first self-supervised pre-trained on a large-scale general multimodal dataset before application. Through pre-training, the model learns general correlation patterns between multimodal signals, providing a good initialization foundation for subsequent fine-tuning. The large-scale general multimodal dataset includes a large number of unannotated speech, electromyography, and inertial signals, covering various scenarios and command types.

[0115] During fine-tuning, the weight parameters of all original linear mapping matrices in the multi-head attention mechanism of the cross-modal attention module remain unchanged. Based on this, low-rank decomposition matrices are added in parallel to the query projection matrix, value projection matrix, and multi-head output integration matrix. Each low-rank decomposition matrix includes a dimension-reduced matrix and a dimension-increasing matrix, with its rank being much smaller than the dimension of the original projection matrix. During training, only the parameters of the added low-rank decomposition matrices are set to trainable, while the original pre-trained parameters remain unchanged.

[0116] The training dataset of the cross-modal attention module is input into the cross-modal attention module, which includes a low-rank factorization matrix. The input partial modal features are randomly masked. The cross-modal attention module reconstructs the features of the masked modality using the unmasked modal features. The error between the reconstructed features and the original features is used as the optimization objective. Training stops when the error converges.

[0117] After fine-tuning, the trained low-rank decomposition matrix is ​​merged with the original frozen projection matrices to obtain the trained cross-modal attention module.

[0118] Step S4: Based on the joint semantic embedding vector and a pre-built knowledge base including action command mapping, parse the operator's interaction intent and decompose it into a sequence of commands with temporal logic, including: The joint semantic embedding vector is combined with the action instruction mapping rules in a pre-built knowledge base including action instruction mapping and preset prompt word templates. Through a large language model, an instruction sequence with clear semantic boundaries and temporal logic is obtained. The prompt word template includes role definition, action instruction mapping list and output format constraints.

[0119] Specifically, by pre-constructing a knowledge base that includes action instruction mapping, structured knowledge support is provided for the intent parsing of large language models.

[0120] When building the knowledge base, firstly, for each unmanned platform in the unmanned swarm, its capability parameters are statistically collected, mainly including: platform configuration (such as rotary-wing UAVs, racing drones, etc.), mission payload (such as cameras, radars, thermal imagers, etc.), autonomy level (such as human control, fully autonomous, semi-autonomous), formation mode (such as changing formation, fixed formation, etc.), load-bearing capacity, etc.

[0121] Based on the capability parameters of each unmanned platform, the data is stored in a structured manner in the form of dictionary data. Each platform corresponds to a dictionary record with multiple fields, resulting in a platform capability parameter library. For example, it can be "{Platform ID: UAV_01, Configuration: Quadrotor, Payload: [Visible light camera, LiDAR], Autonomy level: 3}".

[0122] Furthermore, based on common unmanned platform task types, a requirement template is defined for each task. Each task template includes constraints such as the payload type, minimum autonomy level, and minimum load-bearing capacity required to perform the task, resulting in a task requirement template library for matching and querying during subsequent task allocation. For example, the task template can be "{Area Inspection: Payload must include [visible light camera], load-bearing capacity ≥ 0.5 kg}".

[0123] Furthermore, various interactive actions of the operator are defined, and each action is assigned a clear semantic label. For gesture actions, there are static gestures and dynamic gestures. Static gestures correspond to standardized instructions, such as making a circle with both hands to indicate "start", spreading five fingers to indicate "stop", making a fist to indicate "confirm", and crossing hands to indicate "cancel". Dynamic gestures correspond to directional or trajectory instructions, such as waving the hand to the left to indicate "turn left", raising the arm to indicate "rise", and drawing a circle with the arm to indicate "circle".

[0124] For voice commands, these are commands that are converted into text form through speech recognition, such as "take off," "return to base," and "area patrol."

[0125] For body movements, including whole-body movements and local joint movements, such as leaning forward to indicate "moving forward" and turning the head to indicate "shifting the viewpoint", the focus is on posture and direction information.

[0126] Furthermore, the various executable instructions and parameters within those instructions are defined, including task-level instructions and behavior-level instructions. The task-level instructions correspond to macro-level tasks. For example, "area patrol" includes parameters such as geographical range and altitude; "hover observation" includes parameters such as hovering height and observation angle.

[0127] The behavior-level instructions correspond to atomic actions, such as "take off", "land", "turn left", "move forward", "hover", "turn on camera", etc. Each behavior instruction includes the parameters required to execute the instruction.

[0128] It should be noted that the parameter values ​​for each command are all manually entered.

[0129] Furthermore, based on the various interactive actions of the operator and the various commands that the unmanned platform can execute, an action-command mapping table is constructed from the operator's interactive actions to the commands executed by the drone. For single-modal input, the semantic label of each interactive action is directly mapped to the corresponding drone command. For example, spreading five fingers is mapped to the hover command, making a circle with both hands is mapped to the takeoff command, waving the hand to the left is mapped to the left turn command, and the voice "take off" is mapped to the takeoff command. For multimodal combined input, the semantic combination of multiple actions is mapped to a composite command. For example, drawing a circle with the arm while saying "this area" is mapped to the area patrol command.

[0130] Furthermore, for each predefined task type, a standard behavior sequence template is predefined based on the execution logic. For example, the behavior sequence for {[Area Patrol] is: [Takeoff → Fly to target area → Activate payload → Fly along route → Hover observation → Return → Land]}; the behavior sequence for {[Supply Delivery] is: [Takeoff → Fly to delivery point → Descend to delivery altitude → Release supplies → Hover confirmation → Return → Land]}. It should be noted that when applying the standard behavior sequence template, the behavior sequence needs to be adjusted in conjunction with the platform capability parameter library. If the unmanned platform is not equipped with the payload required for the task, the corresponding behavior should be replaced with an alternative or marked as unexecutable. If the unmanned platform's autonomy level is insufficient, behaviors awaiting operator confirmation need to be inserted at key nodes. The dependencies between behaviors also need to ensure that the execution order of the behavior sequence conforms to the task logic. For example, "Takeoff" must be executed before "Fly to target area". It is understood that the dependencies between behaviors are explicitly defined in the standard behavior sequence template to ensure that the execution order of the behavior sequence conforms to the task logic.

[0131] Furthermore, for each behavioral command, a behavior-control conversion rule is established based on the unmanned platform's dynamics model and control algorithm to translate the command into underlying control parameters. For example, for the "takeoff" command, the desired attitude and thrust required to reach the target altitude are calculated based on the platform's dynamics model, and then converted into specific motor control signals using a PID control algorithm; for the "turn left" command, the yaw rate setpoint is calculated based on the target turning angle and turning speed, and then converted into the speed difference of each motor. Each conversion rule is based on the dynamic parameters corresponding to the platform type, while also considering the impact of real-time environmental information.

[0132] Finally, the platform capability parameter library, the task requirement template library, the action-instruction mapping table, the standard behavior sequence template library, and the behavior-control conversion rule library are integrated in a structured form to obtain the action-instruction mapping knowledge base.

[0133] Furthermore, the Large Language Model (LLM) is a deep learning model based on the Transformer architecture. Through pre-training on massive amounts of text data, it has obtained powerful natural language understanding and generation capabilities. In this embodiment, the LLM is guided to perform intent parsing through preset prompt word templates. The LLM identifies the operator's macro-intent type based on the fused semantic information included in the joint semantic embedding vector, and queries the task requirement template and task decomposition template corresponding to the intent in the knowledge base. Combined with the platform capability parameter library, it selects UAV platforms that can perform the task, decomposes the macro-intent into a series of atomic behaviors and arranges them in sequence according to the physical logic of task execution. Finally, it outputs a structured instruction sequence in the format required by the prompt word template, which can be directly used for subsequent instruction conversion and UAV control.

[0134] For example, the preset prompt word template can be: {Role Definition: Design a cluster task planner. This planner needs to combine a provided list of actions with a function to query the knowledge base, converting the input joint semantic embedding vector into a sequence of actions with temporal logic. The output must strictly adhere to a list representation, where each triple is a dictionary key-value pair. The structure is as follows: seq = [{"drone_id": "drone ID", "task_primitive": "action name", "params":"Optional"}...]} Action list: 1. take_off: Ascend vertically from the ground to the specified height and hover. - Parameter: altitude (height, default 5, unit: meters) -speed (rise rate, default 1.5, unit m / s) 2. fly_to (fly to target): Fly to the target location at the specified speed. - Parameter: speed (speed, default 3, unit m / s) -goal (coordinates, default GPS signal) 3. Surround: Forms an encirclement around the target with a specified radius. - Parameter: radius (enclosing radius, default 10, unit: meters) 4. Detect (Reconnaissance): Perform reconnaissance missions at the current location. - Parameter: duration (duration, default 300, in seconds) 5. hover: Maintain hovering position in the air. - Parameter: duration (duration, default 60, in seconds) 6. land: descend from the current location to the ground. - Parameter: speed (descent speed, default 1.5, unit m / s) Generation rules: 1. Analyze the task to be executed based on joint semantic embedding vectors; 2. All parameters must refer to the definitions in the ontology knowledge base, correspond to the parameters in the knowledge base, or take the default parameters in the knowledge base; 3. Tasks must be arranged in a physically feasible execution order, referring to the task requirement template in the knowledge base; 4. Parameters are expressed in units of distance or height using "meters", time using "seconds", and speed using "m / s"; Input vector: {vector} Output format: JSON list Example output: [ ["UAV001", "takeoff", {"altitude": 5, "speed": 1.5}], ["UAV001", "fly_to", {"speed": "3.0", "goal": {x,y,z}}], ["UAV001", "hover", {"duration": 300}], ["UAV001", "land", {"speed": 1.5}] ]}.

[0135] The prompt word is obtained by replacing “{vector}” in the prompt word template with the joint semantic embedding vector and replacing “{function}” in the prompt word template with the interface function that calls the knowledge base. The prompt word is then input into the large language model to obtain the instruction sequence in JSON format.

[0136] Understandably, the joint semantic embedding vector is converted into a natural language description through a preset prompt word template, and the large language model is constrained to output the instruction sequence in a structured format. In this process, the large language model decomposes the macro intent into a series of atomic behaviors according to the platform capability parameter library, the task requirement template library, the action-instruction mapping table, the standard behavior sequence template library, and the behavior-control conversion rule library defined in the knowledge base, and arranges them in sequence according to the physical logic of task execution, and finally generates an instruction list including UAV identifier, action name and execution parameters.

[0137] Step S5: Convert the instruction sequence into the underlying control instructions of the unmanned cluster and send them to the unmanned cluster.

[0138] Specifically, the instruction sequence is converted into a low-level instruction format that the unmanned platform can recognize by converting the action name and execution parameters in each instruction into the behavior-control conversion rule base in the knowledge base. At the same time, before sending, the instruction sequence is checked for consistency based on the preset security rule base to ensure that there are no logical conflicts or security risks between the instructions.

[0139] More specifically, the preset safety rule base is a series of "If-Then" logical rules constructed from flight data acquired from various normal and abnormal mission scenarios simulated by ROS / GAZEBO. These rules are used to perform conflict detection and safety verification before command execution, ensuring that the generated command sequence will not cause the UAV to enter a dangerous state. When the UAV receives the command sequence to be executed, it extracts the parameters and attributes of each command and inputs them into the safety rule base for matching and detection. If a contradiction is detected between two sequentially executed commands, for example, simultaneously requiring the UAV to "ascend rapidly" and "land," or if a command would cause the UAV to exceed its safe flight envelope, the system will eliminate the erroneous command or issue an alarm according to the rule base, and only send the command sequence that passes the detection to the UAV for execution. Through this consistency detection, it can be ensured that the control logic is always within a safe range, avoiding flight accidents caused by command conflicts or parameter errors.

[0140] Furthermore, the method also includes controlling the unmanned platforms in the unmanned cluster based on the control commands, and dynamically correcting the underlying control commands based on feedback from the unmanned cluster; wherein, the dynamic correction of the control commands based on feedback from the unmanned cluster includes: The status information of each unmanned platform in the unmanned cluster is received in real time, and the deviation value between each status information and the preset status is calculated; wherein, the deviation value includes position deviation value, speed deviation value and formation deviation value.

[0141] Specifically, the position deviation value is used to represent the distance between the current position and the target position of each unmanned platform; the speed deviation value is used to represent the vector difference between the current flight speed and the target speed of each unmanned platform; and the formation deviation value is used to represent the deviation between the actual relative position of each unmanned platform in the formation and the preset formation position.

[0142] When the deviation value exceeds a preset threshold, a control layer compensation command is initiated to correct the flight trajectory and / or speed command of each unmanned platform.

[0143] Specifically, when the deviation exceeds the preset threshold but is still within the convergent range, the unmanned platform's own flight control algorithm is activated, and a control layer compensation algorithm is used to correct the flight trajectory and / or speed commands in real time. This correction does not involve a reinterpretation of the intent, but only fine-tunes the underlying control parameters. It has a fast response speed and is suitable for immediate correction of small deviations.

[0144] When the deviation value persists and cannot converge, the deviation value is converted into a vector value, concatenated with the quality vector of each modality, and then passed through the gating network and the cross-modal attention mechanism in sequence to obtain the updated joint semantic embedding vector.

[0145] Based on the updated joint semantic embedding vector and the knowledge base, the operator's interaction intent is re-parsed and decomposed into an updated instruction sequence.

[0146] The updated instruction sequence is converted into the underlying control instructions for the unmanned swarm, controlling the movement of each unmanned platform.

[0147] Specifically, if the deviation value persists and fails to converge after multiple control cycles of adjustment, it is determined that the current deviation may originate from a problem in the upper-layer intent parsing. The deviation value is then converted into a vector value, concatenated with the quality assessment vectors of each modality, and the modal weights are recalculated based on the concatenated vectors. The joint semantic embedding vector is then updated, and the intent parsing and task decomposition in step S4 are re-executed to obtain the updated instruction sequence. Finally, the instruction sequence is converted into the underlying control instructions of the unmanned platform and reissued to each unmanned platform for execution.

[0148] In summary, the multimodal human-computer interaction intent recognition method for unmanned swarms according to embodiments of the present invention has the following beneficial effects: 1. In this embodiment of the invention, for multi-source heterogeneous interactive signals such as speech, electromyography and inertia, a differentiated preprocessing and unified time window mechanism is used to achieve temporal alignment, and a dedicated feature encoder is used to map different modal signals to a unified semantic space to achieve cross-modal interactive features alignment at the temporal and semantic levels.

[0149] 2. This invention utilizes a dynamic modal gating network to adaptively allocate fusion weights based on the quality of each modal signal. Combined with a cross-modal attention mechanism, it achieves semantic interaction and information complementarity between different modal features, improving the stability and reliability of multimodal intent representation in complex interference environments and effectively avoiding the interruption problem caused by single-modal failure in traditional methods.

[0150] 3. This invention combines fused multimodal features with a knowledge base including action command mapping. A large language model is used to parse and decompose the high-level interactive intent of the operator, mapping the generated task sequence into cluster-executable instructions. Simultaneously, consistency checks are performed on the instruction sequence based on an "If-Then" safety rule base, intercepting conflicting instructions in real time. Dynamic corrections are made based on the execution feedback from the UAV cluster, forming a closed-loop control mechanism of intent understanding, command execution, and feedback adjustment. This improves the system's security and operational stability in dynamic task environments.

[0151] 4. By constructing a knowledge base including platform configuration, task payload and cooperative rules, the embodiments of the present invention formally model the dynamic constraints and task logic of heterogeneous UAVs, realizing the automatic mapping from macroscopic abstract intentions to low-level specific instructions. Compared with existing methods that can only recognize simple actions such as "take-off" and "turn", the present invention can handle macroscopic tasks with complex semantics and temporal logic, improving the accuracy and intelligence level of task execution.

[0152] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0153] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for multimodal human-computer interaction intent recognition for unmanned swarms, characterized in that, Includes the following steps: The multimodal interaction signals of the unmanned swarm operator are acquired, preprocessed, and time-aligned to obtain time-aligned signal segments of each modality; wherein, the multimodality includes speech, electromyography, and inertia; After extracting features from each modal signal segment, cross-modal semantic alignment is performed to obtain aligned feature vectors for each modality. Based on the quality assessment results of each modal signal segment, the feature vectors of each modality are interacted through a cross-modal attention mechanism to obtain a joint semantic embedding vector; Based on the joint semantic embedding vector and a pre-built knowledge base including action instruction mapping, the operator's interaction intention is parsed and decomposed into an instruction sequence with temporal logic; The instruction sequence is converted into the underlying control instructions of the unmanned cluster and sent to the unmanned cluster.

2. The method according to claim 1, characterized in that, The step of extracting features from each modal signal segment in the multimodal signal segment and then performing cross-modal semantic alignment to obtain an aligned multimodal feature vector set includes: inputting each of the modal signal segments into a trained cross-modal alignment model to obtain the aligned multimodal feature vector set; wherein, The cross-modal alignment model includes a speech feature encoder, an electromyography feature encoder, and an inertial feature encoder arranged in parallel, as well as a linear projection layer connected to the output layer of each feature encoder. The speech feature encoder is used to extract feature vectors from speech signal segments; The electromyography feature encoder is used to extract feature vectors from electromyography signal segments; The inertial feature encoder is used to extract feature vectors from inertial signal segments; Each of the linear projection layers is used to map the feature vectors output by each feature encoder to a semantic space of the same dimension, so as to obtain aligned speech feature vectors, electromyographic feature vectors and inertial feature vectors.

3. The method according to claim 2, characterized in that, The cross-modal alignment model was trained using the following method: Acquire multimodal signal segments under multiple different control commands. Each control command corresponds to a set of synchronously acquired speech signal segments, electromyographic signal segments, and inertial signal segments, and label the corresponding intention category. Based on each of the multimodal signal segments and the corresponding intent categories, a training dataset for the cross-modal alignment model is constructed; wherein, for each group of synchronously acquired multimodal signal segments, a signal segment of any one modality is used as an anchor point, a signal segment of another modality in the same group is used as a positive sample, and a signal segment of the same modality as the anchor point in other groups is used as a negative sample, thereby obtaining training data including anchor points, positive samples, negative samples, and intent category labels; The training dataset is loaded into the cross-modal alignment model. The contrast loss is calculated for each pairwise combination of modalities. The average of the contrast losses of each modal combination is used as the total loss to train the cross-modal alignment model, thus obtaining the trained cross-modal alignment model.

4. The method according to claim 2, characterized in that, Quality assessment of each modal signal segment was performed, including: Based on each of the modal signal segments, the signal-to-noise ratio of each modal signal is obtained; Based on each modal signal segment, the feature completeness of each modal signal is obtained; wherein, the feature completeness is the proportion of non-zero frames in the modal signal segment; Based on the feature vectors output by the feature encoders of each modality, the confidence score of each modality signal segment is obtained; The signal-to-noise ratio, feature completeness, and confidence score of each modality are concatenated sequentially to obtain the quality vector of each modal signal segment.

5. The method according to claim 4, characterized in that, Based on the quality assessment results, the modal feature vectors are interacted through a cross-modal attention mechanism to obtain a joint semantic embedding vector, including: The quality vectors of each modal signal segment are input into the trained gating network to obtain the fusion weights of each modal feature vector; The corresponding modal feature vectors are weighted based on the fusion weights to obtain the weighted modal feature vectors. The weighted feature vectors of each modality are input into the trained cross-modal attention module for feature interaction to obtain a joint semantic embedding vector.

6. The method according to claim 5, characterized in that, The gating network was trained using the following method: Construct a training dataset for the gated network; wherein the training dataset includes several sets of sample data, and each set of sample data includes the feature vector of each modality and its corresponding original quality vector; For each set of sample data, the original mass vector of one or more modes is randomly masked by a modal masking mechanism to obtain the masked mass vector of each mode. The masked quality vectors of each modality are input into the gating network to obtain the prediction weights of each modality. Based on the prediction weights, the feature vectors of each modality are weighted and fused to obtain the prediction fusion features; The original mass vectors of each modality are input into the gating network to obtain the target weights of each modality; Based on the target weights, the feature vectors of each modality are weighted and fused to obtain the target fusion features; Based on the predicted fusion features and the target fusion features, the reconstruction loss function is calculated; Based on the predicted fusion features, calculate the KL divergence loss between them and the uniform distribution; The gated network is trained using the sum of reconstruction loss and KL divergence loss as the total loss. When the training converges, the trained gated network is obtained.

7. The method according to claim 5, characterized in that, The step of inputting the weighted modal feature vectors into the trained cross-modal attention module for feature interaction to obtain a joint semantic embedding vector includes: The weighted modal feature vectors are concatenated to obtain a feature vector matrix; where each row of the feature vector matrix corresponds to a feature vector of a modality. The feature vector matrix is ​​linearly mapped into a query matrix, a key matrix, and a value matrix, respectively. The query matrix, key matrix, and value matrix are input into the trained cross-modal attention module, and the attention weight matrix is ​​calculated by multiple parallel attention heads respectively. Based on the attention weight matrix of each attention head, the value matrix corresponding to each attention head is weighted and summed to obtain the updated feature matrix output by each attention head; The updated feature matrices output by each attention head are concatenated and aggregated to obtain a joint semantic embedding vector.

8. The method according to claim 7, characterized in that, The cross-modal attention module is a multi-head attention module; The cross-modal attention module was trained using the following method: Construct a training dataset for the cross-modal attention module; wherein the training dataset includes several sets of sample data, and each set of sample data includes the weighted feature vectors of each modality output by the gating network; The training dataset is input into the cross-modal attention module, and the LoRA fine-tuning method is used to train the cross-modal attention module to obtain the trained cross-modal attention module.

9. The method according to claim 1, characterized in that, The method, based on the joint semantic embedding vector and a pre-built knowledge base including action command mapping, parses the operator's interaction intent and decomposes it into a sequence of commands with temporal logic, including: The joint semantic embedding vector is combined with the action instruction mapping rules in a pre-built knowledge base including action instruction mapping and preset prompt word templates. Through a large language model, an instruction sequence with clear semantic boundaries and temporal logic is obtained. The prompt word template includes role definition, action instruction mapping list and output format constraints.

10. The method according to claim 5, characterized in that, It also includes controlling the unmanned platforms in the unmanned swarm based on the control commands, and dynamically correcting the underlying control commands based on feedback from the unmanned swarm; wherein, the dynamic correction of the control commands based on feedback from the unmanned swarm includes: The status information of each unmanned platform in the unmanned cluster is received in real time, and the deviation value between each status information and the preset state is calculated; wherein, the deviation value includes position deviation value, velocity deviation value and formation deviation value; When the deviation value exceeds the preset threshold, the control layer compensation command is activated to correct the flight trajectory and / or speed command of each unmanned platform. When the deviation value persists and cannot converge, the deviation value is converted into a vector value, concatenated with the quality vector of each modality, and then passed through the gating network and the cross-modal attention mechanism in sequence to obtain the updated joint semantic embedding vector. Based on the updated joint semantic embedding vector and the knowledge base, the operator's interaction intent is re-parsed and decomposed into an updated instruction sequence; The updated instruction sequence is converted into the underlying control instructions for the unmanned swarm, controlling the movement of each unmanned platform.

Images