Unmanned aerial vehicle control method based on multi-modal fusion and related device

Abstract

This application relates to the field of unmanned aerial vehicle (UAV) control technology, and provides a UAV control method and related equipment based on multimodal fusion. The method includes: acquiring multimodal raw data of the UAV during flight, including visual raw data, acoustic raw data, and text raw data; extracting features from the multimodal raw data to obtain visual, acoustic, and textual feature vectors; performing dimensionality reduction and discretization processing on the visual and acoustic feature vectors using k-means clustering to obtain visual discrete features and acoustic discrete features; performing feature alignment and normalization processing on the text feature vector, visual discrete features, and acoustic discrete features to obtain input features; determining weight information based on the stability score, quality score, and communication quality data of the input features; and fusing the input features according to the weight information using attention weighting and residual connections to complete the control of the UAV based on the fused features.

Description

Technical Field

[0001] This application belongs to the field of unmanned aerial vehicle (UAV) control technology, and particularly relates to UAV control methods and related equipment based on multimodal fusion. Background Technology

[0002] The rapid development of low-altitude intelligent networks has placed higher demands on the real-time perception and autonomous decision-making capabilities of core nodes—low-altitude unmanned aerial vehicles (UAVs). Existing technologies are dedicated to integrating data from various sensors on UAVs (such as visual cameras, acoustic microphones, and text modules for recognition) to build a multimodal fusion mechanism, thereby improving the operational efficiency of UAVs in scenarios such as information collection and uploading.

[0003] Currently, the technical solutions surrounding multimodal fusion and data transmission mainly exhibit the following trends: The first type of approach prioritizes ensuring fusion effectiveness but sacrifices real-time performance at the terminal. These approaches typically place complex fusion calculations in the cloud or on the ground. For example, the approach proposed in patent 202310012345.6 employs a feature splicing fusion method based on neural networks. While this achieves a certain degree of modal complementarity, its computational model necessitates the transmission of large amounts of raw data or high-dimensional features from the terminal. This undoubtedly puts immense pressure on the limited communication bandwidth and onboard energy of the UAV, introduces high communication latency, and makes it difficult to meet the UAV's real-time response requirements for unexpected events.

[0004] The second type of approach tends to ensure real-time performance at the terminal side, but simplifies the fusion process. To accommodate the limited onboard computing power of drones, some technologies opt for simple single-modal processing or decision-level fusion at the terminal (such as voting on the recognition results of each modality). While this type of method reduces latency, it fails to fully utilize the rich cross-modal correlation information between video, audio, and text, resulting in a significant decrease in the system's perception accuracy and robustness when sensor data is interfered with (such as blurred video or noisy audio).

[0005] Therefore, existing technologies have not yet achieved a good balance between the onboard computing power constraints of drones and effective multimodal fusion capabilities. Summary of the Invention

[0006] This application provides a UAV control method and related equipment based on multimodal fusion, which can solve the problem that existing technologies cannot balance the onboard computing power constraints of UAVs with effective multimodal fusion capabilities.

[0007] In a first aspect, embodiments of this application provide a UAV control method based on multimodal fusion, comprising: The system acquires multimodal raw data of the UAV during flight, including visual raw data, acoustic raw data and text raw data. The system performs position embedding processing on the visual raw data to obtain a visual feature vector, performs linear projection mapping on the acoustic raw data to obtain an acoustic feature vector, and maps the text raw data to a text feature vector according to a preset vocabulary. The visual and acoustic feature vectors are reduced in dimensionality using the kmeans clustering technique, resulting in visual dimensionality-reduced features for the visual feature vectors and acoustic dimensionality-reduced features for the acoustic feature vectors. The visual and acoustic dimensionality reduction features are discretized using index mapping techniques to obtain visual discrete features and acoustic discrete features for visual dimensionality reduction. The text feature vector, visual discrete feature, and acoustic discrete feature are aligned and normalized to obtain text target features, visual target features, and acoustic target features at the same metric scale. The text target features, visual target features, and acoustic target features at the same metric scale are used as input features. The weight information of the input features is determined by using the stability score, quality score, and communication quality data of the input features. Based on the weight information of the input features, the input features are fused through attention weighting and residual connection to obtain fused features, and the control of the UAV is completed based on the fused features.

[0008] Optionally, the step of performing position embedding processing on the original visual data to obtain a visual feature vector includes: The original visual data is divided into a set of three-dimensional segments of a preset size. Each three-dimensional segment in the set of three-dimensional segments is linearly projected to obtain an initial visual feature vector. By using learnable position embedding, the time corresponding to the initial visual feature vector, the horizontal position information corresponding to the initial visual feature vector, and the vertical position information corresponding to the initial visual feature vector are encoded respectively to obtain a visual feature vector containing spatiotemporal information.

[0009] Optionally, the step of performing dimensionality reduction processing on the visual feature vector and acoustic feature vector using k-means clustering to obtain visual dimensionality-reduced features for the visual feature vector and acoustic dimensionality-reduced features for the acoustic feature vector includes: The optimal number of cluster centers is determined by the elbow rule, and then the visual and acoustic feature vectors are reduced in dimensionality using the k-means clustering technique according to the optimal number of cluster centers, resulting in visual dimensionality reduction features for the visual feature vectors and acoustic dimensionality reduction features for the acoustic feature vectors.

[0010] Optionally, the step of discretizing the visual and acoustic dimensionality reduction features using index mapping technology to obtain visual discrete features for the visual dimensionality reduction features and acoustic discrete features for the acoustic dimensionality reduction features includes: Calculate the Euclidean distance between the visual dimensionality reduction feature and different cluster centers, and select the cluster center closest to the visual dimensionality reduction feature as the center index according to the Euclidean distance. Obtain the visual discrete feature for the visual dimensionality reduction feature through index mapping technology. Calculate the Euclidean distance between the acoustic dimensionality reduction feature and different cluster centers, and select the cluster center closest to the acoustic dimensionality reduction feature as the center index according to the Euclidean distance. Obtain the acoustic discrete feature for the acoustic dimensionality reduction feature through index mapping technology.

[0011] Optionally, the step of determining the weight information of the input feature using the stability score, quality score, and communication quality data of the input feature includes: The probability density of the input feature in different dimensions is obtained by histogram statistics, the entropy of the input feature is calculated based on the probability density, and the entropy of the input feature is used as the stability score of the input feature. Based on the mean and variance of the input features, a quality score for the input features is determined using a two-layer fully connected neural network, wherein the network parameters of the two-layer fully connected neural network are obtained through backpropagation algorithm. The weight information of the input features is determined according to a preset formula based on the stability score, quality score, and communication quality data of the input features. The mathematical representation of the preset formula is as follows:

[0012] in, The entropy of the input feature. To rate the quality, For communication quality data, As preset weights, and .

[0013] Optionally, the step of controlling the drone based on the fused features includes: The fused features are input into a preset recognition model to obtain text data. The text data includes instructions for controlling the movement of the drone and data packets for transmitting back to the cluster center corresponding to the drone. The preset recognition model is a recognition model trained by a preset training set. The preset training set includes preset fused features and preset instructions and preset data packets corresponding to the preset fused features.

[0014] On the other hand, embodiments of this application provide a UAV control device based on multimodal fusion, the device comprising: The data acquisition module is used to acquire multimodal raw data of the UAV during flight. The multimodal raw data includes visual raw data, acoustic raw data and text raw data. The module performs position embedding processing on the visual raw data to obtain a visual feature vector, performs linear projection mapping on the acoustic raw data to obtain an acoustic feature vector, and maps the text raw data to a text feature vector according to a preset vocabulary. The dimensionality reduction module is used to reduce the dimensionality of visual feature vectors and acoustic feature vectors using the kmeans clustering technique, resulting in visual dimensionality reduction features for visual feature vectors and acoustic dimensionality reduction features for acoustic feature vectors. The discrete module is used to discretize visual and acoustic dimensionality reduction features using index mapping technology, resulting in visual discrete features for visual dimensionality reduction features and acoustic discrete features for acoustic dimensionality reduction features. The alignment module is used to perform feature alignment and normalization on text feature vectors, visual discrete features and acoustic discrete features to obtain text target features, visual target features and acoustic target features at the same metric scale, and use text target features, visual target features and acoustic target features at the same metric scale as input features; The weight calculation module is used to determine the weight information of the input feature by using the stability score, quality score and communication quality data of the input feature; The fusion module is used to fuse the input features according to the weight information of the input features through attention weighting and residual connection to obtain fused features, and to complete the control of the UAV based on the fused features.

[0015] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the UAV control method based on multimodal fusion as described in any one of the first aspects above.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the UAV control method based on multimodal fusion as described in any one of the first aspects above.

[0017] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the unmanned aerial vehicle control method based on multimodal fusion as described in any of the first aspects.

[0018] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0019] The beneficial effects of this application's embodiments compared to existing technologies are as follows: By fusing video, audio, and text data and extracting key information at the UAV end, the system only needs to upload filtered event results and highly compressed feature indexes, completely avoiding large-scale backhaul of original data and achieving a breakthrough reduction in communication load. In practical applications, compared to traditional full-stream video transmission modes, this invention can reduce uplink communication data volume by more than 80%, greatly alleviating the pressure on low-altitude spectrum resources. It effectively correlates and verifies complementary information in visual, acoustic, and text commands. Even when the quality of a single modality deteriorates, the system can still make stable judgments based on other reliable modalities. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a UAV control method based on multimodal fusion provided in an embodiment of this application; Figure 2 This is a logical diagram illustrating the feature discretization and clustering process in a multimodal fusion-based UAV control method provided in this application embodiment; Figure 3 The flowchart of noise suppression and dynamic adjustment of modal confidence in a UAV control method based on multimodal fusion is provided for the embodiment. Figure 4 A diagram illustrating the triplet dynamic cross-modal attention mechanism in a multimodal fusion-based UAV control method provided for an embodiment. Figure 5This is a schematic diagram of the structure of a UAV control device based on multimodal fusion provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0022] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0023] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0024] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0025] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [the described condition or event] is detected," or "in response to detection of [the described condition or event]."

[0026] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0027] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0028] like Figure 1 As shown in the figure, this application provides a UAV control method based on multimodal fusion, including: S101. Acquire multimodal raw data of the UAV during flight. The multimodal raw data includes visual raw data, acoustic raw data and text raw data. Perform position embedding processing on the visual raw data to obtain a visual feature vector. Perform linear projection mapping on the acoustic raw data to obtain an acoustic feature vector. Map the text raw data to a text feature vector according to a preset vocabulary. S102. The visual feature vector and acoustic feature vector are reduced in dimensionality using the kmeans clustering technique to obtain visual dimensionality reduction features for the visual feature vector and acoustic dimensionality reduction features for the acoustic feature vector. S103. The visual and acoustic dimensionality reduction features are discretized using index mapping technology to obtain visual discrete features for visual dimensionality reduction features and acoustic discrete features for acoustic dimensionality reduction features. S104. Perform feature alignment and normalization on the text feature vector, visual discrete features and acoustic discrete features to obtain text target features, visual target features and acoustic target features at the same metric scale. Use the text target features, visual target features and acoustic target features at the same metric scale as input features. For example, the alignment and normalization of the three modal features are performed on the text, visual, and acoustic feature vectors to ensure that the features of the three modalities are comparable and have a basis for fusion in a unified semantic space. Finally, three sets of 128-dimensional feature vectors are obtained for each sample, which prepares for cross-modal fusion.

[0029] S105. Determine the weight information of the input feature based on the stability score, quality score, and communication quality data of the input feature; S106. Based on the weight information of the input features, the input features are fused through attention weighting and residual connection to obtain fused features, and the control of the UAV is completed based on the fused features.

[0030] In one possible implementation, the step of performing position embedding processing on the raw visual data to obtain a visual feature vector includes: The original visual data is divided into a set of three-dimensional segments of a preset size. Each three-dimensional segment in the set of three-dimensional segments is linearly projected to obtain an initial visual feature vector. By using learnable position embedding, the time corresponding to the initial visual feature vector, the horizontal position information corresponding to the initial visual feature vector, and the vertical position information corresponding to the initial visual feature vector are encoded respectively to obtain a visual feature vector containing spatiotemporal information.

[0031] For example, a drone captures a video stream at 1920×1080 resolution and 30fps. The video per second (30 frames in total) is divided into 10×10×10 three-dimensional patches, each containing approximately 19.2×10.8×3 pixels. A linear projection is applied to each patch, mapping it to a 512-dimensional feature vector. Simultaneously, a learnable position embedding is introduced to encode temporal, horizontal, and vertical positional information, forming a patch position vector containing spatiotemporal information, which serves as input to the subsequent Transformer encoder.

[0032] Acoustic modality: A microphone array acquires audio waveforms at a sampling rate of 44.1 kHz and a precision of 16 bits. These waveforms are then segmented, with each segment containing 1024 sampling points. Each audio segment is linearly projected into a 512-dimensional feature vector, and audio position embeddings are fused to preserve temporal information.

[0033] Text modality: The received text instructions and the identification information within the target range are processed by word segmentation to build a vocabulary. Each word is converted into a one-hot vector and then a 128-dimensional text embedding is generated through linear projection.

[0034] In one possible implementation, the step of reducing the dimensionality of the visual feature vector and the acoustic feature vector using k-means clustering to obtain visual dimensionality-reduced features for the visual feature vector and acoustic dimensionality-reduced features for the acoustic feature vector includes: The optimal number of cluster centers is determined by the elbow rule, and then the visual and acoustic feature vectors are reduced in dimensionality using the k-means clustering technique according to the optimal number of cluster centers, resulting in visual dimensionality reduction features for the visual feature vectors and acoustic dimensionality reduction features for the acoustic feature vectors.

[0035] For example, K-means clustering is used to cluster the feature space for visual modal features, and K-means clustering is also used for acoustic modal features. The elbow rule is used to automatically determine the optimal number of cluster centers for each modality (e.g., K=512 for visual and M=256 for acoustic) to ensure that the clusters are neither too sparse nor too dense.

[0036] In one possible implementation, the step of discretizing the visual and acoustic dimensionality reduction features using index mapping techniques to obtain visual discrete features for the visual dimensionality reduction features and acoustic discrete features for the acoustic dimensionality reduction features includes: Calculate the Euclidean distance between the visual dimensionality reduction feature and different cluster centers, and select the cluster center closest to the visual dimensionality reduction feature as the center index according to the Euclidean distance. Obtain the visual discrete feature for the visual dimensionality reduction feature through index mapping technology. Calculate the Euclidean distance between the acoustic dimensionality reduction feature and different cluster centers, and select the cluster center closest to the acoustic dimensionality reduction feature as the center index according to the Euclidean distance. Obtain the acoustic discrete feature for the acoustic dimensionality reduction feature through index mapping technology.

[0037] For example, K-means clustering is performed on visual features (initial number of centers 512), and the optimal number of cluster centers K=512 is automatically determined using the elbow rule. The same applies to acoustic modalities, with the number of centers M=256. The clustering process is as follows: Figure 2 As shown, by iteratively optimizing the center position, the feature similarity within the same cluster is maximized, and the differences between different clusters are enhanced.

[0038] Feature discretization and index mapping: The Euclidean distance between each visual / acoustic dimensionality-reduced feature and the cluster center is calculated. The nearest center index is selected (visual modality 0-511, acoustic modality 0-255) to achieve feature discretization. Visual features are reduced from 512 dimensions to 9 dimensions. The acoustic features were reduced from 512 dimensions to 8 dimensions. The discrete semantic vocabulary includes cluster center indexes and their features, which facilitates subsequent modality fusion.

[0039] In one possible implementation, the step of determining the weight information of the input feature using the stability score, quality score, and communication quality data of the input feature includes: The probability density of the input feature in different dimensions is obtained by histogram statistics, the entropy of the input feature is calculated based on the probability density, and the entropy of the input feature is used as the stability score of the input feature. Based on the mean and variance of the input features, a quality score for the input features is determined using a two-layer fully connected neural network, wherein the network parameters of the two-layer fully connected neural network are obtained through backpropagation algorithm. The weight information of the input features is determined according to a preset formula based on the stability score, quality score, and communication quality data of the input features. The mathematical representation of the preset formula is as follows:

[0040] in, The entropy of the input feature. To rate the quality, For communication quality data, As preset weights, and .

[0041] For example, such as Figure 3 As shown, communication link quality is introduced as a key factor in confidence calculation, and missing mode discrimination and gating logic are designed to enable the system to automatically degrade when the modes are incomplete, which greatly improves the robustness and practicality of the system.

[0042] Feature stability and confidence calculation: For each modality's embedded features, the entropy of its distribution is calculated. This entropy reflects the feature's stability, i.e., its confidence, using the formula H = 1. Where D is the feature dimension, the lower the entropy, the more stable the feature, and the higher the confidence level.

[0043] Feature quality prediction utilizes a two-layer fully connected neural network (FC). The mean of the modality features is input, and the quality score of the modality is output. Through end-to-end training, the network can automatically learn to judge the intrinsic quality of features.

[0044] Joint feature entropy and MLP quality score :

[0045] in, The entropy of the input feature. The mean of the modal features.

[0046] By weighting and fusing the feature stability score and the feature quality score, we can set the weight λ=0.6 to obtain the final modality confidence score Cm. The calculated confidence score reflects the reliability of the current modality in the sample. The weights of the three modalities can be adjusted according to the confidence score in the future. Based on the confidence level Cm of each mode, the original mode weights Wm (where m represents one of the three modes) are dynamically adjusted using the following formula:

[0047] in, For the adjusted weights, For a certain mode, For the original modal weights, For the confidence level of a certain mode, the denominator is... and It is the sum of the product of confidence level and weighted modes.

[0048] High-confidence modal weights are amplified, while low-confidence modal weights are suppressed, achieving automatic suppression of noisy modes and enhancement of high-quality information.

[0049] Specifically, by calculating the entropy of the feature distribution, the stability of the features can be quantified, thus providing a basis for subsequent modality weight adjustments. A two-layer fully connected neural network (FC) is used, taking the mean and variance of the modality features as input, and outputting a quality score (between 0 and 1) for that modality. The network parameters are trained using the backpropagation algorithm, enabling the network to automatically learn and judge the intrinsic quality of features. This neural network can evaluate the quality of features based on their statistical characteristics, providing supplementary information for calculating modality confidence.

[0050] The feature stability score and feature quality score are weighted and fused (weight λ = 0.6 for feature stability, 0.4 for feature quality) to obtain the final modal confidence Cm.

[0051] Where λ is the weight value. For feature stability score, For the corresponding modal feature quality score The modal weights are dynamically adjusted based on the confidence level Cm. The modal weight update formula is as follows:

[0052] in These are the original modal weights. It is the sum of confidence scores for all modalities. These are the adjusted modal weights. Modal weights with high confidence are amplified, while those with low confidence are suppressed.

[0053] The original modal features are weighted, and the optimized modal features are output:

[0054] in These are the adjusted modal weights. These are the original modal features. These are the optimized modal features.

[0055] According to the characteristic distribution entropy And predicting quality scores using small neural networks In addition, a new communication quality factor was added. (e.g., based on packet reception rate or signal-to-noise ratio) to ultimately determine modal confidence. :

[0056] in, The entropy of the input feature. To rate the quality, For communication quality data, As preset weights, and ; For missing modes, a gating mechanism is used, and the system monitors the data stream of each mode in real time. When a mode (such as acoustics) is completely missing due to communication interruption or equipment failure, the confidence level of that mode is determined. This will be forcibly set to 0. The fusion mechanism will automatically adjust to a bimodal or monomodal operating mode based on the remaining available modes, ensuring that the system can continue to operate in degraded mode.

[0057] In one possible implementation, such as Figure 4 As shown, the text, visual, and acoustic feature vectors, optimized based on the weight information of the input features, are transformed using a linear transformation (fully connected layer) to obtain three sets of vectors, Q (Query). The text features are then processed by the weight matrix W. Q K (Key) is obtained by transformation, derived from visual features through weight matrix W. k V (Value) is obtained by transformation, derived from acoustic features through a weight matrix W. v The transformation is obtained, where the text features are... Visual features acoustic characteristics :

[0058] in , is the query sequence. , where is the key in the key-value pair. , which is the value in the key-value pair. The optimized text feature vector. The optimized visual feature vector, The optimized acoustic feature vectors are derived from the linear transformation described above, which adjusts the dimensions of each mode. According to the modal compression theorem, Visual feature noise reduction requirements Acoustics has only one diffusion rule. .

[0059] Calculate the dot product of Q and K to obtain the correlation scores between each modality. Scale and normalize the correlation scores, then input them into the Softmax layer to obtain the attention weight distribution matrix. It represents the strength of the dependency between text and visual elements.

[0060] Attention weighting and feature fusion, using the obtained attention weight distribution matrix The V (acoustic features) are weighted and summed to obtain the fused feature vector. This fused feature vector dynamically focuses on and integrates the most relevant information among text, vision, and acoustic elements, enhancing multimodal expressive capabilities.

[0061] in, This represents the result of attention weighting and feature fusion. Let be the attention weight, representing the association strength between the i-th position of text feature T and the j-th position of visual feature V. For acoustic characteristics, a residual connectivity term is added. , This is the Hadamard product (i.e., element-wise multiplication). This is a learnable scaling factor, initially set to 0.3. The residual enhancement term preserves the original interaction pattern between text and vision, preventing the attention mechanism from losing important details.

[0062] In one possible implementation, the step of controlling the drone based on the fused features includes: The fused features are input into a preset recognition model to obtain text data. The text data includes instructions for controlling the movement of the drone and data packets for transmitting back to the cluster center corresponding to the drone. The preset recognition model is a recognition model trained by a preset training set. The preset training set includes preset fused features and preset instructions and preset data packets corresponding to the preset fused features.

[0063] For example, based on edge-side fusion features and normalized text output (including coordinate information, scene category information, and UAV trajectory information), this system constructs a complete output data system on the UAV embedded platform. The terminal execution layer encapsulates trajectory control commands through the MAVLink protocol, including 3D obstacle avoidance vectors and velocity planning parameters based on B-spline interpolation, and outputs real-time situation assessment data in Occupancy Grid format, including a list of obstacle point clouds with semantic labels and area safety level markers based on Voronoi diagrams. The cluster collaboration layer includes standardized event reports (event type enumeration, WGS84 coordinates, PTP precise timestamps, multimodal fusion confidence) and refined feature summaries (64-dimensional visual feature vectors, 32-bit voiceprint hash values, and text semantic embedding vectors), transmitted through a data transmission link conforming to the STANAG 4586 standard. All outputs adopt a plug-and-play interface design, supporting real-time WebSocket push, ensuring that the fusion results can be directly integrated into existing UAV control systems and put into practical applications.

[0064] On the other hand, such as Figure 5 As shown in the figure, this application provides a UAV control device based on multimodal fusion, the device comprising: The data acquisition module 201 is used to acquire multimodal raw data of the UAV during flight. The multimodal raw data includes visual raw data, acoustic raw data and text raw data. The visual raw data is processed by position embedding to obtain a visual feature vector. The acoustic raw data is linearly projected to obtain an acoustic feature vector. The text raw data is mapped to a text feature vector according to a preset vocabulary. The dimension reduction module 202 is used to perform dimension reduction processing on visual feature vectors and acoustic feature vectors using kmeans clustering technology to obtain visual dimension reduction features for visual feature vectors and acoustic dimension reduction features for acoustic feature vectors. Discrete module 203 is used to perform feature discretization processing on visual dimensionality reduction features and acoustic dimensionality reduction features through index mapping technology to obtain visual discrete features for visual dimensionality reduction features and acoustic discrete features for acoustic dimensionality reduction features. Alignment module 204 is used to perform feature alignment and normalization on text feature vectors, visual discrete features and acoustic discrete features to obtain text target features, visual target features and acoustic target features at the same metric scale, and to use text target features, visual target features and acoustic target features at the same metric scale as input features; The weight calculation module 205 is used to determine the weight information of the input feature by means of the stability score, quality score and communication quality data of the input feature; The fusion module 206 is used to fuse the input features according to the weight information of the input features through attention weighting and residual connection to obtain fused features, and to complete the control of the UAV based on the fused features.

[0065] In one possible implementation, such as Figure 6 As shown in the figure, this application provides a terminal device 300, including: a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor 320. When the processor 320 executes the computer program 311, it performs the following: acquiring multimodal raw data of a drone during flight, the multimodal raw data including visual raw data, acoustic raw data, and text raw data; performing position embedding processing on the visual raw data to obtain a visual feature vector; performing linear projection mapping on the acoustic raw data to obtain an acoustic feature vector; mapping the text raw data to a text feature vector according to a preset vocabulary; and performing dimensionality reduction processing on the visual feature vector and acoustic feature vector using k-means clustering technology to obtain visual dimensionality reduction features for the visual feature vector and acoustic feature vector. The acoustic dimensionality reduction features are obtained; the visual and acoustic dimensionality reduction features are discretized using index mapping technology to obtain visual discrete features for visual dimensionality reduction features and acoustic discrete features for acoustic dimensionality reduction features; the text feature vector, visual discrete features, and acoustic discrete features are aligned and normalized to obtain text target features, visual target features, and acoustic target features at the same metric scale, which are used as input features; the weight information of the input features is determined by the stability score, quality score, and communication quality data of the input features; based on the weight information of the input features, the input features are fused using attention weighting and residual connection to obtain fused features, and the control of the UAV is completed based on the fused features.

[0066] In one possible implementation, such as Figure 7As shown, this application embodiment provides a computer-readable storage medium 400, on which a computer program 411 is stored. When executed by a processor, the computer program 411 performs the following: acquiring multimodal raw data of a UAV during flight, the multimodal raw data including visual raw data, acoustic raw data, and text raw data; performing position embedding processing on the visual raw data to obtain a visual feature vector; performing linear projection mapping on the acoustic raw data to obtain an acoustic feature vector; mapping the text raw data to a text feature vector according to a preset vocabulary; performing dimensionality reduction processing on the visual feature vector and acoustic feature vector using k-means clustering technology to obtain visual dimensionality reduction features for the visual feature vector and acoustic dimensionality reduction features for the acoustic feature vector; and performing index mapping technology to... Visual and acoustic dimensionality reduction features are discretized to obtain visual discrete features and acoustic discrete features for visual and acoustic dimensionality reduction. The text feature vector, visual discrete features, and acoustic discrete features are then aligned and normalized to obtain text target features, visual target features, and acoustic target features at the same metric scale. These text target features, visual target features, and acoustic target features at the same metric scale are used as input features. The weight information of the input features is determined using stability scores, quality scores, and communication quality data. Based on the weight information of the input features, the input features are fused using attention weighting and residual connections to obtain fused features. The control of the UAV is then performed based on these fused features.

[0067] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0068] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0069] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0070] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0071] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0072] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0073] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0074] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for controlling a UAV based on multi-modal fusion, characterized in that, include: The system acquires multimodal raw data of the UAV during flight, including visual raw data, acoustic raw data and text raw data. The system performs position embedding processing on the visual raw data to obtain a visual feature vector, performs linear projection mapping on the acoustic raw data to obtain an acoustic feature vector, and maps the text raw data to a text feature vector according to a preset vocabulary. The visual and acoustic feature vectors are reduced in dimensionality using the kmeans clustering technique, resulting in visual dimensionality-reduced features for the visual feature vectors and acoustic dimensionality-reduced features for the acoustic feature vectors. The visual and acoustic dimensionality reduction features are discretized using index mapping techniques to obtain visual discrete features and acoustic discrete features for visual dimensionality reduction features. The text feature vector, visual discrete feature, and acoustic discrete feature are aligned and normalized to obtain text target features, visual target features, and acoustic target features at the same metric scale. The text target features, visual target features, and acoustic target features at the same metric scale are used as input features. The weight information of the input features is determined by using the stability score, quality score, and communication quality data of the input features. Based on the weight information of the input features, the input features are fused through attention weighting and residual connection to obtain fused features, and the control of the UAV is completed based on the fused features.

2. The UAV control method based on multimodal fusion as described in claim 1, characterized in that, The step of performing position embedding processing on the original visual data to obtain visual feature vectors includes: The original visual data is divided into a set of three-dimensional segments of a preset size. Each three-dimensional segment in the set of three-dimensional segments is linearly projected to obtain an initial visual feature vector. By using learnable position embedding, the time corresponding to the initial visual feature vector, the horizontal position information corresponding to the initial visual feature vector, and the vertical position information corresponding to the initial visual feature vector are encoded respectively to obtain a visual feature vector containing spatiotemporal information. 3.The multi-modal fusion based UAV control method of claim 1, wherein, The step of performing dimensionality reduction processing on visual feature vectors and acoustic feature vectors using k-means clustering to obtain visual dimensionality-reduced features for visual feature vectors and acoustic dimensionality-reduced features for acoustic feature vectors includes: The optimal number of cluster centers is determined by the elbow rule, and then the visual and acoustic feature vectors are reduced in dimensionality using the k-means clustering technique according to the optimal number of cluster centers, resulting in visual dimensionality reduction features for the visual feature vectors and acoustic dimensionality reduction features for the acoustic feature vectors. 4.The multi-modal fusion based UAV control method of claim 3, wherein, The step of discretizing visual and acoustic dimensionality reduction features using index mapping technology to obtain visual discrete features for visual dimensionality reduction features and acoustic discrete features for acoustic dimensionality reduction features includes: Calculate the Euclidean distance between the visual dimensionality reduction feature and different cluster centers, and select the cluster center closest to the visual dimensionality reduction feature as the center index according to the Euclidean distance. Obtain the visual discrete feature for the visual dimensionality reduction feature through index mapping technology. Calculate the Euclidean distance between the acoustic dimensionality reduction feature and different cluster centers, and select the cluster center closest to the acoustic dimensionality reduction feature as the center index according to the Euclidean distance. Obtain the acoustic discrete feature for the acoustic dimensionality reduction feature through index mapping technology. 5.The multi-modal fusion based UAV control method of claim 1, wherein, The step of determining the weight information of the input feature using the stability score, quality score, and communication quality data of the input feature includes: The probability density of the input feature in different dimensions is obtained by histogram statistics, the entropy of the input feature is calculated based on the probability density, and the entropy of the input feature is used as the stability score of the input feature. Based on the mean and variance of the input features, a quality score for the input features is determined using a two-layer fully connected neural network, wherein the network parameters of the two-layer fully connected neural network are obtained through backpropagation algorithm. determining the weight information of the input feature according to a preset formula through the stability score, the quality score and the communication quality data of the input feature , and a mathematical representation of the preset formula is wherein, is an entropy of the input features, is a quality score, is a communication quality data, is a preset weight, and . 6.The multi-modal fusion based UAV control method of claim 1, wherein, The steps for controlling the drone based on the fused features include: The fused features are input into a preset recognition model to obtain text data. The text data includes instructions for controlling the movement of the drone and data packets for transmitting back to the cluster center corresponding to the drone. The preset recognition model is a recognition model trained by a preset training set. The preset training set includes preset fused features and preset instructions and preset data packets corresponding to the preset fused features.

7. A multi-modal fusion based unmanned aerial vehicle control device, characterized in that, The device includes: The data acquisition module is used to acquire multimodal raw data of the UAV during flight. The multimodal raw data includes visual raw data, acoustic raw data and text raw data. The module performs position embedding processing on the visual raw data to obtain a visual feature vector, performs linear projection mapping on the acoustic raw data to obtain an acoustic feature vector, and maps the text raw data to a text feature vector according to a preset vocabulary. The dimensionality reduction module is used to reduce the dimensionality of visual feature vectors and acoustic feature vectors using the kmeans clustering technique, resulting in visual dimensionality reduction features for visual feature vectors and acoustic dimensionality reduction features for acoustic feature vectors. The discrete module is used to discretize visual and acoustic dimensionality reduction features using index mapping technology, resulting in visual discrete features for visual dimensionality reduction features and acoustic discrete features for acoustic dimensionality reduction features. The alignment module is used to perform feature alignment and normalization on text feature vectors, visual discrete features and acoustic discrete features to obtain text target features, visual target features and acoustic target features at the same metric scale, and use text target features, visual target features and acoustic target features at the same metric scale as input features; The weight calculation module is used to determine the weight information of the input feature by using the stability score, quality score and communication quality data of the input feature; The fusion module is used to fuse the input features according to the weight information of the input features through attention weighting and residual connection to obtain fused features, and to complete the control of the UAV based on the fused features.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The computer program is executed by the processor to implement the multi-modal fusion-based unmanned aerial vehicle control method in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the multi-modal fusion-based unmanned aerial vehicle control method in any one of claims 1 to 6.

Images