Gesture, Handshape, and Voice Collaborative Multimodal Interaction Perception Method for Humans and Drone Swarms

Abstract

The present invention discloses a method for collaborative multimodal interaction perception between humans and drone swarms using gestures, hand shapes, and voice. The method comprises the following steps: Step 1: constructing a speech recognition model for interaction between humans and drone swarms; Step 2: constructing a gesture recognition model for interaction between humans and drone swarms; Step 3: constructing an interaction framework for autonomous recognition of cluster formation collaborative control based on dual models of voice and gesture, inputting information on three interaction channels (hand shape, posture, and voice) to obtain gesture and voice command actions, and adopting a channel fusion implementation mechanism based on dual-channel switching to achieve multimodal interaction in the interaction framework. The present invention realizes multimodal interaction between humans and drone swarms by constructing an interaction framework for autonomous recognition of cluster formation collaborative control based on dual models of voice and gesture and channel fusion based on dual-channel switching. This method can accurately and efficiently achieve intelligent collaborative multimodal perception of humans and drone swarms through language and gestures.

Description

Technical Field

[0001] The present invention belongs to the field of drone control and relates to a drone swarm intelligent collaborative multimodal data perception method. Background Art

[0002] Compared to individual drones, drone swarms can collaborate to accomplish more complex tasks, and therefore have broad application prospects in both military and civilian fields. However, with the development of counter-drone swarm technology, most existing swarms struggle to meet the requirements of intelligent warfare in highly dynamic environments. Furthermore, human thinking is superior in its ability to analyze and process uncertainties.

[0003] Currently, a series of plans for human-machine interaction and swarm intelligence have been proposed both domestically and internationally. The US Center for Strategic and Budgetary Assessments identifies AI-assisted combat decision-making and command and control, as well as human-machine intelligent collaboration and autonomous operations, as core technologies supporting new winning mechanisms and operational concepts. my country's "New Generation Artificial Intelligence Development Plan" lists eight fundamental theories, many of which are directly related to swarm intelligence. Therefore, how to more efficiently enable humans to command drone swarms for coordinated operations is a current research hotspot, with automated human-machine interaction holding strategic importance for achieving rapid swarm response.

[0004] At present, more and more studies are choosing multimodal human-computer interaction methods to improve interaction efficiency: using information from channels such as vision, text, and action at the same time to achieve human-computer interaction combining multiple channels. Summary of the Invention

[0005] To address the multimodal human-machine interaction issues in drone swarms, this paper provides a method for collaborative multimodal interaction perception between humans and drone swarms using gestures, hand shapes, and voice. This method accurately and efficiently enables intelligent collaborative multimodal perception between humans and drone swarms through language and gestures, effectively improving the multimodal human-machine interaction capabilities of drone swarms.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] A method for collaborative multimodal interaction perception of gestures, hand shapes, and voice between humans and drone swarms includes the following steps:

[0008] Step 1: Build a speech recognition model for human-drone swarm interaction

[0009] Step 1.1: Collect voice commands from multiple males and females of different ages and construct a voice command dataset in a unified format;

[0010] Step 1.2: De-noise the audio time domain signal segments in the voice command dataset constructed in step 1.1 and extract the Mel-frequency cepstral coefficient features of the audio time domain signal;

[0011] Step 1.3: Input the Mel-frequency cepstral coefficient features extracted in step 1.2 into the Keras-based neural network model;

[0012] Step 2: Build a gesture recognition model for human-drone swarm interaction

[0013] Step 2.1: Creation of gesture command database

[0014] Collect data from several experimenters (e.g., ≥5, the more samples, the more accurate the data) under different lighting, background, and distance conditions;

[0015] Step 2.2: Data Preprocessing

[0016] The collected sample images are preprocessed. The specific steps are as follows: the Otsu threshold segmentation method based on the global binarization algorithm is used to adaptively filter out irrelevant features according to the image grayscale features, and then the Canny edge detection algorithm is used to extract the edge information in the image;

[0017] Step 2.3: Creation of Convolutional Neural Network Model for Gesture Recognition

[0018] The gesture recognition convolutional neural network model mainly includes three convolutional layers C1, C2, C3, three pooling layers S1, S2, S3, one batch normalization layer BN and two fully connected layers FC1 and FC2. The first six layers are alternately stacked by C1, C2, C3 and S1, S2, S3, followed by one BN layer, two FC1 and FC2 layers, and finally the output layer Softmax.

[0019] Step 3: Construct an interactive framework for autonomous recognition and collaborative control of cluster formations based on voice and gesture dual models. The input mainly consists of information from three interactive channels: hand shape, posture, and voice, which are used to obtain gesture and voice command actions. A channel fusion implementation mechanism based on dual-channel switching is adopted to realize multimodal interaction in the interactive framework.

[0020] Compared with the prior art, the present invention has the following advantages:

[0021] The present invention realizes multimodal interaction between humans and drone swarms by constructing an interactive framework based on autonomous recognition of cluster formation collaborative control based on voice and gesture dual models and channel fusion based on dual-channel switching. It can accurately and efficiently realize intelligent collaborative multimodal perception of humans and drone swarms through language and gestures. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a diagram of the speech recognition network framework;

[0023] Figure 2 The overall framework diagram of human-computer interaction. DETAILED DESCRIPTION

[0024] The technical solution of the present invention is further described below with reference to the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention that does not depart from the spirit and scope of the technical solution of the present invention should be included in the scope of protection of the present invention.

[0025] The present invention provides a method for intelligent collaborative multimodal data perception of drone swarms. The method first constructs a speech recognition model for human-drone swarm interaction, then designs a speech recognition solution for human-drone swarm interaction. Finally, an overall interaction framework for autonomous recognition of cluster formation collaborative control based on speech and gesture dual models is established. A channel fusion mechanism based on channel switching is also proposed to achieve multimodal interaction between humans and drone swarms. The method specifically includes the following steps:

[0026] Step 1: Build a speech recognition model for human-drone swarm interaction.

[0027] It mainly consists of voice command dataset construction, audio Mel-frequency cepstral coefficient feature extraction, and speech recognition neural network model implementation. The specific steps are as follows:

[0028] Step 1.1: Speech Command Dataset Construction

[0029] Collect voice commands from multiple males and females of different ages and construct a voice command dataset in a unified format. The specific steps are as follows:

[0030] The collected drone voice control commands include 7 types of commands, including triangle formation, line formation, takeoff, landing, constant speed forward, left turn, and right turn, totaling 500 audio segments. These audios were verified by 50 males and females aged 15 to 50. The data set was finally uniformly converted into a 16-bit mono wav file format with a sampling rate of 16kHz.

[0031] Step 1.2: Audio Mel Cepstral Coefficient Feature Extraction

[0032] De-noise the audio time domain signal segments in the voice command dataset constructed in step 1.1, and extract the Mel-frequency cepstral coefficient features of the audio time domain signal.

[0033] Step 1.3: Implementing the Speech Recognition Neural Network Model

[0034] The Mel-frequency cepstral coefficient features extracted in step 1.2 are input into the Keras-based neural network model, where:

[0035] The network input layer of the Keras-based neural network model is (-1, 128, 128, 1), where -1 represents batch_size, (128, 128) represents the number of Mel-cepstral coefficient features, and 1 represents the number of channels.

[0036] The network output layer of the Keras-based neural network model contains 7 neurons (corresponding to 7 types of sound commands) to output the probability of each type of audio recognition;

[0037] The loss function of the Keras-based neural network model is as follows:

[0038]

[0039] Among them, y ic Indicates the true category of the sample. If the category of sample i is equal to c, then y ic Take 1, otherwise take 0; p ic It represents the probability that the predicted sample i belongs to category c, N represents the total number of samples, and c represents the speech category (seven types of sounds: triangle formation, line formation, takeoff, landing, moving at a constant speed, left turn, and right turn).

[0040] Step 2: Build a gesture recognition model for human-drone swarm interaction.

[0041] The construction of the gesture recognition model mainly consists of the creation of a gesture command database, data preprocessing, and the creation of a gesture recognition convolutional neural network model. The specific steps are as follows:

[0042] Step 2.1: Creation of gesture command database

[0043] Data from five subjects were collected under different lighting, background, and distance conditions to maximize the generalization of the model training and experimental evaluation results. The sample image size was 76×179.

[0044] Step 2.2: Data Preprocessing

[0045] Preprocess the collected sample images. The specific steps are as follows:

[0046] The Otsu threshold segmentation method based on the global binarization algorithm is used to adaptively filter out irrelevant features according to the grayscale features of the image, and then the Canny edge detection algorithm is used to extract the edge information in the image.

[0047] The specific steps of Otsu threshold segmentation method are as follows:

[0048] The maximum inter-class variance is used as a criterion for measuring the difference between the background and the foreground, so as to adaptively select the optimal threshold T that separates the foreground and background of the original image. The principle of selecting T is as follows:

[0049] The total average grayscale U of the image is:

[0050] U=W0×U0+W1×U1 (1)

[0051] Among them, W0 and W1 are the ratio of foreground and background points to the image, and U0 and U1 are the average grayscale of foreground and background.

[0052] The variance S of the image foreground and background is:

[0053] S=W0×(U0-U) 2 +W1×(U1-U) 2 (2);

[0054] Substituting formula (1) into formula (2) yields:

[0055] S=W0×W1×(U0-U1) 2 (3);

[0056] When the variance S is the largest, the difference between the foreground and the background is the largest, and the grayscale at this time is the optimal threshold.

[0057] The Canny edge detection algorithm mainly includes image denoising, image gradient calculation, non-maximum suppression, and hysteresis threshold. The specific steps are as follows:

[0058] (1) Use the Sobel operator to calculate the first-order derivative G in the horizontal and vertical directions x and G y , and calculate the gradient G and direction θ of the image. The calculation formula is as follows:

[0059]

[0060]

[0061] (2) Scan the entire image, check the pixels with the same gradient direction in the local range, and only retain the pixels with the largest gradient.

[0062] (3) By setting the double thresholds minVal and maxVal, edge detection and connection are performed on the edge points that have been suppressed by non-maximum values. If the gradient value exceeds maxVal, it is a strong edge; if it is less than maxVal and greater than minVal, it is a weak edge; if it is less than minVal, it is not an edge.

[0063] Step 2.3: Creation of Convolutional Neural Network Model for Gesture Recognition

[0064] The gesture recognition convolutional neural network model mainly includes three convolutional layers (C1, C2, C3), three pooling layers (S1, S2, S3), one batch normalization layer (BN), and two fully connected layers (FC1, FC2). The first six layers are composed of alternating layers of C1, C2, C3 and S1, S2, S3, followed by a layer of BN, two layers of FC1 and FC2, and finally an output layer (Softmax).

[0065] C1 sets the number of convolution kernels to 72, with a size of 5×5, a stride of 1 in all directions, and a padding type of "SAME". ReLU is used as the nonlinear layer activation function. S1 sets the size to 2×2, a sliding stride of 2, and uses max pooling. C2 and C3 both set the number of convolution kernels to 144, with sizes of 5×5 and 3×3, respectively. All other settings are the same as C1. S2 and S3 have the same settings as S1. FC1 and FC2 are set to 1152 and 576 neurons, respectively, and both use ReLU as the activation function. The input is set to 76×179, and the softmax layer is set to 18 neurons.

[0066] Step 3: Construct an interactive framework for autonomous recognition and collaborative control of cluster formations based on the dual-model of voice and gesture. The input mainly includes information from three interactive channels: hand shape, posture, and voice, which are used to obtain gesture and voice command actions.

[0067] The interaction framework integrates a perception system that combines voice and gesture channels. It belongs to multi-modal interactive (MMI) and can be viewed as a multi-tuple:

[0068] MMI=<E,U,T,B,M> (6);

[0069] Where E is the interaction efficiency, U = {u1, u2, ..., u l} is a set of behavioral preferences of the current user in MMI, T = {t1, t2, ..., t m} is the set of m current tasks implemented in MMI, B={b1,b2,...,b k} is the set of k current environments that appear in MMI, M={m1,m2,..,m n} is a set of n control channels supported by MMI.

[0070] According to formula (6), it is known that the channel can be changed to adapt to the change of the environment, so as to complete different tasks. Therefore, the present invention adopts a channel fusion implementation mechanism based on dual-channel switching, that is, the MMI state changes from MMI= <E,U,t j , b i , m i>Switch to MMI <E,U,t j , b j , m j >, thereby realizing the multimodal interaction process in the interaction framework.

[0071] like Figure 2 As shown in the figure, after a person selects an interaction channel based on environmental changes, the acquisition device first collects the current signal characteristics, which are then recognized using gesture and voice perception technologies. The recognized instructions are then matched with the instruction library and mapped into corresponding instruction codes in the human-computer interaction interface. Finally, these instructions are sent to the control module of the drone swarm system for processing. The system then collaboratively controls the drone swarm system to perform basic formations, intelligent obstacle avoidance, heterogeneous collaboration, and time-varying formations. The results are then fed back to the user through the interactive interface.

Claims

1. A collaborative multimodal interaction perception method between human and drone swarm gestures, hand shapes, and voices, characterized by The method comprises the following steps: Step 1: Build a speech recognition model for human-drone swarm interaction Step 1.1: Collect voice commands from multiple males and females of different ages and construct a voice command dataset in a unified format; Step 1.2: De-noise the audio time domain signal segments in the voice command dataset constructed in step 1.1 and extract the Mel-frequency cepstral coefficient features of the audio time domain signal; Step 1.3: Input the Mel-frequency cepstral coefficient features extracted in step 1.2 into the Keras-based neural network model; Step 2: Build a gesture recognition model for human-drone swarm interaction Step 2.1: Creation of gesture command database Collect data from several subjects under different lighting, background, and distance conditions; Step 2.2: Data Preprocessing The collected sample images are preprocessed. The specific steps are as follows: the Otsu threshold segmentation method based on the global binarization algorithm is used to adaptively filter out irrelevant features according to the image grayscale features, and then the Canny edge detection algorithm is used to extract the edge information in the image; Step 2.3: Creation of Convolutional Neural Network Model for Gesture Recognition The gesture recognition convolutional neural network model mainly includes three convolutional layers C1, C2, C3, three pooling layers S1, S2, S3, one batch normalization layer BN and two fully connected layers FC1 and FC2. The first six layers are alternately stacked by C1, C2, C3 and S1, S2, S3, followed by one BN layer, two FC1 and FC2 layers, and finally the output layer Softmax. Step 3: Build an interactive framework for autonomous recognition and collaborative control of cluster formations based on voice and gesture dual models. Input information from three interactive channels: hand shape, posture, and voice. Use it to obtain gesture and voice command actions. Use a channel fusion implementation mechanism based on dual-channel switching to achieve multimodal interaction in the interactive framework.

2. The method for collaborative multimodal interaction perception of gestures, hand shapes, and voices between humans and drones according to claim 1 is characterized in that The control instructions include seven instructions: triangle formation, line formation, take-off, landing, moving forward at a constant speed, turning left, and turning right.

3. The method for collaborative multimodal interaction perception of gestures, hand shapes, and voices between humans and drones according to claim 1 is characterized in that The specific steps of step 1.1 are as follows: The collected drone voice control commands include 7 types of commands, including triangle formation, line formation, takeoff, landing, constant speed forward, left turn, and right turn, totaling 500 audio segments. These audios were verified by 50 males and females aged 15 to 50. The data set was finally uniformly converted into a 16-bit mono wav file format with a sampling rate of 16kHz.

4. The method for collaborative multimodal interaction perception of gestures, hand shapes, and voices between humans and drones according to claim 1 is characterized in that The network input layer of the Keras-based neural network model is (-1, 128, 128, 1), where -1 represents the batch size, (128, 128) represents the feature number of the Mel-cepstral coefficient feature, and 1 represents the number of channels; the network output layer of the Keras-based neural network model contains 7 neurons corresponding to 7 types of sound instructions for outputting the probability of each type of audio recognition.

5. The method for collaborative multimodal interaction perception of gestures, hand shapes, and voices between humans and drones according to claim 1 or 4 is characterized in that The loss function of the Keras-based neural network model is as follows: Among them, y ic Indicates the true category of the sample. If the category of sample i is equal to c, then y ic Take 1, otherwise take 0; p ic It represents the probability that the predicted sample i belongs to category c, N represents the total number of samples, and c represents the speech category.

6. The method for collaborative multimodal interaction perception of gestures, hand shapes, and voices between humans and drones according to claim 1 is characterized in that The specific steps of the Otsu threshold segmentation method are as follows: The maximum inter-class variance is used as a criterion for measuring the difference between background and foreground, so as to adaptively select the optimal threshold T that separates the foreground and background of the original image.

7. The method for collaborative multimodal interaction perception of gestures, hand shapes, and voices between humans and drones according to claim 6 is characterized in that The method for selecting T is as follows: The total average grayscale U of the image is: U=W0×U0+W1×U1 (1) Among them, W0 and W1 are the ratio of the number of foreground and background points to the image, and U0 and U1 are the average grayscale of the foreground and background; The variance S of the image foreground and background is: S=W0×(U0-U) 2 +W1×(U1-U) 2 (2); Substituting formula (1) into formula (2) yields: S=W0×W1×(U0-U1) 2 (3); When the variance S is the largest, the difference between the foreground and the background is the largest, and the grayscale at this time is the optimal threshold.

8. The method for collaborative multimodal interaction perception of gestures, hand shapes, and voices between humans and drones according to claim 1 is characterized in that The Canny edge detection algorithm mainly includes image denoising, image gradient calculation, non-maximum suppression, and hysteresis threshold. The specific steps are as follows: (1) Use the Sobel operator to calculate the first-order derivative G in the horizontal and vertical directions x and G y , and calculate the gradient G and direction θ of the image. The calculation formula is as follows: (2) Scan the entire image, check the pixels with the same gradient direction in the local range, and only retain the pixels with the largest gradient; (3) By setting the double thresholds minVal and maxVal, edge detection and connection are performed on the edge points that have been suppressed by non-maximum values. If the gradient value exceeds maxVal, it is a strong edge; if it is less than maxVal and greater than minVal, it is a weak edge; if it is less than minVal, it is not an edge.

9. The method for collaborative multimodal interaction perception of gestures, hand shapes, and voices between humans and drones according to claim 1 is characterized in that The interaction framework integrates the perception system of voice and gesture channels, which belongs to multimodal interaction and is regarded as a multi-tuple form: MMI=<E,U,T,B,M> (6); Where E is the interaction efficiency, U = {u1, u2, ..., u l } is a set of behavioral preferences of the current user in MMI, T = {t1, t2, ..., t m } is the set of m current tasks implemented in MMI, B={b1,b2,...,b k } is the set of k current environments that appear in MMI, M={m1,m2,...,m n } is a set of n control channels supported by MMI.

Images