Unmanned equipment control method based on gesture recognition and unmanned equipment

Abstract

The invention relates to the field of unmanned equipment control, and particularly discloses an unmanned equipment control method based on gesture recognition and unmanned equipment. The control method comprises the following steps: S1, gesture data acquisition; S2, data preprocessing; S3, feature extraction; S4, feature fusion; the unmanned equipment is controlled by adopting the control method. According to the method, double-flow fusion of IMU data and image data is used, more comprehensive gesture feature information can be provided, and multi-dimensional information is beneficial to improving the accuracy and robustness of gesture recognition.

Description

Technical Field

[0001] The present invention relates to the field of unmanned equipment control, and in particular to an unmanned equipment control method based on gesture recognition and an unmanned equipment. Background Art

[0002] Unmanned equipment is usually used to describe equipment that can perform specific tasks without direct human operation, generally including unmanned aerial vehicles (UAVs), unmanned vehicles (UGVs), unmanned ships (USVs), unmanned underwater vehicles (UUVs) and other types of automated or remote-controlled equipment. Gestures are a natural and intuitive way of interaction. Users do not need to learn complex operating instructions and can quickly get started. Gesture recognition technology is widely used in the remote control of unmanned equipment. Through gesture control, users can flexibly command unmanned equipment in different environments and conditions, without being restricted by physical equipment, which reduces the threshold for use.

[0003] In the prior art, the patent with the announcement number CN111123959B discloses a drone control method based on gesture recognition and a drone using the method, which includes the following steps: controlling the camera module of the drone to take photos and obtain photos; matching the gestures in the photos with the gesture images in the picture set to obtain gesture recognition information; the flight control system receives the gesture recognition information and controls the drone accordingly based on the gesture recognition information. However, it only recognizes gestures through the visual data of the gestures, and the single data source is easily interfered by the factor of switching, and the recognition accuracy is low, especially for more complex gestures.

[0004] The above technical problems need to be solved urgently. Summary of the invention

[0005] In view of this, an object of the present invention is to provide an unmanned equipment control method based on gesture recognition and an unmanned equipment using the control method, so as to improve the accuracy of gesture recognition.

[0006] In order to solve the above technical problems, in a first aspect, the present invention provides an unmanned equipment control method based on gesture recognition, comprising the following steps:

[0007] S1. Gesture data collection: Use a gesture device to capture the user's gesture movements and collect gesture data; the gesture data includes IMU data from the inertial sensor and image data from the camera;

[0008] S2, data preprocessing: perform activity segment detection, data filtering and data normalization on the IMU data and image data obtained in S1; for IMU data, extract its MAV time domain features; for image data, convert it into a grayscale image after region cropping, and adjust the image size;

[0009] S3, feature extraction: The IMU data and image data processed by S2 are respectively input into the two sub-branches of the trained two-stream neural network model; the first sub-branch processes the IMU data and extracts the deep features of the IMU data; the second sub-branch processes the image data and extracts the deep features of the image;

[0010] S4, feature fusion: splice the feature maps extracted by the two sub-branches to integrate IMU data and image data;

[0011] S5, classification and recognition: input the fused features into the classifier, use the Softmax activation function for multi-classification, and obtain the classification result of the gesture;

[0012] S6. Command output: Output corresponding commands to the unmanned equipment based on the classification results.

[0013] Further, in step S1, the gesture device includes a glove worn at the user's hand position and a pair of glasses worn at the user's eye position; the inertial sensor is arranged at the fingertips and / or the back of the hand of the glove to collect gesture motion signals; the camera is arranged at the glasses to collect hand images.

[0014] Furthermore, in step S2, for the collected IMU data, a fifth-order Butterworth filter is first applied for digital filtering, and the quality factor of the filter is set to 0.5. Then, a 50 Hz notch filter is used to remove power line frequency interference, and then a 5-200 Hz bandpass filter is applied to retain the most important frequency components in the gesture action, while filtering out high-frequency noise and low-frequency drift beyond this range; finally, the filtered signal is amplitude normalized to scale the signal value to the range of -1 to 1.

[0015] Furthermore, in step S2, for the collected IMU data, when the activity segment detection is performed, when the instantaneous energy value is greater than 500 and 6 consecutive sampling points exceed the set threshold, it is determined as the gesture start mark and data is intercepted. The intercepted data length is 800ms and the sampling frequency is 100Hz.

[0016] Further, in step S3, when the IMU data flows into the first sub-branch, the dimension of the input data is 80×30×1, and the network front end is composed of three convolutional layers, all of which use the ReLU activation function, where the number of convolution kernels in the first layer is 32, the size is 1×1, the step size is 1, and the input is filled with 1 pixel with a value of 0, the number of convolution kernels in the second layer is 64, the size is 3×3, the step size is 1, and the convolution edge is filled with 0, and the number of convolution kernels in the third layer is 128, the size is 3×3, the step size is 1, and the convolution edge is filled with 0;

[0017] Each convolutional layer is followed by a batch normalization layer and a maximum pooling layer, and the step sizes of the three pooling layers are 2×2, 2×2, and 2×1, respectively; the feature map output by the convolutional layer is reshaped into a two-dimensional sequence and input into a bidirectional LSTM network with an output dimension of 32 and an activation function of ReLU; the deep features of the IMU data are then obtained through the batch normalization layer and the fully connected layer output.

[0018] Furthermore, a Dropout layer is added between the bidirectional LSTM and the fully connected layer with a ratio of 50%, and the features are flattened using a Flatten layer.

[0019] Further, in step S3, when the image data flows into the second sub-branch, the dimension of the input data is 160×120×24×1, and two 3D convolutional layers are used to extract the spatiotemporal features of the image data. The first 3D convolutional layer has 32 convolutional kernels with a size of 3×3×3 and edge padding of 0, and the second 3D convolutional layer has 64 convolutional kernels with a size of 3×3×3 and edge padding of 0; all convolutional layers are followed by ReLU activation functions;

[0020] Each convolutional layer is followed by a batch normalization layer, followed by a maximum pooling layer with a step size of 3×3×3. The feature map output by the 3D convolutional layer is reshaped into a two-dimensional sequence and input into the bidirectional GRU network. The output dimension of each GRU unit is 64, and the ELU activation function is used.

[0021] Furthermore, a 20% Dropout layer is added between the second pooling layer and the bidirectional GRU layer, a 25% Dropout is added from the bidirectional GRU layer to the fully connected layer, and a Flatten layer is used before the fully connected layer to flatten the output of the GRU layer.

[0022] In a second aspect, the present invention further provides an unmanned device, which is controlled by applying the aforementioned control method.

[0023] Compared with the prior art, the unmanned equipment control method based on gesture recognition provided by the present invention can have but is not limited to the following beneficial technical effects:

[0024] (1) Dual-stream fusion network: The present invention uses dual-stream fusion of IMU data and image data to provide more comprehensive gesture feature information. For example, visual data can provide the shape and motion trajectory of a gesture, while IMU data can provide the speed and acceleration information of a gesture. This multi-dimensional information is conducive to improving the accuracy and robustness of gesture recognition, especially in the recognition of dynamic and complex gestures.

[0025] (2) Deep learning model: The present invention uses 3D convolution and bidirectional GRU / LSTM networks. These deep learning models are more effective in capturing spatiotemporal features.

[0026] (3) Feature extraction and fusion: The present invention performs special feature extraction and fusion processing on IMU data and image data, making the model more accurate in processing complex gestures;

[0027] (4) Dropout and activation function optimization: The present invention uses the Dropout layer and the ELU activation function to improve the generalization ability of the model and reduce the overfitting problem during the training process;

[0028] (5) Structured network design: The network design of the present invention includes a complete process from data preprocessing to feature extraction and then to classification and recognition. This structured design helps to optimize and maintain system performance.

[0029] At the same time, the present invention provides an unmanned equipment that adopts the above control method and can therefore have the same technical effect.

[0030] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their description are used to explain the present invention but do not constitute a limitation of the present invention.

[0032] Figure 1 A flowchart of an unmanned equipment control method based on gesture recognition provided by the present invention;

[0033] Figure 2 This is a multi-channel instantaneous energy superposition diagram of the present invention when collecting IMU data

[0034] Figure 3 is a schematic diagram of a visual-inertial fusion control system of the present invention;

[0035] Figure 4 This is a diagram of the image processing network structure of the image data of the present invention. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0037] Embodiment 1

[0038] like Figure 1As shown: This embodiment provides an unmanned equipment control method based on gesture recognition, including the following steps S1-S6.

[0039] Step S1, gesture data collection: using a gesture device to capture the user's gesture movements and collect gesture data; the gesture data includes IMU data from an inertial sensor (IMU) and image data from a camera;

[0040] Step S2, data preprocessing: perform activity segment detection, data filtering and data normalization on the IMU data and image data obtained in S1; for the IMU data, extract its MAV (Mean Amplitude Value, which can be called average amplitude value) time domain features; for the image data, convert it into a grayscale image after region cropping, and adjust the image size;

[0041] Step S3, feature extraction: input the IMU data and image data processed by S2 into the two sub-branches of the trained two-stream neural network model respectively; the first sub-branch processes the IMU data and extracts the deep features of the IMU data; the second sub-branch processes the image data and extracts the deep features of the image;

[0042] Step S4, feature fusion: splicing the feature maps extracted by the two sub-branches to integrate the IMU data and image data;

[0043] Step S5, classification and recognition: input the fused features into the classifier, use the Softmax activation function (soft maximum activation function) to perform multi-classification, and obtain the classification result of the gesture;

[0044] Step S6, command output: output corresponding commands to the unmanned equipment according to the classification results; the control method can be integrated into a controller, and the controller and the unmanned equipment are connected wirelessly.

[0045] This embodiment provides an unmanned equipment control method based on gesture recognition. In step S1, the gesture device includes a glove worn on the user's hand and a pair of glasses worn on the user's eye. The inertial sensor is arranged at the fingertips and the back of the hand of the glove to collect gesture motion signals. The camera is arranged on the glasses to collect hand images.

[0046] The gloves integrate inertial sensors arranged at the fingertips and back of the hands. These sensors are responsible for collecting dynamic signals of hand movements in real time, including but not limited to acceleration and angular velocity, to provide accurate motion data for gesture recognition. The glasses are equipped with a camera, the position and design of which enable it to clearly capture images of the hands. These image data provide the system with visual information of the hand shape and movement, which is crucial for understanding hand movements and gesture recognition.

[0047] This wearable gesture device can collect data from multiple angles and methods at the same time, enhancing the accuracy and reliability of gesture recognition. The inertial sensors of the gloves provide dynamic motion information, while the cameras on the glasses provide static and dynamic visual information. The design of this device takes into account the naturalness and convenience of user operation, and at the same time ensures the comprehensiveness and efficiency of data collection by integrating high-precision sensors and cameras. Through these data, the system can realize accurate recognition of complex gestures to meet the needs of different application scenarios.

[0048] The inertial sensor may be, for example, an LSM6DS3 sensor; the camera may be an RGB (three primary colors) camera or a depth camera as appropriate.

[0049] In the process of dynamic gesture collection, the inertial sensor signal will be affected by factors such as the subject's jitter and the surrounding environment. The information will also be interfered to a certain extent by background noise and inherent noise of electronic equipment during transmission to the computer through the data cable of the control board. These useless signals not only reduce the accuracy of gesture recognition, but also increase the difficulty of gesture activity segment detection and model classification in subsequent online control experiments. Therefore, gesture data is preprocessed. Gesture data preprocessing can remove noise and sharp points in the signal to obtain a smooth signal, so that the signal can clearly and accurately reflect the true characteristics of the gesture and improve the classification accuracy of the algorithm.

[0050] In order to reduce the influence of noise on gesture signals, in step S2, a fifth-order Butterworth filter is first applied to the collected IMU data for digital filtering, and the quality factor of the filter is set to 0.5. Then, a 50 Hz notch filter is used to remove power line frequency interference, and then a 5-200 Hz bandpass filter is applied to retain the most important frequency components in the gesture action, while filtering out high-frequency noise and low-frequency drift beyond this range; finally, the filtered signal is amplitude normalized and the signal value is scaled to the range of -1 to 1 to eliminate the influence of different feature magnitude differences on gesture recognition accuracy; after data normalization and noise reduction filtering, the inertial data can improve the signal-to-noise ratio of the signal.

[0051] The inertial sensor continuously sends data to the computer, and single complete gestures are also mixed in. This chapter uses activity segment detection to quickly and accurately find single complete gestures in the real-time data stream to achieve efficient gesture segmentation; during the static period, the angular velocity and acceleration values ​​of the data glove are close to zero, and when making gestures, there will be a significant increase. The angular velocity and acceleration data of multiple channels of the data glove are all superimposed, such as Figure 2 As shown, it can be clearly seen that the starting point of the key action is when the instantaneous energy value amplitude is about 300-500, an effective gesture starts, the gesture length is 600-800ms, and the subsequent symmetrical energy represents a meaningless action of returning to the initial state after the gesture is executed. After repeated experimental analysis, the threshold is set to 500, that is, when the instantaneous energy value is greater than 500 and 6 consecutive sampling points exceed the set threshold, it is judged as the gesture start mark and data is intercepted, and the intercepted data length is 800ms; since the gesture time window length of each gesture intercepted by a single data sample is 800ms, the sampling frequency is 100Hz.

[0052] This embodiment provides an unmanned equipment control method based on gesture recognition. In step S3, when IMU data flows into the first sub-branch, the dimension of the input data is 80×30×1 (each gesture sample contains 80 time points, and each time point has feature data of 30 channels). The network front end is composed of three convolutional layers, which are specially used to process one-dimensional time series data, and all use ReLU (rectifying linear unit) activation function; wherein, the number of convolution kernels in the first layer is 32, the size is 1×1, the step size is 1, and the input is filled with 1 pixel with a value of 0 to perform spatial mapping of the input data; the number of convolution kernels in the second layer is 64, the size is 3×3, the step size is 1, and the convolution edge is filled with 0, which is used to extract more complex time features; the number of convolution kernels in the third layer is 128, the size is 3×3, the step size is 1, and the convolution edge is filled with 0 to further extract deep-level time features.

[0053] Each convolutional layer is followed by a batch normalization layer to stabilize the learning process and improve generalization ability; followed by a maximum pooling layer, and the step sizes of the three pooling layers are 2×2, 2×2, and 2×1, respectively, which are used to reduce the feature dimension and extract the main features; the feature map output by the convolutional layer is reshaped into a two-dimensional sequence suitable for recurrent neural network processing and input into a bidirectional LSTM (long short-term memory) network with an output dimension of 32 and an activation function of ReLU. The bidirectional LSTM network can extract long-term dependencies and dynamic features in time series data; the features are further integrated through batch normalization layers and fully connected layers, and the deep features of the IMU data are output.

[0054] At the same time, a Dropout layer (also called a "discarding layer" or "random inactivation layer") is added between the bidirectional LSTM and the fully connected layer, with a ratio of 50%, and a Flatten layer (called a "flattening layer" or "flattening layer") is used to flatten the features. Bidirectional LSTM is used to process sequence data, and can simultaneously consider past and future information and capture dynamic features in time series; adding a Dropout layer between the bidirectional LSTM layer and the fully connected layer, with a ratio of 50%, means that 50% of the neuron outputs are randomly discarded (set to zero) during training, which helps prevent the model from overfitting and improves the generalization ability of the model; using the Flatten layer to flatten the three-dimensional features (time step, feature dimension, number of LSTM units) output by the LSTM layer into a one-dimensional feature vector, converting the data into the format required by the fully connected layer; the feature vector after the Flatten layer is then input into the fully connected layer, which linearly combines the features through weights and biases, and introduces nonlinearity through activation functions for final classification or other tasks.

[0055] That is to say, after the bidirectional LSTM layer successfully extracts the features of the time series data, a Dropout layer is introduced with a dropout rate of 50% in order to reduce the model's dependence on the training data and avoid overfitting. Subsequently, the features are flattened into a one-dimensional vector through the Flatten layer to match the input requirements of the fully connected layer. In this way, the model can further learn the high-level representation of the features in the fully connected layer and make accurate predictions for the final task (such as classification). This structure can effectively improve model performance and robustness in deep learning, especially when processing sequence data.

[0056] This embodiment provides an unmanned equipment control method based on gesture recognition. In step S3, when image data flows into the second sub-branch, the dimension of the input data is 160×120×24×1 (indicating that each sample contains a sequence of 24 frames, the size of each frame image is 160×120 pixels, single channel-grayscale image), and two 3D convolution layers are used to extract the spatiotemporal features of the image data. 3D convolution can capture the temporal and spatial information in the image sequence; the first 3D convolution layer has 32 convolution kernels with a size of 3×3×3 and edge padding of 0, which is used to extract preliminary spatiotemporal features; the second 3D convolution layer has 64 convolution kernels with a size of 3×3×3 and edge padding of 0, which is used to extract more complex spatiotemporal features; all convolution layers are followed by a ReLU activation function to introduce nonlinearity and enhance the network's ability to express features.

[0057] Each convolutional layer is followed by a batch normalization layer to stabilize the learning process and accelerate convergence; followed by a maximum pooling layer with a step size of 3×3×3, which is used to reduce the spatial and temporal dimensions of the features while retaining important information; the feature map output by the 3D convolutional layer is reshaped into a two-dimensional sequence to adapt to the subsequent recurrent neural network; the reshaped sequence is input into a bidirectional GRU (gated recurrent unit) network, and the output dimension of each GRU unit is 64. The ELU (exponential linear unit) activation function is used instead of the traditional ReLU or tanh activation function in LSTM, which helps to alleviate the gradient vanishing problem in deep learning and allows negative input, which helps to speed up convergence.

[0058] At the same time, a 20% Dropout layer is added between the second pooling layer and the bidirectional GRU layer. During the training process, 20% of the neuron outputs will be randomly set to zero to reduce the overfitting of the model; a 25% Dropout is added from the bidirectional GRU layer to the fully connected layer to further enhance the generalization ability of the model and ensure that the model does not overfit the training data; a Flatten layer is used before the fully connected layer to flatten the output of the GRU layer. The features output by the bidirectional GRU layer usually have a three-dimensional structure (for example, time step, feature depth). Before sending these features to the fully connected layer, the Flatten layer is used to flatten these three-dimensional features into a one-dimensional feature vector to match the input dimension requirements of the fully connected layer; the feature vector processed by the Flatten layer is then sent to the fully connected layer, which linearly transforms the features through the weight matrix and bias vector, and may be further processed through a nonlinear activation function to complete the final decision or classification task of the model.

[0059] That is to say, Dropout and Flatten layers are used in the key parts of the deep learning model to optimize network performance. A Dropout layer with a ratio of 20% is added after the second pooling layer to reduce the complex co-adaptation between neurons. Subsequently, a Dropout layer with a ratio of 25% is added between the bidirectional GRU layer and the fully connected layer to further enhance the generalization ability of the model. Before the fully connected layer, the Flatten layer is used to convert the output of the GRU layer from a three-dimensional feature map to a one-dimensional feature vector to ensure that the data can flow smoothly into the fully connected layer for final classification or other tasks. Through this design, the model maintains efficient learning ability while also improving its adaptability and robustness to new data.

[0060] In this embodiment, the convolution layer in the network is used to extract gestures, and the global gesture feature information is obtained by integrating different local receptive fields; in order to prevent overfitting, a batch normalization layer is added after each hidden layer, namely the convolution layer and the recurrent layer, to improve the generalization ability of the model. The maximum pooling layer can effectively reduce the parameters and the amount of calculation, reduce information redundancy, and fill 0 for edge feature extraction. Dropout can mask the neurons of the hidden layer in a custom ratio. The entire training process is equivalent to averaging many different neural networks, thereby reducing the possibility of overfitting as a whole; compared with RELU, the ELU activation function has negative values, which can push the value to near 0 when acting on the output, reducing the offset effect so that the gradient is close to the natural gradient, and reducing the amount of calculation. The traditional LSTM structure is used for IMU data. In particular, for image data, the features of the convolution layer are used as the input of the gated recurrent unit (GRU), which is based on the RNN structure to capture long-term dependencies and spatiotemporal order. The modified module in the RNN is similar to the LSTM, but the GRU has a simpler structure, especially because the parameters are less than the LSTM, so the same learning effect can be achieved with less parameters and time.

[0061] GRU is a type of unit used in Recurrent Neural Network (RNN), which is designed to capture long-term dependencies in time series data. GRU was proposed by Kyunghyun Cho et al. in 2014 as a variant of LSTM network, aiming to simplify the model structure while maintaining effective sequence processing capabilities. The structure of GRU is relatively simple. Compared with LSTM, which requires three gating units, it only uses update gate and reset gate. These two gating units determine that certain gesture input information can eventually be used as the output of the gated recurrent unit. By calculating the hidden state, the gating mechanism can learn long sequences well and will not be cleared over time or removed because it is irrelevant to the prediction; the internal structure of GRU contains the input vector at the current moment And the information passed down by the node at the previous moment.

[0062] Embodiment 2

[0063] This embodiment provides an unmanned device, which is controlled by applying the control method shown in the first embodiment, and the same structure and principle are not repeated here.

[0064] Among them, the unmanned equipment can be an unmanned aerial vehicle (UAV), an unmanned vehicle (UGV), an unmanned ship (USV), an unmanned underwater vehicle (UUV) and other types of automated or remote-controlled equipment.

[0065] Finally, it should be noted that this article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only to help understand the core idea of ​​the present invention. Without departing from the principles of the present invention, the present invention may also be improved and modified, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A method for controlling unmanned equipment based on gesture recognition, characterized in that: The following steps are involved: S1. Gesture data collection: Use a gesture device to capture the user's gesture movements and collect gesture data; the gesture data includes IMU data from the inertial sensor and image data from the camera; S2, data preprocessing: perform activity segment detection, data filtering and data normalization on the IMU data and image data obtained in S1; for IMU data, extract its MAV time domain features; for image data, convert it into a grayscale image after region cropping, and adjust the image size; S3, feature extraction: The IMU data and image data processed by S2 are respectively input into the two sub-branches of the trained two-stream neural network model; the first sub-branch processes the IMU data and extracts the deep features of the IMU data; The second sub-branch processes the image data and extracts the deep features of the image; S4, feature fusion: splice the feature maps extracted by the two sub-branches to integrate IMU data and image data; S5, classification and recognition: input the fused features into the classifier, use the Softmax activation function for multi-classification, and obtain the classification result of the gesture; S6. Command output: Output corresponding commands to the unmanned equipment based on the classification results.

2. The unmanned equipment control method based on gesture recognition according to claim 1, characterized in that: In step S1, the gesture device includes a glove worn on the user's hand and a pair of glasses worn on the user's eye; the inertial sensor is arranged on the fingertips and / or the back of the hand of the glove to collect gesture motion signals; the camera is arranged on the glasses to collect hand images.

3. The unmanned equipment control method based on gesture recognition according to claim 1, characterized in that: In step S2, for the collected IMU data, a fifth-order Butterworth filter is first applied for digital filtering, and the quality factor of the filter is set to 0.

5. Then, a 50 Hz notch filter is used to remove power line frequency interference, and then a 5-200 Hz bandpass filter is applied to retain the most important frequency components in the gesture action, while filtering out high-frequency noise and low-frequency drift beyond this range; finally, the filtered signal is amplitude normalized to scale the signal value to the range of -1 to 1.

4. The unmanned equipment control method based on gesture recognition according to claim 1, characterized in that: In step S2, for the collected IMU data, when the activity segment detection is performed, when the instantaneous energy value is greater than 500 and 6 consecutive sampling points exceed the set threshold, it is determined as the gesture start mark and data is intercepted. The intercepted data length is 800ms and the sampling frequency is 100Hz.

5. The unmanned equipment control method based on gesture recognition according to claim 1, characterized in that: In step S3, when the IMU data flows into the first sub-branch, the dimension of the input data is 80×30×1, and the network front end consists of three convolutional layers, all of which use the ReLU activation function. Among them, the number of convolution kernels in the first layer is 32, the size is 1×1, the step size is 1, and the input is filled with 1 pixel with a value of 0. The number of convolution kernels in the second layer is 64, the size is 3×3, the step size is 1, and the convolution edge is filled with 0. The number of convolution kernels in the third layer is 128, the size is 3×3, the step size is 1, and the convolution edge is filled with 0. Each convolutional layer is followed by a batch normalization layer and a maximum pooling layer, and the step sizes of the three pooling layers are 2×2, 2×2, and 2×1, respectively; the feature map output by the convolutional layer is reshaped into a two-dimensional sequence and input into a bidirectional LSTM network with an output dimension of 32 and an activation function of ReLU; the deep features of the IMU data are then obtained through the batch normalization layer and the fully connected layer output.

6. The unmanned equipment control method based on gesture recognition according to claim 5, characterized in that: A Dropout layer is added between the bidirectional LSTM and the fully connected layer with a ratio of 50%, and a Flatten layer is used to flatten the features.

7. The unmanned equipment control method based on gesture recognition according to claim 5, characterized in that: In step S3, when the image data flows into the second sub-branch, the dimension of the input data is 160×120×24×1, and two 3D convolutional layers are used to extract the spatiotemporal features of the image data. The first 3D convolutional layer has 32 convolution kernels with a size of 3×3×3 and edge padding of 0, and the second 3D convolutional layer has 64 convolution kernels with a size of 3×3×3 and edge padding of 0; all convolutional layers are followed by ReLU activation function; Each convolutional layer is followed by a batch normalization layer, followed by a maximum pooling layer with a step size of 3×3×3. The feature map output by the 3D convolutional layer is reshaped into a two-dimensional sequence and input into the bidirectional GRU network. The output dimension of each GRU unit is 64, and the ELU activation function is used.

8. The unmanned equipment control method based on gesture recognition according to claim 7, characterized in that: A 20% Dropout layer is added between the second pooling layer and the bidirectional GRU layer, a 25% Dropout is added from the bidirectional GRU layer to the fully connected layer, and a Flatten layer is used before the fully connected layer to flatten the output of the GRU layer.

9. An unmanned device, characterized in that: The unmanned equipment is controlled by the control method described in any one of claims 1 to 8.

Images