Multi-scale Mamba transform based visual gesture flight control method for UAV

Abstract

The invention discloses a multi-scale Mamba transform unmanned aerial vehicle visual gesture flight control method, comprising the steps of collecting gesture images through a camera of the unmanned aerial vehicle, preprocessing the collected gesture images, labeling gesture categories for the preprocessed gesture images, preparing a gesture data set, extracting features and performing gesture recognition on the gesture images by using a convolutional neural network, performing an average pooling operation on the obtained output values ​​and performing gesture image classification, training to obtain network parameters and weights, and converting gesture recognition results into control instructions and sending them to the unmanned aerial vehicle to realize the control of the flight state of the unmanned aerial vehicle; adopting a pyramid visual Mamba model combined with a multi-scale aggregation module can effectively extract gesture image features and perform recognition, and can improve the efficiency and accuracy of gesture recognition to a certain extent.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle control, and in particular, relates to a multi-scale Mamba transform unmanned aerial vehicle visual gesture flight control method. Background Art

[0002] Traditional drone flight control requires special equipment, generally using remote controls and complex buttons, but has high requirements for operator operation and requires relevant personnel to receive professional training. It can no longer meet the general public's demand for convenient drone control, and to a certain extent limits the development and application of the drone industry.

[0003] In the context of the artificial intelligence era, the development of drone flight control methods based on visual gesture recognition has long-term development significance, and convenient and novel interaction methods are often more popular with users.

[0004] The UAV visual gesture flight control method provides an intuitive and convenient way of control by integrating computer vision, deep learning and flight control technology. This technology not only improves the user experience of UAVs, but also expands their application potential in multiple fields. However, to achieve widespread application, challenges such as environmental adaptability, real-time performance and user diversity need to be overcome. Summary of the invention

[0005] In view of the above defects, the present invention provides a multi-scale Mamba transform UAV visual gesture flight control method, comprising the following steps:

[0006] S1, collecting gesture images through the camera of the drone;

[0007] S2, improving the quality of feature extraction by preprocessing the gesture image collected in step S1;

[0008] S3, labeling the gesture images preprocessed in step S2 with gesture categories and creating a gesture dataset;

[0009] S4. Using a convolutional neural network to perform feature extraction and gesture recognition on gesture images, the collected gesture dataset is trained and tested by a Mamba network model constructed by including an embedding block for reducing the resolution of the input image and multiple visual Mamba encoders for calculating the image attention value;

[0010] A multi-scale aggregation module is placed between each visual Mamba encoder;

[0011] S5, performing an average pooling AvgPool operation on the output value obtained in step S4 through a multi-scale aggregation module, and using a softmax function to classify the gesture image;

[0012] S6. Use the self-collected gesture data set for training to obtain network parameters and weights, and then use the offline trained convolutional neural network for forward inference to obtain the corresponding gesture recognition results;

[0013] S7. Convert the gesture recognition results into control commands, and use TelloSDK to send control commands to the drone, so as to control the flight status of the drone according to predefined gesture actions.

[0014] Furthermore, the Mamba network model includes stages 1 to 4, which are used to expand the feature dimension and compress the spatial dimension of the input image, and output features of different scales, including:

[0015] Stage 1: First, the input gesture image is divided into slices, each of which has a size of , and then the slices are fed into the linear projection layer to obtain The embedding feature block is constructed by using the step size in PatchEmbedding to reduce the resolution of the feature map. Then, learnable positional encoding information is added to the output of the linear projection layer. and sent to the The visual Mamba encoder of the layer finally outputs the feature map The size is ;

[0016] Stage 2: Output feature map of stage 1 As the input of stage 2, repeat the workflow of stage 1 to obtain the stage 2 feature map ;

[0017] Stage 3: Output feature map of stage 2 As the input of stage 3, repeat the workflow of stage 1 to obtain the stage 3 feature map ;

[0018] Stage 4: Output feature map of stage 3 As the input of stage 4, repeat the workflow of stage 1 to obtain the stage 4 feature map .

[0019] Furthermore, each of the visual Mamba encoders includes layer normalization, a state space model and a feedforward neural network, which are used to perform layer normalization LN operations on the input values ​​in sequence, then perform feature extraction through a multi-layer perceptron, and then obtain enhanced local features through deep convolution processing. The obtained local features are then sent to the SiLU activation function to improve the nonlinearity of the model, and the activated features are refined through the state space model SSM. Finally, the original residual input value is added to the intermediate result value after multi-layer transformation after layer normalization and SiLU processing to form a residual connection to obtain the output value.

[0020] Furthermore, each of the visual Mamba encoders includes two residual connections.

[0021] Furthermore, the working steps of the multi-scale aggregation module are:

[0022] P1. Definition Output feature map of the stage ,in Indicates the number of channels, and Represents the height and width of the feature map. Use the spatial aggregation module to enhance the spatial features and transform the feature map Send it to the one-dimensional convolution for channel mapping, and the number of output channels is The feature map of , further, the output feature map is convolutional kernels with different sizes. , as well as The parallel convolution operation obtains multi-scale features, and the output is summed to obtain the feature map .

[0023] P2. Use the average pooling AvgPool operation to obtain the average value of each spatial position of the feature map, concatenate the output and send it to a convolutional layer to learn the weight of each spatial position, and finally use the sigmoid activation function to normalize the output to between 0 and 1. The calculation process of the spatial aggregation module is as follows:

[0024] ；

[0025] .

[0026] in, , , , The convolution kernel sizes are , , , The convolution of represents element-wise multiplication, represents the element-by-element sum operation, Represents the sigmoid activation function, which aggregates the output of the spatial aggregation module After one-dimensional convolution, we get .

[0027] P3, feature map It is sent to the channel aggregation module, and firstly the average pooling AvgPool operation is performed on the channel dimension to obtain The feature map of C is the number of channels, and then these pooling results are sent to The size of the convolution is processed, normalized by the ReLU activation function, and further The convolution obtains the weight matrix for each channel dimension , this calculation process can be expressed as:

[0028] .

[0029] P4. Output of the channel aggregation module and Multiply and combine with the input feature map The sum is the output of the final multi-scale attention aggregation module , expressed as:

[0030] .

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. Use the drone camera to capture the specific movements of the human hand, use the convolutional neural network to recognize the collected gesture images, and further convert the gestures into control commands to achieve real-time drone flight control. The operator can easily control the drone through gestures;

[0033] 2. The pyramid vision Mamba model combined with the multi-scale aggregation module can effectively extract gesture image features and recognize them. Compared with the existing Mamba model, it can improve the efficiency and accuracy of gesture recognition to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0035] In order to facilitate the understanding of the present invention, the device of the present invention will be described more fully below with reference to the relevant drawings. Embodiments of the device are given in the drawings. However, the device can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0036] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "set" should be understood in a broad sense, for example, they can be fixedly connected or set, or detachably connected or set, or integrally connected or set. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0037] Example

[0038] like Figure 1 As shown, this embodiment provides a multi-scale Mamba transform UAV visual gesture flight control method, which specifically includes the following steps:

[0039] S1, using the drone's airborne camera and fixed position to collect gesture images;

[0040] The use of a fixed-position airborne camera ensures image stability and consistency, which facilitates subsequent preprocessing and model training.

[0041] S2, preprocessing the gesture image collected in step S1 to improve the quality of feature extraction;

[0042] The preprocessing operations include but are not limited to: skin color detection, light compensation, binarization, Gaussian filtering, contour extraction, etc., which help to improve the recognizability of gestures.

[0043] S3, using Labelme software to label the gesture images preprocessed in step S2, and create a gesture dataset;

[0044] Specifically, this embodiment defines 10 gesture actions: 0 to 9, corresponding to 10 flight instructions: start, take-off, landing, hovering, vertical upward flight, vertical downward flight, horizontal forward flight, horizontal backward flight, horizontal left flight, and horizontal right flight.

[0045] S4. Use a convolutional neural network (CNN) to perform feature extraction and gesture recognition on the gesture image preprocessed in step S2, and train and test the network model on the gesture data set prepared in step S3.

[0046] like Figure 1As shown in the figure, the network model adopts the pyramid visual Mamba network model. The Mamba network model includes stages 1 to 4, which are used to expand the feature dimension and compress the spatial dimension of the input image, and then output features of different scales. It should be noted that each stage consists of a block embedding (PatchEmbedding) and multiple visual Mamba encoders ( Figure 1 The block embedding is used to reduce the resolution of the input image, and the Mamba encoder is used to calculate the attention value of the image;

[0047] The specific steps are as follows: First, define an input gesture image ,in and Represent the height and width of the input image respectively. 3 means that the input is a three-channel (RGB) color image. The image is input into the Mamba network model for feature encoding.

[0048] Specifically, first enter a The input gesture image of size then enters the following stages:

[0049] Stage 1: First, the input gesture image is divided into slices, each of which has a size of , and then the slices are fed into the linear projection layer to obtain The embedded feature block (this stage reduces the resolution of the feature map by utilizing the step size in PatchEmbedding); then, the learnable position encoding information is added to the output feature map obtained by the linear projection layer and sent to the The visual Mamba encoder of the layer finally outputs the feature map The size is ;

[0050] Stage 2: Output feature map of stage 1 As the input of stage 2, repeat the workflow of stage 1 to obtain the stage 2 feature map ;

[0051] Stage 3: Output feature map of stage 2 As the input of stage 3, repeat the workflow of stage 1 to obtain the stage 3 feature map ;

[0052] Stage 4: Output feature map of stage 3 As the input of stage 4, repeat the workflow of stage 1 to obtain the stage 4 feature map ;

[0053] Among them, the feature map It contains low-level semantic information of the target gesture and retains a lot of detailed information such as hand contour and shape. , , contains high-level semantic information for gesture recognition;

[0054] In the above four stages, each visual Mamba encoder is mainly composed of layer normalization (LayerNormalization, LN), state space model (StateSpaceModel, SSM) and feedforward neural network (FeedForwardNetwork, FFN);

[0055] In detail, the layer normalization (LN) calculation process is as follows:

[0056] ;

[0057] ;

[0058] ;

[0059] In the above formula, is the sample feature dimension, For sample No. dimensional eigenvalue, , The samples are The mean and standard deviation of is the minimum value, , As a trainable parameter, the state space model SSM converts a one-dimensional function (i.e., the feature of each stage) F i ) or sequence Mapping to output The specific process is as follows:

[0060] ;

[0061] ;

[0062] in, is the evolution parameter, , is the projection parameter, is the hidden state, for The derivative of N To provide dimensional information, the state space model SSM uses the time scale parameter The continuous parameters are discretized by the discretization rule. Convert to discrete parameters , the process is as follows:

[0063] ;

[0064] ;

[0065] in, represents the matrix exponential, is the identity matrix, discretized After that, the output is discretized It is expressed as:

[0066] ;

[0067] ;

[0068] Finally, these models compute the output via global convolution , whose expression is:

[0069] ;

[0070] ;

[0071] in, M For the input sequence Length, is a structured convolution kernel.

[0072] like Figure 1 As shown, based on the above calculation principle, the specific workflow of the four stages is:

[0073] The Mamba encoder first inputs the feature map of stage i-1 Perform layer normalization LN operation, then extract features through Multi-Layer Perception (MLP), and then obtain enhanced local features through Depth-Wise Convolution (DWC) processing , further, the local features obtained The sigmoid activation function is used to improve the nonlinearity of the model. The activated features are refined through the state space model SSM. Finally, the original residual is input After layer normalization LN, multi-layer perceptron MLP and sigmoid processing, it is multiplied with the features refined by the SSM model and further multiplied with the input features Add to form a residual connection to get the output , this process can be expressed as:

[0074] ;

[0075] ;

[0076] In the above formula, Represents the sigmoid activation function. FFN consists of two linear layers and an activation function. The calculation formula is as follows:

[0077] ;

[0078] In the above formula, , is the weight matrix of the linear transformation, , is the corresponding bias, represents any input variable, and ReLU is the activation function. Here, the output of the FFN module is combined with The residual connections are added together. Each visual Mamba encoder contains two residual connections, whose main function is to alleviate the problem of network gradient disappearance.

[0079] At the same time, in order to pay more attention to the important areas in the gesture image that are conducive to classification, a multi-scale aggregation module MSAM is also used between each visual Mamba encoder, which can effectively extract gesture image features and perform recognition. Figure 1 As shown in the bottom box, the multi-scale aggregation module includes a spatial aggregation module and a channel aggregation module:

[0080] a) Define Output feature map of the stage ,in Indicates the number of channels, and Represents the height and width of the feature map. Use the spatial aggregation module to enhance the spatial features and transform the feature map Send it to the one-dimensional convolution for channel mapping, and the number of output channels is The calculation process of the spatial aggregation module is as follows:

[0081] ;

[0082] ;

[0083] in, , , , The convolution kernel sizes are , , , The convolution of represents element-wise multiplication, represents the element-by-element sum operation, Represents the sigmoid activation function, which aggregates the output of the spatial aggregation module After one-dimensional convolution, we get .

[0084] Furthermore, the output feature map is convolved with different convolution kernel sizes. , as well as The parallel convolution operation obtains multi-scale features, and the output is summed to obtain the feature map .

[0085] Subsequently, the average pooling AvgPool operation is used to obtain the average value of each spatial position in the feature map, the output is concatenated and sent to a convolutional layer to learn the weight of each spatial position, and finally the sigmoid activation function is used to normalize the output to between 0 and 1.

[0086] b) The feature map It is sent to the channel aggregation module, and firstly the average pooling AvgPool operation is performed on the channel dimension to obtain The feature map of C is the number of channels, and then these pooling results are sent to The size of the convolution is processed, normalized by the ReLU activation function, and further The convolution obtains the weight matrix for each channel dimension , this calculation process can be expressed as:

[0087] ;

[0088] c) Further, the output of the channel aggregation module and Multiply and combine with the input feature map The sum is the output of the final multi-scale attention aggregation module , expressed as:

[0089] ;

[0090] S5, the step S4 obtained in stage 4 ) Perform an average pooling AvgPool operation and use the softmax function to classify gesture images and output gesture category labels.

[0091] S6. Use the self-collected gesture data set for training to obtain network parameters and weights, and then use the offline trained convolutional neural network for forward inference to obtain the corresponding gesture recognition results. It should be noted that the loss function of the above network model training is the classic cross entropy loss function in classification tasks.

[0092] S7. Convert the gesture recognition results into control commands, and use TelloSDK to send control commands to the drone, so as to control the flight status of the drone according to predefined gesture actions.

[0093] It should be noted that the structure described in the present invention can be implemented in a variety of different forms and is not limited to the described embodiments. Any equivalent transformations made by ordinary technicians in this field using the contents of the present invention description and drawings, or directly or indirectly applied to other related technical fields, such as the loading and unloading of other items, are included in the protection scope of the present invention.

Claims

1. A UAV visual gesture flight control method based on multi-scale Mamba transform, characterized by: The following steps are involved: S1, collecting gesture images through the camera of the drone; S2, improving the quality of feature extraction by preprocessing the gesture image collected in step S1; S3, labeling the gesture images preprocessed in step S2 with gesture categories and creating a gesture dataset; S4. Using a convolutional neural network to perform feature extraction and gesture recognition on gesture images, the collected gesture dataset is trained and tested by a Mamba network model constructed by including an embedding block for reducing the resolution of the input image and multiple visual Mamba encoders for calculating the image attention value; The Mamba network model includes stages 1 to 4, which are used to expand the feature dimension and compress the spatial dimension of the input image, and output features of different scales, including: Stage 1: First, the input gesture image is divided into slices, each of which has a size of , and then the slices are fed into the linear projection layer to obtain Then, we add learnable positional encoding information to the output of the linear projection layer. and sent to the The visual Mamba encoder of the layer finally outputs the feature map The size is ; Stage 2: Output feature map of stage 1 As the input of stage 2, repeat the workflow of stage 1 to obtain the stage 2 feature map ; Stage 3: Output feature map of stage 2 As the input of stage 3, repeat the workflow of stage 1 to obtain the stage 3 feature map ; Stage 4: Output feature map of stage 3 As the input of stage 4, repeat the workflow of stage 1 to obtain the stage 4 feature map ; A multi-scale aggregation module is placed between each visual Mamba encoder; S5, performing an average pooling AvgPool operation on the output value obtained in step S4 through a multi-scale aggregation module, and using a softmax function to classify the gesture image; S6. Use the self-collected gesture data set for training to obtain network parameters and weights, and then use the offline trained convolutional neural network for forward inference to obtain the corresponding gesture recognition results; S7. Convert the gesture recognition results into control commands, and use TelloSDK to send control commands to the drone, so as to control the flight status of the drone according to predefined gesture actions.

2. The UAV visual gesture flight control method based on multi-scale Mamba transformation as claimed in claim 1, characterized in that: Each of the visual Mamba encoders includes layer normalization, a state space model and a feedforward neural network, which are used to perform layer normalization LN operations on the input values ​​in sequence, then perform feature extraction through a multi-layer perceptron, and then obtain enhanced local features through deep convolution processing. The obtained local features are then sent to a sigmoid activation function to improve the nonlinearity of the model, and the activated features are refined through a state space model SSM. Finally, the original residual input value is added to the intermediate result value after multi-layer transformation after layer normalization and sigmoid processing to form a residual connection to obtain the output value.

3. The UAV visual gesture flight control method based on multi-scale Mamba transformation as claimed in claim 2, characterized in that: Each of the visual Mamba encoders contains two residual connections.

4. The UAV visual gesture flight control method based on multi-scale Mamba transformation as claimed in claim 2, characterized in that: The working steps of the multi-scale aggregation module are: P1. Definition Output feature map of the stage ,in Indicates the number of channels, and Indicates the height and width of the feature map; Use the spatial aggregation module to enhance spatial features and transform the feature map Send it to the one-dimensional convolution for channel mapping, and the number of output channels is The feature map of , further, the output feature map is convolutional kernels with different sizes. , as well as The parallel convolution operation obtains multi-scale features, and the output is summed to obtain the feature map ; P2. Use the average pooling AvgPool operation to obtain the average value of each spatial position of the feature map, concatenate the output and send it to a convolutional layer to learn the weight of each spatial position, and finally use the sigmoid activation function to normalize the output to between 0 and 1. The calculation process of the spatial aggregation module is as follows: ； ； in, , , , The convolution kernel sizes are , , , The convolution of represents element-wise multiplication, represents the element-by-element sum operation, Represents the sigmoid activation function, which aggregates the output of the spatial aggregation module After one-dimensional convolution, we get ; P3, feature map It is sent to the channel aggregation module, and firstly the average pooling AvgPool operation is performed on the channel dimension to obtain The feature map of C is the number of channels, and then these pooling results are sent to The size of the convolution is processed, normalized by the ReLU activation function, and further The convolution obtains the weight matrix for each channel dimension , this calculation process can be expressed as: ； P4. Output of the channel aggregation module and Multiply and combine with the input feature map The sum is the output of the final multi-scale attention aggregation module , expressed as: 。

Images