Real-time dynamic gesture control of unmanned aerial vehicle system and method

Abstract

The present application relates to a kind of real-time dynamic gesture control unmanned aerial vehicle system and method, first by method to obtain motion information identification neural network, method steps mainly include: the corresponding relationship of gesture and command is specified, the digital image set of gesture x1, the digital image set of gesture x2,..., the digital image set of gesture x n , the input set of gesture x1, the input set of gesture x2,..., the input set of gesture x n , construct and train and test neural network, finally obtain the neural network required. System is based on the content in method and established: the command form represented by gesture can be accurately identified by the present application, to control unmanned aerial vehicle.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) system control, and more specifically, to a real-time dynamic gesture control UAV system and method. Background Technology

[0002] With the rapid development of sensor technology and ubiquitous computing technology, sensor-based human gesture recognition technology is becoming increasingly popular. This technology is widely used in fields such as healthcare, sports management, and especially drone control.

[0003] Current gesture recognition technologies primarily utilize five methods: electromyography (EMG) sensor-based recognition, inertial sensor-based recognition, flexion sensor-based recognition, optical image recognition, and multi-sensor fusion recognition. Each of these methods has its own application scenarios and advantages. Optical image recognition relies on optical sensors to capture images, extracts and fuses features, and then uses deep learning algorithms for classification. Its drawback is that it is limited to specific scenarios, depends on the clarity of the optical image and the specific angle of the optical sensor, and cannot complete recognition tasks in many complex or occluded environments. On the other hand, gesture recognition based on EMG, inertial, and flexion sensors typically places sensors on the arm and hand joints to record dynamic gesture data changes. Feature extraction is performed based on this sensor data, and classification algorithms are used to complete gesture recognition. This method offers stable data and a high recognition rate. However, the weight of the sensors imposes certain limitations on the wearer's arm and hand, which is detrimental to real-time control of drone systems in aviation fields such as drone system control.

[0004] Multi-sensor fusion gesture recognition combines data from two or more types of sensors to compensate for the low recognition rate. It utilizes multimodal data fusion to achieve a higher recognition rate. However, this technology also has certain limitations on hand and arm movements and its sensitivity is not ideal, making it difficult to achieve real-time control of drone systems. Summary of the Invention

[0005] The purpose of this invention is to overcome the deficiencies of the prior art and to provide a real-time dynamic gesture control system and method for unmanned aerial vehicles that can acquire operator gesture information and recognize that the gesture represents a command.

[0006] The present invention provides a method for obtaining human dynamic gesture information recognition neural network, the technical solution of which is as follows:

[0007] A method for acquiring human dynamic gesture information recognition neural network includes the following steps:

[0008] Step 1: Select n gestures, denoted as gesture x1, gesture x2, ..., gesture x nIt is specified that among the n gestures, each gesture represents a drone system operation command, i.e., gesture x i Let i represent the command, where 1 ≤ i ≤ n;

[0009] Step 2: Obtain the operator's data using data gloves. i The gesture x mentioned below i Data on finger flexion, wrist acceleration, and wrist angular velocity at the time of the gesture, and the data on each time the gesture is performed. i The finger flexion data, wrist acceleration data, and wrist angular velocity data obtained at that time are used as a digital image, and further, the digital image is used as elements to form a system with n... i The gesture x of each element i A set of digital images; sequentially forming the set of digital images of gesture x1, the set of digital images of gesture x2, ..., the set of digital images of gesture x1 using the methods described above in this step. n A collection of digital images;

[0010] Step 3: The gesture x i Each digital image in the digital image set is preprocessed, and then the data in each digital image is divided into k blocks according to the physical quantity to which the data in each digital image belongs, thereby obtaining the gesture x. i The input set; sequentially forming the input set of gesture x1, the input set of gesture x2, ..., the input set of gesture x... according to the method described in this step. n The input set;

[0011] Step 4: Construct a neural network consisting of k convolutional networks for acquiring feature images, one fully connected network for summarizing the output images of the k convolutional networks and performing information recognition on the output images, and an output module, arranged sequentially along the running direction.

[0012] Step 5: Press the gesture x as described. i Each element in the input set is input data, and the command i is the ideal output mode. The neural network is trained, and after the training is successful, a test neural network is obtained.

[0013] Step 6: Construct a test set that matches the test neural network, and test the test neural network using the test set. After passing the test, the human dynamic gesture information recognition neural network is obtained.

[0014] Compared with the prior art, the technical solution provided by this invention application offers at least the following advantages: This method acquires finger flexion data, wrist acceleration data, and wrist angular velocity data using a data glove. Based on this, it preprocesses the chokehold data, dividing it into k blocks according to the corresponding physical quantities (k can be 2 or more). Then, it employs a neural network with k convolutional modules, trains and tests the network, and obtains a qualified neural network. This not only achieves the technical effect of recognizing gestures through a neural network to control a drone, but also allows for flexible movement of the fingers and arms throughout the process using a data glove, thus solving the technical problems of inaccurate sampling and inaccurate neural network test input data caused by restricting hand and arm movement.

[0015] Preferably, the data glove used in step two includes an operating glove equipped with a bending sensor for acquiring information about the operator's finger bending, an acceleration sensor for acquiring information about the operator's wrist acceleration, and an angular velocity sensor for acquiring information about the operator's wrist angular velocity. Because this data glove has sensors and easily allows for free hand gestures, the technical problem of restricted hand gestures is solved.

[0016] Preferably, the operators selected in step two include several people of different age groups; this is a requirement for the scientific nature of statistical sampling, which is not only beneficial for training neural networks to accurately recognize gestures, but also for achieving accurate sampling.

[0017] Preferably, in step three, the gesture x i The preprocessing process for each digital image in the digital image set includes the following steps:

[0018] Step 3.1: Eliminate the gesture x i Abnormal data in each of the digital images in the digital image set;

[0019] Step 3.2: Obtain the angular velocity threshold through statistical inference. For all frame data in the digital image processed in Step 3.1, if the angular velocity in a certain frame is less than the angular velocity threshold, then this frame data is regarded as static data; if the angular velocity in a certain frame data is not less than the angular velocity threshold, then this frame data is regarded as dynamic data.

[0020] Step 3.3: Obtain the segmentation length h using a statistical inference method. For all frame data in each digital image processed in Step 3.2, if h frames of static data appear consecutively after a certain frame of dynamic data, then this frame of data is taken as one gesture x. iEnd the data processing and delete all data in the digital image that follows this frame.

[0021] Step 3.4: Calculate the number of data frames contained in each digital image processed in Step 3.3, and set values ​​α and β based on the statistical results, where α < β. Digital images with a number of data frames less than the value α and greater than the value β are considered noise and removed.

[0022] Step 3.5: Using statistical inference methods, take a value γ between the value α and the value β, that is, let...

[0023] α<γ<β

[0024] For the digital images processed in step 3.4, the digital images with a data frame count between the value α and the value γ are upsampled, and the digital images with a data frame count between the value γ and the value β are downsampled.

[0025] This process improves the feasibility of training the neural network and highlights the unique nature of data preprocessing.

[0026] The present invention provides a real-time dynamic gesture control drone system, the technical solution of which is as follows:

[0027] A real-time dynamic gesture control drone system includes a digital signal acquisition module, a digital signal preprocessing module, a dynamic gesture information recognition neural network established based on the human dynamic gesture information recognition neural network acquisition method described in this application, and a terminal execution module arranged sequentially along the digital signal propagation direction.

[0028] The digital signal acquisition module is used to acquire finger bending data, wrist acceleration data, and wrist angular velocity data when the target object makes a gesture, and to form a digital image using the finger bending data, wrist acceleration data, and wrist angular velocity data as elements;

[0029] The digital signal preprocessing module is used to preprocess the digital image and form an input digital image;

[0030] The dynamic gesture information recognition neural network is used to recognize the input digital image and obtain the command represented by the input digital image;

[0031] The terminal execution module is used to receive the command and control the drone according to the command requirements.

[0032] Compared with the prior art, the technical solution provided by this invention application can bring at least the following beneficial effects: by setting up a neural network obtained by the method of this technical solution, since the neural network obtained by this technical solution has a convolution module, it can more accurately identify feature images. By inputting a digital image processed by the information acquisition and preprocessing module, it can accurately output the command represented by the image and send the command to the UAV control system, thereby realizing the technical effect of controlling the UAV by gesture.

[0033] Preferably, the digital signal acquisition module includes a glove, a flexure sensor mounted on the glove for acquiring operator finger flexion data, an accelerometer mounted on the glove for acquiring operator wrist acceleration data, an angular velocity sensor mounted on the glove for acquiring wrist angular velocity data, and a summarizing unit for collecting the output data from the flexure sensor, the accelerometer, and the angular velocity sensor to form the digital image. Thus, by wearing a data glove on the target object's hand, it is beneficial to acquire the target object's wrist acceleration, wrist angular velocity, and finger stretching with almost no external burden. The data is then transmitted to the digital signal preprocessing module via the summarizing unit, thereby facilitating accurate, efficient, and orderly acquisition of gesture motion features and improving efficiency for subsequent processing.

[0034] Preferably, the bending sensor, the acceleration sensor, and the angular velocity sensor are all equipped with signal transmitting units, and the signal transmitting units are connected to the summarizing unit; this facilitates the automated collection and transmission of information.

[0035] Preferably, the digital signal preprocessing module includes an abnormal data elimination unit arranged sequentially along the propagation direction of the digital image for eliminating abnormal data in the digital image; a threshold unit for setting an angular velocity threshold based on the processing result of the abnormal data elimination unit and marking dynamic and static data based on the angular velocity threshold; a segmentation unit for determining the amount of data acquired by each gesture of the target object based on the processing result of the threshold unit; a noise reduction unit for identifying and eliminating noise; a sampling processing unit for dividing the processing result of the noise reduction unit into upsampling and downsampling regions, performing upsampling processing on the upsampling regions, and performing downsampling processing on the downsampling regions; and a classification unit for dividing the processing result of the sampling processing unit into k blocks according to the physical quantity to which the data belongs. This facilitates the preprocessing of digital signals, thereby improving the recognition efficiency of the neural network. Attached Figure Description

[0036] Figure 1This is a flowchart of a method for obtaining human dynamic gesture information recognition neural network in this invention;

[0037] Figure 2 This is a schematic diagram of the structure of a real-time dynamic gesture control drone system according to the present invention;

[0038] Figure 3 This is a schematic diagram of the neural network structure for dynamic gesture information recognition in this invention. Detailed Implementation

[0039] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0040] This invention provides a method for obtaining human dynamic gesture information recognition neural network, comprising the following steps:

[0041] Step 1: Select n gestures, denoted as gesture x1, gesture x2, ..., gesture x n It is stipulated that each of these n gestures represents an unmanned aerial vehicle (UAV) system operation command, i.e., gesture x i Let i represent the command, where 1 ≤ i ≤ n;

[0042] Step 2: Obtain the operator's data using data gloves. i Second gesture x i Data on finger flexion, wrist acceleration, and wrist angular velocity at the time of each gesture, and the x-value of each gesture. i The finger flexion data, wrist acceleration data, and wrist angular velocity data obtained at that time are used as a digital image, and then these digital images are used as elements to form a pattern with n i The element, gesture x i A set of digital images, where 1 ≤ i ≤ n; The set of digital images for gesture x1, the set of digital images for gesture x2, ..., the set of digital images for gesture x1 are formed sequentially according to the method described in this step. n A collection of digital images;

[0043] Step 3: Move the gesture x i Each digital image in the digital image set is preprocessed, and then the data in each digital image is divided into k blocks according to the physical quantity to which the data in each digital image belongs, thereby obtaining the gesture x. i The input set is given by , where 1 ≤ i ≤ n; The input sets for gesture x1, gesture x2, ..., gesture x are formed sequentially according to the method described in this step. n The input set;

[0044] In this embodiment, we set k equal to 2. When dividing the data into blocks, in each digital image, we divide the finger bending data (i.e., the data obtained by the bending sensor) into one block, and the acceleration data and angular velocity data (i.e., the data obtained by the acceleration sensor and angular velocity sensor on the wrist, and the acceleration sensor and angular velocity sensor are collectively referred to as inertial sensors) into another block. Of course, we can also make more refined divisions.

[0045] Step 4: Construct a neural network with k convolutional networks for acquiring feature images, one fully connected network for summarizing the feature images output by the k convolutional networks and performing information recognition on the output images, and an output module, arranged sequentially along the running direction;

[0046] Step 5: Press with the gesture x i Each element in the input set is the input data, and command i is the ideal output method. The neural network is trained, and after the training is successful, a test neural network is obtained, where 1≤i≤n.

[0047] Step 6: Construct a test set that matches the test neural network obtained in Step 5, and test the test neural network using this test set. After passing the test, the human dynamic gesture information recognition neural network is obtained.

[0048] In this embodiment, the data glove used in step two includes an operating glove equipped with a bending sensor for acquiring operator finger bending information, an acceleration sensor for acquiring operator wrist acceleration information, and an angular velocity sensor for acquiring operator wrist angular velocity information.

[0049] Meanwhile, in order to meet the requirement of broad sampling data, the operators selected in step two include several people of different age groups, such as teenagers, middle-aged and elderly people, as well as athletes and pilots.

[0050] In step three of this embodiment, the gesture x i The preprocessing process for each digital image in the digital image collection includes the following steps:

[0051] Step 3.1, Eliminate gesture x i In a collection of digital images, abnormal data in each digital image (such as a frame of data containing null values ​​or other abnormal values);

[0052] Step 3.2: Obtain the angular velocity threshold through statistical inference. For all frame data in the digital image processed in Step 3.1, if the angular velocity in a certain frame is less than the angular velocity threshold, then this frame data is regarded as static data; if the angular velocity in a certain frame data is not less than the angular velocity threshold, then this frame data is regarded as dynamic data.

[0053] Step 3.3: Obtain the segmentation length h using statistical inference methods. For all frame data in each digital image processed in Step 3.2, if h frames of static data appear consecutively after a certain frame of dynamic data, then this frame of data is taken as a gesture x. i End the data and delete all data following this frame in the digital image;

[0054] Step 3.4: Count the number of data frames contained in each digital image processed in Step 3.3, and set the values ​​α and β according to the statistical results, where α < β. Digital images with a number of data frames less than the value α and greater than the value β are regarded as noise and removed.

[0055] Step 3.5: Using statistical inference methods, select a value γ between the values ​​α and β, that is, let γ = γ + β.

[0056] α<γ<β

[0057] For the digital images processed in step 3.4, the digital images with data frame counts between the values ​​α and γ are upsampled, and the digital images with data frame counts between the values ​​γ and β are downsampled.

[0058] The preferred values ​​provided in this embodiment are α = 20, γ = 40, and β = 80, which are obtained by statistically analyzing the number of frames in each loop-locking digital image.

[0059] This embodiment provides a real-time dynamic gesture control drone system, which includes a digital signal acquisition module, a digital signal preprocessing module, a dynamic gesture information recognition neural network established based on the human dynamic gesture information recognition neural network acquisition method described in this embodiment, and a terminal execution module arranged sequentially along the digital signal propagation direction.

[0060] The digital signal acquisition module is used to acquire finger bending data, wrist acceleration data, and wrist angular velocity data when the target object makes a gesture, and to form a digital image using the finger bending data, wrist acceleration data, and wrist angular velocity data as elements;

[0061] The digital signal preprocessing module is used to preprocess digital images and form the input digital image;

[0062] Dynamic gesture information recognition neural networks are used to recognize input digital images and obtain the commands represented by the input digital images;

[0063] The terminal execution module is used to receive commands and control the drone according to the requirements of those commands.

[0064] In this embodiment, the digital signal acquisition module includes a glove, a bending sensor mounted on the glove for acquiring data on the operator's finger bending, an accelerometer mounted on the glove for acquiring data on the operator's wrist acceleration, an angular velocity sensor mounted on the glove for acquiring data on the wrist angular velocity, and a summarizing unit for collecting the output data from the bending sensor, accelerometer, and angular velocity sensor to form a digital image. Each of the bending sensor, accelerometer, and angular velocity sensor is equipped with a signal transmitting unit, and the signal transmitting unit is interconnected with the summarizing unit.

[0065] The digital signal preprocessing module includes, sequentially arranged along the propagation direction of the digital image signal, an abnormal data removal unit for removing abnormal data in the digital image; a threshold unit for setting an angular velocity threshold based on the processing result of the abnormal data removal unit and marking dynamic and static data according to the set angular velocity threshold; a segmentation unit for determining the amount of data acquired for each target object gesture based on the processing result of the threshold unit; a noise reduction unit for identifying and eliminating noise; a sampling processing unit for dividing the processing result of the noise reduction unit into upsampling and downsampling regions, performing upsampling processing on the upsampling regions, and downsampling processing on the downsampling regions; and a classification unit for dividing the processing result of the sampling processing unit into k blocks according to the physical quantity to which the data belongs. In this embodiment, k=2, and the division method is the same as described above.

[0066] In fact, the abnormal data elimination unit, threshold unit, segmentation unit, noise reduction unit and sampling processing unit in this embodiment are the algorithm program units corresponding to steps 3.1 to 3.5 of this embodiment. The statistical inference methods used in this embodiment include taking the mean and dividing by position, etc. The classification unit is actually a common data segmentation algorithm.

[0067] In summary, although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for acquiring human dynamic gesture information recognition neural network, characterized by: Includes the following steps: Step 1: Select n gestures and denote them as gestures. ,gesture ,...,gesture It is stipulated that each of the n gestures represents a drone system operation command, i.e., a gesture. Represents command i, where, ; Step 2: Obtain operator data through data gloves. The gesture mentioned below Data on finger flexion, wrist acceleration, and wrist angular velocity at the time of the gesture, and the data on each time the gesture is performed. The finger flexion data, wrist acceleration data, and wrist angular velocity data obtained at that time are used as a digital image, and further, the digital image is used as elements to form a... The gesture of each element A collection of digital images; forming the gesture The digital image set, the gesture The digital image set, ..., the gesture A collection of digital images; Step 3: Perform the gesture Each of the digital images in the digital image set is preprocessed, and then the data in each digital image is divided into k blocks according to the physical quantity to which the data in each digital image belongs, thereby obtaining the gesture. The input set; forming the gesture The input set, the gesture The input set, ..., the gesture The input set; wherein, the gesture The preprocessing process for each digital image in the digital image set includes the following steps: Step 3.1: Eliminate the gesture Abnormal data in each of the digital images in the digital image set; Step 3.2: Obtain the angular velocity threshold through statistical inference. For all frame data in the digital image processed in Step 3.1, if the angular velocity in a certain frame is less than the angular velocity threshold, then this frame data is regarded as static data; if the angular velocity in a certain frame data is not less than the angular velocity threshold, then this frame data is regarded as dynamic data. Step 3.3: Obtain the segmentation length h using a statistical inference method. For all frame data in each digital image processed in Step 3.2, if h frames of static data appear consecutively after a certain frame of dynamic data, then this frame of data is taken as one gesture. End the data processing and delete all data in the digital image that follows this frame. Step 4: Construct a neural network along the running direction, which includes k convolutional networks for acquiring feature images, one fully connected network for summarizing the output images of the k convolutional networks and performing information recognition on the output images, and an output module. Step 5: Press the gesture described. Each element in the input set is input data, and the command i is the ideal output mode. The neural network is trained, and after the training is successful, a test neural network is obtained. Step 6: Construct a test set that matches the test neural network, and test the test neural network using the test set. After passing the test, the human dynamic gesture information recognition neural network is obtained.

2. The method for acquiring human dynamic gesture information recognition neural network according to claim 1, characterized in that: The data gloves used in step two include gloves equipped with a flexure sensor for acquiring information about the operator's finger flexion, an acceleration sensor for acquiring information about the operator's wrist acceleration, and an angular velocity sensor for acquiring information about the operator's wrist angular velocity.

3. The method for acquiring human dynamic gesture information recognition neural network according to claim 2, characterized in that: The operators selected in step two include several people of different age groups.

4. The method for acquiring human dynamic gesture information recognition neural network according to claim 3, characterized in that: In step three, the gesture The preprocessing of each digital image in the digital image set also includes the following steps: Step 3.4: Count the number of data frames contained in each digital image processed in Step 3.3, and set the value based on the statistical results. Sum of values ,in The number of frames containing data is less than the stated value. and greater than the stated value The digital image is treated as noise and removed; Step 3.5: Using statistical inference methods, in the numerical values... With the stated value Take a value from the middle , that is to say For the digital image processed in step 3.4, the number of data frames is located at the value mentioned above. With the stated value Upsample the digital images between them to bring the data frame count down to the specified value. With the stated value The digital images are downsampled.

5. A real-time dynamic gesture control unmanned aerial vehicle system, characterized in that: The system includes a digital signal acquisition module, a digital signal preprocessing module, a dynamic gesture information recognition neural network established based on the human dynamic gesture information recognition neural network acquisition method described in any one of claims 1-4, and a terminal execution module, arranged sequentially along the direction of digital signal propagation. The digital signal acquisition module is used to acquire finger bending data, wrist acceleration data, and wrist angular velocity data when the target object makes a gesture, and to form a digital image using the finger bending data, wrist acceleration data, and wrist angular velocity data as elements; The digital signal preprocessing module is used to preprocess the digital image and form an input digital image; The dynamic gesture information recognition neural network is used to recognize the input digital image and obtain the command represented by the input digital image; The terminal execution module is used to receive the command and control the drone according to the command requirements.

6. The real-time dynamic gesture control drone system according to claim 5, characterized in that: The digital signal acquisition module includes a glove, a bending sensor mounted on the glove for acquiring bending data of the operator's fingers, an acceleration sensor mounted on the glove for acquiring acceleration data of the operator's wrist, an angular velocity sensor mounted on the glove for acquiring angular velocity data of the wrist, and a summarizing unit for collecting the output data of the bending sensor, the acceleration sensor, and the angular velocity sensor to form the digital image.

7. The real-time dynamic gesture control drone system according to claim 6, characterized in that: Each of the bending sensor, the acceleration sensor, and the angular velocity sensor is equipped with a signal transmitting unit, and the signal transmitting unit communicates with the summarizing unit.

8. The real-time dynamic gesture control drone system according to claim 7, characterized in that: The digital signal preprocessing module includes an abnormal data elimination unit arranged sequentially along the propagation direction of the digital image for eliminating abnormal data in the digital image; a threshold unit for setting an angular velocity threshold based on the processing result of the abnormal data elimination unit and marking dynamic and static data based on the angular velocity threshold; a segmentation unit for determining the amount of data acquired by each gesture of the target object based on the processing result of the threshold unit; a noise reduction unit for identifying and eliminating noise; a sampling processing unit for dividing the processing result of the noise reduction unit into upsampling and downsampling regions, performing upsampling processing on the upsampling regions, and downsampling processing on the downsampling regions; and a classification unit for dividing the processing result of the sampling processing unit into k blocks according to the physical quantity to which the data belongs.

Images