Multi-channel air stroke dictionary learning method based on improved label consistency

Abstract

The invention relates to a multi-channel air stroke dictionary learning method based on improved label consistency, and belongs to the technical field of human-computer interaction. The method comprises the following steps: collecting and obtaining multi-channel data representing multiple types of stroke actions, performing data preprocessing on the data, constructing a multi-channel air stroke dictionary learning model based on improved label consistency, inputting a training set to train the model, and adjusting model parameters through a test set; and completing experimental evaluation of the multi-channel air stroke dictionary learning recognition model based on improved label consistency by using the optimized recognition model, and outputting the optimal model recognition rate for different strokes. The method has the advantages that the real-time requirement of three-dimensional air stroke recognition can be better met, and the problems of low model robustness and low stroke label recognition rate caused by insufficient feature extraction of a single sensor channel are effectively avoided by increasing the diversity of the collected stroke data features.

Description

Technical Field

[0001] The invention belongs to the technical field of human-computer interaction, and in particular refers to a dictionary learning classification method based on improved label consistency for processing multi-channel data, wherein the multi-channel data includes three-dimensional ultrasonic coordinate data, three-axis acceleration data and quaternion. Background Art

[0002] 3D air gesture writing recognition technology is an emerging human-computer interaction technology that aims to input or control by recognizing the gestures or hand gestures written by the user in the air. Air gesture writing increases the freedom of human-computer interaction and improves the human-computer interaction experience, but at the same time puts forward higher requirements for the accuracy and efficiency of its interaction. Efficient and flexible 3D interaction technology has attracted the attention of many researchers as the main development direction of future human-computer interaction. 3D pen-style interaction is one of the important human-computer interaction methods of terminals. With the increasing maturity of computer technology, 3D air pen-style interaction has also been widely used in various fields, and gesture recognition research has also become a hot research direction at present.

[0003] As people's interest in virtual reality and three-dimensional human-computer interaction continues to increase, different from the traditional two-dimensional user graphical interface that uses input signals such as mouse and keyboard to achieve system control, the human-computer interaction input of three-dimensional virtual reality tends to increase the user's real experience, and the amount of information provided by the user as input mapped to the virtual space is larger, accompanied by more data storage and calculation. At present, users mainly use hands and heads as input sources for mapping. In the process of using user hands as input, gesture recognition is more accurate and in line with human natural writing habits than gesture recognition. Three-dimensional air pen-style interaction has the characteristics of three-dimensional space, natural interaction, and efficient interaction. However, the existing gesture recognition method imposes many restrictions on users during the interaction process and cannot bring into play all the characteristics of three-dimensional pen-style interaction, resulting in a significant reduction in the actual user interaction experience. In addition, current research on pen gesture recognition mainly obtains pen gesture data through a single channel, which is easily affected by the device's own errors and external noise interference, and cannot guarantee its high robustness in the recognition system; for example, the method of using accelerometers, ultrasonic sensors, etc. as a single channel to complete pen gesture recognition has relatively simple feature attributes. When completing three-dimensional space pen gesture movements with multiple attributes, the robustness of this method cannot be guaranteed. The multi-channel pen gesture recognition method combines three-dimensional ultrasonic coordinate data and acceleration data that can describe the trajectory shape characteristics of the pen gesture movement, and adds the quaternion of the pen gesture posture to characterize the pen gesture, which effectively improves the robustness of the system. Summary of the invention

[0004] The present invention provides a multi-channel air gesture dictionary learning method based on improved label consistency to solve the problems of inaccurate feature extraction and slow recognition speed in three-dimensional gesture recognition methods.

[0005] The technical solution adopted by the present invention comprises the following steps:

[0006] Step 1. Collect multiple types of pen gesture data samples by having multiple users hold the electronic pen and complete predetermined pen gesture actions, and then obtain multi-channel data representing the multiple types of pen gesture actions;

[0007] Step 2. Preprocess the multi-channel data corresponding to the multiple types of gestures;

[0008] Step 3. Construct a multi-channel air gesture dictionary learning model based on improved label consistency;

[0009] Step 4. Input the training set to train the model and adjust the model parameters through the test set;

[0010] Step 5. Use the optimized recognition model to complete the experimental evaluation of the multi-channel aerial gesture dictionary learning recognition model based on improved label consistency, and output the model's optimal recognition rate for different gestures.

[0011] The data collection method in step 1 of the present invention is specifically as follows:

[0012] The experimenter holds an electronic pen equipped with a nine-axis inertial sensor and performs prescribed gesture movements. He presses the start button to start recording gesture data and transmits it to the host computer. He presses the end button to complete the collection of a single gesture data sample. The above steps are repeated to complete the collection of the entire gesture data set.

[0013] The data preprocessing method in step 2 of the present invention is specifically as follows:

[0014] Step 2.1: Screening of gesture data samples. Since the collected gesture data has data point redundancy and data pollution problems, as well as insufficient gesture feature information due to too few sampling points, it is necessary to remove duplicate points and remove gesture samples with too short gesture length sequences;

[0015] First, read all the gesture sample sequences in a loop, and assume that the gesture sequence without data preprocessing is D[m]=d1,d2,..,d m , where m is the sequence length and the i-th sampling point is d i Next, determine the i-th sampling point d i and the i+1th sampling point d i+1 Are they equal? ​​If they are equal, delete d i+1 data points, then the stroke sequence after removing duplicate points is D[k]=d1,d2,..,d k, where D[k] represents the stroke sequence after deduplication, k≤m;

[0016] Finally, the relationship between the length K of the gesture sequence after removing duplicate points and the sampling point threshold is determined. If K is greater than the sampling point threshold, the gesture sequence sample is retained; if K is less than the sampling point threshold, the gesture sequence sample is deleted.

[0017] Step 2.2: resampling the gesture data. In view of the problems of inconsistent gesture sample sequence length and excessive gesture sample sequence length, which results in information redundancy, and the need for subsequent dictionary learning and sparse representation, the gestures are resampled using linear interpolation.

[0018] Determine the fixed length of the stroke sequence after sampling, that is, the number of sampling points n. Since the number of sampling points in the original stroke sequence is not equal and is less than k according to step 2.1, the number of sampling points after resampling is n, which is calculated as follows:

[0019]

[0020] Where x represents the current position sequence after the target is resampled, y represents the coordinate value corresponding to the resampled sequence, x1 and x2 are the horizontal coordinates of two adjacent sampling points close to the current sampling point x in the original gesture data sequence, and y2 and y1 are the coordinate values ​​corresponding to x1 and x2 in the original gesture data sequence;

[0021] Step 2.3: Smoothing of gesture data. In order to remove noise interference and smooth the tip burrs of the data, a moving average filter is used to smooth the gesture data. The calculation method is as follows:

[0022]

[0023] where H′ r [n] is the gesture sequence after processing in steps 2.1 and 2.1 before filtering, H b [n] is the data sequence after moving mean filtering. The above formula is used to perform mean filtering on the three-dimensional ultrasonic coordinate data and the three-axis acceleration data in the gesture data. N is the number of sampling points in the moving mean filter.

[0024] Step 2.4: Ultrasonic feature extraction, after step 2.3, H is obtained b The [n] data sequence includes ultrasonic data, acceleration and quaternion data, where the ultrasonic coordinate sequence is S[n] = S1, S2, ..., S i ,…,S n ,1≤i≤n; where s i =(x i ,y i ,z i) is the three-axis ultrasonic coordinate of the three-dimensional gesture at any point, and n is the sequence length;

[0025] Calculate the three-dimensional coordinates S of the center point of the trajectory c =(x c ,y c ,z c ):

[0026]

[0027] Calculate each set of ultrasonic coordinates s in the three-dimensional ultrasonic coordinate sequence S[n] in sequence i =(x i ,y i ,z i ) to the center point S of the trajectory c =(x c ,y c ,z c )’s Euclidean distance:

[0028]

[0029] Get the trajectory center distance encoding sequence d n , d[n]=d1,d2,…,d n ,…,d N ;

[0030] Calculate the total length of the trajectory L, and take the sum of the linear distances between two adjacent points in the stroke sequence as the total length of the stroke trajectory:

[0031]

[0032] The trajectory center code is normalized using the full length of the trajectory to obtain the normalized trajectory center distance code sequence C[n]:

[0033]

[0034] Step 2.5: Normalize the gesture data. Since the amplitude deviations between different gesture data measured by the gesture data acquisition device are usually different, in order to normalize the amplitude of the gesture motion signal, the data processed in the above steps is scaled to the interval [-1, 1] using the normalization method. The calculation method is as follows:

[0035]

[0036] Where i represents the i-th sampling point in the gesture sequence, H n [i] is the stroke sequence after normalization, H[i] is the stroke sequence before normalization, and H max Represents the maximum value in the current stroke sequence, H minRepresents the minimum value in the current gesture sequence.

[0037] Step 3 of the present invention is to construct a multi-channel air gesture dictionary learning recognition model based on improved label consistency, including a gesture feature extraction module, a dictionary learning module, and a classifier parameter output module;

[0038] According to the input, the air gesture data set is preprocessed in step 2. The input gesture data contains eight dimensions, namely, the normalized trajectory center distance encoding sequence C[n] after gesture data preprocessing, the three-axis acceleration data A n =[a x ,a y ,a z ] T , quaternion data Q n =[q1,q2,q3,q4] T ; After data preprocessing, the collected gesture data needs to be divided into training set and test set, and the training set and test set are divided according to the ratio of a:b, where a>b; a,n∈N * Where N * is a natural number;

[0039] The gesture feature extraction module uses PCA feature extraction, which completes the feature extraction of the gesture data in the time dimension;

[0040] Assume that there are c types of strokes, and there are p strokes of each type in the training samples. After the above preprocessing, the size of each stroke feature matrix is ​​N = n × 8. Expand each stroke feature matrix into an N × 1-dimensional column vector. Let M = c × p be the number of samples in the training set. Then the overall distribution matrix of the training set is:

[0041]

[0042] where x i represents the i-th gesture sample in a class of samples in the c-class samples, and u represents the average vector of all gesture features in the training samples, which is calculated as follows:

[0043]

[0044] Construct the training sample matrix X and calculate the average value u of the vector sample matrix;

[0045] Calculate the covariance matrix: C = E[(xu)(xu) T ], where E represents the expectation, calculate the eigenvalue and eigenvector; arrange the eigenvalues ​​from large to small, and select the U corresponding to the first k largest eigenvalues ​​that can meet the information volume i As the change matrix W, the training sample is changed to y = W TX obtains the reduced dimension training sample matrix y, where W T represents the dimension reduction matrix;

[0046] The dictionary learning module adopts an improved label consistent dictionary learning algorithm to learn a dictionary with label features, adds the label consistency constraint of "discriminative sparse coding error" improved by cosine similarity and the "optimal" classification performance standard to the objective function, and uses the K-SVD algorithm to optimize;

[0047] The label consistent dictionary learning algorithm is:

[0048]

[0049] The improved label consistent dictionary learning algorithm model is:

[0050]

[0051] Wherein the parameters α and β represent the weights of discriminative sparse coding errors, Q represents the discriminative sparse coding of input samples, D represents the size of the dictionary, X represents the sparse coefficient, Y represents the total number of training samples, W represents the classifier weight parameter, A represents the linear transformation matrix, and T represents the sparse threshold;

[0052] Based on the above formula, the cosine similarity is used to change the initialization of the discriminative sparse coding Q value, and the dimension reduction matrix is ​​trained simultaneously in the dictionary learning process to better extract the discriminative information in the sample. The specific algorithm is:

[0053]

[0054] Where L is the dimension reduction matrix, which is used to reduce the dimension of the sample, the parameter Y is the training sample, D is the dictionary, and X is the sparse coefficient. Q represents the discriminant sparse coding, A represents the linear transformation matrix, W is the classifier parameter, and T is the sparse threshold. The parameters α, β, and γ represent the weights of each item, control the relative contribution coefficient, and are used to adjust the proportions of the three items in the model.

[0055] Set the dimension of the sample to be reduced and initialize the necessary variables. First, use the PCA algorithm to get the initialized dimension reduction matrix L. Let M = LY. At this time, the sample is M and the total number of categories is C. Then set the size of the dictionary, and then use the KSVD algorithm to train the class dictionary of each category. After obtaining the class dictionary of each category, they are used to form the initial dictionary D. (0) , and then the sparse coefficient X of the sample is obtained using the l1 or l2 norm method according to the dictionary;

[0056] Use the multivariate ridge regression model to get the initial transformation matrix A (0) ,as follows:

[0057]

[0058] The following solution is obtained:

[0059] A (0) =QX T (XX T +λ1I) -1

[0060] Similarly, for the classifier parameter W (0) , again using the multivariate ridge regression model we can obtain the following solution:

[0061] W (0) =HX T (XX T +λ2I) -1

[0062] Where λ1, λ2 are positive numbers, and I is the unit matrix;

[0063] After initializing the parameters, the cosine similarity is used to change the initialization of the discriminative sparse coding Q value based on the dictionary learning algorithm based on label consistency, and the dimension reduction matrix is ​​trained simultaneously during the dictionary learning process to better extract the discriminative information in the sample. The specific algorithm is:

[0064]

[0065]

[0066] Keep the initialized dictionary D (0) , classifier parameters W (0) , the dimension reduction matrix L and other parameters remain unchanged, let M = LY, then the above formula changes to:

[0067]

[0068] When updating the dictionary and sparse coefficients, except for the dictionary D and the sparse coefficient X, other parameters remain unchanged, which can be converted into:

[0069]

[0070] Use the initialization parameters to obtain the updated dictionary model parameters D new and X new , specifically:

[0071]

[0072] Import the above formula into the algorithm formula and convert it into a solution formula:

[0073]

[0074] Obtain the dictionary D, change matrix A and classifier parameters W that meet the classification requirements from the updated dictionary, specifically:

[0075]

[0076] Where d1 is the first column of dictionary D, d n is the nth column; a1 is the first column of the transformation matrix A, a n is the nth column of the transformation matrix A; w1 is the first column of the classifier parameter W, w n is the nth column of classifier parameter W;

[0077] In the classifier parameter output module, the label l corresponding to the largest element in the classifier parameter matrix is ​​the output identification label:

[0078]

[0079] Where W is the classifier parameter matrix, X j is the sparse coefficient.

[0080] The specific operation method of training the model with the input training set and adjusting the model parameters with the test set in step 4 of the present invention is as follows:

[0081] The gesture sequence corresponding to the gestures in the test set is used as input to test the mid-air gesture recognition model. The trained multi-channel mid-air gesture recognition model is used to test the gesture recognition accuracy. The training set is iterated multiple times on the mid-air gesture recognition model by adjusting the hyperparameters, so that the mid-air gesture recognition model performs best on the test set.

[0082] The specific adjustable parameters include the number of sampling points k in the data preprocessing in step 2, the dimension reduction matrix W in step 3 T , the ratio of training set and test set a:b, dictionary atom Z, number of iterations R and assignment parameters α, β and γ, and multiple optimization parameters make the air gesture recognition model perform best on the test set.

[0083] The step 5 of the present invention specifically uses the optimized recognition model obtained by optimizing the parameters in the above step 4 to complete the experimental evaluation of the multi-channel aerial gesture recognition model based on sparse representation, and outputs the optimal recognition rate of the model for different gestures.

[0084] The advantages of the present invention are:

[0085] (1) The present invention establishes a multi-channel mid-air gesture recognition model based on sparse representation. The gesture data of different channels increase the diversity of gesture features and improve the robustness of system recognition.

[0086] (2) The multi-channel mid-air gesture recognition model based on sparse representation established by the present invention uses PCA feature extraction to complete preliminary feature extraction, thereby avoiding the problems of gesture feature data redundancy and excessive processing time.

[0087] (3) The multi-channel mid-air gesture recognition model based on sparse representation established by the present invention uses a dictionary learning method to extract the main features of the gestures and perform classification and recognition, thereby improving the recognition speed while ensuring the recognition rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 is a flow chart of the present invention;

[0089] Figure 2 is a schematic diagram of a host according to an embodiment of the present invention;

[0090] Figure 3 is a schematic diagram of the internal structure of an electronic pen according to an embodiment of the present invention;

[0091] Figure 4 is a schematic diagram of multi-channel air gesture collection according to an embodiment of the present invention;

[0092] Figure 5 is a diagram showing the influence of dictionary scale and sparsity parameters on gesture recognition according to an embodiment of the present invention;

[0093] Figure 6 This is a graph showing the influence of sample dimension on gesture recognition according to an embodiment of the present invention. DETAILED DESCRIPTION

[0094] See also Figure 1 , including the following steps:

[0095] Step 1. Collect multiple types of pen gesture data samples by having multiple users hold the electronic pen and complete predetermined pen gesture actions, and then obtain multi-channel data representing the multiple types of pen gesture actions;

[0096] Step 2. Preprocess the multi-channel data corresponding to the multiple types of gestures;

[0097] Step 3. Construct a multi-channel air gesture dictionary learning model based on improved label consistency;

[0098] Step 4. Input the training set to train the model and adjust the model parameters through the test set;

[0099] Step 5. Use the optimized recognition model to complete the experimental evaluation of the multi-channel aerial gesture dictionary learning recognition model based on improved label consistency, and output the model's optimal recognition rate for different gestures.

[0100] The data collection method in step 1 of the present invention is specifically as follows:

[0101] The experimenter holds an electronic pen equipped with a nine-axis inertial sensor and performs prescribed gesture movements. He presses the start button to start recording gesture data and transmits it to the host computer. He presses the end button to complete the collection of a single gesture data sample. The above steps are repeated to complete the collection of the entire gesture data set.

[0102] The data preprocessing method in step 2 of the present invention is specifically as follows:

[0103] Step 2.1: Screening of gesture data samples. Since the collected gesture data has data point redundancy and data pollution problems, as well as insufficient gesture feature information due to too few sampling points, it is necessary to remove duplicate points and remove gesture samples with too short gesture length sequences;

[0104] First, read all the gesture sample sequences in a loop, and assume that the gesture sequence without data preprocessing is D[m]=d1,d2,..,d m , where m is the sequence length and the i-th sampling point is d i Next, determine the i-th sampling point d i and the i+1th sampling point d i+1 Are they equal? ​​If they are equal, delete d i+1 data points, then the stroke sequence after removing duplicate points is D[k]=d1,d2,..,d k , where D[k] represents the stroke sequence after deduplication, k≤m;

[0105] Finally, the relationship between the length K of the gesture sequence after removing duplicate points and the sampling point threshold is determined. If K is greater than the sampling point threshold, the gesture sequence sample is retained; if K is less than the sampling point threshold, the gesture sequence sample is deleted.

[0106] Step 2.2: resampling the gesture data. In view of the problems of inconsistent gesture sample sequence length and excessive gesture sample sequence length, which results in information redundancy, and the need for subsequent dictionary learning and sparse representation, the gestures are resampled using linear interpolation.

[0107] Determine the fixed length of the stroke sequence after sampling, that is, the number of sampling points n. Since the number of sampling points in the original stroke sequence is not equal and is less than k according to step 2.1, the number of sampling points after resampling is n, which is calculated as follows:

[0108]

[0109] Where x represents the current position sequence after the target is resampled, y represents the coordinate value corresponding to the resampled sequence, x1 and x2 are the horizontal coordinates of two adjacent sampling points close to the current sampling point x in the original gesture data sequence, and y2 and y1 are the coordinate values ​​corresponding to x1 and x2 in the original gesture data sequence;

[0110] Step 2.3: Smoothing of gesture data. In order to remove noise interference and smooth the tip burrs of the data, a moving average filter is used to smooth the gesture data. The calculation method is as follows:

[0111]

[0112] where H′ r [n] is the gesture sequence after processing in steps 2.1 and 2.1 before filtering, H b [n] is the data sequence after moving mean filtering. The above formula is used to perform mean filtering on the three-dimensional ultrasonic coordinate data and the three-axis acceleration data in the gesture data. N is the number of sampling points in the moving mean filter.

[0113] Step 2.4: Ultrasonic feature extraction, after step 2.3, H is obtained b The [n] data sequence includes ultrasonic data, acceleration and quaternion data, where the ultrasonic coordinate sequence is S[n] = S1, S2, ..., S i ,…,S n ,1≤i≤n; where s i =(x i ,y i ,z i ) is the three-axis ultrasonic coordinate of the three-dimensional gesture at any point, and n is the sequence length;

[0114] Calculate the three-dimensional coordinates S of the center point of the trajectory c =(x c ,y c ,z c ):

[0115]

[0116] Calculate each set of ultrasonic coordinates s in the three-dimensional ultrasonic coordinate sequence S[n] in sequence i =(x i ,y i ,z i ) to the center point S of the trajectory c =(x c ,y c ,z c )’s Euclidean distance:

[0117]

[0118] Get the trajectory center distance encoding sequence d n , d[n]=d1,d2,…,d n ,…,d N ;

[0119] Calculate the total length of the trajectory L, and take the sum of the linear distances between two adjacent points in the stroke sequence as the total length of the stroke trajectory:

[0120]

[0121] The trajectory center code is normalized using the full length of the trajectory to obtain the normalized trajectory center distance code sequence C[n]:

[0122]

[0123] Step 2.5: Normalize the gesture data. Since the amplitude deviations between different gesture data measured by the gesture data acquisition device are usually different, in order to normalize the amplitude of the gesture motion signal, the data processed in the above steps is scaled to the interval [-1, 1] using the normalization method. The calculation method is as follows:

[0124]

[0125] Where i represents the i-th sampling point in the gesture sequence, H n [i] is the stroke sequence after normalization, H[i] is the stroke sequence before normalization, and H max Represents the maximum value in the current stroke sequence, H min Represents the minimum value in the current gesture sequence.

[0126] Step 3 of the present invention is to construct a multi-channel air gesture dictionary learning recognition model based on improved label consistency, including a gesture feature extraction module, a dictionary learning module, and a classifier parameter output module;

[0127] According to the input, the air gesture data set is preprocessed in step 2. The input gesture data contains eight dimensions, namely, the normalized trajectory center distance encoding sequence C[n] after gesture data preprocessing, the three-axis acceleration data A n =[a x ,a y ,a z ] T , quaternion data Q n =[q1,q2,q3,q4] T ; After data preprocessing, the collected gesture data needs to be divided into training set and test set, and the training set and test set are divided according to the ratio of a:b, where a>b; a,b∈N * Where N * is a natural number;

[0128] The gesture feature extraction module uses PCA feature extraction, which completes the feature extraction of the gesture data in the time dimension;

[0129] Assume that there are c types of strokes, and there are p strokes of each type in the training samples. After the above preprocessing, the size of each stroke feature matrix is ​​N = n × 8. Expand each stroke feature matrix into an N × 1-dimensional column vector. Let M = c × p be the number of samples in the training set. Then the overall distribution matrix of the training set is:

[0130]

[0131] where x i represents the i-th gesture sample in a class of samples in the c-class samples, and u represents the average vector of all gesture features in the training samples, which is calculated as follows:

[0132]

[0133] Construct the training sample matrix X and calculate the average value u of the vector sample matrix;

[0134] Calculate the covariance matrix: C = E[(xu)(xu) T ], where E represents the expectation, calculate the eigenvalue and eigenvector; arrange the eigenvalues ​​from large to small, and select the U corresponding to the first k largest eigenvalues ​​that can meet the information volume i As the change matrix W, the training sample is changed to y = W T X obtains the reduced dimension training sample matrix y, where W T represents the dimension reduction matrix;

[0135] The dictionary learning module adopts an improved label consistent dictionary learning algorithm to learn a dictionary with label features, adds the label consistency constraint of "discriminative sparse coding error" improved by cosine similarity and the "optimal" classification performance standard to the objective function, and uses the K-SVD algorithm to optimize;

[0136] The label consistent dictionary learning algorithm is generally:

[0137]

[0138] The improved label consistent dictionary learning algorithm model is:

[0139]

[0140] Wherein the parameters α and β represent the weights of discriminative sparse coding errors, Q represents the discriminative sparse coding of input samples, D represents the size of the dictionary, X represents the sparse coefficient, Y represents the total number of training samples, W represents the classifier weight parameter, A represents the linear transformation matrix, and T represents the sparse threshold;

[0141] Based on the above formula, the cosine similarity is used to change the initialization of the discriminative sparse coding Q value, and the dimension reduction matrix is ​​trained simultaneously in the dictionary learning process to better extract the discriminative information in the sample. The specific algorithm is:

[0142]

[0143] Where L is the dimension reduction matrix, which is used to reduce the dimension of the sample, the parameter Y is the training sample, D is the dictionary, and X is the sparse coefficient. Q represents the discriminant sparse coding, A represents the linear transformation matrix, W is the classifier parameter, and T is the sparse threshold. The parameters α, β, and γ represent the weights of each item, control the relative contribution coefficient, and are used to adjust the proportions of the three items in the model.

[0144] Set the dimension of the sample to be reduced and initialize the necessary variables. First, use the PCA algorithm to get the initialized dimension reduction matrix L. Let M = LY. At this time, the sample is M and the total number of categories is C. Then set the size of the dictionary, and then use the KSVD algorithm to train the class dictionary of each category. After obtaining the class dictionary of each category, they are used to form the initial dictionary D. (0) , and then the sparse coefficient X of the sample is obtained using the l1 or l2 norm method according to the dictionary;

[0145] Use the multivariate ridge regression model to get the initial transformation matrix A (0) ,as follows:

[0146]

[0147] The following solution is obtained:

[0148] A (0) =QX T (XX T +λ1I) -1

[0149] Similarly, for the classifier parameter W (0) , again using the multivariate ridge regression model we can obtain the following solution:

[0150] W (0) =HX T (XX T +λ2I) -1

[0151] Where λ1, λ2 are positive numbers, and I is the unit matrix;

[0152] After initializing the parameters, the cosine similarity is used to change the initialization of the discriminative sparse coding Q value based on the dictionary learning algorithm based on label consistency, and the dimension reduction matrix is ​​trained simultaneously during the dictionary learning process to better extract the discriminative information in the sample. The specific algorithm is:

[0153]

[0154] Keep the initialized dictionary D (0) , classifier parameters W (0) , the dimension reduction matrix L and other parameters remain unchanged, let M = LY, then the above formula changes to:

[0155]

[0156] When updating the dictionary and sparse coefficients, except for the dictionary D and the sparse coefficient X, other parameters remain unchanged, which can be converted into:

[0157]

[0158] Use the initialization parameters to obtain the updated dictionary model parameters D new and X new , specifically:

[0159]

[0160] Import the above formula into the algorithm formula and convert it into a solution formula:

[0161]

[0162] Obtain the dictionary D, change matrix A and classifier parameters W that meet the classification requirements from the updated dictionary, specifically:

[0163]

[0164] Where d1 is the first column of dictionary D, d n is the nth column; a1 is the first column of the transformation matrix A, a n is the nth column of the transformation matrix A; w1 is the first column of the classifier parameter W, w n is the nth column of classifier parameter W;

[0165] In the classifier parameter output module, the label l corresponding to the largest element in the classifier parameter matrix is ​​the output identification label:

[0166]

[0167] Where W is the classifier parameter matrix, X j is the sparse coefficient.

[0168] The specific operation method of training the model with the input training set and adjusting the model parameters with the test set in step 4 of the present invention is as follows:

[0169] The gesture sequence corresponding to the gestures in the test set is used as input to test the mid-air gesture recognition model. The trained multi-channel mid-air gesture recognition model is used to test the gesture recognition accuracy. The training set is iterated multiple times on the mid-air gesture recognition model by adjusting the hyperparameters, so that the mid-air gesture recognition model performs best on the test set.

[0170] The specific adjustable parameters include the number of sampling points k in the data preprocessing in step 2, the dimension reduction matrix W in step 3 T , the ratio of training set and test set a:b, dictionary atom Z, number of iterations R and assignment parameters α, β and γ, and multiple optimization parameters make the air gesture recognition model perform best on the test set.

[0171] The step 5 of the present invention specifically uses the optimized recognition model obtained by optimizing the parameters in the above step 4 to complete the experimental evaluation of the multi-channel aerial gesture recognition model based on sparse representation, and outputs the optimal recognition rate of the model for different gestures.

[0172] In order to make the purpose, technical solution and advantages of the present invention more obvious and easy to understand, the following is a detailed description with reference to the accompanying drawings:

[0173] The hardware devices used in this method include a host and an electronic pen, wherein the host is composed of a main controller and six receivers, and the six receivers are responsible for receiving ultrasonic signals and infrared signals emitted by the electronic pen, and calculating the three-dimensional ultrasonic positioning coordinates representing the gesture action. The electronic pen integrates a key module, an ultrasonic drive module, an inertial sensor module, a microprocessor module, a data transmission module, etc., which can complete the extraction of three-axis acceleration data and quaternions representing the gesture action. The ultrasonic coordinate data, three-axis acceleration data and quaternion multi-channel data representing the gesture action described in the present invention are provided by the device, and the schematic diagram of the host and the electronic pen is shown as follows: Figure 2 , Figure 3 shown.

[0174] Figure 4 The figure is a schematic diagram of multi-channel data collection of pen gestures. The experimenter holds the electronic pen equipped with a nine-axis inertial sensor and performs the prescribed pen gestures. He presses the start button to start recording the pen gesture data and transmit it to the host computer. He presses the end button to complete the collection of a single pen gesture data sample. The above steps are repeated to complete the collection of the entire pen gesture data set.

[0175] Figure 5This is a diagram showing the influence of dictionary scale and sparsity parameters on gesture recognition in the multi-channel air gesture dictionary learning method based on improved label consistency. The collected gesture dataset is continuously experimented and adjusted through the learning method to find the best dictionary scale and sparsity parameters to improve the recognition rate of air gestures. Within the specific range shown in the figure, the increase of dictionary scale can significantly improve the recognition rate. However, when the amount of training set data is limited, a large dictionary scale may lead to overfitting, excessive sensitivity to training data, and reduced generalization ability. Therefore, the relatively optimal dictionary scale is selected as 500.

[0176] Figure 6 This is a diagram showing the influence of dimensionality reduction on gesture recognition in the multi-channel air gesture dictionary learning method based on improved label consistency. The figure shows the influence of the number of principal components selected on gesture recognition by mapping the original air gesture data to a low-dimensional space through the PCA linear dimensionality reduction method. It can be seen from the figure that if the dimensionality reduction is too large, such as when the dimensionality reduction is 30 in the figure, important features in the data will be lost, resulting in the model being unable to effectively represent the key features of the input data, affecting the model's learning and generalization capabilities and thus reducing the recognition accuracy. In the range of dimensionality reduction greater than 50 in the figure, the effect of dimensionality reduction on improving the recognition rate does not change much, but if the dimensionality reduction is too small, that is, too many original dimensions are retained, the redundant information of the data may not be effectively removed. It will increase the computational burden, so the optimal dimensionality reduction is 50.

[0177] To further verify the effectiveness of the learning method of the present invention in gesture recognition, the recognition rates of different algorithms for 360 sample gestures are calculated under the conditions that the number of dictionary atoms is 500, the sparsity value is 15, the dimensionality reduction sample value is 50, the iteration parameter is set to 50, and the total number of sample training is 3600. Table 1 shows the comparison of gesture recognition accuracy of the multi-channel air gesture dictionary learning method based on improved label consistency (abbreviated as improved LCKSVD in the table) under the same parameters.

[0178] Table 1 Comparison of gesture recognition rates of different algorithms under the same conditions

[0179]

[0180] From Table 1, it can be seen that the accuracy of the improved LCKSVD method proposed in the present invention is 93.80%, which is higher than that of other methods. Therefore, it can be shown that this method has a better classification effect for multi-channel air gesture recognition technology.

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, rather than to limit it.

Claims

1. A multi-channel air gesture dictionary learning method based on improved label consistency, characterized in that: The following steps are involved: Step 1. Collect multiple types of pen gesture data samples by having multiple users hold the electronic pen and complete predetermined pen gesture actions, and then obtain multi-channel data representing the multiple types of pen gesture actions; Step 2. Preprocess the multi-channel data corresponding to the multiple types of gestures; Step 3. Construct a multi-channel air gesture dictionary learning model based on improved label consistency; Step 4. Input the training set to train the model and adjust the model parameters through the test set; Step 5. Use the optimized recognition model to complete the experimental evaluation of the improved label-consistent multi-channel aerial gesture dictionary learning model, and output the model's optimal recognition rate for different gestures.

2. The multi-channel air gesture dictionary learning method based on improved label consistency according to claim 1, characterized in that: The step 1 comprises: The experimenter holds an electronic pen equipped with a nine-axis inertial sensor and performs prescribed gesture movements. He presses the start button to start recording gesture data and transmits it to the host computer. He presses the end button to complete the collection of a single gesture data sample. Repeat the above steps to complete the collection of the entire gesture data set.

3. The multi-channel air gesture dictionary learning method based on improved label consistency according to claim 1, characterized in that: The method for preprocessing the collected multi-channel data in step 2 is specifically as follows: Step 2.1: Screening of gesture data samples. Since the collected gesture data has data point redundancy and data pollution problems, as well as insufficient gesture feature information due to too few sampling points, it is necessary to remove duplicate points and remove gesture samples with too short gesture length sequences; First, read all the gesture sample sequences in a loop, and assume that the gesture sequence without data preprocessing is D[m]=d1,d2,..,d m , where m is the sequence length and the i-th sampling point is d i Next, determine the i-th sampling point d i and the i+1th sampling point d i+1 Are they equal? ​​If they are equal, delete d i+1 data points, then the stroke sequence after removing duplicate points is D[k]=d1,d2,..,d k , where D[k] represents the stroke sequence after deduplication, k≤m; Finally, the relationship between the length k of the gesture sequence after removing duplicate points and the sampling point threshold is determined. If k is greater than the sampling point threshold, the gesture sequence sample is retained; if k is less than the sampling point threshold, the gesture sequence sample is deleted. Step 2.2: Resampling of gesture data: In view of the problems of inconsistent length of gesture sample sequences and excessive length of gesture sample sequences in the collected gesture data, which results in information redundancy and the need for subsequent dictionary learning and sparse representation, the gestures are resampled using linear interpolation; First, determine the fixed length of the stroke sequence after sampling, that is, the number of sampling points n. Since the number of sampling points in the original stroke sequence is not equal and is less than k according to step 2.1, the number of sampling points after resampling is n, which is calculated as follows: Where x represents the current position sequence after the target is resampled, y represents the coordinate value corresponding to the resampled sequence, x1 and x2 are the horizontal coordinates of two adjacent sampling points close to the current sampling point x in the original gesture data sequence, and y2 and y1 are the coordinate values ​​corresponding to x1 and x2 in the original gesture data sequence; Step 2.3: Smoothing of the gesture data. In order to remove noise and other interference and smooth the tip burrs of the data, a moving average filter is used to smooth the gesture data. The calculation method is as follows: where H′ r [n] is the gesture sequence after processing in steps 2.1 and 2.1 before filtering, H b [n] is the data sequence after moving mean filtering. The above formula is used to perform mean filtering on the three-dimensional ultrasonic coordinate data and the three-axis acceleration data in the gesture data. N is the number of sampling points in the moving mean filter. Step 2.4: Ultrasonic feature extraction, after step 2.3, H is obtained b The [n] data sequence includes ultrasonic data, acceleration and quaternion data, where the ultrasonic coordinate sequence is S[n] = S1, S2, ..., S i ,…,S n ,1≤i≤n; where s i =(x i ,y i ,z i ) is the three-axis ultrasonic coordinate of the three-dimensional gesture at any point, and n is the sequence length; Calculate the three-dimensional coordinates S of the center point of the trajectory c =(x c ,y c ,z c ): Calculate each set of ultrasonic coordinates s in the three-dimensional ultrasonic coordinate sequence S[n] in sequence i =(x i ,y i ,z i ) to the center point S of the trajectory c =(x c ,y c ,z c )’s Euclidean distance: Get the trajectory center distance encoding sequence d n , d[n]=d1,d2,…,d i ,…,d n ; Calculate the total length of the trajectory L, and take the sum of the linear distances between two adjacent points in the stroke sequence as the total length of the stroke trajectory: The trajectory center code is normalized using the full length of the trajectory to obtain the normalized trajectory center distance code sequence C[n]: Step 2.5: Normalize the gesture data. Since the amplitude deviations between different gesture data measured by the gesture data acquisition device are usually different, in order to normalize the amplitude of the gesture motion signal, the data processed in the above steps is scaled to the interval [-1, 1] using the normalization method. The calculation method is as follows: Where i represents the i-th sampling point in the gesture sequence, H n [i] is the stroke sequence after normalization, H[i] is the stroke sequence before normalization, and H max Represents the maximum value in the current stroke sequence, H min Represents the minimum value in the current gesture sequence.

4. The multi-channel air gesture dictionary learning method based on improved label consistency according to claim 1, characterized in that: The step 3 constructs the improved label consistent multi-channel air gesture dictionary learning model as follows: The improved label consistent multi-channel air gesture dictionary learning model includes a gesture feature extraction module, a dictionary learning module, and a classifier parameter output module. The input is an air gesture data set that has completed data preprocessing in step 2. The input gesture data contains eight dimensions, namely, the normalized trajectory center distance encoding sequence C[n] after gesture data preprocessing, the three-axis acceleration data A n =[a x ,a y ,a z ] T , quaternion data Q n =[q1,q2,q3,q4] T ; After data preprocessing, the collected gesture data needs to be divided into training set and test set, and the training set and test set are divided according to the ratio of a:b, where a>b; a,b∈N * Where N * is a natural number; The gesture feature extraction module adopts PCA feature extraction, and this layer completes the feature extraction of the gesture data in the time dimension; Assume that there are c types of strokes, and there are p strokes of each type in the training samples. After the above preprocessing, the size of each stroke feature matrix is ​​N = n × 8. Expand each stroke feature matrix into an N × 1-dimensional column vector. Let M = c × p be the number of samples in the training set. Then the overall distribution matrix of the training set is: where x i represents the i-th gesture sample in a class of samples in the c-class samples, and u represents the average vector of all gesture features in the training samples, which is calculated as follows: Construct the training sample matrix X and calculate the average value u of the vector sample matrix; Calculate the covariance matrix: C = E[(xu)(xu) T ], where E represents the expectation, calculate the eigenvalue and eigenvector; arrange the eigenvalues ​​from large to small, and select the U corresponding to the first k largest eigenvalues ​​that can meet the information volume i As the change matrix W, the training sample is changed to y = W T X obtains the reduced dimension training sample matrix y, where W T represents the dimension reduction matrix; The dictionary learning module adopts an improved label consistent dictionary learning algorithm to learn a dictionary with label features, adds the label consistency constraint of "discriminative sparse coding error" improved by cosine similarity and the "optimal" classification performance standard to the objective function, and uses the K-SVD algorithm to optimize; The improved label consistent dictionary learning algorithm model is: Wherein the parameters α and β represent the weights of discriminative sparse coding errors, Q represents the discriminative sparse coding of input samples, D represents the size of the dictionary, X represents the sparse coefficient, Y represents the total number of training samples, W represents the classifier weight parameter, A represents the linear transformation matrix, and T represents the sparse threshold; Based on the above formula, the cosine similarity is used to change the initialization of the discriminative sparse coding Q value, and the dimension reduction matrix is ​​trained simultaneously in the dictionary learning process to better extract the discriminative information in the sample. The specific algorithm is: Where L is the dimension reduction matrix, which is used to reduce the dimension of the sample, the parameter Y is the training sample, D is the dictionary, and X is the sparse coefficient. Q represents the discriminant sparse coding, A represents the linear transformation matrix, W is the classifier parameter, and T is the sparse threshold. The parameters α, β, and γ represent the weights of each item, control the relative contribution coefficient, and are used to adjust the proportions of the three items in the model. Set the dimension of the sample to be reduced and initialize the necessary variables. First, use the PCA algorithm to get the initialized dimension reduction matrix L. Let M = LY. At this time, the sample is M and the total number of categories is C. Then set the size of the dictionary, and then use the KSVD algorithm to train the class dictionary of each category. After obtaining the class dictionary of each category, they are used to form the initial dictionary D. (0) , and then the sparse coefficient X of the sample is obtained using the l1 or l2 norm method according to the dictionary; Use the multivariate ridge regression model to get the initial transformation matrix A (0) ,as follows: The following solution is obtained: A (0) =QX T (XX T +λ1I) -1 Similarly, for the classifier parameter W (0) , again using the multivariate ridge regression model we can obtain the following solution: W (0) =HX T (XX T +λ2I) -1 Where λ1, λ2 are positive numbers, and I is the unit matrix; After initializing the parameters, the cosine similarity is used to change the initialization of the discriminative sparse coding Q value based on the dictionary learning algorithm based on label consistency, and the dimension reduction matrix is ​​trained simultaneously during the dictionary learning process to better extract the discriminative information in the sample. The specific algorithm is: Keep the initialized dictionary D (0) , classifier parameters W (0) , the dimension reduction matrix L and other parameters remain unchanged, let M = LY, then the above formula changes to: When updating the dictionary and sparse coefficients, except for the dictionary D and the sparse coefficient X, other parameters remain unchanged, which can be converted into: Use the initialization parameters to obtain the updated dictionary model parameters D new and X new , specifically: Import the above formula into the algorithm formula and convert it into a solution formula: Obtain the dictionary D, change matrix A and classifier parameters W that meet the classification requirements from the updated dictionary, specifically: Where d1 is the first column of dictionary D, d n is the nth column; a1 is the first column of the transformation matrix A, a n is the nth column of the transformation matrix A; w1 is the first column of the classifier parameter W, w n is the nth column of classifier parameter W; In the classifier parameter output module, the label l corresponding to the largest element in the classifier parameter matrix is ​​the output identification label: Where W is the classifier parameter matrix, X j is the sparse coefficient.

5. The method for learning a dictionary of mid-air gestures based on improved label consistency multi-channel according to claim 1, characterized in that: In step 4, the specific method of training the model by inputting the training set and adjusting the model parameters by the test set is as follows: The gesture sequence corresponding to the gestures in the test set is used as input to test the mid-air gesture recognition model. The trained multi-channel mid-air gesture recognition model is used to test the gesture recognition accuracy. The training set is iterated multiple times on the mid-air gesture recognition model by adjusting the hyperparameters, so that the mid-air gesture recognition model performs best on the test set. The specific adjustable parameters include the number of sampling points n in the data preprocessing in step 2, the dimension reduction matrix W in step 3 T , the ratio of training set and test set a:b, dictionary atom Z, number of iterations R and assignment parameters α, β and γ, and multiple optimization parameters make the air gesture recognition model perform best on the test set.

6. According to claim 1, a multi-channel air gesture dictionary learning method with improved label consistency is characterized by: The step 5 specifically uses the optimized recognition model obtained by optimizing the parameters in the step 4 to complete the experimental evaluation based on the improved label consistent multi-channel air gesture dictionary learning model, and outputs the optimal recognition rate of the model for different gestures.

Images