Keystroke identity recognition method and system based on millimeter wave radar

Abstract

The invention relates to the technical field of identity recognition and classification, and discloses a keystroke identity recognition method and system based on a millimeter wave radar, and the method comprises the following specific steps: S1, detecting a keystroke behavior of a user through the millimeter wave radar; s2, echo information of hand movement is obtained; s3, analyzing echo information of hand movement to obtain knocking position information; s4, extracting a micro-Doppler time-frequency diagram according to the knocking position information; constructing a training data set about hand movement, palm movement and finger movement; s5, constructing an identity recognition network model based on the micro-Doppler time-frequency diagram, training the model by using the training data set, and obtaining a classification model; and S6, constructing an actual measurement data set, and inputting the actual measurement data set into the trained classification model to recognize the identity of the keyboard tapping user. According to the invention, the problem of user privacy invasion caused by using specific physiological information in the prior art is solved, and the method has the characteristic of low requirement on a detection environment.

Description

Technical Field

[0001] The present invention relates to the technical field of identity recognition and classification, and more specifically, to a keystroke identity recognition method and system based on millimeter wave radar. Background Art

[0002] In the era of modern information expansion and rapid technological development, identity recognition and authentication technology has a profound impact on people's lives, privacy, property and other security, and also plays a pivotal role in corporate assets, confidential documents, social credibility and security. Keystroke information encryption is a common form of primary encryption in life. Due to the emergence of technologies for stealing primary encrypted keystroke behavior information, secondary encryption keystroke identity recognition and authentication technology has attracted much attention.

[0003] The current secondary encryption identity recognition methods for keystroke behavior are mainly divided into two categories, including wireless WIFI detection and sensor recognition detection. On the one hand, wireless WIFI often sets antennas on both sides of the keystroke position or places routers in the corner to capture the user's special CSI change information generated by different users during keystrokes, thereby identifying the user's identity information; on the other hand, sensor recognition detection usually uses piezoelectric sensors, accelerometers and gyroscopes to capture the user's keystroke behavior, and uses the different physiological information accompanied by the user's keystrokes or the user's current different physiological states from the user's keystroke behavior to obtain different features to achieve user identity recognition.

[0004] However, the current keystroke identification methods also have some limitations. For the WIFI detection method, it usually has certain requirements for the information acquisition device, the device placement, and it needs to be used in a relatively stable environment; for the sensor detection method, it requires the user to be in a long-term and stable physiological state, and because the sensor needs to capture and read the user's current specific physiological information, it will cause an infringement of the user's privacy.

[0005] Therefore, how to propose a keystroke identity recognition method based on millimeter-wave radar to address the problems of high requirements for the detection environment and the violation of user privacy caused by using specific physiological information is a technical problem that urgently needs to be solved in this technical field. Summary of the invention

[0006] In order to solve the problem in the prior art that the use of specific physiological information may infringe on user privacy, the present invention provides a keystroke identity recognition method and system based on millimeter wave radar, which has the characteristic of low requirements on the detection environment.

[0007] In order to achieve the above-mentioned purpose of the present invention, the technical scheme adopted is as follows:

[0008] A method for keystroke identification based on millimeter wave radar includes the following specific steps:

[0009] S1. Use millimeter wave radar to detect user keystroke behavior;

[0010] S2, obtaining echo information of hand movement;

[0011] S3, analyzing the echo information of the hand movement to obtain the tapping position information;

[0012] S4. Extracting micro-Doppler time-frequency diagrams based on tapping position information; constructing training data sets for hand motion, palm motion, and finger motion;

[0013] S5. Construct an identity recognition network model based on the micro-Doppler time-frequency graph, train the model using the training data set and obtain a classification model;

[0014] S6. Construct a measured data set, and input the measured data set into a trained classification model to identify the identity of the keyboard tapping user.

[0015] Preferably, in step S1, a millimeter wave radar is used to detect the user's keystroke behavior, and the specific steps are:

[0016] S11, let t d =1, calculate the current t d The energy corresponding to a single chirp

[0017]

[0018] in is a single chirp echo signal, where m=1,...,M;

[0019] S12, let t d =t d +1, calculated

[0020]

[0021] S13. Calculate threshold TH low And the energy change ΔE:

[0022]

[0023] S14: Determine ΔE>TH low If yes, execute S17, if no, execute S15;

[0024] S15. Determine t d >T low , if yes, execute S11, if no, execute S16; T low To set the radar detection low power consumption state time;

[0025] S16, t d =t d +1;

[0026] S17, the millimeter wave radar switches from low power consumption state to normal working state, capturing the hand motion data matrix S t (m, n, ch):

[0027] {S t (m, n, ch)m=1,...,M,n=1,...,N,ch=1,...CH}.

[0028] Furthermore, in step S2, the echo information of the hand movement is obtained, and the specific steps are as follows:

[0029] S21, calculation matrix S R，t (m', n, ch):

[0030]

[0031] Where m'=1, ..., M;

[0032] S22. Calculate the expected noise of static clutter eva，t (1) and static clutter correction value

[0033] Among them, β is the correction coefficient;

[0034] Calculate the matrix S after removing static clutter R _ filte r,t(m',n,ch):

[0035]

[0036] S23, calculate the second dimension time-frequency transformation matrix S RD ,t(m',n',ch):

[0037]

[0038] Where n'=1, ..., N;

[0039] S24, calculate compensation phase and the phase compensation matrix S RD_com，t (m', n', ch):

[0040]

[0041] S25, calculate the distance speed accumulation matrix S RD_sum，t(m', n') and the distance accumulation matrix S R_sum，t (m',n'):

[0042]

[0043] S26, through S RD_com，t (m', n', ch) and S RD_sum，t (m', n') is used to locate the knocking position.

[0044] Furthermore, in step S26, the tapping position is located, and the specific steps are as follows:

[0045] S261, determine whether t=1, if yes, execute step S262, if no, execute step S264;

[0046] S262, calculate peak value RD,t :

[0047] peak RD，t ={x|x∈S RD_sum，t (m', n'), x = max(S RD_sum，t (m',n'))}

[0048] Among them, max is the maximum value operation;

[0049] S263, calculate the distance index ranbin t and speed index dopbin t :

[0050] ranbin t ={x|s x，y ∈S RD_sum，t (m', n'), s x，y =peak RD,t}

[0051] dopbin t ={y|s x，y ∈S RD_sum，t (m', n'), s x，y =peak RD，t}

[0052] Where x∈{1, 2, ...M}, y∈{1, 2, ..., N}, subscripts x and y represent the indexes of two dimensions respectively;

[0053] S264, calculate the third dimension time-frequency transformation matrix S RDCh，t (ch'):

[0054]

[0055] Get the angle index angtmpt:

[0056] angtmp t ={x|s x ∈S RDCh，t ,s x =max(S RDCh,t )}

[0057] Among them, the subscript RDCh indicates that all three dimensions have been transformed in time and frequency;

[0058] S265, calculate the distance and speed in each direction after conversion x,t ,ran y,t ,vel x,t ,vel v,t , and let goal t =[ran x,t ,ran y,t ,vel x，t ,vel y，t ],in:

[0059] ran x,t ,ran y，t ,vel x,t ,vel y,t =transform1(ranbin t , dopbin t ,angtmp t )

[0060] Among them, transform1 is the relational transformation function 1, which is used to convert the index into a physical value. Its formula is as follows:

[0061] Calculate the actual distance range t , actual speed velocity t and the actual angle θ t :

[0062]

[0063] Where c is the speed of light, F s is the sampling frequency, η is the emission chirp slope, λ is the wavelength, T c is the chirp duration, angeva is the estimated angle area, and d is the antenna spacing;

[0064] Calculate the distance and speed information in each direction after conversion x,t ,ran y,t ,vel x，t ,vel y,t , and order

[0065] goal t =[ran x,t ,ran y,t ,vel x，t ,vel y,t ],in:

[0066] ran x,t =-range t,i sin(θ t ), ran y，t =range t，i cos(θ t )

[0067] vel x，t =-velocity t，i sin(θ t ),vel y，t = velocity t，i cos(θ t )

[0068] Among them, sin(·) is the sine function, cos(·) is the cosine function;

[0069] S266, t=t+1

[0070] S267, calculate reference point array S tmp :

[0071] S desc ={s1, s2, ..., s k |s k ∈S RD _ sum，t (m', n'), s1>s2>…>s k}

[0072] S tmp =[s1,s2,…s i |s i ∈S desc ]

[0073] Among them, s1>s2>…>s k , k=1,...,M×N,i=1,2,...,I,i is the reference point index, and the reference point is the candidate point of the tracking target;

[0074] Calculate the reference point index rantmp t，i and doptmp t，i :

[0075] rantmp t，i ={x|s x，y，i ∈S RD_sum，t(m', n'), s x，y，i =S tmp [i]}

[0076] doptmp t，i ={y|s x,y,i ∈S RD_sum，t (m', n'), s x，y，i =S tmp [i]}

[0077] S268. Calculate the covariance matrix Cov Ch，i :

[0078] S DCh，t =S RD_com，t (rantmp t,i , n', ch)

[0079] Cov Ch,i =S DCh，t H S DCh,t

[0080] Calculate the angle index angtmp t，i :

[0081]

[0082] angtmp t，i ={x|s x ∈Φ i ,s x =max(Φ i )}

[0083] Where, the superscript H is the conjugate transpose, dd is the antenna direction vector array, and the subscript x is Φ i Index, Φ i The size is equal to the estimated angular area angeva;

[0084] S269, calculate the distance in each direction after conversion, speed information ran x,t,i ,ran y，t,i ,vel x，t,i ,vel y，t,i , and let goal t,i =[ran x,t,i ,ran y，t,i ,vel x，t,i ,vel y，t，i ]:

[0085] ran x,t，i ,ran y，t，i ,vel x，t，i ,vel y,t，i= transform2(ranbin t,i , dopbin t,i ,angtmp)

[0086] Among them, transform2 is a relational transformation function, which is used to convert the index into a physical value;

[0087] Calculate the actual distance range t,i , actual speed velocity t，i and the actual angle θ t，i :

[0088]

[0089]

[0090] Calculate the distance and speed information in each direction after conversion x ,t,i,rany,t,i,vel x ,t,i,vely,t,i,and let goal t , i = [ran x ,t,i,rany,t,i,vel x ,t,i,vely,t,i]:

[0091] ran x，t,i =-range r,i sin(θ t，i ), ran y,t,i =range t,i cos(θ t，i )

[0092] vel x，t,i =-velocity t，i sin(θ t，i ),vel y，t,i = velocity t，i cos(θ t，i )

[0093] S2610, calculate the distance array Dist and tgt_in:

[0094] Dist = [dist1, ..., dist i |dist i ∈norm(goal t-1 , goal t,i )]

[0095] tgt_in={x|s x ∈Dist,s x=min(Dist)}

[0096] Among them, nor m To find the Euclidean distance operation, min is the minimum value operation;

[0097] Calculate Ranbin t and dopbin t :

[0098] ranbin t ={x i |x i ∈ranbin t，i , i = tgt_in}

[0099] dopbin t ={x i |x i ∈dopbin t,i , i = tgt_in}

[0100] S2611, determine whether t=num_frame, if so, output ranbin t With dopbin t If not, execute S2612; for output ranbin t , dopbin t , with t=1,...,num_frame;

[0101] S2612, t=t+1.

[0102] Furthermore, in step S3, the micro-Doppler time-frequency diagram is extracted according to the tapping position information, and the specific steps are as follows:

[0103] S31, decomposition S R_sum，t (m', n):

[0104]

[0105] in, and is the decomposition coefficient, the subscript wdec represents the time-frequency decomposition, the superscript represents the conjugate, and k is the filter ψ app and ψ det Index, k=1,...,K;

[0106] S32. Construct array C wdec , C move and C strike :

[0107]

[0108] Among them, μ1, μ2∈Z + And μ1<μ2 <L;

[0109] S33, reconstruct palm motion matrix S palm，t (m', n) and the finger motion matrix S finger,t (m', n):

[0110]

[0111] S34. Calculate the palm Doppler matrix Spa l m,t(ranbin t , n'), finger Doppler matrix S finger，t (ranbin t , n') and the hand Doppler matrix S hand,t (ranbin t , n'):

[0112]

[0113] S hand,t (ranbin t , n') = S RD_sum，t (ranbin t , n')

[0114] S35. Constructing Micro-Doppler Map palm , Map finger and Map hand :

[0115]

[0116] in, For splicing operation.

[0117] Furthermore, in step S4, a training data set for hand motion, palm motion and finger motion is constructed, and the specific steps are as follows:

[0118] S41. Obtain identity information label of several users num With Micro Doppler Map palm,num , Map finger,num and Map hand,num , and divide the data into:

[0119] Map palm ={Map palm，num |num=1,...,NUM}

[0120] Map finger ={Map finger,num |num=1,...,NUM}

[0121] Map hand ={Map hand,num |num=1,...,NUM}

[0122] NUM=NUM train +NUM test , NUM train =α×NUM,NUM test =(1-α)×NUM

[0123] Among them, NUM is the total number of user samples, NUM train is the total number of user samples used for training, NUM test

[0124] is the total number of user samples used for actual measurement, α is the ratio coefficient between training and actual measurement;

[0125] S42, determine the status of flag, if flag = 0, execute S43, if flag = 1, execute S49;

[0126] Among them, flag is a pre-set flag bit to control the acquisition of training data or measured data;

[0127] S43, the training micro-Doppler map Set as training data

[0128]

[0129] Among them, the subscript num train Index of training user samples;

[0130] S44. Construct training data set and

[0131]

[0132] Set the training round epoch = 1 and obtain the corresponding category label set

[0133]

[0134] in, is the user identity information corresponding to the sample.

[0135] Furthermore, in step S5, the network model model untrained The specific composition is:

[0136] Module 1: Input dataset D finger , Dhand and D palm , output hand movement features Finger movement characteristics Palm movement characteristics

[0137] Module 2: Input Output hand motion depth enhancement features Finger motion depth enhancement feature Palm motion depth enhancement feature

[0138] Module 3: Input Output joint temporal features F time ;

[0139] Module 4: Input F time , Output fusion feature F mix ,Finally, the fused features are flattened and input into the classifier for classification.

[0140] Furthermore, in step S5, the model is trained using the training data set to obtain a classification model, and the specific steps are as follows:

[0141] S45, and Input model model untrained , get the fusion feature F Train :

[0142]

[0143] S46, F Train To classify:

[0144]

[0145] in, Predict the probability of user identity, Class Train is the set of predicted probabilities for the identity of training user samples, classifier is the classifier, and the input is F Train , the output is Class Train ;

[0146] S47, determine epoch = EPOCH? If so, output the trained classification model model trained , if not, execute S48;

[0147] S48. Let epoch = epoch + 1 and calculate the loss:

[0148]

[0149] Among them, Loss is the loss function, which is Its input is Class Train and Input, g class is the sample label, p class is the sample probability, class = 1, 2, ..., CLASS, CLASS is the maximum total number of user identities;

[0150] S49, the measured micro-Doppler map Set as training data

[0151]

[0152] Among them, num test is the measured sample index.

[0153] Furthermore, in step S6, a measured data set is constructed, and the measured data set is input into a trained classification model to identify the identity of the keyboard tapping user. The specific steps are as follows:

[0154] S51. Constructing the measured data set and

[0155]

[0156] S52, use and Model trainea Test and obtain the feature vector F of the measured data set Test :

[0157]

[0158] S53, input F test Go to classifier and get the user identity information Class Test :

[0159]

[0160] A keystroke identification system based on millimeter wave radar includes a detection module, a data processing module, a data parsing module, a data extraction module, a model training module, and an identification module:

[0161] The detection module is used to detect the user's keystroke behavior using a millimeter wave radar;

[0162] The data processing module is used to obtain echo information of hand movement;

[0163] The data analysis module is used to analyze the echo information of the hand movement to obtain the tapping position information;

[0164] The data extraction module is used to extract the micro-Doppler time-frequency diagram according to the tapping position information; construct a training data set and a measured data set for hand movement, palm movement and finger movement;

[0165] The model training module is used to construct an identity recognition network model based on the micro-Doppler time-frequency diagram, and use the training data set to train the model and obtain a classification model;

[0166] The identity recognition module is used to input the measured data set into the trained classification model to recognize the identity of the keyboard tapping user.

[0167] The beneficial effects of the present invention are as follows:

[0168] The present invention discloses a keystroke identification method based on millimeter wave radar. Compared with other methods, the radar keystroke identification technology adopted by the present invention has the advantages of low environmental dependence and privacy protection. On the one hand, radar detection technology has a low degree of dependence on the user environment. Even if the current user is in a noisy or crowded environment, the radar can capture the keystroke information; on the other hand, the radar detection technology adopted by the present invention does not involve the capture of the user's specific privacy data, so it can protect the user's privacy. Therefore, the present invention solves the problem of the existing technology that the use of specific physiological information will cause user privacy infringement, and has the characteristic of low requirements for the detection environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0169] Figure 1 It is a flow chart of a keystroke identification method based on millimeter wave radar of the present invention.

[0170] Figure 2 Schematic diagram of information flow of millimeter wave radar keystroke detection in an embodiment of the present invention.

[0171] Figure 3 Schematic diagram of information flow of hand motion echo analysis according to an embodiment of the present invention.

[0172] Figure 4 Schematic diagram of information flow of tapping location tracking according to an embodiment of the present invention.

[0173] Figure 5 It is a schematic diagram of information flow of obtaining palm and finger information by time-frequency decomposition according to an embodiment of the present invention.

[0174] Figure 6 It is a schematic diagram of the information flow of constructing a training set and model training in an embodiment of the present invention.

[0175] Figure 7It is a schematic diagram of the information flow of the network model data input and output flow in an embodiment of the present invention.

[0176] Figure 8 It is a schematic diagram of the composition of the network model module of an embodiment of the present invention.

[0177] Fig. 9 It is a schematic diagram of information flow for constructing a measured set and model measurement in an embodiment of the present invention. DETAILED DESCRIPTION

[0178] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0179] Example 1

[0180] like Figure 1 As shown, a keystroke identification method based on millimeter wave radar includes the following specific steps:

[0181] S1. Use millimeter wave radar to detect user keystroke behavior;

[0182] S2, obtaining echo information of hand movement;

[0183] S3, analyzing the echo information of the hand movement to obtain the tapping position information;

[0184] S4. Extracting micro-Doppler time-frequency diagrams based on tapping position information; constructing training data sets for hand motion, palm motion, and finger motion;

[0185] S5. Construct an identity recognition network model based on the micro-Doppler time-frequency graph, train the model using the training data set and obtain a classification model;

[0186] S6. Construct a measured data set, and input the measured data set into a trained classification model to identify the identity of the keyboard tapping user.

[0187] In this embodiment, on the one hand, by first analyzing the user's hand movement echo, then locating and tracking the user's tapping, and finally separating the user's finger movement and palm movement signals, the millimeter wave radar can identify the user in a noisy or crowded environment, reducing the degree of dependence on the user's current environment. On the other hand, the millimeter wave radar identity recognition technology does not involve the capture of the user's specific privacy data, thus protecting the user's privacy.

[0188] In a specific embodiment, the millimeter-wave radar has a low power consumption state and a high power consumption state. In step S1, the millimeter-wave radar in the low power consumption state is first used to detect the user's keystroke behavior. If the user's keystroke behavior is detected, it is switched to a normal working state, and the echo information of the hand movement is obtained in the normal working state.

[0189] Example 2

[0190] More specifically, Figure 2 As shown, in a specific embodiment, in step S1, a millimeter wave radar is used to detect the user's keystroke behavior, and the specific steps are:

[0191] S11, let t d =1, calculate the current t d The energy corresponding to a single chirp

[0192]

[0193] in is a single chirp echo signal, where m=1,...,M;

[0194] S12, let t d =t d +1, calculated

[0195]

[0196] S13. Calculate threshold TH low And the energy change ΔE:

[0197]

[0198] S14. Determine ΔE>TH low If yes, execute S17, if no, execute S15;

[0199] S15. Determine t d >T low , if yes, execute S11, if no, execute S16; T low To set the radar detection low power consumption state time;

[0200] S16, t d =t d +1;

[0201] S17, the millimeter wave radar switches from low power consumption state to normal working state, capturing the hand motion data matrix S t (m, n, ch):

[0202] {S t (m, n, ch)m=1,...,M,n=1,...,N,ch=1,...CH}.

[0203] In this embodiment, M=64, N=128, CH=8, and the captured data matrix S t(m, n, ch), m=1,...,64, n=1,...,128, ch=1,...,8.

[0204] like Figure 3 As shown, in a specific embodiment, in step S2, the echo information of the hand movement is obtained, and the specific steps are:

[0205] S21, calculation matrix S R，t (m', n, ch):

[0206]

[0207] Where m'=1, ..., M;

[0208] In this embodiment, M=64, N=128, CH=8, t=1, ..., 30, there is a hand motion echo S t (m, n, Ch), m = 1, ..., 64, n = 1, ..., 128, ch = 1, ..., 8, obtain S R，t (m',n,ch),m'=1,...,64;

[0209] S22. Calculate the expected noise of static clutter eva,t (1) and static clutter correction value

[0210] Among them, β is the correction coefficient;

[0211] Calculate the matrix S after removing static clutter R _ filter ,t(m',n,ch):

[0212]

[0213] In this embodiment, when β=0.8, S R_filter,t (m', n, ch);

[0214] S23, calculate the second dimension time-frequency transformation matrix S RD，t (m', n', ch):

[0215]

[0216] Where n'=1, ..., N;

[0217] In this embodiment, N=128, and S RD,t (m', n', ch), n'=1,...,128;

[0218] S24, calculate compensation phase and the phase compensation matrix S RD_com，t (m', n', ch):

[0219]

[0220] In this embodiment, For an array of length 128, calculate along the second dimension to obtain S RD_com，t (m', n', ch);

[0221] S25, calculate the distance speed accumulation matrix S RD_sum，t (m', n') and the distance accumulation matrix S R_sum，t (m',n'):

[0222]

[0223] In this embodiment, when CH=8,

[0224] S RD_sum,t (m', n') = S RD_com，t (m', n', 1)+…+S RD_com，t (m', n', 8)

[0225] S R_sum，t (m', n) = S R_filter,t (m', n, 1)+…+S R_filter,t (m', n, 8)

[0226] S26, through S RD_com，t (m', n', ch) and S RD_sum，t (m', n') is used to locate the knocking position.

[0227] like Figure 4 As shown, in a specific embodiment, in the step S26, the tapping position is positioned, and the specific steps are:

[0228] S261, determine whether t=1, if yes, execute step S262, if no, execute step S264;

[0229] S262, calculate peak value RD,t :

[0230] peak RD,t ={x|x∈S RD_sum，t (m', n'), x = max(S RD_suum，t (m',n'))}

[0231] Among them, max is the maximum value operation;

[0232] S263, calculate the distance index ranbin t and speed index dopbin t :

[0233] ranbin t ={x|s x，y ∈S RD_sum，t (m', n'), s x，y =peak RD,t}

[0234] dopbin t ={y|s x，y ∈S RD_sum，t (m', n'), s x，y =peak RD，t}

[0235] Where x∈{1, 2, ...M}, y∈{1, 2, ..., N}, subscripts x and y represent the indexes of two dimensions respectively;

[0236] S264, calculate the third dimension time-frequency transformation matrix S RDCh，t (ch'):

[0237]

[0238] Get the angle index angtmp t :

[0239] angtmp t ={x|s x ∈S RDCh，t ,s x =max(S RDCh,t )}

[0240] Among them, the subscript RDCh indicates that all three dimensions have been transformed in time and frequency;

[0241] S265, calculate the distance and speed in each direction after conversion x，t ,ran y,t ,vel x，t ,vel v，t , and let goal t =[ran x,t ,ran y，t ,vel x，t ,vel y，t ],in:

[0242] ran x,t ,ran y，t ,vel x，t ,vel y，t =transform1(ranbint , dopbin t ,angtmp t )

[0243] Among them, transform1 is the relational transformation function 1, which is used to convert the index into a physical value. Its formula is as follows:

[0244] Calculate the actual distance range t , actual speed velocity t and the actual angle θ t :

[0245]

[0246] Where c is the speed of light, F s is the sampling frequency, η is the emission chirp slope, λ is the wavelength, T c is the chirp duration, angeva is the estimated angle area, and d is the antenna spacing;

[0247] Calculate the distance and speed information in each direction after conversion x，t ,ran y,t ,vel x，t ,vel y，t , and let goal t =[ran x,t ,ran y,t ,vel x，t ,vel y,t ],in:

[0248] ran x,t =-range t，i sin(θ t ), ran y，t =range t，i cos(θ t )

[0249] vel x，t =-velocity t，i sin(θ t ),vel y，t = velocity t，i cos(θ t )

[0250] Among them, sin(·) is the sine function, Cos(·) is the cosine function;

[0251] S266, t=t+1

[0252] S267, calculate reference point array S tmp :

[0253] S desc ={s1, s2, ..., s k |s k ∈S RD_sum，t (m', n'), s1>s2>…>s k}

[0254] S tmp =[s1,s2,...s i |s i ∈S desc ]

[0255] Among them, s1>s2>…> s k, k=1, ..., M×N, i=1, 2, ..., I, i, reference point index, the reference point is the candidate point of the tracking target;

[0256] In this embodiment, when M=64, N=128, M×N=64×128=8192, i=2, s1=9000000, s2=8779369, an array S containing two reference points is obtained. tmp ;

[0257] Calculate the reference point index rantmp t，i and doptmp t，i :

[0258] rantmp t，i ={x|s x，y，i ∈S RD_sum，t (m', n'), s x，y，i =S tmp [i]}

[0259] doptmp t，i ={y|s x，y，i ∈S RD_sum，t (m', n'), s x，y，i =S tmp [i]}

[0260] In this embodiment, s1 corresponds to the index rantmp 30，1 is 7, doptmp 30，1 is 68, s2 corresponds to the index rantmp 30，2 is 8, doptmp 30，2 is 64;

[0261] S268. Calculate the covariance matrix Cov Ch,i :

[0262] S DCh，t =S RD_com，t (rantmpt，i , n', ch)

[0263] Cov Ch，i =S DCh,t H S DCh,t

[0264] Calculate the angle index angtmp t,i :

[0265]

[0266] angtmp t,i ={x|s x ∈Φ i， s x =max(Φ i )}

[0267] Where, the superscript H is the conjugate transpose, dd is the antenna direction vector array, and the subscript x is Φ i Index, Φ i The size is equal to the estimated angular area angeva;

[0268] In this embodiment, the covariance matrix Cov Ch，i is an 8×8 matrix, and calculates Φ for the three reference points i , the estimated angle area number angeva is 120, then Φ i is an array of length 1×120, Calculate angtmp 30，1 =62,angtmp 30，2 =52;

[0269] S269, calculate the distance in each direction after conversion, speed information ran x,t,i ,ran y,t,i ,vel x，t,i ,vel y，t,i , and let goal t，i =[ran x,t,i ,ran y,t,i ,vel x，t,i ,vel y，t，i ]:

[0270] ran x，t,i ,ran y,t,i ,vel x，t,i ,vel y，t，i = transform2(ranbin t，i , dopbin t,i ,angtmp)

[0271] Among them, transform2 is a relational transformation function, which is used to convert the index into a physical value;

[0272] Calculate the actual distance range t,i , actual speed velocity t，i and the actual angle θ t，i :

[0273]

[0274] In this embodiment, when c = 3 × 10 8 , λ=0.0039, d=0.00195, F s =2×10 6 , η=1×10 1414 When s0，1 =0°, range 30，1 =0.328, velocity 30，1 =0.571;θ 30，2 =-8.63°range 30，2 =0.375, velocity 30，2 =-0.190;

[0275] Calculate the distance and speed information in each direction after conversion x,t,i ,ran y，t,i ,vel x，t,i ,vel y，t，i , and let goal t，i =[ran x,t,i ,ran y,t,i ,vel x,t,i ,vel y,t,i ]:

[0276] ran x,t,i =-range t,i sin(θ t，i ), ran y,t,i =range r,i cos(θ t，i )

[0277] vel x,t,i =-velocity t，i sin(θ t，i ),vel y,t,i = velocity t，i cos(θ t，i )

[0278] In this embodiment, goal 30，1 =[0, 0.328, 0, 0.571], goal30，2 = [0.056, 0.371, -0.029, -0.188];

[0279] S2610, calculate the distance array Dist and tgt_in:

[0280] Dist = [dist1, ..., dist i |dist i ∈norm(goal t-1 , goal t，i )]

[0281] tgt_in={x|s x ∈Dist,s x =min(Dist)}

[0282] Among them, norm is the Euclidean distance operation, and min is the minimum value operation;

[0283] In this embodiment, when goal 29 =[0, 0.328, 0.011, 0.462], then Dist=[0.110, 0.655], tgt_in=1;

[0284] Calculate Ranbin t and dopbin t :

[0285] ranbin t ={x i |x i ∈ranbin t,i , i = tgt_in}

[0286] dopbin t ={x i |x i ∈dopbin t，i , i = tgt_in}

[0287] In this embodiment, ranbin t =7, dopbin t =68;

[0288] S2611, determine whether t=num_frame, if so, output ranbin t With dopbin t If not, execute S2612; for output ranbin t , dopbin t , with t=1,...,num_frame;

[0289] In this embodiment, when num_frame=30, t=num_frame, the output is ranbin t dopbin t , where t=1, ..., 30;

[0290] S2612, t=t+1.

[0291] like Figure 5 As shown, in a specific embodiment, in step S3, the micro-Doppler time-frequency diagram is extracted according to the tapping position information, and the specific steps are:

[0292] S31, decomposition S R_sum，t (m', n):

[0293]

[0294] in, and is the decomposition coefficient, the subscript wdec represents the time-frequency decomposition, the superscript represents the conjugate, and k is the filter ψ app and ψ aet Index, k=1,...,K;

[0295] In this embodiment, when K=128, L=5, t=1, ..., 30, the decomposition coefficients are obtained.

[0296] S32. Construct array C wdec , C move and C strike :

[0297]

[0298] Among them, μ1, μ2∈Z + And μ1<μ2<L;

[0299] S33, reconstruct palm motion matrix S palm,t (m', n) and the finger motion matrix S finger，t (m', n):

[0300]

[0301] In this embodiment, S is calculated finger,t (m', n) and S palm，t (m', n);

[0302] S34, calculate the palm Doppler matrix S palm，t (ranbin t , n'), finger Doppler matrix S finger，t(ranbin t , n') and the hand Doppler matrix S hand,t (ranbin t , n'):

[0303]

[0304] S hand，t (ranbin t , n') = S RD_sum，t (ranbin t , n')

[0305] In this embodiment, when ranbin t =7, t=1,...,30, the matrix S is calculated palm,t (ranbin t , n'), S hand，t (ranbin t , n') and S hand,t (ranbin t , n');

[0306] S35. Constructing Micro-Doppler Map palm , Map finger and Map hand :

[0307]

[0308] in, For splicing operation.

[0309] In this embodiment, when num_frame=30,

[0310]

[0311] like Figure 6 As shown, in a specific embodiment, in the steps S4 and S5, training data on hand movement, palm movement and finger movement are constructed; an identity recognition network model is constructed based on the micro-Doppler time-frequency diagram, and the model is trained using the training data set to obtain a classification model. The specific steps are:

[0312] S41. Obtain identity information label of several users num With Micro Doppler Map palm,num , Map finger，num and Map hand,num , and divide the data into:

[0313] Map palm ={Map palm,num|num=1,...,NUM}

[0314] Map finer ={Map finger,num |num=1,...,NUM}

[0315] Map hand ={Map hand,num |num=1,...,NUM}

[0316] NUM=NUM train +NUM test , NUM train =α×NUM,NUM test =(1-a)×NUM

[0317] Among them, NUM is the total number of user samples, NUM train is the total number of user samples used for training, NUM test

[0318] is the total number of user samples used for actual measurement, α is the ratio coefficient between training and actual measurement;

[0319] In this embodiment, when NUM=1000, α=0.8, NUM train =800, NUM test =200, get user identity information label num , Micro Doppler Map palm,num , Map finger,num , Map hand,num ;

[0320] S42, determine the status of flag, if flag = 0, execute S43, if flag = 1, execute S49;

[0321] Among them, flag is a pre-set flag bit to control the acquisition of training data or measured data;

[0322] In this embodiment, when flag=1, S43 is executed;

[0323] S43, the training micro-Doppler map Set as training data

[0324]

[0325] Among them, the subscript num train Index of training user samples;

[0326] In this embodiment, the training data is obtained

[0327] S44. Construct training data set and

[0328]

[0329] In this embodiment, when NUM=1000, the training data set is obtained. and

[0330] Set the training round epoch = 1 and obtain the corresponding category label set

[0331]

[0332] in, The user identity information corresponding to the sample;

[0333] In this embodiment, when NUM=1000, we get

[0334] S45, and Input model model untrained , get the fusion feature F Train :

[0335]

[0336] In this embodiment, input and To model untrained , get F Train ;

[0337] S46, F Train To classify:

[0338]

[0339] in, Predict the probability of user identity, Class Train is the set of predicted probabilities for the identity of training user samples, classifier is the classifier, and the input is F Train , the output is Class Train ;

[0340] In this embodiment, when NUM=1000, the identity prediction probability set Class of 1000 user samples is output. Train ;

[0341] S47, determine epoch = EPOCH? If so, output the trained network model modeltrained , if not, execute S48;

[0342] In this embodiment, when EPOCH=100, epoch=EPOCH, the output model trained ;

[0343] S48. Let epoch = epoch + 1 and calculate the loss:

[0344]

[0345] Among them, Loss is the loss function, which is Its input is Class Train and Input, g class is the sample label, p class is the sample probability, class = 1, 2, ..., CLASS, CLASS is the maximum total number of user identities;

[0346] S49, the measured micro-Doppler map Set as training data

[0347]

[0348] Among them, num test is the measured sample index.

[0349] like Figure 7 , as shown in 8, in a specific embodiment, in the step S5, the network model model untrained The specific composition is:

[0350] Module 1: Input dataset D finger , D hand and D palm , output hand movement features Finger movement characteristics Palm movement characteristics Thus, the input time-frequency graph representing different motion information is extracted to obtain multi-depth and deep spatial features of user tapping;

[0351] Module 2: Input Output hand motion depth enhancement features Finger motion depth enhancement feature Palm motion depth enhancement feature This enriches the depth information, highlights the depth information with higher contribution, and expands the difference in feature expression between users;

[0352] Module 3: Input Output joint temporal features F time , combined with the depth enhancement features, the joint temporal features of the joint finger motion and hand motion are obtained, thereby obtaining the joint representation of the user's tapping spatial features and temporal features;

[0353] Module 4: Input F time , Output fusion feature F mix , thereby combining the feature time series feature and the hand motion enhancement feature to obtain the fused feature, and finally flattening the fused feature and inputting it into the classifier for classification.

[0354] like Fig. 9 As shown, in a specific embodiment, in step S6, a measured data set is constructed, and the measured data set is input into a trained classification model to identify the identity of the keyboard tapping user. The specific steps are:

[0355] S51. Constructing the measured data set and

[0356]

[0357] In this embodiment, when NUM test =400, the measured set is composed of 400 measured user samples. and

[0358] S52, use and Model trained Test and obtain the feature vector F of the measured data set Test :

[0359]

[0360] In this embodiment, when NUM test =400, the feature vector F of 400 samples is obtained Test ;

[0361] S53, input F test Go to classifier and get the user identity information Class Test :

[0362]

[0363] In this embodiment, when NUM test =400, the user identity information Class of 400 samples is obtained Test ;

[0364] In summary, the present invention discloses a keystroke identity recognition method based on millimeter-wave radar. Compared with other methods, the radar keystroke identity recognition technology adopted by the present invention has the advantages of low environmental dependence and privacy protection. On the one hand, radar detection technology has a low degree of dependence on the user environment. Even if the current user is in a noisy or crowded environment, the radar can capture the keystroke information; on the other hand, the radar detection technology adopted by the present invention does not involve the capture of the user's specific privacy data, so it can protect the user's privacy. Therefore, the present invention solves the problem that the use of specific physiological information in the prior art will cause user privacy infringement, and has the characteristic of low requirements for the detection environment.

[0365] Example 3

[0366] A keystroke identification system based on millimeter wave radar includes a detection module, a data processing module, a data parsing module, a data extraction module, a model training module, and an identification module:

[0367] The detection module is used to detect the user's keystroke behavior using a millimeter wave radar;

[0368] The data processing module is used to obtain echo information of hand movement;

[0369] The data analysis module is used to analyze the echo information of the hand movement to obtain the tapping position information;

[0370] The data extraction module is used to extract the micro-Doppler time-frequency diagram according to the tapping position information; construct a training data set and a measured data set for hand movement, palm movement and finger movement;

[0371] The model training module is used to construct an identity recognition network model based on the micro-Doppler time-frequency diagram, and use the training data set to train the model and obtain a classification model;

[0372] The identity recognition module is used to input the measured data set into the trained classification model to recognize the identity of the keyboard tapping user.

[0373] Obviously, the above embodiments of the present invention are only examples for clearly explaining the present invention, and are not intended to limit the implementation methods of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A keystroke identification method based on millimeter wave radar, characterized in that: The specific steps include: S1. Use millimeter wave radar to detect user keystroke behavior; S2, obtaining echo information of hand movement; S3, analyzing the echo information of the hand movement to obtain the tapping position information; S4. Extracting micro-Doppler time-frequency diagrams based on tapping position information; constructing training data sets for hand motion, palm motion, and finger motion; S5. Construct an identity recognition network model based on the micro-Doppler time-frequency graph, train the model using the training data set and obtain a classification model; S6. Construct a measured data set, and input the measured data set into a trained classification model to identify the identity of the keyboard tapping user.

2. The method for keystroke identification based on millimeter wave radar according to claim 1, characterized in that: In the step S1, the millimeter wave radar is used to detect the user's keystroke behavior, and the specific steps are as follows: S11, let t d =1, calculate the current t d The energy corresponding to a single chirp in is a single chirp echo signal, where m=1,...,M; S12, let t d =t d +1, calculated S13. Calculate threshold TH low And the energy change ΔE: S14: Determine ΔE>TH low If yes, execute S17, if no, execute S15; S15. Determine t d >T low , if yes, execute S11, if no, execute S16; T low To set the radar detection low power consumption state time; S16、t d =t d +1; S17, the millimeter wave radar switches from low power consumption state to normal working state, capturing the hand motion data matrix S t (m, n, ch): {S t (m,n,ch)|m=1,...,M,n=1,...,N,ch=1,...CH}。 3. The method for keystroke identification based on millimeter wave radar according to claim 2, characterized in that: In step S2, echo information of hand movement is obtained, and the specific steps are as follows: S21, calculation matrix S R，t (m', n, ch): Where m'=1, ..., M; S22. Calculate the expected noise of static clutter eva，t (1) and static clutter correction value Among them, β is the correction coefficient; Calculate the matrix S after removing static clutter R_filter，t (m', n, ch): S23, calculate the second dimension time-frequency transformation matrix S RD，t (m', n, ch): Where n'=1, ..., N; S24, calculate compensation phase and the phase compensation matrix S RD_com，t (m', n', ch): S25, calculate the distance speed accumulation matrix S RD_sum，t (m', n') and the distance accumulation matrix S R_sum，t (m',n'): S26, through S RD_com，t (m', n', ch) and S RD_sum，t (m', n') is used to locate the knocking position.

4. The method for keystroke identification based on millimeter wave radar according to claim 3, characterized in that: In step S26, the tapping position is positioned, and the specific steps are as follows: S261, determine whether t=1, if yes, execute step S262, if no, execute step S264; S262, calculate peak value RD，t : peak RD，t ={x|x∈S RD_sum，t (m’,n’),x=max(S RD_sum，t (m’,n’))} Among them, max is the maximum value operation; S263, calculate the distance index ranbin t and speed index dopbin t : ranbin t ={x|s x，y ∈S RD_sum，t (m’,n’),s x，y =peak RD，t } dopbin t ={y|s x，y ∈S RD_sum，t (m’,n’),s x，y =peak RD,t } Where x∈{1, 2, ...M}, y∈{1, 2, ..., N}, subscripts x and y represent the indexes of two dimensions respectively; S264, calculate the third dimension time-frequency transformation matrix S RDCh，t (ch'): Get the angle index angtmp t : angtmp t ={x|s x ∈S RDCh，t ,s x =max(S RDCh，t )} Among them, the subscript RDCh indicates that all three dimensions have been transformed in time and frequency; S265, calculate the distance and speed in each direction after conversion x，t ,ran y，t ,vel x，t ,vel y，t , and let goal t =[ran x，t ,ran y,t ,vel x，t ,vel y，i ]: robbery x，t ,robbery y,t ,well x，t ,well y，t =transform1(ranbin t ,dopbin t ,angtmp t ) Among them, transform1 is the relational transformation function 1, which is used to convert the index into a physical value. Its formula is as follows: Calculate the actual distance range t , actual speed velocity t and the actual angle θ t : Where c is the speed of light, F s is the sampling frequency, η is the emission chirp slope, λ is the wavelength, T c is the chirp duration, angeva is the estimated angle area, and d is the antenna spacing; Calculate the distance and speed information in each direction after conversion x，t ,ran y，t ,vel x，t ,vel y，t , and let goal t =[ran x，t ,ran y，t ,vel x，t ,vel y，t ],in: run x，t =-range t，i sin(θ t ),ran y,t =range t，i cos(θ t ) vel x，t =-velocity t，i sin(θ t ),vel y，t =velocity t，i cos(θ t ) Among them, sin(·) is the sine function, cos(·) is the cosine function; S266, t=t+1 S267, calculate reference point array S tmp : S desc ={s1,s2,…,s k |s k ∈S RD_sum，t (m’,n’),s1>s2>…>s k } S tmp =[s1,s2,…s i |s i ∈S desc ] Among them, s1>s2>...>s k , k=1,...,M×N,i=1,2,...,I,i is the reference point index, and the reference point is the candidate point of the tracking target; Calculate the reference point index rantmp t，i and doptmp t，i : rantmp t，i ={x|s x，y，i ∈S RD_sum,t (m’,n’),s x，y,i =S tmp [i]} doptmp t，i ={y|s x,y，i ∈S RD_sum,t (m’,n’),s x，y，i =S tmp [i]|} S268. Calculate the covariance matrix Cov Ch，i : S DCh，t =S RD_com，t (rantmp t，i ,n’,ch) Number Ch，i =S DCh，t H S DCh，t Calculate the angle index angtmp t，i : angtmp t，i ={x|s x ∈Φ i ,s x =max(Φ i )} Where, the superscript H is the conjugate transpose, dd is the antenna direction vector array, and the subscript x is Φ i Index, Φ i The size is equal to the estimated angular area angeva; S269, calculate the distance in each direction after conversion, speed information ran x，t，i ,ran y，t，i ,vel x，t，i ,vel y，t，i , and let goal t，i =[ran x，t，i ,ran y，t，i ,vel x，t，i ,vel y，t，i ]: robbery x，t，i ,robbery y，t，i ,well x，t，i ,well y，t，i =transform2(ranbin t，i ,dopbin t，i ,angtmp) Among them, transform2 is a relational transformation function, which is used to convert the index into a physical value; calculate the actual distance range t，i , actual speed velocity t，i and the actual angle θ t，i : Calculate the distance and speed information in each direction after conversion x，t，i ,ran y，t，i ,vel x，t，i ,vel y，t，i , and let goal t，i =[ran x，t，i ,ran y，t，i ,vel x，t，i ,vel y，t，i ]: run x，t，i =-range t，i sin(θ t，i ),ran y，t，i =range t，i cos(θ t，i ) vel x，t，i =-velocity t，i sin(θ t，i ),vel y，t，i =velocity t，i cos(θ t，i ) S2610, calculate the distance array Dist and tgt_in: Dist=[dist1,..,dist i |dist i ∈norm(goal t-1 ,goal t，i )] tgt_in={x|s x ∈Dist,s x =min(Dist)} Among them, norm is the Euclidean distance operation, and min is the minimum value operation; Calculate Ranbin t and dopbin t : ranbin t ={x i |x i ∈ranbin t，i ,i=tgt_in} dopbin t ={x i ||x i ∈dopbin t，i ,i=tgt_in} S2611, determine whether t=num_frame, if so, output ranbin t With dopbin t If not, execute S2612; for output ranbin t , dopbin t , with t=1,...,num_frame; S2612, t=t+1.

5. The method for keystroke identification based on millimeter wave radar according to claim 4, characterized in that: In step S3, the micro-Doppler time-frequency diagram is extracted according to the tapping position information, and the specific steps are as follows: S31, decomposition S R_sum，t (m', n): in, and is the decomposition coefficient, the subscript wdec represents the time-frequency decomposition, the superscript represents the conjugate, and k is the filter ψ app and ψ det Index, k=1,...,K; S32. Construct array C wdec , C move and C strike : Among them, μ1, μ2∈Z + And μ1<μ2<L; S33, reconstruct palm motion matrix S palm，t (m', n) and the finger motion matrix S finger，t (m', n): S34, calculate the palm Doppler matrix S palm，t (ranbin t , n'), finger Doppler matrix S finger，t (ranbin t , n') and the hand Doppler matrix S hand，t (ranbin t , n'): S hand，t (ranbint,n’)=S RD_sum，t (ranbin t ,n’) S35. Constructing Micro-Doppler Map palm , Map finger and Map hand : in, For splicing operation.

6. The method for keystroke identification based on millimeter wave radar according to claim 5, characterized in that: In step S4, a training data set for hand motion, palm motion and finger motion is constructed, and the specific steps are as follows: S41. Obtain identity information label of several users num With Micro Doppler Map palm，num , Map finger，num and Map hand，num , and divide the data into: Map palm ={Map palm，num |num=1,...,NUM} Map finger ={Map jinger，num |nnum=1,...,NUM} Map hand ={Map hand,num |num=1,..,NUM} NUMBER=NUMBER train +NUMBER test ,NUM train =α×NUM,NUM test =(1-α)×NUM Among them, NUM is the total number of user samples, NUM train is the total number of user samples used for training, NUM test is the total number of user samples used for actual measurement, α is the ratio coefficient between training and actual measurement; S42, determine the status of flag, if flag = 0, execute S43, if flag = 1, execute S49; Among them, flag is a pre-set flag bit to control the acquisition of training data or measured data; S43, the training micro-Doppler map Set as training data Among them, the subscript num train Index of training user samples; S44. Construct training data set and Set the training round epoch = 1 and obtain the corresponding category label set in, is the user identity information corresponding to the sample.

7. The method for keystroke identification based on millimeter wave radar according to claim 6, characterized in that: In step S5, the network model model untrained The specific composition is: Module 1: Input dataset D finger , D hand and D palm , output hand movement features Finger movement characteristics Palm movement characteristics Module 2: Input Output hand motion depth enhancement features Finger motion depth enhancement feature Palm motion depth enhancement feature Module 3: Input Output joint temporal features F time ; Module 4: Input F time , Output fusion feature F mix ,Finally, the fused features are flattened and input into the classifier for classification.

8. The method for keystroke identification based on millimeter wave radar according to claim 7, characterized in that: In step S5, the model is trained using the training data set to obtain a classification model, and the specific steps are as follows: S45, and Input model model untrained , get the fusion feature F Train : S46, F Train To classify: in, Predict the probability of user identity, Class Train is the set of predicted probabilities for the identity of training user samples, classifier is the classifier, and the input is F Train , the output is Class Train ; S47, determine epoch = EPOCH? If so, output the trained classification model model trained If not, execute S48; S48. Let epoch = epoch + 1 and calculate the loss: Among them, Loss is the loss function, which is Its input is Class Train and Input, g class is the sample label, p class is the sample probability, class = 1, 2, ..., CLASS, CLASS is the maximum total number of user identities; S49, the measured micro-Doppler map Set as training data Among them, num test is the measured sample index.

9. The method for keystroke identification based on millimeter wave radar according to claim 8, characterized in that: In step S6, a measured data set is constructed, and the measured data set is input into a trained classification model to identify the identity of the keyboard tapping user. The specific steps are: S51. Constructing the measured data set and S52, use and Model trained Test and obtain the feature vector F of the measured data set Test : S53, input F test Go to classifier and get the user identity information Class Test :

10. A keystroke identification system based on millimeter wave radar, characterized in that: Including detection module, data processing module, data analysis module, data extraction module, model training module, and identity recognition module: The detection module is used to detect the user's keystroke behavior using a millimeter wave radar; The data processing module is used to obtain echo information of hand movement; The data analysis module is used to analyze the echo information of the hand movement to obtain the tapping position information; The data extraction module is used to extract the micro-Doppler time-frequency diagram according to the tapping position information; construct a training data set and a measured data set for hand movement, palm movement and finger movement; The model training module is used to construct an identity recognition network model based on the micro-Doppler time-frequency diagram, and use the training data set to train the model and obtain a classification model; The identity recognition module is used to input the measured data set into the trained classification model to recognize the identity of the keyboard tapping user.

Images