A password strength measurement method based on reinforcement learning

Abstract

The application discloses a password strength measurement method based on reinforcement learning, which comprises the following steps: firstly, constructing a password data set, then establishing a reinforcement learning model and a scoring model, training the reinforcement learning model and the scoring model, and obtaining an optimal reinforcement learning model; after the deployment is completed, a user inputs a password into the optimal reinforcement learning model, and outputs a probability; then the probability is compared with a password probability dictionary through a Monte Carlo calculation algorithm to obtain a guessing number of the password; and finally, according to the corresponding relationship between the guessing number of the password and the password strength, the password strength of the password is obtained. The password modeling method based on reinforcement learning regards the process of generating a password by a user as a Markov decision process, calculates the reward brought by each action, and can better model a password sequence; the application can resist online directional guessing attacks and offline wandering guessing attacks at the same time, and is more accurate and more robust.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of information security, and relates to the field of password security authentication, and specifically relates to a password strength measurement method based on reinforcement learning. BACKGROUND

[0002] Nowadays, user password authentication is still the main means of current identity authentication. With the current surge of online services, the number of website accounts that each user needs to manage is increasing, but users can usually only remember 5-7 different passwords. At the same time, it is inevitable that users usually tend to choose popular strings, use personally identifiable information (PII), and are vulnerable to side channel attacks (such as shoulder surfing attacks, key tone feedback, etc.). Therefore, it is very important to measure the security of user passwords in real environments.

[0003] In practice, it is found that the guessing number is a good indicator for evaluating the strength of user passwords. Passwords that can be easily guessed by an adversary are considered weak passwords, while passwords that need a large guessing number to be guessed are considered secure passwords. In order to ensure the security of the system and the user, mainstream Internet service providers will use a password strength meter (PSM) to feedback the strength of the password to the user when the user registers or updates the password. Only a PSM that can accurately feedback the strength of the password can effectively improve the security of the user password, thereby protecting the security of the user account. Incorrectly identifying strong passwords as weak passwords can cause usability problems, while incorrectly identifying weak passwords as strong passwords can pose a security risk and cannot ensure security. In addition, the current mainstream PSM often ignores the use of user personal information by users when setting passwords, but in fact the proportion of user personal information introduced by users when creating passwords is quite high (12.8% to 51.4%). This phenomenon prompts websites and service providers to take effective measures to detect the use of user personal information by users, guide users to avoid using user personal information and some easily guessed passwords when creating passwords, in order to reduce potential security risks.

[0004] The document "Weir, M., Aggarwal, S., De Medeiros, B., & Glodek, B. Password cracking using probabilistic context-free grammars. In Proc. IEEE Symposium on Security and Privacy, pages 391-405." is an unsupervised learning represented by PCFG, and the document "W. Melicher, B. Ur, S. Komanduri, L. Bauer, N. Christin, and L. F. Cranor. Fast, lean and accurate: Modeling password guessability using neural networks. In Proc. USENIX SEC 2016, pages 175-191." is a supervised learning model represented by FLA. Both methods rely on large-scale password data sets, and the problems are: (1) PCFG needs a large-scale training set to fully learn the password division template and template content, and the existing supervised learning needs to learn password features from large-scale data labels. (2) The supervised learning model can only learn through popular password data labels, and non-popular password labels (such as passwor followed by D instead of the common d) are usually ignored by the learning model. In this way, existing guess models can usually only imitate the password generation patterns already existing in the training set. For the unsupervised learning represented by PCFG, it can only generate password templates and related context-free content that appear in the training set. For supervised learning, the training method is character-level autoregression, which can only generate existing paths and lacks diversity. For example, if the training data does not contain or only contains a small amount of a specific password pattern (such as passworD), the existing guess model based on autoregressive learning may not be able to generate passwords with the corresponding pattern, so it cannot model the password strength well. SUMMARY

[0005] In view of the deficiencies of the prior art, the technical problem to be solved by the present application is to provide a password strength measurement method based on reinforcement learning.

[0006] The technical solution of the present application to solve the technical problem is to provide a password strength measurement method based on reinforcement learning, characterized in that the method comprises the following steps:

[0007] Step 1, data construction phase: constructing a password data set, and then extracting user personal information of passwords in the password data set;

[0008] Step 2, training phase: according to the password data set obtained in step 1, a reinforcement learning model and a scoring model are established respectively, and the reinforcement learning model and the scoring model are trained at the same time; during the training process, the reinforcement learning model needs to explore the path of maximizing the reward value sum and continuously generate a guess list, the scoring model continuously scores the guess list output by the reinforcement learning model, and the scoring result is used to feed back and update the reinforcement learning model, and the scoring model is updated according to the KL divergence between the sorting order of the reward value sum of the guess list and the mixed real user password proportion; the training process is repeated until the optimal reinforcement learning model is obtained;

[0009] Step 3, deployment phase: using the optimal reinforcement learning model obtained in step 2 to generate a password probability dictionary Θ, and then deploying the optimal reinforcement learning model, the Monte Carlo calculation algorithm and the password probability dictionary Θ of step 2;

[0010] Step 4, guess number calculation phase: after deployment, the user inputs the password alpha into the optimal reinforcement learning model, and the optimal reinforcement learning model outputs a probability; then the probability and the password probability dictionary Θ are compared through the Monte Carlo calculation algorithm to obtain the guess number of the password; and according to the corresponding relationship between the guess number of the password and the password strength, the password strength of the password is obtained.

[0011] Compared with the prior art, the beneficial effects of the present application are:

[0012] (1) The previous password strength evaluation method is based on traditional statistical method or information entropy estimation, and no novel password modeling method is tried, so it is difficult to capture more user password generation rules. The password modeling method based on reinforcement learning proposed by the present application regards the process of user generating password as a Markov decision process, calculates the reward brought by each action, thereby modeling the user password generation process, can better model the password sequence, and solves the problem of dependence on data set and inability to generate modes not in the data set.

[0013] (2) The present application can resist online targeted guessing attack and offline wandering guessing attack at the same time. The previous password strength evaluation is based on offline password guessing only, and can only resist offline password guessing attack; or can only detect the use of personal identification information by the user, and cannot judge the resistance of the password of the user to online targeted guessing attack. Offline wandering guessing attack and online targeted guessing attack are both threats that may be encountered in the user identity authentication process, so the present application can detect and resist both attack methods at the same time, and protect the password security of the user.

[0014] (3) The application designs a reinforcement learning model and a scoring model. The training steps of the two models are carried out at the same time, the reinforcement learning model is used to measure the password strength, and the scoring model is used to assist the reinforcement learning model training, and the effect of the reinforcement learning model is evaluated to improve the effect.

[0015] (4) More accurate and robust password strength evaluation. By comparing the evaluation effects of different PSMs on the password strength of real users, the results show that the password strength evaluation results output by the application are significantly better than those of other algorithms in accuracy, and at the same time, the robustness of the method, i.e. the strength evaluation of the password in the extreme case will not deviate too much. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is the overall flowchart of the application;

[0017] Figure 2 is a user password and user personal information matching schematic diagram of an embodiment of the application. DETAILED DESCRIPTION

[0018] The specific embodiments of the application are given below. The specific embodiments are only used to further illustrate the application, and do not limit the protection scope of the application.

[0019] The application provides a password strength measurement method based on reinforcement learning (referred to as the method), characterized in that the method comprises the following steps:

[0020] Step 1, data construction phase: constructing a password data set for training, and then extracting user personal information of the password in the password data set;

[0021] Preferably, in step 1, the specific way of constructing the password data set is:

[0022] The password generation process is regarded as a Markov decision process τ:

[0023] τ={s0,a0,s1,a1,…,s t ,a t ,…,s T ,a T} (1)

[0025] In formula (1), the action a t is a character selected from a password character set (i.e. the bag of words of the password data set) , the state s t is the sequence (w0,w1,…,w t-1 ) of the password PW 1:t obtained by the previous action a t-1 , and all characters t∈[0,T] represents the time step; the probability of Markov decision process τ is:

[0026]

[0027] In formula (2), β(a t ∣s t ) represents the policy of the reinforcement learning model;

[0028] Preferably, in step 1, when the password data set only contains passwords, is the set of all printable characters; when the password data set contains passwords and user personal information, is the set of all printable characters and user personal information tags.

[0029] Preferably, in step 1, the specific way of extracting user personal information is: using the pre-defined user personal information tags to match the passwords in the password data set to determine whether there is user personal information in the password; if the user personal information is matched, the user personal information is converted into a field recognizable by the reinforcement learning model in step 2 according to the user personal information tag.

[0030] Preferably, in step 1, the user personal information includes name, username, phone number, ID number, birthday, and email.

[0031] Preferably, in step 1, the pre-defined user personal information tags are:

[0032] (1) In the name, N1-N7 respectively correspond to full name, full name initial, surname, first name, first name initial + surname, first name + surname initial, and surname initial in capital letters;

[0033] (2) In the username, U1-U2 respectively correspond to the username and the username initial in capital letters;

[0034] (3) In the phone number, T1-T3 respectively correspond to the phone number, the first three digits of the phone number, and the last four digits of the phone number;

[0035] (4) In the ID number, I1-I3 respectively correspond to the first three digits of the ID, the first six digits of the ID, and the last four digits of the ID;

[0036] (5) In the birthday, B1-B 13 respectively correspond to month day year, year month day, day month year, month day, year, year month, month year, last two digits of month day year, last two digits of month year, month, day, day month, day year;

[0037] (6) In the email, E1-E3 respectively correspond to the email name, the second-level domain name, and the top-level domain name.

[0038] For example, Figure 2As shown, for example, the password daniel1027. It can be represented in different ways as {N3B4}, {N31,0,2,7}, {N3B 10 ,2,7}、{N31,0,B 11}、{N3B 10 B 11}, {E3B4} and {d,a,n,i,e,l,1,0,2,7} and other 17 forms of representation.

[0039] Step 2, training phase: Based on the password dataset obtained in step 1, a reinforcement learning model and a scoring model are established respectively, and then the reinforcement learning model and the scoring model are trained simultaneously. During the training process, the reinforcement learning model needs to explore the path that maximizes the total reward value while continuously generating guess lists. The scoring model continuously scores the guess lists output by the reinforcement learning model, and the scoring results are used to feedback and update the reinforcement learning model. The scoring model is updated based on the KL (Kullback-Leibler) divergence between the sort order of the total reward value of the guess list and the proportion of mixed real user passwords. The training process is repeated until the optimal reinforcement learning model is obtained.

[0040] Preferably, in step 2, the optimal reinforcement learning model is saved as a binary format file with a compressed size of 1.34 MB.

[0041] Preferably, in step 2, both the reinforcement learning model and the scoring model need to be pre-trained before training, and the pre-training method is to use the maximum likelihood function on the training set for training.

[0042] Preferably, in step 2, the training process of the reinforcement learning model is as follows: the reinforcement learning model needs to be trained according to the current strategy π θ (a t ∣s t ) and the state s of the currently generated password sequence t , from the password character set Select the next character in action a t ; Each state s during command training t Obtained through Monte Carlo tree search; the reward for a single password is calculated using a scoring model Then take the average value of the rewards of n passwords as the current state s t Reward r t ; The goal of reinforcement learning model training is to optimize the strategy π of the reinforcement learning model θ (a t ∣s t ) to maximize the expected total reward:

[0043]

[0044] In formula (3), is the objective function of the training task, is the total reward for the Markov process τ = PW 1:T , that is, the summation of each step reward, find the path that maximizes the reward obtained by exploration of the reinforcement learning model, and thus generate the guess list; the strategy gradient update of the reinforcement learning model is used to update the parameters θ of the reinforcement learning model.

[0045] Preferably, in step 2, the Monte Carlo tree search is specifically: for a certain state s t searches back n paths until the exploration is completely finished.

[0046] Preferably, in step 2, the reward of a single password is The correlation score of the feature vector extracted from y PW and the Top-1w password in the feature space is obtained:

[0047]

[0048] In formula (4), y e is the feature vector of the Top-1w password, is a real password sampled from the training set for comparison; γ is the temperature hyperparameter of the Softmax function, used to control the model training process.

[0049] For example, the password PW 1:T is {P, a, s, s, w, @, r, d}, so set s0 as B s (the start symbol), then select action a0 to obtain the maximum reward 0.62 of the state of character w0 = P, so as to obtain s1 = B s P, then do the next action a1 to obtain s2 = B s Pa, and so on, until the complete password Passw@rd is generated.

[0050] Preferably, in step 2, the scoring model training process is specifically: using passwords mixed according to different proportions of real user passwords and guess lists generated by the reinforcement learning model, using the scoring model to extract features for calculating the KL divergence between the ranking order of the total reward value of the guess list and the mixing proportion as the objective function; the goal of the scoring model training is to minimize the KL divergence objective function.

[0051] Step 3, deployment phase: using the optimal reinforcement learning model obtained in step 2 to generate a password probability dictionary Θ, and then deploying the optimal reinforcement learning model, Monte Carlo calculation algorithm and password probability dictionary Θ of step 2 for user use;

[0052] Preferably, in step 3, the deployment is to a web frontend or a browser plugin.

[0053] Preferably, in step 3, the reference of the Monte Carlo calculation algorithm is “Dell'Amico, M., & Filippone, M.. Monte Carlo strength evaluation: Fast and reliable password checking. In Proc. ACM SIGSAC CCS 2015, pages 158-169.”.

[0054] Step 4, guess number calculation stage: after deployment, the user inputs the password a into the optimal reinforcement learning model, and the optimal reinforcement learning model outputs a probability; then the probability is compared with the password probability dictionary Θ through the Monte Carlo calculation algorithm to obtain the guess number of the password, which is the number of times the attacker needs to guess the password; then according to the corresponding relationship between the guess number of the password and the password strength, the password strength of the password is obtained.

[0055] Preferably, in step 4, after deployment, the user inputs the password, first judges whether the password a is in the blacklist password, if it is in the blacklist password, directly returns not allowed to register, and reminds the user to modify and use a stronger password; when it is judged that the password a is not in the blacklist password, the password a is input into the optimal reinforcement learning model for subsequent steps.

[0056] Preferably, in step 4, the process of calculating the guess number S p (α) of the password a is considered as: calculating the number of passwords in the entire password space that are weaker than the password a:

[0057]

[0058] From equation (5), it is obtained that calculating the guess number S p (α) of the password a is to calculate the number of passwords in the entire password space whose probability p(β) is greater than the probability p(α) of the password a; since β follows the distribution p(β), the estimate C p (α) of the guess number S Δ (α) of the password a is:

[0059]

[0060] In equation (6), n is the sample size of the password probability dictionary Θ. In this embodiment, the sample size of the password probability dictionary Θ is generally 100,000, which can control the detection error within 1%, and the sample size is reduced by 1 / 2 power. Therefore, the password probability dictionary is about 800KB.

[0061] Preferably, in step 4, the high-frequency password list used in the process of training the reinforcement learning model needs to be collected in advance as a blacklisted password of the website, about 100KB. The storage space required by the entire password guessing number calculation application is about 3MB, which can be well deployed to the webpage front end or browser plug-in.

[0062] Preferably, in step 4, the guessing number is divided into online targeted guessing number GN1 and offline wandering guessing number GN2; if user personal information is extracted, which can come from user public accessible username and email and other auto-fill information, or from user actively input field, the calculated guessing number is online targeted guessing number GN1; if no user personal information is extracted, the calculated guessing number is offline wandering guessing number GN2.

[0063] Preferably, in step 4, according to the actual business and scene requirements, the corresponding relationship between the password guessing number and the password strength is as follows:

[0064] (1) Password with online targeted guessing number GN1 ∈ (0, 10 3 ] or offline wandering guessing number GN2 ∈ (0, 10 6 ]: extremely insecure; considering that the network servers of modern major service providers are all equipped with malicious traffic detection mechanisms, and often limit the daily login times of IP, the upper limit of the guessing number of large-scale online targeted guessing by attackers is 10 3 ; offline hash can be guessed out within 10 6 times of guessing, which is also extremely insecure, and such type of password should be prohibited.

[0065] (2) Password with online targeted guessing number GN1 ∈ (10 3 , 10 5 ] or offline wandering guessing number GN2 ∈ (10 6 , 10 9 ]: relatively insecure; for this interval of password, it is still relatively insecure, but for websites with low security requirements, this part of password can be run and used, but websites with high security requirements (such as finance, etc.) still need to be prohibited.

[0066] (3) Password with online targeted guessing number GN1 ∈ (10 5 , 10 7 ] or offline wandering guessing number GN2 ∈ (10 9 , 10 15Passwords with strength PW1: Generally safe; for this part of passwords, it is believed that the attacker is unlikely to guess them in a short time through various means, and it is a safe password.

[0067] (4) Online targeted guess number GN1>10 7 or offline walk guess number GN2>10 15 Passwords with strength PW3: Generally safe; for this part of passwords, it is believed that the attacker is unlikely to guess them in a short time through various means, and it is a safe password.

[0068] Example 1

[0069] The actual operation of the present application and the four better password strength measurement methods of FLA-PSM, Markov-PSM from C. Castelluccia, M. Dürmuth, and D. Perito. Adaptive password-strength meters from Markov models. In Proc. NDSS 2012., Zxcvbn from Wheeler, Daniel Lowe. zxcvbn: Low-Budget Password Strength Estimation. In Proc. USENIX SEC 2016, pages 157-173., Min_auto from B. Ur, S. M. Segreti, L. Bauer, N. Christin, L. F. Cranor, S. Komanduri, D. Kurilova, M. L. Mazurek, W. Melicher, and R. Shay. Measuring real-world accuracies and biases in modeling password guessability. In Proc. USENIX SEC 2015, pages 463-481.

[0070] LinkedIn, CSDN and Dodonew are used as the accuracy test set, and 100,000 password sets sampled from CSDN are used as the robustness test set.

[0071] Accuracy test: The Spearman coefficient between the strength ranking result output by the calculation algorithm and the strength result of the ideal strength ranking is calculated. Experimental results show that the accuracy of the password strength evaluation results output by the present invention is significantly better than that of other algorithms. Specifically, on the English dataset LinkedIn, the similarity of the password strength measurement method proposed by the present invention is 0.652, while FLA-PSM, Zxcvbn, Markov-PSM, and Min_auto are 0.367, 0.579, 0.201, and 0.558, respectively. On the Chinese dataset CSDN, the similarity of the password strength measurement method proposed by the present invention is 0.728, while the other four methods are 0.643, 0.514, 0.268, and 0.575, respectively. On the Chinese dataset Dodonew, the similarity of the password strength measurement method proposed by the present invention is 0.535, while the other four methods are 0.322, 0.475, 0.297, and 0.423, respectively. This shows that the present invention has better accuracy than previous strength measurement methods.

[0072] Robustness test: Mistakenly identifying a strong password as a weak password may lead to usability issues, while misidentifying a weak password as a strong password may bring security risks and fail to ensure the claimed security. Through the 1V1 confusion matrix test on the CSDN dataset, it can be found that Zxcvbn will have 24691 passwords that will be misidentified by the present invention in 10 4 ~10 8 guessed out, and Zxcvbn will think that this part of the password requires 10 8 It takes more than 100 guesses to guess the password. This indicates that Zxcvbn considers a large number of passwords to be strong passwords. Similar phenomena occur in the other algorithms, indicating that the present invention has better stability than the compared algorithms, that is, it is more robust and can avoid some extreme situations.

[0073] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A password strength measurement method based on reinforcement learning, characterized by, The method comprises the following steps: Step 1, data construction phase: constructing a password data set, and then extracting user personal information of passwords in the password data set; Step 2, training phase: according to the password data set obtained in step 1, a reinforcement learning model and a scoring model are established respectively, and then the reinforcement learning model and the scoring model are trained simultaneously; during the training process, the reinforcement learning model needs to explore the path with the maximum reward value sum and continuously generate a guess list, the scoring model continuously scores the guess list output by the reinforcement learning model, and the scoring result is used for feedback and updating the reinforcement learning model, and the scoring model is updated according to the KL divergence between the ranking order of the reward value sum of the guess list and the mixed real user password proportion; the training process is repeated until the optimal reinforcement learning model is obtained; The specific training process of the reinforcement learning model is: the reinforcement learning model needs to be trained according to the current strategy π θ (a t ∣s t ) and the state s of the currently generated password sequence t , from the password character set Select the next character in action a t ; Each state s during command training t Obtained through Monte Carlo tree search; the reward for a single password is calculated using a scoring model Then take the average value of the rewards of n passwords as the current state s t Reward r t ; The goal of reinforcement learning model training is to optimize the strategy π of the reinforcement learning model θ (a t ∣s t ) to maximize the expected total reward: In formula (3), is an objective function of the training task, is the total reward for the Markov process τ = PW 1:T ; the policy gradient update the parameters θ of the reinforcement learning model using the reinforcement learning model. The scoring model training process is specifically: using passwords mixed with real user passwords and the guess list generated by the reinforcement learning model according to different proportions, using the scoring model to extract features for calculating the KL divergence between the ranking order of the reward value sum of the guess list and the mixed proportion as the objective function; the goal of the scoring model training is to minimize the KL divergence; Step 3, deployment phase: using the optimal reinforcement learning model obtained in step 2 to generate a password probability dictionary Θ, and then deploying the optimal reinforcement learning model, the Monte Carlo calculation algorithm and the password probability dictionary Θ in step 2; Step 4, guess number calculation phase: after the deployment is completed, the user inputs a password α into the optimal reinforcement learning model, and the optimal reinforcement learning model outputs a probability; then the probability and the password probability dictionary Θ are compared through the Monte Carlo calculation algorithm to obtain the guess number of the password; and according to the corresponding relationship between the guess number of the password and the password strength, the password strength of the password is obtained.

2. The reinforcement learning based password strength measurement method of claim 1, wherein, In step 1, the specific way of constructing the password data set is: The password generation process is regarded as a Markov decision process τ: T = {s0, a0, s1, a1,..., s t ,a t ,…,s T ,a T}(1) In formula (1), action a t is a character selected from a password character set State s t is a sequence of passwords PW t-1 obtained from the previous action a 1:t ; t ∈ [0, T] represents a time step; the probability of Markov decision process τ is: In formula (2), π θ (a t ∣s t ) represents a policy of the reinforcement learning model.

3. The reinforcement learning based password strength measurement method of claim 1, wherein, In step 1, the specific way of extracting user personal information is: using a pre-defined user personal information label to match the password in the password data set to determine whether the user personal information exists in the password; if the user personal information is matched, the user personal information is transformed into a field recognizable by the reinforcement learning model in step 2 according to the user personal information label.

4. The reinforcement learning based password strength measurement method of claim 1, wherein, In step 2, the reward for a single password By y PW The correlation score of the feature vector extracted from the Top-1w password and the feature vector in the feature space is obtained: In formula (4), y e is a feature vector of Top-1w password, is a real password sampled from the training set for comparison; γ is a temperature hyperparameter of the Softmax function, used to control the model training process.

5. The reinforcement learning based password strength measurement method of claim 1, wherein, In step 4, after the deployment is completed, the user inputs a password, first judges whether the password α is in the blacklist password, if it is in the blacklist password, it is not allowed to register, and the user is reminded to modify the password; When it is judged that the password α is not in the blacklist password, the password α is input into the optimal reinforcement learning model, and the subsequent steps are performed.

6. The reinforcement learning based password strength measurement method of claim 1, wherein, In step 4, the number of guesses S for the password a is computed p The process of (a) is considered as: computing the number of passwords that are weaker than the password a in the whole password space: From equation (5), the number of guesses S to compute the password a is p (a) is the number of passwords whose probability p(β) is greater than the probability p(a) of the password a in the entire password space. Since β is subject to the distribution p(β), the number of guesses S p of the password α is: Δ is: In formula (6), n is the sample size of the password probability dictionary Θ.

7. The reinforcement learning based password strength measurement method of claim 1, wherein, In step 4, the guess number is divided into online directional guess number GN1 and offline wandering guess number GN2.

8. The reinforcement learning based password strength measurement method of claim 7, wherein, In step 4, the corresponding relationship between the guess number of the password and the password strength is as follows: (1) Online directed guess number GN1 e (0, 10 3 ] or offline brute force guess number GN2 e (0, 10 6 ] Password: Extremely insecure; (2) Passwords with online directed guess number GN1 e (10 3 , 10 5 ] or offline brute force guess number GN2 e (10 6 , 10 9 ]: relatively insecure; (3) Passwords with online directed guess number GN1 e (10 5 , 10 7 ] or offline brute force guess number GN2 e (10 9 , 10 15 ]: Generally available; (4) Online directed guess number GN1 > 10 7 Or offline walk guess number GN2 > 10 15 Password: secure.

Images