Mobile phone security authentication method integrating multiple elements of voice mobile network

Abstract

According to the multi-element mobile phone security authentication method fusing the voice mobile network, the usability and the deployability are greatly improved, interaction between a user and the mobile phone is not needed, and meanwhile, the method can be well compatible with most of mobile phones and browsers in the market. An environment audio pair collected by a mobile phone and a browser is used as an enhanced authentication element. When a user inputs a static password to log in, the two devices record audio signals of the surrounding environment through microphones. After acquisition is completed, two audio samples are sent to a server, the server compares the two samples to judge whether the computer and the mobile phone are located at the same place or not, and finally login legality is judged. And moreover, deployment is easy, surrounding audio signals are collected through the microphone, and the collection can be realized as long as a browser supports WebRTCAPIIRI of HTML5 (Hypertext Markup Language 5). No matter whether the mobile phone is indoor or outdoor or even in a pocket or a wallet of a user, the practical application effectiveness and reliability are high, and the mobile phone authentication safety is obviously improved.

Description

Technical Field

[0001] The present application relates to a mobile phone security authentication method integrating sound, and more particularly to a mobile phone security authentication method integrating sound and mobile network multi-factors, belonging to the technical field of multi-factor mobile phone security authentication. Background Art

[0002] The identity authentication security of network services is very important. When protecting personal security and privacy, most users use the form of setting passwords, that is, the "account + static password" identity authentication mechanism. The password in this authentication method is a static password. Attackers can steal users' passwords by guessing, peeping, brute force cracking, dictionary attacks, monitoring, Trojan attacks, etc., resulting in account or password theft and personal information leakage. Although static password verification is relatively simple to use and deploy, from the perspective of security factors, static passwords as a single-factor identity authentication method can no longer meet the identity authentication security requirements of the Internet.

[0003] The multi-factor authentication mechanism uses two-step verification. When a user logs in to a webpage or other service, they provide two authentication factors to verify their identity. The two components of multi-factor authentication are known to the user. Only when both authentication factors are provided can the authentication be passed. Multi-factor authentication means using two independent pieces of evidence to prove an entity, which is intended to reduce the possibility of the requester providing false evidence of their identity.

[0004] The hardware token in multi-factor authentication requires users to carry tokens related to the interaction, and the service provider must provide an interaction token for each user, which greatly increases the deployment cost. The software token mechanism requires users to copy the verification code from their mobile phone to the browser to complete the verification, which increases the interaction between users and mobile phones.

[0005] Multi-factor authentication mechanisms are transitioning from the previous hardware token method to the software token method. The software token method can improve the deployability and ease of use of the multi-factor authentication mechanism. From the perspective of service providers, this authentication method can greatly reduce the manufacturing and transportation costs caused by manufacturing hardware tokens. From the user's perspective, only one mobile phone is needed to integrate different software tokens provided by multiple service providers, avoiding the trouble of carrying extra hardware devices. Although the multi-factor authentication mechanism based on software tokens has been greatly improved compared with the previous authentication methods, if a service does not force the use of multi-factor authentication mechanisms, most users are still accustomed to using a single authentication method with only a static password. Because the multi-factor authentication mechanism requires users to interact with their mobile phones, these extra steps make users less willing to use the multi-factor authentication mechanism, even though the security of this authentication mechanism is much higher than the traditional static password authentication method. At present, many multi-factor authentication research institutions improve its ease of use by reducing the interaction between users and mobile phones. However, most of these solutions cannot be promoted because the mobile phones, computers or browsers on the market cannot meet their deployment requirements.

[0006] The problems that need to be solved by the existing multi-factor mobile phone security authentication method and the key technical difficulties of this application include:

[0007] (1) Most network services require users to authenticate their identities, and the most primitive authentication method is a user name plus a static password. Due to the determinism of static passwords, this login method is very easy to be cracked by hackers using dictionary attacks and other means, and even the password can be obtained by someone peeping behind the back. Based on this, the existing multi-factor authentication mechanism uses two factors, static passwords and physical objects (tokens, verification codes, or biometric markers such as fingerprints) to authenticate user identities, which can effectively protect the user's account security even in the event of password leakage. However, users are still accustomed to using a single-factor static password login instead of a multi-factor authentication mechanism. The reason is that if a user wants to log in to a server with a multi-factor authentication mechanism, many additional operations (in addition to entering a username and static password) are required to ensure a secure login. There are also some multi-factor authentication schemes in the existing technology that reduce user interaction, but these schemes often require additional deployment of some software or hardware, and are less usable and deployable.

[0008] (2) The multi-factor authentication mechanism is transitioning from the previous hardware token method to the software token method. Although the existing multi-factor authentication mechanism based on software tokens has greatly improved over the previous authentication methods, if a service does not force the use of a multi-factor authentication mechanism, most users are still accustomed to using a single authentication method with only a static password, because the multi-factor authentication mechanism requires users to interact with their mobile phones. These extra steps cause users to dislike using the multi-factor authentication mechanism, even though the security of this authentication mechanism is much higher than the traditional static password authentication method. The existing technology improves its usability by reducing the interaction between users and mobile phones, but most of these solutions cannot be promoted because the mobile phones, computers or browsers on the market cannot meet their deployment requirements. The usability and deployability are greatly limited. In addition, network security forms are becoming more and more complex. It is very important to protect the security and privacy of personal information. The existing technology cannot meet the security requirements of Internet identity authentication.

[0009] (3) The prior art has also proposed some audio-based mobile phone security authentication methods, but these methods are too complicated and have high hardware and software requirements. They require users to interact with their mobile phones and require users' mobile phones and computers to frequently collect environmental audio. They cannot determine whether the user's login is legal by comparing the similarity of the samples collected by the microphones of the two devices. At the same time, the two processes of audio collection and comparison require users to interact with their mobile phones. Users feel that it is much more complicated than the single authentication method of logging in with a static password during use. Moreover, not all mobile phones and mainstream browsers currently support audio collection and do not require the installation of any plug-ins. Therefore, the multi-factor authentication mechanism that integrates sound is not easy to deploy. In addition, it is impossible to establish a persistent connection between the client and the server, and the two parties cannot send data to each other at will, and cannot get rid of the limitations of the HTTP protocol request / response data transmission method. Whether indoors or outdoors, especially when the mobile phone is placed in the experimenter's pocket, it is impossible to use the collected audio to well identify the similarity of the environment in which the two devices are located. The actual application effectiveness and reliability are low, and the security of mobile phone authentication has not been significantly improved, and it does not have a good application prospect. Summary of the invention

[0010] The multi-factor authentication of this application greatly improves the ease of use and deployability. It does not require users to interact with mobile phones, and is well compatible with most mobile phones and browsers on the market. The environmental audio pairs collected by the mobile phone and the browser are used as enhanced authentication factors. When the user enters a static password to log in, the two devices record the audio signals of the surrounding environment through the microphone. After the collection is completed, the two audio samples are sent to the server, and the server compares the two samples to determine whether the computer and the mobile phone are in the same place, and finally determines the legitimacy of the login. The authentication mechanism hardly requires users to interact with the mobile phone, and can work normally whether the mobile phone is indoors, outdoors, in the user's wallet or pocket. Moreover, the multi-factor authentication mechanism that integrates sound is easy to deploy, because most mobile phones and browsers currently support the collection of surrounding audio signals through microphones. As long as the browser supports HTML5's WebRTC API I I, the multi-factor authentication mechanism that integrates sound can be implemented. Whether the mobile phone is indoors or outdoors, or even in the user's pocket or wallet, the actual application effectiveness and reliability are high, the security of mobile phone authentication is significantly improved, and it has good application prospects.

[0011] In order to achieve the above technical effects, the technical solutions adopted in this application are as follows:

[0012] The multi-factor mobile phone security authentication method of the integrated sound mobile network is composed of four parts: the server side, the browser side, the Android client side and the database side. The four parts coordinate and cooperate to complete the authentication. The similarity of the audio of the mobile phone and the computer's surrounding environment is used as an enhanced authentication factor. The legitimacy of the login is determined by comparing the audio samples collected by the mobile phone and the browser of the logged-in user. After the user enters the username and password, the server sends a notification to the browser and the mobile phone at the same time to request them to collect audio; after that, the mobile phone and the browser respectively collect audio signals and run the time synchronization protocol to synchronize with the server. After the audio collection is completed, the client adds a timestamp to the sample and sends it to the server. After receiving the sample, the server first determines their time difference. If the time difference is less than l max , then calculate their average power; only if the average power of two audio samples is greater than π db , and then judge their similarity scores. Divide each audio sample according to the 1 / 3 octave rule to obtain the similarity score of each frequency band. Add the similarity scores of these frequency bands and take the average to get the similarity score of each pair of audio. When the audio time difference, average power and similarity score all meet the corresponding critical values, the verification is legal. Otherwise, when the calculation results of the two samples do not meet the l max , π db and π cIf any of them fails, the verification will fail. In addition, after the system is implemented, a large number of simulated logins are carried out under different environments and settings. The training and aggregation of simulated data are used to obtain l max , π db and π c The reasonable critical value can improve the accuracy of the system and reduce the inconvenience caused to user login by unreasonable critical value setting;

[0013] The multi-factor security system that integrates sound requires users to install an application on their mobile phones, but does not require any software to be installed on the computer. It can be implemented as long as the browser implements HTML5's WebRTC API. In addition, both the mobile phone and the computer running the browser must be equipped with a microphone.

[0014] Preferably, the multi-factor security system architecture integrating sound: using environmental sound as an enhanced authentication factor, without the need for additional authentication hardware tokens, using WebSocket to establish a persistent connection between the server and the client, on which both parties can freely transfer data, and through HTML5's WebRTC technology, the browser directly accesses the microphone for audio collection without downloading plug-ins, and by running the time synchronization protocol and calculating the average power, discarding samples with too large a time difference and too low a power, to prevent attackers from providing similar samples to attack the system;

[0015] Determine the system operation parameters to optimize the system model, train and aggregate a large number of audio pairs, collect audio under different environmental settings, and find the frequency range most suitable for audio analysis by comparing the false rejection rate and false acceptance rate of legal audio pairs and illegal audio pairs. Discard the frequency range that is prone to interference in practical applications. On this basis, determine the critical value of the similarity score. The server will divide the audio into 1 / 3 octave frequency bands, calculate the normalized correlation coefficient of each frequency band, and obtain the similarity score of the entire audio pair by summing and averaging the similarity scores of all frequency bands to determine the legitimacy of the login.

[0016] Preferably, the server-side structure is as follows: the server-side notifies the client-side to perform time synchronization, audio collection, and subsequent processing of the audio to determine the legitimacy of the login. When the user enters the username and password and passes the verification, the server immediately starts secondary authentication; the mobile phone and the browser actively establish a WebSocket persistent connection with the server, and then the server-side requires the client to perform audio collection, and the time is set to 3 seconds; while collecting audio, the time synchronization protocol is run to calculate the time difference with the server. After receiving the audio submitted by the client, the server calculates their time difference, average power, and similarity score in turn. Only after the current process is determined to be legal will the next judgment be made. As long as any judgment is illegal, the login is deemed to have failed.

[0017] This application allows the server and the client to run the time synchronization protocol NTP. NTP uses the client / server model for communication: first, the client sends a request data packet, and the server returns a response data packet to the client after receiving it. Both packets contain the timestamps of sending and receiving. Based on these four timestamps, it is easy to determine the network delay and time deviation between the client and the server:

[0018] Routers A and B are connected through a network, and their clock systems are independent of each other. To synchronize the two independent clock systems, make the following assumptions:

[0019] ① Before the clock systems of A and B are synchronized, assume that the current system time of A is 10:50:00am and the current system time of B is 11:00:00am;

[0020] ②B is a time server, and the system time of A needs to be synchronized with B;

[0021] ③The time required for the data packet sent by A to reach B is 1s, that is, the network transmission time is 1s;

[0022] The NTP working process is as follows:

[0023] ①A first sends an NTP data packet to B. The data packet contains the timestamp of when it left A. The timestamp is t1 = 10:50:00am.

[0024] ②After receiving the data packet, B immediately adds its own timestamp t2=11:00:01;

[0025] ③B then sends the ntp data packet to A and adds its own timestamp t3 = 11:00:02am;

[0026] ④A successfully receives the response data packet and adds a new timestamp t4=10:50:03am;

[0027] After the above process, A has enough information to calculate the transmission time δ and time deviation θ of the NTP data packet on the network, which are:

[0028]

[0029] The values ​​of δ and θ are related to the difference between t2, t1 and t3, t4, but have nothing to do with the difference between t3 and t2, that is, they have nothing to do with the time taken by the server to process the request. A uses these four timestamp data to calculate the network transmission time δ and the time deviation θ.

[0030] When the time synchronization is completed, the client obtains the time difference between the local time and the server local time. The timestamps of the two audio samples will be based on the local time of the server. When the server receives the audio pair, it will compare the time difference between the two. When the time difference is less than the preset time difference threshold, the user login is deemed to be legal; otherwise, the user login is illegal.

[0031] When the time difference of the audio pairs is determined to be legal, the server calculates their average power, discards those particularly quiet audio pairs, performs a fast Fourier transform on the time domain audio signal to obtain the frequency domain signal, and obtains the Fourier transform coefficient. The sum of the squares of the coefficients is the average power of the audio signal. After obtaining the average power of the audio pairs, they are compared with the pre-set average power threshold. If the average power of both audio samples is greater than the threshold, the similarity score determination continues; otherwise, the login is determined to be illegal.

[0032] Preferably, the browser-side structure: the browser is responsible for the authentication of the first and enhanced authentication elements. When the username and password are verified to be legal, the browser establishes a persistent connection with the server through the WebSocket protocol. The WebSocket protocol format is "ws: / / IP:Port" or "wss: / / IP:Port", where wss represents the websocket protocol for encrypted transmission. The use of wss ensures the security of data transmission. After that, the WebRTC API is called to access the microphone to obtain the audio stream, and the AudioContext class is used to create an audio processing node to process the audio stream. The audio acquisition time is set to 3 seconds, and the audio is compressed and merged. While ensuring the audio quality, the dual-channel audio is compressed into a single channel. The sampling point bit is changed to 8 bits, the audio volume is compressed, and the transmission time is reduced. While the audio is being collected, the browser runs the time synchronization protocol to calculate the time difference with the server. The browser adjusts the timestamp of the audio sample according to the time difference to ensure that the sample timestamp uses the server local time as the standard time. After the audio acquisition is completed, the audio is sent to the server together with the timestamp.

[0033] Preferably, remote attack security mode: The security of the multi-factor authentication mechanism integrating sound lies in the fact that the attacker cannot guess the sound in the environment when attacking. Assuming that x is the sample collected by the attacked mobile phone, and y is the forged sample submitted by the attacker, if the attacker wants to successfully launch an attack, the average power of the two audio signals must be higher than τ db , and the two signals must be highly correlated, the two audio signal samples must satisfy P wr (x)>τ db , P wr (y)>τ db And S x,y >τ C , where l<lmax , 1 is the time difference between two samples, l max is the maximum time difference allowed between two samples;

[0034] This application sets the delay time value range to be 0 to 1 max Even if the attacker correctly guesses the noise in the environment and can submit similar audio samples, the two samples will not be smaller than l because the lag time does not meet max The application also rejects those samples whose average power is lower than the critical value τ. db Login to prevent fake attacks when the victim's environment is quiet;

[0035] Quantifying the entropy of environmental noise, acoustically distinguishes between legitimate and fraudulent logins, even if the attacker successfully guesses the type of environment the victim is in.

[0036] Preferably, a multi-factor security authentication method integrating sound is as follows: the user opens the login page and enters the username and static password, the server retrieves the username from the database and verifies the legitimacy of the static password, if the username verification is legal, the server returns a prompt message to the client and establishes a WebSocket persistent connection, after the connection is established, the microphone is notified to collect audio, at the same time, the mobile phone and the browser synchronize the clock with the server by running the time synchronization protocol, after the audio collection is completed, each device adjusts the timestamp of its sample by considering the clock difference with the server, and uses the WebSocket wss protocol to establish a communication channel between the client and the server, the audio message is encrypted and sent to the server; the server decrypts the sample after receiving it, and then the server checks the difference between the two sample timestamps, if the time difference is less than l max The fast Fourier transform is used to convert the time domain signal into the frequency domain signal and calculate their average power. If the user's surrounding environment is too quiet, the average power of the two audio samples is required to be greater than π db After that, the server divides the sample into 1 / 3 octaves and calculates the similarity score in each frequency domain. The similarity score S of the audio pair is x,y Equal to the average of the similarity scores in each frequency domain. If S x,y The final result is greater than the pre-set critical value π c ,The server determines that the computer and the mobile phone used by the user to log in are in the same place, confirms that the login is legal, and allows the login.

[0037] Preferably, the server-side authentication method: ① establishes a persistent connection with the client through WebSocket to achieve information interaction; ② runs a time synchronization protocol to calculate the time difference between the client and the server; ③ determines the legitimacy of the time difference between two audio samples; ④ calculates the average power of the audio sample; ⑤ uses 1 / 3 octave to perform frequency band filtering on the audio signal, calculates the similarity score of each frequency band, and obtains the similarity score of the two audios;

[0038] (1) Establishing a WebSocket connection: After the user enters the username and password, the server performs a database search to verify the legitimacy of the user's identity. If the verification is successful, the client will initiate a WebSocket connection request. The server accepts the request and establishes a persistent connection with the client. Each connection is a Session object. The server uses a static List to save each Session object. Data interaction with the client is achieved using Session.

[0039] (2) Run the time synchronization protocol: After the WebSocket connection is established, the server will send a notification to the client to request it to collect audio. While collecting audio, the client and the server will synchronize time. First, the client will send a data packet and record the send time originateTimestamp. After receiving it, the server will immediately record the receive time receiveTimestamp. Then, the time when the server returns the data packet is TransmitTimestamp, and the time when the data packet arrives at the client is destinationTimestamp. The client uses these four times to calculate the time difference between it and the server.

[0040] (3) Time difference comparison: When the server receives the audio samples from the mobile phone and the browser, it uses the timestamps to compare their time differences. If the time difference between the two samples is greater than l max , it is considered that there may be an attack and the verification is illegal. If the time difference between the two samples is less than l max , the server adjusts the start time of the audio according to their time difference.

[0041] (4) Calculation of average power: After the time difference comparison of the audio pairs is completed, the server calculates their average power. The average power of the audio is calculated in the frequency domain by using the ratio of the square of the absolute value of the time domain signal amplitude to the signal length.

[0042] Preferably, the browser-side authentication method: the browser side is written based on HTML5 and JavaScript, the encryption of audio data is implemented by wss using the SSL protocol, the navigator.getUserMedia() API of WebRTC is used to access the local microphone from the browser, and the audio is collected, compressed, merged and formatted by creating a windows.AudioContext environment object; the AudioContext interface represents an audio processing environment composed of audio modules, each module corresponds to an audio node AudioNode, and the audio stream captured by the microphone is obtained through AudioContext, cached and compressed, and then audio data in wav format is generated.

[0043] The browser is responsible for audio collection. When it receives the audio collection notification from the server, it will immediately request the microphone and create a source node from the microphone's input stream. Then, it uses AudioContext to create a processor node for recording.

[0044] Preferably, the Android client authentication method: the Android client uses WebSocket to establish a persistent connection with the server, and performs time synchronization and data transmission on this basis;

[0045] Audio capture uses the AudioRecord class. An AudioRecord class is instantiated through the constructor public AudioRecord(intaudioSource, intsampleRateInHz, int channelConfig, int audioFormat, intbufferSizeInBytes). The parameters represent the audio source, audio sampling frequency, channel settings, recording encoding format, and the size of the buffer where the captured data is saved. After recording is completed, the data is read from the buffer to generate a temporary file in wav format, which is then passed to the server by the send() method of WebSocket.

[0046] Compared with the prior art, the innovations and advantages of this application are:

[0047] (1) This application designs and implements a multi-factor authentication mechanism that integrates sound, uses environmental sound as an enhanced authentication factor, does not require users to purchase additional authentication hardware tokens, and reduces the cost of authentication. When performing the second factor authentication, there is no need for users to interact with the mobile phone. The entire process is as convenient as the traditional "user name + static password" authentication method. The authentication time of the multi-factor authentication mechanism that integrates sound is also greatly shortened compared to other authentication mechanisms. The multi-factor authentication mechanism that integrates sound uses WebSocket to establish a persistent connection between the server and the client, on which both parties can transfer data at will. Through HTML5's WebRTC technology, the browser can directly access the microphone for audio collection without downloading plug-ins, thereby greatly simplifying the redundancy of the program. By running the time synchronization protocol and calculating the average power, the mechanism can discard samples with too large a time difference and too low a power, preventing attackers from providing similar samples to attack the system. In order to optimize the system model and determine the parameters for system operation, a large number of audio pairs are trained and aggregated, and audio is collected under different environmental settings. By comparing the false rejection rate and false acceptance rate of legitimate audio pairs and illegal audio pairs, the frequency range most suitable for audio analysis is found, and the frequency range that is prone to interference in actual applications is discarded. On this basis, the critical value of the similarity score is determined. Through experimental data testing, this application has found that the multi-factor security system that integrates sound can achieve good authentication effects, reduce false acceptance rate and false rejection rate, and at the same time, can effectively resist similar environment attacks launched by attackers.

[0048] (2) The multi-factor authentication of this application has greatly improved the ease of use and deployability. It does not require users to interact with their mobile phones, and is well compatible with most mobile phones and browsers on the market. The ambient audio pairs collected by the mobile phone and the browser are used as enhanced authentication factors. When the user enters a static password to log in, the two devices record the audio signals of the surrounding environment through the microphone. After the collection is completed, the two audio samples are sent to the server, which compares the two samples to determine whether the computer and the mobile phone are in the same location, and finally determines the legitimacy of the login. The authentication mechanism hardly requires users to interact with the mobile phone, and can work normally whether the mobile phone is indoors, outdoors, in the user's wallet or pocket. Moreover, the multi-factor authentication mechanism that integrates sound is easy to deploy, because most mobile phones and browsers currently support the collection of ambient audio signals through microphones. As long as the browser supports HTML5's WebRTC API I I, the multi-factor authentication mechanism that integrates sound can be implemented. Through experiments, this application found that the speed of this authentication mechanism is an average of 5 seconds slower than the authentication method with only a static password. Despite this, this authentication mechanism is still much faster than the multi-factor authentication mechanism based on verification codes (an average of 25 seconds), and the security and convenience of mobile phone authentication are greatly improved.

[0049] (3) The contributions and innovations of this application also include: First, it innovatively proposes a multi-factor authentication mechanism that integrates sound, which has the advantages of reducing user interaction with mobile phones and being easy to deploy. The authentication factor of this authentication mechanism is the proximity of the audio samples collected by the user's mobile phone and the computer he is using to log in. The legitimacy of the login is determined by comparing the surrounding audio signals collected by the mobile phone and the computer. At the same time, the collection and comparison process is transparent to the user. Second, a multi-factor authentication model that integrates sound is implemented in the Android system. The model is used to evaluate the effectiveness of this authentication method on different devices. Through experimental verification, whether the mobile phone is indoors or outdoors, or even in the user's pocket or wallet, the actual application effectiveness and reliability are high, the security of mobile phone authentication is significantly improved, and it has good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a diagram of the architecture of a multi-factor security system that integrates sound.

[0051] Figure 2 This is the server-side flow chart of this application.

[0052] Figure 3 This is a graph of audio similarity score calculation based on 1 / 3 octave.

[0053] Figure 4 It is a framework diagram of a multi-factor security authentication method that integrates sound.

[0054] Figure 5 It is a schematic diagram of similarity scores in different environments for false rejection rate analysis.

[0055] Figure 6 This is a diagram of the similarity scores of the false rejection rate analysis when the mobile phone is placed in different positions. DETAILED DESCRIPTION

[0056] The following, in conjunction with the accompanying drawings, further describes the technical solution of the integrated voice mobile network multi-factor mobile phone security authentication method provided by the present application, so that technical personnel in the field can better understand the present application and implement it.

[0057] The present application proposes a simple and easy-to-implement multi-factor authentication mechanism that integrates sound. The mechanism does not require the user to interact with the mobile phone, and its enhanced authentication factor is the ambient audio collected by the user's mobile phone and computer. By comparing the similarity of the samples collected by the microphones of the two devices, it can be determined whether the user's login is legal. At the same time, the two processes of audio collection and comparison do not require the user to interact with the mobile phone, so the user feels like a single authentication method that only logs in with a static password during use, which is also the biggest difference between this mechanism and other multi-factor authentication mechanisms. Since all current mobile phones and mainstream browsers support audio collection and do not require the installation of any plug-ins, the multi-factor authentication mechanism that integrates sound is easy to deploy. In addition, by using WebSocket technology to establish a persistent connection between the client and the server, both parties can send data to each other at will, getting rid of the limitations of the HTTP protocol request / response data transmission method. Through experimental verification, the present application found that whether indoors or outdoors, even if the mobile phone is placed in the experimenter's pocket, the collected audio can well identify the similarity of the environment in which the two devices are located, verifying that the system has good application prospects.

[0058] 1. Multi-factor security system architecture integrating sound

[0059] The multi-factor security system that integrates sound consists of four parts: server side, browser side, Android client side and database side. The four parts work together to complete the authentication work. The overall design of the system is as follows: Figure 1 shown.

[0060] The multi-factor security system that integrates sound uses the similarity of the audio of the mobile phone and the computer's surrounding environment as an enhanced authentication factor. It determines the legitimacy of the login by comparing the audio samples collected by the user's mobile phone and browser. After the user enters the username and password, the server sends a notification to the browser and mobile phone at the same time to request them to collect audio; then, the mobile phone and browser collect audio signals respectively and run the time synchronization protocol to synchronize with the server. After the audio collection is completed, the client adds a timestamp to the sample and sends it to the server. After receiving the sample, the server first determines their time difference. If the time difference is less than l max , then calculate their average power; only if the average power of two audio samples is greater than π db, and then judge their similarity scores. Divide each audio sample according to the 1 / 3 octave rule to obtain the similarity score of each frequency band. Add the similarity scores of these frequency bands and take the average to get the similarity score of each pair of audio (in actual applications, not all frequency bands are used for similarity scores. Some frequency bands are not suitable for audio comparison due to serious attenuation, so only frequency bands with better robustness are extracted for calculation). When the audio time difference, average power and similarity score all meet the corresponding critical values, the verification is legal. Otherwise, when the calculation results of the two samples do not meet the l max , π db and π c If any of them fails, the verification will fail. In addition, after the system is implemented, a large number of simulated logins are carried out under different environments and settings. The training and aggregation of simulated data are used to obtain l max , π db and π c Reasonable critical value can be set to improve the accuracy of the system and reduce the inconvenience caused to user login by unreasonable critical value setting.

[0061] The multi-factor security system that integrates voice requires users to install an application on their mobile phones, but does not require any software to be installed on the computer. It can be implemented as long as the browser implements HTML5's WebRTC API. In addition, both the mobile phone and the computer running the browser must be equipped with a microphone.

[0062] Multi-factor authentication mechanism integrating voice: ① Ease of use: It makes users feel that they are logging in with only a username and a static password. Users are not required to interact with their phones, even if they pick up their phones or take them out of their pockets or wallets. ② Deployability: It can work properly on mobile phones, computers, and browsers. Users do not need to download additional software or install browser plug-ins. Plug-in-based solutions limit the availability of the system. Different servers may require different plug-ins, so users need to install the plug-in every time they log in on a new computer. This mechanism should be widely used in smartphones, so this application abandons the solution that requires the use of special hardware (such as NFC chips or biometric sensors) on mobile phones.

[0063] 2. Server-side structure

[0064] The server notifies the client to perform time synchronization, audio collection, and subsequent processing of the audio to determine the legitimacy of the login. When the user enters the username and password and passes the verification, the server immediately starts secondary authentication; the mobile phone and browser actively establish a WebSocket persistent connection with the server. After that, the server requires the client to collect audio, and the time is set to 3 seconds; while collecting audio, the time synchronization protocol is run to calculate the time difference with the server. After receiving the audio submitted by the client, the server calculates their time difference, average power, and similarity score in turn. Only when the current process is determined to be legal will the next judgment be made. As long as any judgment is illegal, the login is deemed to have failed. The server-side process is as follows: Figure 2 .

[0065] The time synchronization between the server and the client is used to determine the timestamp of the audio sample. Since the local time of the browser may be different from the local time of the Android client, if their respective local times are used as the timestamp of the audio sample, the time difference cannot be compared later. In view of the above reasons, this application allows the server and the client to run the time synchronization protocol NTP. NTP uses the client / server mode for communication: First, the client sends a request data packet, and the server returns a response data packet to the client after receiving it. Both data packets contain the timestamps of sending and receiving. Based on these four timestamps, it is easy to determine the network delay and time deviation between the client and the server:

[0066] Routers A and B are connected through a network, and their clock systems are independent of each other. To synchronize the two independent clock systems, make the following assumptions:

[0067] ① Before the clock systems of A and B are synchronized, assume that the current system time of A is 10:50:00am and the current system time of B is 11:00:00am;

[0068] ②B is a time server, and the system time of A needs to be synchronized with B;

[0069] ③The time required for the data packet sent by A to reach B is 1s, that is, the network transmission time is 1s;

[0070] The NTP working process is as follows:

[0071] ①A first sends an NTP data packet to B. The data packet contains the timestamp of when it left A. The timestamp is t1 = 10:50:00am.

[0072] ②After receiving the data packet, B immediately adds its own timestamp t2=11:00:01;

[0073] ③B then sends the ntp data packet to A and adds its own timestamp t3 = 11:00:02am;

[0074] ④A successfully receives the response data packet and adds a new timestamp t4=10:50:03am;

[0075] After the above process, A has enough information to calculate the transmission time δ and time deviation θ of the NTP data packet on the network, which are:

[0076]

[0077] The values ​​of δ and θ are related to the difference between t2, t1 and t3, t4, but have nothing to do with the difference between t3 and t2, that is, they have nothing to do with the time taken by the server to process the request. A uses these four timestamp data to calculate the network transmission time δ and the time deviation θ.

[0078] When the time synchronization is completed, the client obtains the time difference between the local time and the server local time. The timestamps of the two audio samples will be based on the local time of the server. When the server receives the audio pair, it will compare the time difference between the two. When the time difference is less than the preset time difference threshold, the user login is deemed to be legal; otherwise, the user login is illegal.

[0079] When the time difference of the audio pairs is determined to be legal, the server calculates their average power, discards those particularly quiet audio pairs, performs a fast Fourier transform on the time domain audio signal to obtain the frequency domain signal, and obtains the Fourier transform coefficient. The sum of the squares of the coefficients is the average power of the audio signal. After obtaining the average power of the audio pairs, they are compared with the pre-set average power threshold. If the average power of both audio samples is greater than the threshold, the similarity score determination continues; otherwise, the login is determined to be illegal.

[0080] Determination of similarity score: First, the frequency domain signal is divided into frequency bands with constant bandwidth ratio according to 1 / 3 octave. Through a large number of experiments, the most common and anti-attenuation frequency band set is selected, and unreasonable frequency bands are discarded. The cross-correlation function of each frequency band in the frequency band set is calculated and normalized to obtain the normalization coefficient of each frequency band. The weighted average of the normalized cross-correlation coefficients of all frequency bands is the similarity score of the two audio samples. After obtaining the similarity score of the audio pair, it will be compared with the critical value of the similarity score. If their similarity score is greater than the visual value, the two audios are similar and the login is legal; otherwise, the user login is determined to be illegal. 1 / 3 octave is a frequency band division method that conforms to the characteristics of the human ear. At the same time, compared with 1 octave, it divides the sound frequency into more frequency bands. The frequency band division in the low-frequency range is denser, while the high-frequency range is sparser, which is convenient for more detailed research. Figure 3Calculation diagram of audio similarity score based on 1 / 3 octave.

[0081] 3. Browser-side structure

[0082] The browser is responsible for the authentication of the first and enhanced authentication factors. When the username and password are verified to be legitimate, the browser establishes a persistent connection with the server through the WebSocket protocol. The WebSocket protocol format is "ws: / / IP:Port" or "wss: / / IP:Port", where wss represents the websocket protocol for encrypted transmission. The use of wss ensures the security of data transmission. After that, the WebRTC API is called to access the microphone to obtain the audio stream, and the AudioContext class is used to create an audio processing node to process the audio stream. The audio acquisition time is set to 3 seconds, and the audio is compressed and merged. While ensuring the audio quality, the dual-channel audio is compressed into a single channel. The sampling point bit is changed to 8 bits, the audio volume is compressed, and the transmission time is reduced. While the audio is being collected, the browser runs the time synchronization protocol to calculate the time difference with the server. The browser adjusts the timestamp of the audio sample according to the time difference to ensure that the sample timestamp uses the server local time as the standard time. After the audio acquisition is completed, the audio is sent to the server together with the timestamp.

[0083] 4. Android Client Structure

[0084] The Android client is responsible for collecting and sending audio, and performs time synchronization protocol at the same time. After establishing a persistent connection with the server, it will wait for the server to collect audio messages. The audio collection process compresses and merges the audio to reduce the volume.

[0085] 5. Remote Attack Safe Mode

[0086] The security of the multi-factor authentication mechanism that integrates sound lies in the fact that the attacker cannot guess the sound in the environment when attacking. Assume that x is the sample collected by the attacker's mobile phone, and y is the forged sample submitted by the attacker. If the attacker wants to successfully launch an attack, the average power of the two audio signals must be higher than τ. db , and the two signals must be highly correlated, the two audio signal samples must satisfy P wr (x)>τ db , P wr (y)>τ db And S x,y >τ C , where l<l max , 1 is the time difference between two samples, l max is the maximum time difference allowed between two samples;

[0087] In order to prevent the attacker from successfully guessing the victim's environment information and improve the security of the scheme against attacks, this application sets the delay time value range to 0 to 1. max Even if the attacker correctly guesses the noise in the environment and can submit similar audio samples, the two samples are rejected because the lag time does not meet the requirement of less than lmax. This application also rejects those samples whose average power is lower than the critical value τ db login, preventing impersonation attacks when the victim's environment is quiet (for example, when the victim is sleeping).

[0088] Quantifying the entropy of environmental noise and guessing the victim's environment information is a challenging or even impossible task. The results often depend on factors such as the victim's environment, voice, gender, and age. This application uses experimental data to show that sound can distinguish between legitimate and fraudulent logins, even if the attacker successfully guesses the type of environment the victim is in.

[0089] 6. Multi-factor security authentication method integrating sound

[0090] Figure 4 The implementation process of the authentication mechanism is demonstrated. The user opens the login page and enters the username and static password. The server retrieves the username from the database and verifies the legitimacy of the static password. If the username is verified to be legitimate, the server returns a prompt message to the client and establishes a WebSocket persistent connection. After the connection is established, the microphone is notified to collect audio (the browser uses WebRTC, and the mobile client uses AudioRecord). At the same time, the mobile phone and browser synchronize the clock with the server by running the time synchronization protocol. After the audio collection is completed, each device adjusts the timestamp of its sample by considering the clock difference with the server. The communication channel between the client and the server is established using the WebSocket wss protocol. The audio message is encrypted and sent to the server; the server decrypts the sample after receiving it. After that, the server checks the difference between the two sample timestamps. If the time difference is less than l max The fast Fourier transform is used to convert the time domain signal into the frequency domain signal and calculate their average power. If the user's surrounding environment is too quiet, the attacker can easily forge a similar audio and carry out the attack. Therefore, the average power of the two audio samples is required to be greater than π db After that, the server divides the sample into 1 / 3 octaves and calculates the similarity score in each frequency domain. The similarity score S of the audio pair is x,y Equal to the average of the similarity scores in each frequency domain. If S x,y The final result is greater than the pre-set critical value π c,The server determines that the computer and the mobile phone used by the user to log in are in the same place, confirms that the login is legal, and allows the login.

[0091] In a noisy environment, the authentication process is completely transparent to the user. If the environment is quiet, the multi-factor authentication mechanism that integrates sound requires the user to make some noise, such as clearing the throat.

[0092] 7. Server-side authentication method

[0093] ① Establish a persistent connection with the client through WebSocket to realize information interaction; ② Run the time synchronization protocol to calculate the time difference between the client and the server; ③ Determine the legitimacy of the time difference between two audio samples; ④ Calculate the average power of the audio sample; ⑤ Use 1 / 3 octave to perform frequency band filtering on the audio signal, calculate the similarity score of each frequency band, and obtain the similarity score of the two audios.

[0094] Considering that the efficiency of using Java for 1 / 3 octave division and similarity score calculation is not high enough, Java needs to call Matlab functions to perform related operations. First, use Matlab to write the corresponding calculation function, then package it to generate a Java-callable jar package, and copy it together with Matlab's own javabuilder.jar package to the project directory, so that Java can successfully use Matlab functions for calculations.

[0095] (1) Establishing a WebSocket Connection

[0096] After the user enters the username and password, the server performs a database search to verify the legitimacy of the user's identity. If the verification is successful, the client will initiate a WebSocket connection request. The server accepts the request and establishes a persistent connection with the client. Each connection is a Session object. The server uses a static List to save each Session object. Data interaction with the client is achieved using Session.

[0097] (2) Runtime Synchronization Protocol

[0098] After the WebSocket connection is established, the server will send a notification to the client to request it to collect audio. While collecting audio, the client and the server will synchronize time. First, the client will send a data packet and record the send time originateTimestamp. After receiving it, the server will immediately record the receive time receiveTimestamp. Then the time when the server returns the data packet is TransmitTimestamp, and the time when the data packet arrives at the client is destinationTimestamp. The client uses these four times to calculate the time difference between it and the server.

[0099] (3) Time difference comparison

[0100] When the server receives the audio samples from the mobile phone and the browser, it compares their time difference using the timestamps. If the time difference between the two samples is greater than l max , it is considered that there may be an attack and the verification is illegal. If the time difference between the two samples is less than l max , the server adjusts the start time of the audio according to their time difference.

[0101] (4) Calculation of average power

[0102] After the time difference comparison of the audio pairs is completed, the server calculates their average power and uses the ratio of the square of the absolute value of the time domain signal amplitude to the signal length to calculate the average power of the audio in the frequency domain.

[0103] First, the audio signal in the time domain is Fourier transformed to obtain the frequency domain signal. The ratio of the frequency domain signal to the Fourier transform length is the Fourier transform coefficient. The sum of the squares of the coefficients is the average power of the audio signal. When both audio samples meet the set power threshold, their similarity scores are calculated. The frequency domain signal is divided into frequency bands using 1 / 3 octaves. 20 frequency bands within the frequency range of 50Hz to 4000Hz are selected to calculate their cross-correlation functions. The xcorr function is used to calculate the cross-correlation coefficient of the two audio samples in the form of cross-correlation = xcorr (x, y, option), where x and y represent two audio bands respectively, and opt The optional values ​​of ion are baised, unbaised, coeff and none, where baised and unbaised are biased and unbiased estimates of the computer cross-correlation function, respectively. None is the default case. Coeff is the normalized cross-correlation function, that is, the cross-correlation coefficient, with a value between -1 and 1. Coeff is used to normalize the result and scale the result to between -1 and 1 to become a scalar, thereby obtaining the correlation between the two signal quantities. After normalization, the larger the absolute value of the result, the stronger the similarity between the two audio signals. After the similarity score is obtained, it is compared with the set critical value to determine the legitimacy of the login.

[0104] 8. Browser-side authentication method

[0105] The browser is written based on HTML5 and JavaScript. The encryption of audio data is implemented by wss using the SSL protocol. The navigator.getUserMedia() API of WebRTC is used to access the local microphone from the browser. The audio is collected, compressed, merged and formatted by creating a windows.AudioContext environment object. The AudioContext interface represents an audio processing environment composed of audio modules. Each module corresponds to an audio node AudioNode. The audio stream captured by the microphone is obtained through AudioContext, cached and compressed, and then audio data in wav format is generated.

[0106] The browser is responsible for audio collection. When it receives the audio collection notification from the server, it will immediately request the microphone and create a source node from the microphone's input stream. Then, it uses AudioContext to create a processor node for recording.

[0107] 9. Android Client Authentication Method

[0108] The Android client uses WebSocket to establish a persistent connection with the server, based on which time synchronization and data transmission are performed.

[0109] Audio capture uses the AudioRecord class. An AudioRecord class is instantiated through the constructor public AudioRecord(intaudioSource, intsampleRateInHz, int channelConfig, int audioFormat, intbufferSizeInBytes). The parameters represent the audio source, audio sampling frequency, channel settings, recording encoding format, and the size of the buffer where the captured data is saved. After recording is completed, the data is read from the buffer to generate a temporary file in wav format, which is then passed to the server by the send() method of WebSocket.

[0110] 10. Experimental Analysis

[0111] 1. Data Collection

[0112] In order to determine several important critical values ​​of the system operation (time difference, average power, similarity score), after the system is successfully designed, it is used to collect a large number of audio pairs. This application sets up a server that supports multi-factor authentication of fused sound. It runs the time synchronization protocol and notifies the mobile phone and computer to collect audio at each login. The time for each audio collection is set to 3 seconds. After receiving the audio pair, the server will insert it into the database for storage, divide these audio pairs into 1 / 3 octaves, and calculate the similarity score, power critical value, etc.

[0113] Considering that the collected audio data may vary due to differences in the type of computer operating system, environmental factors, and the placement of the mobile phone, audio samples were collected in the following settings based on various factors. Environment: in a quiet laboratory, a laboratory with a computer playing music, a living room with a TV (TV), a classroom with a teacher giving a lecture (classroom), a running subway (subway), and a coffee shop (coffee shop). User behavior: not making any sound, talking normally with people, coughing, and whistling. Mobile phone placement: on a table or bench next to the user, in the user's pocket or wallet.

[0114] 2. System Performance Evaluation

[0115] 1. False Rejection Rate Analysis

[0116] Discuss the impact of different settings (environment, user activity, phone location, phone model, and computer model) on the false rejection rate. Figure 5 and Figure 6Box and whisker plots for each setting. The whiskers mark the 5th and 95th percentiles of similarity scores. The upper and lower ends of the boxes show the 25th and 75th percentiles. The line and solid inside each box represent the median and mean, respectively, of the similarity scores for that setting. The dashed line represents the similarity score cutoff, and the dots in the plot represent login attempts with a similarity score below the cutoff (i.e., falsely rejected login attempts).

[0117] (1) Environment

[0118] Figure 5 The similarity scores in different environments are shown, and the multi-factor security system that integrates sound can work well both indoors and outdoors. There were no false rejections in the music (432 authentications in total), lecture (122 authentications in total), and TV (430 authentications in total) environments; the false rejection rate in the laboratory environment (310 authentications in total) was 0.003; the false rejection rate in the subway environment (370 authentications in total) was 0.003; and the false rejection rate in the cafe environment (338 authentications in total) was 0.006.

[0119] (2) User Activities

[0120] Similarity scores based on different user activities. If the user makes some noise, the similarity score will be increased. In the subway and cafe, if the user is relatively quiet and there is no other sound around, the authentication will fail; when listening to a lecture, the user can only remain quiet (but there is a teacher giving a lecture), so other situations were not tested; similarly, in order not to affect others, whistling was not tested in the cafe. The statistical results show that the false rejection rate when the user is quiet is 0.005 (a total of 579 authentications); the false rejection rate when the user coughs is 0.002 (a total of 529 authentications); the false rejection rate when the user speaks is 0 (a total of 541 authentications); the false rejection rate when the user whistles is 0 (a total of 353 authentications).

[0121] (3) Mobile phone placement

[0122] Figure 6 The similarity scores of the mobile phone in different positions are shown. The authentication effect is better when the mobile phone is placed on the table; on the contrary, the authentication effect is not particularly ideal when the mobile phone is placed in the user's pocket or wallet. This is because when the mobile phone is placed in the pocket or wallet, it will cause friction with the textile, resulting in attenuation of the sound signal, which is significantly different from the audio collected by the computer. The false rejection rates when the mobile phone is placed in different positions are: 0.001 on the table (a total of 667 authentications), 0.001 in the pocket (a total of 675 authentications), and 0.003 in the wallet (a total of 660 authentications).

[0123] (4) The distance between the phone and the computer

[0124] In some settings, the user's phone may be far away from his computer. The computer was placed next to a turned-on TV and the multi-factor security system integrating sound was tested with the phone away from the computer. For this set of experiments, the average noise power of the TV was 50db. Three different distances were tested: 4 meters, 6 meters and 10 meters (20 login attempts were made at each distance). The results showed that all login attempts were successful; but when the phone was in another room with the door closed, the login was rejected.

[0125] Based on the above experimental results, it can be inferred that the false rejection rate can be completely ignored in practical applications, and the security system can play a good authentication effect in applications.

[0126] 2. Safety performance analysis

[0127] A successful attack requires the attacker to submit a sample that is very similar to the user's mobile phone audio. If the victim is in a coffee shop, then the attacker needs to submit a typical audio sample in a similar environment (for example, the attacker and the victim are in two different coffee shops). This application assumes that the attacker successfully guesses the victim's environment or that the attacker and the victim are in different locations with the same sound source. Analyze the false acceptance rates of the above two scenarios.

[0128] (1) Similar environment attack

[0129] Assume that the attacker and the victim are in similar environments. Calculate its false acceptance rate. After a large number of experiments, the following results are obtained: the false acceptance rates in the music and TV environments are 0.004 (430 attempts in total) and 0.005 (409 attempts in total); the false acceptance rate in the lecture environment is 0.006 (460 attempts in total); when the attacker and the victim are both on the subway, the false acceptance rate is 0.002 (470 attempts in total); the false acceptance rate in the office environment is 0.004 (472 attempts in total); when the attacker and the victim are both in a cafe, the false acceptance rate is 0.007 (569 attempts in total).

[0130] The above results show that even if the attacker correctly guesses the victim's environment and submits similar samples for attack, the false acceptance rate is still very low. This is because the audio noise in a specific environment is affected by random events (such as background music, clinking of cups, voices of people around, etc.), which are factors that the attacker cannot predict and imitate.

[0131] (2) Same audio source attack

[0132] Assume that the victim and the attacker access the same audio source from different locations. For example, when the victim is watching TV, the attacker correctly guesses the victim's TV channel. Put the attacker's phone and the victim's computer in different locations, but they are both next to the smart TV. Because both devices can access the same audio source, if the time difference between their submissions is less than l max , then the attack will be successful. This application tested 4 cable TV channels, 2 Internet radios and 2 Internet TV channels. For each scenario, this application simulated 100 attacks and recorded the false acceptance rate. In the case of cable TV channels, since all content is provided by the same provider, the signals are tightly synchronized and the similarity score is higher than the critical value. In the network environment, when the same content is provided by different network providers, due to the delay of the signal, the similarity score is basically lower than the critical value; if it is provided by the same provider, the similarity score will be significantly higher than the similarity score.

Claims

1. A multi-factor mobile phone security authentication method integrating voice mobile network, It is characterized in that The multi-factor security system integrating sound consists of four parts: server side, browser side, Android client side and database side. The four parts coordinate and cooperate to complete the authentication. The similarity of the audio of the mobile phone and the computer's surrounding environment is used as an enhanced authentication factor. The legitimacy of the login is determined by comparing the audio samples collected by the mobile phone and browser of the logged-in user. After the user enters the username and password, the server sends a notification to the browser and mobile phone at the same time to request them to collect audio; then, the mobile phone and browser collect audio signals respectively and run the time synchronization protocol to synchronize with the server. After the audio collection is completed, the client adds a timestamp to the sample and sends it to the server. After receiving the sample, the server first determines their time difference. If the time difference is less than l max , then calculate their average power; only if the average power of two audio samples is greater than π db , and then judge their similarity scores. Each audio sample is divided according to the 1 / 3 octave rule to obtain the similarity score of each frequency band. The similarity scores of these frequency bands are added and averaged to obtain the similarity score of each pair of audio. When the audio time difference, average power and similarity score all meet the corresponding critical values, the verification is legal. Otherwise, when the calculation results of the two samples do not meet l max , π db and π c If any of them fails, the verification will fail. In addition, after the system is implemented, a large number of simulated logins are carried out under different environments and settings. The training and aggregation of simulated data are used to obtain l max , π db and π c The reasonable critical value can improve the accuracy of the system and reduce the inconvenience caused to user login by unreasonable critical value setting; The multi-factor security system that integrates sound requires users to install an application on their mobile phones, but does not require any software to be installed on the computer. It can be implemented as long as the browser implements HTML5's WebRTC API. In addition, both the mobile phone and the computer running the browser must be equipped with a microphone.

2. According to the method for fusion voice mobile network multi-factor mobile phone security authentication in claim 1, It is characterized in that Multi-factor security system architecture integrating sound: Environmental sound is used as an enhanced authentication factor, without the need for additional authentication hardware tokens. WebSocket is used to establish a persistent connection between the server and the client. On this connection, both parties can pass data at will. Through HTML5's WebRTC technology, the browser directly accesses the microphone for audio collection without downloading plug-ins. By running the time synchronization protocol and calculating the average power, samples with too large a time difference and too low a power are discarded to prevent attackers from providing similar samples to attack the system. Determine the system operation parameters to optimize the system model, train and aggregate a large number of audio pairs, collect audio under different environmental settings, and find the frequency range most suitable for audio analysis by comparing the false rejection rate and false acceptance rate of legal audio pairs and illegal audio pairs. Discard the frequency range that is prone to interference in practical applications. On this basis, determine the critical value of the similarity score. The server will divide the audio into 1 / 3 octave frequency bands, calculate the normalized correlation coefficient of each frequency band, and obtain the similarity score of the entire audio pair by summing and averaging the similarity scores of all frequency bands to determine the legitimacy of the login.

3. According to the method for fusion voice mobile network multi-factor mobile phone security authentication in claim 1, It is characterized in that Server-side structure: The server notifies the client to perform time synchronization, audio collection, and subsequent processing of the audio to determine the legitimacy of the login. When the user enters the username and password and passes the verification, the server immediately starts the secondary authentication; the mobile phone and browser actively establish a WebSocket persistent connection with the server, and then the server requires the client to collect audio, and the time is set to 3 seconds; While collecting audio, the time synchronization protocol is run to calculate the time difference with the server. After receiving the audio submitted by the client, the server calculates their time difference, average power and similarity score in turn. Only when the current process is judged to be legal will the next judgment be made. If any judgment is illegal, the login is considered to have failed. This application allows the server and the client to run the time synchronization protocol NTP. NTP uses the client / server model for communication: first, the client sends a request data packet, and the server returns a response data packet to the client after receiving it. Both packets contain the timestamps of sending and receiving. Based on these four timestamps, it is easy to determine the network delay and time deviation between the client and the server: Routers A and B are connected through a network, and their clock systems are independent of each other. To synchronize the two independent clock systems, make the following assumptions: ① Before the clock systems of A and B are synchronized, assume that the current system time of A is 10:50:00am and the current system time of B is 11:00:00am; ②B is a time server, and the system time of A needs to be synchronized with B; ③The time required for the data packet sent by A to reach B is 1s, that is, the network transmission time is 1s; The NTP working process is as follows: ①A first sends an NTP data packet to B. The data packet contains the timestamp of when it left A. The timestamp is t1 = 10:50:00am. ②After receiving the data packet, B immediately adds its own timestamp t2=11:00:01; ③B then sends the ntp data packet to A and adds its own timestamp t3=11: 00:02am; ④A successfully receives the response data packet and adds a new timestamp t4=10:50:03am; After the above process, A has enough information to calculate the transmission time δ and time deviation θ of the NTP data packet on the network, which are: The values ​​of δ and θ are related to the difference between t2, t1 and t3, t4, but have nothing to do with the difference between t3 and t2, that is, they have nothing to do with the time taken by the server to process the request. A uses these four timestamp data to calculate the network transmission time δ and the time deviation θ. When the time synchronization is completed, the client obtains the time difference between the local time and the server local time. The timestamps of the two audio samples will be based on the local time of the server. When the server receives the audio pair, it will compare the time difference between the two. When the time difference is less than the preset time difference threshold, the user login is deemed to be legal; otherwise, the user login is illegal. When the time difference of the audio pairs is determined to be legal, the server calculates their average power, discards those particularly quiet audio pairs, performs a fast Fourier transform on the time domain audio signal to obtain the frequency domain signal, and obtains the Fourier transform coefficient. The sum of the squares of the coefficients is the average power of the audio signal. After obtaining the average power of the audio pairs, they are compared with the pre-set average power threshold. If the average power of both audio samples is greater than the threshold, the similarity score determination continues; otherwise, the login is determined to be illegal.

4. According to the method for fusion voice mobile network multi-factor mobile phone security authentication of claim 1, It is characterized in that Browser-side structure: The browser is responsible for the authentication of the first and enhanced authentication factors. When the username and password are verified to be legal, the browser establishes a persistent connection with the server through the WebSocket protocol. The WebSocket protocol format is "ws: / / IP:Port" or "wss: / / IP:Port", where wss represents the websocket protocol for encrypted transmission. Wss is used to ensure the security of data transmission. After that, the WebRTC API is called to access the microphone to obtain the audio stream, and the AudioContext class is used to create an audio processing node to process the audio stream. The audio acquisition time is set to 3 seconds, and the audio is compressed and merged. While ensuring the audio quality, the dual-channel audio is compressed into a single channel. The sampling point bit is changed to 8 bits, the audio volume is compressed, and the transmission time is reduced. While the audio is being collected, the browser runs the time synchronization protocol to calculate the time difference with the server. The browser adjusts the timestamp of the audio sample according to the time difference to ensure that the sample timestamp uses the server local time as the standard time. After the audio acquisition is completed, the audio is sent to the server together with the timestamp.

5. According to claim 1, the method for fusion voice mobile network multi-factor mobile phone security authentication, It is characterized in that Remote attack security mode: The security of the multi-factor authentication mechanism that integrates sound lies in the fact that the attacker cannot guess the sound in the environment when attacking. Assume that x is the sample collected by the attacked mobile phone, and y is the forged sample submitted by the attacker. If the attacker wants to successfully launch an attack, the average power of the two audio signals must be higher than τ. db , and the two signals must be highly correlated, the two audio signal samples must satisfy P wr (x)>τ db , P wr (y)>τ db And S x，y >τ C , where l<l max , 1 is the time difference between two samples, l max is the maximum time difference allowed between two samples; This application sets the delay time value range to be 0 to 1 max Even if the attacker correctly guesses the noise in the environment and can submit similar audio samples, the two samples will not be smaller than l because the lag time does not meet max The application also rejects those samples whose average power is lower than the critical value τ. db Login to prevent fake attacks when the victim's environment is quiet; Quantifying the entropy of environmental noise, acoustically distinguishes between legitimate and fraudulent logins, even if the attacker successfully guesses the type of environment the victim is in.

6. According to claim 1, the method for fusion voice mobile network multi-factor mobile phone security authentication, It is characterized in that Multi-factor security authentication method integrating sound: the user opens the login page and enters the username and static password. The server retrieves the username from the database and verifies the legitimacy of the static password. If the username is verified to be legitimate, the server returns a prompt message to the client and establishes a WebSocket persistent connection. After the connection is established, the microphone is notified to collect audio. At the same time, the mobile phone and browser synchronize the clock with the server by running the time synchronization protocol. After the audio collection is completed, each device adjusts the timestamp of its sample by considering the clock difference with the server. The communication channel between the client and the server is established using the WebSocket wss protocol, and the audio message is encrypted and sent to the server. After receiving the sample, the server decrypts it. Then, the server checks the difference between the two sample timestamps. If the time difference is less than l max The fast Fourier transform is used to convert the time domain signal into the frequency domain signal and calculate their average power. If the user's surrounding environment is too quiet, the average power of the two audio samples is required to be greater than π db After that, the server divides the sample into 1 / 3 octaves and calculates the similarity score in each frequency domain. The similarity score S of the audio pair is x,y Equal to the average of the similarity scores in each frequency domain. If S x,y The final result is greater than the pre-set critical value π c ,The server determines that the computer and the mobile phone used by the user to log in are in the same place, confirms that the login is legal, and allows the login.

7. According to claim 1, the method for fusion voice mobile network multi-factor mobile phone security authentication, It is characterized in that Server-side authentication method: ① Establish a persistent connection with the client through WebSocket to achieve information interaction; ② Run the time synchronization protocol to calculate the time difference between the client and the server; ③ Determine the legitimacy of the time difference between two audio samples; ④ Calculate the average power of the audio sample; ⑤ Use 1 / 3 octave to perform frequency band filtering on the audio signal, calculate the similarity score of each frequency band, and obtain the similarity score of the two audios; (1) Establishing a WebSocket connection: After the user enters the username and password, the server performs a database search to verify the legitimacy of the user's identity. If the verification is successful, the client will initiate a WebSocket connection request. The server accepts the request and establishes a persistent connection with the client. Each connection is a Session object. The server uses a static List to save each Session object. Data interaction with the client is achieved using Session. (2) Run the time synchronization protocol: After the WebSocket connection is established, the server will send a notification to the client to request it to collect audio. While collecting audio, the client and the server will synchronize time. First, the client will send a data packet and record the send time originateTimestamp. After receiving it, the server will immediately record the receive time receiveTimestamp. Then, the time when the server returns the data packet is TransmitTimestamp, and the time when the data packet arrives at the client is destinationTimestamp. The client uses these four times to calculate the time difference between it and the server. (3) Time difference comparison: When the server receives the audio samples from the mobile phone and the browser, it uses the timestamps to compare their time differences. If the time difference between the two samples is greater than l max , it is considered that there may be an attack and the verification is illegal. If the time difference between the two samples is less than l max , the server adjusts the start time of the audio according to their time difference. (4) Calculation of average power: After the time difference comparison of the audio pairs is completed, the server calculates their average power. The average power of the audio is calculated in the frequency domain by using the ratio of the square of the absolute value of the time domain signal amplitude to the signal length.

8. According to claim 1, the method for fusion voice mobile network multi-factor mobile phone security authentication, It is characterized in that Browser-side authentication method: The browser side is written based on HTML5 and JavaScript. The encryption of audio data is implemented by wss using the SSL protocol. The navigator.getUserMedia() API of WebRTC is used to access the local microphone from the browser. The audio is collected, compressed, merged and formatted by creating a windows.AudioContext environment object. The AudioContext interface represents an audio processing environment composed of audio modules. Each module corresponds to an audio node AudioNode. The audio stream captured by the microphone is obtained through AudioContext, cached and compressed, and then audio data in wav format is generated. The browser is responsible for audio collection. When it receives the audio collection notification from the server, it will immediately request the microphone and create a source node from the microphone's input stream. Then, it uses AudioContext to create a processor node for recording.

9. According to claim 1, the method for fusion voice mobile network multi-factor mobile phone security authentication, It is characterized in that Android client authentication method: The Android client uses WebSocket to establish a persistent connection with the server, based on which time synchronization and data transmission are performed; Audio capture uses the AudioRecord class. An AudioRecord class is instantiated through the constructor public AudioRecord(int audioSource, int sampleRateInHz, int channelConfig, int audioFormat, int bufferSizeInBytes). The parameters represent the audio source, audio sampling frequency, channel settings, recording encoding format, and the size of the buffer where the captured data is saved. After recording is completed, the data is read from the buffer to generate a temporary file in wav format, which is then passed to the server by the send() method of WebSocket.

Images