Voiceprint authentication method and device and storage medium

Abstract

The present disclosure provides a voiceprint authentication method, device and storage medium, belonging to the technical field of computer. The voiceprint authentication method is based on recording first audio data by using the local microphone of the terminal, superimposes the signal generated by the vibration in the first audio data through the vibration motor of the terminal, and the server performs anti-fake detection on the first audio data based on whether the signal generated by the vibration in the first audio data conforms to the vibration strategy, and performs voiceprint authentication if the anti-fake detection passes. Since the vibration motor is installed in the terminal, only the local microphone of the terminal can record the signal generated by the vibration of the vibration motor, therefore, the above process can ensure that the audio data used for voiceprint authentication is the audio data recorded by the local microphone, and can avoid the case that the attacker bypasses the local microphone and directly injects fake audio data to the collection layer, thereby improving the reliability of voiceprint authentication.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a voiceprint authentication method, apparatus and storage medium. Background Technology

[0002] Voiceprint authentication is a biometric authentication technology that verifies identity based on the structure of a speaker's vocal organs, vocal tract characteristics, and pronunciation habits. It has broad application prospects in scenarios such as mobile account security, financial risk control, and real-name verification.

[0003] Currently, in the voiceprint authentication process, the terminal sends audio data input by the user through its microphone to the server. The server receives the audio data and analyzes its spectral characteristics, phase noise, and acoustic features to determine whether the audio data is synthesized speech or a recorded playback. If the server determines that the audio data is synthesized speech or a recorded playback, it refuses to perform voiceprint authentication. If the server determines that the audio data is not synthesized speech or a recorded playback, it processes the audio data using a backend voiceprint model to authenticate the user based on the audio data.

[0004] While the aforementioned voiceprint authentication scheme can detect whether the audio data input from the local microphone is synthesized speech or a recorded playback, it fails when an attacker bypasses the local microphone and injects forged audio data directly into the server's audio data acquisition layer, resulting in low reliability of voiceprint authentication. Summary of the Invention

[0005] This disclosure provides a voiceprint authentication method, apparatus, and storage medium to improve the reliability of voiceprint authentication. The technical solution is as follows: According to one aspect of the embodiments of this disclosure, a voiceprint authentication method is provided, the method comprising: Receive first audio data from the terminal, the first audio data including external sound recorded by the terminal’s native microphone and signals generated by the vibration of the terminal’s vibration motor; If the signal generated by the vibration of the vibration motor in the first audio data conforms to the vibration strategy, voiceprint authentication is performed on the first audio data.

[0006] In some embodiments, the above method further includes: Receive second audio data from the terminal, which is recorded through the terminal's external audio input device; Perform a consistency check on the first audio data and the second audio data; If the first audio data and the second audio data fail the consistency check, the voiceprint authentication process will be stopped.

[0007] In some embodiments, the first audio data and the second audio data each have a corresponding channel marker, and the channel marker of the first audio data indicates that the first audio data is audio data acquired through the main audio acquisition channel.

[0008] In some embodiments, the above method further includes: Receive the vibration timestamp corresponding to the first audio data from the terminal. The vibration timestamp indicates the time point in the first audio data when the vibration signal appears. If the signal generated by the vibration motor in the first audio data conforms to the vibration strategy, the voiceprint authentication of the first audio data includes: Based on the vibration timestamp, vibration acoustic features are extracted from the time points indicated by the vibration timestamps in the first audio data. If the vibration acoustic characteristics match the vibration strategy, voiceprint authentication is performed on the first audio data.

[0009] In some embodiments, the aforementioned vibration acoustic feature is the energy peak of the first audio data at the time indicated by the vibration timestamp.

[0010] In some embodiments, the above method further includes: Receive auxiliary data from the terminal, the auxiliary data including at least one of the following: accelerometer data of the terminal and gyroscope data of the terminal; Based on auxiliary data, verify whether the signal generated by the vibration of the vibration motor in the first audio data matches the vibration information indicated by the auxiliary data; If the signal generated by the vibration motor in the first audio data does not match the vibration information indicated by the auxiliary data, the voiceprint authentication process will be stopped.

[0011] In some embodiments, the above method further includes: In response to an operation request from the terminal to trigger voiceprint authentication, a vibration strategy is randomly generated, and a vibration strategy and audio acquisition command are sent to the terminal. The vibration strategy includes at least one of the following: vibration trigger time, vibration amplitude, duration of each vibration, vibration interval, vibration frequency, and vibration mode.

[0012] In some embodiments, the above-mentioned voiceprint authentication of the first audio data includes: Vibration frequency band filtering is performed on the first audio data. Vibration frequency band filtering is used to filter out the signal generated by the vibration of the vibration motor from the first audio data. Voiceprint authentication is performed on the first audio data after filtering out the vibration frequency band.

[0013] According to another aspect of the embodiments of this disclosure, a voiceprint authentication method is provided, the method comprising: In response to the audio acquisition command, the terminal's local microphone is activated to record audio, and the terminal's vibration motor is triggered to vibrate according to the vibration strategy. After recording is completed, the first audio data is sent to the server. The first audio data includes external sounds recorded through the local microphone and signals generated by the vibration motor. The first audio data is used for voiceprint authentication.

[0014] In some embodiments, when the terminal is connected to at least one external audio input device, the above method further includes: Recording is performed via at least one external audio input device when it is detected that the terminal is connected to at least one external audio input device. After recording is completed, the recorded second audio data is sent to the server. The second audio data is used for voiceprint authentication.

[0015] In some embodiments, the first audio data and the second audio data each have a corresponding channel marker, and the channel marker of the first audio data indicates that the first audio data is audio data acquired through the main audio acquisition channel.

[0016] In some embodiments, the above method further includes: Send the vibration timestamp corresponding to the first audio data to the server. The vibration timestamp indicates the time point when the vibration signal appears in the first audio data.

[0017] In some embodiments, the vibration strategy is randomly generated, and the vibration strategy includes at least one of the following: vibration trigger time, vibration amplitude, duration of each vibration, vibration interval, vibration frequency, and vibration mode.

[0018] In some embodiments, the above method further includes: During the recording process, auxiliary data of the terminal is collected. The auxiliary data includes at least one of the following: accelerometer data of the terminal and gyroscope data of the terminal. The auxiliary data is used to verify whether the signal generated by the vibration of the vibration motor in the first audio data matches the actual vibration of the terminal. Send the collected auxiliary data to the server.

[0019] According to another aspect of the present disclosure, a voiceprint authentication device is provided, the device comprising: An audio receiving module is used to receive first audio data from the terminal, the first audio data including external sound recorded by the terminal's local microphone and signals generated by the vibration of the terminal's vibration motor; The voiceprint authentication module is used to perform voiceprint authentication on the first audio data if the signal generated by the vibration of the vibration motor in the first audio data conforms to the vibration strategy.

[0020] In some embodiments, the audio receiving module described above is further configured to receive second audio data from the terminal, the second audio data being recorded by an external audio input device of the terminal; The above-mentioned device also includes: The consistency verification module is used to verify the consistency between the first audio data and the second audio data; if the first audio data and the second audio data fail the consistency verification, the voiceprint authentication process is stopped.

[0021] In some embodiments, the first audio data and the second audio data each have a corresponding channel marker, and the channel marker of the first audio data indicates that the first audio data is audio data acquired through the main audio acquisition channel.

[0022] In some embodiments, the above-described apparatus further includes: The timestamp receiving module is used to receive the vibration timestamp corresponding to the first audio data from the terminal. The vibration timestamp indicates the time point when the vibration signal appears in the first audio data. The aforementioned voiceprint authentication module is used for: Based on the vibration timestamp, vibration acoustic features are extracted from the time points indicated by the vibration timestamps in the first audio data. If the vibration acoustic characteristics match the vibration strategy, voiceprint authentication is performed on the first audio data.

[0023] In some embodiments, the aforementioned vibration acoustic feature is the energy peak of the first audio data at the time indicated by the vibration timestamp.

[0024] In some embodiments, the above-described apparatus further includes: An auxiliary data receiving module is used to receive auxiliary data from the terminal, the auxiliary data including at least one of the following: accelerometer data of the terminal and gyroscope data of the terminal; The auxiliary verification module is used to verify, based on auxiliary data, whether the signal generated by the vibration of the vibration motor in the first audio data conforms to the vibration information indicated by the auxiliary data; if the signal generated by the vibration of the vibration motor in the first audio data does not conform to the vibration information indicated by the auxiliary data, the voiceprint authentication is stopped.

[0025] In some embodiments, the above-described apparatus further includes: The strategy generation module is used to respond to the operation request from the terminal to trigger voiceprint authentication, randomly generate a vibration strategy, and send the vibration strategy and audio acquisition command to the terminal. The vibration strategy includes at least one of the following: vibration trigger time, vibration amplitude, duration of each vibration, vibration interval, vibration frequency, and vibration mode.

[0026] In some embodiments, the above-described voiceprint authentication module is used for: Vibration frequency band filtering is performed on the first audio data. Vibration frequency band filtering is used to filter out the signal generated by the vibration of the vibration motor from the first audio data. Voiceprint authentication is performed on the first audio data after filtering out the vibration frequency band.

[0027] According to another aspect of the present disclosure, a voiceprint authentication device is provided, the device comprising: The recording module is used to respond to the audio acquisition command, activate the terminal's local microphone to record, and trigger the terminal's vibration motor to vibrate according to the vibration strategy. The audio sending module is used to send the first audio data recorded to the server after the recording is completed. The first audio data includes external sounds recorded through the local microphone and signals generated by the vibration motor. The first audio data is used for voiceprint authentication.

[0028] In some embodiments, the recording module described above is further configured to record audio through at least one external audio input device when it is detected that the terminal is connected to at least one external audio input device; The audio sending module is also used to send the recorded second audio data to the server after the recording is completed. The second audio data is used for voiceprint authentication.

[0029] In some embodiments, the first audio data and the second audio data each have a corresponding channel marker, and the channel marker of the first audio data indicates that the first audio data is audio data acquired through the main audio acquisition channel.

[0030] In some embodiments, the above-described apparatus further includes: The timestamp sending module is used to send the vibration timestamp corresponding to the first audio data to the server. The vibration timestamp indicates the time point when the vibration signal appears in the first audio data.

[0031] In some embodiments, the vibration strategy is randomly generated, and the vibration strategy includes at least one of the following: vibration trigger time, vibration amplitude, duration of each vibration, vibration interval, vibration frequency, and vibration mode.

[0032] In some embodiments, the above-described apparatus further includes: An auxiliary data acquisition module is used to acquire auxiliary data of the terminal during the recording process. The auxiliary data includes at least one of the following: accelerometer data of the terminal and gyroscope data of the terminal. The auxiliary data is used to verify whether the signal generated by the vibration of the vibration motor in the first audio data matches the actual vibration of the terminal. The auxiliary data sending module is used to send the collected auxiliary data to the server.

[0033] According to another aspect of the present disclosure, a server is provided, the server comprising: one or more processors; a memory for storing processor-executable program code; wherein the processor is configured to execute the program code to implement the above-described voiceprint authentication method.

[0034] According to another aspect of the present disclosure, a terminal is provided, the terminal comprising: one or more processors; a memory for storing processor-executable program code; wherein the processor is configured to execute the program code to implement the above-described voiceprint authentication method.

[0035] According to another aspect of the present disclosure, a computer-readable storage medium is provided, wherein program code in the computer-readable storage medium is executed by a server's processor, enabling the server to perform the above-described voiceprint authentication method.

[0036] According to another aspect of the present disclosure, a computer-readable storage medium is provided, wherein program code in the computer-readable storage medium is executed by a terminal's processor, enabling the terminal to perform the aforementioned voiceprint authentication method.

[0037] According to another aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a server's processor, implements the above-described voiceprint authentication method.

[0038] According to another aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a terminal's processor, implements the above-described voiceprint authentication method.

[0039] The voiceprint authentication method provided in this disclosure, based on recording first audio data using the terminal's local microphone, superimposes a vibration signal onto the first audio data using the terminal's vibration motor. The server performs anti-counterfeiting detection on the first audio data based on whether the vibration signal conforms to a vibration strategy, and performs voiceprint authentication if the anti-counterfeiting detection passes. Since the vibration motor is installed in the terminal, only the terminal's local microphone can record the signal generated by the vibration motor. Therefore, the above process, based on the vibration signal for anti-counterfeiting detection, ensures that the audio data used for voiceprint authentication is recorded by the local microphone. This prevents attackers from bypassing the local microphone and directly injecting forged audio data into the acquisition layer, thereby improving the reliability of voiceprint authentication.

[0040] Based on the implementation methods provided in the above aspects, this disclosure can be further combined to provide more implementation methods. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram illustrating the implementation environment of a voiceprint authentication method according to an exemplary embodiment; Figure 2 This is a schematic diagram illustrating an injection attack according to an exemplary embodiment; Figure 3 This is a data interaction diagram illustrating a voiceprint authentication method according to an exemplary embodiment; Figure 4 This is a flowchart illustrating a voiceprint authentication method according to an exemplary embodiment; Figure 5 This is a flowchart illustrating another voiceprint authentication method according to an exemplary embodiment; Figure 6 This is a data interaction diagram illustrating another voiceprint authentication method according to an exemplary embodiment; Figure 7 This is a schematic diagram of audio data acquisition according to an exemplary embodiment; Figure 8 This is a block diagram illustrating a voiceprint authentication device according to an exemplary embodiment; Figure 9 This is a block diagram illustrating another voiceprint authentication device according to an exemplary embodiment; Figure 10 This is a schematic diagram of the structure of a server according to an exemplary embodiment; Figure 11 This is a schematic diagram of the structure of a terminal according to an exemplary embodiment. Detailed Implementation

[0043] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0044] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0045] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the audio data involved in this disclosure were all obtained with full authorization.

[0046] Figure 1 This is a schematic diagram illustrating an implementation environment for a voiceprint authentication method according to an exemplary embodiment. See also... Figure 1 The implementation environment specifically includes: terminal 101 and server 102.

[0047] In some embodiments, terminal 101, also referred to as a mobile device or mobile terminal, can be at least one of the following: smartphone, smartwatch, desktop computer, laptop, MP3 (Moving Picture Experts Group Audio Layer III) player, MP4 (Moving Picture Experts Group Audio Layer IV) player, and laptop computer. Applications are installed and running on terminal 101, such as social networking applications, shopping applications, multimedia applications, office applications, or game applications, etc., which are not limited in this embodiment. When a user operates an application through terminal 101, an operation request is triggered, which instructs the user to perform corresponding operations on the application. Terminal 101 can connect to server 102 via a wireless network or wired network, and can then send operation requests to server 102, which performs corresponding operations based on the operation requests. During this process, if server 102 detects that an operation request triggers voiceprint authentication, it sends an audio acquisition command to terminal 101. In response to the audio acquisition command, terminal 101 acquires audio data and returns the audio data to server 102. Server 102 performs voiceprint authentication based on the audio data, and if the voiceprint authentication is successful, executes the operation indicated in the aforementioned operation request.

[0048] Voiceprint authentication, a biometric authentication technology based on the structure of a speaker's vocal organs, vocal tract characteristics, and pronunciation habits, offers advantages such as being contactless, having natural interaction, and requiring low device specifications. It has broad application prospects in scenarios such as mobile account security, financial risk control, and real-name verification. For example, when a user operates an application running on terminal 101 and wants to log in to an account or execute a transaction, terminal 101 responds to the user's operation by sending a login request or transaction request to server 102 (both login and transaction requests are operation requests). Upon receiving the login or transaction request, server 102 detects that the request triggers voiceprint authentication and sends an audio acquisition command to terminal 101. Terminal 101 responds to the audio acquisition command by acquiring audio data and returning the audio data to server 102. Server 102 performs voiceprint authentication based on this audio data to verify the user's identity. If the voiceprint authentication is successful, the user logs in to the account based on the login request or executes a transaction based on the transaction request. This disclosure does not limit the scope of the embodiments described herein.

[0049] Terminal 101 generally refers to one of a plurality of terminals; this embodiment uses terminal 101 as an example. Those skilled in the art will understand that the number of terminals can be more or less. For example, there may be several terminals, or dozens or hundreds of terminals, or even more. This disclosure does not limit the number of terminals or the type of device.

[0050] In some embodiments, server 102 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), big data, and artificial intelligence platforms. Server 102 is used to provide background services for the above applications, such as executing operation requests, issuing audio collection instructions to terminal 101 when the operation request triggers voiceprint authentication, and performing voiceprint authentication based on the audio data collected by terminal 101. In some embodiments, server 102 undertakes the main computing work, and terminal 101 undertakes the secondary computing work; or, server 102 undertakes the secondary computing work, and terminal 101 undertakes the main computing work; or, server 102 and terminal 101 collaborate on computing using a distributed computing architecture.

[0051] The audio data acquisition process of the aforementioned terminal is the starting point for the voiceprint authentication system and also the most vulnerable environment in the entire security chain. Once an attacker can inject forged audio data into the acquisition environment, no matter how advanced the backend voiceprint model is, it will be rendered useless. To address this issue, current mainstream anti-spoofing detection solutions mainly include: (1) Detection based on audio features: Detection is performed by analyzing the spectral characteristics, phase noise, and acoustic features of audio data to determine whether the audio data is fake audio data such as TTS (Text-To-Speech) or recording playback. For example, by detecting features such as MFCC (Mel-Frequency Cepstral Coefficients), LPC (Linear Predictive Coding), and CQT (Constant-Q Transform) of audio data, the differences between fake audio data and real audio data can be detected, thereby identifying fake audio data. Another example is the use of end-to-end detection models based on deep learning (such as RawNet and AASIST) to process audio data to determine whether it is fake audio data.

[0052] (2) Detection based on deep learning detection models: such as SafeEar (ACM CCS 2024) (a content privacy-preserving speech forgery detection framework) decouples semantic information and acoustic information, and only uses prosody and timbre features to perform deep forgery detection, thereby reducing EER (Equal Error Rate) to 2.02% while achieving privacy protection.

[0053] (3) Multimodal liveness detection: Combines multiple dimensions such as lip movement, face and audio-visual synchronization for joint judgment.

[0054] (4) Text-dependent challenge-response voiceprint authentication: The server sends random text in real time and asks the user to read it aloud. The recording playback attack is prevented by detecting whether the user's voice content matches the text.

[0055] The above detection schemes all detect whether the audio data itself is forged. However, if an attacker bypasses the terminal's native microphone, such as... Figure 2 As shown, when forged audio data (such as synthesized speech, recording playback, or remote audio streams) is directly injected into the server's audio data acquisition layer through external audio input devices (such as Bluetooth audio devices, wired audio devices, or OTG (On-The-Go) external devices), or when audio streams or audio files are directly transmitted at the terminal's system layer or network transmission layer, bypassing the hardware configuration, the acquisition layer considers the acquired audio data to be real (non-forged). The above detection schemes fail, resulting in low reliability of voiceprint authentication.

[0056] To address the aforementioned issue of low reliability in voiceprint authentication, this disclosure provides a voiceprint authentication method based on local microphone acquisition and dynamic vibration anti-spoofing. By combining local microphone acquisition with dynamic vibration anti-spoofing, it ensures that the audio data used by the server for voiceprint authentication is audio data acquired by the local microphone, rather than audio data injected by external audio input devices. This solves the problems of external audio input device injection attacks, recording playback attacks, and real-time synthesized speech attacks that exist in the current terminal audio data acquisition stage, thereby improving the reliability of voiceprint authentication.

[0057] The voiceprint authentication method provided in the embodiments of this disclosure is described below.

[0058] Figure 3 This is a data interaction diagram illustrating a voiceprint authentication method according to an exemplary embodiment, such as... Figure 3 As shown, the method includes the following steps.

[0059] In step 301, in response to the audio acquisition command, the terminal activates its local microphone to record audio and triggers the terminal's vibration motor to vibrate according to the vibration strategy.

[0060] The terminal's built-in microphone is the terminal's native microphone. The vibration motor is located inside the terminal, and only the terminal's built-in microphone can record the sound generated by the vibration motor. External audio input devices connected via wired or wireless means cannot record the sound generated by the vibration motor. The sound generated by the vibration motor is reflected in the audio data as a vibration signal. The vibration strategy includes at least one of the following: vibration trigger time, vibration amplitude, duration of each vibration, vibration interval, vibration frequency, and vibration mode.

[0061] In this embodiment of the disclosure, the terminal responds to an audio acquisition command by activating its local microphone to record audio. During the recording process, the terminal's vibration motor is triggered to vibrate according to a vibration strategy. Correspondingly, the terminal's local microphone synchronously captures the physical vibration acoustic signal generated when the vibration motor is working during the recording process, and this signal is naturally superimposed on the sound track recorded by the local microphone.

[0062] In step 302, after the terminal finishes recording, it sends the first audio data recorded to the server. The first audio data includes external sounds recorded through the local microphone and signals generated by the vibration motor. The first audio data is used for voiceprint authentication.

[0063] The external sounds recorded by the local microphone include at least one of the following: target human voice, non-target human voice, background noise, and physiological noise. Target human voice is the sound emitted by the user; non-target human voice is the sound emitted by people other than the user; background noise, also known as ambient sound, is non-target sound in the recording environment, such as air conditioner noise, fan noise, and traffic noise. Physiological noise includes, for example, breathing sounds, but this embodiment does not limit the scope of the invention.

[0064] In this embodiment of the disclosure, after the terminal completes the recording, it sends the first audio data recorded to the server so that the server can perform voiceprint authentication based on the first audio data.

[0065] In step 303, the server receives the first audio data from the terminal. If the signal generated by the vibration of the vibration motor in the first audio data conforms to the vibration strategy, the server performs voiceprint authentication on the first audio data.

[0066] In this embodiment of the disclosure, the server receives first audio data from the terminal and extracts the signal generated by the vibration of the vibration motor from the first audio data. If the extracted signal conforms to the vibration strategy, the server extracts the voiceprint features of the first audio data and compares the voiceprint features with the features in the voiceprint database to obtain the voiceprint authentication result.

[0067] The voiceprint authentication method provided in this disclosure, based on recording first audio data using the terminal's local microphone, superimposes a vibration signal onto the first audio data using the terminal's vibration motor. The server performs anti-counterfeiting detection on the first audio data based on whether the vibration signal conforms to a vibration strategy, and performs voiceprint authentication if the anti-counterfeiting detection passes. Since the vibration motor is installed in the terminal, only the terminal's local microphone can record the signal generated by the vibration motor. Therefore, the above process, based on the vibration signal for anti-counterfeiting detection, ensures that the audio data used for voiceprint authentication is recorded by the local microphone. This prevents attackers from bypassing the local microphone and directly injecting forged audio data into the acquisition layer, thereby improving the reliability of voiceprint authentication.

[0068] The above Figure 3 The diagram shown is only the basic process of this disclosure. The following is a further explanation of the solution provided in this disclosure based on a specific implementation method. Figure 4 This is a flowchart of a voiceprint authentication method provided in an embodiment of this disclosure, such as... Figure 4As shown, in response to the audio acquisition command and vibration strategy from the server, the terminal activates its local microphone to record audio. If an external audio input device is detected connected to the terminal, it simultaneously records audio through the external audio input device. During this recording process, the terminal triggers vibration according to the vibration strategy, and the local microphone captures the vibration audio characteristics, thus superimposing vibration sound patterns onto the audio data recorded by the local microphone. Through this recording process, the terminal acquires audio data from both the local microphone track and the external audio input device track. It performs dual-channel audio merging on these two sets of audio data to obtain a single audio file. The terminal then sends this audio file, along with the channel markers for each audio data point within the file and a vibration timestamp, to the server. The channel markers indicate the track from which the audio data originated, and the vibration timestamps indicate the time points in the audio data recorded by the local microphone where vibration signals appeared.

[0069] After the server receives the audio file, it executes the following: Figure 5 The verification process is shown below. It is illustrative, as... Figure 5 As shown, the server receives an audio file, channel markers for each audio data point within the audio file, and vibration timestamps. The server performs consistency checks on the audio data from different tracks within the audio file and verifies whether the local microphone track is the primary channel. If the local microphone track is the primary channel, and the audio data from different tracks passes the consistency check, the server extracts vibration acoustic features from the audio data from the local microphone track based on the vibration timestamps. It then performs vibration acoustic feature verification based on these features to check if the vibrations in the audio data conform to the vibration policy. If the vibrations in the audio data conform to the vibration policy, the server extracts voiceprint features from the audio data, compares the extracted features with features in the voiceprint database, and obtains and outputs the voiceprint authentication result.

[0070] The following is combined Figure 6 Taking the audio data recorded by the local microphone as the first audio data and the audio data recorded by an external audio input device as the second audio data as an example, the above... Figure 4 and Figure 5 The process shown will be described in detail.

[0071] Figure 6 This is a data interaction diagram illustrating a voiceprint authentication method according to an exemplary embodiment, such as... Figure 6 As shown, the method includes the following steps.

[0072] In step 601, the terminal responds to the user's operation by sending an operation request to the server.

[0073] The user's operation may include clicking the screen, entering text, or entering voice, etc., and this embodiment does not limit this. The user's operation instructs to perform operations such as account login, item transaction, or real-name verification, and correspondingly, the operation request instructs to perform the operation such as account login, item transaction, or real-name verification.

[0074] In this embodiment of the disclosure, the terminal responds to the user's operation, generates an operation request, and sends the operation request to the server.

[0075] In step 602, the server responds to the operation request by triggering voiceprint authentication, randomly generates a vibration strategy, and sends an audio acquisition command and vibration strategy to the terminal.

[0076] The voiceprint authentication method provided in this disclosure offers a dynamic vibration anti-counterfeiting mechanism. This mechanism adds an anti-counterfeiting identifier to the external sound collected by the local microphone by superimposing a vibration signal generated based on a vibration strategy. The vibration strategy includes at least one of the following: vibration trigger time, vibration amplitude, duration of each vibration, vibration interval, vibration frequency, and vibration mode. Taking a vibration strategy that includes a vibration parameter combination and a vibration trigger time strategy as an example, the vibration parameter combination includes vibration amplitude, duration of each vibration, and vibration interval, while the vibration trigger time strategy indicates the vibration trigger time. The design of the vibration strategy follows the following principles to ensure that the vibration acoustic characteristics are not prepared in advance or simulated in real time.

[0077] (1) Randomization of vibration amplitude: The vibration amplitude is generated randomly. The vibration amplitude is divided into multiple levels, such as the weak level (motor operates at low power) and the strong level (motor operates at high power). Different levels correspond to different acoustic amplitude characteristics. The server identifies the vibration amplitude level through spectrum analysis.

[0078] (2) Randomization of vibration duration: The duration of each vibration is randomly generated. The duration of each vibration is divided into multiple levels, such as short level (e.g., less than 50ms (millisecond)) and long level (e.g., more than 100ms). The duration of vibration corresponds to the temporal duration of the vibration acoustic features in the audio data.

[0079] (3) Randomization of vibration interval: The interval between multiple vibrations is randomly generated (e.g., randomly generated within 200ms to 2000ms), making it impossible for attackers to simulate vibration by a fixed beat.

[0080] (4) Randomized vibration triggering time: Vibration can be triggered at any time during the recording process, including at the beginning of the recording, at random times in the middle of the recording, and at the end of the recording. It can be triggered multiple times. The triggering time and the number of triggers are randomly and dynamically determined, making it impossible for attackers to predict.

[0081] (5) Uniqueness of parameter combination: The vibration amplitude, vibration duration, vibration interval and vibration trigger time used for each voiceprint authentication are generated by the server in real time, with sufficient random entropy to ensure that historical parameters cannot be reused.

[0082] (6) Vibration parameter dimensions can be expanded: In addition to vibration amplitude, duration of each vibration and vibration interval, vibration frequency (motor speed) and vibration mode (such as single, pulse or gradual) can be added to further improve the recognition accuracy of vibration acoustic features and the attack difficulty of voiceprint recognition.

[0083] In this embodiment, the server responds to an operation request that triggers voiceprint authentication, randomly generates a vibration strategy, and sends an audio acquisition command and the vibration strategy used for this voiceprint authentication to the terminal. The vibration strategy issued for each authentication is different and cannot be predicted in advance by the terminal. For example, an operation request triggering voiceprint authentication might occur when the server detects that the operation requested requires voiceprint authentication. For instance, if the server detects that the requested operation is real-name verification, and the server has pre-recorded operations requiring voiceprint authentication, the server retrieves the record and finds that the real-name verification operation requires voiceprint authentication, thus triggering the voiceprint authentication. The voiceprint authentication method provided in this embodiment dynamically issues random vibration strategies by the server, making the characteristics of the vibration signals generated by the terminal during audio data recording unpredictable and preventing attackers from preparing in advance or simulating vibrations in real time.

[0084] The above process is illustrated using the example of a server generating and sending a vibration strategy to the terminal. In some embodiments, the terminal, in response to the following audio acquisition command, generates its own vibration strategy and triggers a vibration motor to vibrate according to that strategy. Accordingly, after recording is completed, the terminal not only returns the acquired audio data to the server but also sends the vibration strategy to the server so that the server can verify the audio data based on the vibration strategy.

[0085] In step 603, the terminal responds to the audio acquisition command by detecting the terminal. If an external audio input device is detected, the terminal records audio through its own microphone and the external audio input device, and triggers the terminal's vibration motor to vibrate according to the vibration strategy.

[0086] External audio input devices include, for example, Bluetooth audio devices, wired audio devices, and OTG external devices. Bluetooth audio devices include, for example, Bluetooth headsets and Bluetooth speakers; wired audio devices include, for example, 3.5mm (millimeter) headphones and USB (Universal Serial Bus) sound cards; and OTG external devices include, for example, external microphones and audio capture cards. The terminal may have one or more built-in microphones. When activating a built-in microphone for recording, the terminal may activate one microphone or multiple microphones for recording in parallel; this disclosure does not limit this approach.

[0087] In this embodiment, in response to an audio acquisition command, the terminal detects the connection status of an external audio input device before initiating audio data acquisition. If an external audio input device is detected, the terminal's local microphone is set as the primary audio acquisition channel to ensure that audio data acquired through the local microphone is prioritized by the server. Furthermore, when the external audio input device is enabled, recording occurs simultaneously with the local microphone, forming a dual-channel recording (local microphone track and external audio input device track). Simultaneously, based on the vibration strategy issued by the server, the terminal triggers the vibration motor according to the vibration amplitude, duration, and interval specified in the vibration strategy by calling the vibration motor API (Application Programming Interface). During recording, the terminal's local microphone simultaneously captures the physical vibration acoustic signal generated by the vibration motor, which is naturally superimposed on the sound track recorded by the local microphone.

[0088] The above process provides a method to ensure the security of the sound acquisition channel by activating the local microphone for recording, ensuring that the acquired audio data definitely contains the audio data recorded by the local microphone.

[0089] The audio data recorded via the local microphone is designated as the first audio data, and the audio data recorded via an external audio input device is designated as the second audio data. Each audio data set has a corresponding channel marker, indicating which track it originated from. For example, the channel marker for the first audio data indicates that it was acquired via the main audio acquisition channel (the local microphone track), and the channel marker for the second audio data indicates that it was acquired via an external audio input device.

[0090] Step 603 above is illustrated using the example of a terminal connected to an external audio input device. In some embodiments, the terminal is not connected to any external audio input device. When the terminal detects that no external audio input device is connected, it directly activates its local microphone to record. During recording, the terminal controls the vibration motor to vibrate according to the following vibration strategy: one weak vibration (vibrating at the beginning of recording, vibration duration 80ms), one strong vibration (vibrating at a random moment in the middle of recording, vibration duration 120ms), and one weak vibration (vibrating at the end of recording, vibration duration 60ms). The interval between the first and second vibrations is 800ms, and the interval between the second and third vibrations is 600ms. The local microphone captures external sounds (such as user voice) and the sounds generated by the three vibrations during recording. After recording is complete, the terminal sends the recorded audio data and vibration timestamps to the server. The vibration timestamp indicates the time point in the audio data where the vibration signal appears. The server extracts vibration acoustic features from the time point indicated by the vibration timestamp in the audio data. The vibration acoustic features include vibration amplitude, duration of each vibration, and vibration interval. The server compares the vibration acoustic features with the vibration amplitude, duration of each vibration, and vibration interval indicated by the vibration strategy. If the vibration acoustic features match the vibration strategy, the audio data passes the vibration acoustic feature verification. The server then performs voiceprint comparison based on the audio data to complete voiceprint authentication. The vibration acoustic feature verification process and the voiceprint authentication process are similar to the related content described below, and will not be repeated here.

[0091] During the above process, the vibration signal collected by the terminal's local microphone is as follows: Figure 7 As shown. Figure 7 This is a schematic diagram of audio data acquisition according to an exemplary embodiment. For example... Figure 7 As shown, the server sends an audio acquisition command and vibration policy to the terminal. In response to the audio acquisition command, the terminal activates its local microphone to record. According to the vibration policy, it controls the vibration motor to perform a weak and short vibration at the start of recording, a strong and long vibration at a certain point during recording, and a weak and short vibration at the end of recording. The local microphone captures these vibrations and superimposes the vibration signals onto the acquired external sound to obtain audio data. The terminal sends the audio data and vibration timestamps to the server. The server performs verification and voiceprint authentication based on the received audio data.

[0092] The above process, when no external device is connected, uses only the local microphone for single-channel recording, reducing the amount of data uploaded to the server, improving transmission efficiency, and thus improving voiceprint authentication efficiency.

[0093] Step 603 above uses the terminal's local microphone to collect audio data and detects and controls external audio input devices, blocking injection attack paths from the acquisition layer. Furthermore, the vibration process shown in step 603 provides natural notifications to the user of the recording status at the start, middle, and end of the acquisition process, balancing security and user experience. Additionally, the voiceprint authentication method provided in this embodiment is entirely based on the terminal's existing hardware (local microphone + vibration motor), requiring no additional sensors or other devices for the terminal, resulting in extremely low deployment costs.

[0094] In some embodiments, during the recording process, the terminal synchronously acquires auxiliary data and sends the auxiliary data to the server after the recording is completed. The auxiliary data includes at least one of the following: the terminal's IMU (Accelerometer) data and the terminal's gyroscope data. This auxiliary data is used to verify whether the signal generated by the vibration motor in the first audio data matches the actual vibration of the terminal.

[0095] In step 604, after recording is completed, the terminal sends an audio file and a vibration timestamp to the server. The audio file includes first audio data recorded through the local microphone, second audio data recorded through an external audio input device, and channel markers for each audio data.

[0096] The first audio data includes external sounds recorded by the local microphone and signals generated by the vibration of the vibration motor. Both the first and second audio data are used for voiceprint authentication. The channel markers for the first audio data indicate that it originates from the local microphone track (main audio acquisition channel), and the channel markers for the second audio data indicate that it originates from an external audio input device track.

[0097] In this embodiment, after recording is completed, the terminal merges the first and second audio data into a single audio file by performing dual-channel audio merging. This audio file is then sent to the server, along with the channel markers for each audio data item. Additionally, the terminal sends a vibration timestamp corresponding to the first audio data to the server. This vibration timestamp is implemented as a list of time points, recording the precise timestamps at which the vibration signal appears in the first audio data, which are also the precise timestamps at which the terminal's vibration is triggered.

[0098] In step 605, the server performs a consistency check on the first audio data and the second audio data in the audio file, and the first audio data and the second audio data pass the consistency check.

[0099] The consistency check between the first and second audio data involves comparing their acoustic features. If there is a significant difference in the acoustic features of the first and second audio data, the server determines that an injection attack has occurred and refuses to proceed with voiceprint authentication. If there is no significant difference in the acoustic features of the first and second audio data, the server determines that no injection attack has occurred, the first and second audio data pass the consistency check, and the server proceeds to the subsequent vibration acoustic feature verification process. A significant difference in the acoustic features of the first and second audio data could be, for example, the presence of the user's voice in the first audio data but not in the second audio data. If there is a significant difference in the acoustic features of the first and second audio data, it indicates that the local microphone used to collect the first audio data and the external audio input device used to collect the second audio data are not in the same space. In this case, it is highly likely that an attacker has launched an injection attack from a remote location, and continuing voiceprint authentication poses a risk; the server should refuse to execute the subsequent voiceprint authentication process.

[0100] In this embodiment, the server extracts first audio data and second audio data from an audio file. It performs feature extraction on the first audio data to obtain its acoustic features, and then performs feature extraction on the second audio data to obtain its acoustic features. The server compares the acoustic features of the first and second audio data. If there is no significant difference between the acoustic features of the first and second audio data, the first and second audio data pass a consistency check, and the server proceeds to step 606. For example, if a terminal is connected to a Bluetooth headset, the terminal detects the presence of the Bluetooth headset and simultaneously activates both its own microphone and the Bluetooth headset for dual-channel recording, obtaining and sending dual-channel audio to the server. After receiving the dual-channel audio, the server extracts and compares the acoustic features of the two audio data sets. If no significant difference is found between the two acoustic features, the consistency check is passed, and voiceprint authentication continues.

[0101] The above process involves the server identifying the first audio data and the second audio data through channel marking. In some embodiments, the first audio data recorded by the local microphone contains vibration acoustic features that conform to the vibration strategy. However, the external audio input device is physically decoupled from the terminal and cannot sense vibration. The second audio data recorded by the external audio input device does not contain vibration acoustic features. Therefore, vibration acoustic features can be used as a physical channel watermark, which can be used not only for anti-counterfeiting verification but also as a unique marker of the source of the audio data. Accordingly, the server identifies the first audio data and the second audio data based on whether the audio data contains vibration acoustic features. This disclosure does not limit this aspect.

[0102] The above process is illustrated using the example of the first and second audio data passing the consistency check. In some embodiments, if the first and second audio data fail the consistency check, the server stops performing voiceprint authentication. For example, if the audio data recorded through the Bluetooth headset and the audio data recorded through the local microphone have different sources (e.g., an attacker injects synthesized speech through the Bluetooth headset), the server detects a significant difference between their acoustic characteristics, determines that an injection attack has occurred, and refuses to perform the subsequent voiceprint authentication process.

[0103] In some embodiments, the server not only refuses to perform subsequent voiceprint authentication processes, but also returns risk warning information to the terminal to alert the user to the existence of injection attacks, thereby improving the reliability of voiceprint authentication and increasing the user's trust in it.

[0104] Steps 603 to 605 above combine the local microphone acquisition with channel consistency verification. This not only ensures that the acquired audio data contains audio data acquired through the local microphone, but also compares the audio data acquired through the local microphone with the audio data acquired through other channels. Based on the comparison results, it determines whether an injection attack exists. If an injection attack is found, authentication is rejected, thus improving the reliability of voiceprint authentication.

[0105] It should be noted that step 605 above is an optional step. The server can further improve the reliability of voiceprint authentication by performing the consistency check shown in step 605, or it can choose not to perform the consistency check shown in step 605 to save computing resources and improve the efficiency of voiceprint authentication. This disclosure does not limit this.

[0106] In step 606, the server performs vibration acoustic feature verification on the first audio data based on the vibration timestamp, and the first audio data passes the vibration acoustic feature verification.

[0107] In this embodiment, the server extracts vibration acoustic features from the first audio data based on a vibration timestamp at the time point indicated by the vibration timestamp. These vibration acoustic features include the amplitude and duration of each vibration, and the interval between multiple vibrations. The server compares these vibration acoustic features with a vibration strategy. If the vibration acoustic features match the vibration strategy, the server determines that the first audio data has passed the vibration acoustic feature verification. The first audio data is audio data recorded by the local microphone.

[0108] In some embodiments, during the vibration acoustic feature verification process, the server compares the vibration amplitude in the vibration acoustic features with the vibration strategy to determine whether the vibration amplitude of each vibration in the first audio data matches the vibration amplitude indicated by the vibration strategy. The server compares the vibration duration in the vibration acoustic features with the vibration strategy to determine whether the vibration duration of each vibration in the first audio data matches the vibration duration indicated by the vibration strategy. The server compares the vibration interval between adjacent vibrations in the vibration acoustic features with the vibration strategy to determine whether the vibration interval between adjacent vibrations in the first audio data matches the vibration interval indicated by the vibration strategy. Furthermore, the server uses the aforementioned vibration timestamp, or analyzes the first audio data, to determine whether the time of vibration occurrence in the first audio data matches the vibration trigger time indicated by the vibration strategy. Through the aforementioned vibration acoustic feature verification, the server determines whether the vibration signal in the first audio data conforms to the vibration strategy, and based on this, determines whether the first audio data is audio data recorded by the local microphone, thereby ensuring that the server performs voiceprint authentication based on audio data recorded by the local microphone, improving the accuracy of voiceprint authentication. If the first audio data fails any of the above verification processes, the server determines that the first audio data has failed the vibration acoustic feature verification, and that the first audio data was not recorded locally, and refuses to proceed with the subsequent voiceprint authentication process.

[0109] For example, an attacker connects to an external microphone via the OTG interface and injects their own voice through the microphone. Simultaneously, the attacker routes the system audio source to the external microphone connected via the OTG interface. In this scenario, the terminal can detect the connected external audio input device (i.e., the external microphone) and simultaneously activate its own microphone to record, sending both the audio data recorded via its own microphone and the audio data recorded via the external microphone to the server. However, because the attacker routes the system audio source to the external microphone, the server, upon receiving the audio data recorded via the external microphone, interprets it as audio data recorded via its own microphone. Since the audio recorded via the own microphone is the user's voice and the ambient sound of the user's environment (the user may not be speaking, in which case the recorded audio is the ambient sound), while the audio recorded via the external microphone is the attacker's voice and the ambient sound of the attacker's environment, the server can use the aforementioned consistency check process to determine the presence of an injection attack and thus refuse voiceprint authentication. Given that the vibration acoustic features in the audio data recorded through the local microphone conform to the vibration policy, while the audio data recorded through the external microphone does not contain vibration acoustic features that conform to the vibration policy, the server can determine that the audio data is injected by an attacker based on the absence of vibration acoustic features that conform to the vibration policy in the audio data recorded through the external microphone, and thus refuse to perform voiceprint authentication.

[0110] The above process is illustrated using vibration acoustic features including vibration amplitude, duration of each vibration, and vibration interval as an example. In some embodiments, the server only detects the energy peak at the time indicated by the vibration timestamp. Based on the detected energy peak, it determines whether the amplitude of the vibration signal in the first audio data, the vibration interval between adjacent vibrations, and the time when the vibration occurs conform to the vibration strategy. This reduces computational overhead through lightweight vibration detection, enabling the voiceprint authentication method provided in this embodiment to be applied to scenarios with limited computing resources.

[0111] The above-described vibration acoustic feature verification process is illustrated by taking the server's extraction of vibration acoustic features from audio data based on vibration timestamps as an example. In some embodiments, during the vibration acoustic feature verification process, the server directly parses the first audio data to extract features such as the occurrence time, amplitude, interval, and duration of the vibration signal. The extracted features are then compared with a vibration strategy to achieve the verification process. In other embodiments, the server compares the occurrence time of the vibration signal extracted from the first audio data with the time indicated by the vibration timestamp uploaded by the terminal to verify the authenticity of the vibration signal in the first audio data, ensuring the authenticity of the vibration signal.

[0112] The aforementioned vibration acoustic feature verification ensures that the audio device used for voiceprint authentication records audio data using the local microphone, thus improving the reliability of voiceprint authentication. Illustratively, because external audio input devices are physically decoupled from the local device, they cannot sense the mechanical vibrations generated by the local vibration motor; therefore, the audio data they collect will not contain vibration acoustic features. Furthermore, remote audio streams transmitted over the network cannot carry the physical vibration acoustic features of the local device, and the vibration strategy issued by the server cannot be predicted, thus making it impossible to generate matching vibration acoustic features in real time. Moreover, the vibration strategy issued by the server is different each time; attackers cannot pre-record audio data with the correct vibration acoustic features for recording and playback attacks, nor can they synthesize audio data containing the correct vibration acoustic features in real time. Furthermore, attackers remotely inject synthesized speech in real time over the network. Even if the attackers know part of the vibration strategy, because the remote audio stream is completely decoupled from the local physical vibration, the attackers cannot generate physical vibration signals in the audio stream that are consistent with the vibration acoustic features captured by the local microphone in real time. Based on this, the vibration acoustic features extracted by the server from the remotely injected audio data do not match the vibration strategy (or there are no vibration acoustic features at all). The server determines that the audio data is not recorded locally and refuses authentication.

[0113] The processes described in steps 603 to 606 above construct a voiceprint authentication protection mechanism from two dimensions: "acquisition channel security" and "physical device binding." Regarding acquisition channel security, the voiceprint authentication method provided in this embodiment detects external audio input devices and uses the local microphone as the main audio acquisition channel. When both channels are acquired simultaneously, consistency verification is performed on the dual-channel audio. If an abnormal audio data source is detected, voiceprint authentication is rejected. By using the local microphone at the physical level, a fundamental defense is formed against attacks such as injection attacks from external audio input devices and remote audio stream injection attacks, filling the blind spots of existing anti-spoofing detection. Regarding physical device binding, the voiceprint authentication method provided in this embodiment triggers a vibration motor to vibrate at the start, end, and during audio data acquisition. The terminal's local microphone synchronously captures the physical acoustic features generated by the vibration. The server verifies whether the vibration acoustic features in the audio data conform to the previously issued vibration strategy to determine whether the acquired audio data comes from the real local microphone, rather than externally injected audio data. The aforementioned vibration acoustic characteristics are strongly bound to the device's physical components (vibration motor). Any audio data not recorded by the device itself cannot contain the correct vibration acoustic characteristics, thus constructing an unforgeable channel marker based on the device's physical characteristics. The voiceprint authentication protection mechanism built from these two dimensions ensures that the audio data used for voiceprint authentication is recorded by the device's microphone, greatly improving the reliability of voiceprint authentication.

[0114] In some embodiments, the server receives auxiliary data from the terminal. Before or after performing vibration acoustic feature verification, the server verifies, based on the auxiliary data, whether the signal generated by the vibration of the vibration motor in the first audio data conforms to the vibration information indicated by the auxiliary data. This vibration information may be, for example, the displacement of the terminal, such as the translational acceleration of the terminal, the speed and direction of the terminal's rotation or tilt, etc. This vibration information can indirectly reflect the vibration status of the terminal. If the signal generated by the vibration of the vibration motor in the first audio data does not conform to the vibration information indicated by the auxiliary data, the server stops performing voiceprint authentication. The above process cross-validates the auxiliary data with the vibration acoustic features, and through dual verification, confirms the physical authenticity of the vibration in the first audio data, further improving the reliability of voiceprint authentication.

[0115] In step 607, the server performs voiceprint authentication on the first audio data to obtain the voiceprint authentication result.

[0116] In this embodiment of the disclosure, the server extracts the voiceprint features from the first audio data and compares these voiceprint features with features in the voiceprint database to obtain a voiceprint authentication result. Specifically, if the voiceprint features extracted from the first audio data match the features pre-stored by the user in the voiceprint database, the voiceprint authentication result is successful; if the voiceprint features extracted from the first audio data do not match the features pre-stored by the user in the voiceprint database, the voiceprint authentication result is unsuccessful.

[0117] In some embodiments, during the process of extracting the voiceprint features of the first audio data, the server extracts a speech segment from the first audio data, performs an embedding operation on the speech segment, and obtains the voiceprint features of the first audio data.

[0118] In some embodiments, during the voiceprint comparison process, the server obtains the cosine similarity between the voiceprint features of the first audio data and the features pre-stored by the user in the voiceprint database. If the cosine similarity is greater than or equal to a preset threshold, the voiceprint authentication is successful; if the cosine similarity is less than the preset threshold, the voiceprint authentication fails.

[0119] In some embodiments, the server processes the first audio data using a pre-trained voiceprint model to obtain a voiceprint authentication result.

[0120] In some embodiments, before performing voiceprint authentication, the server performs vibration frequency band filtering on the first audio data to filter out the signal generated by the vibration motor vibration from the first audio data, retaining pure voice for voiceprint feature extraction, avoiding vibration noise interference with voiceprint authentication, and improving the accuracy of voiceprint authentication.

[0121] The voiceprint authentication method provided in this disclosure consists of three layers of protection: consistency verification, vibration anti-counterfeiting, and voiceprint comparison. A single attack method cannot bypass all three layers of protection at the same time, and this method significantly improves the overall security of voiceprint authentication.

[0122] In step 608, the server returns the voiceprint authentication result to the terminal.

[0123] In this embodiment of the disclosure, the server returns a voiceprint authentication result to the terminal as either successful or unsuccessful. Optionally, if the voiceprint authentication result is unsuccessful, the server also links with the risk control system to return a risk warning to the terminal, thereby alerting the user through the terminal that the current voiceprint authentication poses a risk.

[0124] The voiceprint authentication method provided in this disclosure, based on recording first audio data using the terminal's own microphone, superimposes a vibration signal onto the first audio data using the terminal's vibration motor. The server performs anti-counterfeiting detection on the first audio data based on whether the vibration signal conforms to a vibration strategy, and performs voiceprint authentication if the anti-counterfeiting detection passes. Since the vibration motor is installed in the terminal, only the terminal's own microphone can record the signal generated by the vibration motor. Therefore, the above method provides an audio data source identification mechanism, enabling the server to distinguish whether the audio data originates from the terminal's own microphone or an external audio input device. Indicatively, the server's anti-counterfeiting detection based on the vibration signal ensures that the audio data used for voiceprint authentication is recorded using the terminal's own microphone, preventing attackers from bypassing the microphone and directly injecting forged audio data into the acquisition layer, thereby improving the reliability of voiceprint authentication. Furthermore, the voiceprint authentication method provided in this disclosure requires no additional user operation, providing a higher level of security for voiceprint authentication without significantly increasing the user's operational burden. The voiceprint authentication method provided in this disclosure is no longer based entirely on software-level audio analysis. Instead, it utilizes the physical hardware characteristics of the terminal itself (such as a vibration motor) to construct an unspoofable channel marker. This ensures that the audio data used for voiceprint authentication is collected by the terminal's native microphone, rather than forged audio data injected by attackers through external audio input devices or networks. This improves the reliability of voiceprint authentication without increasing costs.

[0125] Figure 8 This is a block diagram illustrating a voiceprint authentication device according to an exemplary embodiment. The device is used to perform the steps of the above-described voiceprint authentication method. See also... Figure 8 The device includes: The audio receiving module 801 is used to receive first audio data from the terminal, the first audio data including external sound recorded by the terminal's local microphone and signals generated by the vibration of the terminal's vibration motor. The voiceprint authentication module 802 is used to perform voiceprint authentication on the first audio data if the signal generated by the vibration of the vibration motor in the first audio data conforms to the vibration strategy.

[0126] In some embodiments, the audio receiving module 801 described above is further configured to receive second audio data from the terminal, the second audio data being recorded by an external audio input device of the terminal; The above-mentioned device also includes: The consistency verification module is used to verify the consistency between the first audio data and the second audio data; if the first audio data and the second audio data fail the consistency verification, the voiceprint authentication process is stopped.

[0127] In some embodiments, the first audio data and the second audio data each have a corresponding channel marker, and the channel marker of the first audio data indicates that the first audio data is audio data acquired through the main audio acquisition channel.

[0128] In some embodiments, the above-described apparatus further includes: The timestamp receiving module is used to receive the vibration timestamp corresponding to the first audio data from the terminal. The vibration timestamp indicates the time point when the vibration signal appears in the first audio data. The aforementioned voiceprint authentication module 802 is used for: Based on the vibration timestamp, vibration acoustic features are extracted from the time points indicated by the vibration timestamps in the first audio data. If the vibration acoustic characteristics match the vibration strategy, voiceprint authentication is performed on the first audio data.

[0129] In some embodiments, the aforementioned vibration acoustic feature is the energy peak of the first audio data at the time indicated by the vibration timestamp.

[0130] In some embodiments, the above-described apparatus further includes: An auxiliary data receiving module is used to receive auxiliary data from the terminal, the auxiliary data including at least one of the following: accelerometer data of the terminal and gyroscope data of the terminal; The auxiliary verification module is used to verify, based on auxiliary data, whether the signal generated by the vibration of the vibration motor in the first audio data conforms to the vibration information indicated by the auxiliary data; if the signal generated by the vibration of the vibration motor in the first audio data does not conform to the vibration information indicated by the auxiliary data, the voiceprint authentication is stopped.

[0131] In some embodiments, the above-described apparatus further includes: The strategy generation module is used to respond to the operation request from the terminal to trigger voiceprint authentication, randomly generate a vibration strategy, and send the vibration strategy and audio acquisition command to the terminal. The vibration strategy includes at least one of the following: vibration trigger time, vibration amplitude, duration of each vibration, vibration interval, vibration frequency, and vibration mode.

[0132] In some embodiments, the voiceprint authentication module 802 described above is used for: Vibration frequency band filtering is performed on the first audio data. Vibration frequency band filtering is used to filter out the signal generated by the vibration of the vibration motor from the first audio data. Voiceprint authentication is performed on the first audio data after filtering out the vibration frequency band.

[0133] It should be noted that the device provided in the above embodiments is only illustrated by the division of the above functional modules when performing voiceprint authentication. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0134] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0135] Figure 9 This is a block diagram illustrating a voiceprint authentication device according to an exemplary embodiment. The device is used to perform the steps of the above-described voiceprint authentication method. See also... Figure 9 The device includes: The recording module 901 is used to respond to an audio acquisition command, activate the terminal's local microphone to record, and trigger the terminal's vibration motor to vibrate according to a vibration strategy. The audio sending module 902 is used to send the first audio data recorded to the server after the recording is completed. The first audio data includes external sound recorded by the local microphone and signals generated by the vibration of the vibration motor. The first audio data is used for voiceprint authentication.

[0136] In some embodiments, the recording module 901 described above is further configured to record audio through at least one external audio input device when it is detected that the terminal is connected to at least one external audio input device; The audio sending module 902 is also used to send the recorded second audio data to the server after the recording is completed. The second audio data is used for voiceprint authentication.

[0137] In some embodiments, the first audio data and the second audio data each have a corresponding channel marker, and the channel marker of the first audio data indicates that the first audio data is audio data acquired through the main audio acquisition channel.

[0138] In some embodiments, the above-described apparatus further includes: The timestamp sending module is used to send the vibration timestamp corresponding to the first audio data to the server. The vibration timestamp indicates the time point when the vibration signal appears in the first audio data.

[0139] In some embodiments, the vibration strategy is randomly generated, and the vibration strategy includes at least one of the following: vibration trigger time, vibration amplitude, duration of each vibration, vibration interval, vibration frequency, and vibration mode.

[0140] In some embodiments, the above-described apparatus further includes: An auxiliary data acquisition module is used to acquire auxiliary data of the terminal during the recording process. The auxiliary data includes at least one of the following: accelerometer data of the terminal and gyroscope data of the terminal. The auxiliary data is used to verify whether the signal generated by the vibration of the vibration motor in the first audio data matches the actual vibration of the terminal. The auxiliary data sending module is used to send the collected auxiliary data to the server.

[0141] It should be noted that the device provided in the above embodiments is only illustrated by the division of the above functional modules when performing voiceprint authentication. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0142] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0143] Figure 10 This is a schematic diagram of a server structure according to an exemplary embodiment. The server 1000 can vary considerably due to different configurations or performance. It may include one or more CPUs (Central Processing Units) 1001 and one or more memories 1002. The memory 1002 stores at least one computer program, which is loaded and executed by the processor 1001 to implement the voiceprint authentication method provided in the above-described method embodiments. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated here.

[0144] Figure 11This is a schematic diagram of a terminal structure according to an exemplary embodiment. The terminal 1100 can vary considerably due to different configurations or performance. It may include one or more CPUs (Central Processing Units) 1101, one or more memories 1102, one or more microphones 1103, and vibration motors 1104. The memory 1102 stores at least one computer program, which is loaded and executed by the processor 1101 to implement the voiceprint authentication method provided in the above-described method embodiments. Of course, the terminal may also have wired or wireless network interfaces, keyboards, and input / output interfaces for input and output. The terminal may also include other components for implementing device functions, which will not be elaborated here.

[0145] This disclosure also provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a server's processor to implement the operations performed by the server in the voiceprint authentication method of the above embodiments. For example, the computer-readable storage medium may be ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage device, etc.

[0146] This disclosure also provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a server's processor to implement the operations performed by the terminal in the voiceprint authentication method of the above embodiments. For example, the computer-readable storage medium may be ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage device, etc.

[0147] This disclosure also provides a computer program product or computer program including computer program code stored in a computer-readable storage medium. A server's processor reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the server to perform the voiceprint authentication method provided in the various optional implementations described above.

[0148] This disclosure also provides a computer program product or computer program including computer program code stored in a computer-readable storage medium. The terminal's processor reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the terminal to perform the voiceprint authentication method provided in the various optional implementations described above.

[0149] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0150] The above description is merely an optional embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. A voiceprint authentication method, characterized in that, The method includes: Receive first audio data from the terminal, the first audio data including external sound recorded through the terminal's native microphone and signals generated by the vibration of the terminal's vibration motor; If the signal generated by the vibration of the vibration motor in the first audio data conforms to the vibration strategy, voiceprint authentication is performed on the first audio data.

2. The method according to claim 1, characterized in that, The method further includes: Receive second audio data from the terminal, the second audio data being recorded via an external audio input device of the terminal; Perform a consistency check on the first audio data and the second audio data; If the first audio data and the second audio data fail the consistency check, the voiceprint authentication process will be stopped.

3. The method according to claim 2, characterized in that, The first audio data and the second audio data each have a corresponding channel marker. The channel marker of the first audio data indicates that the first audio data is audio data acquired through the main audio acquisition channel.

4. The method according to claim 1, characterized in that, The method further includes: Receive a vibration timestamp corresponding to the first audio data from the terminal, the vibration timestamp indicating the time point in the first audio data when a vibration signal appears; If the signal generated by the vibration of the vibration motor in the first audio data conforms to the vibration strategy, the voiceprint authentication of the first audio data includes: Based on the vibration timestamp, vibration acoustic features are extracted at the time points indicated by the vibration timestamp in the first audio data; If the vibration acoustic characteristics match the vibration strategy, voiceprint authentication is performed on the first audio data.

5. The method according to claim 4, characterized in that, The vibration acoustic feature is the energy peak value of the first audio data at the time indicated by the vibration timestamp.

6. The method according to claim 4, characterized in that, The method further includes: Receive auxiliary data from the terminal, the auxiliary data including at least one of the following: accelerometer data of the terminal and gyroscope data of the terminal; Based on the auxiliary data, verify whether the signal generated by the vibration of the vibration motor in the first audio data matches the vibration information indicated by the auxiliary data; If the signal generated by the vibration of the vibration motor in the first audio data does not match the vibration information indicated by the auxiliary data, the voiceprint authentication process will be stopped.

7. The method according to claim 1, characterized in that, The method further includes: In response to an operation request from the terminal to trigger voiceprint authentication, a vibration strategy is randomly generated, and the vibration strategy and audio acquisition command are sent to the terminal. The vibration strategy includes at least one of the following: vibration trigger time, vibration amplitude, duration of each vibration, vibration interval, vibration frequency, and vibration mode.

8. The method according to claim 1, characterized in that, The voiceprint authentication of the first audio data includes: The first audio data is subjected to vibration frequency band filtering, which is used to filter out the signal generated by the vibration of the vibration motor from the first audio data; Voiceprint authentication is performed on the first audio data after filtering out the vibration frequency band.

9. A voiceprint authentication method, characterized in that, The method includes: In response to an audio acquisition command, the terminal's local microphone is activated to record audio, and the terminal's vibration motor is triggered to vibrate according to a vibration strategy. After recording is completed, the first audio data is sent to the server. The first audio data includes external sounds recorded through the local microphone and signals generated by the vibration of the vibration motor. The first audio data is used for voiceprint authentication.

10. The method according to claim 9, characterized in that, When the terminal is connected to at least one external audio input device, the method further includes: If it is detected that the terminal is connected to at least one external audio input device, recording is performed through the at least one external audio input device; After the recording is completed, the recorded second audio data is sent to the server. The second audio data is used for voiceprint authentication.

11. The method according to claim 10, characterized in that, The first audio data and the second audio data each have a corresponding channel marker. The channel marker of the first audio data indicates that the first audio data is audio data acquired through the main audio acquisition channel.

12. The method according to claim 9, characterized in that, The method further includes: Send a vibration timestamp corresponding to the first audio data to the server. The vibration timestamp indicates the time point in the first audio data when a vibration signal appears.

13. The method according to claim 9, characterized in that, The vibration strategy is randomly generated and includes at least one of the following: vibration trigger time, vibration amplitude, duration of each vibration, vibration interval, vibration frequency, and vibration mode.

14. The method according to claim 9, characterized in that, The method further includes: During the recording process, auxiliary data of the terminal is collected. The auxiliary data includes at least one of the following: accelerometer data of the terminal and gyroscope data of the terminal. The auxiliary data is used to verify whether the signal generated by the vibration of the vibration motor in the first audio data matches the actual vibration of the terminal. The collected auxiliary data is sent to the server.

15. A voiceprint authentication device, characterized in that, The device includes: An audio receiving module is used to receive first audio data from a terminal, the first audio data including external sound recorded by the terminal's local microphone and signals generated by the vibration of the terminal's vibration motor; The voiceprint authentication module is used to perform voiceprint authentication on the first audio data if the signal generated by the vibration of the vibration motor in the first audio data conforms to the vibration strategy.

16. A voiceprint authentication device, characterized in that, The device includes: The recording module is used to respond to an audio acquisition command by activating the terminal's local microphone to record audio and triggering the terminal's vibration motor to vibrate according to a vibration strategy. An audio sending module is used to send the first audio data recorded to the server after recording is completed. The first audio data includes external sounds recorded through the local microphone and signals generated by the vibration of the vibration motor. The first audio data is used for voiceprint authentication.

17. A server, characterized in that, The server includes: One or more processors; Memory used to store the executable program code of the processor; The processor is configured to execute the program code to implement the voiceprint authentication method as described in any one of claims 1 to 8.

18. A terminal, characterized in that, The terminal includes: One or more processors; Memory used to store the executable program code of the processor; The processor is configured to execute the program code to implement the voiceprint authentication method as described in any one of claims 9 to 14.

19. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the server, the server is enabled to perform the voiceprint authentication method as described in any one of claims 1 to 8.

20. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the terminal, the terminal is enabled to perform the voiceprint authentication method as described in any one of claims 9 to 14.

21. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the voiceprint authentication method according to any one of claims 1 to 8.

22. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the voiceprint authentication method according to any one of claims 9 to 14.

Images