Voiceprint registration method, voiceprint authentication method and corresponding devices

Abstract

Embodiments of the present application disclose a voiceprint registration method, a voiceprint authentication method and corresponding devices. The main technical solutions include: obtaining historical works of a target account and a reference visual biological feature of the target account, the historical works including video data; based on the reference visual biological feature, extracting a target voice segment from the video data, the vocal subject of the target voice segment being consistent with the subject corresponding to the target account; performing voiceprint feature extraction on the target voice segment to obtain a voiceprint feature corresponding to the target voice segment; and based on the voiceprint feature, generating and storing a voiceprint template corresponding to the target account. The voiceprint registration can be automatically completed without the user performing a live recording action, effectively reducing the user interaction cost, simplifying the registration interaction required for constructing a voiceprint template, and making it possible to construct a large-scale voiceprint library.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to a voiceprint registration method, a voiceprint authentication method, and a corresponding device. Background Technology

[0002] In the field of internet identity authentication and biometric recognition technology, voiceprint authentication has broad application prospects in scenarios such as account security and risk control due to its advantages of being contactless, having natural interaction, and requiring low equipment. However, the prerequisite for the effective operation of any voiceprint authentication system is to establish a reliable voiceprint registration database, which requires a way to obtain user voiceprint templates in a low-cost, high-coverage, and reliable manner.

[0003] In related technologies, the construction of a voiceprint registration database typically relies on the user's active cooperation and on-site recording at specific stages. For example, a common approach requires the user to read designated content aloud as prompted, the system then collects the audio and extracts voiceprint features to obtain the user's voiceprint template. This approach requires the user to add a separate on-site recording step, resulting in a long registration interaction chain, high costs, and the quality of the registered voiceprint template is limited by the on-site recording environment. This technical bottleneck restricts the large-scale implementation and widespread application of voiceprint authentication. Summary of the Invention

[0004] In view of this, this application provides a voiceprint registration method, a voiceprint authentication method, and a corresponding apparatus to simplify the registration interaction required for constructing a voiceprint template.

[0005] This application provides the following solution: According to the first aspect, a voiceprint registration method is provided, comprising: obtaining the historical works of a target account and reference visual biometric features used to identify the subject identity of the target account, wherein the historical works include video data; extracting a target speech segment from the video data based on the reference visual biometric features, wherein the speaker of the target speech segment is consistent with the subject corresponding to the target account; extracting voiceprint features from the target speech segment to obtain the voiceprint features corresponding to the target speech segment; and generating and storing a voiceprint template corresponding to the target account based on the voiceprint features.

[0006] Optionally, based on reference visual biometrics, the target speech segment is extracted from the video data, including: performing visual biometric detection on the video data, generating one or more candidate visual biometrics and video content corresponding to each candidate visual biometric based on the detection results; determining the similarity between each candidate visual biometric and the reference visual biometric, determining the video content corresponding to the candidate visual biometric that meets the similarity requirement as the target video content; and extracting the target speech segment from the target video content.

[0007] Optionally, extracting target audio segments from target video content includes: performing consistency detection on the speaker of each video segment contained in the target video content to determine the target video segments whose speaker matches the consistency requirement with the subject corresponding to the target account; and extracting audio segments from the target video segments as target audio segments.

[0008] Optionally, the consistency of the speaker in each video segment contained in the target video content is detected, including: determining the temporal correlation between lip movement and speech signal in the video segment based on lip movement detection; determining the synchronization between video frame and speech signal in the video segment; determining the similarity between the visual biometrics of the speaker in the video segment and the reference visual biometrics; and determining the consistency between the speaker in the video segment and the subject corresponding to the target account based on at least one of temporal correlation, synchronization, and similarity.

[0009] Optionally, extracting target audio segments from target video content includes: distinguishing multiple audio tracks from target video content; and performing at least one of background audio track filtering, background music audio track filtering, transferred audio track filtering, and dubbing audio track filtering on the multiple audio tracks to separate the audio segments of the main vocal audio track as the target audio segments.

[0010] Optionally, there are multiple target speech segments; based on voiceprint features, a voiceprint template corresponding to the target account is generated, including: determining quality evaluation indicators for multiple target speech segments respectively; weighted fusion of the voiceprint features corresponding to multiple target speech segments based on the quality evaluation indicators of each target speech segment to obtain the voiceprint template corresponding to the target account; or, based on the quality evaluation indicators of each target speech segment, selecting voiceprint features from the voiceprint features corresponding to multiple target speech segments as the voiceprint template corresponding to the target account.

[0011] Optionally, the method further includes: in response to the target account's updated historical works meeting preset requirements, extracting new target speech segments from the updated historical works based on reference visual biometrics; determining multiple target speech segments within a preset time window; extracting voiceprints from the multiple target speech segments within the preset time window to obtain voiceprint features corresponding to the multiple target speech segments within the preset time window; and for the multiple target speech segments within the preset time window, proceeding to the step of determining the quality evaluation indicators of the multiple target speech segments respectively, thereby obtaining the updated voiceprint template corresponding to the target account.

[0012] Optionally, the method further includes: in response to the target account's updated historical works meeting preset requirements, extracting new target speech segments from the updated historical works based on reference visual biometrics; determining quality evaluation indicators for the new target speech segments respectively; selecting the N target speech segments with the highest quality evaluation indicators from all extracted speech segments; and performing weighted fusion of the voiceprint features corresponding to the N target speech segments based on the quality evaluation indicators of the N target speech segments to obtain the updated voiceprint template corresponding to the target account, where N is a preset positive integer.

[0013] Optionally, the quality evaluation index includes at least one of visual biometric matching score, voice credibility score, and audio quality score; wherein, the visual biometric matching score is obtained based on the similarity between visual biometrics extracted from the video content corresponding to the target speech segment and reference visual biometrics; the voice credibility score is obtained by performing voice subject consistency detection on the target speech segment, and the voice subject consistency detection is used to detect the consistency between the voice subject of the target speech segment and the subject corresponding to the target account.

[0014] Optionally, there are multiple target speech segments; based on the voiceprint features, a voiceprint template corresponding to the target account is generated, including: clustering the voiceprint features corresponding to multiple target speech segments; and using the cluster centers obtained from the clustering as the voiceprint template corresponding to the target account.

[0015] Optionally, the method further includes: in response to the target account's updated historical works meeting preset requirements, extracting new target speech segments from the updated historical works based on reference visual biometrics; extracting voiceprint features from the new target speech segments to obtain voiceprint features corresponding to the new target speech segments; determining the ratio between the first voiceprint feature and the second voiceprint feature, wherein the first voiceprint feature is the voiceprint feature corresponding to the new target speech segment that is greater than or equal to a preset distance threshold from the cluster center, and the second voiceprint feature is the voiceprint feature corresponding to the new target speech segment that is less than a preset distance threshold from the cluster center; if the ratio is less than a preset ratio threshold, incrementally updating the cluster center using the second voiceprint feature, and using the updated cluster center as the updated voiceprint template corresponding to the target account; if the ratio is greater than or equal to a preset ratio threshold, clustering using the first voiceprint feature, and using the new cluster center obtained by clustering as the updated voiceprint template corresponding to the target account.

[0016] According to the second aspect, a voiceprint authentication method is provided, comprising: extracting voiceprint features from the voice to be authenticated to obtain the voiceprint features corresponding to the voice to be authenticated; determining the similarity between the voiceprint features corresponding to the voice to be authenticated and the voiceprint template corresponding to the target account; and, in response to the similarity being greater than or equal to a similarity threshold, passing the authentication; otherwise, failing the authentication; wherein the voiceprint template corresponding to the target account is registered using the method provided by any implementation of the first aspect above.

[0017] Optionally, the method further includes: identifying the risk level of the authentication scenario for the voice to be authenticated; and determining a similarity threshold based on the identified risk level, wherein the risk level is positively correlated with the similarity threshold.

[0018] Optionally, the method further includes: identifying the risk level of the authentication scenario for the voice to be authenticated; determining the authentication strategy to be adopted based on the identified risk level, wherein the authentication strategy includes: a voiceprint authentication method, or a voiceprint authentication method combined with at least one other authentication method; wherein the other authentication methods include authentication methods based on visual biometrics, authentication methods based on voice liveness detection, authentication methods based on the device from which the voice to be authenticated originates, authentication methods based on the behavior corresponding to the target account, or manual authentication methods.

[0019] According to a third aspect, a voiceprint registration device is provided, the device comprising: The reference acquisition unit is configured to acquire the historical works of the target account and the reference visual biometric features used to identify the subject identity corresponding to the target account, wherein the historical works include video data; The segment acquisition unit is configured to extract target speech segments from the video data based on the reference visual biometrics, wherein the speaker of the target speech segment is consistent with the subject corresponding to the target account; The voiceprint extraction unit is configured to extract voiceprint features from the target speech segment to obtain the voiceprint features corresponding to the target speech segment. The voiceprint maintenance unit is configured to generate and store a voiceprint template corresponding to the target account based on the voiceprint features.

[0020] According to a fourth aspect, a voiceprint authentication device is provided, the device comprising: The voiceprint extraction unit is configured to extract voiceprint features from the speech to be authenticated, thereby obtaining the voiceprint features corresponding to the speech to be authenticated. The voiceprint authentication unit is configured to determine the similarity between the voiceprint features corresponding to the voice to be authenticated and the voiceprint template corresponding to the target account; The result acquisition unit is configured to pass authentication if the similarity is greater than or equal to a similarity threshold; otherwise, authentication fails. The voiceprint template corresponding to the target account is registered by the voiceprint registration device described in the third aspect above.

[0021] According to a fifth aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first and second aspects above.

[0022] According to a sixth aspect, an electronic device is provided, comprising: One or more processors; and A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any one of the first and second aspects above.

[0023] According to a seventh aspect, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of the method described in either the first or second aspect.

[0024] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application makes full use of the target account's historical works in video format. By introducing reference visual biometrics as a bridge, it binds the video data of the target account's historical works with the identity of the speaker, thereby enabling automatic voiceprint registration without the user having to perform on-site recording. This effectively reduces user interaction costs, simplifies the registration interaction required to build voiceprint templates, and makes the construction of a large-scale voiceprint library possible.

[0025] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a system architecture diagram applicable to the embodiments of this application.

[0028] Figure 2 A flowchart of the voiceprint authentication method provided in the embodiments of this application.

[0029] Figure 3 This is a schematic flowchart illustrating a voiceprint authentication method provided in an embodiment of this application.

[0030] Figure 4 A schematic block diagram of a voiceprint registration device provided in an embodiment of this application.

[0031] Figure 5 A schematic block diagram of a voiceprint authentication device provided in an embodiment of this application.

[0032] Figure 6 A schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0034] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0035] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0036] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0037] In the field of internet account authentication technology, the commonly used technical solution for voiceprint template registration is on-site audio recording. Specifically, this solution requires users to read aloud a specified text as prompted by the system during registration. The system then records the user's audio in real time, extracts voiceprint features from the recording, and stores the registration template for the user's account. Its basic working principle relies on the user's active cooperation to complete a one-time voiceprint template registration in a controlled or semi-controlled environment. Its widespread application stems from its ability to directly capture speech containing known text content linked to the user's identity, thereby establishing a basic voiceprint comparison benchmark.

[0038] However, this solution's performance is less than ideal when applied to internet platforms with large user bases and historical content assets. A fundamental contradiction lies in the fact that, in pursuit of controllability in the registration process and direct data sources, the solution's design inevitably requires users to actively record their voices on-site. This inherent design inevitably leads to high overall registration interaction costs, long processing times, and an inability to achieve large-scale coverage of the platform's massive existing accounts. Specifically, in application scenarios requiring the creation of voiceprint databases for hundreds of millions of existing accounts on the platform, guiding each user to complete on-site voice recording registration individually presents extremely high user cooperation barriers and enormous interaction costs, resulting in low registration rates and ultimately failing to establish a voiceprint authentication capability with practical value.

[0039] In view of this, this application provides a new approach. To facilitate understanding of this application, the system architecture on which this application is based will first be described. Figure 1 An exemplary system architecture that can be applied to embodiments of this application is shown, such as Figure 1 As shown, the system architecture may include: a user terminal set on the user terminal and a voiceprint registration device and a voiceprint authentication device set on the server.

[0040] The server side and the client side are the two main parts of an application service. The server side uses a server as the main hardware infrastructure and may include one or more software service modules, forming a front-end and back-end that work together with the client side.

[0041] User terminals can include, but are not limited to, smart mobile terminals, wearable devices, and PCs (Personal Computers). Smart mobile devices can include devices such as mobile phones, tablets, PDAs (Personal Digital Assistants), and connected car terminals. Wearable devices can include devices such as smartwatches, smart glasses, smart bracelets, VR (Virtual Reality) devices, AR (Augmented Reality) devices, and mixed reality devices (devices that support both virtual and augmented reality), etc.

[0042] The user end can be a client, mini-program, or web application located on the user's terminal.

[0043] A server can be a standalone server, a server cluster, or a cloud server. A cloud server, also known as a cloud computing server or cloud host, is a hosting product within the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Servers (VPS) services, such as high management difficulty and weak service scalability.

[0044] In this embodiment, the user terminal can capture, record, or acquire works and upload them to the server for storage or publication, enabling the user and / or other users to play the works. When using related services, the user can input voice to be authenticated, which is then sent to the server via the user terminal for voiceprint authentication.

[0045] The server-side voiceprint registration device can use the historical works of each account to generate and store the voiceprint template corresponding to each account, as provided in the embodiments of this application. After obtaining the voice to be authenticated, the voiceprint authentication device can authenticate the voice to be authenticated using the methods provided in the embodiments of this application, and the server then determines whether to provide the corresponding service based on the authentication result.

[0046] It should be understood that Figure 1 The server-side, user-side, user terminal, voiceprint registration device, and voiceprint authentication device shown are merely illustrative. Depending on implementation needs, any number of server-side, user-side, user terminal, voiceprint registration device, and voiceprint authentication device can be included.

[0047] Figure 2 This is a flowchart of a voiceprint authentication method provided in an embodiment of this application. The method can be... Figure 1 The voiceprint authentication device in the system shown is executed. For example... Figure 2 As shown, the method may include the following steps: Step 201: Obtain the target account's historical works and reference visual biometrics used to identify the target account's subject identity. Historical works include video data.

[0048] Step 203: Based on reference visual biometrics, extract the target speech segment from the video data. The speaker of the target speech segment is consistent with the subject corresponding to the target account.

[0049] Step 205: Extract voiceprint features from the target speech segment to obtain the corresponding voiceprint features.

[0050] Step 207: Generate and store the voiceprint template corresponding to the target account based on voiceprint features.

[0051] As can be seen from the above process, this application makes full use of the target account's historical works in video format. By introducing reference visual biometrics as a bridge, the video data included in the target account's historical works is bound to the identity of the speaker, thereby realizing automatic voiceprint registration without the user having to perform on-site recording. This effectively reduces user interaction costs, simplifies the registration interaction required to build voiceprint templates, and makes the construction of a large-scale voiceprint library possible.

[0052] The following describes in detail each step of the above process and the effects that can be further produced, with reference to the embodiments.

[0053] First, the above step 201, namely "obtaining the target account's historical works and reference visual biometrics used to identify the target account's subject identity, the historical works including video data", will be described in detail with reference to the embodiments.

[0054] The target account refers to a user-registered account on the service platform corresponding to the server. The target account's historical works refer to user-generated content (UGC) containing video data that the account has published or associated with on the platform. As a specific implementation, the historical works may include, but are not limited to: short videos published by the target account, live stream replays, and other video data containing audio information uploaded. When performing this step, the voiceprint registration device can retrieve the aforementioned historical works data generated by the target account within a preset time range from the server's content library.

[0055] Visual biometrics refers to visual features that can uniquely or significantly identify a speaker, including but not limited to facial features, iris texture, lip shape and movement trajectory, and body movement patterns. These features are used to locate and track specific individuals in video data.

[0056] In this application, "reference visual biometrics" refers to any pre-stored or accessible feature data used to visually characterize the identity of an account holder. Specifically, it can refer to a facial photograph retained during the account's real-name authentication process, a facial feature vector extracted from that photograph, or a facial image or feature of the real-name holder saved by the system after verification through a liveness detection process conducted by an authoritative institution. Besides facial images, other types of visual biometrics are also acceptable. Regardless of the method used to obtain the reference visual biometrics, they must be authoritative and authentic, and originate from the entity corresponding to the target account.

[0057] The order in which reference visual biometrics are acquired and historical works are acquired is not limited; they can be performed sequentially or in parallel.

[0058] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application, such as the reference visual biometrics, video data, voiceprint features, etc. involved in the embodiments of this application, are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data (including the reference visual biometrics, video data, voiceprint features, etc. involved in the embodiments of this application) must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0059] The following describes in detail step 203, namely, "based on reference visual biometrics, extracting target speech segments from video data, wherein the speaker of the target speech segment is consistent with the subject corresponding to the target account", with reference to the embodiments.

[0060] This step aims to establish a correlation between reference visual biometrics and audio information in historical works, thereby locating the voice of the originating subject that matches the target account without requiring the user to perform on-site recording. The specific means of achieving this function may involve a series of complex processing logics including visual analysis and audiovisual correlation analysis.

[0061] In this application, the extracted "target audio segment" refers to any audio segment whose speaker is determined to be the same as the speaker of the target account. The speaker of the target account can be a natural person confirmed through real-name authentication. For example, for an account that has completed real-name authentication, the speaker is that real-name natural person. However, for an unauthenticated account, the speaker can be the main speaker determined by a combination of multiple works under that account.

[0062] As one feasible approach, when extracting target speech segments from video data based on reference visual biometrics, one can first perform visual biometric detection on the video data, generate one or more candidate visual biometrics and the corresponding video content based on the detection results; then, determine the similarity between each candidate visual biometric and the reference visual biometric, and determine the video content corresponding to the candidate visual biometric that meets the similarity requirement as the target video content; and then extract the target speech segment from the target video content.

[0063] In the process of visual biometric detection of video data, a pre-trained object detection model can scan the video frame by frame or at intervals to identify and record all appearing visual biometric features and their corresponding spatiotemporal locations, generating one or more candidate visual biometric features, such as facial features. Each candidate visual biometric feature is accompanied by its corresponding video content segment, including timestamps, frame sequences, and associated audio. The aforementioned object detection model can be, for example, FaceNet (based on convolutional neural networks) or MTCNN (Multi-Task Cascaded Convolutional Networks).

[0064] The similarity between candidate visual biometrics and reference visual biometrics can be represented by vector distance (such as Euclidean distance, cosine similarity, etc.). When the similarity between a candidate visual biometric and a reference visual biometric exceeds a preset threshold (e.g., 0.85) or meets the similarity ranking requirements (e.g., ranking in the top 3), the video segment corresponding to the candidate visual biometric is marked as the target video content.

[0065] By using visual biometrics (such as faces) similarity comparison as a cross-modal indexing mechanism, abstract accounts are precisely associated with specific, visually represented individuals in video streams. This allows for the automated and highly accurate selection of video clips featuring voices corresponding to the visual biometrics from a vast amount of historical works, preventing the misuse of irrelevant individuals' voices for voiceprint registration.

[0066] In addition to this approach, other alternative methods can be used, such as similarity-based target video content extraction, or directly extracting the video content corresponding to all candidate visual biometrics and then determining the speech attribution one by one through subsequent speech activity detection (VAD) or speaker recognition models.

[0067] When extracting target audio segments from target video content, one feasible approach is to further introduce speaker consistency detection. This involves performing speaker consistency detection on each video segment within the target video content to determine target video segments whose speaker's consistency with the entity corresponding to the target account meets consistency requirements (such as consistency exceeding a preset consistency threshold); then, audio segments are extracted from these target video segments as the target audio segments.

[0068] This approach, by adding an audio-visual correlation verification layer of "voice subject consistency" on top of the visual dimension, can effectively eliminate interference commonly found in videos, ensuring that the voice segments used for voiceprint modeling do indeed come from the subject corresponding to the target account, thereby improving the purity of the voiceprint template and the reliability of authentication.

[0069] The following explains the consistency detection of the sound-generating entity, which may include, but is not limited to, the following detection mechanisms used independently or in combination: (1) Temporal correlation between lip movement detection and speech signal: Lip movement detection can be used to determine the temporal correlation between lip movements and speech signals in video clips. For example, by analyzing the temporal alignment between the trajectory of a person's lips in a video clip (such as the distance between the upper and lower lips and the rate of change of the opening angle) and the energy envelope or phoneme boundary of the corresponding speech signal. If the correlation coefficient (such as the Pearson correlation coefficient) is high (e.g., >0.7), it indicates that the speaker in the video clip is consistent with the speaker in the video.

[0070] (2) The synchronization relationship between video footage and audio signal in a video clip: Calculate the offset between the timestamp and the sampling point of the audio signal in the video clip, and assess whether there is an offset (such as the audio track being moved backward or the dubbing being out of sync) and the degree of offset (inversely reflecting the degree of synchronization).

[0071] (3) Similarity between the visual biometrics of the speaker in the video clip and the reference visual biometrics: Visual biometrics are extracted from each frame or keyframe within a video segment, and a matching score is calculated between the extracted visual biometrics and a reference visual biometric. The average score can be used as a consistency indicator for the video segment.

[0072] The consistency between the speaker of the video clip and the subject corresponding to the target account can be determined by comprehensively considering at least one of the three dimensions mentioned above, or by combining other consistency detection mechanisms.

[0073] For example, weighted voting or logistic regression can be used to combine the scores of the above three dimensions to obtain the final consistency judgment result.

[0074] For example, the synchronization relationship between the video frame and the voice signal in the video clip can be determined first. Based on this synchronization relationship, lip movement detection can be performed to determine the actual speaker in the video frame corresponding to the voice signal. Then, the visual biometrics of the actual speaker and the reference visual biometrics can be used to calculate the similarity. Based on the similarity, it can be determined whether the speaker is consistent with the subject corresponding to the target account.

[0075] By incorporating cross-validation of multiple technologies, including lip movement, audio-visual synchronization, and visual feature similarity, a comprehensive judgment is made on the question of "who is speaking." This multimodal fusion judgment method is more robust than relying solely on facial recognition or sound source localization, and can significantly improve the accuracy of identifying the speaker in complex acoustic environments and visual scenes.

[0076] When extracting target audio segments from target video content, one possible approach is to first distinguish multiple audio tracks from the target video content; then, at least one of the following can be applied to the multiple audio tracks: background audio track filtering, background music audio track filtering, transferred audio track filtering, and dubbing audio track filtering, in order to separate the audio segments from the main vocal audio track as the target audio segments.

[0077] The background audio filtering primarily identifies and discards audio tracks containing non-speech components such as environmental noise, wind noise, and footsteps. The background music filtering mainly removes deliberately added background music, opening and ending theme songs, and other irrelevant audio. The copied audio track filtering detects and removes audio tracks that are clearly copied from other videos and are irrelevant to the video's theme (e.g., through audio fingerprint comparison or voice content repetition rate detection). The dubbing audio track filtering primarily excludes voices that are not the original speaker's due to post-production human dubbing (such as narration or foreign language dubbing).

[0078] Finally, the remaining main audio track after filtering is retained, and audio segments within it are used as target audio segments. This method is particularly suitable for scenarios with independent recording sources or post-production mixing, such as film editing and multi-camera recording.

[0079] By actively identifying and separating non-target sound sources such as background music, platform music, and dubbing from a multi-track mixed environment, it is possible to extract relatively pure, unmodified original speech signals from the speaker. This avoids the problem of distortion in voiceprint feature extraction caused by interference from background music, sound effects, etc., and provides high-quality original audio input for the subsequent generation of highly discriminative voiceprint templates.

[0080] Besides the multi-track separation methods mentioned above, other alternative approaches can be used. For example, a deep learning-based target speaker extraction model can be used, guided by visual biometrics, to directly extract target speech segments from the target video content.

[0081] The following describes step 205, namely "extracting voiceprint features from the target speech segment to obtain the voiceprint features corresponding to the target speech segment", in detail with reference to the embodiments.

[0082] Voiceprint features can be represented as speaker embeddings, which are compact vectors that can characterize a speaker's unique physiological and behavioral characteristics, such as vocal tract structure and pronunciation habits.

[0083] As one feasible approach, this step can begin by preprocessing the target speech segment. Preprocessing may include noise reduction, Voice Activity Detection (VAD), and audio normalization. The system then uses a pre-trained speaker feature extraction model to process the preprocessed speech segment. This model can be an x-vector model based on a Time Delay Neural Network (TDNN), a speaker embedding model based on a Transformer architecture, or any other deep neural network (DNN) model capable of mapping variable-length speech segments to fixed-dimensional speaker feature vectors. The model's processing involves calculating and compressing the input Mel-frequency cepstral coefficients or other acoustic features layer by layer through the network, ultimately outputting a feature vector representing the speaker's identity.

[0084] After this step, each target speech segment will be converted into a corresponding voiceprint feature vector. These voiceprint feature vectors are temporarily stored for subsequent voiceprint template generation.

[0085] The following describes step 207, namely "generating and storing the voiceprint template corresponding to the target account based on voiceprint features," in detail with reference to the embodiments.

[0086] A voiceprint template is a baseline model used for subsequent voiceprint authentication comparison. It is generated based on voiceprint features extracted from one or more target speech segments. When there is a single target speech segment, the voiceprint features of that segment can be directly used as the voiceprint template. When multiple target speech segments are extracted, their corresponding voiceprint features are fused or selected to generate a more representative and stable registration voiceprint template.

[0087] As one feasible approach, when generating a voiceprint template corresponding to a target account based on voiceprint features, a quality evaluation index for each target speech segment can be determined first. The quality evaluation index is used to quantify the purity, completeness, and reliability of the target speech segment. Subsequently, based on the quality evaluation index of each target speech segment, a voiceprint template is generated using one of the following two methods: Weighted fusion strategy: The speaker features of each target speech segment are weighted and calculated with the corresponding quality evaluation index, such as by weighted averaging or weighted splicing (introducing an attention mechanism to allocate feature importance according to weight), and finally fused into a unified speaker template.

[0088] Selection Strategy: Select voiceprint features from multiple target speech segments as the voiceprint template for the target account. For example, select the segment with the highest quality evaluation index from all target speech segments and directly use its voiceprint features as the voiceprint template for the target account.

[0089] By introducing quality evaluation indicators as weights or selection criteria, higher contribution of high-quality speech segments to the speaker template can be given, while suppressing or eliminating the interference of low-quality speech segments on the speaker template. This data-driven optimization strategy, compared to simply averaging all segments, can significantly improve the stability and discriminativeness of the final generated speaker template, and reduce the false rejection rate and false recognition rate during authentication.

[0090] As the target account's works are continuously updated, this application also introduces an automatic voiceprint template update mechanism to adapt to changes in the voiceprint characteristics of the speaker over time or in the environment (such as aging, hoarseness, device replacement, etc.). Under the above-mentioned method of generating voiceprint templates, the update mechanism that can be adopted may include, but is not limited to, the following two: The first update mechanism: the time window-based update mechanism.

[0091] First, in response to the detection that the target account's update history meets preset requirements (e.g., more than 5 new works, the latest work was published within 24 hours, and the work type is consistent with the previous ones), new target audio segments are extracted from these update history works based on reference visual biometrics. The extraction process can reuse the method used in step 203 above to ensure that the new target audio segments also have consistency with the speaker subject of the reference visual biometrics.

[0092] Subsequently, multiple target speech segments are identified within a preset time window. This time window can be of a fixed length (e.g., all segments from the last 30 days) or a dynamically sliding window (e.g., retaining only the most recent segments), thus balancing the timeliness and historical stability of the voiceprint template. For example, if the window size is set to 20 target speech segments, the oldest segments are automatically discarded when the number of new target speech segments exceeds 20.

[0093] Next, voiceprint extraction is performed on multiple target speech segments within the preset time window to obtain the voiceprint features corresponding to each segment.

[0094] Finally, for multiple target speech segments within a preset time window, the process proceeds to the step described above, "determining the quality evaluation indicators for each target speech segment." Then, through a weighted fusion or selection strategy, an updated voiceprint template corresponding to the target account is obtained. This updated voiceprint template replaces the original one.

[0095] By establishing a time window and regenerating voiceprint templates using newly generated data, a rolling update mechanism for voiceprint templates is achieved. This mechanism automatically adapts to voiceprint feature drift caused by factors such as aging and changes in physiological state, ensuring the voiceprint database remains synchronized with the user's latest voiceprint status. This effectively reduces the authentication failure rate due to voiceprint aging and guarantees the accuracy of voiceprint templates. For example, if a news anchor's voice changes due to a cold, the voiceprint template can be automatically updated to match the new timbre, avoiding recognition failure.

[0096] The second update mechanism: high-quality replacement.

[0097] First, in response to the detection that the target account's update history meets preset requirements (e.g., more than 5 new works, the latest work was published within 24 hours, and the work type is consistent with the previous ones), new target audio segments are extracted from these update history works based on reference visual biometrics. The extraction process can reuse the method used in step 203 above to ensure that the new target audio segments also have consistency with the speaker subject of the reference visual biometrics.

[0098] Next, quality evaluation indicators for the new target speech segments were determined.

[0099] Subsequently, from all extracted target speech segments (including historically extracted and new target speech segments), the N target speech segments with the highest quality evaluation index are selected. For example, if N=10, and a total of 50 target speech segments have been extracted, the top 10 segments are selected by sorting them from highest to lowest quality evaluation index.

[0100] Finally, based on the quality evaluation indicators of each of the N target speech segments, the voiceprint features corresponding to these N target speech segments are weighted and fused to obtain the updated voiceprint template corresponding to the target account.

[0101] This method of selecting the best based on quality evaluation metrics ensures that the set of optimal target speech segments collected historically is always used to generate the voiceprint template. This not only controls computational and storage costs but also guarantees that the voiceprint template is composed of the most representative high-quality voiceprint features, maintaining its high discriminative power. For example, when a broadcaster changes to a professional microphone, the quality of the new target speech segment significantly improves. The system will automatically discard the previous low-quality target speech segments, retaining only the optimal ones, thus obtaining a purer voiceprint template.

[0102] The aforementioned quality evaluation indicators may specifically include at least one of the following scores, providing a multi-dimensional quality assessment basis for the generation or updating of voiceprint templates: (1) Visual biometric matching score This score reflects the similarity between visual biometric features (such as facial features) extracted from the video content corresponding to the target audio segment and reference visual biometric features. In other words, the visual biometric matching score of the target audio segment is calculated based on the similarity between the visual biometric features extracted from the video content corresponding to the target audio segment and the reference visual biometric features. Specifically, for each target audio segment, its corresponding timestamp and screen region can be located. Facial features are extracted from this screen region, and the similarity between these facial features and the reference facial features collected during real-name authentication is calculated. Based on this similarity, the visual biometric matching score of the target audio segment is determined. The higher the visual biometric matching score, the more consistent the speaker of the target audio segment is with the speaker of the target account, thus making the voiceprint features generated by the target audio segment more reliable.

[0103] (2) Voice credibility score This score is obtained by performing a speaker consistency detection on the target speech segment, which is used to detect the consistency between the speaker of the segment and the person in the video frame. The speaker consistency detection can be found in the relevant description in the previous embodiments. The higher the consistency obtained from the speaker consistency detection, the higher the speech credibility score.

[0104] (3) Audio quality score This score is independent of visual information and directly measures the acoustic quality of the target speech segment itself. For example, it can be calculated by determining the signal-to-noise ratio of the target speech segment, or by using a neural network-based speech quality prediction model to output the mean opinion score. The audio quality score is used to eliminate target speech segments whose voiceprint features are distorted due to environmental noise, microphone pops, data compression distortion, etc. For example, the audio quality score of the target speech segment extracted from a video recorded in a quiet studio can reach 0.9; while the audio quality score of the target speech segment extracted from a video recorded on a noisy street is 0.4, and its voiceprint features may be severely interfered with by noise.

[0105] By establishing a multi-dimensional evaluation system that integrates visual, audio-visual correlation, and audio data, a comprehensive quantitative assessment of target speech segments is achieved. This multi-dimensional scoring mechanism avoids the limitations of evaluating target speech segments based on a single indicator, and can more accurately identify target speech segments that not only sound clear but also have visuals that are indeed of the target person, thus possessing high credibility. This provides precise numerical basis for the weighted fusion or optimal selection of target speech segments.

[0106] As another feasible approach, when generating the voiceprint template corresponding to the target account based on voiceprint features, a clustering method can be adopted, that is, clustering the voiceprint features corresponding to multiple target speech segments; and using the cluster centers obtained from the clustering as the voiceprint template corresponding to the target account.

[0107] Clustering algorithms can include K-means or adaptive clustering algorithms such as DBSCAN. During clustering, each voiceprint feature is treated as a point in a high-dimensional space, and the algorithm groups them based on the distance between feature vectors (such as cosine distance or Euclidean distance). From the resulting clusters, the cluster center of the cluster containing the most voiceprint features can be selected as the voiceprint template.

[0108] By clustering multiple voiceprint vectors in the feature space and taking the cluster centers, a geometric center that truly reflects the distribution of the speaker's essential acoustic features can be found. This method is naturally robust to outliers caused by randomly occurring noise fragments, because noise features typically do not cluster together and therefore do not affect the location of the cluster centers. The voiceprint templates generated in this way are more statistically representative and stable, and can better summarize the user's typical voiceprint patterns.

[0109] In this approach, the automatic update mechanism for the voiceprint template may include the following steps: First, in response to the detection that the target account's update history meets preset requirements (e.g., more than 5 new works, the latest work was published within 24 hours, and the work type is consistent with the previous ones), new target audio segments are extracted from these update history works based on reference visual biometrics. The extraction process can reuse the method used in step 203 above to ensure that the new target audio segments also have consistency with the speaker subject of the reference visual biometrics.

[0110] Next, voiceprint features are extracted from the new target speech segments to obtain the voiceprint features corresponding to each new target speech segment.

[0111] Next, the ratio between the first and second voiceprint features is determined. Specifically, the distance (e.g., cosine distance) between each new voiceprint feature and the current cluster center is calculated. If the distance is greater than or equal to a preset distance threshold, it is classified as a first voiceprint feature (deviation feature); if the distance is less than the preset distance threshold, it is classified as a second voiceprint feature (consistency feature). The number of features in each category is counted, and the ratio is calculated as: Number of first features / Number of second features (or Number of first features / Total number). For example, if a new target speech segment has 3 deviations and 7 consistency features, the ratio is 3 / 7 ≈ 0.43.

[0112] Then, based on the relationship between the quantity ratio and the preset ratio threshold, two cases are handled: Scenario 1: If the proportion is less than a preset threshold (e.g., the threshold is set to 0.4, but the actual proportion is 0.3, indicating that most new target speech segments still conform to the existing pattern), then the cluster centers are incrementally updated using the second voiceprint feature. Incremental updates can employ methods such as exponential moving averages. New cluster center = α × old cluster center + (1-α) × weighted average of second voiceprint features (weights are based on the quality evaluation index of each target speech segment).

[0113] The updated cluster centers serve as the updated voiceprint templates.

[0114] Scenario 2: If the proportion is greater than or equal to the preset proportion threshold (e.g., the proportion threshold is 0.4, but the actual value is 0.6, indicating that most new target speech segments deviate from the existing pattern, possibly due to voiceprint drift of the speaker or drastic environmental changes), then the old cluster centers are abandoned, and the first voiceprint feature is used directly for clustering. The new cluster centers obtained by clustering are used as the updated voiceprint template.

[0115] For example: Suppose a broadcaster's voiceprint template is represented by cluster center C0. One day, they changed to a higher-quality microphone (but the main speaker remained the same). Of the 10 newly extracted target speech segments, the voiceprint features of 8 target speech segments are less than a preset distance threshold from C0, belonging to the second voiceprint feature category, while the voiceprint features of 2 target speech segments are greater than the preset distance threshold from C0, belonging to the first voiceprint feature category. The ratio is 0.25, which is less than the preset ratio threshold of 0.4. Therefore, scenario one is triggered: C0 is incrementally updated using the 8 second voiceprint features, resulting in a fine-tuned voiceprint template C1. If the anchor's voice changes drastically after catching a cold, and the distance between the voiceprint features of 7 target voice segments and C0 is greater than the preset distance threshold, they belong to the first voiceprint feature, while the distance between the voiceprint features of 3 target voice segments and C0 is less than the preset distance threshold, they belong to the second voiceprint feature. The proportion of these three is approximately 2.3, which is greater than the preset proportion threshold of 0.4. Then, scenario two is triggered: the 7 first voiceprint features are re-clustered to obtain a new cluster center C2 as the voiceprint template, and the entire feature is updated.

[0116] By establishing a distance threshold to distinguish between the first and second voiceprint features and monitoring the ratio of the first and second voiceprint features, an adaptive update mechanism with intelligent judgment capabilities is constructed. For minor changes in voiceprint features, an incremental update strategy is used to smoothly adjust the voiceprint template, which has low computational cost and can maintain the continuity of the voiceprint template; when a drastic change in feature distribution is detected, re-clustering is triggered, achieving an effective balance between maintaining the stability of the voiceprint template state and the risk of mutation.

[0117] In another optional embodiment, to further enhance the security and reliability of the voiceprint database, the method of the present invention includes a step of verifying the authenticity of the audio data or target speech segment before extracting voiceprint features. Specifically, a pre-trained audio deep fake detection model can be used to identify the audio data or target speech segment and calculate its probability score of belonging to real human speech. When the score is lower than a preset deep fake detection threshold, it indicates that the audio data or target speech segment is highly likely to be generated by attack methods such as AI synthesis, speech conversion, or recording playback, and is therefore filtered out and not used for subsequent voiceprint feature extraction and voiceprint template generation.

[0118] Furthermore, to ensure user privacy and security, this application embodiment may employ a feature-level encryption storage strategy when generating and storing the voiceprint template corresponding to the target account. The generated voiceprint template is encrypted, and only the encrypted voiceprint template is associated with the target account identifier and stored in the voiceprint database. The original voice data and voiceprint features are not retained locally or in memory. This ensures that even if the voiceprint database is accidentally leaked, attackers cannot directly obtain the user's voiceprint features or voice data, minimizing privacy risks.

[0119] Figure 3 The diagram shown is a schematic flowchart of a voiceprint authentication method provided in an embodiment of this application. This voiceprint authentication method can be used in conjunction with the aforementioned voiceprint registration method. Figure 3 As shown, the method may include the following steps: Step 301: Receive the voice to be authenticated and extract the voiceprint features of the voice to be authenticated to obtain the voiceprint features corresponding to the voice to be authenticated.

[0120] The voice to be authenticated is typically captured by the user's microphone in scenarios requiring identity verification and then sent to the server. Upon receiving the voice, the server performs preprocessing and voiceprint feature extraction. The model used for voiceprint feature extraction should maintain the same architecture as the model used in the registration phase to ensure comparability of the feature space. The extracted voiceprint features are denoted as the voiceprint features corresponding to the voice to be authenticated.

[0121] Step 303: Determine the similarity between the voiceprint features corresponding to the voice to be authenticated and the voiceprint template corresponding to the target account.

[0122] The server retrieves the corresponding voiceprint template from the voiceprint database based on the target account's identifier. Retrieval can be based on the account ID or the real-name entity identifier bound to the account. Subsequently, the similarity between the voiceprint features corresponding to the speech to be authenticated and the voiceprint template corresponding to the target account is calculated. Similarity can be calculated using methods such as cosine similarity or Euclidean distance.

[0123] Step 305: Based on the comparison result between the similarity and the preset similarity threshold, output the authentication result.

[0124] Authentication is considered successful if the calculated similarity score is greater than or equal to the similarity threshold; otherwise, authentication fails. The server then decides whether to grant the user the requested permissions or services based on this result.

[0125] As can be seen, this application provides a voiceprint authentication process that seamlessly integrates with the aforementioned automatic registration method, enabling the constructed voiceprint database to be directly applied to speaker verification tasks. It can quickly and accurately determine whether the current speaker's identity matches the target account holder, thereby transforming the zero-operation-cost registration advantage into usable security authentication capabilities.

[0126] Furthermore, to address the varying security risks across different service scenarios, this application also provides a dynamic security strategy. It can identify the risk level of the authentication scenario in which the voice message to be authenticated occurs. The risk level of the scenario can be comprehensively assessed based on multi-dimensional information such as service type (e.g., browsing, payment, password modification), device environment (e.g., whether it's a frequently used device), network environment, and user behavior habits. Then, based on the identified risk level, the similarity threshold used in the authentication process is set in reverse. The setting logic follows a positive correlation principle: the higher the risk level, the higher the similarity threshold required for authentication.

[0127] By establishing a dynamic adjustment mechanism that positively correlates the similarity threshold used in the authentication process with the risk of the scenario, a differentiated security strategy is achieved. For example, in low-risk scenarios (such as browsing content), a lenient similarity threshold is used to improve user experience and smoothness; in high-risk scenarios (such as transaction payments), the similarity threshold is automatically tightened to increase the requirement for consistency of speaker identity, thereby improving the system's security against fraud and theft without increasing the user's interaction burden.

[0128] Furthermore, this application also supports multimodal joint authentication decision-making. After identifying the risk level of the authentication scenario, the authentication strategy can be dynamically selected. Risk level identification can be achieved through analysis of multiple dimensions, such as: detecting whether the source device of the current request (e.g., authentication request or service request) is a frequently used device (e.g., if the device is newly registered or has never been associated with the target account, the risk is increased); analyzing the network environment of the request (e.g., public Wi-Fi, proxy IP); assessing the deviation between the collection time of the voice to be authenticated and the user's historical active time (e.g., abnormal requests at 3 AM); and checking whether there are multiple authentication failure records recently. Based on the above information, the risk level can be divided into three levels: low, medium, and high.

[0129] Authentication strategies may include: voiceprint authentication methods, or voiceprint authentication methods combined with at least one other authentication method; wherein, the other authentication methods may include: Authentication methods based on visual biometrics: such as requiring users to perform real-time facial recognition (comparing with reference visual biometrics) to ensure that the speaker of the voice to be authenticated is consistent with the face during real-name authentication, and to prevent recording replay or speech synthesis attacks.

[0130] Authentication methods based on voice liveness detection: Detect whether the voice to be authenticated is a real human voice rather than a recording playback or TTS synthesis, for example by randomly playing challenge-response phrases (such as asking the user to read random numbers) or analyzing breathing or lip noise features in the voice.

[0131] Authentication methods based on the device from which the voice message to be authenticated originates: verifying the device ID, SIM card information, security certificate, etc., to ensure that the request comes from a trusted terminal (such as a mobile phone or smart speaker that has been bound to the target account).

[0132] Authentication methods based on the behavior of the target account: Analyze whether the current behavior matches the historical pattern of the target account (such as frequently used login locations, payment habits, and social operation sequences).

[0133] Manual authentication methods include: transferring the call to customer service via video call, requesting identification documents, or manual verification by an administrator.

[0134] For example, a user attempts to initiate a voiceprint transfer command from a new phone that has never been logged into before. The device is detected as abnormal, and the risk level is determined to be high, thus triggering multi-factor authentication: the user is required to first complete voiceprint authentication (similarity calculation against a voiceprint template), then a random three-digit number "247" is played and the user is asked to repeat it (i.e., voice liveness detection), and simultaneously, the front-facing camera is activated for facial recognition (visual biometric authentication). The transaction is only allowed after all three steps are successful. If only single voiceprint authentication is used, an attacker could easily bypass it using speech synthesis.

[0135] As can be seen, by combining scenario risk assessment with a multi-authentication strategy orchestration engine, it is possible to intelligently and seamlessly overlay authentication dimensions such as facial recognition, liveness detection, and device fingerprinting according to the risk level, forming a defense-in-depth system. In low-risk scenarios, the convenience of voiceprint authentication is maintained, while in high-risk scenarios, the fusion of multiple authentication strategies significantly increases the difficulty for attackers to bypass multiple authentications simultaneously, achieving a balance between security and user convenience.

[0136] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0137] Figure 4 A schematic block diagram of the voiceprint registration device provided in the embodiments of this application, such as Figure 4 As shown, the device 400 includes: a reference acquisition unit 401, a segment acquisition unit 402, a voiceprint extraction unit 403, and a voiceprint maintenance unit 404. The main functions of each component are as follows: The reference acquisition unit 401 is configured to acquire the historical works of the target account and the reference visual biometric features used to identify the identity of the subject corresponding to the target account, wherein the historical works include video data.

[0138] The segment acquisition unit 402 is configured to extract a target speech segment from the video data based on the reference visual biometrics, wherein the speaker of the target speech segment is consistent with the subject corresponding to the target account.

[0139] The voiceprint extraction unit 403 is configured to extract voiceprint features from the target speech segment to obtain the voiceprint features corresponding to the target speech segment.

[0140] The voiceprint maintenance unit 404 is configured to generate and store the voiceprint template corresponding to the target account based on the voiceprint features.

[0141] As one possible implementation, the segment acquisition unit 402 can be configured to: perform visual biometric detection on the video data; generate one or more candidate visual biometrics and video content corresponding to each candidate visual biometric based on the detection results; determine the similarity between each candidate visual biometric and the reference visual biometric; determine the video content corresponding to the candidate visual biometric that meets the similarity requirement as the target video content; and extract the target speech segment from the target video content.

[0142] As one possible implementation method, when the segment acquisition unit 402 extracts the target audio segment from the target video content, it can be specifically configured to: perform a consistency detection of the speaker subject on each video segment contained in the target video content, determine the target video segment whose consistency with the subject corresponding to the target account meets the consistency requirements; and extract the audio segment from the target video segment as the target audio segment.

[0143] As one possible implementation method, when the segment acquisition unit 402 performs voice subject consistency detection on each video segment contained in the target video content, it can be specifically configured to: determine the temporal correlation between lip movement and speech signal in the video segment based on lip movement detection; determine the synchronization relationship between video frame and speech signal in the video segment; determine the similarity between the visual biometric features of the voice subject in the video segment and the reference visual biometric features; and determine the consistency between the voice subject of the video segment and the subject corresponding to the target account based on at least one of the temporal correlation, the synchronization relationship, and the similarity.

[0144] As another possible implementation, the segment acquisition unit 402 can be configured to: distinguish multiple audio tracks from the target video content; and perform at least one of background audio track filtering, background music audio track filtering, transported audio track filtering, and dubbing audio track filtering on the multiple audio tracks to separate the audio segment of the main vocal audio track as the target audio segment.

[0145] As one possible implementation, the target speech segments are multiple; the voiceprint maintenance unit 404 can be specifically configured to: determine the quality evaluation index of each of the multiple target speech segments; based on the quality evaluation index of each target speech segment, perform weighted fusion of the voiceprint features corresponding to the multiple target speech segments to obtain the voiceprint template corresponding to the target account; or, based on the quality evaluation index of each target speech segment, select voiceprint features from the voiceprint features corresponding to the multiple target speech segments as the voiceprint template corresponding to the target account.

[0146] Furthermore, the segment acquisition unit 402 can be specifically configured to: in response to the target account's updated historical works meeting preset requirements, extract new target speech segments from the updated historical works based on the reference visual biometric features. The voiceprint extraction unit 403 can also be configured to: determine multiple target speech segments within a preset time window; extract voiceprints from the multiple target speech segments within the preset time window respectively, obtaining voiceprint features corresponding to the multiple target speech segments within the preset time window. Correspondingly, the voiceprint maintenance unit 404, for the multiple target speech segments within the preset time window, switches to the processing of determining the quality evaluation indicators of the multiple target speech segments respectively, thereby obtaining the updated voiceprint template corresponding to the target account.

[0147] Furthermore, the segment acquisition unit 402 can be specifically configured to: in response to the target account's updated historical works meeting preset requirements, extract new target speech segments from the updated historical works based on the reference visual biometrics. The voiceprint maintenance unit 404 can be configured to: determine the quality evaluation index of the new target speech segments respectively; select the N target speech segments with the highest quality evaluation index from all extracted target speech segments; and, based on the quality evaluation index of the N target speech segments, perform weighted fusion of the voiceprint features corresponding to the N target speech segments to obtain the updated voiceprint template corresponding to the target account, where N is a preset positive integer.

[0148] The quality evaluation indicators include at least one of visual biometric matching score, vocal credibility score, and audio quality score; The visual biometric matching score is obtained based on the similarity between the visual biometrics extracted from the video content corresponding to the target speech segment and the reference visual biometrics. The speech credibility score is obtained by performing a speech subject consistency detection on the target speech segment. The speech subject consistency detection is used to detect the consistency between the speech subject of the target speech segment and the person in the video frame.

[0149] As another possible approach, there are multiple target speech segments; the voiceprint maintenance unit 404 can be specifically configured to: cluster the voiceprint features corresponding to the multiple target speech segments; and use the cluster centers obtained from the clustering as the voiceprint template corresponding to the target account.

[0150] Furthermore, the segment acquisition unit 402 can also be specifically configured to: in response to the target account's updated historical works meeting preset requirements, extract new target speech segments from the updated historical works based on the reference visual biometric features. The voiceprint extraction unit 403 is further configured to extract voiceprint features from the new target speech segments to obtain the voiceprint features corresponding to the new target speech segments. Correspondingly, the voiceprint maintenance unit 404 can also be configured to determine the quantity ratio between the first voiceprint feature and the second voiceprint feature, wherein the first voiceprint feature is the voiceprint feature corresponding to the new target speech segment that is greater than or equal to a preset distance threshold from the cluster center, and the second voiceprint feature is the voiceprint feature corresponding to the new target speech segment that is less than the preset distance threshold from the cluster center; if the quantity ratio is less than the preset ratio threshold, the cluster center is incrementally updated using the second voiceprint feature, and the updated cluster center is used as the updated voiceprint template corresponding to the target account; if the quantity ratio is greater than or equal to the preset ratio threshold, clustering is performed using the first voiceprint feature, and the new cluster center obtained by clustering is used as the updated voiceprint template corresponding to the target account.

[0151] Figure 5 A schematic block diagram of the voiceprint authentication device provided in the embodiments of this application, such as Figure 5 As shown, the device 400 includes: a voiceprint extraction unit 501, a voiceprint authentication unit 502, and a result acquisition unit 503. It may also include a risk identification unit 504 and a strategy determination unit 505. The main functions of each component are as follows: The voiceprint extraction unit 501 is configured to extract voiceprint features from the speech to be authenticated, thereby obtaining the voiceprint features corresponding to the speech to be authenticated.

[0152] The voiceprint authentication unit 502 is configured to determine the similarity between the voiceprint features corresponding to the voice to be authenticated and the voiceprint template corresponding to the target account.

[0153] The result acquisition unit 503 is configured to pass authentication if the similarity is greater than or equal to the similarity threshold; otherwise, authentication fails.

[0154] The voiceprint template corresponding to the target account is registered by the voiceprint registration device 400.

[0155] Furthermore, the risk identification unit 504 can be configured to identify the risk level of the authentication scenario for the voice to be authenticated.

[0156] The result acquisition unit 503 can also be configured to determine the similarity threshold based on the identified risk level, wherein the risk level is positively correlated with the similarity threshold.

[0157] Furthermore, the risk identification unit 504 can be configured to identify the risk level of the authentication scenario for the voice to be authenticated.

[0158] The strategy determination unit 505 can be configured to: determine the authentication strategy to be adopted based on the identified risk level, wherein the authentication strategy includes: the voiceprint authentication method, or the voiceprint authentication method combined with at least one other authentication method; wherein the other authentication method includes an authentication method based on visual biometrics, an authentication method based on voice liveness detection, an authentication method based on the device from which the voice to be authenticated originates, an authentication method based on the behavior corresponding to the target account, or a manual authentication method.

[0159] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or device embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the description of the method embodiments. The system and device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0160] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0161] And an electronic device, comprising: One or more processors; and A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0162] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0163] in, Figure 6 An exemplary architecture of an electronic device is shown, which may include a processor 610, a video display adapter 611, a disk drive 612, an input / output interface 613, a network interface 614, and a memory 620. The processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, and memory 620 can communicate with each other via a communication bus 630.

[0164] The processor 610 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs and implement the technical solution provided in this application.

[0165] The memory 620 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 620 can store the operating system 621 for controlling the operation of the electronic device 600, and the basic input / output system (BIOS) 622 for controlling the low-level operations of the electronic device 600. Additionally, it can store a web browser 623, a data storage management system 624, and a voiceprint registration device 400 / voiceprint authentication device 500, etc. The aforementioned voiceprint registration device 400 / voiceprint authentication device 500 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 620 and executed by the processor 610.

[0166] Input / output interface 613 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0167] Network interface 614 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0168] Bus 630 includes a pathway for transmitting information between various components of the device, such as processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, and memory 620.

[0169] It should be noted that although the above-described device only shows the processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, memory 620, bus 630, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0170] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer program product. This computer program product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0171] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A voiceprint registration method, characterized in that, The method includes: Obtain the target account's historical works and reference visual biometric features used to identify the subject corresponding to the target account, wherein the historical works include video data; Based on the reference visual biometrics, a target audio segment is extracted from the video data, and the speaker of the target audio segment is consistent with the subject corresponding to the target account; The target speech segment is subjected to voiceprint feature extraction to obtain the voiceprint features corresponding to the target speech segment; Based on the voiceprint features, a voiceprint template corresponding to the target account is generated and stored.

2. The method according to claim 1, characterized in that, The step of extracting the target speech segment from the video data based on the reference visual biometric features includes: Visual biometrics detection is performed on the video data, and one or more candidate visual biometrics and the corresponding video content for each candidate visual biometric are generated based on the detection results. The similarity between each candidate visual biometric feature and the reference visual biometric feature is determined, and the video content corresponding to the candidate visual biometric feature whose similarity meets the similarity requirement is determined as the target video content. Extract the target audio segment from the target video content.

3. The method according to claim 2, characterized in that, The step of extracting the target audio segment from the target video content includes: Perform a voice subject consistency detection on each video segment contained in the target video content to determine the target video segments whose voice subject and the subject corresponding to the target account meet the consistency requirements. Extract the audio segment from the target video segment as the target audio segment.

4. The method according to claim 3, characterized in that, The step of performing speaker consistency detection on each video segment contained in the target video content includes: Based on lip movement detection, the temporal correlation between lip movements and speech signals in video clips is determined; Determine the synchronization relationship between the video frame and the audio signal in the video clip; Determine the similarity between the visual biometrics of the speaker in the video clip and the reference visual biometrics; The consistency between the speaker of the video segment and the subject corresponding to the target account is determined based on at least one of the time correlation, the synchronization relationship, and the similarity.

5. The method according to claim 2, characterized in that, The step of extracting the target audio segment from the target video content includes: Distinguish multiple audio tracks from the target video content; At least one of the following is applied to the multiple speech tracks: background audio track filtering, background music track filtering, transferred audio track filtering, and dubbing audio track filtering, in order to separate the speech segment of the main vocal track as the target speech segment.

6. The method according to claim 1, characterized in that, The target speech segments are multiple; based on the voiceprint features, a voiceprint template corresponding to the target account is generated, including: Determine the quality evaluation indicators for the multiple target speech segments respectively; Based on the quality evaluation index of each target speech segment, the voiceprint features corresponding to the multiple target speech segments are weighted and fused to obtain the voiceprint template corresponding to the target account; or, based on the quality evaluation index of each target speech segment, voiceprint features are selected from the voiceprint features corresponding to the multiple target speech segments as the voiceprint template corresponding to the target account.

7. The method according to claim 6, characterized in that, The method further includes: In response to the target account's updated historical works meeting preset requirements, new target audio segments are extracted from the updated historical works based on the reference visual biometrics; Identify multiple target speech segments within a preset time window; Voiceprint extraction is performed on multiple target speech segments within the preset time window to obtain the voiceprint features corresponding to the multiple target speech segments within the preset time window. For multiple target speech segments within the preset time window, proceed to the step of determining the quality evaluation indicators of the multiple target speech segments respectively, and then obtain the updated voiceprint template corresponding to the target account.

8. The method according to claim 6, characterized in that, The method further includes: In response to the target account's updated historical works meeting preset requirements, new target audio segments are extracted from the updated historical works based on the reference visual biometrics; Determine the quality evaluation indicators for the new target speech segments respectively; Select the N target speech segments with the highest quality evaluation index from all extracted target speech segments. Based on the quality evaluation index of the N target speech segments, perform weighted fusion on the voiceprint features corresponding to the N target speech segments to obtain the updated voiceprint template corresponding to the target account. N is a preset positive integer.

9. The method according to any one of claims 6 to 8, characterized in that, The quality evaluation indicators include at least one of visual biometric matching score, vocal credibility score, and audio quality score; The visual biometric matching score is obtained based on the similarity between the visual biometrics extracted from the video content corresponding to the target speech segment and the reference visual biometrics. The speech credibility score is obtained by performing a speech subject consistency detection on the target speech segment. The speech subject consistency detection is used to detect the consistency between the speech subject of the target speech segment and the person in the video frame.

10. The method according to claim 1, characterized in that, The target speech segments are multiple; based on the voiceprint features, a voiceprint template corresponding to the target account is generated, including: Cluster the voiceprint features corresponding to multiple target speech segments; The cluster centers obtained from the clustering are used as the voiceprint templates corresponding to the target account.

11. The method according to claim 10, characterized in that, The method further includes: In response to the target account's updated historical works meeting preset requirements, new target audio segments are extracted from the updated historical works based on the reference visual biometrics; The new target speech segment is subjected to voiceprint feature extraction to obtain the voiceprint features corresponding to the new target speech segment; Determine the ratio between the first voiceprint feature and the second voiceprint feature, wherein the first voiceprint feature is the voiceprint feature corresponding to the new target speech segment that is greater than or equal to a preset distance threshold from the cluster center, and the second voiceprint feature is the voiceprint feature corresponding to the new target speech segment that is less than the preset distance threshold from the cluster center. If the quantity ratio is less than the preset ratio threshold, the cluster center is incrementally updated using the second voiceprint feature, and the updated cluster center is used as the updated voiceprint template corresponding to the target account. If the quantity ratio is greater than or equal to a preset ratio threshold, then clustering is performed using the first voiceprint feature, and the new cluster center obtained by clustering is used as the updated voiceprint template corresponding to the target account.

12. A voiceprint authentication method, characterized in that, The method includes: Voiceprint features are extracted from the speech to be authenticated to obtain the voiceprint features corresponding to the speech to be authenticated; Determine the similarity between the voiceprint features corresponding to the voice to be authenticated and the voiceprint template corresponding to the target account; If the similarity is greater than or equal to the similarity threshold, authentication is successful; otherwise, authentication fails. The voiceprint template corresponding to the target account is registered using the method described in any one of claims 1 to 11.

13. The method according to claim 12, characterized in that, The method further includes: The risk level of the authentication scenario is identified for the voice to be authenticated; The similarity threshold is determined based on the identified risk level, wherein the risk level is positively correlated with the similarity threshold.

14. The method according to claim 12, characterized in that, The method further includes: The risk level of the authentication scenario is identified for the voice to be authenticated; Based on the identified level of risk, an authentication strategy is determined, which includes: the voiceprint authentication method, or the voiceprint authentication method combined with at least one other authentication method; The other authentication methods include authentication methods based on visual biometrics, authentication methods based on voice liveness detection, authentication methods based on the device from which the voice to be authenticated originates, authentication methods based on the behavior corresponding to the target account, or manual authentication methods.

15. A voiceprint registration device, characterized in that, The device includes: The reference acquisition unit is configured to acquire the historical works of the target account and the reference visual biometric features used to identify the subject identity corresponding to the target account, wherein the historical works include video data; The segment acquisition unit is configured to extract target speech segments from the video data based on the reference visual biometrics, wherein the speaker of the target speech segment is consistent with the subject corresponding to the target account; The voiceprint extraction unit is configured to extract voiceprint features from the target speech segment to obtain the voiceprint features corresponding to the target speech segment. The voiceprint maintenance unit is configured to generate and store a voiceprint template corresponding to the target account based on the voiceprint features.

16. A voiceprint authentication device, characterized in that, The device includes: The voiceprint extraction unit is configured to extract voiceprint features from the speech to be authenticated, thereby obtaining the voiceprint features corresponding to the speech to be authenticated. The voiceprint authentication unit is configured to determine the similarity between the voiceprint features corresponding to the voice to be authenticated and the voiceprint template corresponding to the target account; The result acquisition unit is configured to pass authentication if the similarity is greater than or equal to a similarity threshold; otherwise, authentication fails. The voiceprint template corresponding to the target account is registered by the voiceprint registration device of claim 15.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1 to 14.

18. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 14.

19. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1 to 14.

Images