Method for normalizing speaker similarity using meta data

Abstract

The present invention relates to a method for normalizing speaker similarity using metadata, wherein speaker embeddings are extracted from a pre-stored dataset based on the metadata of voice data to select a cohort, and statistical values ​​calculated from the cohort for each voice data are applied to the similarity between voice data to calculate a normalized similarity based on environmental factors.

Description

Technology Field

[0001] The present invention relates to a technique for calculating similarity between speaker embeddings extracted from speech for speaker recognition. Background Technology

[0002] Speaker recognition is a technique that estimates speaker information from a given speech. It extracts speaker embeddings from each speech, calculates similarity by comparing two speaker embeddings, and recognizes the speaker by determining whether they are the same person based on whether the calculated similarity exceeds a threshold.

[0003] Speaker embeddings refer to vectors of a fixed size created by extracting features from speech input. Speaker embedding extraction is mostly implemented by artificial neural network models, which are designed to extract speaker embeddings of a fixed size regardless of the length of the input speech through pooling layers. The similarity between speaker embeddings is generally calculated using cosine similarity, and speaker recognition is performed by determining whether the speakers are the same based on whether the similarity exceeds a threshold.

[0004] In this regard, an Adaptive Score Normalization technique is being introduced to improve speaker recognition performance. The Adaptive Score Normalization technique can improve speaker identification performance by reducing the variability in similarity between speaker embeddings. This is achieved by dividing the data into subsets called cohorts based on the similarity of speaker embeddings and identifying speakers based on the similarity between the speech data and each cohort.

[0005] However, since cohorts are selected based on the similarity of speaker embeddings, there is a limitation in that environmental information of the speech is not reflected in the normalization. Specifically, the similarity between speaker embeddings can vary depending on environmental factors such as language, gender, speech length, and the presence of noise. Existing speech similarity normalization techniques perform normalization simply based on similarity without considering these environmental factors, which risks degrading speaker identification performance due to distorted similarity distributions. Prior art literature

[0006] Republic of Korea Patent Application No. 10-2023-0013550 Republic of Korea Patent Application No. 10-2021-0194701 Republic of Korea Patent Application No. 10-2019-0051557 The problem to be solved

[0007] The objective of the present invention is to solve the above problem by providing a speaker similarity normalization method using metadata that performs similarity normalization based on environmental factors by utilizing metadata about speech.

[0008] The objectives of the present invention are not limited to those mentioned above, and other unmentioned objectives will be clearly understood from the description below. means of solving the problem

[0009] A method for normalizing speaker similarity using metadata according to one aspect of the present invention for achieving the aforementioned purpose is a method performed by an electronic device, comprising: a step of calculating similarity between two voice data from speaker embeddings extracted from two voice data; a step of constructing a cohort for each voice data by extracting speaker embeddings from a dataset that has been stored in advance including speaker embeddings and metadata for each speaker based on metadata consisting of at least one key-value pair for each voice data; a step of calculating statistics from the cohort constructed for each voice data; and a step of normalizing the similarity between two voice data using the statistics calculated for each voice data. Effects of the invention

[0010] According to the present invention, by utilizing metadata for speech and normalizing speaker similarity by considering the similarity distribution according to environmental factors, there is an effect of improving speaker identification performance.

[0011] The effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description in the claims. Brief explanation of the drawing

[0012] FIG. 1 is a flowchart of a speaker similarity normalization method using metadata according to an embodiment of the present invention. FIG. 2 is a block diagram of an electronic device in which a speaker similarity normalization method using metadata according to an embodiment of the present invention is performed. Figure 3 is an illustrative diagram for explaining the process of extracting speaker embeddings from a dataset in an embodiment of the present invention. Figure 4 is an example diagram illustrating the process of selecting speaker embeddings extracted from a dataset as a cohort in an embodiment of the present invention. FIG. 5 is an example diagram illustrating the process of calculating statistics using speaker embeddings belonging to a cohort in an embodiment of the present invention. Specific details for implementing the invention

[0013] The advantages and features of the present invention, and the methods for achieving them, will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. Meanwhile, the terms used in this specification are for describing the embodiments and are not intended to limit the present invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text.

[0014] The present invention relates to a technique for normalizing the similarity between speaker embeddings to improve speaker identification performance by reducing the variability of speaker similarity.

[0015] In particular, the present invention is characterized by a technical feature of improving speaker identification performance by normalizing similarity according to the similarity distribution with speaker embeddings in an environment similar to voice data.

[0016] These technical features can be achieved by a configuration that extracts speaker embeddings from a pre-stored dataset based on metadata for voice data to select a cohort, and calculates statistics for normalization from the speaker embeddings of the cohort.

[0017] Hereinafter, a method for normalizing speaker similarity using metadata according to an embodiment of the present invention will be specifically described with reference to the attached drawings.

[0018] FIG. 1 is a flowchart of a speaker similarity normalization method using metadata according to an embodiment of the present invention, and the steps illustrated in the flowchart can be implemented as computer instructions that perform specified functions by being loaded into a processor or memory of an electronic device capable of data processing (e.g., a general-purpose computer, a specialized computer, a portable notebook computer, a network computer).

[0019] Since computer instructions can be stored in computer-readable memory, the functions described in the steps of the flowchart may be produced as manufactured products containing means of instruction to perform them.

[0020] Referring to FIG. 2, the electronic device (100) may include a memory (110), an input / output interface (120), and a processor (130).

[0021] The memory (110), input / output interface (120), and processor (130) can be connected to each other and communicate through the bus (101).

[0022] The memory (910) may be volatile memory or non-volatile memory.

[0023] The memory (110) can store computer commands for performing a speaker similarity normalization method using metadata according to an embodiment of the present invention.

[0024] The input / output interface (120) is connected to a device for input or output and can receive user input through an input device or output data processing results through an output device.

[0025] Here, input devices may include a mouse, keyboard, etc., and output devices may include a display, etc., but are not limited thereto.

[0026] A processor (130) is one or more devices that execute instructions or programs consisting of instructions, such as a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), etc.

[0027] The processor (130) can perform a speaker similarity normalization method using metadata according to an embodiment of the present invention by executing instructions stored in memory (910).

[0028] The processor (130) can receive at least two voice data and metadata for each voice data (S100).

[0029] Here, voice data consists of amplitude values ​​for multiple points in time over time and refers to voice data input by the user.

[0030] Metadata for each voice data may consist of at least one key-value pair to distinguish environmental factors of the voice.

[0031] For example, the key may include gender, channel, language, and voice segment length.

[0032] If the key is gender, the value for the key is for either male or female; if the key is channel, the value for the key is for any one of multiple channels including telephone; if the key is language, the value for the key is for any one of multiple languages ​​including Korean, English, Japanese, and Chinese; and if the key is speech interval length, the value for the key may be a value based on the time at which speech is recognized as existing in the speech data.

[0033] The processor (130) can receive at least two metadata consisting of at least two pre-generated voice data and at least one key-value pair for each voice data according to user input (S100).

[0034] The processor (130) can extract speaker embeddings from each of two voice data among the input voice data (S200).

[0035] The processor (130) can extract a first speaker embedding of a predetermined size from the first voice data and extract a second speaker embedding of a predetermined size from the second voice data (S200).

[0036] The processor (130) receives the first voice data and the second voice data, respectively, and inputs them into a preset embedding extraction model to extract a speaker embedding in the form of a vector consisting of a predetermined number of values ​​regardless of the number of samples in the voice data, thereby extracting the first speaker embedding and the second speaker embedding (S200).

[0037] For example, the processor (130) First speech data consisting of samples is input into a preset embedding extraction model to obtain a first speaker embedding consisting of F values ​​( Extract ) and, ( By inputting the second speech data consisting of ) samples into a preset embedding extraction model, the second speaker embedding consisting of F values ​​( ) can be extracted.

[0038] Here, the embedding extraction model may utilize existing technologies developed and used to extract embeddings from speech data.

[0039] For example, embedding extraction models such as x-vector, ECAPA-TDNN, and WavLM may be used, but are not limited to these.

[0040] Afterwards, the processor (130) can calculate the similarity between two voice data from speaker embeddings extracted from two voice data (S300).

[0041] The processor (130) can calculate the cosine similarity between the first speaker embedding extracted from the first voice data and the second speaker embedding extracted from the second voice data as the similarity between the two voice data (S300).

[0042] The processor (130) can calculate the cosine similarity between the first speaker embedding and the second speaker embedding according to the following mathematical formula (S300).

[0043]

[0044] Here, is the i-th value included in the first speaker embedding, is the i-th value included in the second speaker embedding, represents the size of the speaker embedding.

[0045] Afterwards, the processor (130) can construct a cohort for each voice data by extracting speaker embeddings from a pre-stored dataset containing speaker embeddings and metadata based on metadata consisting of at least one key-value pair for each voice data (S400).

[0046] The dataset may be stored in memory (110) in advance and may include an identifier for identifying each speaker corresponding to multiple voice data in which the speaker is identified, a speaker embedding extracted from the voice data, and metadata consisting of at least one key-value pair.

[0047] Identifiers, speaker embeddings, and metadata included in the dataset can be matched by speaker.

[0048] The processor (130) can form a cohort by extracting speaker embeddings included in the dataset in which the key-value pairs included in the matched metadata match the key-value pairs included in the metadata for each voice data (S400).

[0049] Specifically, the processor (130) can extract speaker embeddings containing the same key and value pairs among the speaker embeddings included in the dataset for each pair of key and value included in the metadata for each voice data (first voice data, second voice data) (S410).

[0050] For example, referring to FIG. 3, if the key-value pairs included in the metadata (1000) for the first voice data are gender-male (1010), channel-phone (1020), and voice segment length-4-8 seconds (1030), then in the dataset, metadata (2100) containing key-value pairs matching gender-male, metadata (2200) containing key-value pairs matching channel-phone, and metadata (2300) containing key-value pairs matching voice segment length-4-8 seconds can be searched to extract speaker embeddings matched to the searched metadata.

[0051] And, the processor (130) can calculate the number of key-value pairs that match the key-value included in the metadata for each voice data in the metadata of the extracted speaker embeddings (S420).

[0052] The processor (130) can calculate the number of key-value pairs that match the key-value pairs included in the metadata for each voice data by counting the extracted speaker embeddings by identifier (S420).

[0053] For example, referring to FIG. 3, since a total of two speaker embeddings are extracted by extracting one for gender-male and one for channel-phone, the processor (130) can calculate the number of speaker embeddings matched to A001 as 2 by counting the speaker embeddings matched to identifier A001 among the extracted speaker embeddings.

[0054] Afterwards, the processor (130) can add speaker embeddings to the cohort for each voice data in order of the largest number of calculated embeddings (S430).

[0055] Referring to FIG. 4, the processor (130) calculates a predetermined number of cohorts (in order of largest to smallest number) Speaker embeddings of the size of ) can be added to the cohort for each voice data (S430).

[0056] According to the above configuration, the processor (130) can form a cohort for the first voice data based on megadata for the first voice data and form a cohort for the second voice data based on megadata for the second voice data.

[0057] Afterward, the processor (130) can calculate statistics from the cohort configured for each voice data (S500).

[0058] The processor (130) can calculate the similarity between the speaker embedding extracted from each voice data and each speaker embedding belonging to a cohort of voice data (S510).

[0059] The processor (130) can calculate the similarity between a speaker embedding extracted from one of the two voice data and each speaker embedding belonging to a cohort for the other voice data (S510).

[0060] The processor (130) has a first speaker embedding extracted from the first voice data and a predetermined number of cohorts belonging to the cohort for the second voice data ( Calculate cosine similarity with each speaker embedding by the amount of ), and with respect to the first speech data, with respect to the cohort of the second speech data Calculate the number of similarities, and the number of predetermined cohorts belonging to the cohort for the second speaker embedding extracted from the second voice data and the first voice data ( Calculate cosine similarity with each speaker embedding by the amount of ), and with respect to the second speech data and the cohort of the first speech data Similarity can be calculated (S510).

[0061] And, the processor (130) can calculate statistics for each voice data by calculating the mean and standard deviation of the similarity of each speaker embedding in the cohort (S520).

[0062] The processor (130) can calculate statistics for each voice data by calculating the mean and standard deviation from the similarities calculated for one of the two voice data, for a predetermined number of cohorts belonging to the other cohort (S520).

[0063] For example, referring to FIG. 5, the processor (130) has a first speaker embedding extracted from the first voice data as a statistical value for the first voice data ( ) and the cohort of the second voice data ( Each speaker embedding belonging to ) , , , , Average of similarity between ) ) and standard deviation( Calculate ) and the second speaker embedding extracted from the second voice data as a statistical value for the second voice data ( ) and the cohort of the first voice data ( Each speaker embedding belonging to ) , , , , Average of similarity between ) ) and standard deviation( ) can be produced.

[0064] Afterwards, the processor (130) can normalize the similarity between two voice data using the statistics calculated for each voice data (S600).

[0065] The processor (130) can calculate a normalized similarity by subtracting the average calculated for each voice data from the similarity between two voice data and dividing by the standard deviation calculated for each voice data (S600).

[0066] The processor (130) can calculate a first normalized value by subtracting the mean of the first voice data from the similarity between the first voice data and the second voice data and dividing the result by the standard deviation of the first voice data, calculate a second normalized value by subtracting the mean of the second voice data from the similarity between the first voice data and the second voice data and dividing the result by the standard deviation of the second voice data, and calculate a normalized similarity by averaging the first normalized value and the second normalized value (S600).

[0067] The processor (130) normalizes the similarity according to the following mathematical formula ( ) can be calculated (S600).

[0068]

[0069] Here, is the similarity between the first voice data and the second voice data, is the average for the first voice data, is the standard deviation for the first voice data, is the average for the second voice data, represents the standard deviation for the second voice data.

[0070] According to the present invention, in normalizing the similarity of two voice data, the similarity of the two voice data can be normalized based on the similarity distribution of speaker embeddings belonging to a cohort extracted based on the speaker embedding of each voice data and the metadata of the other voice data.

[0071] That is, according to the present invention, similarity can be normalized to reduce variability in similarity by utilizing the similarity distribution between the voice data being compared and a speaker embedding in an environment similar to that of the voice data.

[0072] By recognizing speakers based on normalized similarity according to the present invention, the problem of misidentifying the same speaker as a different speaker or misidentifying different speakers as the same speaker due to differences in environmental factors is prevented, and the advantage of improving speaker recognition performance can be expected.

[0073] A person skilled in the art to which the present invention pertains will understand that the present invention may be implemented in other specific forms without altering its technical concept or essential features. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the claims and their equivalents should be interpreted as being included within the scope of the present invention. Explanation of the symbols

[0074] 100 : Electronic device 101 : Bus 110 : Memory 120: Input / Output Interface 130 : Processor

Claims

Claim 1 A method for normalizing speaker similarity using metadata, comprising: a step of calculating similarity between two voice data from speaker embeddings extracted from two voice data, performed by an electronic device; a step of extracting speaker embeddings containing the same key-value pair among speaker embeddings included in a dataset that is pre-stored including speaker embeddings and metadata for each speaker based on metadata consisting of at least one key-value pair for each voice data, for each key-value pair included in the metadata for each voice data, calculating the number of key-value pairs that match the key-value included in the metadata for each voice data from the metadata of the extracted speaker embeddings, and constructing a cohort for each voice data by adding speaker embeddings to a predetermined number of cohorts in order of the calculated number, up to the largest number; a step of calculating a statistical value from the cohort constructed for each voice data; and a step of normalizing the similarity between the two voice data using the statistical value calculated for each voice data. Claim 2 delete Claim 3 delete Claim 4 A method for normalizing speaker similarity using metadata, wherein, in claim 1, the step of calculating statistical values ​​from the cohort comprises: a step of calculating the similarity of speaker embeddings belonging to the cohort formed in the step of forming a cohort for each of the two voice data with respect to a speaker embedding extracted from one of the two voice data and the other voice data, and a step of calculating the mean and standard deviation of the similarity of speaker embeddings belonging to the cohort using statistical values ​​for one of the voice data. Claim 5 In claim 4, the step of normalizing the similarity between the two voice data is to calculate the normalized similarity by subtracting the mean for each voice data calculated in the step of calculating statistics from the cohort from the similarity between the two voice data and dividing by the standard deviation for each voice data. This is a speaker similarity normalization method utilizing metadata.

Images