Speaker classification method and device and storage medium
By using speech recognition model and clustering technology in speaker log technology, the speaker vector in the voice fragment and input it into the hybrid neural network, the problem of low classification accuracy caused by phoneme information combination is solved, and the further development is achieved Accurate speaker classification.
Patent Information
- Application Number
- CN202411274706.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-11
- Publication Date
- 2024-12-27
AI Technical Summary
In the existing speaker log technology, when generating voice fragments based on phoneme information, phoneme information of different speakers may be combined, resulting in low accuracy of speaker classification results.
By entering the target voice fragment into the preset speech recognition model, the phoneme vector and text vector are determined, and the voiceprint vectors are clustered, the speaker vector is determined, and then these vectors are input into the hybrid neural network model to generate speaker classification results, avoiding Phoneme information combination.
It improves the accuracy of the classification results of speakers, avoids the combination of phoneme information of different speakers, and enhances the accuracy of the classification results.
Smart Images
Figure CN119207425A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of communication technology, and in particular to a speaker classification method, device and storage medium. Background Art
[0002] Speaker Diarization (SD) technology is used to detect the time periods of speech activity of different speakers, that is, to answer the question of "who spoke at what time". This technology involves the application of audio signal processing and machine learning technology, aiming to identify the voices of each speaker from complex voice interactions and mark their speaking time periods.
[0003] At present, speaker log technology usually determines the different speakers corresponding to text information based on the combination of text information and acoustic features, and then determines the speaker classification result, and constructs a speaker log based on the speaker classification result; however, when generating text information of speech fragments based on phoneme information, the phoneme information of different speakers may be combined, resulting in the generation of erroneous text information and low accuracy of speaker classification results. Summary of the invention
[0004] The present disclosure provides a speaker classification method, device and storage medium, which solve the technical problem of low accuracy of speaker classification results in related technologies.
[0005] In order to achieve the above objectives, the present invention adopts the following technical solutions:
[0006] In a first aspect, a speaker classification method is provided, the method comprising: inputting a target speech segment into a preset speech recognition model, determining a phoneme vector and a text vector of the target speech segment; the phoneme vector is used to characterize the phoneme features in the target speech segment; the text vector is used to characterize the text information of the target speech segment; clustering multiple voiceprint vectors of the target speech segment, determining the clustering center of multiple clusters after clustering the multiple voiceprint vectors as the speaker vector of the target speech segment; the speaker vector is used to identify the speaker in the target speech segment; inputting the phoneme vector, the text vector, the speaker vector, and the acoustic feature into a preset hybrid neural network model, determining the speaker classification result of the target speech segment; the acoustic feature is the acoustic feature determined by performing acoustic feature extraction on the target speech segment; the speaker classification result includes: the speaker contained in the target speech segment, the phoneme features corresponding to the speaker, and the text information determined based on the phoneme features.
[0007] In combination with the above-mentioned first aspect, in a possible implementation method, the method specifically includes: clustering multiple voiceprint vectors of the target speech segment to determine at least one first clustering cluster, and determining the number of first clustering clusters; the number of first clustering clusters is used to characterize the number of speakers in the target speech segment; based on the number of first clustering clusters, determining whether the target speech segment is a non-aliased speech segment; the non-aliased speech segment contains only one speaker; when it is determined that the target speech segment is a non-aliased speech segment, determining the voiceprint vector located at the cluster center in the first clustering cluster as the speaker vector.
[0008] In combination with the above-mentioned first aspect, in a possible implementation method, the method specifically includes: when it is determined that the number of first clustering clusters is greater than one, determining that the target speech segment is an aliased speech segment containing multiple speakers; when it is determined that the number of first clustering clusters is equal to one, determining that the target speech segment is a non-aliased speech segment containing only one speaker.
[0009] In combination with the above-mentioned first aspect, in a possible implementation method, the method specifically includes: inputting the target speech segment into a preset speech recognition model to determine the phoneme information of the target speech segment, and the text information of the target speech segment; performing phoneme embedding processing on the phoneme information to determine the phoneme vector of the target speech segment, and performing text embedding processing on the text information to determine the text vector of the target speech segment.
[0010] In combination with the above-mentioned first aspect, in a possible implementation method, the preset speech recognition model includes: an acoustic model and a language model; the method specifically includes: determining the phoneme information of the target speech segment based on the acoustic model; performing speech recognition on the phoneme information based on the language model to determine the text information of the target speech segment.
[0011] In combination with the above-mentioned first aspect, in a possible implementation, the method specifically includes: step 1, determining multiple first clustering centers, the first clustering center is a clustering center randomly selected from multiple voiceprint vectors, or the first clustering center is a clustering center updated in the previous iteration; step 2, based on the multiple first clustering centers, clustering the multiple voiceprint vectors to determine multiple second clustering clusters; step 3, determining whether the distance from each voiceprint vector in the multiple second clustering clusters to the first clustering center meets the convergence condition; step 4, if satisfied, determining the multiple second clustering clusters as the first clustering cluster; step 5, if not satisfied, determining the clustering centers of the multiple second clustering clusters, and using the clustering centers of the multiple second clustering clusters as the multiple first clustering centers and iteratively executing steps 1, step 2, step 3, step 4, and step 5 until at least one first clustering cluster is determined.
[0012] In combination with the above-mentioned first aspect, in a possible implementation method, the method also includes: performing voice activity detection on the speech to be processed to determine the voiced speech segment in the speech to be processed; dividing the voiced speech segment into multiple speech segments of the same length based on a preset duration; the target speech segment is any speech segment among the multiple speech segments.
[0013] In a second aspect, a speaker classification device is provided, the device comprising: a processing unit; the processing unit is used to input a target speech segment into a preset speech recognition model, and determine a phoneme vector and a text vector of the target speech segment; the phoneme vector is used to characterize the phoneme features in the target speech segment; the text vector is used to characterize the text information of the target speech segment; the processing unit is used to cluster multiple voiceprint vectors of the target speech segment, and determine that the clustering center of multiple clusters after clustering the multiple voiceprint vectors is the speaker vector of the target speech segment; the speaker vector is used to identify the speaker in the target speech segment; the processing unit is used to input the phoneme vector, the text vector, the speaker vector, and the acoustic feature into a preset hybrid neural network model to determine the speaker classification result of the target speech segment; the acoustic feature is the acoustic feature determined by performing acoustic feature extraction on the target speech segment; the speaker classification result includes: the speaker contained in the target speech segment, the phoneme features corresponding to the speaker, and the text information determined based on the phoneme features.
[0014] In combination with the above-mentioned second aspect, in a possible implementation method, the processing unit is specifically used to: cluster multiple voiceprint vectors of the target speech segment to determine at least one first clustering cluster, and determine the number of first clustering clusters; the number of first clustering clusters is used to characterize the number of speakers in the target speech segment; based on the number of first clustering clusters, determine whether the target speech segment is a non-aliased speech segment; the non-aliased speech segment contains only one speaker; when it is determined that the target speech segment is a non-aliased speech segment, determine that the voiceprint vector located at the center of the cluster in the first clustering cluster is the speaker vector.
[0015] In combination with the above-mentioned second aspect, in a possible implementation method, the processing unit is specifically used to: when it is determined that the number of first clustering clusters is greater than one, determine that the target speech segment is an aliased speech segment containing multiple speakers; when it is determined that the number of first clustering clusters is equal to one, determine that the target speech segment is a non-aliased speech segment containing only one speaker.
[0016] In combination with the above-mentioned second aspect, in a possible implementation method, the processing unit is specifically used to: input the target speech segment into a preset speech recognition model to determine the phoneme information of the target speech segment, and the text information of the target speech segment; perform phoneme embedding processing on the phoneme information to determine the phoneme vector of the target speech segment, and perform text embedding processing on the text information to determine the text vector of the target speech segment.
[0017] In combination with the above-mentioned second aspect, in a possible implementation method, the preset speech recognition model includes: an acoustic model, a language model, and a processing unit, which is specifically used to: determine the phoneme information of the target speech segment based on the acoustic model; perform speech recognition on the phoneme information based on the language model to determine the text information of the target speech segment.
[0018] In combination with the above-mentioned second aspect, in a possible implementation method, the processing unit is specifically used to: Step 1, determine multiple first clustering centers, the first clustering center is a clustering center randomly selected from multiple voiceprint vectors, or the first clustering center is a clustering center updated in the previous iteration; Step 2, based on the multiple first clustering centers, cluster the multiple voiceprint vectors to determine multiple second clustering clusters; Step 3, determine whether the distance from each voiceprint vector in the multiple second clustering clusters to the first clustering center meets the convergence condition; Step 4, if satisfied, determine the multiple second clustering clusters as the first clustering cluster; Step 5, if not satisfied, determine the clustering centers of the multiple second clustering clusters, and use the clustering centers of the multiple second clustering clusters as the multiple first clustering centers and iteratively execute steps 1, step 2, step 3, step 4, and step 5 until at least one first clustering cluster is determined.
[0019] In combination with the above-mentioned second aspect, in a possible implementation method, the processing unit is specifically used to: perform voice activity detection on the speech to be processed to determine the voiced speech segment in the speech to be processed; divide the voiced speech segment into multiple speech segments of the same length based on a preset duration; the target speech segment is any speech segment among the multiple speech segments.
[0020] In a third aspect, a speaker classification device is provided, comprising: a processor and a memory; wherein the memory is used to store computer execution instructions, and when the speaker classification device is running, the processor executes the computer execution instructions stored in the memory, so that the speaker classification device performs the speaker classification method recorded in the first aspect and any possible implementation method thereof.
[0021] In a fourth aspect, a computer-readable storage medium is provided, in which instructions are stored. When the instructions in the computer-readable storage medium are executed by a processor of a speaker classification device, the speaker classification device performs the speaker classification method as described in the first aspect and any possible implementation thereof.
[0022] In a fifth aspect, a chip is provided, the chip including a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run a computer program or instruction to implement the speaker classification method recorded in the first aspect and any possible implementation method thereof.
[0023] In the present disclosure, the name of the speaker classification device does not limit the device or functional module itself. In actual implementation, these devices or functional modules may appear with other names. As long as the functions of each device or functional module are similar to those of the present disclosure, they belong to the scope of the present disclosure and its equivalent technologies.
[0024] The technical solution provided by the present disclosure brings at least the following beneficial effects: the speaker classification device in the present disclosure first inputs the target speech segment into a preset speech recognition model, determines the phoneme vector used to characterize the phoneme features in the target speech segment, and the text vector used to characterize the text information of the target speech segment; then clusters the multiple voiceprint vectors of the target speech segment to determine the speaker vector used to identify the speaker in the target speech segment; then inputs the phoneme vector, the text vector, the speaker vector, and the acoustic features into a preset hybrid neural network model to determine the speaker of the target speech segment and the phoneme features corresponding to the speaker, determines the text information based on the phoneme features corresponding to the speaker, avoids combining the phoneme information of different speakers, and thus solves the technical problem of low accuracy of the speaker classification result. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.
[0026] Figure 1 A schematic diagram of the structure of a speaker classification system provided by an embodiment of the present disclosure;
[0027] Figure 2 A schematic diagram of the hardware structure of a speaker classification device provided in an embodiment of the present disclosure;
[0028] Figure 3 A flowchart of a speaker classification method provided by an embodiment of the present disclosure;
[0029] Figure 4 A flowchart of another speaker classification method provided in an embodiment of the present disclosure;
[0030] Figure 5 A flowchart of another speaker classification method provided in an embodiment of the present disclosure;
[0031] Figure 6 A flowchart of another speaker classification method provided in an embodiment of the present disclosure;
[0032] Figure 7 A flowchart of another speaker classification method provided in an embodiment of the present disclosure;
[0033] Figure 8A flowchart of another speaker classification method provided in an embodiment of the present disclosure;
[0034] Fig. 9 A flowchart of another speaker classification method provided in an embodiment of the present disclosure;
[0035] Fig.10 A schematic diagram of the structure of a speaker classification device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0036] A speaker classification method, device and storage medium provided by an embodiment of the present disclosure are described in detail below with reference to the accompanying drawings.
[0037] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.
[0038] The terms “first”, “second” and the like in the specification and drawings of the present disclosure are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects.
[0039] In addition, the terms "including" and "having" and any variations thereof mentioned in the description of the present disclosure are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes other steps or units that are not listed, or optionally includes other steps or units that are inherent to these processes, methods, products or devices. It should be noted that in the embodiments of the present disclosure, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present disclosure should not be interpreted as being more preferred or more advantageous than other embodiments or design solutions. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0040] In the description of the present disclosure, unless otherwise specified, “plurality” means two or more.
[0041] The terms involved in the embodiments of the present disclosure are explained below to facilitate the reader's understanding.
[0042] 1. Speaker diarization (SD) technology
[0043] Speaker log technology, also known as speaker separation and clustering or speaker segmentation technology, is a technology widely used in the field of speech recognition. The main purpose of this technology is to distinguish the segments of different speakers from a continuous multi-speaker speech and solve the problem of "who speaks when".
[0044] 2. Acoustic Model
[0045] The acoustic model is a key component in the speech recognition system. It can associate the acoustic characteristics in the speech signal with speech units (such as phonemes, syllables, etc.) to achieve the recognition of the speech content. Specifically, the acoustic model generates an acoustic model score by receiving the speech features extracted by the feature extraction module, providing key information for the subsequent decoding and recognition process. The acoustic model is mainly modeled using the Hidden Markov Model (HMM). HMM is a model based on probability statistics that can effectively model the timing and dynamics of speech signals. In HMM, the speech signal is regarded as a sequence consisting of a series of state transitions and observations. Each state corresponds to a specific speech unit, and the observation corresponds to the speech feature generated from the state. In addition, the Gaussian mixture model (GMM) is also often used in HMM to describe the probability distribution of the observations in each state. Through training data, the parameters of HMM can be estimated, and then an accurate acoustic model can be constructed.
[0046] 3. Language Model
[0047] Language Model (LM) is a core concept in Natural Language Processing (NLP), which is used to model the probability model of natural language. The main task of the language model is to evaluate the probability of a given word sequence appearing in the real world. This model plays a key role in many NLP applications such as machine translation, speech recognition, and text generation.
[0048] At present, speaker log technology usually determines the different speakers corresponding to text information based on the combination of text information and acoustic features, and then determines the speaker classification result, and constructs a speaker log based on the speaker classification result; however, when generating text information of speech fragments based on phoneme information, the phoneme information of different speakers may be combined, resulting in the generation of erroneous text information and low accuracy of speaker classification results.
[0049] In order to solve the above technical problems, the present disclosure provides a speaker classification method, device and storage medium, which are used to solve the technical problem that when generating text information of a voice segment based on phoneme information, the phoneme information of different speakers may be combined, resulting in the generation of erroneous text information, thereby affecting the accuracy of the speaker classification result. The method includes: the speaker classification device first inputs the target voice segment into a preset voice recognition model, determines the phoneme vector used to characterize the phoneme features in the target voice segment, and the text information text vector used to characterize the target voice segment; then clusters multiple voiceprint vectors of the target voice segment to determine the speaker vector used to identify the speaker in the target voice segment; then inputs the phoneme vector, text vector, speaker vector, and acoustic features into a preset hybrid neural network model, determines the speaker of the target voice segment, and the phoneme features corresponding to the speaker, determines the text information based on the phoneme features corresponding to the speaker, avoids combining the phoneme information of different speakers, and thus solves the technical problem of low accuracy of the speaker classification result.
[0050] In a possible implementation, the above speaker classification method can be applied to the speaker classification system 100. Figure 1 A speaker classification system 100 provided in an embodiment of the present application is described in detail. Figure 1 As shown, Figure 1 A speaker classification system 100 provided in an embodiment of the present disclosure includes: a terminal 101 and a speaker classification device 102 .
[0051] The terminal 101 is used to send the speech to be processed to the speaker classification device 102, so that the speaker classification device 102 divides the speech to be processed into multiple speech segments of the same length and then performs speaker classification on the speech to be processed.
[0052] The speaker classification device 102 is used to: input the target speech segment into a preset speech recognition model to determine the phoneme vector and text vector of the target speech segment; the phoneme vector is used to represent the phoneme features in the target speech segment; the text vector is used to represent the text information of the target speech segment; clustering multiple voiceprint vectors of the target speech segment, and determining the clustering center of multiple clusters after clustering the multiple voiceprint vectors as the speaker vector of the target speech segment; the speaker vector is used to identify the speaker in the target speech segment; input the phoneme vector, text vector, speaker vector, and acoustic features into a preset hybrid neural network model to determine the speaker classification result of the target speech segment; the acoustic features are acoustic features determined by performing acoustic feature extraction on the target speech segment; the speaker classification result includes: the speaker contained in the target speech segment, the phoneme features corresponding to the speaker, and text information determined based on the phoneme features.
[0053] In a possible implementation, the hardware structure of the speaker classification device includes: Figure 2 The speaker classification device 200 shown in FIG. 2 includes the following components. Figure 2 The speaker classification device 200 shown in the figure is used as an example to introduce the hardware structure of the speaker classification device. Figure 2 As shown, the speaker classification device 200 includes at least one processor 201, a communication line 202, and at least one communication interface 204, and may also include a memory 203. The processor 201, the memory 203 and the communication interface 204 may be connected via the communication line 202.
[0054] The processor 201 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present disclosure, such as one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA).
[0055] The communication link 202 may include a pathway for transmitting information between the above-mentioned components.
[0056] The communication interface 204 is used to communicate with other devices or communication networks, and can use any transceiver-like device, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.
[0057] The memory 203 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to include or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.
[0058] In one possible design, the memory 203 can exist independently of the processor 201, that is, the memory 203 can be a memory outside the processor 201. In this case, the memory 203 can be connected to the processor 201 through the communication line 202, and is used to store execution instructions or application code, and the processor 201 controls the execution to implement the speaker classification method provided in the following embodiment of the present disclosure. In another possible design, the memory 203 can also be integrated with the processor 201, that is, the memory 203 can be the internal memory of the processor 201, for example, the memory 203 is a high-speed cache, which can be used to temporarily store some data and instruction information.
[0059] As an implementation method, the processor 201 may include one or more CPUs, for example Figure 2 As another possible implementation, the speaker classification device 200 may include multiple processors, such as Figure 2 As another implementation, the speaker classification apparatus 200 may further include an output device 205 and an input device 206.
[0060] In the following, a speaker classification method provided by an embodiment of the present disclosure is described in detail.
[0061] like Figure 3 As shown, Figure 3 A speaker classification method provided in an embodiment of the present disclosure can be applied to Figure 2 In the speaker classification device shown, the method includes the following S301-S303, which are described in detail below.
[0062] S301: A speaker classification device inputs a target speech segment into a preset speech recognition model to determine a phoneme vector and a text vector of the target speech segment.
[0063] Among them, the phoneme vector is used to represent the phoneme features in the target speech segment; the text vector is used to represent the text information of the target speech segment.
[0064] It is understandable that the preset speech recognition model may include multiple models for performing speech recognition on the target speech segment.
[0065] In a possible implementation, the speaker classification device inputs the target speech segment into a preset speech recognition model, determines a phoneme vector for representing phoneme features in the target speech segment and a text vector for representing text information of the target speech segment.
[0066] S302: The speaker classification device clusters the multiple voiceprint vectors of the target speech segment, and determines the clustering center of the multiple clusters after the multiple voiceprint vectors are clustered as the speaker vector of the target speech segment.
[0067] Among them, the speaker vector is used to identify the speaker in the target speech segment.
[0068] In a possible implementation, the speaker classification device determines all voiceprint vectors contained in the target speech segment, and clusters all voiceprint vectors to determine at least one cluster, the cluster center of which is the speaker vector of the target speech segment.
[0069] It can be understood that after clustering multiple voiceprint vectors of the target speech segment, the voiceprint vectors in each cluster may be the voiceprint vectors of the same speaker, among which the voiceprint vector located at the cluster center of the cluster has the highest match with the speaker and is the speaker vector of the speaker corresponding to the cluster.
[0070] S303: The speaker classification device inputs the phoneme vector, the text vector, the speaker vector, and the acoustic feature into a preset hybrid neural network model to determine the speaker classification result of the target speech segment.
[0071] In one possible implementation, the speaker classification device extracts the acoustic features of the target speech segment based on a convolutional neural network (CNN) model and a recurrent neural network (RNN) model to determine the acoustic features of the target speech segment, and then inputs the phoneme vector, text vector, speaker vector, and acoustic features into a preset hybrid neural network model to determine the speaker classification result of the target speech segment.
[0072] The speaker classification result includes: the speaker contained in the target speech segment, the phoneme features corresponding to the speaker, and text information determined based on the phoneme features.
[0073] It can be understood that the speaker phoneme vector, text vector, speaker vector, and acoustic features are input into the preset hybrid neural network model, and the output speaker classification result is obtained by classifying the speakers at the phoneme level, that is, determining the factor features corresponding to each speaker, so that the text information can be determined based on the phoneme features corresponding to the speaker, avoiding the combination of phoneme information of different speakers.
[0074] The technical solution provided by the above embodiment can at least bring the following beneficial effects: the speaker classification device first inputs the target voice segment into a preset voice recognition model, determines the phoneme vector used to characterize the phoneme features in the target voice segment, and the text vector used to characterize the text information of the target voice segment; then clusters the multiple voiceprint vectors of the target voice segment to determine the speaker vector used to identify the speaker in the target voice segment; then inputs the phoneme vector, text vector, speaker vector, and acoustic features into a preset hybrid neural network model to determine the speaker of the target voice segment and the phoneme features corresponding to the speaker, determines the text information based on the phoneme features corresponding to the speaker, avoids combining the phoneme information of different speakers, and thus solves the technical problem of low accuracy of the speaker classification result.
[0075] In one possible implementation, combining Figure 3 ,like Figure 4 As shown, the above S302, the speaker classification device clusters multiple voiceprint vectors of the target speech segment, and determines that the clustering center of multiple clusters after the multiple voiceprint vectors are clustered is the speaker vector of the target speech segment. This process can be specifically implemented through the following S401-S403, which is described in detail below.
[0076] S401: A speaker classification device clusters multiple voiceprint vectors of a target speech segment to determine at least one first cluster, and determines the number of first clusters.
[0077] In one possible implementation, the speaker classification device clusters multiple voiceprint vectors of the target speech segment to determine at least one first cluster, each first cluster containing multiple voiceprint vectors, and determines the number of each first cluster, and the number of first clusters is used to characterize the number of speakers in the target speech segment.
[0078] In one possible implementation, the speaker classification device first globally clusters multiple voiceprint vectors of the target speech segment using an unsupervised algorithm based on a hybrid method of a clustering algorithm and a neural network, such as spectral clustering, hierarchical clustering, etc., and then performs local speaker classification using a neural network.
[0079] S402: The speaker classification apparatus determines whether the target speech segment is a non-aliased speech segment based on the number of the first clusters.
[0080] Among them, the non-aliased speech segment contains only one speaker.
[0081] In a possible implementation, the speaker classification device determines the number of speakers in the target speech segment based on the number of first clusters, and determines whether the target speech segment is a non-aliased speech segment based on the number of speakers in the target speech segment.
[0082] S403: When determining that the target speech segment is a non-aliased speech segment, the speaker classification device determines the voiceprint vector located at the center of the cluster in the first cluster as the speaker vector.
[0083] In one possible implementation, the speaker classification device determines the number of speakers in the target speech segment based on the number of first clusters, determines that the target speech segment is a non-aliased speech segment based on the number of speakers in the target speech segment, and when the target speech segment is a non-aliased speech segment, determines that the voiceprint vector located at the center of the cluster in the first cluster is the speaker vector.
[0084] It can be understood that the non-aliased speech segment contains only one speaker, and the speaker vector determined by the voiceprint vector in the non-aliased speech segment has a higher accuracy. Therefore, when the speaker classification device determines that the target speech segment is a non-aliased speech segment, it determines that the voiceprint vector located at the center of the cluster in the first cluster is the speaker vector.
[0085] The technical solution provided by the above embodiment can at least bring about the following beneficial effects: the speaker classification device clusters multiple voiceprint vectors of the target speech segment to determine at least one first clustering cluster, and determines that the target speech segment is a non-aliased speech segment, the non-aliased speech segment contains only one speaker, and the speaker vector determined by the voiceprint vector in the non-aliased speech segment has a high accuracy. When the speaker classification device determines that the target speech segment is a non-aliased speech segment, the voiceprint vector located at the center of the cluster in the first clustering cluster is determined as the speaker vector.
[0086] In one possible implementation, combining Figure 4 ,like Figure 5As shown, the above S402, the process in which the speaker classification device determines whether the target speech segment is a non-aliased speech segment based on the number of the first clusters, can be specifically determined through the following S501-S502, which is described in detail below.
[0087] S501: When determining that the number of the first clusters is greater than one, the speaker classification apparatus determines that the target speech segment is an aliased speech segment containing multiple speakers.
[0088] In one possible implementation, the speaker classification device clusters multiple voiceprint vectors of the target speech segment, determines the number of first clustering clusters after the multiple voiceprint vectors are clustered, and the number of first clustering clusters is the number of speakers. When the speaker classification device determines that the number of first clustering clusters is greater than one, it determines that the target speech segment is an aliased speech segment containing multiple speakers.
[0089] S502: When determining that the number of the first clusters is equal to one, the speaker classification apparatus determines that the target speech segment is a non-aliased speech segment containing only one speaker.
[0090] In one possible implementation, the speaker classification device clusters multiple voiceprint vectors of the target speech segment, determines the number of first clustering clusters after the multiple voiceprint vectors are clustered, and the number of first clustering clusters is the number of speakers. When the speaker classification device determines that the number of first clustering clusters is equal to one, it determines that the target speech segment is a non-aliasing speech segment containing only one speaker.
[0091] The technical solution provided by the above embodiment can at least bring the following beneficial effects: the speaker classification device determines the number of speakers of the target speech segment based on the number of the first clusters, and further determines whether the target speech segment is a non-aliasing speech segment.
[0092] In one possible implementation, combining Figure 5 ,like Figure 6 As shown, the above S301, the speaker classification device inputs the target speech segment into a preset speech recognition model to determine the phoneme vector and text vector of the target speech segment, which can be specifically implemented through the following S601-S602, which is described in detail below.
[0093] S601: The speaker classification device inputs a target speech segment into a preset speech recognition model to determine the phoneme information of the target speech segment and the text information of the target speech segment.
[0094] In one possible implementation, the speaker classification device inputs the target speech segment into a preset speech recognition model, determines the vowel phonemes and consonant phonemes in the target speech segment, and obtains text information of part of the target speech segment based on the vowel phonemes and consonant phonemes in the target speech segment.
[0095] S602: The speaker classification device performs phoneme embedding processing on the phoneme information to determine the phoneme vector of the target speech segment, and performs text embedding processing on the text information to determine the text vector of the target speech segment.
[0096] In a possible implementation, the speaker classification device converts the phoneme information of the target speech segment into a phoneme vector based on a preset phoneme embedding conversion model.
[0097] In one possible implementation, the speaker classification device inputs the one-hot encoding of the phoneme label into the embedding layer, and generates a phoneme vector through the embedding layer. The phoneme vector can be used as the input of other neural network layers to further learn and predict acoustic features or speech signals.
[0098] In a possible implementation, the speaker classification device can calculate the feature vector of the text information by using a preset text embedding model, and each word or phrase in the text embedding is assigned a vector. These vectors can form a vector representation of the text, and the text vector can represent key feature information such as the semantic information of the text.
[0099] The technical solution provided by the above embodiment can at least bring about the following beneficial effects: the speaker classification device determines the phoneme information of the target speech segment and the text information of the target speech segment based on a preset speech recognition model, and performs phoneme embedding processing on the phoneme information to determine the phoneme vector of the target speech segment, and performs text embedding processing on the text information to determine the text vector of the target speech segment, so that the speaker classification device determines the speaker classification result of the target speech segment based on the phoneme vector and text vector of the target speech segment.
[0100] In one possible implementation, combining Figure 6 ,like Figure 7 As shown, the preset speech recognition model includes: an acoustic model and a language model; the above S601, the speaker classification device inputs the target speech segment into the preset speech recognition model, determines the phoneme information of the target speech segment, and the process of the text information of the target speech segment, which can be specifically implemented through the following S701-S702, which is described in detail below.
[0101] S701: A speaker classification device determines phoneme information of a target speech segment based on an acoustic model.
[0102] In a possible implementation, a speaker classification device determines the phoneme information of a target speech segment based on a hidden Markov model and a deep neural network model.
[0103] S702: The speaker classification device performs speech recognition on the phoneme information based on the language model to determine the text information of the target speech segment.
[0104] In a possible implementation, the speaker classification device performs speech recognition on phoneme information based on a neural network language model (NNLM) to determine text information of the target speech segment.
[0105] The technical solution provided by the above embodiment can at least bring about the following beneficial effects: the speaker classification device determines the phoneme information of the target speech segment based on the acoustic model; then performs speech recognition on the phoneme information based on the language model, and determines the text information of the target speech segment, so that the subsequent speaker classification device determines the phoneme vector of the target speech segment and the text vector of the target speech segment based on the phoneme information of the target speech segment and the text information of the target speech segment.
[0106] In one possible implementation, Figure 8 As shown, the process of the speaker classification device clustering multiple voiceprint vectors of the target speech segment to determine at least one first cluster can be specifically implemented through the following S801-S805, which is described in detail below.
[0107] S801: A speaker classification device determines a plurality of first cluster centers.
[0108] The first cluster center is a cluster center randomly selected from multiple voiceprint vectors, or the first cluster center is a cluster center updated in the previous iteration.
[0109] In a possible implementation, when the second cluster does not satisfy the condition, the cluster center of the second cluster is iterated to be the first cluster center.
[0110] S802: The speaker classification device clusters the multiple voiceprint vectors based on the multiple first cluster centers to determine multiple second cluster clusters.
[0111] In a possible implementation, the speaker classification device determines that each voiceprint vector and the first cluster center that is closest to each other is a cluster, and determines a plurality of second clusters respectively.
[0112] S803: The speaker classification device determines whether the distance from each voiceprint vector in the plurality of second clusters to the center of the first cluster meets a convergence condition.
[0113] In a possible implementation, when the distance between the current cluster center of each of the plurality of second cluster clusters and the previous cluster center no longer changes, it can be determined that the convergence condition is met.
[0114] S804: If the conditions are met, the speaker classification apparatus determines the plurality of second clusters as the first cluster.
[0115] It can be understood that when the distance from each voiceprint vector in the second cluster to the first cluster center meets the convergence condition, that is, when the distance between the cluster centers of multiple second clusters and the corresponding multiple first cluster centers no longer changes, it can be determined that the clustering has been completed, and at this time, the multiple second clusters are the first clusters.
[0116] S805. If not, the speaker classification apparatus determines cluster centers of multiple second clusters, and uses the cluster centers of multiple second clusters as multiple first cluster centers and iteratively executes steps S801, S802, S803, S804, and S805 until at least one first cluster is determined.
[0117] It can be understood that when the distance from each voiceprint vector in the second cluster cluster to the first cluster center does not meet the convergence condition, that is, when the distance between the cluster centers of multiple second cluster clusters and the corresponding multiple first cluster centers still changes, it can be determined that clustering has not been completed at this time. At this time, the cluster centers of multiple second cluster clusters need to be used as multiple first cluster centers to iteratively execute S801, S802, S803, S804, and S805 to continue clustering until clustering is completed.
[0118] The technical solution provided by the above embodiment can at least bring the following beneficial effects: the speaker classification device clusters multiple voiceprint vectors of the target speech segment based on the distance between each voiceprint vector and the nearest first cluster center, and by judging whether the distance between each voiceprint vector in multiple second clusters and the first cluster center meets the convergence condition, the first cluster center is iteratively updated for clustering until each first cluster is completely converged. Thus, the first cluster to which each voiceprint vector belongs is accurately determined.
[0119] In one possible implementation, combining Figure 3 ,like Fig. 9 As shown, in the above S301, the speaker classification device inputs the target speech segment into the preset speech recognition model, and before determining the phoneme vector and text vector of the target speech segment, the speaker classification device needs to determine the target speech segment. This process can be specifically implemented through the following S901-S902, which is described in detail below.
[0120] S901: A speaker classification device performs voice activity detection on a speech to be processed, and determines an audible speech segment in the speech to be processed.
[0121] In a possible implementation, the speaker classification device determines the voiced speech segment in the speech to be processed based on voice activity detection (VAD) technology.
[0122] S902: The speaker classification device divides the voiced speech segment into multiple speech segments of the same duration based on a preset duration.
[0123] The target voice segment is any voice segment among the multiple voice segments.
[0124] In a possible implementation, the speaker classification device divides the voiced speech segment into multiple speech segments of the same duration based on a preset duration, and classifies the speech segments based on the speaker in each speech segment using power set coding.
[0125] Exemplarily, the multiple speech segments include speaker A, speaker B, and speaker C; the types of speech segments include: a speech segment containing only speaker A, a speech segment containing only speaker B, a speech segment containing only speaker C, a speech segment containing speaker A and speaker B, a speech segment containing speaker A and speaker C, a speech segment containing speaker B and speaker C, and a speech segment containing speaker A, speaker B, and speaker C. Among them: a speech segment containing only speaker A, a speech segment containing only speaker B, and a speech segment containing only speaker C are non-aliased speech segments, and the speaker vector is determined based on the non-aliased speech segment.
[0126] It can be seen that the above mainly introduces the technical solution provided by the embodiment of the present disclosure from the perspective of the method. In order to achieve the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the modules and algorithm steps of each example described in the embodiment disclosed herein, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present disclosure.
[0127] The embodiment of the present disclosure can divide the functional modules of the speaker classification device according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. Optionally, the division of modules in the embodiment of the present disclosure is schematic and is only a logical functional division. There may be other division methods in actual implementation.
[0128] In one possible implementation, Fig.10 , Fig.10 A schematic diagram of the structure of a speaker classification device 1000 provided by the present disclosure.
[0129] The speaker classification device 1000 includes: a processing unit 1001; the processing unit 1001 is used to input a target speech segment into a preset speech recognition model to determine a phoneme vector and a text vector of the target speech segment; the phoneme vector is used to represent the phoneme features in the target speech segment; the text vector is used to represent the text information of the target speech segment; the processing unit 1001 is used to cluster multiple voiceprint vectors of the target speech segment, and determine that the clustering center of multiple clusters after the multiple voiceprint vectors are clustered is the speaker vector of the target speech segment; the speaker vector is used to identify the speaker in the target speech segment; the processing unit 1001 is used to input the phoneme vector, the text vector, the speaker vector, and the acoustic feature into a preset hybrid neural network model to determine the speaker classification result of the target speech segment; the acoustic feature is the acoustic feature determined by performing acoustic feature extraction on the target speech segment; the speaker classification result includes: the speaker contained in the target speech segment, the phoneme features corresponding to the speaker, and the text information determined based on the phoneme features.
[0130] In one possible implementation, the processing unit 1001 is specifically used to: cluster multiple voiceprint vectors of the target speech segment to determine at least one first clustering cluster, and determine the number of first clustering clusters; the number of first clustering clusters is used to characterize the number of speakers in the target speech segment; based on the number of first clustering clusters, determine whether the target speech segment is a non-aliased speech segment; the non-aliased speech segment contains only one speaker; when it is determined that the target speech segment is a non-aliased speech segment, determine that the voiceprint vector located at the cluster center in the first clustering cluster is the speaker vector.
[0131] In a possible implementation, the processing unit 1001 is specifically used to: when it is determined that the number of the first clustering clusters is greater than one, determine that the target speech segment is an aliased speech segment containing multiple speakers; when it is determined that the number of the first clustering clusters is equal to one, determine that the target speech segment is a non-aliased speech segment containing only one speaker.
[0132] In one possible implementation, the processing unit 1001 is specifically used to: input the target speech segment into a preset speech recognition model to determine the phoneme information of the target speech segment and the text information of the target speech segment; perform phoneme embedding processing on the phoneme information to determine the phoneme vector of the target speech segment, and perform text embedding processing on the text information to determine the text vector of the target speech segment.
[0133] In one possible implementation, the preset speech recognition model includes: an acoustic model, a language model, and a processing unit 1001, which is specifically used to: determine the phoneme information of the target speech segment based on the acoustic model; perform speech recognition on the phoneme information based on the language model to determine the text information of the target speech segment.
[0134] In one possible implementation, the processing unit 1001 is specifically used to: step 1, determine multiple first clustering centers, where the first clustering center is a clustering center randomly selected from multiple voiceprint vectors, or the first clustering center is a clustering center updated in the previous iteration; step 2, clustering multiple voiceprint vectors based on the multiple first clustering centers to determine multiple second clusters; step 3, determining whether the distance from each voiceprint vector in the multiple second clusters to the first clustering center meets the convergence condition; step 4, if satisfied, determining the multiple second clusters as the first clustering cluster; step 5, if not satisfied, determining the clustering centers of the multiple second clusters, and using the clustering centers of the multiple second clusters as the multiple first clustering centers and iteratively executing steps 1, 2, 3, 4, and 5 until at least one first cluster is determined.
[0135] In a possible implementation, the processing unit 1001 is specifically used to: perform voice activity detection on the speech to be processed to determine the voiced speech segment in the speech to be processed; divide the voiced speech segment into multiple speech segments of the same duration based on a preset duration; the target speech segment is any speech segment among the multiple speech segments.
[0136] The embodiment of the present disclosure also provides a speaker classification device, which includes a processor and a memory; wherein the memory is used to store computer execution instructions. When the speaker classification device is running, the processor executes the computer execution instructions stored in the memory, so that the speaker classification device performs the speaker classification method recorded in the embodiment of the present disclosure.
[0137] An embodiment of the present disclosure provides a computer program product including instructions. When the instructions are executed on a computer, the computer is enabled to execute the speaker classification method in the above method embodiment.
[0138] An embodiment of the present disclosure provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run a computer program or instruction to implement a speaker classification method as in the above method embodiment.
[0139] Among them, the computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, and a hard disk. Random Access Memory (RAM), Read-Only Memory (ROM), Erasable Programmable Read Only Memory (EPROM), registers, hard disks, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any other form of computer-readable storage medium in a suitable combination of the above, or numerical values in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an Application Specific Integrated Circuit (ASIC). In the embodiments of the present disclosure, computer-readable storage media may be any tangible media that contains or stores a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.
[0140] Since the apparatus, device, computer-readable storage medium, and computer program product in the embodiments of the present disclosure can be applied to the above-mentioned method, the technical effects that can be obtained can also refer to the above-mentioned method embodiments, and the embodiments of the present disclosure will not be repeated here.
[0141] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. A speaker classification method, characterized in that: include: Inputting the target speech segment into a preset speech recognition model to determine the phoneme vector and text vector of the target speech segment; The phoneme vector is used to represent the phoneme features in the target speech segment; The text vector is used to represent the text information of the target speech segment; Clustering multiple voiceprint vectors of the target speech segment, and determining the cluster centers of multiple clusters after the multiple voiceprint vectors are clustered as the speaker vector of the target speech segment; the speaker vector is used to identify the speaker in the target speech segment; Inputting the phoneme vector, the text vector, the speaker vector, and the acoustic feature into a preset hybrid neural network model to determine a speaker classification result of the target speech segment; the acoustic feature is an acoustic feature determined by performing acoustic feature extraction on the target speech segment; The speaker classification result includes: the speaker contained in the target speech segment, the phoneme features corresponding to the speaker, and text information determined based on the phoneme features.
2. The method according to claim 1, characterized in that The step of clustering the multiple voiceprint vectors of the target speech segment and determining the cluster centers of the multiple clusters after the multiple voiceprint vectors are clustered as the speaker vector of the target speech segment includes: Clustering the multiple voiceprint vectors of the target speech segment to determine at least one first cluster, and determining the number of the first clusters; the number of the first clusters is used to characterize the number of speakers in the target speech segment; Determining whether the target speech segment is a non-aliased speech segment based on the number of the first clustering clusters; the non-aliased speech segment contains only one speaker; When it is determined that the target speech segment is a non-aliased speech segment, the voiceprint vector located at the center of the cluster in the first cluster is determined as the speaker vector.
3. The method according to claim 2, characterized in that The determining whether the target speech segment is a non-aliased speech segment based on the number of the first clustering clusters includes: In a case where it is determined that the number of the first clusters is greater than one, determining that the target speech segment is an aliased speech segment containing multiple speakers; When it is determined that the number of the first clusters is equal to one, the target speech segment is determined to be a non-aliased speech segment containing only one speaker.
4. The method according to claim 3, characterized in that: The step of inputting the target speech segment into a preset speech recognition model and determining the phoneme vector and text vector of the target speech segment includes: Inputting the target speech segment into the preset speech recognition model to determine the phoneme information of the target speech segment and the text information of the target speech segment; The phoneme information is subjected to phoneme embedding processing to determine the phoneme vector of the target speech segment, and the text information is subjected to text embedding processing to determine the text vector of the target speech segment.
5. The method according to claim 4, characterized in that The preset speech recognition model includes: an acoustic model and a language model; the step of inputting the target speech segment into the preset speech recognition model and determining the phoneme information and text information of the target speech segment includes: Determining phoneme information of the target speech segment based on the acoustic model; Speech recognition is performed on the phoneme information based on the language model to determine text information of the target speech segment.
6. The method according to claim 5, characterized in that The step of clustering the plurality of voiceprint vectors of the target speech segment to determine at least one first clustering cluster comprises: Step 1: determining a plurality of first cluster centers, wherein the first cluster centers are cluster centers randomly selected from the plurality of voiceprint vectors, or the first cluster centers are cluster centers updated in a previous iteration; Step 2: clustering the plurality of voiceprint vectors based on the plurality of first cluster centers to determine a plurality of second cluster clusters; Step 3: Determine whether the distance from each voiceprint vector in the plurality of second clusters to the first cluster center meets the convergence condition; Step 4: If satisfied, determine the plurality of second clusters as the first cluster; Step 5: If not satisfied, determine the cluster centers of the multiple second clusters, and use the cluster centers of the multiple second clusters as the multiple first cluster centers and iteratively execute step 1, step 2, step 3, step 4, and step 5 until at least one first cluster is determined.
7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Performing voice activity detection on the speech to be processed to determine the voiced speech segments in the speech to be processed; The voiced speech segment is divided into a plurality of speech segments of the same duration based on a preset duration; and the target speech segment is any speech segment among the plurality of speech segments.
8. A speaker classification device, characterized in that: include: Processing unit; The processing unit is used to input the target speech segment into a preset speech recognition model to determine the phoneme vector and text vector of the target speech segment; the phoneme vector is used to characterize the phoneme features in the target speech segment; The text vector is used to represent the text information of the target speech segment; The processing unit is used to cluster multiple voiceprint vectors of the target speech segment, and determine the cluster centers of multiple clusters after the multiple voiceprint vectors are clustered as the speaker vector of the target speech segment; the speaker vector is used to identify the speaker in the target speech segment; The processing unit is used to input the phoneme vector, the text vector, the speaker vector, and the acoustic feature into a preset hybrid neural network model to determine a speaker classification result of the target speech segment; The acoustic feature is an acoustic feature determined by performing acoustic feature extraction on the target speech segment; The speaker classification result includes: the speaker contained in the target speech segment, the phoneme features corresponding to the speaker, and text information determined based on the phoneme features.
9. A speaker classification device, characterized in that: include: A processor and a memory; wherein the memory is used to store computer-executable instructions, and when the speaker classification device is running, the processor executes the computer-executable instructions stored in the memory, so that the speaker classification device performs the speaker classification method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions. When the instructions in the computer-readable storage medium are executed by a processor of the speaker classification device, the speaker classification device executes the speaker classification method according to any one of claims 1 to 7.