Speaker segmentation and clustering method, device, electronic device and storage medium

Abstract

The present invention relates to the technical field of speaker segmentation and clustering, and in particular to a speaker segmentation and clustering method, device, electronic device and storage medium, the method comprising: selecting a target speaker transition point from the initial speaker transition point according to the difference between the sound source direction of the previous initial audio data segment and the sound source direction of the next initial audio data segment at the initial speaker transition point in the audio data; segmenting the audio data based on all target speaker transition points to obtain target audio data segments; constructing a similarity matrix between the voiceprint vectors of all target audio data segments and the voiceprint vectors corresponding to all historical audio data segments; performing spectral clustering on all target audio data segments and all historical audio data segments based on the similarity matrix to obtain speaker clustering results. The present application can perform speaker segmentation and clustering on audio data, thereby improving the accuracy of speaker segmentation and clustering.

Description

Technical Field

[0001] The present invention relates to the technical field of speaker segmentation and clustering, and in particular to a speaker segmentation and clustering method, device, electronic equipment and storage medium. Background Art

[0002] Speaker segmentation and clustering is mainly used to segment the speech of multiple speakers into speech segments of a single speaker and identify who is speaking in each time period.

[0003] Existing speaker segmentation and clustering methods usually rely on a series combination of subtasks such as voice activity detection, embedding code extractor and speaker clustering. Existing speaker segmentation and clustering methods are mainly based on single-channel speech signal processing. For speech data recorded by multi-channel microphone arrays, the segmentation and clustering effect is poor and the accuracy is low. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a speaker segmentation and clustering method, device, electronic device and storage medium, which can perform spectral clustering on audio data through the similarity matrix between the voiceprint vector of the target audio data segment obtained based on the target conversion point division and the voiceprint vector corresponding to all historical audio data segments, thereby improving the accuracy of speaker segmentation and clustering.

[0005] In a first aspect, an embodiment of the present application provides a speaker segmentation and clustering method, the method comprising:

[0006] Obtain the initial speaker transition point corresponding to the audio data in the current time period in this voice conversation, and the voiceprint vector corresponding to each historical audio data segment in this voice conversation;

[0007] Selecting a target speaker change point from the initial speaker change points according to a difference between a sound source direction of a previous initial audio data segment and a sound source direction of a subsequent initial audio data segment at the initial speaker change point in the audio data;

[0008] Segment the audio data based on all target speaker transition points to obtain target audio data segments;

[0009] Constructing a similarity matrix between the voiceprint vectors of all target audio data segments and the voiceprint vectors corresponding to all historical audio data segments;

[0010] Based on the similarity matrix, spectral clustering is performed on all target audio data segments and all historical audio data segments to obtain speaker clustering results.

[0011] In a possible implementation, constructing a similarity matrix between the voiceprint vectors of all target audio data segments and the voiceprint vectors corresponding to all historical audio data segments includes:

[0012] If the duration of the current voice conversation exceeds the preset duration, the similarity weights between each audio data segment and each historical audio data segment are calculated based on the difference between the sound source direction corresponding to each audio data segment and the speaker sound source direction of each historical audio data segment;

[0013] According to the similarity weights between each audio data segment and each historical audio data segment, a similarity matrix of the voiceprint vectors of all audio data segments is constructed; the audio data segments include target audio data segments and historical audio data segments.

[0014] In a possible implementation, the speaker sound source direction of each historical audio data segment is obtained by the following steps:

[0015] Obtaining all historical speaker clustering results corresponding to each historical audio data segment; the historical speaker clustering results include the speakers corresponding to each historical audio data segment;

[0016] Determine the target speaker corresponding to each historical audio data segment according to the clustering results of all historical speakers corresponding to each historical audio data segment;

[0017] The average value of the sound source directions of all historical audio data segments corresponding to each target speaker is determined as the sound source direction of each target speaker;

[0018] The sound source direction of the target speaker corresponding to each historical audio data segment is determined as the speaker sound source direction of each historical audio data segment.

[0019] In a possible implementation, determining a target speaker corresponding to each historical audio data segment according to clustering results of all historical speakers corresponding to each historical audio data segment includes:

[0020] Count the number of times each historical audio data corresponds to each speaker;

[0021] The speaker with the most times is taken as the target speaker corresponding to each historical audio data segment.

[0022] In a possible implementation, calculating the similarity weight between each audio data segment and each historical audio data segment according to the difference between the sound source direction corresponding to each audio data segment and the speaker sound source direction of each historical audio data segment includes:

[0023] Substitute the difference between the sound source direction corresponding to each audio data segment and the speaker sound source direction of each historical audio data segment into the following formula to calculate the similarity weight between each audio data segment and each historical audio data segment:

[0024] ;

[0025] in, is the similarity weight between the i-th audio data segment and the j-th historical audio data segment, is the sound source direction of the i-th audio data segment, is the speaker source direction of the jth historical audio data segment, is the preset weight coefficient.

[0026] In a possible implementation, the voiceprint vector corresponding to the target audio data segment or the voiceprint vector corresponding to the historical audio data segment is obtained by the following steps:

[0027] Inputting the target audio data segment or the historical audio data segment into the voiceprint extraction model to obtain the voiceprint vector corresponding to the target audio data segment or the historical audio data segment;

[0028] Among them, the voiceprint extraction model is trained through audio sample data and the corresponding voiceprint vector.

[0029] In a second aspect, an embodiment of the present application further provides a speaker segmentation and clustering device, the speaker segmentation and clustering device comprising:

[0030] An acquisition module, used to acquire the initial speaker switching point corresponding to the audio data in the current time period in this voice conversation, and the voiceprint vector corresponding to each historical audio data segment in this voice conversation;

[0031] A screening module, for screening a target speaker change point from the initial speaker change points according to a difference between a sound source direction of a previous initial audio data segment and a sound source direction of a subsequent initial audio data segment corresponding to the initial speaker change point in the audio data;

[0032] A segmentation module, configured to segment the audio data based on all target speaker transition points to obtain target audio data segments;

[0033] A construction module, used to construct a similarity matrix between the voiceprint vectors of all target audio data segments and the voiceprint vectors corresponding to all historical audio data segments;

[0034] The clustering module is used to perform spectral clustering on all target audio data segments and all historical audio data segments based on the similarity matrix to obtain speaker clustering results.

[0035] In a possible implementation, a construction module is specifically used to calculate the similarity weight between each audio data segment and each historical audio data segment based on the difference between the sound source direction corresponding to each audio data segment and the speaker sound source direction of each historical audio data segment if the conversation duration of the current voice conversation exceeds the preset conversation duration; construct a similarity matrix of the voiceprint vectors of all audio data segments based on the similarity weight between each audio data segment and each historical audio data segment; the audio data segments include target audio data segments and historical audio data segments.

[0036] In a possible implementation, a construction module is specifically used to obtain all historical speaker clustering results corresponding to each historical audio data segment; the historical speaker clustering results include the speakers corresponding to each historical audio data segment; according to all historical speaker clustering results corresponding to each historical audio data segment, a target speaker corresponding to each historical audio data segment is determined; the average value of the sound source directions of all historical audio data segments corresponding to each target speaker is determined as the sound source direction of each target speaker; and the sound source direction of the target speaker corresponding to each historical audio data segment is determined as the speaker sound source direction of each historical audio data segment.

[0037] In a possible implementation, the module is constructed to specifically count the number of times each historical audio data segment corresponds to each speaker; and the speaker with the largest number of times is used as the target speaker corresponding to each historical audio data segment.

[0038] In a possible implementation, the construction module is specifically used to substitute the difference between the sound source direction corresponding to each audio data segment and the speaker sound source direction of each historical audio data segment into the following formula to calculate the similarity weight between each audio data segment and each historical audio data segment:

[0039] ;

[0040] in, is the similarity weight between the i-th audio data segment and the j-th historical audio data segment, is the sound source direction of the i-th audio data segment, is the speaker source direction of the jth historical audio data segment, is the preset weight coefficient.

[0041] In one possible implementation, the acquisition module is specifically used to input the target audio data segment or the historical audio data segment into the voiceprint extraction model to obtain the voiceprint vector corresponding to the target audio data segment or the historical audio data segment; wherein the voiceprint extraction model is obtained by training with audio sample data and the corresponding voiceprint vector.

[0042] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of any speaker segmentation and clustering method as described in the first aspect.

[0043] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any speaker segmentation and clustering method in the first aspect are executed.

[0044] The embodiment of the present application provides a speaker segmentation and clustering method, device, electronic device and storage medium, the method comprising: obtaining the initial speaker switching point corresponding to the audio data in the current time period of this voice conversation, and the voiceprint vector corresponding to each historical audio data segment in this voice conversation; selecting the target speaker switching point from the initial speaker switching point according to the difference between the sound source direction of the previous initial audio data segment and the sound source direction of the next initial audio data segment corresponding to the initial speaker switching point in the audio data; segmenting the audio data based on all target speaker switching points to obtain the target audio data segment; constructing a similarity matrix between the voiceprint vectors of all target audio data segments and the voiceprint vectors corresponding to all historical audio data segments; performing spectral clustering on all target audio data segments and all historical audio data segments based on the similarity matrix to obtain the speaker clustering result. The present application can perform spectral clustering on the audio data by using the similarity matrix between the voiceprint vectors of the target audio data segment obtained by dividing the target switching point in this voice conversation and the voiceprint vectors corresponding to all historical audio data segments, thereby improving the accuracy of speaker segmentation and clustering. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A flowchart of a speaker segmentation and clustering method provided in an embodiment of the present application is shown;

[0047] Figure 2 A schematic diagram of the network structure of a speaker transition point detection model provided in an embodiment of the present application is shown;

[0048] Figure 3A schematic diagram of the structure of a speaker segmentation and clustering device provided in an embodiment of the present application is shown;

[0049] Figure 4 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0050] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0051] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0052] In order to enable those skilled in the art to use the contents of this application, the following implementation is provided in conjunction with a specific application scenario, "the technical field of speaker segmentation and clustering". For those skilled in the art, the general principles defined herein can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application. Although this application is mainly described around the "technical field of speaker segmentation and clustering", it should be understood that this is only an exemplary embodiment.

[0053] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0054] A speaker segmentation and clustering method provided in an embodiment of the present application is described in detail below.

[0055] Reference Figure 1As shown, it is a flowchart of a speaker segmentation and clustering method provided in an embodiment of the present application. The specific execution process of the speaker segmentation and clustering method is as follows:

[0056] S101, obtaining an initial speaker switching point corresponding to audio data in a current time period in this voice conversation, and a voiceprint vector corresponding to each historical audio data segment in this voice conversation.

[0057] S102: Filter a target speaker change point from the initial speaker change points according to a difference between a sound source direction of a previous initial audio data segment and a sound source direction of a subsequent initial audio data segment corresponding to the initial speaker change point in the audio data.

[0058] S103: Segment the audio data based on all target speaker transition points to obtain target audio data segments.

[0059] S104: construct a similarity matrix between the voiceprint vectors of all target audio data segments and the voiceprint vectors corresponding to all historical audio data segments.

[0060] S105 , performing spectral clustering on all target audio data segments and all historical audio data segments based on the similarity matrix to obtain speaker clustering results.

[0061] The embodiment of the present application provides a speaker segmentation and clustering method, which includes: obtaining the initial speaker switching point corresponding to the audio data in the current time period in this voice conversation, and the voiceprint vector corresponding to each historical audio data segment in this voice conversation; selecting the target speaker switching point from the initial speaker switching point according to the difference between the sound source direction of the previous initial audio data segment and the sound source direction of the next initial audio data segment corresponding to the initial speaker switching point in the audio data; segmenting the audio data based on all target speaker switching points to obtain the target audio data segment; constructing a similarity matrix between the voiceprint vectors of all target audio data segments and the voiceprint vectors corresponding to all historical audio data segments; performing spectral clustering on all target audio data segments and all historical audio data segments based on the similarity matrix to obtain speaker clustering results. The present application can perform spectral clustering on the audio data by using the similarity matrix between the voiceprint vectors of the target audio data segment obtained by dividing the target switching point in this voice conversation and the voiceprint vectors corresponding to all historical audio data segments, thereby improving the accuracy of speaker segmentation and clustering.

[0062] The following is a description of the exemplary steps of the present application embodiment:

[0063] S101, obtaining an initial speaker switching point corresponding to audio data in a current time period in this voice conversation, and a voiceprint vector corresponding to each historical audio data segment in this voice conversation.

[0064] In the implementation manner of the present application, the audio data in this voice conversation can be recorded based on a multi-channel microphone, or it can be recorded by a multi-channel microphone. In the process of online real-time processing of this voice conversation, the audio data of this voice conversation is collected based on a sliding window of a fixed duration (generally 10 seconds of audio data) and a fixed window shift (generally 5 seconds); this sliding window and window shift strategy can ensure the real-time segmentation and clustering of audio data. The initial speaker transition point is used to characterize the time when the speaker in the audio data changes in the current time period. That is, the speaker of the audio data at the moment before the initial speaker transition point is not the same person as the speaker of the audio data at the moment after the initial speaker transition point.

[0065] Specifically, the audio data in the current time period is input into a speaker transition point detection model to obtain an initial speaker transition point of the audio data; the speaker transition point detection model is obtained by training the audio sample data and the corresponding speaker transition points.

[0066] In the implementation mode of the present application, the input of the speaker transition point detection model is: audio data with the same duration as the fixed duration of the aforementioned sliding window. The output of the speaker transition point detection model is: the probability of each initial audio data segment with the same duration as the target preset duration from the beginning to the end in the audio data corresponding to each speaker; and based on the probability of the initial audio data segment corresponding to each speaker, the speaker corresponding to each initial audio data segment is determined; finally, the initial speaker transition point is determined according to the speaker corresponding to each initial audio data segment.

[0067] Here, the speaker transition point detection model is based on the multi-label classification problem and is optimized through permutation invariant training. Figure 2 As shown, it is a schematic diagram of the network structure of the speaker transition point detection model provided in an embodiment of the present application; the speaker transition point detection model includes 18 layers of residual network, four layers of bidirectional long short-term memory network, two layers of fully connected layers, a time pooling layer and a final classification layer. The 18 convolutional layers of the residual network are used to extract the audio features of the audio data. Each bidirectional long short-term memory network layer includes 128 units, and the first three bidirectional long short-term memory network layers use 50% dropout random inactivation. Each fully connected layer includes 128 units and a Leaky ReLU activation function. The classification layer uses a sigmoid activation function to output the speaker probability of the Kmax dimension. Kmax is the maximum number of speakers that the speaker transition point detection model can detect.

[0068] Among them, Dropout is a regularization technique commonly used in deep learning, which aims to prevent overfitting of neural networks during training. Its working principle is to randomly discard a part of neurons with a certain probability in each training iteration, so that the model will not be overly dependent on certain specific neurons, thereby reducing the risk of overfitting. This technology increases the generalization ability of the model by reducing the co-adaptability between neurons, making the model perform better on unseen data.

[0069] Specifically, the historical audio data segment is input into the voiceprint extraction model to obtain the voiceprint vector corresponding to the historical audio data segment; wherein the voiceprint extraction model is obtained by training the audio sample data and the corresponding voiceprint vector.

[0070] In the implementation manner of the present application, the voiceprint extraction model is a lightweight pre-trained residual network 34-layer model. After pre-training and lightweight optimization, the model can quickly extract the voiceprint vector corresponding to high-quality audio data and reduce the waste of computing resources.

[0071] S102: Filter a target speaker change point from the initial speaker change points according to a difference between a sound source direction of a previous initial audio data segment and a sound source direction of a subsequent initial audio data segment corresponding to the initial speaker change point in the audio data.

[0072] In the implementation mode of the present application, since the speaker switching point detection model is not 100% accurate, in order to improve the accuracy of the speaker switching point detection, when the difference between the sound source direction of the previous initial audio data segment and the sound source direction of the next initial audio data segment corresponding to the initial speaker switching point in the audio data is greater than the preset difference, the initial speaker switching point is determined as the target speaker switching point. The sound source direction is expressed in angles, and the preset difference is also an angle difference.

[0073] Furthermore, the sound source direction of the initial audio data segment is calculated by the following formula:

[0074] ;

[0075] in, is the sound source direction of the initial audio data segment, is the number of microphones, is a weight function (usually a PHAT weight (Phase Transform weight), which is used to remove the amplitude information of the initial audio data segment and retain only the phase information.) is the frequency of the initial audio data segment, is the Fourier transform of the initial audio data segment received by the i-th microphone, is the Fourier transform of the initial audio data segment received by the jth microphone, is the phase offset term (this is a phase compensation factor that takes into account the sound source position of the initial audio data segment The time difference between the arrival at the i-th microphone and the j-th microphone ).

[0076] Here, the sound source direction of the initial audio data segment is calculated by the phase-transformed directional response power algorithm. The phase-transformed directional response power algorithm utilizes the data of the multi-channel microphone array and calculates the time delay of each microphone through Fourier transform, thereby estimating the spatial position of the initial audio data segment.

[0077] S103: Segment the audio data based on all target speaker transition points to obtain target audio data segments.

[0078] In the implementation manner of the present application, the audio data is segmented at the position corresponding to the target speaker's switching point to obtain the target audio data segment.

[0079] S104: construct a similarity matrix between the voiceprint vectors of all target audio data segments and the voiceprint vectors corresponding to all historical audio data segments.

[0080] In the implementation manner of the present application, the similarity matrix is ​​an n*n matrix, where n is the sum of the total number of target audio data segments and the total number of historical audio data segments. The similarity matrix includes the similarity between the voiceprint vector of each target audio data segment and the voiceprint vector of each target audio data segment, the similarity between the voiceprint vector of each target audio data segment and the voiceprint vector of each historical audio data segment, and the similarity between the voiceprint vector of each historical audio data segment and the voiceprint vector of each historical audio data segment.

[0081] Specifically, the target audio data segments are input into the voiceprint extraction model to obtain the voiceprint vectors corresponding to the target audio data segments; wherein the voiceprint extraction model is trained by audio sample data and the corresponding voiceprint vectors.

[0082] In the implementation manner of the present application, the voiceprint extraction model is a lightweight pre-trained residual network 34-layer model. After pre-training and lightweight optimization, the model can quickly extract the voiceprint vector corresponding to high-quality audio data and reduce the waste of computing resources.

[0083] Specifically, if the duration of the current voice conversation does not exceed the preset duration (which may be 5 minutes), a similarity matrix is ​​constructed between the voiceprint vectors of all target audio data segments and the voiceprint vectors corresponding to all historical audio data segments, including: the similarities between the voiceprint vectors of each audio data segment are combined into a similarity matrix without weighting each similarity.

[0084] Specifically, if the duration of the current voice conversation exceeds the preset duration (which may be 5 minutes), the similarity matrix between the voiceprint vectors of all target audio data segments and the voiceprint vectors corresponding to all historical audio data segments is constructed through the following steps:

[0085] Step 1: Calculate the similarity weights between each audio data segment based on the difference between the sound source direction corresponding to each audio data segment and the speaker sound source direction of each historical audio data segment.

[0086] In the implementation manner of the present application, the audio data segment includes a target audio data segment and a historical audio data segment. The historical audio data segment used to calculate the similarity weight can be the earliest historical audio data segment of the target preset session duration; the target preset session duration is greater than or equal to the preset session duration. The speaker sound source direction of each historical audio data segment is determined based on the clustering results of all historical speakers corresponding to each historical audio data segment.

[0087] The difference between the sound source direction corresponding to each audio data segment and the speaker sound source direction of each historical audio data segment includes: the difference between the sound source direction of each target audio data segment and the speaker sound source direction of each historical audio data segment, and the difference between the sound source direction of each historical audio data segment and the speaker sound source direction of each historical audio data segment. The similarity weights between each audio data segment include the similarity weights between the sound source direction of each target audio data segment and the speaker sound source direction of each historical audio data segment, and the similarity weights between the sound source direction of each historical audio data segment and the speaker sound source direction of each historical audio data segment.

[0088] Here, if the target preset session duration is too short, the calculated similarity weight may be inaccurate, so the accuracy of the calculated similarity weight needs to be ensured by setting the preset session duration and the target preset session duration.

[0089] Specifically, the speaker sound source direction of each historical audio data segment is obtained through the following steps:

[0090] a. Obtain the clustering results of all historical speakers corresponding to each historical audio data segment.

[0091] In the implementation manner of the present application, the historical speaker clustering result includes the speaker corresponding to each historical audio data segment.

[0092] b. Determine the target speaker corresponding to each historical audio data segment based on the clustering results of all historical speakers corresponding to each historical audio data segment.

[0093] In the implementation manner of the present application, the number of times each historical audio data corresponds to each speaker is counted; and the speaker with the largest number of times is used as the target speaker corresponding to each historical audio data segment.

[0094] c. The average value of the sound source directions of all historical audio data segments corresponding to each target speaker is determined as the sound source direction of each target speaker.

[0095] d. Determine the sound source direction of the target speaker corresponding to each historical audio data segment as the sound source direction of the speaker of each historical audio data segment.

[0096] Further, according to the difference between the sound source direction corresponding to each audio data segment and the speaker sound source direction of each historical audio data segment, the similarity weight between each audio data segment is calculated, including:

[0097] In the implementation manner of the present application, the difference between the sound source direction corresponding to each audio data segment and the speaker sound source direction of each historical audio data segment is substituted into the following formula to calculate the similarity weight between each audio data segment and each audio data segment:

[0098] ;

[0099] in, is the similarity weight between the i-th audio data segment and the j-th historical audio data segment, is the sound source direction of the i-th audio data segment, is the speaker source direction of the jth historical audio data segment, is the preset weight coefficient.

[0100] , Less than 1. The similarity weights between the target audio data segments can be preset, but the similarity weights between the target audio data segments are less than 1.

[0101] Step 2: Construct a similarity matrix of the voiceprint vectors of all audio data segments based on the similarity weights between each audio data segment and each historical audio data segment.

[0102] In an implementation manner of the present application, the similarity between the voiceprint vector of each audio data segment and the voiceprint vector of each historical audio data segment is weighted based on the similarity weight between each audio data segment and each historical audio data segment, and the similarity between the voiceprint vector of each target audio data segment and the voiceprint vector of each target audio data segment is weighted based on the preset similarity weight between each target audio data segment and each target audio data segment, so as to obtain a similarity matrix of the voiceprint vectors of all audio data segments.

[0103] S105 , performing spectral clustering on all target audio data segments and all historical audio data segments based on the similarity matrix to obtain speaker clustering results.

[0104] In the implementation manner of the present application, the speaker clustering result includes the speaker corresponding to each target audio data segment and the speaker corresponding to each historical audio data segment. The speaker corresponding to each target audio data segment is displayed on the user page.

[0105] Based on the same inventive concept, the embodiment of the present application also provides a speaker segmentation and clustering device corresponding to the speaker segmentation and clustering method. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned speaker segmentation and clustering method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0106] Reference Figure 3 FIG. 1 is a schematic diagram of a speaker segmentation and clustering device provided in an embodiment of the present application, wherein the speaker segmentation and clustering device comprises:

[0107] The acquisition module 301 is used to acquire the initial speaker switching point corresponding to the audio data in the current time period in this voice conversation, and the voiceprint vector corresponding to each historical audio data segment in this voice conversation;

[0108] A screening module 302, configured to screen a target speaker change point from the initial speaker change points according to a difference between a sound source direction of a previous initial audio data segment and a sound source direction of a subsequent initial audio data segment corresponding to the initial speaker change point in the audio data;

[0109] A segmentation module 303 is used to segment the audio data based on all target speaker transition points to obtain target audio data segments;

[0110] A construction module 304 is used to construct a similarity matrix between the voiceprint vectors of all target audio data segments and the voiceprint vectors corresponding to all historical audio data segments;

[0111] The clustering module 305 is used to perform spectral clustering on all target audio data segments and all historical audio data segments based on the similarity matrix to obtain speaker clustering results.

[0112] In a possible implementation, the construction module 304 is specifically used to calculate the similarity weight between each audio data segment and each historical audio data segment according to the difference between the sound source direction corresponding to each audio data segment and the speaker sound source direction of each historical audio data segment if the conversation duration of the current voice conversation exceeds the preset conversation duration; construct a similarity matrix of the voiceprint vectors of all audio data segments according to the similarity weight between each audio data segment and each historical audio data segment; the audio data segments include target audio data segments and historical audio data segments.

[0113] In a possible implementation, the construction module 304 is specifically used to obtain all historical speaker clustering results corresponding to each historical audio data segment; the historical speaker clustering results include the speakers corresponding to each historical audio data segment; according to all historical speaker clustering results corresponding to each historical audio data segment, the target speaker corresponding to each historical audio data segment is determined; the average value of the sound source directions of all historical audio data segments corresponding to each target speaker is determined as the sound source direction of each target speaker; the sound source direction of the target speaker corresponding to each historical audio data segment is determined as the speaker sound source direction of each historical audio data segment.

[0114] In a possible implementation, the construction module 304 is specifically used to count the number of times each historical audio data corresponds to each speaker; and the speaker with the largest number of times is used as the target speaker corresponding to each historical audio data segment.

[0115] In a possible implementation, the construction module 304 is specifically configured to substitute the difference between the sound source direction corresponding to each audio data segment and the speaker sound source direction of each historical audio data segment into the following formula to calculate the similarity weight between each audio data segment and each historical audio data segment:

[0116] ;

[0117] in, is the similarity weight between the i-th audio data segment and the j-th historical audio data segment, is the sound source direction of the i-th audio data segment, is the speaker source direction of the jth historical audio data segment, is the preset weight coefficient.

[0118] In one possible implementation, the acquisition module 301 is specifically used to input the target audio data segment or the historical audio data segment into the voiceprint extraction model to obtain the voiceprint vector corresponding to the target audio data segment or the historical audio data segment; wherein the voiceprint extraction model is obtained by training with audio sample data and the corresponding voiceprint vector.

[0119] The embodiment of the present application provides a speaker segmentation and clustering device, which includes: an acquisition module 301, which is used to obtain an initial speaker transition point corresponding to audio data in a current time period in this voice conversation, and a voiceprint vector corresponding to each historical audio data segment in this voice conversation; a screening module 302, which is used to screen a target speaker transition point from the initial speaker transition point according to the difference between the sound source direction of the previous initial audio data segment and the sound source direction of the next initial audio data segment corresponding to the initial speaker transition point in the audio data; a segmentation module 303, which is used to segment the audio data based on all target speaker transition points to obtain target audio data segment; a construction module 304, used to construct a similarity matrix between the voiceprint vectors of all target audio data segments and the voiceprint vectors corresponding to all the historical audio data segments; a clustering module 305, used to perform spectral clustering on all the target audio data segments and all the historical audio data segments based on the similarity matrix to obtain speaker clustering results; the present application can perform spectral clustering on audio data through the similarity matrix between the voiceprint vectors of the target audio data segments obtained by dividing based on the target conversion points in this voice conversation and the voiceprint vectors corresponding to all the historical audio data segments, thereby improving the accuracy of speaker segmentation and clustering.

[0120] like Figure 4 As shown, an electronic device 400 provided in an embodiment of the present application includes: a processor 401, a memory 402 and a bus, the memory 402 stores machine-readable instructions executable by the processor 401, when the electronic device is running, the processor 401 and the memory 402 communicate through the bus, and the processor 401 executes the machine-readable instructions to perform the steps of the above-mentioned speaker segmentation clustering method.

[0121] Specifically, the memory 402 and the processor 401 can be general-purpose memories and processors, which are not specifically limited here. When the processor 401 runs the computer program stored in the memory 402, the speaker segmentation and clustering method can be executed.

[0122] Corresponding to the above-mentioned speaker segmentation and clustering method, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned speaker segmentation and clustering method are executed.

[0123] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0124] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0125] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0126] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the information processing method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0127] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A speaker segmentation and clustering method, characterized in that: The method comprises: Obtain the initial speaker transition point corresponding to the audio data in the current time period in this voice conversation, and the voiceprint vector corresponding to each historical audio data segment in this voice conversation; Filtering a target speaker change point from the initial speaker change points according to a difference between a sound source direction of a previous initial audio data segment and a sound source direction of a subsequent initial audio data segment corresponding to the initial speaker change point in the audio data; Segment the audio data based on all the target speaker transition points to obtain target audio data segments; Constructing a similarity matrix between the voiceprint vectors of all the target audio data segments and the voiceprint vectors corresponding to all the historical audio data segments, including: if the conversation duration of the current voice conversation exceeds the preset conversation duration, calculating the similarity weight between each audio data segment and each historical audio data segment according to the difference between the sound source direction corresponding to each audio data segment and the speaker sound source direction of each historical audio data segment; constructing a similarity matrix of the voiceprint vectors of all audio data segments according to the similarity weight between each audio data segment and each historical audio data segment; the audio data segments include the target audio data segments and the historical audio data segments; Performing spectral clustering on all the target audio data segments and all the historical audio data segments based on the similarity matrix to obtain speaker clustering results; The speaker sound source direction of each historical audio data segment is obtained by the following steps: obtaining all historical speaker clustering results corresponding to each historical audio data segment; the historical speaker clustering results include the speakers corresponding to each historical audio data segment; determining the target speaker corresponding to each historical audio data segment according to all historical speaker clustering results corresponding to each historical audio data segment; determining the average value of the sound source directions of all historical audio data segments corresponding to each target speaker as the sound source direction of each target speaker; determining the sound source direction of the target speaker corresponding to each historical audio data segment as the speaker sound source direction of each historical audio data segment; The calculating the similarity weight between each audio data segment and each historical audio data segment according to the difference between the sound source direction corresponding to each audio data segment and the sound source direction of the speaker of each historical audio data segment comprises: substituting the difference between the sound source direction corresponding to each audio data segment and the sound source direction of the speaker of each historical audio data segment into the following formula to calculate the similarity weight between each audio data segment and each historical audio data segment: ； in, is the similarity weight between the i-th audio data segment and the j-th historical audio data segment, is the sound source direction of the i-th audio data segment, is the speaker source direction of the jth historical audio data segment, is the preset weight coefficient.

2. The speaker segmentation and clustering method according to claim 1, characterized in that: The step of determining the target speaker corresponding to each historical audio data segment according to the clustering results of all historical speakers corresponding to each historical audio data segment includes: Count the number of times each historical audio data corresponds to each speaker; The speaker with the most times is taken as the target speaker corresponding to each historical audio data segment.

3. The speaker segmentation and clustering method according to claim 2, characterized in that: The voiceprint vector corresponding to the target audio data segment or the voiceprint vector corresponding to the historical audio data segment is obtained by the following steps: Inputting the target audio data segment or the historical audio data segment into a voiceprint extraction model to obtain a voiceprint vector corresponding to the target audio data segment or the historical audio data segment; The voiceprint extraction model is obtained by training with audio sample data and corresponding voiceprint vectors.

4. A speaker segmentation and clustering device, characterized in that: The device is used to implement the speaker segmentation and clustering method according to claim 1, and the device comprises: An acquisition module, used to acquire the initial speaker switching point corresponding to the audio data in the current time period in this voice conversation, and the voiceprint vector corresponding to each historical audio data segment in this voice conversation; a screening module, configured to screen a target speaker change point from the initial speaker change points according to a difference between a sound source direction of a previous initial audio data segment and a sound source direction of a subsequent initial audio data segment corresponding to the initial speaker change point in the audio data; A segmentation module, configured to segment the audio data based on all the target speaker transition points to obtain target audio data segments; A construction module, for constructing a similarity matrix between the voiceprint vectors of all the target audio data segments and the voiceprint vectors corresponding to all the historical audio data segments, including: if the duration of the current voice conversation exceeds the preset duration, then according to the difference between the sound source direction corresponding to each audio data segment and the speaker sound source direction of each historical audio data segment, calculate the similarity weight between each audio data segment and each historical audio data segment; according to the similarity weight between each audio data segment and each historical audio data segment, construct a similarity matrix of the voiceprint vectors of all audio data segments; the audio data segments include target audio data segments and historical audio data segments; A clustering module, configured to perform spectral clustering on all the target audio data segments and all the historical audio data segments based on the similarity matrix to obtain speaker clustering results; The construction module is specifically used to obtain all historical speaker clustering results corresponding to each historical audio data segment; the historical speaker clustering results include speakers corresponding to each historical audio data segment; according to all historical speaker clustering results corresponding to each historical audio data segment, determine the target speaker corresponding to each historical audio data segment; determine the average value of the sound source direction of all historical audio data segments corresponding to each target speaker as the sound source direction of each target speaker; determine the sound source direction of the target speaker corresponding to each historical audio data segment as the speaker sound source direction of each historical audio data segment; The construction module is specifically used to substitute the difference between the sound source direction corresponding to each audio data segment and the speaker sound source direction of each historical audio data segment into the following formula to calculate the similarity weight between each audio data segment and each historical audio data segment: ； in, is the similarity weight between the i-th audio data segment and the j-th historical audio data segment, is the sound source direction of the i-th audio data segment, is the speaker source direction of the jth historical audio data segment, is the preset weight coefficient.

5. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of the speaker segmentation and clustering method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the speaker segmentation and clustering method according to any one of claims 1 to 3 are executed.

Images