Speaker transformation point detection method and device, electronic equipment and storage medium

Abstract

The invention relates to a speaker transformation point detection method and device, electronic equipment and a storage medium. The speaker transformation point detection method comprises the following steps: acquiring a voice audio to be processed; the to-be-processed voice audio is processed through a preset student model, a posterior probability result is obtained, and the posterior probability result comprises the posterior probability of each time node in the voice audio, the posterior probability represents the probability that voices contained in the voice audio corresponding to the corresponding time node and voices contained in the voice audio corresponding to the previous time node belong to different speakers. And determining a speaker transformation point in the voice audio according to the posterior probability of each time node and a preset probability threshold. Through the speaker transformation point detection method and device, on the premise that computing resources consumed by the student model are limited in a certain range, the detection precision of the student model when the student model executes a speaker transformation point detection task is improved.

Description

Technical Field

[0001] This disclosure relates to the field of audio processing, and in particular to speaker change point detection methods, apparatus, electronic devices, and storage media. Background Technology

[0002] For audio obtained in multi-person speaking scenarios, speech recognition technology is usually required to post-process the audio, obtain segmented speech, and provide sentence segmentation information.

[0003] In related technologies, speaker change detection (SCD) models are generally used to identify speaker change points in an audio clip, thereby providing phrasing information. However, directly trained speaker change detection models suffer from poor recognition accuracy. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides a speaker change point detection method, apparatus, electronic device, and storage medium.

[0005] According to a first aspect of the present disclosure, a speaker transition point detection method is provided, comprising: acquiring speech audio to be processed, the speech audio including the speech of at least two different speakers; processing the speech audio to be processed through a preset student model to obtain a posterior probability result, the posterior probability result including the posterior probability of each time point in the speech audio, the posterior probability representing the probability that the speech audio corresponding to the corresponding time point belongs to a different speaker than the speech audio corresponding to the previous time point, the student model being trained based on an initial model and a teacher model, the model structure of the initial model being different from the model structure of the teacher model; determining speaker transition points in the speech audio based on the posterior probabilities of each time point and a preset probability threshold, wherein the speech audio corresponding to the speaker transition point belongs to a different speaker than the speech audio corresponding to the previous time point.

[0006] In one embodiment, the student model is trained as follows: the initial model processes training audio to obtain the output layer output and intermediate layer output of the initial model, and the teacher model processes the training audio to obtain the second weighted output of the teacher model; a first loss is obtained based on the output layer output, and a second loss is obtained based on the intermediate layer output and the second weighted output; the sum of the first loss and the second loss is determined as the target loss; the initial model is trained until the target loss meets a preset condition, and the initial model is determined as the student model.

[0007] In one embodiment, the initial model includes multiple first feature extraction layers; the step of processing training audio through the initial model to obtain the output layer output and intermediate layer output of the initial model includes: acquiring the filter bank (fbank) features of the training audio; extracting features from the filter bank (fbank) features step by step through the multiple first feature extraction layers; determining the feature data extracted by a preset first feature extraction layer among the multiple first feature extraction layers as the intermediate layer output; and determining the feature data extracted by the last first feature extraction layer among the multiple first feature extraction layers as the output layer output.

[0008] In one embodiment, the teacher model includes multiple second feature extraction layers; the step of processing the training audio through the teacher model to obtain the second weighted output of the teacher model includes: acquiring audio features of the training audio, wherein the audio features are distinct from the filter bank (fbank) features of the training audio; extracting features from the audio features step by step through multiple second feature extraction layers, acquiring the output of each second feature extraction layer, and acquiring the weight corresponding to each first feature extraction layer, wherein the output of each second feature extraction layer represents a different granularity of the training audio; and performing a weighted summation on the multiple second feature extraction layers based on the output of each second feature extraction layer and the weight corresponding to each first feature extraction layer, and determining the parameter obtained by the weighted summation as the second weighted output.

[0009] In one embodiment, obtaining the second loss based on the intermediate layer output and the second weighted output includes: processing the intermediate layer output through a preset adapter to obtain a first weighted output, wherein the adapter includes a single-layer convolutional neural network and an activation function, and the shape of the adapter output corresponds to the shape of the teacher model output; and obtaining the second loss based on the first weighted output, the second weighted output, and a preset second loss acquisition method.

[0010] In one implementation, during model training, the parameters of the teacher model remain unchanged, while the weights corresponding to each of the plurality of second feature extraction layers included in the teacher model change as training progresses.

[0011] In one embodiment, determining the speaker transition point in the speech audio based on the posterior probability of each time point and a preset probability threshold includes: for the posterior probability of each time point, in response to the posterior probability being greater than or equal to the probability threshold, determining the time point in the speech audio corresponding to the posterior probability as the speaker transition point.

[0012] In one implementation, the teacher model is a self-supervised pre-trained model.

[0013] According to a second aspect of the present disclosure, a speaker transition point detection device is provided, comprising: an acquisition unit for acquiring speech audio to be processed, the speech audio including the speech of at least two different speakers; a processing unit for processing the speech audio to be processed using a preset student model to obtain a posterior probability result, the posterior probability result including the posterior probability of each time point in the speech audio, the posterior probability representing the probability that the speech audio at the corresponding time point belongs to a different speaker than the speech audio at the previous time point, the student model being trained based on an initial model and a teacher model, the model structure of the initial model being different from the model structure of the teacher model; and a determination unit for determining speaker transition points in the speech audio based on the posterior probabilities of each time point and a preset probability threshold, wherein the speech audio at the speaker transition point belongs to a different speaker than the speech audio at the previous time point.

[0014] In one embodiment, the student model is trained by the processing unit in the following manner: the initial model processes the training audio to obtain the output layer output and intermediate layer output of the initial model, and the teacher model processes the training audio to obtain the second weighted output of the teacher model; a first loss is obtained based on the output layer output, and a second loss is obtained based on the intermediate layer output and the second weighted output; the sum of the first loss and the second loss is determined as the target loss; the initial model is trained until the target loss meets a preset condition, and the initial model is determined as the student model.

[0015] In one embodiment, the initial model includes multiple first feature extraction layers; the processing unit processes the training audio through the initial model in the following manner to obtain the output layer output and intermediate layer output of the initial model: obtaining the filter bank fbank features of the training audio, and extracting features from the filter bank fbank features step by step through the multiple first feature extraction layers; determining the feature data extracted by a preset first feature extraction layer among the multiple first feature extraction layers as the intermediate layer output, and determining the feature data extracted by the last first feature extraction layer among the multiple first feature extraction layers as the output layer output.

[0016] In one embodiment, the teacher model includes multiple second feature extraction layers; the processing unit processes the training audio through the teacher model in the following manner to obtain the second weighted output of the teacher model: acquiring the audio features of the training audio, wherein the audio features are distinct from the filter bank (fbank) features of the training audio;

[0017] The audio features are extracted step by step through multiple second feature extraction layers. The output of each second feature extraction layer is obtained, and the weights corresponding to each first feature extraction layer are also obtained. The output of each second feature extraction layer represents the training audio at different granularities. For the multiple second feature extraction layers, a weighted sum is performed based on the output of each second feature extraction layer and the weights corresponding to each first feature extraction layer. The parameters obtained by the weighted sum are determined as the second weighted output.

[0018] In one embodiment, the processing unit obtains a second loss based on the intermediate layer output and the second weighted output in the following manner: processing the intermediate layer output through a preset adapter to obtain a first weighted output, wherein the adapter includes a single-layer convolutional neural network and an activation function, and the shape of the adapter output corresponds to the shape of the teacher model output; and obtaining the second loss based on the first weighted output, the second weighted output, and a preset second loss acquisition method.

[0019] In one implementation, during model training, the parameters of the teacher model remain unchanged, while the weights corresponding to each of the plurality of second feature extraction layers included in the teacher model change as training progresses.

[0020] In one embodiment, the processing unit determines the speaker transition point in the speech audio based on the posterior probability of each time point and a preset probability threshold as follows: for the posterior probability of each time point, in response to the posterior probability being greater than or equal to the probability threshold, the time point in the speech audio corresponding to the posterior probability is determined as the speaker transition point.

[0021] In one implementation, the teacher model is a self-supervised pre-trained model.

[0022] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: execute the speaker change point detection method described in the first aspect or any embodiment of the first aspect.

[0023] According to a fourth aspect of the present disclosure, a storage medium is provided, the storage medium storing instructions that, when executed by a processor, enable the processor to perform the speaker change point detection method described in the first aspect or any embodiment of the first aspect.

[0024] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: After acquiring speech audio including at least two different speakers, the speech audio to be processed is processed through a preset student model to obtain the posterior probability of each time point in the speech audio. The posterior probability represents the probability that the speech audio corresponding to the corresponding time point belongs to a different speaker than the speech audio corresponding to the previous time point. The student model is trained based on the initial model and the teacher model, and its model structure is different from that of the teacher model. Based on the posterior probability of each time point and a preset probability threshold, the speaker transition points in the speech audio are determined. Through this disclosure, the detection accuracy of the student model when performing speaker transition point detection tasks is improved while limiting the computational resources consumed by the student model to a certain range.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0027] Figure 1 This is a flowchart illustrating a speaker change point detection method according to an exemplary embodiment.

[0028] Figure 2 This is a schematic diagram illustrating a speaker change point detection method according to an exemplary embodiment of the present disclosure.

[0029] Figure 3 This is a flowchart illustrating a method for training a student model according to an exemplary embodiment.

[0030] Figure 4 This is a flowchart illustrating a method for obtaining the output of the initial model's output layer and the output of the intermediate layer, according to an exemplary embodiment.

[0031] Figure 5 This is a flowchart illustrating a method for obtaining a second weighted output of a teacher model according to an exemplary embodiment.

[0032] Figure 6This is a flowchart illustrating a method for obtaining a second loss based on intermediate layer output and a second weighted output, according to an exemplary embodiment.

[0033] Figure 7 This is a flowchart illustrating a method for determining speaker transition points in speech audio according to an exemplary embodiment.

[0034] Figure 8 This is a schematic diagram of a model architecture for training a speaker change point detection model according to an exemplary embodiment of the present disclosure.

[0035] Figure 9 This is a block diagram illustrating a speaker change point detection device according to an exemplary embodiment.

[0036] Figure 10 This is a block diagram illustrating an apparatus for speaker change point detection according to an exemplary embodiment.

[0037] Figure 11 This is a block diagram illustrating an apparatus for speaker change point detection according to an exemplary embodiment. Detailed Implementation

[0038] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure.

[0039] The speaker change point detection method provided in this disclosure is applied to scenarios where a neural network model is trained and the obtained neural network model is used to identify speaker change points in audio.

[0040] For audio obtained in multi-person speaking scenarios, speech recognition technology is typically used for post-processing to obtain segmented speech and provide sentence segmentation information. In related technologies, for audio processing tasks involving speech segmentation, speaker change detection (SCD) is generally used to identify speaker change points in an audio segment, thereby obtaining sentence segmentation information for speech segmentation. Speaker change points are the time points where speakers switch within an audio segment. Speaker change detection is a technique for detecting the temporal boundaries between different speaker speech segments. SCD has various applications, such as speaker diarization (detecting who speaks and when), speaker tracking, and speech recognition in multi-person dialogue scenarios. Traditional speaker change detection methods use a sliding window approach, explicitly comparing the left and right windows of a speech frame for speaker changes, which suffers from low processing efficiency. With the development of deep neural networks, sequence-labeling methods based on neural networks, i.e., using the trained speaker change detection model to identify speaker change points in audio, have become the mainstream approach.

[0041] In related technologies, there are techniques that use a single model to obtain sentence segmentation information. This involves directly training a randomly initialized model using training audio data to perform an SCD task, resulting in a speaker change detection model. This trained model is then used to process the audio data and determine speaker change points. However, the directly trained speaker change detection model has a relatively simple structure, such as containing two Long Short-Term Memory (LSTM) layers and two fully connected layers, with a data size of only 0.18M. This limits its inference ability and results in poor recognition accuracy; the inferred speaker change points deviate somewhat from the actual speaker change points in the audio.

[0042] In recent years, self-supervised pre-trained models have attracted widespread attention due to their powerful modeling capabilities and ability to effectively utilize large amounts of unlabeled data. Many models have demonstrated very strong audio representation capabilities through training methods that predict masking parts and employing specifically designed loss functions. Many downstream tasks have achieved excellent results with the help of self-supervised pre-trained models. Given the powerful audio representation capabilities of self-supervised pre-trained models, and their wide applicability to various downstream audio tasks, a related technology exists that combines speaker change detection models with self-supervised pre-trained models for speaker change detection. This involves fine-tuning a self-supervised pre-trained model (such as wav2vec2.0) on a speaker change detection task, thereby leveraging the capabilities of the self-supervised pre-trained model in speaker change detection. However, this approach requires a self-supervised pre-trained model for inference when performing speaker change detection. This means the size of the self-supervised pre-trained model is the same as the model size used for inference, and these models are generally much larger than typical speaker change detection models. For example, a self-supervised pre-trained model is approximately 95MB, while a typical speaker change detection model is about 0.18MB. Therefore, the computational resources consumed by the speaker change detection task using a self-supervised pre-trained model are far greater than those consumed by a typical speaker change detection model. Furthermore, in real-world products, the module performing speaker change detection is typically a pre-processing module for audio processing, requiring a small model size and minimal computational resources. Therefore, the approach of using a self-supervised pre-trained model for speaker change detection is unacceptable for practical applications of speaker change detection models in real-world products.

[0043] In summary, the techniques using single-speaker change point detection models to obtain sentence segmentation information suffer from limited reasoning ability and poor recognition accuracy, resulting in significant discrepancies between the inferred sentence information and the actual audio. Furthermore, the approach combining self-supervised pre-trained models in related technologies suffers from large model data volumes and high computational resource consumption for inference, making it unsuitable for practical products.

[0044] In view of this, this disclosure proposes a speaker transition point detection method. After acquiring audio recordings containing at least two different speakers, the audio recordings are processed using a pre-set student model to obtain the posterior probability of each time point in the audio recordings. The posterior probability represents the probability that the audio recordings at the corresponding time point belong to different speakers than the audio recordings at the previous time point. The student model is trained based on an initial model and a teacher model, and its model structure is different from that of the teacher model. Based on the posterior probabilities of each time point and a pre-set probability threshold, speaker transition points in the audio recordings are determined. Through this disclosure, the detection accuracy of the student model when performing speaker transition point detection tasks is improved while limiting the computational resources consumed by the student model to a certain range.

[0045] The speaker change point detection method proposed in this disclosure is applicable to smart electronic devices such as mobile phones and tablets. Its main application scenarios are those where speaker change point detection algorithms are used. These scenarios include, but are not limited to: speech segmentation in multi-person dialogue scenarios, providing sentence segmentation information; and speech segmentation in spatial stereo tasks where different speakers provide different orientations, providing sentence segmentation information.

[0046] Figure 1 This is a flowchart illustrating a speaker transition point detection method according to an exemplary embodiment. Figure 1 As shown, the method includes steps S101 to S103.

[0047] In step S101, the audio to be processed is acquired, which includes the voices of at least two different speakers.

[0048] In step S102, the speech audio to be processed is processed by a preset student model to obtain the posterior probability result. The posterior probability result includes the posterior probability of each time point in the speech audio. The posterior probability represents the probability that the speech audio at the corresponding time point and the speech audio at the previous time point belong to different speakers.

[0049] The student model is trained based on the initial model and the teacher model, and the model structure of the initial model is different from that of the teacher model.

[0050] In step S103, based on the posterior probability of each time point and the preset probability threshold, the speaker change point in the speech audio is determined. The speech audio corresponding to the speaker change point belongs to a different speaker than the speech audio corresponding to the previous time point.

[0051] In this embodiment, after obtaining the speech audio to be processed, the long audio sequence is divided into multiple audio segments of preset length (e.g., 1.5 seconds). Adjacent audio segments of these preset lengths overlap (e.g., with a step size of 0.1 seconds). The segmented audio segments are processed by a student model, and the posterior probability (posterior probability result) corresponding to each time point in the speech audio to be processed is obtained from the student model output. The posterior probability is compared one by one with a preset probability threshold to determine the speaker transition points among the multiple time points corresponding to the speech audio. These transition points represent the time points in the speech audio to be processed where the speech belongs to a different speaker than the speech audio corresponding to the previous time point.

[0052] In an exemplary embodiment of this disclosure, such as Figure 2 A schematic diagram of the speaker change point detection method. This disclosure uses the speaker change point detection model to process the speech audio to be processed in the following way to obtain speaker change points: The long audio sequence (the speech audio to be processed) is divided into audio segments with overlapping regions of the same length as the training (e.g., 1.5 seconds) (e.g., the step size between adjacent segments is 0.1 seconds). Multiple audio segments are processed by the speaker change point detection model to obtain the curve corresponding to each audio segment. The curve includes the posterior probability corresponding to each time node of each audio segment in multiple audio segments. Each time step (e.g., 0.1 seconds) of the audio segment has multiple candidate probabilities [e.g., using speakerchange(1) to represent that the speaker has changed, and using no speaker change(0) to represent that the speaker has not changed]. The posterior probability corresponding to each time node is the probability that the speaker has changed at that time point. The so-called speaker change means that the speaker corresponding to the speech in the audio at the current time point is different from the speaker corresponding to the speech in the audio at the previous time point. Based on the posterior probability of each time point in the speech audio and a preset adjustable threshold θ, the speaker transition point can be determined from the posterior probability of each time point.

[0053] In this embodiment, the student model is obtained through synchronous training in conjunction with the teacher model, and the model structure of the student model differs from that of the teacher model. This disclosure trains the student model using a knowledge distillation (KD) method, combining it with a pre-trained teacher model to train the initial model corresponding to the student model, thus obtaining the student model. During model training, the initial model corresponding to the student model learns information from both the training task itself and the knowledge from each feature extraction layer of the teacher model, thereby improving the student model's performance in handling speaker change detection tasks. Through this disclosure, the student model is obtained through synchronous training in conjunction with the teacher model. The student model processes the acquired speech audio to be processed, obtaining the posterior probability corresponding to each time point in the speech audio. Based on the posterior probability and probability threshold corresponding to each time point, the speaker change points representing the changes between different voices in the speech audio are determined. By combining other models during model training and performing model inference based on the student model itself during model inference, the detection accuracy of the student model in performing speaker change point detection tasks is improved while limiting the computational resources consumed by the student model to a certain range.

[0054] In this embodiment of the disclosure, during model training, the initial model corresponding to the student model is trained in conjunction with a well-developed teacher model. This allows the initial model to learn information from both the training task itself and the feature extraction layer of the teacher model. Therefore, during model training, it is necessary to consider the model loss of the initial model itself in processing the training audio, as well as the loss between the output of the initial model and the output of the teacher model. The following embodiments of this disclosure illustrate the method for training the student model.

[0055] Figure 3 This is a flowchart illustrating a method for training a student model according to an exemplary embodiment. Figure 3 As shown, the method includes steps S201 to S204.

[0056] In step S201, the training audio is processed by the initial model to obtain the output layer output and intermediate layer output of the initial model, and the training audio is processed by the teacher model to obtain the second weighted output of the teacher model.

[0057] In step S202, a first loss is obtained based on the output of the output layer, and a second loss is obtained based on the output of the intermediate layer and the second weighted output.

[0058] In step S203, the sum of the first loss and the second loss is determined as the target loss.

[0059] In step S204, the initial model is trained until the target loss meets the preset conditions, and the initial model is determined as the student model.

[0060] In this embodiment of the disclosure, during the model training process for the initial model, the long audio data (training audio) is segmented into multiple adjacent audio segments with overlapping storage regions, and random noise is added to each of the multiple audio segments for data augmentation. It is understood that the training process of the initial model corresponds to the inference process of the first model in actual application, and the same segmentation principles are set for both the training audio and the speech audio to be processed.

[0061] In this embodiment of the disclosure, during model training, the initial model and the teacher model respectively extract features from the training audio. For the initial model, the disclosure obtains the final output (output layer output) of the initial model processing the training audio, and obtains the loss (first loss) corresponding to the initial model's own execution of the training task based on the final output of the initial model. The disclosure also obtains the output of a certain layer (intermediate layer output) during the initial model's processing of the training audio, used to obtain the loss between the initial model output and the teacher model output (second loss). For the teacher model, the disclosure obtains the final output (second weighted output) of the teacher model, used to calculate the second loss by combining it with the obtained intermediate layer output.

[0062] In an exemplary embodiment of this disclosure, the first loss of the initial model is obtained based on the cross-entropy loss function.

[0063] In this embodiment, the sum of the first loss and the second loss is determined as the target loss. The initial model is continuously trained based on the training audio and the teacher model until the target loss meets a preset condition, at which point the initial model is identified as the student model. This disclosure considers the loss between the output of the initial model and the output of the teacher model during training, as well as the loss corresponding to the initial model's own execution of the training task. This allows the initial model corresponding to the student model to learn information from both the training task itself and the knowledge from the feature extraction layer of the teacher model. Thus, while limiting the computational resources consumed by the student model to a certain range, the detection accuracy of the student model when performing speaker change point detection tasks is improved.

[0064] It is understandable that excessive training iterations during the initial model training process can lead to overfitting in the resulting student model, negatively impacting its processing performance. Therefore, this disclosure allows setting conditions for training completion for parameters other than the target loss to prevent overfitting in the trained student model. For example, a training iteration threshold can be set; once the threshold is reached for the initial model, training is considered complete, and the student model is obtained. Alternatively, a training audio library can be provided. The initial model is trained using training audio from this library, and the initial model is trained using all training audio from the library in conjunction with the teacher model. After iterating through the training audio one or more times, training is considered complete, and the student model is obtained.

[0065] In this embodiment, the initial model includes multiple first feature extraction layers. These multiple first feature extraction layers are used to extract features from different levels / dimensions of the training audio. The output layer output in this disclosure is the output of the last first feature extraction layer among the multiple first feature extraction layers, while the intermediate output layer is the output of a specific first feature extraction layer among the multiple first feature extraction layers. The following embodiments of this disclosure illustrate the method for obtaining the output layer output and intermediate layer output of the initial model.

[0066] Figure 4 This is a flowchart illustrating a method for obtaining the output of the initial model's output layer and the output of the intermediate layer, according to an exemplary embodiment. Figure 4 As shown, the method includes steps S301 to S302.

[0067] In step S301, the filter bank fbank features of the training audio are obtained, and the filter bank fbank features are extracted step by step through multiple first feature extraction layers.

[0068] In step S302, the feature data extracted by the preset first feature extraction layer in the multiple first feature extraction layers is determined as the intermediate layer output, and the feature data extracted by the last first feature extraction layer in the multiple first feature extraction layers is determined as the output layer output.

[0069] In this embodiment of the disclosure, during model training, after obtaining multiple audio segments corresponding to the training audio, the multiple audio segments are input into the initial model and the teacher model respectively. For the initial model, the filter bank (fbank) features of the multiple audio segments are extracted through the feature extraction layer in the model architecture. Then, the filter bank (fbank) features are extracted step by step through multiple first feature extraction layers in the model architecture to obtain the output layer features extracted by the last first feature extraction layer in the multiple first feature extraction layers, and the intermediate layer output extracted by the preset first feature extraction layer in the multiple first feature extraction layers is also obtained.

[0070] In an exemplary embodiment of this disclosure, the initial model architecture includes multiple first feature extraction layers and a filter bank fbank feature extraction layer [feat(fbank)extractor]. The multiple first feature extraction layers, arranged sequentially, include two bidirectional long short-term memory (B-LSTM) layers, two nonlinear fully connected layers (Feed Forward Network), and a linear fully connected layer serving as the output layer. The B-LSTM layer (64 nodes) is used to acquire a representation that integrates the temporal dimension information of the training audio. The two nonlinear fully connected layers (128 nodes) are used to acquire a representation that integrates all information of the training audio. The output of the output layer is either speaker change [speaker change(1)] or non-speaker change [no speaker change(0)]. The features acquired by the first B-LSTM layer in the initial model architecture are used as the intermediate layer output, and the output of the linear fully connected layer serving as the output layer is used as the output layer output.

[0071] In this embodiment, the teacher model includes multiple second feature extraction layers. Features at different levels of the training audio can be extracted through these multiple second feature extraction layers. The second weighted output in this disclosure is the feature obtained by combining the features acquired by the multiple second feature extraction layers. The following embodiments of this disclosure illustrate the method for obtaining the second weighted output of the teacher model.

[0072] Figure 5 This is a flowchart illustrating a method for obtaining a second weighted output of a teacher model according to an exemplary embodiment. Figure 5 As shown, the method includes steps S401 to S403.

[0073] In step S401, the audio features of the training audio are obtained, and the audio features are different from the filter bank features of the training audio.

[0074] In step S402, the audio features are extracted step by step through multiple second feature extraction layers, the output of each second feature extraction layer is obtained, and the weights corresponding to each first feature extraction layer are obtained. The output of each second feature extraction layer is a representation of the training audio at different granularities.

[0075] In step S403, for multiple second feature extraction layers, a weighted sum is performed based on the output of each second feature extraction layer and the weight corresponding to each first feature extraction layer, and the parameter obtained by the weighted sum is determined as the second weighted output.

[0076] In this embodiment, during model training, after obtaining multiple audio segments corresponding to the training audio, these audio segments are input into the initial model and the teacher model respectively. For the teacher model, audio features corresponding to the multiple audio segments are obtained through the feature extraction layer in the model architecture. Then, multiple second feature extraction layers in the teacher model architecture extract audio features with different granularities at each level, and simultaneously obtain the weights corresponding to each second feature extraction layer. An activation function maps all weights corresponding to the second feature extraction layers to values ​​between 0 and 1. Based on the output of each second feature extraction layer and the mapped weights of each first feature extraction layer, a weighted sum is performed to obtain a second weighted output. By obtaining the weights corresponding to each second feature extraction layer and the second weighted output obtained by weighted summing the weights and outputs of each feature extraction layer, the information from each layer of the pre-trained model is fully utilized during the training of the student model, and information adapted to the task is automatically selected, ensuring the model training effect.

[0077] In an exemplary embodiment of this disclosure, the teacher model includes a pretrained feature extractor for extracting low-order audio features and 12 pretrained encoders. The 12 pretrained encoders (or encoder layers) provide representations of speech at different granularities, and each encoder outputs a learnable weight. All weights are mapped to values ​​between 0 and 1 using a sigmoid function. The weighted sum of the outputs of the 12 encoder layers and their corresponding weights is then used as the output of the teacher model (the second weighted output, softmax output).

[0078] The following embodiments of this disclosure illustrate a method for obtaining a second loss based on the intermediate layer output and the second weighted output.

[0079] Figure 6 This is a flowchart illustrating a method for obtaining a second loss based on intermediate layer output and a second weighted output, according to an exemplary embodiment. Figure 6 As shown, the method includes steps S501 to S502.

[0080] In step S501, the intermediate layer output is processed by a preset adapter to obtain the first weighted output. The adapter includes a single-layer convolutional neural network and an activation function, and the shape of the adapter output corresponds to the shape of the teacher model output.

[0081] In step S502, the second loss is obtained based on the first weighted output, the second weighted output, and the preset second loss acquisition method.

[0082] In this embodiment, to enable the initial model to learn the knowledge of the teacher model during model training—that is, to allow the student model (actually the teacher model) to effectively teach the initial model (actually the student model)—this disclosure aligns the output of a certain layer in the initial model (intermediate layer output) with the output of a certain layer in the teacher model (second weighted output) in both time and feature dimensions. This is achieved by using a single-layer convolutional neural network (CNN) followed by an activation function (such as the hyperbolic tangent tanh activation function) as an adapter. The adapter's output is made to have a corresponding shape to the teacher model's output. This disclosure processes the intermediate layer output of the initial model through the adapter to obtain a first weighted output whose shape corresponds to the second weighted output. Then, based on the first weighted output, the second weighted output, and a preset second loss acquisition method, a second loss is obtained.

[0083] In this embodiment of the disclosure, model training is performed on an initial model based on knowledge distillation. The initial model is used as the student model, and the teacher model is used as the teacher model. The second loss, obtained from the first weighted output, the second weighted output, and a preset second loss acquisition method, is the loss from knowledge distillation. In one example, the loss function (second loss acquisition method) for knowledge distillation is shown in the following formula:

[0084]

[0085] Where, x i Corresponding to the training audio, t(x) i s(.) is the softmax output (second weighted output) of the teacher model, s(.) is the softmax output (first weighted output) of the adapter layer, X is the training sample set, β is a preset constant, and sg[.] is the stopping gradient operator, which is defined as the identity operator during forward computation and has a partial derivative of zero, so that this part of the parameters will not be updated during backpropagation. KL(.) is the KL divergence between distributions p and q. KL divergence is a function that measures the difference between two distributions. It measures the distance between two probability distributions. The closer the two distributions are, the smaller the KL divergence; the farther apart they are, the larger the KL divergence. KL divergence is defined by the following formula:

[0086]

[0087] That is, KL[sg <t(x i )>,s(xi )] is sg <t(x i )> and s(x i The KL divergence between ) and KL[t(x) i ), sg <s(x i )>] is equivalent to t(x i ) and sg <s(x i KL divergence between )>.

[0088] Since updating the SCD main network (initial model) has a higher priority than updating the weights (the weights corresponding to each second feature layer in the teacher model), the value of β should be less than 1. In one example, β is set to 0.25. This disclosure uses different learning rates for the weights and the SCD main network, such as setting the learning rate of the weights to 0.1 times the learning rate of the main network.

[0089] The following embodiments further illustrate the teacher model.

[0090] In one embodiment of this disclosure, during model training, the parameters of the teacher model remain unchanged, while the weights corresponding to each of the multiple second feature extraction layers included in the teacher model change as training progresses.

[0091] In this embodiment, model training for the initial model is performed using knowledge distillation. The initial model is used as the student model, and the teacher model is used as the teacher model. Since the teacher model is used as the teacher model, its parameters are not updated during training; only the weights multiplied on the output of each layer (e.g., w1 to wn) are learnable (changing as training progresses).

[0092] The following embodiments of this disclosure further illustrate the method for determining speaker transition points in speech audio.

[0093] Figure 7 This is a flowchart illustrating a method for determining speaker transition points in speech audio according to an exemplary embodiment. Figure 7 As shown, the method includes steps S601 to S602.

[0094] In step S601, the speech audio to be processed is processed by a preset student model to obtain the posterior probability result, which includes the posterior probability of each time point in the speech audio.

[0095] In step S602, for each time point, in response to the posterior probability being greater than or equal to the probability threshold, the time point in the speech audio corresponding to the posterior probability is determined as the speaker transition point.

[0096] In this embodiment, the posterior probability represents the probability that the speech audio at a corresponding time point belongs to a different speaker than the speech audio at the previous time point, i.e., the probability of a speaker change occurring at the corresponding time point in the speech audio. The preset posterior probability threshold is a critical value representing the probability of a speaker change occurring at the corresponding time point in the speech audio. Based on this, this disclosure obtains the posterior probability result by processing the speech audio to be processed using a preset student model. For each time point in the speech audio included in the posterior probability result, if the posterior probability is greater than or equal to the probability threshold, then the time point in the speech audio corresponding to the posterior probability is determined to be a speaker change point.

[0097] The following embodiments further illustrate the teacher model.

[0098] In one embodiment of this disclosure, the teacher model is a self-supervised pre-trained model.

[0099] Given the powerful audio representation capabilities of self-supervised pre-trained models, which can be widely applied to various downstream audio tasks, this disclosure uses a self-supervised pre-trained model as the teacher model. During model training based on knowledge distillation, the initial model learns the knowledge from the self-supervised pre-trained model, resulting in a student model that outperforms general change-point detection models in speaker change point detection tasks.

[0100] In an exemplary embodiment of this disclosure, such as Figure 8The diagram illustrates the architecture of the speaker change detection model. This model consists of two parts: a self-supervised pre-trained model (teacher model) and a main speaker change detection model (Main SCD Model, or SCD main network, initial model), also known as the student model. Speech data (audio, or audio sequence, training audio) is simultaneously fed into both the self-supervised pre-trained model and the main SCD model. The self-supervised pre-trained model first obtains a low-level representation of the speech through a pretrained feature extractor, and then obtains representations of different granularities of the training audio through 12 pretrained encoder layers. The output of each pretrained encoder layer is multiplied by a learnable weight. All weights are mapped to values ​​between 0 and 1 using a sigmoid function. The outputs of all pretrained encoder layers, combined with the mapped weights (values ​​between 0 and 1) for each pretrained encoder layer, are weighted and summed to serve as the output of the teacher model. The SCD speaker transition detection model is a very small network (0.18M parameters). After extracting Fbank features from the training audio, it passes through two B-LSTM layers (64 nodes) (bidirectional long short-term memory network layers 1 and 2) to obtain a representation that integrates temporal information. Then, it passes through two nonlinear fully connected layers (128 nodes) (feedforward layers FFN) to obtain a representation that integrates all information. Finally, a linear fully connected layer is added as the output layer. The output of the output layer is the speaker switching (1) and non-speaker switching (0). In addition, the output of a certain layer in the student model is aligned with the output of the teacher model in terms of time and feature dimensions. A single-layer convolutional neural network (CNN) followed by a tanh activation function is used as an adapter. The output of the adapter has the same shape as the output of the self-supervised pre-trained model. The knowledge distillation loss (KD Criterion) can be calculated based on the output of the adapter and the output of the self-supervised pre-trained model, thereby achieving the purpose of teaching knowledge to the speaker transition detection model. The model loss (KD Criterion) can be obtained from the output of the SCD speaker transition point detection model, thereby optimizing the speaker transition point detection model.

[0101] In this embodiment, after acquiring audio data including at least two different speakers, the audio data is processed by a preset student model to obtain the posterior probability of each time point in the audio data. The posterior probability represents the probability that the audio data at the corresponding time point belongs to a different speaker than the audio data at the previous time point. The student model is trained based on the initial model and the teacher model, and its model structure is different from that of the teacher model. Speaker transition points in the audio data are determined based on the posterior probabilities of each time point and a preset probability threshold. During the training of the student model, the training audio is processed by the initial model corresponding to the student model to obtain the output layer features of the last layer output in the initial model and the intermediate layer outputs of the preset layers in the initial model. The corresponding training data is then processed by the trained teacher model. A weighted sum is obtained by weighting the outputs of each layer of the teacher model and the weights corresponding to each layer. A first loss is obtained based on the output layer output of the initial model, and a second loss is obtained based on the intermediate layer outputs of the initial model and the weighted output of the teacher model. The sum of the first loss and the second loss is determined as the target loss. The initial model is trained using the teacher model until the target loss meets the preset conditions. Once the training is complete, the trained initial model is designated as the student model.

[0102] This disclosure describes a method to train an initial model using a teacher model to obtain a student model. The student model possesses reasoning capabilities equivalent to the teacher model, but in actual reasoning, only the student model's own model structure performs the reasoning without the teacher model's involvement. This improves the student model's detection accuracy when performing speaker change point detection tasks while limiting the computational resources consumed by the student model. Furthermore, during model training, weighted outputs are obtained by combining the outputs and corresponding weights of each level of the teacher model and participate in model training. This allows the student model to learn the parameters of each level in the teacher model, further enhancing its detection accuracy when performing speaker change point detection tasks.

[0103] Based on the same concept, this disclosure also provides a speaker change point detection device 100.

[0104] It is understood that the speaker change point detection device 100 provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0105] Figure 9 This is a block diagram illustrating a speaker change point detection device 100 according to an exemplary embodiment. (Refer to...) Figure 9 The device includes an acquisition unit 101, a processing unit 102, and a determination unit 103.

[0106] The acquisition unit 101 is used to acquire the speech audio to be processed, which includes the speech of at least two different speakers.

[0107] The processing unit 102 is used to process the speech audio to be processed through a preset student model to obtain the posterior probability result. The posterior probability result includes the posterior probability of each time point in the speech audio. The posterior probability represents the probability that the speech audio corresponding to the corresponding time point and the speech audio corresponding to the previous time point belong to different speakers. The student model is trained based on the initial model and the teacher model. The model structure of the initial model is different from the model structure of the teacher model.

[0108] The determining unit 103 is used to determine the speaker change point in the speech audio based on the posterior probability of each time node and a preset probability threshold. The speech audio corresponding to the speaker change point belongs to a different speaker than the speech audio corresponding to the previous time node.

[0109] In one implementation, the student model is trained by the processing unit 102 in the following manner: The initial model processes training audio to obtain the output layer output and intermediate layer output of the initial model; the teacher model processes the training audio to obtain the second weighted output of the teacher model. A first loss is obtained based on the output layer output, and a second loss is obtained based on the intermediate layer output and the second weighted output. The sum of the first and second losses is determined as the target loss. The initial model is trained until the target loss meets a preset condition, at which point the initial model is determined as the student model.

[0110] In one embodiment, the initial model includes multiple first feature extraction layers. The processing unit 102 processes the training audio through the initial model in the following manner to obtain the output layer output and intermediate layer output of the initial model: It acquires the filter bank (fbank) features of the training audio, and extracts features from the filter bank (fbank) features step-by-step through multiple first feature extraction layers. The feature data extracted by a preset first feature extraction layer among the multiple first feature extraction layers is determined as the intermediate layer output, and the feature data extracted by the last first feature extraction layer among the multiple first feature extraction layers is determined as the output layer output.

[0111] In one implementation, the teacher model includes multiple second feature extraction layers. The processing unit 102 processes the training audio through the teacher model in the following manner to obtain the second weighted output of the teacher model: acquiring the audio features of the training audio, which are distinct from the filter bank (fbank) features of the training audio.

[0112] Audio features are extracted step-by-step through multiple second feature extraction layers. The output of each second feature extraction layer and the weights corresponding to each first feature extraction layer within the second feature extraction layers are obtained. The output of each second feature extraction layer represents the training audio at different granularities. For each of the multiple second feature extraction layers, a weighted sum is performed based on the output of each second feature extraction layer and the weights corresponding to each first feature extraction layer. The parameter obtained from this weighted sum is determined as the second weighted output.

[0113] In one embodiment, the processing unit 102 obtains a second loss based on the intermediate layer output and the second weighted output in the following manner: processing the intermediate layer output through a preset adapter to obtain a first weighted output, wherein the adapter includes a single-layer convolutional neural network and an activation function, and the shape of the adapter output corresponds to the shape of the teacher model output. The second loss is then obtained based on the first weighted output, the second weighted output, and the preset second loss acquisition method.

[0114] In one implementation, the parameters of the teacher model remain unchanged during model training, while the weights of each of the multiple second feature extraction layers in the teacher model change as training progresses.

[0115] In one embodiment, the processing unit 102 determines the speaker transition point in the speech audio based on the posterior probability of each time point and a preset probability threshold as follows: for the posterior probability of each time point, in response to the posterior probability being greater than or equal to the probability threshold, the time point in the speech audio corresponding to the posterior probability is determined as the speaker transition point.

[0116] In one implementation, the teacher model is a self-supervised pre-trained model.

[0117] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0118] Figure 10 This is a block diagram illustrating an apparatus 200 for speaker change point detection according to an exemplary embodiment. The apparatus 200 can be provided as a terminal. For example, the apparatus 200 can be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0119] Reference Figure 10 The device 200 may include one or more of the following components: processing component 202, memory 204, power component 206, multimedia component 208, audio component 210, input / output (I / O) interface 212, sensor component 214, and communication component 216.

[0120] Processing component 202 typically controls the overall operation of device 200, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 202 may include one or more modules to facilitate interaction between processing component 202 and other components. For example, processing component 202 may include a multimedia module to facilitate interaction between multimedia component 208 and processing component 202.

[0121] Memory 204 is configured to store various types of data to support the operation of device 200. Examples of such data include instructions for any application or method operating on device 200, contact data, phonebook data, messages, pictures, videos, etc. Memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0122] The power supply component 206 provides power to the various components of the device 200. The power supply component 206 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 200.

[0123] Multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 208 includes a front-facing camera and / or a rear-facing camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0124] Audio component 210 is configured to output and / or input audio signals. For example, audio component 210 includes a microphone (MIC) configured to receive external audio signals when device 200 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 204 or transmitted via communication component 216. In some embodiments, audio component 210 also includes a speaker for outputting audio signals.

[0125] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0126] Sensor assembly 214 includes one or more sensors for providing status assessments of various aspects of device 200. For example, sensor assembly 214 may detect the on / off state of device 200, the relative positioning of components such as the display and keypad of device 200, changes in the position of device 200 or a component of device 200, the presence or absence of user contact with device 200, the orientation or acceleration / deceleration of device 200, and temperature changes of device 200. Sensor assembly 214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 214 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0127] Communication component 216 is configured to facilitate wired or wireless communication between device 200 and other devices. Device 200 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 216 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0128] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0129] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, which can be executed by a processor 220 of the device 200 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0130] Figure 11 This is a block diagram illustrating an apparatus 300 for speaker change point detection according to an exemplary embodiment. For example, apparatus 300 may be provided as a server. (Refer to...) Figure 11 The apparatus 300 includes a processing component 322, which further includes one or more processors, and memory resources represented by memory 332 for storing instructions, such as application programs, that can be executed by the processing component 322. The application programs stored in memory 332 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 322 is configured to execute instructions to perform the aforementioned speaker change point detection method.

[0131] Device 300 may also include a power supply component 326 configured to perform power management of device 300, a wired or wireless network interface 350 configured to connect device 300 to a network, and an input / output (I / O) interface 358. Device 300 may operate on an operating system stored in memory 332, such as Windows Server™, MacOSX™, Unix™, Linux™, FreeBSD™, or similar.

[0132] It is understood that in this disclosure, "multiple" refers to two or more, and other quantifiers are similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.

[0133] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0134] It is further understood that the terms “center,” “longitudinal,” “lateral,” “front,” “rear,” “up,” “down,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” and “outer,” etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this embodiment and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation.

[0135] It can be further understood that, unless otherwise specified, "connection" includes both direct connections where no other components exist between the two parties and indirect connections where other components exist between them.

[0136] It is further understood that although operations are described in a specific order in the accompanying drawings in the embodiments of this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0137] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0138] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for detecting speaker transition points, characterized in that, include: Acquire the audio to be processed, wherein the audio includes the voices of at least two different speakers; The speech audio to be processed is processed by a preset student model to obtain a posterior probability result. The posterior probability result includes the posterior probability of each time point in the speech audio. The posterior probability represents the probability that the speech audio at the corresponding time point and the speech audio at the previous time point belong to different speakers. The student model is trained based on an initial model and a teacher model. The model structure of the initial model is different from that of the teacher model. Based on the posterior probability of each time point and a preset probability threshold, the speaker change point in the speech audio is determined. The speech audio corresponding to the speaker change point belongs to a different speaker than the speech audio corresponding to the previous time point.

2. The method according to claim 1, characterized in that, The student model was trained in the following manner: The initial model processes the training audio to obtain the output layer output and intermediate layer output of the initial model, and the teacher model processes the training audio to obtain the second weighted output of the teacher model. The first loss is obtained based on the output of the output layer, and the second loss is obtained based on the output of the intermediate layer and the second weighted output. The sum of the first loss and the second loss is determined as the target loss; The initial model is trained until the target loss meets the preset conditions, and then the initial model is determined as the student model.

3. The method according to claim 2, characterized in that, The initial model includes multiple first feature extraction layers; The process of processing the training audio using the initial model to obtain the output layer output and intermediate layer output of the initial model includes: The filter bank (fbank) features of the training audio are obtained, and the filter bank (fbank) features are extracted step by step through the multiple first feature extraction layers. The feature data extracted by the preset first feature extraction layer in the plurality of first feature extraction layers is determined as the output of the intermediate layer, and the feature data extracted by the last first feature extraction layer in the plurality of first feature extraction layers is determined as the output of the output layer.

4. The method according to claim 2, characterized in that, The teacher model includes multiple second feature extraction layers; The step of processing the training audio through a teacher model to obtain the second weighted output of the teacher model includes: Obtain the audio features of the training audio, which are distinct from the filter bank (fbank) features of the training audio; The audio features are extracted step by step through multiple second feature extraction layers. The output of each second feature extraction layer is obtained, and the weight corresponding to each first feature extraction layer is obtained. The output of each second feature extraction layer is a different granularity representation of the training audio. For the plurality of second feature extraction layers, a weighted sum is performed based on the output of each second feature extraction layer and the weight corresponding to each first feature extraction layer, and the parameter obtained by the weighted sum is determined as the second weighted output.

5. The method according to claim 2, characterized in that, The step of obtaining the second loss based on the intermediate layer output and the second weighted output includes: The intermediate layer output is processed by a preset adapter to obtain a first weighted output. The adapter includes a single-layer convolutional neural network and an activation function, and the shape of the adapter output corresponds to the shape of the teacher model output. The second loss is obtained based on the first weighted output, the second weighted output, and the preset second loss acquisition method.

6. The method according to claim 4, characterized in that, During model training, the parameters of the teacher model remain unchanged, while the weights of each of the multiple second feature extraction layers in the teacher model change as training progresses.

7. The method according to claim 4, characterized in that, Determining the speaker transition points in the speech audio based on the posterior probability of each time point and a preset probability threshold includes: For each time point, in response to a posterior probability greater than or equal to the probability threshold, the time point in the speech audio corresponding to the posterior probability is determined as the speaker transition point.

8. The method according to any one of claims 1, 2, 4, 5 and 6, characterized in that, The teacher model is a self-supervised pre-trained model.

9. A speaker change point detection device, characterized in that, include: An acquisition unit is used to acquire the speech audio to be processed, wherein the speech audio includes the voices of at least two different speakers; The processing unit is used to process the speech audio to be processed through a preset student model to obtain a posterior probability result. The posterior probability result includes the posterior probability of each time point in the speech audio. The posterior probability represents the probability that the speech audio at the corresponding time point and the speech audio at the previous time point belong to different speakers. The student model is trained based on an initial model and a teacher model. The model structure of the initial model is different from that of the teacher model. The determining unit is used to determine the speaker change point in the speech audio based on the posterior probability of each time point and a preset probability threshold. The speech audio corresponding to the speaker change point belongs to a different speaker than the speech audio corresponding to the previous time point.

10. The apparatus according to claim 9, characterized in that, The student model is trained by the processing unit in the following manner: The initial model processes the training audio to obtain the output layer output and intermediate layer output of the initial model, and the teacher model processes the training audio to obtain the second weighted output of the teacher model. The first loss is obtained based on the output of the output layer, and the second loss is obtained based on the output of the intermediate layer and the second weighted output. The sum of the first loss and the second loss is determined as the target loss; The initial model is trained until the target loss meets the preset conditions, and then the initial model is determined as the student model.

11. The apparatus according to claim 10, characterized in that, The initial model includes multiple first feature extraction layers; The processing unit processes the training audio through the initial model in the following manner to obtain the output layer output and intermediate layer output of the initial model: The filter bank (fbank) features of the training audio are obtained, and the filter bank (fbank) features are extracted step by step through the multiple first feature extraction layers. The feature data extracted by the preset first feature extraction layer in the plurality of first feature extraction layers is determined as the output of the intermediate layer, and the feature data extracted by the last first feature extraction layer in the plurality of first feature extraction layers is determined as the output of the output layer.

12. The apparatus according to claim 10, characterized in that, The teacher model includes multiple second feature extraction layers; The processing unit processes the training audio through the teacher model in the following manner to obtain the second weighted output of the teacher model: Obtain the audio features of the training audio, which are distinct from the filter bank (fbank) features of the training audio; The audio features are extracted step by step through multiple second feature extraction layers. The output of each second feature extraction layer is obtained, and the weight corresponding to each first feature extraction layer is obtained. The output of each second feature extraction layer is a different granularity representation of the training audio. For the plurality of second feature extraction layers, a weighted sum is performed based on the output of each second feature extraction layer and the weight corresponding to each first feature extraction layer, and the parameter obtained by the weighted sum is determined as the second weighted output.

13. The apparatus according to claim 10, characterized in that, The processing unit obtains the second loss based on the intermediate layer output and the second weighted output in the following manner: The intermediate layer output is processed by a preset adapter to obtain a first weighted output. The adapter includes a single-layer convolutional neural network and an activation function, and the shape of the adapter output corresponds to the shape of the teacher model output. The second loss is obtained based on the first weighted output, the second weighted output, and the preset second loss acquisition method.

14. The apparatus according to claim 12, characterized in that, During model training, the parameters of the teacher model remain unchanged, while the weights of each of the multiple second feature extraction layers in the teacher model change as training progresses.

15. The apparatus according to claim 12, characterized in that, The processing unit determines the speaker transition points in the speech audio based on the posterior probability of each time point and a preset probability threshold in the following manner: For each time point, in response to a posterior probability greater than or equal to the probability threshold, the time point in the speech audio corresponding to the posterior probability is determined as the speaker transition point.

16. The apparatus according to any one of claims 9, 10, 12, 13, and 14, characterized in that, The teacher model is a self-supervised pre-trained model.

17. An electronic device, characterized in that, include: processor: Memory used to store processor-executable instructions; The processor is configured to execute the speaker change point detection method according to any one of claims 1 to 8.

18. A storage medium, characterized in that, The storage medium stores instructions that, when executed by a processor, enable the processor to perform the speaker change point detection method according to any one of claims 1 to 8.

Images