Speech segment model training method, speech segment method, device and electronic equipment

Abstract

The application provides a speaker segmentation model training method, a speaker segmentation method, a device and an electronic device. The speaker segmentation model training method comprises: inputting an audio feature into a to-be-trained model for iterative training; wherein, in the process of the Nth iteration training, the to-be-trained model performs feature mapping on the audio feature to obtain a first feature vector, and then performs attention mechanism fitting on the first feature vector to obtain a second feature vector, N being a positive integer; cross-entropy loss values and clustering loss values are obtained respectively according to the second feature vector; and in the case that the cross-entropy loss values and the clustering loss values satisfy a preset condition, the to-be-trained model after the Nth iteration training is determined as a speaker segmentation model. In this way, the accuracy of the speaker segmentation result can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of voiceprint recognition, and particularly relates to a segmentation model training method, a speaker segmentation method, a device and an electronic device. BACKGROUND

[0002] With the progress of voiceprint recognition technology, voiceprint recognition is more and more widely used in people's life. Voiceprint recognition can be applied to the scene of speaker segmentation. In the actual application process, the current voice is usually processed by using a voice activity detection technology (VAD) to obtain VAD data, then the VAD data is subjected to event detection, and the VAD data and the event detection result are combined to obtain the speaker segmentation result. However, the accuracy of the speaker segmentation result obtained by this method is low. SUMMARY

[0003] The embodiments of the present application provide a segmentation model training method, a speaker segmentation method, a device and an electronic device to solve the problem of low accuracy of the speaker segmentation result.

[0004] To solve the above technical problems, the present application is implemented as follows:

[0005] In a first aspect, the embodiments of the present application provide a speaker segmentation model training method, comprising:

[0006] inputting an audio feature into a to-be-trained model for iterative training; wherein in the process of the Nth iteration training, after the to-be-trained model performs feature mapping on the audio feature to obtain a first feature vector, the to-be-trained model performs attention mechanism fitting on the first feature vector to obtain a second feature vector, N is a positive integer;

[0007] determining a cross-entropy loss value and a clustering loss value according to the second feature vector, respectively;

[0008] in the case where the cross-entropy loss value and the clustering loss value meet a preset condition, determining the to-be-trained model after the Nth iteration training as a speaker segmentation model.

[0009] As can be seen, in the embodiment, after the audio feature is sequentially subjected to feature mapping and attention mechanism fitting, the expression performance of the feature can be enhanced, so that the first feature vector and the second feature vector obtained can be more accurately recognized, and at the same time, the relationship between the speaker information included in the feature can be captured, so that the accuracy of the cross-entropy loss value and the clustering loss value obtained finally is high, and thus the accuracy of the speaker segmentation result of the speaker segmentation model obtained finally is high.

[0010] In a second aspect, an embodiment of the present application provides a speaker segmentation method, the method comprising:

[0011] obtaining an audio feature;

[0012] inputting the audio feature into a speaker segmentation model to output a target feature vector; wherein the speaker segmentation model comprises a first linear layer, a first encoding layer, and a target network; the first linear layer is configured to perform feature mapping on the input audio feature to output a first feature vector; the first encoding layer is configured to perform attention mechanism fitting on the input first feature vector to output a second feature vector; and the target network layer is configured to process the input second feature vector to output the target feature vector;

[0013] determining a speaker segmentation result according to the target feature vector.

[0014] As can be seen, in the embodiment, after the audio feature passes through the speaker segmentation model, feature mapping and attention mechanism fitting are sequentially performed to obtain the target feature vector, and the speaker segmentation result is determined according to the target feature vector. In this way, through feature mapping and attention mechanism fitting, the expression performance of the feature can be enhanced, so that each vector obtained can be more accurately recognized, and at the same time, the relationship between the speaker information included in the feature can be captured, so that the accuracy of the obtained speaker segmentation result is higher.

[0015] In a third aspect, an embodiment of the present application further provides a speaker segmentation model training device, comprising:

[0016] a training module configured to input an audio feature into a to-be-trained model for iterative training; wherein in the process of the Nth iteration training, after the to-be-trained model performs feature mapping on the audio feature to obtain a first feature vector, the to-be-trained model performs attention mechanism fitting on the first feature vector to obtain a second feature vector, N being a positive integer;

[0017] a first determination module configured to determine a cross-entropy loss value and a clustering loss value according to the second feature vector, respectively;

[0018] a second determination module configured to determine the to-be-trained model after the Nth iteration training as a speaker segmentation model in a case where the cross-entropy loss value and the clustering loss value satisfy a preset condition.

[0019] In a fourth aspect, an embodiment of the present application further provides a speaker segmentation device, comprising:

[0020] a obtaining module configured to obtain an audio feature;

[0021] The vector acquisition module is configured to input the audio feature into a speaker segmentation model to output a target feature vector; wherein the speaker segmentation model comprises a first linear layer, a first encoding layer, and a target network; the first linear layer is configured to perform feature mapping on the input audio feature to output a first feature vector; the first encoding layer is configured to perform attention mechanism fitting on the input first feature vector to output a second feature vector; and the target network layer is configured to process the input second feature vector to output the target feature vector.

[0022] The determination module is configured to determine a speaker segmentation result according to the target feature vector.

[0023] In a fifth aspect, an embodiment of the present application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the speaker segmentation model training method or the speaker segmentation method when executing the computer program.

[0024] In a sixth aspect, an embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executable on a processor to implement the steps in the speaker segmentation model training method or the speaker segmentation method. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0026] Figure 1 is a flowchart of a speaker segmentation model training method provided by an embodiment of the present application;

[0027] Figure 2 is a structural schematic diagram of a to-be-trained model provided by an embodiment of the present application;

[0028] Figure 3 is a flowchart of a speaker segmentation method provided by an embodiment of the present application;

[0029] Figure 4 is a structural schematic diagram of a speaker segmentation model training device provided by an embodiment of the present application;

[0030] Figure 5 is a structural schematic diagram of a speaker segmentation device provided by an embodiment of the present application;

[0031] Figure 6 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0033] Referring to Figure 1 , Figure 1 is a flowchart of a speaker segmentation model training method provided by an embodiment of the present application, as shown in Figure 1 , comprising the following steps:

[0034] Step 101, input an audio feature to a to-be-trained model for iterative training; wherein in the process of the Nth iterative training, after the to-be-trained model performs feature mapping on the audio feature to obtain a first feature vector, the to-be-trained model performs attention mechanism fitting on the first feature vector to obtain a second feature vector, and N is a positive integer.

[0035] The audio feature can be a 23-dimensional Mel-frequency cepstral coefficient (MFCC) audio feature.

[0036] In the present embodiment, since a logarithmic mapping operation method is used when the MFCC audio feature is extracted, the extraction effect is more inclined to the effect of human ear hearing, and therefore the effect in speaker segmentation will be better. In addition, using the 23-dimensional MFCC audio feature can effectively reduce the overfitting of the finally obtained speaker segmentation model.

[0037] Of course, the audio feature can also be a 40-dimensional MFCC audio feature or an 80-dimensional MFCC audio feature.

[0038] The audio feature can be a pre-existing feature in a sample pool, and each audio feature can include labeling information; the labeling information is used to indicate the speaker to which the audio feature belongs, or the labeling information is used to indicate the speaker to which a time period in the audio feature belongs.

[0039] Of course, the audio feature can be an audio feature extracted from an audio file. As an optional implementation, before the audio feature is input to the to-be-trained model for iterative training, the method further comprises:

[0040] Obtain L to-be-trained audio files, and L is a positive integer.

[0041] generating a speech index of the audio file to be trained;

[0042] determining a target audio file according to the speech index, the target audio file being an audio file in the L audio files to be trained;

[0043] extracting a feature of the target audio file to obtain the audio feature.

[0044] The audio file to be trained can be an audio file in a sample pool. The content of the audio file to be trained is not specifically limited herein. For example, the audio file to be trained can be an audio file of a conversation between a customer service and a user, or an audio file including a customer service, a user, and noise, where the noise can refer to at least one of other customer services and environmental noise.

[0045] Each audio file to be trained has at least one label, and the speech index of each audio file to be trained includes the at least one label of the audio file to be trained. In this way, the corresponding target audio file can be determined by searching for the label included in the speech index. The target audio file can be an audio file in the audio files to be trained, and the target audio file can be determined as a file that needs to be extracted and input to the audio model to be trained for iterative training.

[0046] The label can be, for example, an audio identity document (ID), a speech channel id, a speech segment start time, a time increment, a speaker id, or the like.

[0047] In the embodiment, the audio file to be trained can be classified or stored in a region in the sample pool. The target audio file can be searched for according to the speech index of the audio file to be trained, and the feature of the target audio file can be extracted to obtain the audio feature. In this way, the speech index can improve the searching efficiency of the target audio file, and further improve the extraction efficiency of the audio feature.

[0048] It should be noted that the storage format of the audio file to be trained in the sample pool is not limited herein. The storage format can be, for example, a speech wav file format.

[0049] The format of the speech index is not limited herein. As an optional embodiment, the format of the speech index can be, for example, a kaldi format.

[0050] The specific structure of the speech index in the Kaldi format can be seen from the following table: the first column of the speech index is an audio id (identifier), the second column is a speech channel id (the speech channel refers to a single channel, a double channel or a multi-channel, and the speech channel id can refer to an identifier of each speech channel), the third column is a speech segment start time (which can be understood as the start time of the speech segment), the fourth column is a time increment (which can be understood as the length of the speech segment, and the unit can be seconds), and the fifth column is a speaker id (which can be understood as an identifier of a speaker to which the speech segment belongs).

[0051] Of course, the structure of the speech index in the Kaldi format is not limited to the above manner, and the above manner is only an exemplary description. For example, the content of the fourth column can be replaced by the end time of the speech segment on the basis of the above manner.

[0052] In this embodiment, the format of the speech index is the Kaldi format, which can make the efficiency of searching for a target audio file higher, and the reliability of the speech index in the Kaldi format is better.

[0053] As another optional embodiment, the format of the speech index can also be the espnet format or the librosa format.

[0054] It should be noted that the specific structure of the to-be-trained model is not limited herein.

[0055] In step 102, cross-entropy loss values and clustering loss values are respectively determined according to the second feature vectors.

[0056] As an optional embodiment, referring to Figure 2 , the to-be-trained model comprises a first linear layer, a first encoding layer and a target network layer connected in sequence. It should be noted that the connection mode of the first linear layer, the first encoding layer and the target network layer included in the to-be-trained model is not limited herein. As another optional embodiment, the connection mode of the first linear layer, the first encoding layer and the target network layer can also be understood as the following mode: the first encoding layer is connected with the first linear layer, the first encoding layer is connected with the cross-entropy loss network through the target network layer, and the first encoding layer is connected with the supervised clustering loss network.

[0057] The first linear layer is configured to perform feature mapping on the input audio features and output the first feature vectors; the first encoding layer is configured to perform attention mechanism fitting on the input first feature vectors and output the second feature vectors; and the target network layer is configured to process the input second feature vectors and output target feature vectors.

[0058] The determining the cross-entropy loss value and the clustering loss value according to the second feature vector respectively comprises: determining the cross-entropy loss value through a cross-entropy loss function and the target feature vector; and determining the clustering loss value through a supervised clustering loss and the second feature vector.

[0059] Wherein, the existing speaker segmentation network cannot accurately capture the speaker information relationship between the speech sequences corresponding to each feature, and the expression performance of the feature is poor, so that the segmentation accuracy is poor when performing segmentation. The speaker segmentation network of the present application is different from the existing speaker segmentation network. The existing speaker segmentation network does not have a combination of linear layers and transformers, and the speaker segmentation network of the present application combines linear layers and transformers. In the present application, the first linear layer is used to map the features of the audio features, thereby enhancing the expression performance of the audio features. In addition, the first encoding layer connected to the first linear layer can use a transformer encoding layer. The attention structure of the transformer encoding layer can effectively capture the speaker information relationship between the speech sequences in the audio features, and the composed network is efficient. This can more accurately identify each speaker information of the audio features, thereby improving the segmentation accuracy of the segmentation result. In addition, the supervised clustering loss network included in the present application can also perform automatic clustering of speakers in a supervised manner, thereby reducing the loss of clustering results and labeled information, thereby more facilitating the judgment of whether the model converges and improving the accuracy of the result of the model judgment whether to converge.

[0060] It should be noted that the first linear layer can map the features of the audio features, thereby enhancing the feature expression performance of the first feature vector. The first encoding layer can fit the attention mechanism to the first feature vector, and the attention mechanism fitting can include feature mapping and dynamic weighting attention operations, thereby obtaining the second feature vector. The supervised clustering loss network can automatically cluster the second feature vector under the supervision of the label, thereby reducing the loss of the clustering result and the label, i.e. reducing the clustering loss value. The cross-entropy loss network can reduce the cross-entropy loss value of the second feature vector, i.e. the cross-entropy loss network can reduce the cross-entropy loss value in the form of minimizing the information entropy, so as to automatically determine that the output result belongs to a certain speaker, reduce the gap between the output result and the cross-entropy loss function, and improve the accuracy of the output result.

[0061] It should be noted that the supervised clustering loss network can use a self-supervised clustering loss network, and the self-supervised clustering is to automatically cluster the speakers in a supervised manner to reduce the loss of the clustering result and the label.

[0062] In the embodiment, the first encoding layer can effectively capture the speaker information relationship between the speech sequences in the audio features, so that the accuracy of the speaker segmentation result output by the speaker segmentation model is high.

[0063] The specific structure of the target network layer is not limited herein, for example, the target network layer can be a transfer layer for transferring vectors.

[0064] As an optional embodiment, the target network layer comprises M second encoding layers and M second linear layers, and M is a positive integer.

[0065] If M is greater than 1, one second linear layer is arranged between any two adjacent second encoding layers.

[0066] If M is equal to 1, the output of the second encoding layer is the input of the second linear layer.

[0067] The second encoding layer can perform the same operation as the first encoding layer, and the second linear layer can perform the same operation as the first linear layer.

[0068] The second encoding layer can perform the same operation as the first encoding layer, and the second linear layer can perform the same operation as the first linear layer. Figure 2 The structure connected with the first encoding layer can be the second encoding layer, and the structure connected with the cross-entropy loss network can be the second linear layer.

[0069] In the embodiment, the target network layer comprises M second encoding layers and M second linear layers, so that the vectors can be further subjected to feature mapping and attention mechanism fitting, thereby further improving the accuracy of the final cross-entropy loss value.

[0070] As an optional embodiment, M is equal to 1, and the second encoding layer is configured to perform attention mechanism fitting on the input second feature vector and output a third feature vector.

[0071] The second linear layer is configured to perform feature mapping on the input third feature vector and output the target feature vector.

[0072] In the embodiment, the target network layer comprises the second encoding layer and the second linear layer, so that the vectors can be further subjected to feature mapping and attention mechanism fitting through the second encoding layer and the second linear layer, thereby further improving the accuracy of the final cross-entropy loss value. In addition, the target network layer only comprises one second encoding layer and one second linear layer, thereby reducing the number of network layers of the target network layer and saving the computing resources of the target network layer.

[0073] In a case where the cross-entropy loss value and the clustering loss value satisfy a preset condition, the trained model after the Nth iteration training is determined as the speaker segmentation model.

[0074] As an optional implementation, the preset condition comprises that a sum of the cross-entropy loss value and the clustering loss value is less than a first preset value.

[0075] It can also be understood that, in a case where the sum of the cross-entropy loss value and the clustering loss value is less than the first preset value, the trained model after the Nth iteration training is determined as the speaker segmentation model.

[0076] In this embodiment, when the sum of the cross-entropy loss value and the clustering loss value is less than the first preset value, it indicates that the loss has dropped to less than the first preset value and the loss has converged, and at this time, the trained model after the Nth iteration training can be determined as the speaker segmentation model, so that the accuracy of the speaker segmentation result of the final obtained speaker segmentation model is higher.

[0077] As another optional implementation, the preset condition comprises that a weighted sum of the cross-entropy loss value and the clustering loss value is less than a second preset value.

[0078] It can also be understood that, in a case where the weighted sum of the cross-entropy loss value and the clustering loss value is less than the second preset value, the trained model after the Nth iteration training is determined as the speaker segmentation model.

[0079] The weight of the weighting can be determined according to the influence degree of the cross-entropy loss value and the clustering loss value on the speaker segmentation result, for example, when the influence degree of the cross-entropy loss value is greater, the corresponding weight is greater, and vice versa, the corresponding weight is smaller.

[0080] In this embodiment, the accuracy of the speaker segmentation result of the final obtained speaker segmentation model is also higher, and the size of the adjusted weight can be adjusted, so that the obtained speaker segmentation model can be more in line with the actual scene requirements. In addition, the flexibility of the training method of the speaker segmentation model is also increased.

[0081] In the embodiment of the application, after the feature mapping and the attention mechanism fitting are sequentially performed on the audio features, the accuracy of the final obtained cross-entropy loss value and clustering loss value is higher, so that the accuracy of the speaker segmentation result of the final obtained speaker segmentation model is higher.

[0082] Referring to Figure 3 , Figure 3is a flowchart of a speaker segmentation method provided by an embodiment of the present application. The method is applied to a speaker segmentation model. Optionally, the speaker segmentation model can be trained according to the speaker segmentation model training method described above, as shown in Figure 3 The speaker segmentation method includes the following steps.

[0083] Step 301: Obtain audio features.

[0084] The audio features can refer to the related descriptions in the above embodiments, and will not be described here again.

[0085] Step 302: Input the audio features into a speaker segmentation model to output a target feature vector. The speaker segmentation model includes a first linear layer, a first encoding layer, and a target network. The first linear layer is configured to perform feature mapping on the input audio features to output a first feature vector. The first encoding layer is configured to perform attention mechanism fitting on the input first feature vector to output a second feature vector. The target network layer is configured to process the input second feature vector to output the target feature vector.

[0086] Step 303: Determine a speaker segmentation result according to the target feature vector.

[0087] The target feature vector can also be referred to as a segmented voice or a voiceprint segmentation segment. The voiceprint segmentation segment can be segmented according to a user. For example, all voices of user A included in the audio features are segmented to obtain a plurality of first target feature vectors, and the plurality of first target feature vectors are used to determine the speaker segmentation result of user A. Similarly, all voices of user B included in the audio features are segmented to obtain a plurality of second target feature vectors, and the plurality of second target feature vectors are used to determine the speaker segmentation result of user B. The identities of user A and user B are not limited here. For example, user A can be a customer service, and user B can be a customer.

[0088] In the embodiment of the present application, after the audio features pass through the speaker segmentation model, feature mapping and attention mechanism fitting are performed in sequence, and segmented voices are output. In this way, the feature mapping and attention mechanism fitting can enhance the expression performance of the features, so that the obtained vectors can be more accurately recognized, and the relationship between the speaker information included in the features can also be captured, so that the accuracy of the speaker segmentation result is higher.

[0089] As an optional implementation, the speaker segmentation model is connected with a voiceprint recognition model. After the speaker segmentation result is determined according to the target feature vector, the method further includes:

[0090] input the speaker segmentation result into the voiceprint recognition model, and output voiceprint recognition information.

[0091] The voiceprint recognition information can be identification information, for example, when the audio features include two persons, the identification information of one of the persons is 0, and the identification information of the other person is 1, the voiceprint recognition information can be 0 or 1.

[0092] Of course, the specific type of voiceprint recognition information is not limited here, for example, the voiceprint recognition information can also be the probability of the speaker to which the audio features belong.

[0093] Since the voiceprint recognition information has corresponding identity features in the voiceprint library, the voiceprint recognition information can be searched in the voiceprint database to confirm the identity of the user corresponding to the identification information, thereby completing the identification of the user of the audio features.

[0094] It should be noted that the voiceprint recognition model in the embodiment can also be a model running a softmax algorithm, and of course, the voiceprint recognition model can also be a softmax classifier.

[0095] As an optional implementation, the target network includes a second encoding layer and a second linear layer connected in sequence.

[0096] The second encoding layer is configured to perform attention mechanism fitting on the input second feature vector, and output a third feature vector.

[0097] The second linear layer is configured to perform feature mapping on the input third feature vector, and output the target feature vector.

[0098] In the embodiment, since the target network layer includes the second encoding layer and the second linear layer connected in sequence, the vector can be further feature mapped and attention mechanism fitted through the second encoding layer and the second linear layer, thereby further improving the accuracy of the target feature vector obtained finally. In addition, since the target network layer only includes one second encoding layer and one second linear layer, the number of network layers of the target network layer is reduced, and the computing resources of the target network layer are saved.

[0099] In the embodiment, through steps 301 to 303, on the basis of determining the speaker segmentation result in the speaker segmentation model, the voiceprint recognition model can identify the speaker segmentation result to obtain voiceprint recognition information. In this way, through the cooperative operation of the speaker segmentation model and the voiceprint recognition model, the efficiency of segmented speech recognition is improved, and the accuracy of the voiceprint recognition information obtained finally is high.

[0100] Referring to Figure 4 , Figure 4is a structural diagram of a speaker segmentation model training device provided by an embodiment of the present application, which can realize the details of the speaker segmentation model training method in the above embodiments and achieve the same effect. As shown in Figure 4 The speaker segmentation model training device 400 comprises:

[0101] The training module 401 is configured to input the audio features into a to-be-trained model for iterative training. In the process of the Nth iterative training, the to-be-trained model performs feature mapping on the audio features to obtain a first feature vector, and then performs attention mechanism fitting on the first feature vector to obtain a second feature vector, where N is a positive integer.

[0102] The first determination module 402 is configured to determine a cross-entropy loss value and a clustering loss value based on the second feature vector.

[0103] The second determination module 403 is configured to determine the to-be-trained model after the Nth iterative training as a speaker segmentation model when the cross-entropy loss value and the clustering loss value meet a preset condition.

[0104] Optionally, the to-be-trained model comprises a first linear layer, a first encoding layer and a target network layer connected in sequence.

[0105] The first linear layer is configured to perform feature mapping on the input audio features and output the first feature vector. The first encoding layer is configured to perform attention mechanism fitting on the input first feature vector and output the second feature vector. The target network layer is configured to process the input second feature vector and output a target feature vector.

[0106] The second determination module 403 is further configured to determine the cross-entropy loss value through a cross-entropy loss function and the target feature vector, and determine the clustering loss value through a supervised clustering loss and the second feature vector.

[0107] Optionally, the target network layer comprises M layers of second encoding layers and M layers of second linear layers, where M is a positive integer.

[0108] If M is greater than 1, one layer of second linear layers is arranged between any two adjacent layers of second encoding layers.

[0109] If M is equal to 1, the output of the second encoding layer is the input of the second linear layer.

[0110] Optionally, M is equal to 1, and the second encoding layer is configured to perform attention mechanism fitting on the input second feature vector and output a third feature vector.

[0111] The second linear layer is configured to perform feature mapping on the input third feature vector to output the target feature vector.

[0112] Optionally, the preset condition comprises:

[0113] The sum of the cross-entropy loss value and the clustering loss value is less than a first preset value.

[0114] Alternatively, the weighted sum of the cross-entropy loss value and the clustering loss value is less than a second preset value.

[0115] Optionally, the speaker segmentation model training apparatus 400 further comprises:

[0116] The acquisition module is configured to acquire L audio files to be trained;

[0117] The generation module is configured to generate a speech index of the audio file to be trained;

[0118] The third determination module is configured to determine a target audio file according to the speech index, the target audio file being an audio file in the L audio files to be trained;

[0119] The extraction module is configured to extract a feature of the target audio file to obtain the audio feature;

[0120] wherein L is a positive integer.

[0121] Optionally, the speech index is a speech index in kaldi format.

[0122] Optionally, the audio feature is a 23-dimensional MFCC audio feature.

[0123] The speaker segmentation model training apparatus provided by the embodiments of the present application can realize Figure 1 The processes implemented by the speaker segmentation model training apparatus in the method embodiments are not repeated here to avoid repetition.

[0124] In the present embodiment, since the audio feature is sequentially subjected to feature mapping and attention mechanism fitting, the expression performance of the feature can be enhanced, so that the first feature vector and the second feature vector obtained can be more accurately recognized, and the relationship between the speaker information included in the feature can also be captured, so that the accuracy of the cross-entropy loss value and the clustering loss value obtained finally is relatively high, and thus the accuracy of the speaker segmentation result of the speaker segmentation model obtained finally is relatively high.

[0125] Referring to Figure 5 , Figure 5A structural schematic diagram of a speaker segmentation device provided in an embodiment of the present application is provided. The speaker segmentation device in the embodiment of the present application is applied to a speaker segmentation model, which is trained according to the speaker segmentation model training method described above, as shown in FIG. 5. The speaker segmentation device 500 includes: Figure 5

[0126] The acquisition module 501 is configured to acquire audio features.

[0127] The vector acquisition module 502 is configured to input the audio features into a speaker segmentation model to output a target feature vector. The speaker segmentation model includes a first linear layer, a first encoding layer, and a target network. The first linear layer is configured to perform feature mapping on the input audio features to output a first feature vector. The first encoding layer is configured to perform attention mechanism fitting on the input first feature vector to output a second feature vector. The target network layer is configured to process the input second feature vector to output the target feature vector.

[0128] The determination module 503 is configured to determine a speaker segmentation result according to the target feature vector.

[0129] Optionally, the target network includes a second encoding layer and a second linear layer connected in sequence.

[0130] The second encoding layer is configured to perform attention mechanism fitting on the input second feature vector to output a third feature vector.

[0131] The second linear layer is configured to perform feature mapping on the input third feature vector to output the target feature vector.

[0132] The speaker segmentation device provided in the embodiment of the present application can implement the method embodiment. Figure 3 The speaker segmentation device implements various processes in the method embodiment, and thus repeated descriptions are not provided herein.

[0133] In the embodiment, after the audio features pass through the speaker segmentation model, feature mapping and attention mechanism fitting are performed in sequence, and segmented speech is output. In this way, the expression performance of the features can be enhanced through feature mapping and attention mechanism fitting, so that the obtained vectors can be more accurately recognized, and the relationship between the speaker information included in the features can also be captured, so that the accuracy of the segmentation result of the obtained segmented speech is higher.

[0134] Figure 6 A hardware structural schematic diagram of an electronic device for implementing various embodiments of the present application is provided.

[0135] ​The electronic device 600 includes, but is not limited to, a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, a processor 610, and a power supply 611, and the like. Those skilled in the art can understand that Figure 6 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than the figure, or combine certain components, or different component arrangements. In the embodiments of the present application, the electronic device includes, but is not limited to, a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle terminal, a wearable device, and a pedometer, etc.

[0136] The processor 610 is configured to:

[0137] input the audio feature to a to-be-trained model for iterative training; wherein in the process of the Nth iteration training, after the to-be-trained model performs feature mapping on the audio feature to obtain a first feature vector, the to-be-trained model performs attention mechanism fitting on the first feature vector to obtain a second feature vector, N is a positive integer;

[0138] determine a cross-entropy loss value and a clustering loss value according to the second feature vector, respectively;

[0139] In the case where the cross-entropy loss value and the clustering loss value meet a preset condition, determine the to-be-trained model after the Nth iteration training as a speaker segmentation model.

[0140] Optionally, the to-be-trained model includes a first linear layer, a first encoding layer, and a target network layer connected in sequence;

[0141] The first linear layer is configured to perform feature mapping on the input audio feature to output the first feature vector; the first encoding layer is configured to perform attention mechanism fitting on the input first feature vector to output the second feature vector; and the target network layer is configured to process the input second feature vector to output a target feature vector.

[0142] The processor 610, in the process of determining the cross-entropy loss value and the clustering loss value according to the second feature vector, respectively, includes: determining the cross-entropy loss value through a cross-entropy loss function and the target feature vector; and determining the clustering loss value through a supervised clustering loss and the second feature vector.

[0143] Optionally, the target network layer includes M second encoding layers and M second linear layers, and M is a positive integer;

[0144] If the M is greater than 1, a second linear layer is arranged between any two adjacent second encoding layers;

[0145] If the M is equal to 1, the output of the second encoding layer is the input of the second linear layer.

[0146] Optionally, the M is equal to 1, and the second encoding layer is configured to perform attention mechanism fitting on the input second feature vector to output a third feature vector.

[0147] The second linear layer is configured to perform feature mapping on the input third feature vector to output the target feature vector.

[0148] Optionally, the preset condition comprises:

[0149] The sum of the cross-entropy loss value and the clustering loss value is less than a first preset value.

[0150] Alternatively, the weighted sum of the cross-entropy loss value and the clustering loss value is less than a second preset value.

[0151] Optionally, the processor 610 is further configured to:

[0152] Obtain L audio files to be trained;

[0153] Generate a speech index of the audio files to be trained;

[0154] Determine a target audio file according to the speech index, the target audio file being an audio file in the L audio files to be trained;

[0155] Extract features of the target audio file to obtain the audio features;

[0156] wherein L is a positive integer.

[0157] Optionally, the speech index is a speech index in kaldi format.

[0158] Optionally, the audio features are 23-dimensional MFCC audio features.

[0159] Alternatively, the electronic device comprises a speaker diarization model, which can be trained according to the speaker diarization model training method described above.

[0160] wherein the processor 610 is configured to:

[0161] Obtain audio features;

[0162] input the audio feature into a speaker segmentation model to output a target feature vector; wherein the speaker segmentation model comprises a first linear layer, a first encoding layer, and a target network; the first linear layer is configured to perform feature mapping on the input audio feature to output a first feature vector; the first encoding layer is configured to perform attention mechanism fitting on the input first feature vector to output a second feature vector; and the target network is configured to process the input second feature vector to output the target feature vector.

[0163] determine a speaker segmentation result according to the target feature vector.

[0164] Optionally, the target network comprises a second encoding layer and a second linear layer connected in sequence.

[0165] The second encoding layer is configured to perform attention mechanism fitting on the input second feature vector to output a third feature vector.

[0166] The second linear layer is configured to perform feature mapping on the input third feature vector to output the target feature vector.

[0167] The embodiments of the present application also have the same beneficial technical effects as the above Figure 1 or Figure 3 embodiments, which will not be repeated here in detail.

[0168] It should be understood that in the embodiments of the present application, the radio frequency unit 601 can be used for receiving and transmitting signals in the process of information transmission or conversation. Specifically, after receiving the downlink data from the base station, the processor 610 processes it. In addition, the uplink data is sent to the base station. Generally, the radio frequency unit 601 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc. In addition, the radio frequency unit 601 can also communicate with the network and other devices through a wireless communication system.

[0169] The electronic device provides the user with wireless broadband Internet access through the network module 602, such as helping the user to send and receive emails, browse web pages, and access streaming media, etc.

[0170] The audio output unit 603 can convert audio data received by the radio frequency unit 601 or the network module 602 or stored in the memory 609 into an audio signal and output it as a sound. Moreover, the audio output unit 603 can also provide audio output related to a specific function performed by the electronic device 600 (e.g., a call signal reception sound, a message reception sound, etc.). The audio output unit 603 includes a speaker, a buzzer, and a receiver, etc.

[0171] The input unit 604 is configured to receive audio or video signals. The input unit 604 can include a graphic processing unit (GPU) 6041 and a microphone 6042. The graphic processing unit 6041 processes image data of a still picture or a video obtained by an image capture apparatus (e.g., a camera) in a video capture mode or an image capture mode. The processed image frame can be displayed on the display unit 606. The image frame processed by the graphic processing unit 6041 can be stored in the memory 609 (or other storage medium) or transmitted via the radio frequency unit 601 or the network module 602. The microphone 6042 can receive sound and can process such sound as audio data. The processed audio data can be converted into a format that can be transmitted to a mobile communication base station via the radio frequency unit 601 in a telephone call mode.

[0172] The electronic device 600 further includes at least one sensor 605, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor, where the ambient light sensor can adjust the brightness of the display panel 6061 according to the brightness of ambient light, and the proximity sensor can turn off the display panel 6061 and / or the backlight when the electronic device 600 is moved to the ear. As one of the motion sensors, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and when at rest, can detect the magnitude and direction of gravity, and can be used to identify the electronic device posture (such as screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, knock), and the like. The sensor 605 can also include a fingerprint sensor, a pressure sensor, an iris sensor, a molecular sensor, a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, and the like, which will not be described here.

[0173] The display unit 606 is configured to display information input by a user or information provided to the user. The display unit 606 can include a display panel 6061, which can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0174] The user input unit 607 can be used to receive inputted digital or character information, and to generate key signal input related to user settings of the electronic device and control of functions. Specifically, the user input unit 607 includes a touch panel 6071 and other input devices 6072. The touch panel 6071, also called a touch screen, can collect a user's touch operation (such as a user's operation on or near the touch panel 6071 using a finger, a stylus, or any suitable object or accessory) on or near it. The touch panel 6071 can include two parts, a touch detection device and a touch controller. The touch detection device detects the user's touch position and detects a signal caused by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch coordinates, and sends it to the processor 610, receives commands from the processor 610 and executes them. In addition, the touch panel 6071 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 6071, the user input unit 607 can also include other input devices 6072. Specifically, the other input devices 6072 can include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, on / off buttons, etc.), trackballs, mice, joysticks, and the like, which will not be described here.

[0175] Further, the touch panel 6071 can be overlaid on the display panel 6061, and when the touch panel 6071 detects a touch operation on or near it, it transmits to the processor 610 to determine the type of touch event, and then the processor 610 provides corresponding visual output on the display panel 6061 according to the type of touch event. Although in the Figure 6 In some embodiments, the touch panel 6071 and the display panel 6061 can be integrated to realize the input and output functions of the electronic device, which is not limited here.

[0176] The interface unit 608 is an interface for connecting external devices to the electronic device 600. For example, the external devices can include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device having an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, and the like. The interface unit 608 can be used to receive input (e.g., data information, power, etc.) from external devices and transmit the received input to one or more elements within the electronic device 600, or can be used to transmit data between the electronic device 600 and external devices.

[0177] The memory 609 can be used to store software programs and various data. The memory 609 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), and the like. In addition, the memory 609 can include a high-speed random access memory, and can also include a nonvolatile memory, for example, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device.

[0178] The processor 610 is a control center of the electronic device, connects all parts of the electronic device through various interfaces and lines, executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 609 and calling data stored in the memory 609, and thus monitors the whole electronic device. The processor 610 can include one or more processing units; preferably, the processor 610 can integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 610.

[0179] The electronic device 600 can further include a power supply 611 (such as a battery) for supplying power to various components; preferably, the power supply 611 can be logically connected to the processor 610 through a power management system, so as to realize the functions of managing charging, discharging, and power consumption management, and the like through the power management system.

[0180] In addition, the electronic device 600 includes some functional modules that are not shown and will not be described here.

[0181] Preferably, the embodiments of the present application further provide an electronic device, including a processor 610, a memory 609, and a computer program stored in the memory 609 and executable on the processor 610, wherein the computer program is executed by the processor 610 to implement each process of the above-mentioned speaker segmentation model training method or speaker segmentation method embodiments, and can achieve the same technical effects. To avoid repetition, it will not be described here.

[0182] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement each process of the speaker segmentation model training method or the speaker segmentation method embodiment, and achieves the same technical effects. To avoid repetition, details are not described herein. The computer readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, or the like.

[0183] It should be noted that in this document, the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article, or device that includes a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article, or device. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or device that includes the element.

[0184] From the above description of the embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by software and a necessary general hardware platform, and of course, they can also be implemented by hardware, but in many cases, the former is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, an optical disk), and includes a plurality of instructions for causing a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the methods described in the various embodiments of the present application.

[0185] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above specific embodiments, and the above specific embodiments are only illustrative, not limiting. Those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the protection scope of the claims.

Claims

1. A method for training a speaker diarization model, the method comprising: The method comprises: inputting an audio feature into a to-be-trained model for iterative training; wherein, in the process of the Nth iteration training, after the to-be-trained model performs feature mapping on the audio feature to obtain a first feature vector, the to-be-trained model performs attention mechanism fitting on the first feature vector to obtain a second feature vector, N being a positive integer; determining a cross-entropy loss value and a clustering loss value according to the second feature vector and a cross-entropy loss function and a supervised clustering loss respectively; in the case that the cross-entropy loss value and the clustering loss value meet a preset condition, determining the to-be-trained model after the Nth iteration training as a speaker segmentation model.

2. The method of claim 1, wherein, The to-be-trained model comprises a first linear layer, a first encoding layer and a target network layer connected in sequence; wherein, the first linear layer is configured to perform feature mapping on the input audio feature to output the first feature vector; the first encoding layer is configured to perform attention mechanism fitting on the input first feature vector to output the second feature vector; and the target network layer is configured to process the input second feature vector to output a target feature vector. The method comprises:

3. The method of claim 2, wherein, determining the cross-entropy loss value by the cross-entropy loss function and the target feature vector; and determining the clustering loss value by the supervised clustering loss and the second feature vector. The target network layer comprises M layers of second encoding layers and M layers of second linear layers, M being a positive integer; if M is greater than 1, one layer of second linear layer is arranged between any two adjacent layers of second encoding layers; 4. The method of claim 3, wherein, if M is equal to 1, the output of the second encoding layer is the input of the second linear layer. The M is equal to 1, and the second encoding layer is configured to perform attention mechanism fitting on the input second feature vector to output a third feature vector; 5. The method of claim 1, wherein, the second linear layer is configured to perform feature mapping on the input third feature vector to output the target feature vector. The preset condition comprises: the sum of the cross-entropy loss value and the clustering loss value is less than a first preset value; or 6. The method of claim 1, wherein, the weighted sum of the cross-entropy loss value and the clustering loss value is less than a second preset value. Before inputting the audio feature into the to-be-trained model for iterative training, the method further comprises: obtaining L to-be-trained audio files, L being a positive integer; generating a speech index of the to-be-trained audio files; determining a target audio file according to the speech index, the target audio file being an audio file in the L to-be-trained audio files; 7. A method of speaker diarization, the method comprising: extracting features of the target audio file to obtain the audio feature. The method comprises: obtaining an audio feature; input the audio feature into a speaker segmentation model to output a target feature vector; wherein the speaker segmentation model comprises a first linear layer, a first encoding layer, and a target network layer; the first linear layer is configured to perform feature mapping on the input audio feature to output a first feature vector; the first encoding layer is configured to perform attention mechanism fitting on the input first feature vector to output a second feature vector; and the target network layer is configured to process the input second feature vector to output the target feature vector; determine a speaker segmentation result according to the target feature vector.

8. The method of claim 7, wherein, The target network layer comprises a second encoding layer and a second linear layer connected in sequence. The second encoding layer is configured to perform attention mechanism fitting on the input second feature vector to output a third feature vector. The second linear layer is configured to perform feature mapping on the input third feature vector to output the target feature vector.

9. A speaker diarization model training apparatus, comprising: The method comprises: a training module configured to input an audio feature into a to-be-trained model for iterative training; wherein in the process of the Nth iterative training, after the to-be-trained model performs feature mapping on the audio feature to obtain a first feature vector, the to-be-trained model performs attention mechanism fitting on the first feature vector to obtain a second feature vector, N being a positive integer; a first determination module configured to determine a cross-entropy loss value and a clustering loss value according to the second feature vector and a cross-entropy loss function and a supervised clustering loss, respectively; a second determination module configured to determine the to-be-trained model after the Nth iterative training as a speaker segmentation model in the case that the cross-entropy loss value and the clustering loss value satisfy a preset condition.

10. A speaker diarization apparatus characterized by comprising: The device comprises: an acquisition module configured to acquire an audio feature; a vector acquisition module configured to input the audio feature into a speaker segmentation model to output a target feature vector; wherein the speaker segmentation model comprises a first linear layer, a first encoding layer, and a target network layer; the first linear layer is configured to perform feature mapping on the input audio feature to output a first feature vector; the first encoding layer is configured to perform attention mechanism fitting on the input first feature vector to output a second feature vector; and the target network layer is configured to process the input second feature vector to output the target feature vector; a determination module configured to determine a speaker segmentation result according to the target feature vector.

11. An electronic device, comprising: The computer program is stored on the computer readable storage medium and is executed by the processor to implement the steps in the method of any one of claims 1 to 8. The computer program is stored on the computer readable storage medium and is executed by the processor to implement the steps in the method of any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that, ​

Images