Role recognition method and device based on speaker segmentation

Abstract

The present invention provides a method and device for role recognition based on speaker segmentation. The method comprises: converting the conversational speech to be recognized into text data, segmenting the text data into multiple sentences based on a sentence segmentation model, and extracting the text features of each sentence; segmenting the conversational speech to be recognized, obtaining an audio segment corresponding to each sentence, and extracting the acoustic features of each audio segment; aligning the text features and acoustic features corresponding to each sentence based on an attention mechanism to generate an alignment vector corresponding to each sentence; and obtaining the speaker category corresponding to each sentence based on a classification model based on the alignment vector, text features, and acoustic features corresponding to each sentence. The present invention utilizes the interactive features between text and audio for role recognition, improving the accuracy of role recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method and device for role recognition based on speaker segmentation. Background Art

[0002] Currently, role recognition in voice conversations primarily consists of two modules: speaker segmentation and role recognition. Both modules play a crucial role in tasks such as automated customer service speech segmentation and recognition and intelligent meeting recording.

[0003] Speaker segmentation involves identifying and labeling the speaking duration of different speakers within a collected speech signal stream. Role identification involves labeling the speakers of individual speech segments in a multi-person conversation. Existing techniques for role identification in speech use audio features or semantic features derived from audio transcription, resulting in low accuracy. Summary of the Invention

[0004] The present invention provides a method and device for role recognition based on speaker segmentation, which is used to solve the defect of low role recognition accuracy in the prior art of using audio features or semantic features after audio transcription for role recognition, and to improve the accuracy of role recognition.

[0005] The present invention provides a role recognition method based on speaker segmentation, comprising:

[0006] Converting the conversational speech to be recognized into text data, segmenting the text data into multiple sentences based on a sentence segmentation model, and extracting text features of each sentence;

[0007] Segmenting the conversation speech to be recognized, obtaining audio segments corresponding to each sentence, and extracting acoustic features of each audio segment;

[0008] Based on the attention mechanism, the text features and acoustic features corresponding to each sentence are aligned to generate the alignment vector corresponding to each sentence;

[0009] According to the alignment vector, text features and acoustic features corresponding to each sentence, the speaker category corresponding to each sentence is obtained based on the classification model.

[0010] According to a method for role recognition based on speaker segmentation provided by the present invention, the text data is segmented into multiple sentences based on a sentence segmentation model, and text features of each sentence are extracted, including:

[0011] Encoding the text data based on a language model to obtain a hidden state of each word in the text data;

[0012] According to the hidden state of each word in the text data, the text data is divided into a plurality of sentences based on a sentence segmentation model;

[0013] The hidden states of all words in each sentence are fused to obtain the text features of each sentence.

[0014] According to a method for role recognition based on speaker segmentation provided by the present invention, the text features and acoustic features corresponding to each sentence are aligned based on an attention mechanism to generate an alignment vector corresponding to each sentence, including:

[0015] Encode the text features and acoustic features corresponding to each sentence based on a bidirectional long short-term memory neural network to obtain the text feature vector corresponding to each sentence and the acoustic feature vector of each audio segment;

[0016] Based on the attention mechanism, the text feature vector of each sentence and the acoustic feature vector of the audio segment corresponding to each sentence are aligned to generate an alignment vector corresponding to each sentence.

[0017] According to a method for role recognition based on speaker segmentation provided by the present invention, the text feature vector of each sentence and the acoustic feature vector of the audio segment corresponding to each sentence are aligned based on the attention mechanism described in the following formula to generate an alignment vector corresponding to each sentence:

[0018]

[0019]

[0020] p ij =tanh(u T s i +v T h j +b);

[0021] in, represents the alignment vector corresponding to the i-th word in any sentence, tanh is the hyperbolic tangent function, u, v and b represent parameters, u T represents the transpose of u, v T represents the transpose of v, s i represents the acoustic feature vector corresponding to the i-th word in any sentence, h j represents the text feature vector corresponding to the jth word in any sentence, and N represents the number of words in any sentence.

[0022] According to a method for role recognition based on speaker segmentation provided by the present invention, the speaker category corresponding to each sentence is obtained based on a classification model according to the alignment vector, text features, and acoustic features corresponding to each sentence, including:

[0023] Encoding the alignment vector based on the bidirectional long short-term memory neural network;

[0024] According to the text feature vector, acoustic feature vector and encoded alignment vector corresponding to each sentence, the speaker category corresponding to each sentence is obtained based on the classification model.

[0025] According to a method for role recognition based on speaker segmentation provided by the present invention, the speaker category corresponding to each sentence is obtained based on a classification model according to the text feature vector, acoustic feature vector and encoded alignment vector corresponding to each sentence, including:

[0026] Take the text feature vector, acoustic feature vector and encoded alignment vector corresponding to each sentence as input information;

[0027] Connect the input information corresponding to the context of each sentence with the input information corresponding to each sentence;

[0028] The connection result is input into the classification model to obtain the speaker category corresponding to each sentence.

[0029] According to a method for role recognition based on speaker segmentation provided by the present invention, before dividing the text data into a plurality of sentences based on a sentence segmentation model and extracting text features of each sentence, the method further includes:

[0030] The conversational speech sample is used as a sample, and the actual sentence segmentation mark of the text data corresponding to the conversational speech sample and the actual speaker category corresponding to each sentence are used as labels to train the sentence segmentation model, the attention mechanism and the classification model;

[0031] The training loss function is obtained based on the loss between the sentence segmentation token output by the sentence segmentation model and the actual sentence segmentation token, and the loss between the speaker category output by the classification model and the actual speaker category.

[0032] The present invention also provides a role recognition device based on speaker segmentation, comprising:

[0033] A first segmentation module is used to convert the conversation speech to be recognized into text data, segment the text data into multiple sentences based on a sentence segmentation model, and extract text features of each sentence;

[0034] A second segmentation module is used to segment the conversation speech to be recognized, obtain the audio segment corresponding to each sentence, and extract the acoustic features of each audio segment;

[0035] The alignment module is used to align the text features and acoustic features corresponding to each sentence based on the attention mechanism and generate the alignment vector corresponding to each sentence;

[0036] The recognition module is used to obtain the speaker category corresponding to each sentence based on the alignment vector, text features and acoustic features corresponding to each sentence based on the classification model.

[0037] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-described methods for role recognition based on speaker segmentation are implemented.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described methods for role recognition based on speaker segmentation.

[0039] The present invention provides a method and device for role recognition based on speaker segmentation. The method converts the conversational speech to be recognized into text and then divides it into sentences. After extracting the text features and acoustic features corresponding to each sentence, an attention mechanism is used to further extract the interaction features between the text features and the acoustic features. The interaction features, text features and acoustic features are integrated to perform speaker recognition, thereby improving the accuracy of role recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 This is one of the flow charts of the role recognition method based on speaker segmentation provided by the present invention;

[0042] Figure 2 1. It is a schematic diagram of the sentence segmentation process in the role recognition method based on speaker segmentation provided by the present invention;

[0043] Figure 3 This is the second flow chart of the role recognition method based on speaker segmentation provided by the present invention;

[0044] Figure 4 2 is a schematic structural diagram of a role recognition device based on speaker segmentation provided by the present invention;

[0045] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0046] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0047] The following combination Figure 1 The present invention describes a method for role recognition based on speaker segmentation, comprising: step 101, converting a conversation speech to be recognized into text data, segmenting the text data into a plurality of sentences based on a sentence segmentation model, and extracting text features of each sentence;

[0048] The conversational speech to be recognized is the conversational speech that needs to be recognized, and is audio stream data with a timestamp.

[0049] After the speech to be recognized is converted into text data, the text data is preprocessed. The preprocessed text data may or may not contain punctuation.

[0050] Since text data contains encoding formats, network labels, and useless punctuation marks, the text data is cleaned and noise is removed through preprocessing.

[0051] The preprocessing includes one or more of removing network tags, removing useless punctuation marks, converting between traditional and simplified Chinese characters, and converting between full-width and half-width characters.

[0052] After the text data corresponding to the conversational speech to be recognized is segmented into sentences using the sentence segmentation model, feature extraction is performed on each sentence. This embodiment is not limited to the type of sentence segmentation model and the specific extraction method.

[0053] Step 102: segment the conversation speech to be recognized, obtain the audio segment corresponding to each sentence, and extract the acoustic features of each audio segment;

[0054] The speech to be recognized is segmented based on the timestamp of each sentence, generating multiple audio segments. Each sentence corresponds to an audio segment. Each word in each sentence is aligned with its pronunciation in the corresponding audio segment, facilitating the subsequent fusion of the sentence's textual features with the acoustic features of the corresponding audio segment.

[0055] Acoustic features in audio can be extracted through various methods, such as MFCC (Mel-Frequency Cepstral Coefficients) and Filter bank.

[0056] Taking the Filter bank as an example, the acoustic feature extraction steps are as follows:

[0057] 1. Framing each audio segment to smooth the acoustic signal and enable fast Fourier transform;

[0058] 2. Perform fast Fourier transform on each segment of the framed signal, and a spectrum diagram can be obtained after the transformation;

[0059] 3. Use Mel filter to filter the signal;

[0060] 4. Take the logarithm of the output of the triangular filter to generate the filter bank vector, which is generally 40-dimensional.

[0061] 5. You can directly use the generated filter bank vector as the acoustic feature of each audio segment. Alternatively, you can perform a discrete cosine transform on the filter bank vector to obtain an MFCC feature vector, which can be used as the acoustic feature.

[0062] Step 103: align the text features and acoustic features corresponding to each sentence based on the attention mechanism to generate an alignment vector corresponding to each sentence;

[0063] Using the attention mechanism, the text features and acoustic features are fused according to their attention weights, and the interaction features between the text features and acoustic features are extracted, and the interaction features are used as the alignment vector.

[0064] Step 104 : Obtain the speaker category corresponding to each sentence based on the classification model according to the alignment vector, text features, and acoustic features corresponding to each sentence.

[0065] Optionally, the classification model is a softmax neural network model. The alignment vector, text features, and acoustic features corresponding to each sentence are used as input, and the classification model outputs the speaker category corresponding to each sentence. For example, the output format is: Speaker A: ..., ...; Speaker B: ..., ....

[0066] This embodiment converts the conversational speech to be recognized into text and then divides it into sentences. After extracting the text features and acoustic features corresponding to each sentence, the attention mechanism is used to further extract the interaction features between the text features and acoustic features. The interaction features, text features and acoustic features are integrated to perform speaker recognition, thereby improving the accuracy of role recognition.

[0067] Based on the above embodiment, the text data is segmented into multiple sentences based on the sentence segmentation model in this embodiment, and the text features of each sentence are extracted, including: encoding the text data based on the language model to obtain the hidden state of each word in the text data;

[0068] Use the idea of ​​transfer learning to represent text data based on a pre-trained language model.

[0069] The language model can be BERT, Transformer, or Elmo, etc.

[0070] Taking BERT as an example, we can obtain the hidden state of each word in the text data. During the language model training process, we will fine-tune the parameters to make the extracted text features more suitable for the sentence segmentation task. The formula is as follows:

[0071]

[0072] Where n is the total number of words in the text data, w i is the word vector of the i-th word, is the hidden state of the i-th word vector after being encoded by the pre-trained model Bert.

[0073] According to the hidden state of each word in the text data, the text data is divided into a plurality of sentences based on a sentence segmentation model;

[0074] Optionally, the sentence segmentation model is a sequence labeling model. The sequence labeling model calculates the scores of all possible label sequences and selects the sequence with the highest score as the output sequence. The labels of the output sequence are used as the basis for sentence segmentation, for separating the upper and lower sentences. The principle of segmentation is to segment the sentence as finely as possible.

[0075] The encoded word-level hidden state is passed through the fully connected neural network and the sequence labeling network in sequence, and the label of each word is output, represented by y, y∈{B,O}. Among them, B represents the beginning of a sentence and O represents the middle of a sentence. The sentence segmentation can be performed by predicting the label of each word. The specific process is as follows Figure 2 shown.

[0076] The hidden states of all words in each sentence are fused to obtain the text features of each sentence.

[0077] Based on the above embodiment, the present embodiment aligns the text features and acoustic features corresponding to each sentence based on the attention mechanism to generate an alignment vector corresponding to each sentence, including: encoding the text features and acoustic features corresponding to each sentence based on a bidirectional long short-term memory neural network to obtain a text feature vector corresponding to each sentence and an acoustic feature vector for each audio segment;

[0078] Optionally, the network parameters of the bidirectional long short-term memory neural network for encoding text features and acoustic features are different, and they are independent encoding networks.

[0079] The formula for encoding the text features of each sentence based on the bidirectional long short-term memory neural network is as follows:

[0080]

[0081]

[0082]

[0083] Among them, e i The text feature of the i-th word in any sentence is encoded by the forward and reverse long short-term memory neural network to obtain the forward hidden layer state and the reverse hidden layer state The two vectors are connected as text encoding representation, using h i express.

[0084] The formula for encoding acoustic features through a bidirectional long short-term memory neural network is as follows:

[0085]

[0086]

[0087]

[0088] Among them, v i The acoustic features corresponding to the i-th word in any sentence are encoded by the forward and reverse long short-term memory neural network to obtain the forward hidden layer state and the reverse hidden layer state The two vectors are connected as text encoding representation, using s i express.

[0089] Based on the attention mechanism, the text feature vector of each sentence and the acoustic feature vector of the audio segment corresponding to each sentence are aligned to generate an alignment vector corresponding to each sentence.

[0090] To fully utilize the interaction between acoustic and text features, they need to be aligned. Therefore, an attention mechanism is used to learn the alignment weights between acoustic and text features, thereby capturing the interaction between these two features at the phonetic and character level.

[0091] There are rich interactive features between audio and semantics, but existing technologies do not utilize the interactive information between audio and semantics, resulting in low character recognition accuracy.

[0092] Based on the above embodiment, in this embodiment, the text feature vector of each sentence and the acoustic feature vector of the audio segment corresponding to each sentence are aligned based on the attention mechanism as described in the following formula to generate an alignment vector corresponding to each sentence:

[0093]

[0094]

[0095] p ij =tanh(u T s i +v T h j +b);

[0096] in, represents the alignment vector corresponding to the i-th word in any sentence, tanh is the hyperbolic tangent function, u, v and b represent parameters, u T represents the transpose of u, v T represents the transpose of v, s i represents the acoustic feature vector corresponding to the i-th word in any sentence, h j represents the text feature vector corresponding to the jth word in any sentence, and N represents the number of words in any sentence.

[0097] Based on the above embodiment, the present embodiment obtains the speaker category corresponding to each sentence based on the alignment vector, text features, and acoustic features corresponding to each sentence based on a classification model, including: encoding the alignment vector based on the bidirectional long short-term memory neural network;

[0098] Use a bidirectional long short-term memory neural network to encode the alignment vector and obtain a comprehensive vector of text features and acoustic features. The calculation formula is as follows:

[0099]

[0100]

[0101]

[0102] in, Represents the alignment vector corresponding to the i-th word in any sentence, and the forward hidden layer state is obtained through forward and reverse long short-term memory neural network encoding and the reverse hidden layer state The two vectors are connected as text encoding representation, using m i represents, and N represents the total number of words in any sentence.

[0103] According to the text feature vector, acoustic feature vector and encoded alignment vector corresponding to each sentence, the speaker category corresponding to each sentence is obtained based on the classification model.

[0104] Optionally, the vector corresponding to the i-th word in each sentence is c i =[m i ,s i ,h i ], the vector corresponding to all words in each sentence is fused to obtain the vector, such as C = c1 + c2 + ... + c N Input into the classification model to obtain the speaker category corresponding to each sentence.

[0105] Based on the above embodiments, the present embodiment obtains the speaker category corresponding to each sentence based on a classification model according to the text feature vector, acoustic feature vector and encoded alignment vector corresponding to each sentence, including: obtaining the input information corresponding to each sentence according to the text feature vector, acoustic feature vector and encoded alignment vector corresponding to each sentence; connecting the input information corresponding to the context of each sentence with the input information corresponding to each sentence; and inputting the connection result into the classification model to obtain the speaker category corresponding to each sentence.

[0106] Among them, the input information corresponding to each sentence is C=c1+c2+...+c N , c i =[m i ,s i ,h i ]. Optionally, the role identity is determined based on the features of consecutive sentences, that is, a sliding window method is used. When calculating the role to which the j-th sentence belongs, the input information corresponding to the n1 sentences above and below the window is used, and n1 generally takes a value in the range of 3 to 5.

[0107] After connecting the input information corresponding to 2*n1+1 sentences, the fully connected neural network and the softmax neural network are used to calculate the probability that the j-th sentence belongs to different speaker categories. The formula is as follows:

[0108] y=softmax(w T Q j +d);

[0109] Q j =concat(C j-n1 ,...,C j-1 ,C j ,C j+1 ,...,C j+n1 ).

[0110] Among them, C jrepresents the input information corresponding to the j-th sentence, w and d are the parameters to be learned, and y is the predicted result.

[0111] This embodiment adopts a prediction method based on the features of continuous sentences at the output layer of role recognition, effectively utilizing the information of context sentences to improve the accuracy of role recognition.

[0112] Based on the above embodiments, this embodiment further includes, before segmenting the text data into multiple sentences based on the sentence segmentation model and extracting text features of each sentence, using a conversational speech sample as a sample, and using the actual sentence segmentation marks of the text data corresponding to the conversational speech sample and the actual speaker category corresponding to each sentence as labels to train the sentence segmentation model, the attention mechanism, and the classification model;

[0113] The training loss function is obtained based on the loss between the sentence segmentation token output by the sentence segmentation model and the actual sentence segmentation token, and the loss between the speaker category output by the classification model and the actual speaker category.

[0114] When performing character recognition on conversational speech, existing techniques for speaker segmentation and character recognition are completely independent of each other, leading to cumulative errors. This is primarily due to the fact that existing speaker segmentation methods cannot guarantee accuracy. Any problems with speaker segmentation directly impact the quality of character recognition.

[0115] This embodiment uses the attention mechanism and classification model as a whole as the speaker detection model. The loss of the sentence segmentation model and the loss of the speaker detection model are combined as the final loss. The parameters of the sentence segmentation model and the speaker detection model are updated based on the final loss. The specific formula is as follows:

[0116] Loss=α1l1+α2l2

[0117] Among them, Loss represents the final loss, l1 represents the loss of the sentence segmentation model, which reflects the difference between the sentence segmentation tokens output by the sentence segmentation model and the actual sentence segmentation tokens, l2 represents the loss of the speaker detection model, which reflects the difference between the speaker category output by the classification model and the actual speaker category, α1 and α2 are the corresponding weight parameters, and α1+α2=1. The complete flow chart of this embodiment is shown in the figure. Figure 3 shown.

[0118] This embodiment employs a joint training approach, first segmenting sentences into the smallest possible units to ensure each sentence contains only one speaker, and then performing role identification. This approach requires only a single model to simultaneously perform speaker segmentation and role identification, effectively resolving the existing problem of training a separate model for each role, which consumes excessive storage space and increases response time. During training, the sentence segmentation and role identification tasks are trained simultaneously, completing both speaker segmentation and role identification, effectively resolving the problem of error accumulation caused by completely separating the two tasks.

[0119] The following describes a role recognition device based on speaker segmentation provided by the present invention. The role recognition device based on speaker segmentation described below and the role recognition method based on speaker segmentation described above can be referenced to each other.

[0120] like Figure 4 As shown, the apparatus includes a first segmentation module 401, a second segmentation module 402, an alignment module 403 and a recognition module 404, wherein:

[0121] The first segmentation module 401 is used to convert the conversation speech to be recognized into text data, segment the text data into multiple sentences based on the sentence segmentation model, and extract text features of each sentence;

[0122] The second segmentation module 402 is used to segment the conversation speech to be recognized, obtain the audio segment corresponding to each sentence, and extract the acoustic features of each audio segment;

[0123] The alignment module 403 is used to align the text features and acoustic features corresponding to each sentence based on the attention mechanism to generate an alignment vector corresponding to each sentence;

[0124] The recognition module 404 is configured to obtain the speaker category corresponding to each sentence based on the classification model according to the alignment vector, text features, and acoustic features corresponding to each sentence.

[0125] This embodiment converts the conversational speech to be recognized into text and then divides it into sentences. After extracting the text features and acoustic features corresponding to each sentence, the attention mechanism is used to further extract the interaction features between the text features and acoustic features. The interaction features, text features and acoustic features are integrated to perform speaker recognition, thereby improving the accuracy of role recognition.

[0126] Figure 5 An example of a physical structure diagram of an electronic device is shown below. Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute a role recognition method based on speaker segmentation, which includes: converting the conversation speech to be recognized into text data, segmenting the text data into multiple sentences based on a sentence segmentation model, and extracting text features of each sentence; segmenting the conversation speech to be recognized, obtaining an audio segment corresponding to each sentence, and extracting acoustic features of each audio segment; aligning the text features and acoustic features corresponding to each sentence based on an attention mechanism to generate an alignment vector corresponding to each sentence; and obtaining the speaker category corresponding to each sentence based on a classification model according to the alignment vector, text features, and acoustic features corresponding to each sentence.

[0127] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0128] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the role recognition method based on speaker segmentation provided by the above methods, the method including: converting the conversational speech to be recognized into text data, dividing the text data into multiple sentences based on a sentence segmentation model, and extracting text features of each sentence; segmenting the conversational speech to be recognized, obtaining an audio segment corresponding to each sentence, and extracting acoustic features of each audio segment; aligning the text features and acoustic features corresponding to each sentence based on an attention mechanism, and generating an alignment vector corresponding to each sentence; and obtaining the speaker category corresponding to each sentence based on a classification model according to the alignment vector, text features and acoustic features corresponding to each sentence.

[0129] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned speaker segmentation-based role recognition method, the method comprising: converting the conversational speech to be recognized into text data, segmenting the text data into multiple sentences based on a sentence segmentation model, and extracting text features of each sentence; segmenting the conversational speech to be recognized, obtaining an audio segment corresponding to each sentence, and extracting acoustic features of each audio segment; aligning the text features and acoustic features corresponding to each sentence based on an attention mechanism, and generating an alignment vector corresponding to each sentence; and obtaining the speaker category corresponding to each sentence based on a classification model according to the alignment vector, text features, and acoustic features corresponding to each sentence.

[0130] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0131] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A role recognition method based on speaker segmentation, characterized in that: include: Converting the conversational speech to be recognized into text data, segmenting the text data into multiple sentences based on a sentence segmentation model, and extracting text features of each sentence; Segmenting the conversation speech to be recognized, obtaining audio segments corresponding to each sentence, and extracting acoustic features of each audio segment; Based on the attention mechanism, the text features and acoustic features corresponding to each sentence are aligned to generate the alignment vector corresponding to each sentence; Based on the alignment vector, text features, and acoustic features corresponding to each sentence, the speaker category corresponding to each sentence is obtained based on the classification model; The attention mechanism is used to align the text features and acoustic features corresponding to each sentence to generate an alignment vector corresponding to each sentence, including: Encode the text features and acoustic features corresponding to each sentence based on a bidirectional long short-term memory neural network to obtain the text feature vector corresponding to each sentence and the acoustic feature vector of each audio segment; Based on the attention mechanism, the text feature vector of each sentence is aligned with the acoustic feature vector of the audio segment corresponding to each sentence to generate an alignment vector corresponding to each sentence. The method of obtaining the speaker category corresponding to each sentence based on the classification model according to the alignment vector, text features, and acoustic features corresponding to each sentence includes: Encoding the alignment vector based on the bidirectional long short-term memory neural network; According to the text feature vector, acoustic feature vector and encoded alignment vector corresponding to each sentence, the speaker category corresponding to each sentence is obtained based on the classification model.

2. The role recognition method based on speaker segmentation according to claim 1, characterized in that The sentence segmentation model is used to segment the text data into multiple sentences, and the text features of each sentence are extracted, including: Encoding the text data based on a language model to obtain a hidden state of each word in the text data; According to the hidden state of each word in the text data, the text data is divided into a plurality of sentences based on a sentence segmentation model; The hidden states of all words in each sentence are fused to obtain the text features of each sentence.

3. The method for role recognition based on speaker segmentation according to claim 1, characterized in that: The text feature vector of each sentence and the acoustic feature vector of the audio segment corresponding to each sentence are aligned based on the attention mechanism described in the following formula to generate the alignment vector corresponding to each sentence: p ij =tanh(u T S i +v T h j +b); in, represents the alignment vector corresponding to the i-th word in any sentence, tanh is the hyperbolic tangent function, u, v and b represent parameters, u T represents the transpose of u, v T represents the transpose of v, s i represents the acoustic feature vector corresponding to the i-th word in any sentence, h j represents the text feature vector corresponding to the jth word in any sentence, and N represents the number of words in any sentence.

4. The method for role recognition based on speaker segmentation according to claim 1, characterized in that: The method of obtaining the speaker category corresponding to each sentence based on the classification model according to the text feature vector, acoustic feature vector and encoded alignment vector corresponding to each sentence includes: Obtain the input information corresponding to each sentence based on the text feature vector, acoustic feature vector, and encoded alignment vector corresponding to each sentence; Connect the input information corresponding to the context of each sentence with the input information corresponding to each sentence; The connection result is input into the classification model to obtain the speaker category corresponding to each sentence.

5. The method for role recognition based on speaker segmentation according to any one of claims 1 to 4, characterized in that: Before dividing the text data into a plurality of sentences based on the sentence segmentation model and extracting the text features of each sentence, the method further includes: The conversational speech sample is used as a sample, and the actual sentence segmentation mark of the text data corresponding to the conversational speech sample and the actual speaker category corresponding to each sentence are used as labels to train the sentence segmentation model, the attention mechanism and the classification model; The training loss function is obtained based on the loss between the sentence segmentation token output by the sentence segmentation model and the actual sentence segmentation token, and the loss between the speaker category output by the classification model and the actual speaker category.

6. A role recognition device based on speaker segmentation, characterized in that: include: A first segmentation module is used to convert the conversation speech to be recognized into text data, segment the text data into multiple sentences based on a sentence segmentation model, and extract text features of each sentence; A second segmentation module is used to segment the conversation speech to be recognized, obtain the audio segment corresponding to each sentence, and extract the acoustic features of each audio segment; The alignment module is used to align the text features and acoustic features corresponding to each sentence based on the attention mechanism and generate the alignment vector corresponding to each sentence; The recognition module is used to obtain the speaker category corresponding to each sentence based on the alignment vector, text features, and acoustic features corresponding to each sentence based on the classification model; The method aligns the text features and acoustic features corresponding to each sentence based on the attention mechanism to generate an alignment vector corresponding to each sentence, including: encoding the text features and acoustic features corresponding to each sentence based on a bidirectional long short-term memory neural network to obtain a text feature vector corresponding to each sentence and an acoustic feature vector for each audio segment; aligning the text feature vector of each sentence with the acoustic feature vector of the audio segment corresponding to each sentence based on the attention mechanism to generate an alignment vector corresponding to each sentence; The method of obtaining the speaker category corresponding to each sentence based on the classification model according to the alignment vector, text features and acoustic features corresponding to each sentence includes: encoding the alignment vector based on the bidirectional long short-term memory neural network; and obtaining the speaker category corresponding to each sentence based on the classification model according to the text feature vector, acoustic feature vector and encoded alignment vector corresponding to each sentence.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method for role recognition based on speaker segmentation according to any one of claims 1 to 5 are implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for role recognition based on speaker segmentation according to any one of claims 1 to 5 are implemented.

Images