Method, device, equipment, and medium for role separation based on voice data

Abstract

The embodiment of the present application provides a method, device, equipment, and medium for role separation based on voice data, which belongs to the field of artificial intelligence technology. The method includes: extracting the voiceprint vector and text vector of each voice segment to determine the preliminary fusion vector of the voice segment, inputting the preliminary fusion vector into a preset neural network model, and the neural network model calculates the vector weight for each preliminary fusion vector based on the attention mechanism to obtain multiple target fusion vectors, and then obtains the separation result of the voice data to be processed through the target fusion vector and the neural network model. The embodiment of the present application realizes role separation through the joint action of the text vector of the voice data, the voiceprint vector and the attention mechanism of the neural network model, thereby improving the accuracy of role separation. The embodiment of the present application specifically includes the following advantages: First, more text information is added, and the information source path is expanded. Second, based on the attention mechanism, the role separation task is better achieved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for separating roles based on voice data. Background Art

[0002] Currently, in the field of phone conversations, it is often necessary to distinguish between two or more parties. Only by correctly distinguishing the text in speech can we achieve good results in subsequent natural language processing tasks. For example, if a customer's speech is mistakenly attributed to an agent, this will have a significant impact on the evaluation of the agent's speech compliance. Therefore, how to provide a role separation method based on speech data that can improve the accuracy of role separation has become a pressing technical problem. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose a method, device, equipment, and medium for role separation based on voice data, which can improve the separation accuracy of role separation.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application provides a method for separating roles based on speech data, the method comprising:

[0005] Acquire voice data to be processed, where the voice data to be processed includes voice data of at least two characters;

[0006] Preprocessing the speech data to be processed to obtain a plurality of speech segments;

[0007] Performing voiceprint recognition on each of the speech segments to obtain a voiceprint vector for each of the speech segments;

[0008] Converting each of the speech segments into a text segment, and performing semantic recognition on each of the text segments to obtain a text vector for each of the speech segments;

[0009] Performing vector calculation on the voiceprint vector and the text vector belonging to the same speech segment to obtain a preliminary fusion vector;

[0010] Inputting the obtained multiple preliminary fusion vectors into a preset neural network model, the neural network model calculates a corresponding vector weight for each of the preliminary fusion vectors based on an attention mechanism to obtain multiple target fusion vectors;

[0011] Determine the role separation result of the speech data to be processed based on the multiple target fusion vectors and the neural network model.

[0012] In some embodiments, converting each of the speech segments into a text segment and performing semantic recognition on each of the text segments to obtain a text vector for each of the speech segments includes:

[0013] Converting each of the speech segments into a text segment, and performing semantic recognition on each of the text segments to obtain a preliminary text vector;

[0014] constructing a vector matrix based on the obtained multiple preliminary text vectors, and determining the neighboring preliminary text vectors of each preliminary text vector through the vector matrix;

[0015] Calculating the similarity between each of the preliminary text vectors and the adjacent preliminary text vectors;

[0016] A weight is applied to the preliminary text vector according to the similarity to obtain a text vector for each of the speech segments.

[0017] In some embodiments, converting each of the speech segments into a text segment and performing semantic recognition on each of the text segments to obtain a text vector for each of the speech segments includes:

[0018] Converting each of the speech segments into a text segment, and performing semantic recognition on each of the text segments to obtain a preliminary text vector for each of the speech segments;

[0019] Inputting the text segment into a preset text regularization model, wherein the text regularization model regularizes the word components of the text segment based on grammatical rules to obtain a preliminary text segment;

[0020] calculating the similarity between the text segment and the preliminary text segment;

[0021] A weight is applied to the preliminary text vector according to the similarity to obtain a text vector for each of the speech segments.

[0022] In some embodiments, performing vector calculation on the voiceprint vector and the text vector belonging to the same speech segment to obtain a preliminary fusion vector includes:

[0023] Performing vector concatenation on the voiceprint vector and the text vector belonging to the same speech segment to obtain a preliminary fusion vector;

[0024] or,

[0025] Perform vector summation on the voiceprint vector and the text vector belonging to the same speech segment, and then take an average of the summation results to obtain a preliminary fusion vector.

[0026] In some embodiments, performing vector calculation on the voiceprint vector and the text vector belonging to the same speech segment to obtain a preliminary fusion vector includes:

[0027] Calculating the speech duration of each speech segment;

[0028] Counting the number of characters in each of the text segments;

[0029] Determine a single word duration vector according to the speech duration and the number of characters belonging to the same speech segment;

[0030] Vector calculation is performed on the single word duration vector, the voiceprint vector and the text vector belonging to the same speech segment to obtain a preliminary fusion vector.

[0031] In some embodiments, performing vector calculation on the voiceprint vector and the text vector belonging to the same speech segment to obtain a preliminary fusion vector includes:

[0032] Obtaining the original voiceprint dimension of the voiceprint vector and obtaining the original text dimension of the text vector;

[0033] If the original voiceprint dimension is different from the original text dimension, processing the original voiceprint dimension and the original text dimension by a preset linear transformation method to obtain a target dimension;

[0034] Updating the dimension of the voiceprint vector according to the target dimension to obtain a target voiceprint vector;

[0035] Updating the dimension of the text vector according to the target dimension to obtain a target text vector;

[0036] A preliminary fusion vector is obtained according to the target voiceprint vector and the target text vector.

[0037] In some embodiments, the neural network model includes an MLP fully connected layer and a softmax layer, and determining the role separation result according to the plurality of target fusion vectors and the neural network model includes:

[0038] Map each of the target fusion vectors to N categories through the MLP fully connected layer to obtain a classification result, where N is a positive integer;

[0039] Calculating the probability values ​​of the classification results through the softmax layer to obtain N probability values ​​for each classification result;

[0040] The role of each of the speech segments is determined according to the N probability values ​​to obtain a role separation result of the speech data to be processed.

[0041] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a role separation device based on voice data, the device comprising:

[0042] An acquisition module, configured to acquire voice data to be processed, wherein the voice data to be processed includes voice data of at least two characters;

[0043] A preprocessing module, configured to preprocess the speech data to be processed to obtain a plurality of speech segments;

[0044] a voiceprint extraction module, configured to perform voiceprint recognition on each of the speech segments to obtain a voiceprint vector for each of the speech segments;

[0045] A text extraction module, configured to convert each of the speech segments into a text segment and perform semantic recognition on each of the text segments to obtain a text vector for each of the speech segments;

[0046] a vector calculation module, configured to perform vector calculation on the voiceprint vector and the text vector belonging to the same speech segment to obtain a preliminary fusion vector;

[0047] A weight calculation module is used to input the obtained multiple preliminary fusion vectors into a preset neural network model, and the neural network model calculates a vector weight for each of the preliminary fusion vectors based on an attention mechanism to obtain multiple target fusion vectors;

[0048] A role separation module is used to determine the role separation result of the speech data to be processed based on the multiple target fusion vectors and the neural network model.

[0049] To achieve the above-mentioned objectives, the third aspect of an embodiment of the present application proposes a computer device, which includes a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the method described in the first aspect above is implemented.

[0050] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in the first aspect above.

[0051] The present application proposes a method, device, equipment, and medium for role separation based on speech data. By extracting the voiceprint vector and text vector of each speech segment, the preliminary fusion vector of the speech segment can be determined. The preliminary fusion vector includes both the voiceprint information and the semantic information of the speech segment. The preliminary fusion vector is input into a preset neural network model. The neural network model calculates vector weights for each preliminary fusion vector based on the attention mechanism to obtain multiple target fusion vectors, and then the separation results of the speech data to be processed are obtained through the target fusion vectors and the neural network model. The embodiment of the present application realizes role separation through the joint action of the text vector of the speech data, the voiceprint vector and the attention mechanism of the neural network model. The embodiment of the present application specifically includes the following advantages: First, more text information is added, and the information source path is expanded. Second, based on the attention mechanism, the role separation task is better achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flow chart of a method for separating roles based on voice data provided by an embodiment of the present application;

[0053] Figure 2 yes Figure 1 Flowchart of step S140 in FIG.

[0054] Figure 3 yes Figure 1 Flowchart of step S140 in FIG.

[0055] Figure 4 yes Figure 1 Flowchart of step S150 in FIG.

[0056] Figure 5 yes Figure 1 Flowchart of step S150 in FIG.

[0057] Figure 6 This is a module structure diagram of a device for separating roles based on voice data provided by an embodiment of the present application;

[0058] Figure 7 This is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0060] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0062] First, let’s analyze some of the terms used in this application:

[0063] Artificial Intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and create new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses theories, methods, technologies, and application systems that use digital computers or digital computer-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0064] Natural Language Processing (NLP): NLP uses computers to process, understand, and apply human languages ​​(such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information image processing, information extraction and filtering, text classification and clustering, public opinion analysis, and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0065] Information Extraction (NER): A text processing technology that extracts specified types of entity, relationship, event, and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of some specific units, such as sentences, paragraphs, and chapters. Text information is composed of some small specific units, such as characters, words, phrases, sentences, paragraphs, or a combination of these specific units. Extracting noun phrases, names, place names, etc. from text data is all text information extraction. Of course, the information extracted by text information extraction technology can be of various types.

[0066] Corpus: Language materials. Corpus is the basic unit of a corpus and is typically a collection of text resources of a certain quantity and size. Corpus size can range from tens of millions, even hundreds of millions of sentences, to just a few hundred. People simply use text as a proxy, and the contextual relationships within text serve as a proxy for the contextual relationships in real-world language. A collection of texts can be called a corpus, and when there are several such collections, it can be called a collection of corpora. The internet itself is a vast and complex corpus. Corpora can be categorized in many ways based on different criteria. For example, corpora can be monolingual or multilingual.

[0067] The BERT model (Bidirectional Encoder Representations from Transformers) is a deep learning model based on the Transformer architecture and encoders. After being pre-trained on unlabeled training data, the BERT model only requires a small amount of training on sample data for specific downstream processing tasks before being applied to them. This feature makes the BERT model well-suited for applications in fields such as natural language processing (NLP).

[0068] Currently, in the field of phone calls, it is often necessary to distinguish between the two or more parties in a call. Only by correctly distinguishing the text in the speech can better results be achieved in the subsequent natural language processing tasks. For example, if the customer's words are mistakenly classified as the agent's words in the speech, it will have a certain impact on the judgment of the compliance of the agent's speech. In related technologies, considering that each person's speaking voice is different, the roles in the speech are generally separated by the voiceprint vector in the speech. However, if the speaking voices of the characters in the speech are similar, that is, there are different roles but similar voiceprint vectors, the accuracy of role separation achieved only by voiceprint vectors is not high enough, and there is a possibility of misjudgment. Therefore, how to provide a role separation method based on speech data that can improve the separation accuracy of role separation has become a technical problem that needs to be solved urgently.

[0069] Based on this, the main purpose of the embodiments of this application is to propose a role separation method and device, equipment, and medium based on voice data, aiming to achieve role separation through the joint action of the text vector of voice data, voiceprint vector and the attention mechanism of the neural network model. This embodiment can improve the separation accuracy of role separation.

[0070] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0071] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0072] The embodiment of the present application provides a method for role separation based on voice data, which relates to the field of artificial intelligence technology. The embodiment of the present application provides a method for role separation based on voice data that can be applied to a terminal, or to a server, or it can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements a method for role separation based on voice data, etc., but is not limited to the above forms.

[0073] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0074] The embodiments of the present application provide a method, apparatus, device, and medium for role separation based on voice data, which are specifically illustrated by the following embodiments. First, a method for role separation based on voice data in the embodiments of the present application is described.

[0075] Figure 1 This is an optional flowchart of a method for separating roles based on voice data provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S110 to S170.

[0076] Step S110, obtaining voice data to be processed, where the voice data to be processed includes voice data of at least two characters;

[0077] Step S120, pre-processing the speech data to be processed to obtain a plurality of speech segments;

[0078] Step S130, performing voiceprint recognition on each speech segment to obtain a voiceprint vector for each speech segment;

[0079] Step S140: convert each speech segment into a text segment, and perform semantic recognition on each text segment to obtain a text vector for each speech segment;

[0080] Step S150, performing vector calculation on the voiceprint vector and text vector belonging to the same speech segment to obtain a preliminary fusion vector;

[0081] Step S160: Input the obtained multiple preliminary fusion vectors into a preset neural network model. The neural network model calculates a vector weight for each preliminary fusion vector based on an attention mechanism to obtain multiple target fusion vectors.

[0082] Step S170: determining a character separation result of the speech data to be processed based on the multiple target fusion vectors and the neural network model.

[0083] In steps S110 to S170 illustrated in this embodiment of the present application, character separation is achieved through the combined action of the text vectors and voiceprint vectors of the speech data and the attention mechanism of the neural network model. This embodiment can improve the accuracy of character separation. Specifically, by extracting the voiceprint vector and text vector of each speech segment, a preliminary fusion vector for that speech segment can be determined. This preliminary fusion vector includes both the voiceprint information and the semantic information corresponding to the speech segment. It should be noted that each speech segment includes at least one sentence. The preliminary fusion vector is input into a preset neural network model. The neural network model calculates vector weights for each preliminary fusion vector based on the attention mechanism to obtain multiple target fusion vectors. The target fusion vectors and the neural network model are then used to obtain the separation results of the processed speech data. For example, the preliminary fusion vector is input into the encoder of a transformer. The transformer encoder utilizes the attention mechanism and can calculate the preliminary fusion vector of each speech segment with the preliminary fusion vectors of multiple speech segments to obtain vector weights. In theory, the weight values ​​will be higher for similar sentences, which helps improve the accuracy of character separation. It is understandable that the embodiment of the present application can combine voice and text information to implement the role separation task for the information of the entire call. It has the following advantages: First, more text information is added, and the information source is expanded. Second, based on the attention mechanism, the role separation task can be better implemented. It should be noted that the Attention mechanism is a method of extracting specific vectors from a vector expression set according to certain rules or certain additional information for weighted combination, and strengthening the attention to certain important words by assigning different weights to different word vectors. In the embodiment of the present application, different preliminary fusion vectors (sentence vectors) are weighted according to the Attention mechanism to obtain the target fusion vector.

[0084] In step S110 of some embodiments, the voice data to be processed refers to voice data containing at least two characters. The voice data to be processed can be directly obtained from existing corpus, or voice data can be collected to obtain the voice data to be processed. It is understood that the voice data can be collected on-site using a microphone, using a recording device, or remotely using a communication device such as a mobile phone.

[0085] In step S120 of some embodiments, the voice data to be processed is preprocessed to obtain multiple voice segments. Generally, the voice data to be processed corresponds to the voice data of the entire call, which contains at least two characters and includes many pauses and noises. The voice data to be processed is preprocessed, specifically including segmenting the voice data to be processed to obtain multiple preliminary voice segments; removing preliminary voice segments with a text length of less than 3 words, preliminary voice segments with garbled characters after text conversion, preliminary voice segments with a recording time of less than 3 seconds, and preliminary voice segments with an average single-word duration of less than 0.1 seconds to obtain multiple voice segments.

[0086] In step S130 of some embodiments, voiceprint recognition is performed on each speech segment to obtain a voiceprint vector for each speech segment. Specifically, voiceprint recognition is performed on each speech segment using a preset voiceprint model to obtain a voiceprint vector. For example, the voiceprint vector of each speech segment is extracted using an XVector model, and the original dimension of the voiceprint vector is 1×512. It is understandable that the preset voiceprint model can also be a Gaussian mixture model, a hidden Markov model, a dynamic time warping model, a vectorization model, etc., which is not limited in the embodiments of the present application. The extraction of voiceprint vectors can also be achieved by constructing a voiceprint model. For example, the voiceprint features of the speech data are first extracted, and then a voiceprint model is constructed based on the acoustic features, such as constructing a voiceprint model based on a speaker factor vector.

[0087] In step S140 of some embodiments, each speech segment is converted into a text segment, and semantic recognition is performed on each text segment to obtain a text vector for each speech segment. Specifically, each text segment corresponds to at least one sentence, and the text vector of each speech segment can also be called a preliminary sentence vector. A preliminary sentence vector can be extracted for each text using a preset semantic recognition model. For example, a preliminary sentence vector is extracted for a text segment using a pre-trained neural network model (BERT model), and the original dimension of the preliminary sentence vector is 1×768.

[0088] Specifically, refer to Figure 2 In some embodiments, step S140 includes but is not limited to steps S210 to S240:

[0089] Step S210 , converting each speech segment into a text segment, and performing semantic recognition on each text segment to obtain a preliminary text vector;

[0090] Step S220, constructing a vector matrix based on the obtained multiple preliminary text vectors, and determining the neighboring preliminary text vectors of each preliminary text vector through the vector matrix;

[0091] Step S230 , calculating the similarity between each preliminary text vector and adjacent preliminary text vectors;

[0092] Step S240 , applying weights to the preliminary text vectors according to the similarity to obtain a text vector corresponding to each speech segment.

[0093] In steps S210 to S240 shown in the embodiment of the present application, multiple preliminary text vectors are mapped into a vector matrix, taking into account the upper and lower semantic information between multiple preliminary text vectors, wherein the dimension of the vector matrix can be set according to actual needs, and the embodiment of the present application does not make specific limitations. The vector matrix can be used to determine the adjacent preliminary texts of each preliminary text vector, and the similarity between each preliminary text vector and the adjacent preliminary text vector can be calculated. The degree of semantic association between each preliminary text vector and the adjacent preliminary text vector can be determined by the size of the similarity, and weights are applied to the preliminary text vectors according to the similarity, and finally the text vector of each speech segment is obtained. In actual applications, weights can be applied to the preliminary text vectors in proportion to the size of the obtained similarity, and the specific weight values ​​are not specifically limited in the embodiment of the present application.

[0094] Specifically, refer to Figure 3 In some embodiments, step S140 includes but is not limited to steps S310 to S340:

[0095] Step S310: convert each speech segment into a text segment, and perform semantic recognition on each text segment to obtain a preliminary text vector for each speech segment;

[0096] Step S320: Input the text segment into a preset text regularization model. The text regularization model regularizes the word components of the text segment based on grammatical rules to obtain a preliminary text segment.

[0097] Step S330, calculating the similarity between the text segment and the preliminary text segment;

[0098] Step S340 , weighting the preliminary text vectors according to the similarity to obtain a text vector for each speech segment.

[0099] In the steps S310 to S340 shown in the embodiment of the present application, considering that each role has different grammatical habits when speaking, the grammatical structure of the text segment can be identified by a preset text regularization model, and a new text segment, i.e., a preliminary text segment, can be regenerated. The similarity between the text segment and the preliminary text is used to represent the grammatical difference of each text segment based on the standard grammatical structure. Therefore, weights are applied to the preliminary text vector according to the similarity, with the purpose of distinguishing the different grammatical habits of different roles. Specifically, the text regularization model can adopt a neural network model, for example, including: an input layer, a feature vector conversion layer, a BLSTM layer, a classifier layer, and an output layer. In an actual example, taking the voice data to be processed as the voice data including customers and agents as an example, after the voice segments are converted into text segments, the text segments of the agents are generally complete and the sentences are fluent, while the text segments of the customers are relatively incomplete and the grammar is chaotic. For example, there are the following two text segments: "Do you have any other business to handle?" and "Business A, I want to know about it." After the above two text segments are regularized by the text regularization model, the following two preliminary text segments are obtained: "Do you have any other business to handle?" and "I want to know about Business A." It can be found that the first text segment and the first preliminary text segment are almost the same, and the calculated similarity value is high, while the second text segment and the second preliminary text segment have different word component orders, and the calculated similarity value is low. Therefore, adding weights to the preliminary text vectors by similarity is beneficial to improving the information representation ability of the text vector of each speech segment, so as to improve the accuracy of subsequent role separation.

[0100] In step S150 of some embodiments, vector calculation is performed on the voiceprint vector and the text vector belonging to the same speech segment to obtain a preliminary fusion vector, which specifically includes one of the following steps:

[0101] Perform vector concatenation on the voiceprint vector and text vector belonging to the same speech segment to obtain a preliminary fusion vector;

[0102] Alternatively, the voiceprint vector and the text vector belonging to the same speech segment are summed, and then the average of the summed results is taken to obtain a preliminary fusion vector.

[0103] Specifically, a preliminary fusion vector is generated by fusing the voiceprint vector with the text vector. This preliminary fusion vector includes both the voiceprint information and the semantic information corresponding to the speech segment. This preliminary fusion information helps improve the accuracy of subsequent character separation of the speech data.

[0104] Specifically, refer to Figure 4 In some embodiments, step S150 includes but is not limited to steps S410 to S440:

[0105] Step S410, calculating the speech duration corresponding to each speech segment;

[0106] Step S420, calculating the number of characters corresponding to each text segment;

[0107] Step S430, determining a single word duration vector based on the speech duration and the number of characters belonging to the same speech segment;

[0108] Step S440 , performing vector calculation on the single word duration vector, voiceprint vector, and text vector belonging to the same speech segment to obtain a preliminary fusion vector.

[0109] In steps S410 to S440 shown in the embodiment of the present application, the speaking characteristics of different characters are taken into consideration, such as different pauses in speaking. The embodiment of the present application uses single-word duration vectors to represent the speaking characteristics of different characters. It can be understood that integrating the single-word duration vectors into the preliminary fusion vector can improve the information representation ability of the preliminary fusion vector, and further help to improve the accuracy of subsequent character separation.

[0110] Specifically, refer to Figure 5 In some embodiments, step S150 includes but is not limited to steps S510 to S550:

[0111] Step S510, obtaining the original voiceprint dimension of the voiceprint vector and the original text dimension of the text vector;

[0112] Step S520: If the original voiceprint dimension is different from the original text dimension, the original voiceprint dimension and the original text dimension are processed by a preset linear transformation method to obtain a target dimension;

[0113] Step S530, updating the dimension of the voiceprint vector according to the target dimension to obtain the target voiceprint vector;

[0114] Step S540, updating the dimension of the text vector according to the target dimension to obtain a target text vector;

[0115] Step S550: obtaining a preliminary fusion vector according to the target voiceprint vector and the target text vector.

[0116] In steps S510 to S550 shown in the embodiment of the present application, considering that the dimension in the calculation process between vectors will affect the calculation results, the original voiceprint dimension and the original text dimension are processed by a linear transformation method in the embodiment of the present application to obtain the target dimension. For example, the original voiceprint dimension is 1×512, and the original text dimension is 1×768. The target dimension that can be obtained by the linear transformation method is 1×640. The dimension of the voiceprint vector is updated according to the target dimension, and the dimension size of the target voiceprint vector obtained is the target dimension. The dimension of the text vector is updated according to the target dimension, and the dimension size of the target text vector obtained is the target dimension. In the experiment, it was found that by unifying the dimension of the text vector and the dimension of the voiceprint vector and then performing vector calculation, the preliminary fusion vector obtained can better represent the role information of the speech segment, which helps to improve the accuracy of subsequent role separation.

[0117] In step S160 of some embodiments, the obtained multiple preliminary fusion vectors are input into a preset neural network model, and the neural network model calculates a vector weight for each preliminary fusion vector based on the attention mechanism to obtain multiple target fusion vectors. Specifically, the attention mechanism, also known as the Attention mechanism, is a method of extracting specific vectors from a vector expression set for weighted combination according to certain rules or certain additional information, and strengthening the degree of attention to certain important words by assigning different weights to different word vectors. The multiple preliminary fusion vectors in the embodiment of the present application are obtained by the voiceprint vector and text vector corresponding to the speech segment, so the preliminary fusion vector specifically refers to the sentence vector, not the word vector. The embodiment of the present application introduces the Attention mechanism to calculate the sentence vector of each speech segment and the sentence vector of the entire call (voice data to be processed), obtain the sentence vector i and the weights of N sentence vectors, and further obtain the target fusion vector. It should be noted that for similar sentence vectors, the weight value will be higher, so it is beneficial for separating the identities of the characters.

[0118] In step S170 of some embodiments, determining a character separation result of the speech data to be processed based on the multiple target fusion vectors and the neural network model specifically includes the following steps:

[0119] Map each target fusion vector to N categories through the MLP fully connected layer to obtain the classification result, where N is a positive integer;

[0120] The probability value of the classification results is calculated through the softmax layer to obtain N probability values ​​for each classification result;

[0121] The role of each speech segment is determined according to the N probability values ​​to obtain a role separation result of the speech data to be processed.

[0122] Specifically, N is a preset value and can be set according to actual needs. For example, if the voice data to be processed includes two roles, N is set to 2. If the voice data to be processed includes 3 roles, N is set to 3. For example, if N is 2, each target fusion vector is mapped to 2 classifications through the MLP fully connected layer, and the classification result includes two role information, specifically [A, B], A refers to role A, and B refers to role B. The probability value of the classification result is calculated through the softmax layer to obtain 2 probability values ​​in each classification result, such as [0.6, 0.4]. The role of the current voice segment is determined according to the role corresponding to the maximum probability value. For example, in this embodiment, the role of the current voice segment is determined by role A corresponding to the probability value of 0.6. When the role separation of all voice segments in the voice data to be processed is completed, the role separation result of the voice data to be processed is obtained.

[0123] Specifically, in some embodiments, the method further includes but is not limited to the following steps:

[0124] Calculate the probability difference between N probability values. If the probability difference is less than a preset difference threshold, obtain the text vector and voiceprint vector of the speech segment.

[0125] Input the text vector and the voiceprint vector into a preset neural network model. The neural network model calculates the corresponding vector weight for the text vector based on the attention mechanism to obtain a weighted text vector. The neural network model calculates the corresponding vector weight for the voiceprint vector based on the attention mechanism to obtain a weighted voiceprint vector. The target fusion vector is obtained according to the weighted text vector and the weighted voiceprint vector.

[0126] Map each target fusion vector to N categories through the MLP fully connected layer to obtain the classification result, where N is a positive integer;

[0127] The probability value of the classification results is calculated through the softmax layer to obtain N probability values ​​for each classification result;

[0128] The role of each speech segment is determined according to the N probability values ​​to obtain a role separation result of the speech data to be processed.

[0129] It is understandable that if the probability difference is less than the preset difference threshold, it means that the current role separation effect is not obvious and the role separation is not accurate enough. Therefore, in the embodiment of the present application, by performing separate vector weight calculations on the text vector and voiceprint vector of the speech segment, the similarity information between the vectors can be further retained. The vector calculation based on the weighted text vector and weighted voiceprint vector is used to obtain the target fusion vector, which can further improve the role separation effect and improve the accuracy of role separation of speech data.

[0130] See also Figure 6The embodiment of the present application also provides a role separation device based on voice data, which can implement the above-mentioned role separation method based on voice data. Figure 6 This is a module structure block diagram of a role separation device based on voice data provided in an embodiment of the present application, which includes: an acquisition module 610, a preprocessing module 620, a voiceprint extraction module 630, a text extraction module 640, a vector calculation module 650, a weight calculation module 660, and a role separation module 670. Among them, the acquisition module 610 is used to obtain the voice data to be processed, and the voice data to be processed includes voice data of at least two roles; the preprocessing module 620 is used to preprocess the voice data to be processed to obtain multiple voice segments; the voiceprint extraction module 630 is used to perform voiceprint recognition on each of the voice segments to obtain a voiceprint vector for each of the voice segments; the text extraction module 640 is used to convert each of the voice segments into a text segment, and perform semantic recognition on each of the text segments to obtain a text vector for each of the voice segments; the vector calculation module 650 is used to perform vector calculation on the voiceprint vector and the text vector belonging to the same voice segment to obtain a preliminary fusion vector; the weight calculation module 660 is used to input the obtained multiple preliminary fusion vectors into a preset neural network model, and the neural network model calculates the vector weight for each of the preliminary fusion vectors based on the attention mechanism to obtain multiple target fusion vectors; the role separation module 670 is used to determine the role separation result of the voice data to be processed based on the multiple target fusion vectors and the neural network model.

[0131] A voice data-based role separation device, according to an embodiment of the present application, achieves role separation through the combined effects of the voice data's text vectors, voiceprint vectors, and the attention mechanism of a neural network model. This embodiment improves the accuracy of role separation. Specifically, by extracting the voiceprint vector and text vector of each voice segment, a preliminary fusion vector for that segment is determined. This preliminary fusion vector includes both the voiceprint information and the semantic information corresponding to the segment. The preliminary fusion vector is input into a preset neural network model, which calculates vector weights for each preliminary fusion vector based on the attention mechanism to obtain multiple target fusion vectors. The target fusion vectors and the neural network model are then used to determine the separation results for the processed voice data. This embodiment of the present application combines voice and text information to perform role separation on the entire call. It offers the following advantages: First, it adds text information, expanding the information source. Second, it utilizes the attention mechanism to better implement the role separation task.

[0132] It should be noted that the specific implementation of the role separation device based on voice data is basically the same as the specific embodiment of the above-mentioned role separation method based on voice data, and will not be repeated here.

[0133] The present application also provides a computer device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. When the program is executed by the processor, the aforementioned method for role separation based on voice data is implemented. The computer device can be any intelligent terminal, including a tablet computer and an in-vehicle computer.

[0134] See also Figure 7 , Figure 7 The hardware structure of a computer device according to another embodiment is shown. The computer device includes:

[0135] The processor 701 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0136] The memory 702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 702 and is called by the processor 701 to execute a role separation method based on voice data in the embodiments of this application.

[0137] Input / output interface 703, used to implement information input and output;

[0138] Communication interface 704, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0139] Bus 705 , which transmits information between various components of the device (e.g., processor 701 , memory 702 , input / output interface 703 , and communication interface 704 );

[0140] The processor 701 , the memory 702 , the input / output interface 703 and the communication interface 704 are connected to each other in communication within the device via a bus 705 .

[0141] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned role separation method based on voice data.

[0142] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0143] Embodiments of the present application provide a method, device, and storage medium for role separation based on speech data. These methods achieve role separation through the combined effects of text vectors, voiceprint vectors, and the attention mechanism of a neural network model. This method improves the accuracy of role separation. Specifically, by extracting the voiceprint vector and text vector of each speech segment, a preliminary fusion vector for that segment is determined. This preliminary fusion vector includes both the voiceprint information and semantic information corresponding to the segment. The preliminary fusion vector is input into a preset neural network model, which calculates vector weights for each preliminary fusion vector based on the attention mechanism to obtain multiple target fusion vectors. The target fusion vectors are then combined with the neural network model to obtain the separation results for the processed speech data. Embodiments of the present application combine speech and text information to perform role separation on the entire call. This method offers the following advantages: First, it adds text information, expanding the information source. Second, it utilizes the attention mechanism to better achieve role separation. It should be understood that the attention mechanism in this application is equivalent to the attention mechanism.

[0144] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0145] It will be understood by those skilled in the art that Figure 1-5The technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figures, or a combination of certain steps, or different steps.

[0146] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0147] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0148] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0149] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0150] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0151] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0152] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0153] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0154] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A method for character separation based on speech data, characterized in that: The method comprises: Acquire voice data to be processed, where the voice data to be processed includes voice data of at least two characters; Preprocessing the speech data to be processed to obtain a plurality of speech segments; Performing voiceprint recognition on each of the speech segments to obtain a voiceprint vector for each of the speech segments; Converting each of the speech segments into a text segment, and performing semantic recognition on each of the text segments to obtain a text vector for each of the speech segments; Performing vector calculation on the voiceprint vector and the text vector belonging to the same speech segment to obtain a preliminary fusion vector; Inputting the obtained multiple preliminary fusion vectors into a preset neural network model, and calculating a corresponding vector weight for each of the preliminary fusion vectors through the attention mechanism of the neural network model to obtain multiple target fusion vectors; Mapping each of the target fusion vectors to N categories through the MLP fully connected layer of the neural network model to obtain a classification result, where N is a positive integer; Calculating the probability values ​​of the classification results through the softmax layer of the neural network model to obtain N probability values ​​for each classification result; The role of each of the speech segments is determined according to the N probability values ​​to obtain a role separation result of the speech data to be processed.

2. The method according to claim 1, characterized in that The converting each of the speech segments into a text segment and performing semantic recognition on each of the text segments to obtain a text vector for each of the speech segments includes: Converting each of the speech segments into a text segment, and performing semantic recognition on each of the text segments to obtain a preliminary text vector; constructing a vector matrix based on the obtained multiple preliminary text vectors, and determining the neighboring preliminary text vectors of each preliminary text vector through the vector matrix; Calculating the similarity between each of the preliminary text vectors and the adjacent preliminary text vectors; A weight is applied to the preliminary text vector according to the similarity to obtain a text vector for each of the speech segments.

3. The method according to claim 1, characterized in that The converting each of the speech segments into a text segment and performing semantic recognition on each of the text segments to obtain a text vector for each of the speech segments includes: Converting each of the speech segments into a text segment, and performing semantic recognition on each of the text segments to obtain a preliminary text vector for each of the speech segments; Inputting the text segment into a preset text regularization model, wherein the text regularization model regularizes the word components of the text segment based on grammatical rules to obtain a preliminary text segment; calculating the similarity between the text segment and the preliminary text segment; A weight is applied to the preliminary text vector according to the similarity to obtain a text vector for each of the speech segments.

4. The method according to claim 1, wherein The performing vector calculation on the voiceprint vector and the text vector belonging to the same speech segment to obtain a preliminary fusion vector includes: Performing vector concatenation on the voiceprint vector and the text vector belonging to the same speech segment to obtain a preliminary fusion vector; or, Perform vector summation on the voiceprint vector and the text vector belonging to the same speech segment, and then take an average of the summation results to obtain a preliminary fusion vector.

5. The method according to claim 1, wherein The performing vector calculation on the voiceprint vector and the text vector belonging to the same speech segment to obtain a preliminary fusion vector includes: Calculating the speech duration of each speech segment; Counting the number of characters in each of the text segments; Determine a single word duration vector according to the speech duration and the number of characters belonging to the same speech segment; Vector calculation is performed on the single word duration vector, the voiceprint vector and the text vector belonging to the same speech segment to obtain a preliminary fusion vector.

6. The method according to any one of claims 1 to 5, characterized in that The performing vector calculation on the voiceprint vector and the text vector belonging to the same speech segment to obtain a preliminary fusion vector includes: Obtaining the original voiceprint dimension of the voiceprint vector and obtaining the original text dimension of the text vector; If the original voiceprint dimension is different from the original text dimension, processing the original voiceprint dimension and the original text dimension by a preset linear transformation method to obtain a target dimension; Updating the dimension of the voiceprint vector according to the target dimension to obtain a target voiceprint vector; Updating the dimension of the text vector according to the target dimension to obtain a target text vector; A preliminary fusion vector is obtained according to the target voiceprint vector and the target text vector.

7. A role separation device based on voice data, characterized in that: The device comprises: An acquisition module, configured to acquire voice data to be processed, wherein the voice data to be processed includes voice data of at least two characters; A preprocessing module, configured to preprocess the speech data to be processed to obtain a plurality of speech segments; a voiceprint extraction module, configured to perform voiceprint recognition on each of the speech segments to obtain a voiceprint vector for each of the speech segments; A text extraction module, configured to convert each of the speech segments into a text segment and perform semantic recognition on each of the text segments to obtain a text vector for each of the speech segments; a vector calculation module, configured to perform vector calculation on the voiceprint vector and the text vector belonging to the same speech segment to obtain a preliminary fusion vector; A weight calculation module is used to input the obtained multiple preliminary fusion vectors into a preset neural network model, and calculate a vector weight for each of the preliminary fusion vectors through the attention mechanism of the neural network model to obtain multiple target fusion vectors; Role separation module for: Mapping each of the target fusion vectors to N categories through the MLP fully connected layer of the neural network model to obtain a classification result, where N is a positive integer; Calculating the probability values ​​of the classification results through the softmax layer of the neural network model to obtain N probability values ​​for each classification result; The role of each of the speech segments is determined according to the N probability values ​​to obtain a role separation result of the speech data to be processed.

8. A computer device, characterized in that: The computer device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for implementing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A storage medium, which is a computer-readable storage medium and is used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any one of claims 1 to 6.

Images