Speech separation method and electronic device for one doctor with multiple patients

Abstract

The present invention relates to the field of speech signal processing technology, and in particular, provides a method and electronic device for voice separation of one doctor with multiple patients. The method includes inputting multiple consultation source audios of different patients corresponding to the same doctor into a voiceprint segmentation clustering model to obtain an RTTM file corresponding to each consultation source audio; cutting and splicing the corresponding consultation original audio according to the RTTM file to obtain a sub-separated audio corresponding to each consultation original audio; performing voiceprint recognition on the sub-separated audio corresponding to each consultation original audio to determine that the audio that appears multiple times is the doctor's audio and the audio that appears once is the patient's audio; splicing the doctor's audio to obtain multiple separated audios of different patients and doctors, and naming each separated audio with an identity and a speaker label. While realizing the separation of the communication audio between doctors and patients, the method can obtain voice samples according to time series labels and determine the identity information of the speaker.

Description

Technical Field

[0001] The present invention relates to the technical field of speech signal processing, and in particular to a speech separation method and electronic equipment for one doctor with multiple patients. Background Art

[0002] When performing speech recognition and natural language processing on the audio of communication between doctors and patients, it is crucial to distinguish the audio content of doctors and patients. For example, due to the lack of professional medical knowledge, the patient's description of the disease is not accurate, and the final treatment plan must be formulated by the doctor. Differentiating the voices of doctors and patients through voiceprint segmentation and clustering can help us better generate structured information to improve electronic medical records or perform automatic diagnosis and analysis.

[0003] In addition, speech data contains acoustic and linguistic features that can be recognized by machine learning models to measure the behavioral health of the speaker. Moreover, the collection of speech data is relatively simple, convenient and inexpensive for patients, requiring only a microphone, a quiet place and a device to collect audio samples. Therefore, speech-based biomarkers can pre-screen diseases, monitor disease progression and response to treatment, and become useful surrogate markers for clinical research with informed consent, but the premise of this process is also that the audio of doctors and patients needs to be distinguished when collecting audio samples. In actual situations, the voices of most doctors and patients are not collected separately, but mixed together. In the prior art, the speaker's log is usually used to separate sounds in the time domain. However, this separation method cannot obtain speech samples based on time series labels, and cannot determine the speaker's identity information. Summary of the invention

[0004] In view of this, the present invention provides a speech separation method and electronic device for one doctor and multiple patients, which can separate the communication audio between the doctor and the patient, obtain speech samples according to time series labels, and determine the identity information of the speaker.

[0005] In a first aspect, the present invention provides a method for speech separation of multiple patients with one doctor, the method comprising:

[0006] Step 1: Input multiple consultation source audios of different patients corresponding to the same doctor into the voiceprint segmentation clustering model to obtain the RTTM file corresponding to each consultation source audio;

[0007] Step 2: Cut and splice the corresponding original audio of the medical consultation according to the RTTM file to obtain the sub-separated audio corresponding to each original audio of the medical consultation;

[0008] Step three: perform voiceprint recognition on the sub-separated audio corresponding to each original consultation audio, determine that the audio that appears multiple times is the doctor's audio, and the audio that appears once is the patient's audio; splice the doctor's audio to obtain multiple separated audios of different patients and doctors, and name each of the separated audios with an identity and speaker label.

[0009] Optionally, the step one includes:

[0010] Step S11, using a voice activity detection model, dividing each audio frame of the medical consultation source into two categories: voice signal audio and non-voice signal audio;

[0011] Step S12, dividing the speech signal audio into N segments according to the two categories classified in step S11, and detecting speaker switching in the speech signal audio to obtain a time point at which the speaker switching occurs, wherein N is a positive integer;

[0012] Step S13, calculating the voiceprint embedding codes corresponding to the N segments according to the N segments divided in step S12;

[0013] Step S14, performing cluster analysis on each voiceprint embedding code in step S13, classifying each voiceprint embedding code into a category, and obtaining preliminary data of voiceprint segmentation and clustering, wherein one category corresponds to one speaker;

[0014] Step S15, using the preliminary data in step S14 as training data to obtain a trained classifier model; performing secondary segmentation on the speech signal audio to obtain M segments; applying the classifier model to the M segments to obtain classification data of voiceprint segmentation and clustering, wherein M is a positive integer and M>N;

[0015] Step S16: Save the start time, end time and speaker label corresponding to the M segments in step S15 into an RTTM file to obtain an RTTM file.

[0016] Optionally, the step 2 includes:

[0017] Step S21, according to the start time, end time and speaker tag in the RTTM file, save the single start time and end time in the loop body;

[0018] Step S22, cutting the corresponding original audio of the medical consultation according to the start time and end time in step S21, and saving a temporary segmentation file;

[0019] Step S23, determining whether there is an audio file named with the speaker tag in step S21 in the storage directory;

[0020] Step S24: if it is determined that there is no audio file named with the speaker tag in the storage directory, the temporary segmented file in step S22 is named and saved according to the speaker tag;

[0021] Step S25: If it is determined that there is an audio file named with the speaker tag in the save directory, then the temporarily segmented file in step S22 is spliced ​​after the existing audio file named with the speaker tag; continue to execute step S21, and after traversing the start time, end time and speaker tag in the RTTM file, two sub-audio files named with the speaker tag are obtained.

[0022] Optionally, the step three includes:

[0023] Step S31, extracting any 3-second continuous segment from the sub-separated audio corresponding to each original audio of the medical consultation;

[0024] Step S32, extracting the voiceprint embedded code in any 3-second continuous segment in step S31, and performing cluster analysis on the voiceprint embedded code to obtain the identity identification serial number of each time segment;

[0025] Step S33: concatenate the sub-audio files in which the identity appears repeatedly, and name the files with the doctor and the identity identification number; for the sub-audio files in which the identity appears only once, name the files with the patient and the identity identification number.

[0026] Optionally, the voice activity detection model in step S11 adopts a Gaussian mixture model (GMM-VAD);

[0027] Gaussian mixture model for voice activity detection The formula is:

[0028] ;

[0029] in, is the weight of the speech signal; is the weight of non-speech signal; is the mean value of the speech signal; is the variance of the speech signal; is the mean of non-speech signals; is the variance of the non-speech signal.

[0030] Optionally, the speaker switch detection in step S12 adopts left-right window comparison, wherein the left-right window comparison includes:

[0031] Step S121, for each audio frame of the consultation source, two left and right windows of the same size are selected, and the voiceprint embedding codes of the left and right windows and the cosine similarity of the left and right voiceprint embedding codes are calculated to obtain a two-dimensional curve with the time of the audio frame as the x-axis and the cosine similarity as the y-axis;

[0032] Step S122, performing peak detection on the two-dimensional curve in step S121, selecting all local minimum points, and determining whether the cosine similarity corresponding to the local minimum points is less than a specific threshold;

[0033] Step S123: If the cosine similarity corresponding to the local minimum point at the judgment location is less than a specific threshold, the speaker is switched.

[0034] Optionally, the calculation of the voiceprint embedding code in step S13 adopts factor analysis based on the identity vector i-vector. Given a speaker's speech segment, the corresponding Gaussian mean supervector is defined as:

[0035] ;

[0036] Where s is the Gaussian mean supervector of a given speaker's speech segment; m is a supervector that is independent of both the speaker and the channel; T is the global difference space matrix; and w is a random vector that follows a standard multivariate normal distribution. , the dimension is usually selected between 400 and 600.

[0037] Optionally, the cluster analysis in step S14 adopts K-means clustering; the K-means clustering includes:

[0038] Step S141: N random vectors Divide into K subsets S = {S1, •••, Sk}, so that the sum of squares within the class is minimized, and select the centers of the initial K classes, where 1≤i≤N;

[0039] Step S142, calculating the cosine similarity between each data and the center of the K classes in step S141, and assigning it to the class with the highest cosine similarity;

[0040] Step S143, according to the data assigned to each class in step S142, calculate the mean vector of the data, take it as the center of the class, and continue to execute step S142 until the convergence criterion is met and the change of the sum of squares within the class is less than the threshold.

[0041] In a second aspect, the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is run, the electronic device where the computer-readable storage medium is located is controlled to execute the one-doctor-multiple-patient speech separation method in the first aspect or any possible implementation of the first aspect.

[0042] In a third aspect, the present invention provides an electronic device comprising: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, enable the electronic device to perform the one-doctor-multiple-patient speech separation method in the first aspect or any possible implementation of the first aspect.

[0043] In the technical solution provided by the present invention, the method includes inputting multiple consultation source audios of different patients corresponding to the same doctor into a voiceprint segmentation clustering model to obtain an RTTM file corresponding to each consultation source audio; cutting and splicing the corresponding consultation original audio according to the RTTM file to obtain a sub-separated audio corresponding to each consultation original audio; performing voiceprint recognition on the sub-separated audio corresponding to each consultation original audio to determine that the audio that appears multiple times is the doctor's audio and the audio that appears once is the patient's audio; splicing the doctor's audio to obtain multiple separated audios of different patients and doctors, and naming each separated audio with an identity and speaker label. While realizing the separation of the doctor-patient communication audio, the method can obtain voice samples according to time series labels and determine the identity information of the speaker. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 A flow chart of a method for voice separation of multiple patients provided by one doctor in an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of speech segmentation provided by an embodiment of the present invention;

[0047] Figure 3 A framework diagram of a voiceprint segmentation clustering model provided in an embodiment of the present invention;

[0048] Figure 4 A flowchart of a method for obtaining sub-separated audio provided by an embodiment of the present invention;

[0049] Figure 5 A flowchart of an identity recognition method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] It should be clear that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings.

[0053] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0054] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0055] Figure 1 A flowchart of a method for voice separation of multiple patients with one doctor provided in an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0056] Step 1: Input multiple consultation source audios of different patients corresponding to the same doctor into the voiceprint segmentation clustering model to obtain the RTTM file corresponding to each consultation source audio.

[0057] In the embodiment of the present invention, the RTTM file includes a start time, an end time, and a speaker tag.

[0058] RTTM files are space-delimited text files, with one lap per line and each line containing ten fields:

[0059] •type – The segment type; should always be SPEAKER.

[0060] • File ID – File name; base name of the record minus the extension (for example, rec1_a).

[0061] •Channel ID – Channel index; should always be 1.

[0062] • Transition start – The transition starts in seconds from the start of recording.

[0063] • Duration – Duration of the speech in seconds.

[0064] • Orthography field -- should always be <na>.

[0065] • Speaker Type - should always be <na>.

[0066] • Speaker Name - The name of the speaker for this turn; should be unique within the scope of each document.

[0067] • Confidence - the confidence (probability) that the system believes the information is correct; should always be <na>.

[0068] •Signal Advance Time - should always be <na>.

[0069] Figure 3 The framework diagram of the voiceprint segmentation clustering model provided by the embodiment of the present invention is as follows: Figure 3 As shown, step one includes:

[0070] Step S11: using a voice activity detection model, classify each consultation source audio frame frame by frame into two categories: voice signal audio and non-voice signal audio.

[0071] In the embodiment of the present invention, the voice activity detection model in step S11 adopts a Gaussian mixture model (GMM-VAD);

[0072] Gaussian mixture model for voice activity detection The formula is:

[0073] ;

[0074] in, is the weight of the speech signal; is the weight of non-speech signal; is the mean value of the speech signal; is the variance of the speech signal; is the mean of non-speech signals; is the variance of the non-speech signal.

[0075] In the embodiment of the present invention, the principle of using the Gaussian mixture model for speech detection is to assume that both the speech signal and the non-speech signal conform to the Gaussian distribution, and to assume that the non-speech signal is more stable than the speech signal and has less energy than the speech signal. That is, the mean variance of the non-speech signal is less than that of the speech signal. As shown in the above formula, GMM-VAD includes 6 parameters: the mean, variance and weight of the speech signal, and the mean, variance and weight of the non-speech signal. The signal is divided into 6 frequency bands: 80Hz~250Hz, 250Hz~500Hz, 500Hz~1KHz, 1KHz~2KHz, 2KHz~3KHz, 3KHz~4KHz. GMM is used to fit the signal in each frequency band. When the GMM-VAD object is initialized, the above 6 parameters will use the default values. When the signal comes, the similarity probability is calculated according to the current GMM model and it is determined whether the current frame is speech or noise; then, based on the judgment, the maximum likelihood estimation is used to update the above 6 parameters and update the GMM model.

[0076] Step S12: Segment the speech signal audio into N segments according to the two categories classified in step S11, and detect speaker switching in the speech signal audio to obtain the time point when the speaker switches, where N is a positive integer.

[0077] In the embodiment of the present invention, after completing the voice activity detection, the audio containing the voice signal needs to be segmented into a number of segments in order to calculate the voiceprint embedding code and perform clustering. The shorter the segment, the less information it contains; and the longer the segment, the more likely it is to contain multiple speakers. Through speaker transition detection, the specific time point when the speaker changes is detected, so that the segmented segments meet the above two conditions at the same time.

[0078] In the embodiment of the present invention, the speaker switch detection in step S12 adopts left-right window comparison, wherein the left-right window comparison includes:

[0079] Step S121, for each consultation source audio frame, two left and right windows of the same size are selected, and the voiceprint embedding codes of the left and right windows and the cosine similarity of the left and right voiceprint embedding codes are calculated to obtain a two-dimensional curve with the time of the audio frame as the x-axis and the cosine similarity as the y-axis.

[0080] Step S122: perform peak detection on the two-dimensional curve in step S121, select all local minimum points, and determine whether the cosine similarity corresponding to the local minimum points is less than a specific threshold.

[0081] Step S123: If the cosine similarity corresponding to the local minimum point at the judgment location is less than a specific threshold, the speaker is switched.

[0082] In the embodiment of the present invention, the detection density of the audio frequency frame is set to 100 milliseconds, the left and right window sizes are 2 seconds respectively, and the window for peak detection is 1 second.

[0083] Step S13: Calculate the voiceprint embedding codes corresponding to the N segments according to the N segments divided in step S12.

[0084] In the embodiment of the present invention, the calculation of the voiceprint embedding code in step S13 adopts factor analysis based on the identification vector (i-vector). Given a speaker's speech segment, the corresponding Gaussian mean supervector is defined as:

[0085] ;

[0086] Where s is the Gaussian mean supervector of a given speaker's speech segment; m is a supervector that is independent of both the speaker and the channel; T is the global difference space matrix; and w is a random vector that follows a standard multivariate normal distribution. , the dimension is usually selected between 400 and 600.

[0087] In the embodiment of the present invention, w is also called "identity vector", abbreviated as i-vector. By calculating the Baum-Welch statistic corresponding to the target speaker, estimating the global difference space matrix T, and calculating the posterior mean of w, i.e., i-vector.

[0088] Step S14: perform cluster analysis on each voiceprint embedded code in step S13, classify each voiceprint embedded code into a category, and obtain preliminary data of voiceprint segmentation and clustering, where one category corresponds to one speaker.

[0089] In the embodiment of the present invention, the cluster analysis in step S14 adopts K-means clustering; the K-means clustering includes:

[0090] Step S141: N random vectors Divide into K subsets S = {S1, •••, Sk} so that the sum of squares within the class is minimized, and select the centers of the initial K classes, where 1≤i≤N.

[0091] In the embodiment of the present invention, the value of K is 2.

[0092] Step S142: Calculate the cosine similarity between each data and the centers of the K classes in step S141, and assign it to the class with the highest cosine similarity.

[0093] In the embodiment of the present invention, the cosine distance formula of cosine similarity is:

[0094] ,

[0095] ,

[0096] in Represents x and The cosine value of the angle between two vectors can be used to indicate the similarity between the two vectors. The smaller the angle is, the closer it is to 0 degrees, and the closer the cosine value is to 1, the more their directions match, and the more similar they are; when the directions of the two vectors are completely opposite, the cosine of the angle takes the minimum value of -1; when the cosine value is 0, the two vectors are orthogonal, and the angle is 90 degrees.

[0097] Step S143, according to the data assigned to each class in step S142, calculate the mean vector of the data, take it as the center of the class, and continue to execute step S142 until the convergence criterion is met and the change of the sum of squares within the class is less than the threshold.

[0098] In the embodiment of the present invention, the optimized initialization strategy can greatly reduce the number of allocations and updates required for convergence. Therefore, the initialization strategy uses the K-Means++ method:

[0099] (1) Initial point: Randomly select one from all the data according to a uniform distribution as the center of the first class.

[0100] (2) Calculate the distance. For each point in the data, calculate its distance to the center of the existing class and take the minimum value of these distances.

[0101] (3) Random sampling. A point is randomly selected from all the data as the center of the next class, with a probability proportional to the square of the minimum distance calculated above.

[0102] (4) Iteration: Repeat steps 2 to 3 until all K centers are selected.

[0103] From the above steps, we can see that the advantage of the K-Means++ method is that it can avoid selecting points that are too close as the center of the initial class at the same time, and can take care of points that are far away from most of the data.

[0104] Step S15, using the preliminary data in step S14 as training data to obtain a trained classifier model; performing secondary segmentation on the speech signal audio to obtain M segments; applying the classifier model to the M segments to obtain classification data of voiceprint segmentation clustering, wherein M is a positive integer and M>N.

[0105] In the embodiment of the present invention, after the cluster analysis is completed, each segment has obtained the corresponding speaker, but the result at this time may still be not ideal. Secondary segmentation is a post-processing method after the clustering is completed, which is used to further adjust the boundaries of the segments and the clustering results to obtain better results.

[0106] Secondary segmentation uses the simplest secondary segmentation based on classification, that is, the results of the clustering algorithm are used as training data to train a classifier model, and then the classifier is applied to the same data to obtain the classification result. The initial segmentation and clustering results are , where 1≤i≤N, that is, the fixed-length audio segment corresponding to each speaker (there are N segments in total). Figure 2 As shown in Figure 2, features are extracted from a denser set of M > N audio clips. , where 1≤j≤M, find the speakers corresponding to these features and train a two-class classifier. Finally, apply the trained classifier to the M dense features {z j }, a new classification result is obtained as the final result of the secondary segmentation, and the start time, end time and speaker label corresponding to each segment segment are saved in the RTTM file.

[0107] Step S16: Save the start time, end time and speaker label corresponding to the M segments in step S15 into an RTTM file to obtain an RTTM file.

[0108] Step 2: Crop and splice the corresponding original audio of the consultation according to the RTTM file to obtain the sub-separated audio corresponding to each original audio of the consultation.

[0109] Step 3: Perform voiceprint recognition on the sub-separated audio corresponding to each original consultation audio to determine that the audio that appears multiple times is the doctor's audio and the audio that appears once is the patient's audio; splice the doctor's audio to obtain multiple separated audios of different patients and doctors, and name each of the separated audios with an identity and speaker label.

[0110] In the embodiment of the present invention, Figure 1 As shown, 00 and 01 are speaker labels or identity identification numbers, and the identity and speaker labels are such as doctor 00, patient 00, and patient 01.

[0111] Figure 4 A flowchart of a method for obtaining sub-separated audio provided by an embodiment of the present invention, such as Figure 4 As shown, step two includes:

[0112] Step S21: According to the start time, end time and speaker tag in the RTTM file, a single start time and end time are saved in the loop body.

[0113] Step S22: Cut the corresponding original audio of the medical consultation according to the start time and end time in step S21, and save the temporary segmented file.

[0114] Step S23, determine whether there is an audio file named with the speaker tag in step S21 in the storage directory, if not, execute step S24; if yes, execute step S25.

[0115] Step S24, naming and saving the temporary segmented files in step S22 according to the speaker labels;

[0116] Step S25, after the existing audio file named with the speaker tag, splice the temporarily segmented file in step S22; continue to execute step S21, after traversing the start time, end time and speaker tag in the RTTM file, obtain two sub-audio files named with the speaker tag.

[0117] In the embodiment of the present invention, the cropping and splicing function calls AudioSegment in the pydub function library, and the os library is called to determine the files with the same name in the directory.

[0118] Figure 5 A flowchart of an identity recognition method provided by an embodiment of the present invention, such as Figure 5 As shown, step three includes:

[0119] Step S31, extract any 3-second continuous segment from the sub-separated audio corresponding to each original audio of the medical consultation.

[0120] Step S32: extract the voiceprint embedded code in any 3-second continuous segment in step S31, and perform cluster analysis on the voiceprint embedded code to obtain the identity identification serial number of each time segment.

[0121] Step S33: concatenate the sub-audio files in which the identity appears repeatedly, and name the files with the doctor and the identity identification number; for the sub-audio files in which the identity appears only once, name the files with the patient and the identity identification number.

[0122] In the embodiment of the present invention, each step may be performed by an electronic device, such as a tablet computer, a portable PC, a desktop PC, etc.

[0123] In the technical solution provided by the present invention, the method includes inputting multiple consultation source audios of different patients corresponding to the same doctor into a voiceprint segmentation clustering model to obtain an RTTM file corresponding to each consultation source audio; cutting and splicing the corresponding consultation original audio according to the RTTM file to obtain a sub-separated audio corresponding to each consultation original audio; performing voiceprint recognition on the sub-separated audio corresponding to each consultation original audio to determine that the audio that appears multiple times is the doctor's audio and the audio that appears once is the patient's audio; splicing the doctor's audio to obtain multiple separated audios of different patients and doctors, and naming each separated audio with an identity and speaker label. While realizing the separation of the doctor-patient communication audio, the method can obtain voice samples according to time series labels and determine the identity information of the speaker.

[0124] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is run, the electronic device where the computer-readable storage medium is located is controlled to execute the embodiment of the above-mentioned one-doctor-multiple-patient speech separation method.

[0125] An embodiment of the present invention provides an electronic device, which includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the method for separating speech of multiple patients from one doctor in the embodiment is implemented. To avoid repetition, the method is not described one by one here.

[0126] The electronic device includes, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that this embodiment is merely an example of an electronic device and does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or may combine certain components, or different components, for example, the electronic device may also include an input / output device, a network access device, a bus, etc.

[0127] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0128] The memory can be an internal storage unit of an electronic device, such as a hard disk or memory of the electronic device. The memory can also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the memory can also include both the internal storage unit of the electronic device and the external storage device. The memory is used to store computer programs and other programs and data required by network devices. The memory can also be used to temporarily store data that has been output or is to be output.

[0129] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0130] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.< / na> < / na> < / na> < / na>

Claims

1. A method for voice separation for multiple patients with one doctor, characterized in that: The method comprises: Step 1: Input multiple consultation source audios of different patients corresponding to the same doctor into the voiceprint segmentation clustering model to obtain the RTTM file corresponding to each consultation source audio; Step 2: Cut and splice the corresponding original audio of the consultation according to the RTTM file to obtain the sub-separated audio corresponding to each original audio of the consultation; Step 3: Perform voiceprint recognition on the sub-separated audio corresponding to each original audio of the consultation, determine that the audio that appears multiple times is the doctor's audio, and the audio that appears once is the patient's audio; splice the doctor's audio to obtain multiple separated audios of different patients and doctors, and name each of the separated audios with an identity and speaker label; The step one comprises: Step S11, using a voice activity detection model, dividing each audio frame of the medical consultation source into two categories: voice signal audio and non-voice signal audio; Step S12, dividing the speech signal audio into N segments according to the two categories classified in step S11, and detecting speaker switching in the speech signal audio to obtain a time point at which the speaker switching occurs, wherein N is a positive integer; Step S13, calculating the voiceprint embedding codes corresponding to the N segments according to the N segments divided in step S12; Step S14, performing cluster analysis on each voiceprint embedding code in step S13, classifying each voiceprint embedding code into a category, and obtaining preliminary data of voiceprint segmentation and clustering, wherein one category corresponds to one speaker; Step S15, using the preliminary data in step S14 as training data to obtain a trained classifier model; performing secondary segmentation on the speech signal audio to obtain M segments; applying the classifier model to the M segments to obtain classification data of voiceprint segmentation and clustering, wherein M is a positive integer and M>N; Step S16: Save the start time, end time and speaker label corresponding to the M segments in step S15 into an RTTM file to obtain an RTTM file.

2. The method according to claim 1, characterized in that The second step comprises: Step S21, according to the start time, end time and speaker tag in the RTTM file, save the single start time and end time in the loop body; Step S22, cutting the corresponding original audio of the medical consultation according to the start time and end time in step S21, and saving a temporary segmentation file; Step S23, determining whether there is an audio file named with the speaker tag in step S21 in the storage directory; Step S24: if it is determined that there is no audio file named with the speaker tag in the storage directory, the temporary segmented file in step S22 is named and saved according to the speaker tag; Step S25: If it is determined that there is an audio file named with the speaker tag in the save directory, then the temporarily segmented file in step S22 is spliced ​​after the existing audio file named with the speaker tag; continue to execute step S21, and after traversing the start time, end time and speaker tag in the RTTM file, two sub-audio files named with the speaker tag are obtained.

3. The method according to claim 1, characterized in that The step three comprises: Step S31, extracting any 3-second continuous segment from the sub-separated audio corresponding to each original audio of the medical consultation; Step S32, extracting the voiceprint embedded code in any 3-second continuous segment in step S31, and performing cluster analysis on the voiceprint embedded code to obtain the identity identification serial number of each time segment; Step S33: concatenate the sub-audio files in which the identity appears repeatedly, and name the files with the doctor and the identity identification number; for the sub-audio files in which the identity appears only once, name the files with the patient and the identity identification number.

4. The method according to claim 1, characterized in that: The voice activity detection model in step S11 adopts Gaussian mixture model GMM-VAD; Gaussian mixture model for voice activity detection The formula is: ； in, is the weight of the speech signal; is the weight of non-speech signal; is the mean value of the speech signal; is the variance of the speech signal; is the mean of non-speech signals; is the variance of the non-speech signal.

5. The method according to claim 1, characterized in that The speaker switch detection in step S12 adopts left-right window comparison, wherein the left-right window comparison includes: Step S121, for each consultation source audio frame, two left and right windows of the same size are selected, and the voiceprint embedding codes of the left and right windows and the cosine similarity of the left and right voiceprint embedding codes are calculated to obtain a two-dimensional curve with the time of the audio frame as the x-axis and the cosine similarity as the y-axis; Step S122, performing peak detection on the two-dimensional curve in step S121, selecting all local minimum points, and determining whether the cosine similarity corresponding to the local minimum points is less than a specific threshold; Step S123: If the cosine similarity corresponding to the local minimum point at the judgment location is less than a specific threshold, the speaker is switched.

6. The method according to claim 1, characterized in that The calculation of the voiceprint embedding code in step S13 is based on the identity vector i-vector factor analysis. Given a speaker's speech segment, the corresponding Gaussian mean supervector is defined as: ； Where s is the Gaussian mean supervector of a given speaker's speech segment; m is a supervector that is independent of both the speaker and the channel; T is the global difference space matrix; and w is a random vector that follows a standard multivariate normal distribution. , the dimension is usually selected between 400 and 600.

7. The method according to claim 1, characterized in that The cluster analysis in step S14 adopts K-means clustering; the K-means clustering includes: Step S141: N random vectors Divide into K subsets S = {S1, •••, Sk}, so that the sum of squares within the class is minimized, and select the centers of the initial K classes, where 1≤i≤N; Step S142, calculating the cosine similarity between each data and the center of the K classes in step S141, and assigning it to the class with the highest cosine similarity; Step S143, according to the data assigned to each class in step S142, calculate the mean vector of the data, take it as the center of the class, and continue to execute step S142 until the convergence criterion is met and the change of the sum of squares within the class is less than the threshold.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the electronic device where the computer-readable storage medium is located is controlled to execute the method for speech separation of one doctor for multiple patients according to any one of claims 1 to 7.

9. An electronic device, characterized in that: include: one or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, enable the electronic device to perform the one-doctor-multiple-patient speech separation method described in any one of claims 1 to 7.

Images