Speaker separation method and related equipment

Abstract

The embodiment of the invention provides a speaker separation method and related equipment, and belongs to the technical field of speech recognition. The method comprises the following steps: acquiring current audio data; inputting the current audio data into the voice activity detection model to obtain a voice segment set; performing voice text recognition and voice text segmentation processing on each current voice segment to obtain a conversion text of the current voice segment and a time interval corresponding to the text segment; obtaining a vector set of historical voice, performing embedded vectorization processing on each current voice segment to obtain a current voice vector set, and clustering the vector set of historical voice and the current voice vector set to obtain a current label set; and obtaining an initial speaker tag of each time interval according to the time interval corresponding to the text fragment and the current tag set. According to the embodiment of the invention, a multi-speaker separation task can be completed in real time, the efficiency, accuracy and real-time performance of speaker separation are improved, and the processing time is saved.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and in particular to a speaker separation method and related equipment. Background Technology

[0002] In related technologies, existing speaker separation techniques typically involve step-by-step processing, which cannot meet the requirements of low latency and on-the-fly processing scenarios, thus affecting the real-time performance of speaker separation. Furthermore, the sliding window approach, based solely on the single-speaker assumption, cannot meet the needs of multi-speaker scenarios.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The main objective of this application is to propose a speaker separation method and related equipment that can complete multi-speaker separation tasks in real time, improve the efficiency, accuracy and real-time performance of speaker separation, and save processing time.

[0005] To achieve the above objectives, one aspect of this application proposes a speaker separation method, the method comprising: Get the current audio data; The current audio data is input into the speech activity detection model to obtain a set of speech segments, which includes the current speech segment. For each current speech segment, speech-to-text recognition and speech-to-text segmentation are performed to obtain the converted text of the current speech segment and the time interval corresponding to the text segment, wherein the converted text includes the text segment; Obtain the vector set of historical speech, and perform embedding vectorization processing on each current speech segment to obtain the current speech vector set. Cluster the vector set of historical speech and the current speech vector set to obtain the current tag set. Based on the time interval corresponding to the text segment and the current tag set, the initial speaker tag for each time interval is obtained.

[0006] In some embodiments, obtaining the current audio data includes: Get the current audio stream resource; If a historical voice cache exists, the current audio stream resource is concatenated with the historical voice cache to obtain the current audio data; If no historical audio cache exists, the current audio stream resource will be used as the current audio data.

[0007] In some embodiments, inputting the current audio data into the speech activity detection model to obtain a set of speech segments includes: If the speech activity detection model detects that there is truncated speech in the current audio data input speech, then the truncated speech is saved as a historical speech cache.

[0008] In some embodiments, the current tag set includes a first current tag set of the historical speech and a second current tag set of the current audio data, and the method further includes: Get the set of historical tags for historical voice recordings; The first current tag set is corrected based on the historical tag set of the historical speech to obtain the second current tag set of the historical speech; A tag mapping table is constructed based on the historical tag set of the historical speech and the second current tag set. The historical tag set includes historical tags, the second current tag set includes second current tags, and the tag mapping table includes the mapping relationship between the second current tags and the historical tags.

[0009] In some embodiments, the method further includes: The historical speaker labels for each time interval are determined based on the initial speaker labels for each time interval, the second current label set of the historical speech, and the label mapping table; By integrating the initial speaker tag and the historical speaker tag for each time interval, the current speaker tag corresponding to the current audio data is obtained.

[0010] In some embodiments, the method further includes: If the historical speaker tag is the same as the historical tag of the historical speech, then the historical speaker tag is mixed with the vector set of the historical speech with the historical tag. Calculate the center of the mixed vector set, and update the vector set of the historical speech based on the first few vectors that are closest to the center of the vector set.

[0011] To achieve the above objectives, another aspect of the embodiments of this application proposes a speaker separation device, the device comprising: The audio acquisition module is used to acquire the current audio data; The activity detection module is used to input the current audio data into the speech activity detection model to obtain a speech segment set, the speech segment set including the current speech segment; The text recognition module is used to perform speech-text recognition and speech-text segmentation processing on each of the current speech segments to obtain the converted text of the current speech segment and the time interval corresponding to the text segment, wherein the converted text includes the text segment; The clustering module is used to obtain the vector set of historical speech, and to perform embedding vectorization processing on each current speech segment to obtain the current speech vector set. The vector set of historical speech and the current speech vector set are clustered to obtain the current tag set. The speaker separation module is used to obtain the initial speaker label for each time interval based on the time interval corresponding to the text segment and the current label set.

[0012] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0013] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0014] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0015] The embodiments of this application include at least the following beneficial effects: This application provides a speaker separation method, apparatus, electronic device, storage medium, and program product. This solution acquires current audio data; inputs the current audio data into a speech activity detection model to obtain a set of speech segments; performs speech-text recognition and speech-text segmentation processing on each current speech segment to obtain the converted text of the current speech segment and the time interval corresponding to the text segment, which is beneficial for understanding the speech content and provides the alignment relationship between text and audio; acquires a set of vectors of historical speech, and performs embedding vectorization processing on each current speech segment to obtain a set of current speech vectors; clusters the set of vectors of historical speech and the set of current speech vectors to obtain a set of current labels, which is beneficial for improving recognition efficiency and improving the accuracy and robustness of speaker labels; and obtains the initial speaker label for each time interval based on the time interval corresponding to the text segment and the current label set, which can complete the multi-speaker separation task in real time, improve the efficiency, accuracy, and real-time performance of speaker separation, and save processing time. Attached Figure Description

[0016] Figure 1 This is a flowchart of the speaker separation method provided in the embodiments of this application; Figure 2 yes Figure 1 The flowchart of step S102 in the document; Figure 3 yes Figure 1 The flowchart of step S103 in the process; Figure 4 yes Figure 1 The flowchart of step S104 in the process; Figure 5 This is a flowchart of the updating steps of the speaker separation method provided in the embodiments of this application; Figure 6 This is a flowchart illustrating a specific implementation of the speaker separation method provided in this application when applied to a streaming audio real-time speaker separation system. Figure 7 This is a schematic diagram of the output effect of the speech activity detection provided in the embodiments of this application; Figure 8 This is a schematic diagram illustrating the effect of speech-to-text recognition and speech-to-text segmentation provided in the embodiments of this application; Figure 9 This is a schematic diagram illustrating the effect of speaker separation provided in the embodiments of this application; Figure 10 This is a schematic diagram of the speaker separation device provided in the embodiments of this application; Figure 11 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0018] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0019] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0021] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0022] 1) Speaker Diarization is a process in speech recognition. The goal of this task is to distinguish information from different speakers in audio.

[0023] 2) Voice Activity Detection (VAD) is used to locate continuous audio segments of human voices.

[0024] In related technologies, existing speaker separation techniques typically involve step-by-step processing, which cannot meet the requirements of low latency and on-the-fly processing scenarios, thus affecting the real-time performance of speaker separation. Furthermore, the sliding window approach, based solely on the single-speaker assumption, cannot meet the needs of multi-speaker scenarios.

[0025] In summary, the technical problems existing in the relevant technologies need to be improved.

[0026] In view of this, this application provides a speaker separation method and related device. This method acquires current audio data; inputs the current audio data into a speech activity detection model to obtain a set of speech segments; performs speech-to-text recognition and speech-to-text segmentation on each current speech segment to obtain the converted text of the current speech segment and the corresponding time interval of the text segment, which is beneficial for understanding the speech content and provides the alignment relationship between text and audio; acquires a set of vectors from historical speech, and performs embedding vectorization processing on each current speech segment to obtain a set of current speech vectors; clusters the set of vectors from historical speech and the set of current speech vectors to obtain a set of current labels, which helps improve recognition efficiency and the accuracy and robustness of speaker labels; based on the time interval corresponding to the text segment and the current label set, it obtains the initial speaker label for each time interval, enabling real-time completion of multi-speaker separation tasks, improving the efficiency, accuracy, and real-time performance of speaker separation, and saving processing time.

[0027] The speaker separation method provided in this application relates to the field of speech recognition technology. The speaker separation method provided in this application can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the speaker separation method, but is not limited to the above forms.

[0028] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0029] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0030] Figure 1 This is an optional flowchart of the speaker separation method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S105.

[0031] Step S101: Obtain the current audio data.

[0032] In some embodiments, the current audio stream resource is obtained; if a historical voice cache exists, the current audio stream resource is concatenated with the historical voice cache to obtain the current audio data; if no historical voice cache exists, the current audio stream resource is used as the current audio data.

[0033] It should be noted that if a historical voice cache exists, and the length of the current audio stream resource concatenated with the historical voice cache exceeds the preset window length, then the current audio stream resource will be concatenated with the historical voice cache to obtain the current audio data.

[0034] If there is no historical voice cache and the length of the current audio stream resource exceeds the preset window length, then the current audio stream resource will be used as the current audio data.

[0035] In some embodiments, audio stream resources are acquired in real time, and the acquired audio streams are concatenated and buffered until the length of the concatenated audio buffer exceeds a preset window length. The previously unprocessed voice buffer is concatenated with the current audio buffer to obtain the audio data for the current batch.

[0036] In this embodiment, the current audio data is acquired to prepare for subsequent speaker recognition.

[0037] Step S102: Input the current audio data into the speech activity detection model to obtain a set of speech segments.

[0038] Specifically, the speech segment set includes the current speech segment. Speech activity detection is used to filter out audio segments such as noise and silence. These segments will be ignored and skipped in subsequent processing, thereby reducing resource consumption and improving recognition accuracy.

[0039] In some embodiments, if the speech activity detection model detects that there is truncated speech in the current audio data input speech, the truncated speech is saved as a historical speech cache.

[0040] In this embodiment, the current audio data is input into the speech activity detection model to obtain a set of speech segments, avoiding speaker identification in silent segments without speech, saving computing resources, and improving recognition efficiency and accuracy.

[0041] Step S103: Perform speech-to-text recognition and speech-to-text segmentation on each current speech segment to obtain the converted text of the current speech segment and the time interval corresponding to the text segment.

[0042] Specifically, text conversion includes text fragments.

[0043] It should be noted that the speech segments are converted into natural language text, and the text is segmented according to semantics to form individual text segments.

[0044] Optionally, the text and timestamps obtained by the text recognition model are output, and the time interval corresponding to each text segment in the original audio is recorded.

[0045] In some embodiments, each current speech segment is input into a speech-to-text recognition model for speech-to-text recognition, and the recognized text is input into a speech-to-text segmentation model to obtain the converted text of the current speech segment and the time interval corresponding to the text segment.

[0046] It is understandable that after processing all the current speech segments, the converted text of the current audio data and the time interval corresponding to the text segments are obtained.

[0047] Specifically, features are extracted and sampled from the current speech segment through processes such as Fourier transform and Mel filtering. The speech-text recognition model adopts the transformer encoder framework, which sequentially inputs each current speech segment in time sequence and outputs the token of the next time sequence sample. The output token is decoded to obtain the text corresponding to the speech segment.

[0048] Furthermore, the text corresponding to the speech segment is encoded into a token, which is then input into the speech-to-text segmentation model. The model outputs a label for each token, and the labels are decoded to obtain the punctuation marks of the text segment. Each punctuation mark is considered a text segment. The speech-to-text segmentation model uses the BERT framework, but it can also employ a punctuation annotation model.

[0049] In this embodiment, speech-to-text recognition and speech-to-text segmentation are performed on each current speech segment to obtain the converted text of the current speech segment and the time interval corresponding to the text segment. The text is helpful for understanding the speech content and provides the alignment relationship between the text and the audio, which prepares for subsequent tag extraction.

[0050] Step S104: Obtain the vector set of historical speech, and perform embedding vectorization processing on each current speech segment to obtain the current speech vector set. Cluster the vector set of historical speech and the current speech vector set to obtain the current label set.

[0051] Specifically, the current tag set includes a first current tag set of historical speech and a second current tag set of current audio data.

[0052] In some embodiments, each speech segment of the current audio data is embedded and vectorized to obtain a current speech vector set. This vector set is then mixed with the vector sets of historical speech to obtain a mixed current vector set. This vector set is then clustered to obtain a current tag set.

[0053] It is understandable that the time of the speech corresponding to the label is determined by the index of the vector, and thus the label corresponds to the time of the current speech segment.

[0054] In this embodiment, a set of vectors of historical speech is obtained, and each current speech segment is embedded and vectorized to obtain a set of current speech vectors. The set of vectors of historical speech and the set of current speech vectors are clustered to obtain a set of current labels, which helps to improve recognition efficiency and improve the accuracy and robustness of speaker labels.

[0055] Step S105: Based on the time interval corresponding to the text segment and the current tag set, obtain the initial speaker tag for each time interval.

[0056] In some embodiments, a historical tag set of historical speech is obtained; a first current tag set is corrected based on the historical tag set of historical speech to obtain a second current tag set of historical speech; a tag mapping table is constructed based on the historical tag set of historical speech and the second current tag set, wherein the historical tag set includes historical tags and the second current tag set includes second current tags, and the tag mapping table includes the mapping relationship between the second current tags and historical tags.

[0057] Furthermore, based on the initial speaker labels for each time interval, the second current label set of historical speech, and the historical speaker labels for each time interval in the label mapping table, the initial speaker labels and historical speaker labels for each time interval are integrated to obtain the current speaker label corresponding to the current audio data.

[0058] Specifically, the historical speaker labels corresponding to the initial speaker labels for each time interval are obtained, and the initial speaker labels are replaced with the historical speaker labels to obtain the current speaker labels corresponding to the current audio data.

[0059] Optionally, if the historical speaker label is the same as the historical label of the historical speech, then the historical speaker label is mixed with the vector set of the historical speech with the historical label; the center of the mixed vector set is calculated, and the vector set of the historical speech is updated according to the first few vectors that are closest to the center of the vector set.

[0060] In this embodiment, the initial speaker label for each time interval is obtained based on the time interval corresponding to the text segment and the current label set. This enables the multi-speaker separation task to be completed in real time, improving the efficiency, accuracy and real-time performance of speaker separation and saving processing time.

[0061] Steps S101 to S105 as shown in the embodiments of this application involve: acquiring current audio data; inputting the current audio data into a speech activity detection model to obtain a set of speech segments; performing speech-text recognition and speech-text segmentation on each current speech segment to obtain the converted text of the current speech segment and the time interval corresponding to the text segment, which is beneficial for understanding the speech content and provides the alignment relationship between the text and the audio; acquiring a set of vectors of historical speech and embedding vectorization processing on each current speech segment to obtain a set of current speech vectors; clustering the set of vectors of historical speech and the set of current speech vectors to obtain a set of current labels, which is beneficial for improving recognition efficiency and improving the accuracy and robustness of speaker labels; obtaining the initial speaker label for each time interval based on the time interval corresponding to the text segment and the current label set, which can complete the multi-speaker separation task in real time, improve the efficiency, accuracy and real-time performance of speaker separation, and save processing time.

[0062] Please see Figure 2 In some embodiments, step S102 may include, but is not limited to, steps S201 to S205: Step S201: Extract features from the current audio data using Fourier transform or Mel filtering and sample the data to obtain data samples.

[0063] In step S201 of some embodiments, the current audio data is subjected to Fourier transform or Mel filtering.

[0064] Step S202: Input each data sample into the speech discrimination model to obtain the speech probability score of the sample. If the score exceeds the threshold, it is judged as a speech segment; otherwise, it is a noise segment.

[0065] In step S202 of some embodiments, each data sample is input into the speech discrimination model to obtain the speech probability score of the sample.

[0066] Furthermore, if the speech probability score exceeds the threshold, it is determined to be a speech segment; otherwise, it is a noise segment.

[0067] The threshold can be either a default threshold or a user-defined threshold.

[0068] It is understandable that noisy segments are removed.

[0069] Step S203: Calculate the decibel value for each data sample. If the value exceeds the threshold, the data sample is considered a valid sound segment; otherwise, the data sample is a silent segment.

[0070] In step S203 of some embodiments, a decibel value is calculated for each data sample. If the value exceeds a threshold, the data sample is considered a valid sound segment; otherwise, the data sample is a silent segment.

[0071] Understandably, if a sample is both a speech segment and a valid sound segment, then the sample can be used for speaker recognition.

[0072] Optionally, a decibel value is calculated for each speech segment, and if it exceeds a threshold, the speech segment is considered valid and can be used for speaker recognition.

[0073] Step S204: If there are more than n consecutive samples that are both speech segments and valid sound segments, then merge them into one speech segment.

[0074] Where n is the minimum continuous speech threshold.

[0075] In step S204 of some embodiments, segments that are both speech segments and valid sound segments are merged.

[0076] Step S205: For the merged speech segments, if two adjacent speech segments are separated by less than m samples, they are merged into one speech segment.

[0077] Where m is the maximum interval speech threshold.

[0078] It should be noted that the speech segments are repeatedly merged until all speech segments are separated by no less than m samples.

[0079] Understandably, due to the use of a fixed window, the last segment of audio in the window may be truncated. To avoid semantic incompleteness and speaker label misjudgment in subsequent processes, the incomplete audio is removed from the current audio data, cached, and then concatenated with the audio of the next batch for processing.

[0080] Please see Figure 3 In some embodiments, step S103 may include, but is not limited to, steps S301 to S305: Step S301: Extract features from the current speech segment using Fourier transform or Mel filtering and sample them.

[0081] In step S301 of some embodiments, features are extracted and sampled from the current speech segment by Fourier transform and Mel filtering to obtain data samples.

[0082] Step S302: Input each data sample into the speech-text recognition model in chronological order and output the token of the next time-series sample.

[0083] Specifically, the speech-to-text recognition model uses the transformer encoder framework.

[0084] In step S302 of some embodiments, the acoustic feature sequence of the speech is input into the Transformer encoder, the model predicts the corresponding text token frame by frame, and decodes the text token into a complete text sentence.

[0085] Step S303: Decode the output token to obtain the text corresponding to the speech segment.

[0086] In step S303 of some embodiments, the output token is decoded to obtain the text corresponding to the data sample.

[0087] Furthermore, the converted text corresponding to the current speech segment is obtained.

[0088] The token output by the speech-text recognition model includes a timestamp.

[0089] Step S304: Encode the text corresponding to the speech segment into a token, input the encoded token into the speech-text segmentation model, and the speech-text segmentation model outputs the label corresponding to each token.

[0090] Specifically, the speech-to-text segmentation model uses the BERT framework.

[0091] Understandably, considering the need to output punctuation marks in real-world scenarios, the punctuation annotation model can be directly used as the segmentation model here.

[0092] Step S305: Decode the tag corresponding to the token to obtain the punctuation of the text segment.

[0093] In step S305 of some embodiments, the tag corresponding to the token is decoded to obtain the punctuation of the text segment, and the text segment is divided based on the converted text using the punctuation.

[0094] Furthermore, based on the timestamp output by the speech-to-text recognition model, the text segment of the current speech segment and the corresponding time interval are determined.

[0095] Please see Figure 4 In some embodiments, step S104 may include, but is not limited to, steps S401 to S403: Step S401: Extract features from the current speech segment using Fourier transform and Mel filtering processes and sample them.

[0096] Optionally, features are extracted and sampled from all current speech segments using Fourier transform and Mel filtering to obtain data samples.

[0097] Step S402: Input each sample into an audio embedding model and output the representation vector of each sample.

[0098] In step S402 of some embodiments, each data sample is input into an audio embedding model, and the representation vector of each sample (i.e., the current set of speech vectors) is output.

[0099] Step S403: Mix the representation vector with the historical cache vector, perform spectral clustering, and obtain the current label set.

[0100] In step S403 of some embodiments, the representation vector is mixed with the historical cache vector, and spectral clustering is performed using the SpectralClustering method in sklearn to obtain a first current label set of historical speech and a second current label set of current audio data.

[0101] Please see Figure 5 In some embodiments, the speaker separation method provided in this application further includes an update step, which may include, but is not limited to, steps S501 to S503: Step S501: Determine the speaker label for the time interval based on the time interval, data sample, and current label set.

[0102] In step S501 of some embodiments, the time intervals obtained after speech-text segmentation are combined, assuming that each time interval corresponds to only one speaker.

[0103] Furthermore, the data samples are projected onto time intervals, resulting in a current tag set for each interval. The tag with the highest frequency in the current tag set within an interval is taken as the speaker tag for that interval. Step S502: Obtain the historical tag set of historical speech, correct the first current tag set of historical speech, and create a current tag-historical tag mapping table.

[0104] In step S502 of some embodiments, the current tag set of historical speech is taken, and each speech sample of historical speech has a historical speaker tag.

[0105] Optionally, the speech samples of historical speech can be grouped according to the historical speaker labels to obtain the current label set corresponding to the speech samples of historical speech in each group.

[0106] Furthermore, the label with the highest frequency in the current label set within the group is taken, and all samples within the group are revised and updated to that label to obtain a new current label set for historical speech (i.e., the second current label set).

[0107] Optionally, a mapping table of "current tag - historical tags" can be created.

[0108] Step S503: For each interval's speaker label, if it exists in the new current label set, then the current label for that interval is relabeled as the corresponding historical label according to the current label-historical label mapping table; otherwise, it is treated as a new label. This process ultimately yields the speaker labels for each interval.

[0109] It's important to note that for each output speaker tag, it's mixed with the historical cached speech vectors containing the same historical tags. The center of the vector set is calculated, and the k vectors closest to the center are used to update the historical cached speech vectors. Furthermore, a value of k that is too small will affect the final result; a value that is too large will result in a high computational cost per batch. After feature sampling, approximately 100 samples are obtained from 1 second of audio. Based on experimental testing, considering both performance and computational cost, a value of k between 100 and 300 is suitable.

[0110] Figure 6 This is a flowchart illustrating a specific implementation of the speaker separation method provided in this application when applied to a streaming audio real-time speaker separation system. Figure 6 The methods may include, but are not limited to, the following steps: Step 1: Acquire audio stream resources (such as m3u8 streams) in real time. Concatenate the acquired audio streams into a buffer until the length of the concatenated audio buffer exceeds the preset window length. Concatenate the previously unprocessed voice buffer with the current audio buffer to obtain the audio data for the current batch.

[0111] Step 2: The audio data of the current batch from Step 1 is fed into the Voice Activity Detection (VAD) model. This model is used to locate continuous audio segments of human voices, resulting in a set of speech segments for the current batch. If any truncated speech is detected, it is removed from the current batch of audio data and cached for use in the next batch of speech processing.

[0112] Among them, speech activity detection is used to locate continuous audio segments (speech segments) of human voices. For example, the output effect of speech activity detection is illustrated in the diagram below. Figure 7 As shown.

[0113] It should be noted that the voice activity detection step is often used to filter out audio segments such as noise and silence. These segments will be ignored and skipped in subsequent processing, thereby reducing resource consumption and improving recognition accuracy.

[0114] The speech activity detection process adopted in this solution is as follows: 1. Extract features and sample the audio data of the current batch through processes such as Fourier Transform (STFT) and Mel filtering; 2. Input each data sample into the speech discrimination model to obtain the speech probability score of that sample. If the score exceeds the threshold, it is judged as a speech segment; otherwise, it is a noise segment. 3. Calculate the decibel value for each data sample. If the value exceeds the threshold, it is considered a valid sound segment; otherwise, it is a silent segment. 4. Combining the results of steps 2 and 3, if there are more than n consecutive samples that are both speech segments and valid sound segments, then merge them into one speech segment; 5. Smoothing. After merging the speech segments in step 4, if two adjacent speech segments are less than m samples apart, they are merged into one speech segment. Step 5 is repeated until all speech segments are at least m samples apart.

[0115] It should be noted that, due to the use of a fixed window, the last segment of audio in the window may be truncated (e.g., ...). Figure 2 (Audio segment 3) To avoid semantic incompleteness and speaker label misjudgment in subsequent processes, incomplete speech is removed from the current batch of audio data and cached, and then concatenated with the audio of the next batch for processing.

[0116] For example, because there is a mechanism for buffering unprocessed audio streams, the impact on the final effect is not significant as long as the window length is not set too small. Furthermore, considering that the real-time latency should not be too high, the window length is generally set to 1 to 1.5 seconds.

[0117] Step 3: For each speech segment in the current batch from Step 2, perform speech-to-text recognition and speech-to-text segmentation to obtain the converted text of each speech segment and the corresponding time interval for each text segment.

[0118] For example, speech-to-text recognition and speech-to-text segmentation are used to convert speech segments into natural language text, and then segment the text according to semantics to form individual text segments. An example illustration of the effect of speech-to-text recognition and speech-to-text segmentation is shown below. Figure 8 As shown.

[0119] The speech activity detection process adopted in this solution is as follows: 1. Extract features and sample the current batch of speech data using processes such as Fourier Transform (STFT) and Mel filtering; 2. The speech-to-text recognition model uses the transformer encoder framework, which inputs each data sample sequentially in time sequence and outputs the token of the next time sequence sample; 3. Decode the token output in step 2 to obtain the text corresponding to the speech segment; 4. The speech-to-text segmentation model uses the BERT framework. Considering the need to output punctuation marks in real-world scenarios, a punctuation annotation model can be directly used as the segmentation model. The text obtained in step 3 is encoded into tokens. The encoded tokens are input, and the label corresponding to each token is output. 5. Decode the tags output in step 4 to obtain the punctuation marks of the text segment. Each punctuation mark is treated as a text segment.

[0120] Step 4: For each speech segment in the current batch obtained in Step 2, perform embedding vectorization processing to obtain a vector set of the current batch of speech. Mix this vector set with the vector set of historical speech to obtain a vector set of the current batch. Cluster this vector set to obtain the current tag set.

[0121] Speaker separation, in particular, distinguishes different speakers in audio. An example illustration of the speaker separation effect is shown below. Figure 9 As shown.

[0122] The speech activity detection process adopted in this solution is as follows: 1. Extract features and sample the current batch of speech data using processes such as Fourier Transform (STFT) and Mel filtering; 2. Input each sample into an audio embedding model and output a representation vector for each sample; 3. Combine the representation vector obtained in step 2 with the vector from the historical cache, and use the SpectralClustering method in sklearn to perform spectral clustering to obtain the current label of the speech samples in the current batch and the current label set of the samples in the historical cache. 4. Combining the time intervals obtained after speech-text segmentation, assuming that each time interval corresponds to only one speaker, the data samples from step 1 are projected into the time intervals, resulting in a current label set for each interval. The label with the highest frequency in the current label set within the interval is taken as the speaker label for that interval; 5. Obtain the current tag set from the historical cache obtained in step 3. Additionally, each historical cached speech sample has a historical speaker tag. First, group the historical cached samples according to their historical tags, resulting in a current tag set for each group. Take the tag with the highest frequency in the current tag set within each group, and update all samples within that group with that tag, resulting in a new current tag set. Finally, create a mapping table between "current tag" and "historical tag". 6. For each interval's speaker tag, if it exists in the new current tag set from step 5, then re-label the current tag of that interval as the corresponding historical tag according to the "current tag - historical tag" mapping table; otherwise, treat it as a new tag. This yields the speaker tags for each interval. 7. For each output speaker tag, mix it with the speech vectors from the historical cache that share the same historical tags. Calculate the center of the vector set, and take the k vectors closest to the center of the set to update the speech vectors in the historical cache. Note that if the value of k is too small, it will affect the final result; if it is too large, the computational cost per batch may be very high. After feature sampling, approximately 100 samples will be obtained from 1 second of audio. Based on experimental testing, considering both performance and computational cost, a value of k between 100 and 300 is suitable.

[0123] Step 5: Combine the time intervals from Step 3 and the current tag set from Step 4 to obtain the speaker tags for each time interval.

[0124] Step 6: Combine the current tag set of historical speech and the historical tag set of historical speech from Step 4, and revise the current tag set of historical speech while keeping the historical tag set of historical speech unchanged, to obtain the mapping table of "current tag - historical tag" for the current batch.

[0125] Step 7: Combining the current speaker tags for each time interval in Step 5, the current tag set from the historical cache in Step 6, and the "current tag - historical tag" mapping table for the current batch, obtain the historical speaker tags for each interval. Integrate the speaker tags from all intervals and output the live speaker tags for the current batch.

[0126] Step 8: Combine the historical speaker tags obtained in Step 7 with the vector set in Step 4 to update the vector set of historical speech.

[0127] Step 9: Return to Step 1 until the audio stream resource acquisition is complete.

[0128] This invention performs real-time processing of streaming audio to separate speaker information, and has the following beneficial effects: 1) High real-time performance. Speaker separation can be completed online in real time without offline processing.

[0129] 2) High accuracy. By combining text information, multiple speaker information can be output within a fixed window, solving the problem of inaccurate speaker segmentation. Furthermore, it is not limited to recognizing a finite number of speakers; theoretically, it can identify an infinite number of speakers.

[0130] 3) High efficiency and convenience. No pre-training is required, and it can be used with zero registration; moreover, it can run concurrently with other speech processing processes, greatly saving processing time.

[0131] Please see Figure 10 This application also provides a speaker separation device that can implement the above-described method. The device includes: Audio acquisition module 1001 is used to acquire current audio data; The activity detection module 1002 is used to input the current audio data into the speech activity detection model to obtain a speech segment set, wherein the speech segment set includes the current speech segment; The text recognition module 1003 is used to perform speech-text recognition and speech-text segmentation processing on each of the current speech segments to obtain the converted text of the current speech segment and the time interval corresponding to the text segment, wherein the converted text includes the text segment; Clustering module 1004 is used to obtain a vector set of historical speech, and to perform embedding vectorization processing on each current speech segment to obtain a current speech vector set. The vector set of historical speech and the current speech vector set are clustered to obtain a current tag set. The speaker separation module 1005 is used to obtain the initial speaker label for each time interval based on the time interval corresponding to the text segment and the current label set.

[0132] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0133] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0134] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0135] Please see Figure 11 , Figure 11 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 1101 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1102 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1102 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1102 and is called and executed by the processor 1101 using the methods described in the embodiments of this application. Input / output interface 1103 is used to implement information input and output; The communication interface 1104 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1105 transmits information between various components of the device (e.g., processor 1101, memory 1102, input / output interface 1103, and communication interface 1104); The processor 1101, memory 1102, input / output interface 1103 and communication interface 1104 are connected to each other within the device via bus 1105.

[0136] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0137] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0138] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0139] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0140] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0141] The speaker separation method, apparatus, electronic device, storage medium, and program product provided in this application embodiment acquire current audio data; input the current audio data into a speech activity detection model to obtain a set of speech segments; perform speech-text recognition and speech-text segmentation processing on each current speech segment to obtain the converted text of the current speech segment and the time interval corresponding to the text segment, which is beneficial for understanding the speech content and provides the alignment relationship between text and audio; acquire a set of vectors of historical speech, and perform embedding vectorization processing on each current speech segment to obtain a set of current speech vectors; cluster the set of vectors of historical speech and the set of current speech vectors to obtain a set of current labels, which is beneficial for improving recognition efficiency and improving the accuracy and robustness of speaker labels; and obtain the initial speaker label for each time interval based on the time interval corresponding to the text segment and the current label set. This enables real-time completion of multi-speaker separation tasks, improving the efficiency, accuracy, and real-time performance of speaker separation, and saving processing time.

[0142] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0143] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0144] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0145] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0146] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0147] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0148] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0149] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0150] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0151] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0152] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A speaker separation method, characterized in that, The method includes the following steps: Get the current audio data; The current audio data is input into the speech activity detection model to obtain a set of speech segments, which includes the current speech segment. For each current speech segment, speech-to-text recognition and speech-to-text segmentation are performed to obtain the converted text of the current speech segment and the time interval corresponding to the text segment, wherein the converted text includes the text segment; Obtain the vector set of historical speech, and perform embedding vectorization processing on each current speech segment to obtain the current speech vector set. Cluster the vector set of historical speech and the current speech vector set to obtain the current tag set. Based on the time interval corresponding to the text segment and the current tag set, the initial speaker tag for each time interval is obtained.

2. The method according to claim 1, characterized in that, The process of obtaining the current audio data includes: Get the current audio stream resource; If a historical voice cache exists, the current audio stream resource is concatenated with the historical voice cache to obtain the current audio data; If no historical audio cache exists, the current audio stream resource will be used as the current audio data.

3. The method according to claim 1, characterized in that, The step of inputting the current audio data into the speech activity detection model to obtain a set of speech segments includes: If the speech activity detection model detects that there is truncated speech in the current audio data input speech, then the truncated speech is saved as a historical speech cache.

4. The method according to claim 1, characterized in that, The current tag set includes a first current tag set of the historical speech and a second current tag set of the current audio data; the method further includes: Get the set of historical tags for historical voice recordings; The first current tag set is corrected based on the historical tag set of the historical speech to obtain the second current tag set of the historical speech; A tag mapping table is constructed based on the historical tag set of the historical speech and the second current tag set. The historical tag set includes historical tags, the second current tag set includes second current tags, and the tag mapping table includes the mapping relationship between the second current tags and the historical tags.

5. The method according to claim 4, characterized in that, The method further includes: The historical speaker labels for each time interval are determined based on the initial speaker labels for each time interval, the second current label set of the historical speech, and the label mapping table; By integrating the initial speaker tag and the historical speaker tag for each time interval, the current speaker tag corresponding to the current audio data is obtained.

6. The method according to claim 5, characterized in that, The method further includes: If the historical speaker tag is the same as the historical tag of the historical speech, then the historical speaker tag is mixed with the vector set of the historical speech with the historical tag. Calculate the center of the mixed vector set, and update the vector set of the historical speech based on the first few vectors that are closest to the center of the vector set.

7. A speaker separation device, characterized in that, The device includes: The audio acquisition module is used to acquire the current audio data; The activity detection module is used to input the current audio data into the speech activity detection model to obtain a speech segment set, wherein the speech segment set includes the current speech segment; The text recognition module is used to perform speech-text recognition and speech-text segmentation processing on each of the current speech segments to obtain the converted text of the current speech segment and the time interval corresponding to the text segment, wherein the converted text includes the text segment; The clustering module is used to obtain the vector set of historical speech, and to perform embedding vectorization processing on each current speech segment to obtain the current speech vector set. The vector set of historical speech and the current speech vector set are clustered to obtain the current tag set. The speaker separation module is used to obtain the initial speaker label for each time interval based on the time interval corresponding to the text segment and the current label set.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images