Method for recognizing speaker through long voice transliteration character separation
By employing speech preprocessing and voiceprint feature extraction methods, the problem of speaker indistinguishability in conference systems has been solved, achieving efficient speech transcription and speaker recognition, thereby improving recognition accuracy and user experience.
Patent Information
- Application Number
- CN202511687659.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-18
- Publication Date
- 2026-02-10
AI Technical Summary
In the conference system, it is impossible to effectively distinguish who said each sentence, resulting in the inability to accurately separate the process of corresponding speakers, text, and voice.
This paper achieves speaker recognition for long speech by employing methods such as speech preprocessing, endpoint detection, speakerprint feature extraction, clustering, text transcription, and visualization. Specific steps include speech format standardization, resampling, noise suppression, speakerprint feature extraction, clustering, label correction, and visualization.
It improves the accuracy and efficiency of speech recognition, reduces misjudgments and label jitter, supports face matching in videos, and enhances the user experience.
Smart Images
Figure CN121506148A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of speech technology processing, specifically a method for long speech transcription, text separation, and speaker recognition. Background Technology
[0002] In common conferencing systems, meeting records such as voice, text, and video are archived. The process of identifying speakers, text, and voice is called role separation in meeting records. When recognizing the content of the audio, it is impossible to distinguish who said each sentence. Summary of the Invention
[0003] The purpose of this invention is to provide a method for separating and recognizing the speaker in long speech-to-text transcription, so as to solve the problems mentioned in the background art.
[0004] To achieve the above objectives, the present invention provides the following technical solution: a method for long speech-to-text transcription, text separation, and speaker recognition, comprising the following steps: Step S1: Speech preprocessing, which involves uniformly converting the speech format; Step S2: Perform endpoint detection on the long speech signal, segment it into multiple speech segments, and extract the voiceprint feature vector of each speech segment; the main steps of audio voiceprint feature extraction are as follows: a. Resample the audio to 16kHz, normalize it to [-1,1], output AudioSegment object, and extract the samples attribute of the audio; b. Dynamically fill the voiceprint, create a filling matrix according to the different durations of the audio; c. Calculate FBank features and perform feature extraction; d. Batch predict the voiceprint and output the voiceprint embedding vector. Step S3: Based on voiceprint feature clustering, classify the speech segments of the same speaker and obtain the clustered voiceprint label array; Step S4: Process the audio segmentation results, including label correction, segment merging, and smoothing. Step S5: Transcribe the processed audio into text to obtain the result text content and the corresponding time period. Combine the voiceprint tag result set and the transcribed text result set and process them to match the voiceprint tags corresponding to the text content. Step S6: Add a visual dashboard display, showing content including speaker tags, timeline, and text relationship diagram.
[0005] Preferably, in step S1, during speech preprocessing, the audio data is extracted from and retained from the collected audio and video data, and the sampling rate of the audio is detected. If the audio sampling rate is not 16k, it needs to be resampled and calculated, and the 16k audio data is converted and output. The speech enhancement module based on RNN suppresses environmental noise and uses nonlinear filtering to eliminate interference.
[0006] Preferably, in step S2, the detection of long speech signals also includes silence detection. During silence detection, the energy value of each 0.5-second speech segment is calculated. When the energy of three consecutive segments is lower than the threshold, it is marked as a silence segment to avoid misjudging breathing sounds as speech. During the speech detection process, the speech is adaptively segmented: the speech is divided into segments of 5-15 seconds using the silence segment as the dividing point. If the segment exceeds 15 seconds, it is forcibly segmented at the point of sudden change in speech rate.
[0007] Preferably, in step S2, the main steps for identifying the same speaker are as follows: using an incremental clustering algorithm, a. initializing an empty cluster set; b. for each new speech segment, calculating the cosine similarity with the existing clusters; c. if the highest similarity is greater than the threshold, then it is assigned to that cluster; otherwise, a new cluster is created; d. dynamically adjusting the threshold θ = 0.85 - 0.05 × the current number of clusters; for audio with a duration of more than 1 hour, when extracting the voiceprint feature vector, the data needs to be first divided into blocks for coarse clustering, with one cluster center assigned to every 100 samples, and then the clustering algorithm is applied to the center points. The results are used to map the labels to the original data, and similar speakers with a cosine similarity greater than the threshold are merged.
[0008] Preferably, when identifying the same person speaking, a multimodal verification step is added. The steps mainly include: a. extracting the text content features of each speech segment; b. triggering voiceprint re-examination when the semantic conflict degree of consecutive speech segments of the same speaker is greater than the threshold; c. adjusting the classification results through a voiceprint-text bimodal decision tree.
[0009] Preferably, in step S3, the audio is segmented. Short-term noise interference causes the same speaker to be mislabeled as multiple different IDs, which needs to be corrected. The same speaker is segmented into multiple segments due to short pauses. Multiple segments are merged into one. Smoothing effectively handles the phenomenon of label jitter, that is, adjacent segments are alternately labeled as different speakers.
[0010] Preferably, when processing video, the speaker's avatar information is found by analyzing the voiceprint features of the audio; if the frontal face appears once, the speaker's voiceprint feature information for the current time period is recorded, and the avatar information corresponding to the voiceprint is matched according to the number of times the frontal face appears, and the identity of the person is determined.
[0011] Preferably, for cases with multiple faces in a video frame, the voiceprint label and multiple face labels corresponding to the time point are recorded, the entire voiceprint result set and face result set are obtained, and the erroneous labels of multiple face labels appearing in the same second are compared, merged, and removed. The algorithm outputs a recognition result set with voiceprint label and face label.
[0012] Preferably, the median of the time period of the obtained voiceprint tag result set is compared and classified with the median of the time period of the speech recognition text result set, and the merged result sets of the same interval are output as a recognition result set with voiceprint tags.
[0013] Preferably, when using the visual dashboard for querying, a dynamic timeline is displayed, showing the speaker, timeline, and text in a dynamic manner; the ASR transcribed text is dynamically associated with the voiceprint tag, the text color matches the speaker's color block, and hovering over the text displays the corresponding audio's voiceprint matching degree.
[0014] Compared with the prior art, the beneficial effects of the present invention are: (1) This invention ensures the consistency of input data by uniformly converting audio formats, detecting sampling rate and calculating resampling; it combines the speech enhancement module of RNN to suppress environmental noise, uses nonlinear filtering to eliminate interference and improve speech quality, and silence detection avoids misjudging breathing sounds as speech; adaptive segmentation divides speech into 5-15 second segments and forces segmentation at the point of sudden change in speech rate to ensure processing efficiency and accuracy. (2) This invention generates a reliable speaker embedding vector by resampling, normalization, dynamic filling and FBank feature calculation, and dynamically adjusts the threshold to adapt to different numbers of speakers; it performs block coarse clustering for long audio to improve processing efficiency and enhance the accuracy of speaker recognition. (3) For long audio recordings exceeding 1 hour, this invention adopts a block coarse clustering strategy, assigning one cluster center to every 100 samples, which optimizes processing efficiency and accuracy, ensures stable performance under large data volumes, and realizes users' desire for reliable and usable AI transcription. (4) This invention solves the problem of mislabeling caused by short-term noise interference and pauses, merges segments of the same speaker, reduces label jitter, and performs multimodal verification: combining text content features, triggering re-examination when semantic conflicts occur, and adjusting the classification results through a bimodal decision tree to improve the accuracy and reliability of the results; (5) This invention associates voiceprints with faces: it analyzes the voiceprint features of audio, combines them with the speaker's avatar information in the video, and matches the avatar corresponding to the voiceprint. Multiple face processing: it records the voiceprint and face tags corresponding to the time points, removes incorrect tags, and outputs the results with voiceprint and face tags, thereby supporting video processing and face matching; (6) This invention provides an intuitive visualization display that shows the dynamic relationship between the speaker, timeline and text, with text color matching the speaker's color block and hovering to display the voiceprint matching degree, thereby improving the user experience. Attached Figure Description
[0015] Figure 1 This is a flowchart from the present invention; Detailed Implementation
[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example
[0017] like Figure 1 As shown, this invention provides a method for separating and recognizing speakers in long speech-to-text transcription. Speech exceeding one hour in duration is defined as long speech. This long speech requires processing to separate and recognize the voices of different speakers, including the following steps: Step S1: When the received speech is a long speech, the speech is first preprocessed. The speech format is uniformly converted. The speech format conversion mainly involves extracting and retaining the audio data from the acquired audio and video data, and detecting the audio sampling rate. If the audio sampling rate is not 16kHz, it needs to be resampled and calculated, and the 16kHz audio data is converted and output. The speech enhancement module based on RNN suppresses environmental noise and uses nonlinear filtering to eliminate noise interference. Step S2: Then, perform endpoint detection on the long speech signal to segment it into multiple speech segments, including valid and invalid speech segments. Valid speech segments are defined as: Invalid speech segments are defined as: Then, extract the voiceprint feature vector of each valid speech segment. The main steps for extracting the voiceprint features of each speech segment are as follows: a. Resample the audio to 16kHz, normalize it to [-1,1], output an AudioSegment object, and extract the samples attribute of the audio; b. Dynamically fill the voiceprint, creating a filling matrix according to the different durations of the audio; mainly, the batch audio length is statistically analyzed. When processing each batch, the actual length of all audio in the current batch (e.g., the number of frames) is obtained first, and then the length of the longest audio in the current batch is used as the filling target. This target is not a fixed value, thus determining the maximum length; when filling short audio, for audio shorter than the maximum length, zeros (or specific values) are padded at the end of the time axis (or frame axis) to align it with the longest audio, generating a filling matrix, and finally obtaining a matrix with the shape [batch_size, max_length, The matrix of [feature_dim] can be directly input into the model; c. Calculate the FBank features of the voiceprint and perform feature extraction; d. Batch predict the voiceprint and output the voiceprint embedding vector; Among them, when detecting long speech signals, silence detection is also included. When detecting silence, the energy value of each 0.5-second speech segment is calculated. When the energy of 3 consecutive segments is lower than the threshold, the threshold is 0.85, and it is marked as a silence segment to avoid misjudging breathing sounds as speech; Adaptive segmentation of speech is performed during speech detection: the speech is divided into segments of 5-15 seconds with silence segments as the dividing point. If the segment exceeds 15 seconds, it is forcibly segmented at the point of sudden change in speech rate, for example; The main steps to identify the same speaker are: using an incremental clustering algorithm, a. initialize an empty cluster set b. for each new speech segment, calculate the cosine similarity with the existing clusters; c. if the highest similarity is > the threshold, it is assigned to that cluster; otherwise, a new cluster is created; d. Dynamically adjust the threshold θ = 0.85 - 0.05 × current number of clusters; For audio with a duration of more than 1 hour, when extracting the voiceprint feature vector, the data needs to be divided into blocks for coarse clustering first, and a cluster center is assigned to every 100 samples. Then, the clustering algorithm is applied to the center points. The results are used to map the labels to the original data, and similar speakers with a cosine similarity greater than the threshold are merged, so that the voiceprints of the same speaker can be identified. Step S3: Based on voiceprint feature clustering, classify the speech segments of the same speaker and obtain the clustered voiceprint label array; when identifying the same person speaking in step S2, a multimodal verification step can be added. This step mainly involves: a. extracting the text content features of each speech segment; b. when the text semantic conflict degree of consecutive speech segments of the same speaker is greater than the threshold (text semantic conflict degree refers to semantic coherence), if the threshold is 0.85, then when the text semantic conflict degree is greater than 0.85, voiceprint re-examination will be triggered; c. adjusting the classification results through a voiceprint-text bimodal decision tree. For example, if there is a speech segment, the result is identified as two speech segments A and B. Speech segment A is: "Mom, eat tonight"; speech segment B is: "What"; then the text semantics of these two speech segments show conflict, and voiceprint recognition will identify this as the words spoken by two people, which will trigger voiceprint re-examination.
[0018] Step S4: Process the audio segmentation results, including label correction, segment merging and smoothing. The audio is segmented. Short-term noise interference causes the same speaker to be mislabeled as multiple different IDs, which needs to be corrected. The same speaker is segmented into multiple segments due to short pauses. Multiple segments are merged into one. Smoothing can effectively deal with the phenomenon of label jitter, that is, adjacent segments are alternately labeled as different speakers. Step S5: Transcribe the processed audio into text, obtaining the corresponding text content and the time period. Process the voiceprint tag result set and the speech recognition-transcribed text result set to match the voiceprint tags corresponding to the text content. For video processing, analyze the voiceprint features of the audio to find the speaker's avatar information. If a face appears in the video, and it is a frontal face, record the speaker's voiceprint features for the current time period each time the frontal face appears. Then, match the avatar information corresponding to the voiceprint based on the number of times the frontal face appears, and determine which face it is based on the avatar information. If multiple faces appear in the video, record them in the same time period. The algorithm obtains the entire voiceprint and face result sets by acquiring a voiceprint tag and multiple face tags corresponding to a given time point. Then, it compares and merges the voiceprint and face result sets, removing erroneous tags that include multiple face tags appearing in the same second. The algorithm outputs a recognition result set with both voiceprint and face tags. Next, it compares the median value of the obtained voiceprint tag result set within a given time period with the median value of the speech recognition-to-text result set within the same time period, categorizing and merging them. During categorization and merging, result sets within the same interval are merged. Finally, the algorithm outputs a recognition result set with voiceprint tags. This ensures that even when multiple faces and voices appear, different voices corresponding to different faces can be identified. Step S6: After identifying and separating the voices of different people, a visual dashboard can be added to display the relevant content of the speech, including speaker tags, timelines, text association diagrams, and other information. When using the visual dashboard, a dynamic timeline can be used to display the speaker, timeline, and text dynamically. Furthermore, the ASR transcribed text is dynamically associated with the voiceprint tags, using different text colors to match the speaker's color block. For example, when the same speaker is speaking, the text color displayed is the same as the voiceprint tag color. Hovering the mouse over the transcribed text displays the corresponding audio's voiceprint matching score, thus greatly improving the user experience.
[0019] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments are to be regarded in all respects as exemplary and not restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.
Claims
1. A method for long speech-to-text transcription, text separation, and speaker recognition, characterized in that: Includes the following steps: Step S1: Speech preprocessing, which involves uniformly converting the speech format; Step S2: Perform endpoint detection on the long speech signal, segment it into multiple speech segments, and extract the voiceprint feature vector of each speech segment; the main steps of audio voiceprint feature extraction are as follows: a. Resample the audio to 16kHz, normalize it to [-1,1], output AudioSegment object, and extract the samples attribute of the audio; b. Dynamically fill the voiceprint, create a filling matrix according to the different durations of the audio; c. Calculate FBank features and perform feature extraction. d. Batch predict voiceprints and output voiceprint embedding vectors; Step S3: Based on voiceprint feature clustering, classify the speech segments of the same speaker and obtain the clustered voiceprint label array; Step S4: Process the audio segmentation results, including label correction, segment merging, and smoothing. Step S5: Transcribe the processed audio into text to obtain the result text content and the corresponding time period. Combine the voiceprint tag result set and the transcribed text result set and process them to match the voiceprint tags corresponding to the text content. Step S6: Add a visual dashboard display, showing content including speaker tags, timeline, and text relationship diagram.
2. The method for long speech-to-text separation and speaker recognition according to claim 1, characterized in that: In step S1, during speech preprocessing, the audio data is extracted and retained from the collected audio and video data, and the sampling rate of the audio is detected. If the audio sampling rate is not 16k, it needs to be resampled and calculated, and the 16k audio data is converted and output. The speech enhancement module based on RNN suppresses environmental noise and uses nonlinear filtering to eliminate interference.
3. The method for long speech-to-text separation and speaker recognition according to claim 1, characterized in that: In step S2, silence detection is also included when detecting long speech signals. During silence detection, the energy value of each 0.5-second speech segment is calculated. When the energy of three consecutive segments is lower than the threshold, it is marked as a silence segment to avoid misjudging breathing sounds as speech. During the speech detection process, the speech is adaptively segmented: the speech is divided into segments of 5-15 seconds using silence segments as the dividing point. If a segment exceeds 15 seconds, it is forcibly segmented at the point of sudden change in speech rate.
4. The method for long speech-to-text separation and speaker recognition according to claim 1, characterized in that: In step S2, the main steps for identifying the same speaker are as follows: using an incremental clustering algorithm, a. initialize an empty cluster set; b. for each new speech segment, calculate the cosine similarity with the existing clusters; c. if the highest similarity is greater than the threshold, then it is assigned to that cluster; otherwise, a new cluster is created; d. dynamically adjust the threshold θ = 0.85 - 0.05 × the current number of clusters. For audio with a duration of more than 1 hour, when extracting the voiceprint feature vector, the data needs to be first divided into blocks for coarse clustering, with one cluster center assigned to every 100 samples. Then, the clustering algorithm is applied to the center points, and the labels are mapped to the original data based on the results. Similar speakers with a cosine similarity greater than the threshold are merged.
5. The method for long speech-to-text separation and speaker recognition according to claim 4, characterized in that: When identifying the same person speaking, a multimodal verification step is added. The steps mainly include: a. extracting the text content features of each speech segment; b. triggering voiceprint re-examination when the semantic conflict of consecutive speech segments of the same speaker is greater than the threshold; c. adjusting the classification results through a voiceprint-text bimodal decision tree.
6. The method for long speech-to-text separation and speaker recognition according to claim 1, characterized in that: In step S3, the audio is segmented. Short-term noise interference causes the same speaker to be mislabeled as multiple different IDs, which needs to be corrected. The same speaker is segmented into multiple segments due to short pauses. Multiple segments are merged into one. Smoothing effectively handles the phenomenon of label jitter, that is, adjacent segments are alternately labeled as different speakers.
7. The method for long speech-to-text separation and speaker recognition according to claim 2, characterized in that: When processing video, the speaker's face information is found by analyzing the voiceprint features of the audio. If the face appears once, the speaker's voiceprint feature information for the current time period is recorded. The face information corresponding to the voiceprint is matched according to the number of times the face appears, and the face is determined.
8. The method for long speech-to-text separation and speaker recognition according to claim 2, characterized in that: For situations involving multiple faces in a video frame, the system records the voiceprint tags and multiple face tags corresponding to each time point, obtains the entire voiceprint result set and face result set, compares and merges them, removes erroneous tags that appear in the same second with multiple face tags, and outputs a recognition result set with voiceprint tags and face tags through the algorithm.
9. The method for long speech-to-text separation and speaker recognition according to claim 1, characterized in that: The median value of the time period in the voiceprint tag result set is compared and classified with the median value of the time period in the speech recognition text result set. The merged result sets in the same interval are output as the recognition result set with voiceprint tags.
10. The method for long speech-to-text separation and speaker recognition according to claim 1, characterized in that: When using the visual dashboard, a dynamic timeline is displayed, showing the speaker, timeline, and text in a dynamic way; the ASR transcribed text is dynamically associated with the voiceprint tag, the text color matches the speaker's color block, and hovering over the text displays the voiceprint matching degree of the corresponding audio.