Voice multi-role separation method and device based on large model, equipment and medium

Abstract

The invention discloses a voice multi-role separation method and device based on a large model, equipment and a medium, and relates to the technical field of voice recognition, and the method comprises the steps: obtaining an initial affiliation result based on the division of original audio data; calculating multi-modal abnormal scores of adjacent text segments in the text transfer sequence, and identifying candidate error points; for each candidate error point, determining a corresponding local time window, identifying a sub-segmentation point in the local time window, and re-dividing the text segment in the local time window based on the sub-segmentation point to update the text transcription sequence; constructing structured input data containing audio features, text features and initial speaker tags based on the updated text transcription sequence, and inputting the structured input data into a preset large language model; and based on the operation suggestion of the large language model, solving the globally optimal speaker tag sequence through a preset optimization algorithm to obtain a speaker attribution result. According to the invention, the accuracy of performing multi-role separation on the voice can be improved.

Description

A method, apparatus, device, and medium for speech multi-role separation based on a large model. Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a method, apparatus, device, and medium for speech multi-role separation based on a large model. Background Technology

[0002] With the continuous development of information technology, long speech recognition and multi-speaker speech processing technologies are increasingly widely used in daily life and professional fields. For example, in scenarios such as teleconferences, academic interviews, court hearing recordings, and customer service centers, automatically transcribing speech content and accurately distinguishing the speeches of different speakers has become the foundation for supporting key functions such as subsequent content retrieval, process auditing, and risk control. Currently, the industry typically follows a relatively mature workflow chain when implementing multi-speaker speech recognition and role separation. This process generally begins with speech activity detection, aiming to locate valid speech segments from continuous audio. Acoustic segmentation is then performed, dividing the speech signal into representative analysis units. Next, automatic speech recognition technology is used to transcribe each speech segment into text, usually retaining timestamp information. To distinguish different speakers, the system extracts speaker embedding vectors from the speech segments, which can represent the speaker's acoustic features. Finally, a clustering algorithm is used to group speech segments belonging to the same speaker into the same category, and speaker labels are assigned to the transcribed text based on the clustering results.

[0003] However, in practical applications, it has been found that although the above process can work under certain conditions, the speaker attribution results still exhibit several typical error patterns, hindering further improvements in system performance. One common error is the incorrect role separation order; that is, although speech segments are correctly clustered to their corresponding speakers, their order on the timeline may be disordered, disrupting the logical flow of the dialogue—for example, the response may appear before the question. This role separation disorder is more pronounced when processing longer speech passages. Introducing a large language model to directly generate or correct the separation results may improve local semantic coherence, but the inherent illusionary nature of large language models may lead them to arbitrarily adjust the speech order or even add or delete text content without basis, severely deviating from the true content of the original speech.

[0004] Another deeper error stems from the mismatch between semantic structure and acoustic features. Specifically, a single, long speech segment may exhibit acoustic consistency, thus being attributed to the same speaker, even though its semantic content clearly indicates the existence of multiple speaker roles. A typical example is a continuous pronunciation containing both question and answer statements, such as "What's your name? My name is [name]." Semantically, this should be divided into question and answer parts and attributed to different speakers, but current processes tend to merge them into one. Furthermore, existing methods perform poorly when dealing with overlapping speaker pronunciations, easily leading to omissions or incorrect attributions. Additionally, inaccurate punctuation or sentence breaks generated by automatic speech recognition can mislead subsequent role determination processes, and brief responses are often inappropriately merged into the preceding statement, causing a chain reaction of speaker attribution errors.

[0005] These issues collectively reveal a fundamental limitation of existing technologies: their processing often relies excessively on acoustic clustering results or solely on the text generation capabilities of language models. Therefore, in practical applications, especially in scenarios with extremely high requirements for accuracy and reliability, effectively integrating multimodal information to accurately detect and correct various errors in speaker attribution without altering the original transcription, and particularly ensuring the logical correctness and global consistency of role switching in long dialogues, has become a critical technical problem urgently needing to be solved in this field. Summary of the Invention

[0006] This invention provides a method, apparatus, device, and medium for speech multi-role separation based on a large model, aiming to solve the problem of how to accurately separate speech into multiple roles.

[0007] In a first aspect, embodiments of the present invention provide a speech multi-role separation method based on a large model, comprising:

[0008] Obtain raw audio data, and obtain initial attribution results based on the raw audio data. The initial attribution results include text transcription sequences with initial speaker labels assigned.

[0009] Calculate the multimodal anomaly scores of adjacent text segments in the text transcription sequence, and identify candidate error points based on the multimodal anomaly scores, wherein the multimodal anomaly scores integrate acoustic features, semantic features, and pause features;

[0010] For each candidate error point, a corresponding local time window is determined, sub-segmentation points are identified within the local time window, and the text segments within the local time window are re-divided based on the sub-segmentation points to update the text transcription sequence;

[0011] Based on the updated text transcription sequence, structured input data containing audio features, text features, and initial speaker labels is constructed and input into a preset large language model. The large language model is constrained by preset prompt words to output operation suggestions for segmentation, reallocation, or swapping without changing the original text content.

[0012] Based on the operational suggestions of the large language model, the globally optimal speaker label sequence is solved through a preset optimization algorithm to obtain the speaker attribution result.

[0013] A further technical solution is that obtaining the initial attribution result based on the original audio data includes:

[0014] The original audio data is preprocessed to obtain an audio segment sequence;

[0015] The audio segment sequence is subjected to speech recognition processing to generate a timestamped text transcription sequence, wherein the text transcription sequence contains start time, end time and confidence information at the word or phrase level;

[0016] Speaker embedding vectors are extracted based on the audio segment sequence, and cluster analysis is performed on the speaker embedding vectors to assign initial speaker labels to each segment in the text transcription sequence, thereby obtaining the initial attribution result.

[0017] A further technical solution is that the multimodal anomaly score is calculated based on the following formula:

[0018]

[0019] Where A(i) represents the anomaly score between the i-th adjacent text segments. sem_div(t) represents the cosine similarity of the centroid vectors of speaker clusters corresponding to adjacent text segments. i ,t i+1 ) represents the semantic difference between adjacent text segments, pause_score(i) represents the score of the pause duration between adjacent text segments, and α, β, and γ are preset adjustable weight coefficients.

[0020] A further technical solution is that identifying sub-segmentation points within the local time window includes:

[0021] Calculate word-level prosodic variation features within the local time window;

[0022] Detect question-and-answer patterns or read-aloud quoting patterns within the local time window;

[0023] Sub-segmentation points are determined based on the word-level prosodic variation features and the detected question-and-answer patterns or self-quoting patterns.

[0024] A further technical solution is that the preset prompt words include examples, and explicitly restrict the large language model to only output operation suggestions for segmentation, reallocation, or exchange of order, and prohibits adding, deleting, or rewriting text content.

[0025] A further technical solution is that the optimization algorithm is a dynamic programming algorithm or an integer linear programming algorithm; the objective function of the optimization algorithm is used to maximize acoustic similarity and semantic coherence, minimize unreasonable speaker switching frequency, and introduce a reward term for satisfying the operation suggestions of the large language model.

[0026] A further technical solution is that the method further includes:

[0027] Overlapping speech is detected in the audio segments of the audio segment sequence. When overlapping speech is detected, source separation or parallel annotation operation is performed.

[0028] An audit log is generated, which records the modification process, modification type, and evidence used from the initial attribution result to the final speaker attribution result.

[0029] Secondly, embodiments of the present invention also provide a speech multi-role separation device based on a large model, which includes a unit for performing the above-described method.

[0030] Thirdly, embodiments of the present invention also provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0031] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.

[0032] This invention provides a method, apparatus, device, and medium for speech multi-role separation based on a large model. The method includes: acquiring raw audio data; dividing the raw audio data to obtain an initial attribution result, the initial attribution result including a text transcription sequence with assigned initial speaker labels; calculating the multimodal anomaly scores of adjacent text segments in the text transcription sequence, and identifying candidate error points based on the multimodal anomaly scores, wherein the multimodal anomaly scores integrate acoustic features, semantic features, and pause features; for each candidate error point, determining a corresponding local time window, identifying sub-segmentation points within the local time window, and re-dividing the text segments within the local time window based on the sub-segmentation points to update the text transcription sequence; constructing structured input data containing audio features, text features, and initial speaker labels based on the updated text transcription sequence, and inputting it into a preset large language model, wherein the large language model is constrained by preset prompt words to output operation suggestions for segmentation, reallocation, or swapping sorting without changing the original text content; and based on the operation suggestions of the large language model, solving for the globally optimal speaker label sequence using a preset optimization algorithm to obtain the speaker attribution result. In this invention, firstly, multimodal anomaly scores are calculated by fusing acoustic, semantic, and pause features to accurately locate and separate error points; then, fine-grained analysis using local windows is employed to effectively identify complex errors such as multiple roles within a single sentence; further, constraint prompts guide the large language model to output safe and reliable operational suggestions, ensuring that the original text is not tampered with; finally, a global optimization algorithm coordinates local evidence and semantic logic to obtain results with accurate role attribution, natural turn order, and high overall consistency. Therefore, this method significantly improves the accuracy of multi-speaker separation in long speech. Attached Figure Description

[0033] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 is a flowchart illustrating a speech multi-role separation method based on a large model provided in an embodiment of the present invention;

[0035] Figure 2 is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0038] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0039] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0040] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0041] Please refer to Figure 1. This embodiment of the invention provides a speech multi-role separation method based on a large model. The method includes the following steps:

[0042] S1, Obtain the original audio data, and obtain the initial attribution result based on the original audio data. The initial attribution result includes a text transcription sequence with initial speaker labels assigned.

[0043] In practice, the original audio data is a standardized audio stream file (such as WAV or FLAC format), which is not specifically limited in this invention. Obtaining the initial attribution result based on the original audio data processing is the fundamental step in the entire correction process, constructing a structured data object that can be used for subsequent in-depth analysis.

[0044] While the initial attribution result may contain errors, it transforms continuous audio signals into serialized information with timestamps, text content, and preliminary speaker information through conventional speech activity detection, automatic speech recognition, and speaker clustering techniques. This is equivalent to establishing a preliminary index and skeleton for the raw, unstructured speech data, enabling all subsequent calculations and analyses to be performed within a unified framework that includes acoustic, textual, and temporal information.

[0045] In some preferred embodiments, the above step "obtaining initial attribution results based on the original audio data" specifically includes the following steps: preprocessing the original audio data to obtain an audio segment sequence; performing speech recognition processing on the audio segment sequence to generate a timestamped text transcription sequence, wherein the text transcription sequence contains word or phrase level start time, end time, and confidence information; extracting speaker embedding vectors based on the audio segment sequence, and performing cluster analysis on the speaker embedding vectors to assign initial speaker labels to each segment in the text transcription sequence, thereby obtaining the initial attribution results.

[0046] In practice, preprocessing the raw audio data can effectively improve the quality of the audio signal and reduce the interference of environmental noise and volume fluctuations on subsequent speech recognition and speaker feature extraction, thus laying an acoustic foundation for the accuracy of subsequent steps.

[0047] For example, preprocessing includes noise suppression, speech activity detection, and gain normalization, which are not specifically limited in this invention.

[0048] Noise suppression: Background noise is suppressed using spectral gating, Wiener filtering, or deep learning noise reduction models (such as RNNoise), and the improvement in signal-to-noise ratio is recorded.

[0049] Gain normalization: RMS normalization of audio volume to achieve the target energy level.

[0050] Voice Activity Detection (VAD): Performs coarse-grained segmentation of speech / silence intervals and outputs activity segment boundary information in the format {segment_id, start, end, speech_flag}. Suggested parameters: minimum speech segment length 0.25 seconds, minimum silence interval 0.15 seconds (adjustable according to the actual scenario).

[0051] After preprocessing, the output includes audio and a list of VAD segments.

[0052] Furthermore, the preprocessed audio is subjected to automatic speech recognition (ASR) and a timestamped text transcription sequence is generated. This not only transforms speech information into computable text information, but more importantly, it preserves precise time information at the word or phrase level. These timestamps are like marking the dialogue text with precise time coordinates, providing key temporal constraints for subsequent calculation of pause scores, fine-grained segmentation, and global optimization, so that all operations can be performed on an accurate timeline.

[0053] For example, the preprocessed audio and VAD segmentation results are input into the ASR model, and the output is a timestamped text transcription sequence. The output format is a sequence of words or phrases, each containing a word, start time, end time, and confidence score. Example: ("hello", 12.34, 12.60, 0.92). The recognition confidence score is retained for weighted decision-making in subsequent processing.

[0054] Examples of ASR models include Whisper (which supports timestamps), Conformer-Transducer, and wav2vec2+ language models; this invention does not specifically limit the types. Furthermore, ASR models should ideally output word-level or sub-word-level timestamps.

[0055] Further, a forced alignment step is performed, as follows:

[0056] Objective: To achieve precise word-level alignment between ASR transcribed text and audio signals, thereby improving segmentation accuracy.

[0057] Optional tools: Montreal Forced Aligner or CTC-based forced alignment methods; this invention is not specifically limited to any particular tool.

[0058] Output: A set of precise timestamps for each word {word, start, end}, used for subsequent prosodic analysis (such as the alignment of fundamental frequency f0 with energy at the word or frame level).

[0059] Finally, speaker embedding vectors are extracted from the audio segment sequences and cluster analysis is performed to assign initial speaker labels, establishing a preliminary association between voice features and speaker identity. Although errors may exist, it provides a baseline model that can be detected and corrected in subsequent processes. For example, the specific steps include the following:

[0060] Embedding extraction: Speaker embedding vectors e_i∈R^d are extracted using a sliding window (e.g., 0.5–1.0 seconds, step size 0.25 seconds). Models such as ECAPA-TDNN or x-vector can be used, but this invention is not specifically limited to them. Suggested parameters: embedding dimension d is 192 or 256; minimum merging threshold for short segments is 0.5 seconds.

[0061] Similarity calculation: Cosine similarity is used to measure the similarity between embedded vectors. The formula is:

[0062] cos_sim(e_i,e_j)=(e_i·e_j) / (||e_i||2·||e_j||2), where e_i and e_j are speaker embedding vectors.

[0063] Clustering method: First, perform spectral clustering to construct candidate clusters, and then use aggregated hierarchical clustering (AHC) to merge short time segments.

[0064] Cluster number estimation: Feature gap method (eigengap), silhouette coefficient or shortest occurrence duration prior (e.g., each speaker appears for at least 3 seconds) can be used, but the present invention does not specifically limit this.

[0065] Initial output: Assign initial speaker labels to each time period or word based on the clustering results, and generate preliminary speaker attribution transcription (preserving confidence).

[0066] S2, calculate the multimodal anomaly scores of adjacent text segments in the text transcription sequence, and identify candidate error points based on the multimodal anomaly scores, wherein the multimodal anomaly scores integrate acoustic features, semantic features and pause features.

[0067] In practice, after obtaining the initial attribution results, the crucial candidate error detection stage begins. This invention innovatively introduces the concept of a multimodal anomaly score, which does not rely on a single-dimensional feature but rather integrates a weighted consideration of acoustic features, semantic features, and pause features. Acoustic feature similarity directly reflects the essential correlation between adjacent speech segments, serving as the most direct physical evidence for determining whether a speaker has switched roles. Semantic feature difference analyzes the language's connotation, identifying situations where acoustic features are similar but semantically they clearly belong to a dialogue turn transition, such as question-and-answer pairs. Pause features introduce important contextual information about dialogue rhythm, as turn transitions are usually accompanied by specific pause patterns.

[0068] By fusing these three types of heterogeneous information into a quantified multimodal anomaly score, this method can quickly and accurately scan the entire transcribed text, much like an experienced proofreader, precisely locating "suspicious points" that are contradictory or incoherent in acoustics, semantics, or rhythm. This multimodal fusion-based screening significantly reduces the false negative rate of genuine errors compared to traditional methods that rely solely on acoustic clustering, and effectively avoids performing ineffective operations on correct paragraphs. This allows subsequent computational resources to be focused on the areas most in need of correction, thus laying a solid foundation for improving the overall efficiency and accuracy of the method.

[0069] For example, in some preferred embodiments, the multimodal anomaly score is calculated based on the following formula:

[0070]

[0071] Where A(i) represents the anomaly score between the i-th adjacent text segments. sem_div(t) represents the cosine similarity of the centroid vectors of speaker clusters corresponding to adjacent text segments. i ,t i+1 ) represents the semantic difference between adjacent text segments, pause_score(i) represents the score of the pause duration between adjacent text segments, and α, β, and γ are preset adjustable weight coefficients.

[0072] This formula does not rely on a single feature for judgment, but innovatively integrates acoustic features, semantic features, and pause features through a weighted linear combination. Among these, the acoustic similarity difference term... This directly reflects the essential differences in the sound features of adjacent segments and is the most direct evidence for determining whether a speaker has switched. For example, if the centroid similarity of adjacent sound segments is extremely low, this value will increase, strongly suggesting a possible role separation error. Semantic Difference Item (sem_div(t) i ,t i+1 The method provides auxiliary judgment at the linguistic content level. For example, if the first part is a question, "What is your opinion?", and the second part is an answer, "I agree," even if the acoustic features are similar (like one person quickly asking and answering themselves), the significant semantic difference will cause the score of this item to increase, thus triggering an alarm. This is particularly effective for detecting "multiple roles in a single sentence" errors. The pause duration score (pause_score(i)) introduces common-sense rules of dialogue rhythm and turn-taking. Generally, a longer pause is more likely to indicate a change in speaker. By adjusting the weight coefficients α, β, and γ, this method can adjust the relative importance of the three types of evidence according to different application scenarios (such as formal meetings and fast-paced conversations). For example, in a fast-paced debate, the weight of semantic differences can be appropriately increased.

[0073] This multimodal fusion-based quantitative scoring mechanism, compared to methods that rely solely on acoustic clustering or simple rules, can more comprehensively capture various types of anomalies, greatly reducing the probability of missed and false detections. It provides the entire method with highly reliable and flexible error localization capabilities, and is a key first step in achieving accurate correction.

[0074] In some preferred embodiments, the semantic dissimilarity is obtained by calculating the cosine distance or KL divergence of the semantic embedding vectors of adjacent text segments.

[0075] In practice, semantic embedding vectors map text to a high-dimensional vector space, making semantically similar texts closer together in space. Calculating cosine distance essentially measures the difference in direction between two vectors, and is insensitive to the absolute magnitude of the vectors. This is well-suited for measuring the semantic similarity of texts because it focuses more on the relevance of content than the length of expression. For example, the vector directions of "How are you?" and "How have you been lately?" would be very close, resulting in a small cosine distance.

[0076] KL divergence measures the difference between two probability distributions. If the semantic embedding is interpreted as a probability distribution, KL divergence can capture more subtle differences in information. Providing these two optional metrics increases the flexibility of the method, allowing the selection of the most suitable algorithm based on the actual embedding model used and specific needs. This difference calculation based on deep semantic embedding understands the deeper meaning of language far better than simple keyword matching or bag-of-words models, thus accurately identifying adjacent paragraphs that differ superficially in word choice but belong to the same turn (such as question and answer), or that appear similar but have actually switched topics (potentially indicating a change in speakers).

[0077] Furthermore, threshold determination: if A(i) > τ_detect (e.g., initially set to 0.6), then the point is included as a candidate error point in the candidate error point set.

[0078] Furthermore, additional rules are provided: if adjacent paragraphs exhibit sentence patterns such as "question-statement" or "imperative-response", the semantic difference weight should be increased; if the ASR confidence of two adjacent paragraphs is low, their priority for entering manual or LLM review should be increased.

[0079] S3. For each candidate error point, determine the corresponding local time window, identify sub-segmentation points within the local time window, and re-divide the text segments within the local time window based on the sub-segmentation points to update the text transcription sequence.

[0080] In its implementation, this invention does not employ a coarse global adjustment strategy for the screened candidate errors. Instead, it enters a more refined local time window analysis stage. This minimizes interference with the correct parts while achieving in-depth analysis of specific types of complex errors (especially multi-role errors in single sentences). This method defines a finite local time window (e.g., 3-10 seconds, which can be dynamically expanded according to the speech rate; this invention does not specifically limit this) for each candidate error. Within this local time window, fine-grained analysis is performed to identify potential sub-segmentation points. The advantage of this localized processing is that it constrains the analysis scope to the local context in which the problem may occur, avoiding error propagation that might result from a global model and reducing computational complexity.

[0081] Within a local time window, a multi-dimensional investigation is conducted, including prosodic feature analysis (such as fundamental frequency and energy changes) and deep semantic analysis (such as semantic role labeling to identify question-and-answer and citation structures). This surface-to-depth analytical approach is particularly effective in addressing the thorny issue of "single-person discourse containing multiple semantic roles." For example, for sentences like "What's your name? My name is Zhang San," which are mistakenly attributed to the same speaker, analyzing the micro-prosodic changes and semantic roles between words can provide strong evidence to support segmentation between the question and answer. Furthermore, the text segments within the local time window are re-divided based on the sub-segmentation points to update the text transcription sequence.

[0082] In some preferred embodiments, the above step "identifying sub-segmentation points within the local time window" specifically includes the following steps: calculating word-level prosodic variation features within the local time window; detecting question-and-answer patterns or self-quote patterns within the local time window; and determining sub-segmentation points based on the word-level prosodic variation features and the detected question-and-answer patterns or self-quote patterns.

[0083] In practice, the analysis of word-level prosodic features starts from the physical level of the speech signal. For example, when a person imitates or switches to different roles in the same sentence, their pitch (fundamental frequency), volume (energy), and rhythm (inter-word pauses) usually undergo unconscious, subtle, but detectable changes. Detecting question-and-answer patterns or self-quotation patterns is judged from the logical level of linguistics. For example, semantic role labeling identifies the "questioner" and "responder" roles in the sentence, or detects the structure of direct quotations (such as "He said: 'I agree'"). Combining these two types of evidence greatly improves the reliability of sub-segmentation point identification. For example, for a sentence like "What's your name? My name is Zhang San," which is incorrectly attributed to the same speaker, the method first uses prosodic analysis to potentially find a slight fundamental frequency jump and a short pause between the word "word" and "I"; simultaneously, semantic analysis identifies it as a typical question-and-answer structure, with the first part being a question and the second part a statement. The physical and logical evidence corroborate each other, strongly indicating that sub-segmentation should be performed at this point. This refined analysis goes beyond the limitations of traditional acoustic clustering, which only focuses on differences between segments, and can delve into the internal structure of speech segments to reveal their inherent role switching clues.

[0084] For example, the specific steps include the following:

[0085] Prosodic variation detection steps: The word-level prosodic variation features include inter-word energy difference, fundamental frequency difference, and inter-word pause interval. Accordingly, the inter-word energy difference Δenergy_j, fundamental frequency difference Δf0_j, pause interval word_gap_j, etc., are calculated, and a prosodic variation score is constructed.

[0086] prosody_change(j)=w_p·Δenergy_j+w_f·Δf0_j+w_s·I[word_gap_j>t_pause]

[0087] Wherein, prosody_change(j) represents the prosodic change score calculated at the position between the j-th words (i.e., between the j-th word and the (j+1)-th word). This score is used to quantify the probability of a speaker switch occurring at this position; the larger the value, the more significant the prosodic feature change, and the more likely it is to be a sub-segmentation point.

[0088] w_p, w_f, and w_s are weighting coefficients used to adjust the contribution of energy variation, fundamental frequency variation, and pause features to the overall score, respectively. These weights are hyperparameters and are typically tuned based on experimental results on a development set to optimize detection performance.

[0089] Δenergy_j: Represents the energy change (or volume change) calculated at the position of the j-th word. Specifically, it is the difference in average energy between two adjacent words (the j-th word and the (j+1)-th word).

[0090] Δf0_j: Represents the change in fundamental frequency (or pitch change) calculated at the j-th word position. Fundamental frequency is the basic frequency of a speech signal and is one of the key parameters for distinguishing the acoustic characteristics of a speaker. The change here is also the difference in the average fundamental frequency between two adjacent words.

[0091] I[word_gap_j>t_pause]: This is an indicator function.

[0092] word_gap_j represents the duration of silence or pause between the j-th word and the (j+1)-th word.

[0093] t_pause is a preset pause duration threshold, for example, 0.20–0.40 seconds, which is not specifically limited in this invention.

[0094] This function works as follows: if the word gap (word_gap_j) is greater than the threshold (t_pause), the function takes a value of 1; otherwise, it takes a value of 0. It is used to capture speaker switching that may be caused by obvious pauses.

[0095] If prosody_change(j) > τ_prosody and the text contains semantic breakpoints (such as commas, interrogative words, etc.), then it is considered a sub-segmentation point. τ_prosody is a preset threshold, for example, set to 0.3–0.6, which is not specifically limited in this invention.

[0096] Steps for identifying question-and-answer patterns or self-quote patterns: Perform syntactic dependency analysis or semantic role labeling on the statements in the window to detect question-and-answer patterns or self-quote patterns (such as strong prompts for segmentation when a question word is followed by a first-person statement).

[0097] If a question-and-answer pattern is detected, a sub-segmentation point is added between the question and the answer. For example, for the text "What's your name? My name is Zhang San.", a sub-segmentation point is inserted between "What's your name?" and "My name is Zhang San".

[0098] If a self-quoting pattern is detected, a sub-segmentation point is added before the self-quoting sentence and the quoted sentence. For example, for the text "He turned to me and said, 'I will definitely come back.'"

[0099] In this sentence, "He turned to me and said" is a statement by speaker A (self-narrative sentence), while "I will definitely come back" is a quotation sentence, with "he" as the agent. Therefore, a sub-separation point should be added between "He turned to me and said" and "I will definitely come back".

[0100] Verification steps: Recalculate the word embedding vector e_left of the left text segment and the word embedding vector e_right of the right text segment on both sides of the candidate segmentation point. If cos_sim(e_left,e_right)<τ_split (e.g., 0.6), then the segmentation is supported, meaning that the sub-segmentation point is reliable.

[0101] Output steps: For each text segment segmented by a sub-segmentation point, generate the sub-segment sequence of the text segment and update the corresponding time span and word embedding vector. At the same time, update the text transcription sequence according to the text segmentation result.

[0102] S4. Based on the updated text transcription sequence, construct structured input data containing audio features, text features, and initial speaker labels, and input it into a preset large language model. The large language model is constrained by preset prompt words to output operation suggestions for segmentation, reallocation, or exchange sorting without changing the original text content.

[0103] In its implementation, the powerful tool Large Language Model (LLM) was introduced, but its application was subject to key innovative constraints, achieving a safe, controllable and efficient intelligent decision-making assistance. It cleverly utilizes the excellent semantic understanding capabilities of the Large Language Model while completely avoiding the "illusion" problem that it may produce.

[0104] By using carefully designed preset prompts, the behavioral boundaries of the large language model are strictly constrained. It is clearly limited to outputting operation suggestions such as segmentation, reassignment, or order swapping, and any addition, deletion, or rewriting of the original text content is strictly prohibited. This ensures that the powerful capabilities of the large language model are precisely guided to solve the specific task of role attribution, making it an extremely professional and reliable "intelligent advisor" without arbitrarily altering the original content. This ensures that the output of the correction process is strictly faithful to the original speech recognition results.

[0105] In some preferred embodiments, the preset prompts include examples and explicitly constrain the large language model to only output operation suggestions for segmentation, reallocation, or swapping, and prohibit the addition, deletion, or rewriting of text content.

[0106] In practical implementation, the example-containing prompts (i.e., Few-shot Learning) provide the large language model with a clear task template, enabling it to quickly and accurately understand the task boundaries and expected output format of this invention. For example, presenting an "input-output" pair, where the input is structured data containing acoustic and semantic evidence, and the output is an operation suggestion in a specified JSON format, significantly reduces the possibility of the model misunderstanding the task and improves the accuracy of the first response. More importantly, explicitly constraining the large language model to only output specific operation suggestions and prohibiting the addition, deletion, or rewriting of text content is equivalent to drawing an inviolable red line for its operations. Without this constraint, the large language model might arbitrarily modify word choice or supplement content in pursuit of text fluency. Through this constraint, the method ensures that the correction process does not introduce any new distortions, and all corrections are strictly based on the text generated by the original speech recognition, thus strictly guaranteeing the "authenticity" of the content while utilizing artificial intelligence to improve the level of intelligence.

[0107] For example, structured input data can be specifically in JSON format. An example of structured input data is shown below:

[0108] The meaning of the above structured input data is explained as follows:

[0109] 1. window_id

[0110] Meaning: The unique identifier of the current analysis window.

[0111] Purpose: To track and record which specific dialogue segment is being processed by the LLM, facilitating subsequent log auditing and result matching.

[0112] 2. segments (array)

[0113] Meaning: Contains detailed information on all speech segments to be analyzed within the current window.

[0114] Each fragment object contains:

[0115] segment_id: The unique ID of the segment (e.g., "s1", "s2").

[0116] start / end: ​​The start and end times (in seconds) of this segment in the audio.

[0117] text: The ASR transcription text corresponding to this fragment.

[0118] word_timestamps: An array of timestamps at the word level (used for fine alignment).

[0119] embed_centroid: The speaker embedding vector (acoustic feature) of this segment.

[0120] cluster_label: Speaker labels assigned after initial clustering (e.g., "A", "B").

[0121] acoustic_conf: The confidence level of the acoustic model in the speaker to whom the segment belongs.

[0122] asr_conf: The confidence level of ASR in transcribing this text segment.

[0123] 3. Adjacency (adjacency relationship)

[0124] Meaning: It provides quantitative evidence of the relationships between segments.

[0125] Include:

[0126] pause_durations: The duration of silent pauses between adjacent segments.

[0127] cos_sim_adj: Speaker embedding cosine similarity (acoustic similarity) of adjacent segments.

[0128] semantic_sim: The semantic similarity between adjacent text segments (a lower similarity usually indicates that the text may have been written by someone else).

[0129] 4. Constraints

[0130] Meaning: Strict instructions given to the LLM to ensure that its output conforms to the requirements of this method.

[0131] Key constraints:

[0132] Adding or modifying text content is prohibited (to prevent the "illusion" of LLM and ensure the authenticity of the content).

[0133] Only three operations are allowed:

[0134] split: to divide a segment into multiple speakers.

[0135] reassign: Changes the speaker label of a segment.

[0136] swap_order (swap order): Adjusts the order of segments.

[0137] The output must be in JSON format: this facilitates programmatic parsing and integration.

[0138] The structured input data described above is essentially a highly structured "problem description package," which integrates the following information and is then fed into a large language model for intelligent judgment:

[0139] "What it is": the original transcribed text and preliminary speaker labels.

[0140] "Where is the evidence?": Multimodal data including acoustic features, temporal information, and semantic relationships.

[0141] "What can it do": By strictly limiting the operational scope of the large language model through constraints, we can ensure that its output is controllable, parsable, and meets business requirements.

[0142] The ultimate goal is to enable large language models to provide correction suggestions for speaker attribution, segment segmentation, and order without altering the original text, based on multimodal evidence and semantic understanding, thereby solving the common multi-role separation errors in traditional speech recognition systems.

[0143] Furthermore, key design considerations for Prompt prompts:

[0144] Provide 2–4 sets of "input → correct adjustment" examples to clearly define the JSON output format;

[0145] Emphasis on constraints: Only splitting, redistribution, or swapping the order is allowed; adding, deleting, or modifying the text is prohibited.

[0146] The required output fields include: segment_id, action, split_offsets, new_speaker, confidence, and evidence_summary.

[0147] Among them, 1. The meaning of segment_id: the unique identifier of the audio segment to be processed.

[0148] 2. Meaning of action: The specific type of operation that LLM recommends to be performed.

[0149] Typical operations include:

[0150] split: This option suggests splitting the current segment in two at the specified location.

[0151] reassign: Reassign. It is recommended to change the speaker label of the current segment from A to B.

[0152] swap_order: The order in which the segments are swapped. It is recommended to adjust the order of two adjacent segments.

[0153] 3. The meaning of split_offsets: When the action is "split", the suggested specific split point position.

[0154] Purpose: This position can be a timestamp (e.g., 12.5 seconds after the start of the segment) or a word index (e.g., after the 3rd word of the segment). This provides precise coordinates for subsequent fine-grained processing.

[0155] 4. The meaning of new_speaker: When a new segment is generated after the action is "reassignment" or "splitting", it is recommended to assign the new speaker label to the segment.

[0156] Function: Labels can be abstract (e.g., "B") or associated with preliminary clustering results (e.g., "SPK_2"). LLM determines which speaker a given segment is more likely to belong to based on semantic and acoustic evidence from the input.

[0157] 5. Meaning of confidence: The confidence score of the LLM for its recommendations.

[0158] Purpose: This is a numerical value (e.g., from 0.0 to 1.0) that reflects the degree of confidence that the LLM (Lesson Learning Model) believes the suggestion is correct based on the input evidence. This score will serve as a weighting factor (Reward_LLM) in the subsequent global combinatorial optimization objective function. High-confidence suggestions will receive higher reward scores, thus being more likely to be adopted.

[0159] 6. Meaning of evidence_summary: A brief textual summary of the key evidence on which the LLM made this recommendation.

[0160] Function: This is a readable reason for explanation. For example: "Evidence: The first part is an interrogative sentence ('What is your name?') and the second part is a declarative sentence containing the first person 'I' ('My name is Zhang San'), which conforms to the question-and-answer pattern."

[0161] S5. Based on the operational suggestions of the large language model, the globally optimal speaker label sequence is solved through a preset optimization algorithm to obtain the speaker attribution result.

[0162] In practice, after obtaining local correction suggestions from the large language model, the system enters a global combinatorial optimization phase. This ensures the global consistency and naturalness of the final output at the system level, resolving potential logical conflicts or sequence disorder caused by local optimization. Specifically, through a pre-defined optimization algorithm, all local evidence (acoustic similarity, semantic coherence) and local suggestions (large language model output) are incorporated into a unified objective function for global planning. This objective function comprehensively considers acoustic matching degree, semantic fluency, the naturalness of turn-taking (e.g., penalizing unreasonable frequent turns), and rewards for adopting reasonable suggestions from the large language model. Using mathematical tools such as dynamic programming or integer linear programming, the speaker label sequence that maximizes this objective function is solved.

[0163] In some preferred embodiments, the optimization algorithm is a dynamic programming algorithm or an integer linear programming algorithm, but the present invention is not specifically limited to these.

[0164] Furthermore, the objective function of the optimization algorithm is used to maximize acoustic similarity and semantic coherence, and minimize unreasonable speaker switching frequency, while introducing a reward term for satisfying the operational suggestions of the large language model.

[0165] In practice, this objective function is not a simple maximization of a single metric, but rather a reflection of multi-objective trade-offs. Maximizing acoustic similarity ensures that the speaker labels assigned to each segment are the most matching in terms of vocal features, which is the foundation of accuracy. Maximizing semantic coherence (especially when adjacent segments are assigned the same speaker labels) considers the natural flow of the dialogue, avoiding the counterintuitive result of forcibly splitting semantically closely connected sentences to different speakers. Introducing a speaker switching penalty term simulates the inertia of turn-taking in real dialogue, avoiding unnatural and frequent speaker switching caused by excessive pursuit of local acoustic matching. The design of this term being inversely proportional to the pause duration further aligns with the dialogue rule that "the longer the pause, the greater the likelihood of switching." Finally, introducing a reward term for satisfying LLM recommendations integrates the local intelligent judgments of the large language model based on semantics and logic into the global optimization framework in the form of soft constraints, so that the global decision-making respects both acoustic evidence and semantic intelligence. By adjusting the weight coefficients (λ1, λ2, λ3) of these terms, this method can adapt to application scenarios with different styles.

[0166] Specifically, the objective function is as follows:

[0167]

[0168] Where L represents the global speaker label sequence, i.e., L = {l1, l2, ..., l...} N}, where l i Speaker label assigned to the i-th time period (or speech segment).

[0169] N: Represents the total number of time periods, i.e. the total number of audio segments.

[0170] Sim acoustic (i,l i ): Represents the acoustic features of the i-th time period and the speaker label l i The similarity between the corresponding acoustic models. This similarity is usually calculated based on the cosine similarity of the speaker embedding vectors (such as vectors extracted by ECAPA-TDNN or x-vector), using the formula: Sim acoustic (i,l)=cos_sim(e i ,μ li );

[0171] Where e i It is the embedding vector for the i-th time period, μ li It is the centroid vector of the speaker's label li.

[0172] λ1: Weighting coefficient of the semantic coherence term, used to adjust the importance of semantic coherence in the objective function.

[0173] Sim semantic (i,i+1|li,li+1): Represents the semantic coherence between adjacent time segments i and i+1, taking into account their speaker labels li and li+1. This value is typically calculated based on the similarity of the semantic embeddings of the transcribed text (such as SBERT vectors), with higher expected semantic coherence if adjacent segments belong to the same speaker.

[0174] λ2: Weighting coefficient of speaker switching penalty term, used to adjust the importance of switching penalty in the objective function.

[0175] SwitchPenalty(i): Represents the penalty for switching speakers between time periods i and i+1. This penalty is usually inversely proportional to the pause duration between adjacent segments; that is, the shorter the pause duration, the greater the penalty, in order to suppress unreasonable frequent switching.

[0176] λ3: Weighting coefficient of LLM reward terms, used to adjust the importance of LLM recommendation rewards in the objective function.

[0177] S LLM : Represents the set of all operation suggestions output by the large language model. Each suggestion s may involve operations such as splitting, reassignment, or changing the order.

[0178] Reward LLM(s,L): This represents the reward value given if the speaker label sequence L satisfies the suggestion s of the Large Language Model (LLM). This reward term is used to encourage sequence L to conform to the local correction suggestions of the LLM.

[0179] Solution method: If the state space is moderate, Viterbi dynamic programming can be used to find an approximate optimal solution;

[0180] If complex hard constraints exist (such as prohibiting switching at certain positions), it can be transformed into an integer linear programming (ILP) problem, solved using solvers such as Gurobi or CBC. Example constraint: The "hard_split" position in the LLM output must not be merged back into the original segment.

[0181] The output after solving is: the global optimal label sequence and the confidence of each label (through score normalization or posterior probability estimation), which gives the speaker attribution result.

[0182] In some preferred embodiments, the method further includes the following steps: performing overlapping speech detection on the audio segments in the audio segment sequence; when overlapping speech is detected, performing source separation or parallel annotation operation; generating an audit log, the audit log recording the modification process, modification type and evidence used from the initial attribution result to the final speaker attribution result.

[0183] In practical implementation, overlapping speech detection is performed on the audio segments in the audio segment sequence. When overlapping speech is detected, source separation is performed to attempt to separate the mixed speech signals, and then they are identified and attributed separately. This is a method that fundamentally solves the problem. When the separation effect is not ideal or computational resources are limited, using parallel labeling (such as speaker_A+speaker_B) is a pragmatic and information-free remedial measure, which significantly enhances the robustness of the method in scenarios such as heated discussions and interruptions, as well as the completeness of the output information. For example, as follows:

[0184] Overlapping speech detection: If multiple sound sources or multi-channel evidence indicate that there is overlap in an audio segment, multiple speaker labels are allowed to be output.

[0185] Overlap processing strategy: Prioritize speech source separation methods (such as Conv-TasNet, DPRNN or multi-channel beamforming) to process the overlapping parts, and then perform ASR and alignment on the separated signals respectively; if this is not feasible, use parallel labeling (such as speaker_A+speaker_B) in the transcription.

[0186] Furthermore, it also includes logging and quality auditing steps: generating an audit log for each modification, including the original paragraph, modification type, LLM output, evidence indicators, and processing time, to facilitate manual spot checks and rollbacks.

[0187] Generating audit logs is of significant management and quality control importance. It meticulously records the complete modification chain from the initial attribution result to the final result, including where the modification occurred, why it was modified (based on acoustic evidence or LLM recommendations), how it was modified (segmentation, reallocation, etc.), and the level of confidence in the modification, providing a complete "source archive" for the output. For users, it allows for quick identification of potentially uncertain correction points for manual review, improving the credibility of the results. For system developers, logs are a valuable data source for optimizing algorithms and adjusting thresholds. In high-risk fields such as the judiciary and finance, this auditability is a fundamental requirement for compliance.

[0188] This invention proposes a speech multi-role separation method based on a large model, comprising: acquiring raw audio data; dividing the raw audio data to obtain an initial attribution result, the initial attribution result including a text transcription sequence assigned initial speaker labels; calculating the multimodal anomaly scores of adjacent text segments in the text transcription sequence, and identifying candidate error points based on the multimodal anomaly scores, wherein the multimodal anomaly scores integrate acoustic features, semantic features, and pause features; for each candidate error point, determining a corresponding local time window, identifying sub-segmentation points within the local time window, and re-segmenting the text segments within the local time window based on the sub-segmentation points to update the text transcription sequence; constructing structured input data containing audio features, text features, and initial speaker labels based on the updated text transcription sequence, and inputting it into a preset large language model, wherein the large language model is constrained by preset prompt words to output operation suggestions for segmentation, reallocation, or exchange sorting without changing the original text content; and based on the operation suggestions of the large language model, solving for the globally optimal speaker label sequence through a preset optimization algorithm to obtain the speaker attribution result. In this invention, firstly, multimodal anomaly scores are calculated by fusing acoustic, semantic, and pause features to accurately locate and separate error points; then, fine-grained analysis using local windows is employed to effectively identify complex errors such as multiple roles within a single sentence; further, constraint prompts guide the large language model to output safe and reliable operational suggestions, ensuring that the original text is not tampered with; finally, a global optimization algorithm coordinates local evidence and semantic logic to obtain results with accurate role attribution, natural turn order, and high overall consistency. Therefore, this method significantly improves the accuracy of multi-speaker separation in long speech.

[0189] Corresponding to the above-described large-model-based speech multi-role separation method, this invention also provides a large-model-based speech multi-role separation device. This large-model-based speech multi-role separation device includes a unit for executing the above-described large-model-based speech multi-role separation method, and can be configured in a desktop computer, tablet computer, laptop computer, or other terminal. Specifically, the large-model-based speech multi-role separation device includes:

[0190] A segmentation unit is used to acquire raw audio data and segment it based on the raw audio data to obtain an initial attribution result, wherein the initial attribution result includes a text transcription sequence with initial speaker labels assigned.

[0191] The computing unit is used to calculate the multimodal anomaly scores of adjacent text segments in the text transcription sequence, and to identify candidate error points based on the multimodal anomaly scores, wherein the multimodal anomaly scores integrate acoustic features, semantic features and pause features;

[0192] The determining unit is configured to determine a corresponding local time window for each candidate error point, identify sub-segmentation points within the local time window, and re-divide the text segments within the local time window based on the sub-segmentation points to update the text transcription sequence.

[0193] The input unit is used to construct structured input data containing audio features, text features and initial speaker labels based on the updated text transcription sequence, and input it into a preset large language model. The large language model is constrained by preset prompt words to output operation suggestions for segmentation, reallocation or exchange sorting without changing the original text content.

[0194] The solution unit is used to solve for the globally optimal speaker label sequence based on the operation suggestions of the large language model and through a preset optimization algorithm to obtain the speaker attribution result.

[0195] In some preferred embodiments, obtaining the initial attribution result based on the original audio data includes:

[0196] The original audio data is preprocessed to obtain an audio segment sequence;

[0197] The audio segment sequence is subjected to speech recognition processing to generate a timestamped text transcription sequence, wherein the text transcription sequence contains start time, end time and confidence information at the word or phrase level;

[0198] Speaker embedding vectors are extracted based on the audio segment sequence, and cluster analysis is performed on the speaker embedding vectors to assign initial speaker labels to each segment in the text transcription sequence, thereby obtaining the initial attribution result.

[0199] In some preferred embodiments, the multimodal anomaly score is calculated based on the following formula:

[0200]

[0201] Where A(i) represents the anomaly score between the i-th adjacent text segments. sem_div(t) represents the cosine similarity of the centroid vectors of speaker clusters corresponding to adjacent text segments. i ,t i+1 ) represents the semantic difference between adjacent text segments, pause_score(i) represents the score of the pause duration between adjacent text segments, and α, β, and γ are preset adjustable weight coefficients.

[0202] In some preferred embodiments, identifying sub-segmentation points within the local time window includes:

[0203] Calculate word-level prosodic variation features within the local time window;

[0204] Detect question-and-answer patterns or read-aloud quoting patterns within the local time window;

[0205] Sub-segmentation points are determined based on the word-level prosodic variation features and the detected question-and-answer patterns or self-quoting patterns.

[0206] In some preferred embodiments, the preset prompts include examples and explicitly constrain the large language model to only output operation suggestions for segmentation, reallocation, or swapping, and prohibit the addition, deletion, or rewriting of text content.

[0207] In some preferred embodiments, the optimization algorithm is a dynamic programming algorithm or an integer linear programming algorithm; the objective function of the optimization algorithm is used to maximize acoustic similarity and semantic coherence, minimize unreasonable speaker switching frequency, and introduce a reward term for satisfying the operational suggestions of the large language model.

[0208] In some preferred embodiments, the large-model-based speech multi-role separation device further includes:

[0209] Overlapping speech is detected in the audio segments of the audio segment sequence. When overlapping speech is detected, source separation or parallel annotation operation is performed.

[0210] An audit log is generated, which records the modification process, modification type, and evidence used from the initial attribution result to the final speaker attribution result.

[0211] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned large-model-based voice multi-role separation device and its various units can be found in the corresponding descriptions in the foregoing method embodiments. For the sake of convenience and brevity, these details will not be repeated here.

[0212] The aforementioned large-model-based voice multi-role separation device can be implemented as a computer program that can run on the computer device shown in Figure 2.

[0213] Please refer to Figure 2, which is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server. The terminal can be an electronic device with communication functions, such as a smartphone, tablet, laptop, desktop computer, personal digital assistant, or wearable device. The server can be a standalone server or a server cluster composed of multiple servers.

[0214] The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0215] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it causes the processor 502 to execute a speech multi-role separation method based on a large model.

[0216] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0217] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a speech multi-role separation method based on a large model.

[0218] The network interface 505 is used for network communication with other devices. Those skilled in the art will understand that the above structure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. A specific computer device 500 may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.

[0219] The processor 502 is used to run a computer program 5032 stored in the memory to perform the following steps:

[0220] Obtain raw audio data, and obtain initial attribution results based on the raw audio data. The initial attribution results include text transcription sequences with initial speaker labels assigned.

[0221] Calculate the multimodal anomaly scores of adjacent text segments in the text transcription sequence, and identify candidate error points based on the multimodal anomaly scores, wherein the multimodal anomaly scores integrate acoustic features, semantic features, and pause features;

[0222] For each candidate error point, a corresponding local time window is determined, sub-segmentation points are identified within the local time window, and the text segments within the local time window are re-divided based on the sub-segmentation points to update the text transcription sequence;

[0223] Based on the updated text transcription sequence, structured input data containing audio features, text features, and initial speaker labels is constructed and input into a preset large language model. The large language model is constrained by preset prompt words to output operation suggestions for segmentation, reallocation, or swapping without changing the original text content.

[0224] Based on the operational suggestions of the large language model, the globally optimal speaker label sequence is solved through a preset optimization algorithm to obtain the speaker attribution result.

[0225] In some preferred embodiments, obtaining the initial attribution result based on the original audio data includes:

[0226] The original audio data is preprocessed to obtain an audio segment sequence;

[0227] The audio segment sequence is subjected to speech recognition processing to generate a timestamped text transcription sequence, wherein the text transcription sequence contains start time, end time and confidence information at the word or phrase level;

[0228] Speaker embedding vectors are extracted based on the audio segment sequence, and cluster analysis is performed on the speaker embedding vectors to assign initial speaker labels to each segment in the text transcription sequence, thereby obtaining the initial attribution result.

[0229] In some preferred embodiments, the multimodal anomaly score is calculated based on the following formula:

[0230]

[0231] Where A(i) represents the anomaly score between the i-th adjacent text segments. sem_div(t) represents the cosine similarity of the centroid vectors of speaker clusters corresponding to adjacent text segments. i,t i+1 ) represents the semantic difference between adjacent text segments, pause_score(i) represents the score of the pause duration between adjacent text segments, and α, β, and γ are preset adjustable weight coefficients.

[0232] In some preferred embodiments, identifying sub-segmentation points within the local time window includes:

[0233] Calculate word-level prosodic variation features within the local time window;

[0234] Detect question-and-answer patterns or read-aloud quoting patterns within the local time window;

[0235] Sub-segmentation points are determined based on the word-level prosodic variation features and the detected question-and-answer patterns or self-quoting patterns.

[0236] In some preferred embodiments, the preset prompts include examples and explicitly constrain the large language model to only output operation suggestions for segmentation, reallocation, or swapping, and prohibit the addition, deletion, or rewriting of text content.

[0237] In some preferred embodiments, the optimization algorithm is a dynamic programming algorithm or an integer linear programming algorithm; the objective function of the optimization algorithm is used to maximize acoustic similarity and semantic coherence, minimize unreasonable speaker switching frequency, and introduce a reward term for satisfying the operational suggestions of the large language model.

[0238] In some preferred embodiments, the method further includes:

[0239] Overlapping speech is detected in the audio segments of the audio segment sequence. When overlapping speech is detected, source separation or parallel annotation operation is performed.

[0240] An audit log is generated, which records the modification process, modification type, and evidence used from the initial attribution result to the final speaker attribution result.

[0241] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0242] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0243] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program. When executed by a processor, the computer program causes the processor to perform the following steps:

[0244] Obtain raw audio data, and obtain initial attribution results based on the raw audio data. The initial attribution results include text transcription sequences with initial speaker labels assigned.

[0245] Calculate the multimodal anomaly scores of adjacent text segments in the text transcription sequence, and identify candidate error points based on the multimodal anomaly scores, wherein the multimodal anomaly scores integrate acoustic features, semantic features, and pause features;

[0246] For each candidate error point, a corresponding local time window is determined, sub-segmentation points are identified within the local time window, and the text segments within the local time window are re-divided based on the sub-segmentation points to update the text transcription sequence;

[0247] Based on the updated text transcription sequence, structured input data containing audio features, text features, and initial speaker labels is constructed and input into a preset large language model. The large language model is constrained by preset prompt words to output operation suggestions for segmentation, reallocation, or swapping without changing the original text content.

[0248] Based on the operational suggestions of the large language model, the globally optimal speaker label sequence is solved through a preset optimization algorithm to obtain the speaker attribution result.

[0249] In some preferred embodiments, obtaining the initial attribution result based on the original audio data includes:

[0250] The original audio data is preprocessed to obtain an audio segment sequence;

[0251] The audio segment sequence is subjected to speech recognition processing to generate a timestamped text transcription sequence, wherein the text transcription sequence contains start time, end time and confidence information at the word or phrase level;

[0252] Speaker embedding vectors are extracted based on the audio segment sequence, and cluster analysis is performed on the speaker embedding vectors to assign initial speaker labels to each segment in the text transcription sequence, thereby obtaining the initial attribution result.

[0253] In some preferred embodiments, the multimodal anomaly score is calculated based on the following formula:

[0254]

[0255] Where A(i) represents the anomaly score between the i-th adjacent text segments. sem_div(t) represents the cosine similarity of the centroid vectors of speaker clusters corresponding to adjacent text segments. i ,t i+1 ) represents the semantic difference between adjacent text segments, pause_score(i) represents the score of the pause duration between adjacent text segments, and α, β, and γ are preset adjustable weight coefficients.

[0256] In some preferred embodiments, identifying sub-segmentation points within the local time window includes:

[0257] Calculate word-level prosodic variation features within the local time window;

[0258] Detect question-and-answer patterns or read-aloud quoting patterns within the local time window;

[0259] Sub-segmentation points are determined based on the word-level prosodic variation features and the detected question-and-answer patterns or self-quoting patterns.

[0260] In some preferred embodiments, the preset prompts include examples and explicitly constrain the large language model to only output operation suggestions for segmentation, reallocation, or swapping, and prohibit the addition, deletion, or rewriting of text content.

[0261] In some preferred embodiments, the optimization algorithm is a dynamic programming algorithm or an integer linear programming algorithm; the objective function of the optimization algorithm is used to maximize acoustic similarity and semantic coherence, minimize unreasonable speaker switching frequency, and introduce a reward term for satisfying the operational suggestions of the large language model.

[0262] In some preferred embodiments, the method further includes:

[0263] Overlapping speech is detected in the audio segments of the audio segment sequence. When overlapping speech is detected, source separation or parallel annotation operation is performed.

[0264] An audit log is generated, which records the modification process, modification type, and evidence used from the initial attribution result to the final speaker attribution result.

[0265] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.

[0266] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0267] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0268] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0269] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0270] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0271] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Since these modifications and variations fall within the scope of the claims and their equivalents, this invention also intends to include these modifications and variations.

[0272] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A speech multi-role separation method based on a large model, characterized in that, include: The process involves: acquiring raw audio data; dividing the raw audio data to obtain initial attribution results, which include a text transcription sequence with assigned initial speaker labels; calculating multimodal anomaly scores for adjacent text segments in the text transcription sequence and identifying candidate error points based on these scores, where the multimodal anomaly scores integrate acoustic features, semantic features, and pause features; determining a corresponding local time window for each candidate error point; identifying sub-segmentation points within the local time window; re-dividing the text segments within the local time window based on these sub-segmentation points to update the text transcription sequence; constructing structured input data containing audio features, text features, and initial speaker labels based on the updated text transcription sequence and inputting it into a pre-defined large language model; constraining the large language model with pre-defined prompts to output segmentation, reassignment, or sorting operation suggestions without changing the original text content; and using the operation suggestions from the large language model to solve for the globally optimal speaker label sequence through a pre-defined optimization algorithm to obtain the speaker attribution result.

2. The speech multi-role separation method based on a large model according to claim 1, characterized in that, The initial attribution result obtained based on the original audio data includes: preprocessing the original audio data to obtain an audio segment sequence; performing speech recognition processing on the audio segment sequence to generate a timestamped text transcription sequence, wherein the text transcription sequence contains word or phrase level start time, end time, and confidence information; extracting speaker embedding vectors based on the audio segment sequence, and performing cluster analysis on the speaker embedding vectors to assign initial speaker labels to each segment in the text transcription sequence, thereby obtaining the initial attribution result.

3. The speech multi-role separation method based on a large model according to claim 1, characterized in that, The multimodal anomaly score is calculated based on the following formula: Where A(i) represents the anomaly score between the i-th adjacent text segments. sem_div(t) represents the cosine similarity of the centroid vectors of speaker clusters corresponding to adjacent text segments. i ,t i+1 ) represents the semantic difference between adjacent text segments, pause_score(i) represents the score of the pause duration between adjacent text segments, and α, β, and γ are preset adjustable weight coefficients.

4. The speech multi-role separation method based on a large model according to claim 1, characterized in that, The step of identifying sub-segmentation points within the local time window includes: calculating word-level prosodic variation features within the local time window; detecting question-and-answer patterns or self-quote patterns within the local time window; and determining sub-segmentation points based on the word-level prosodic variation features and the detected question-and-answer patterns or self-quote patterns.

5. The speech multi-role separation method based on a large model according to claim 1, characterized in that, The preset prompts include examples and explicitly restrict the large language model to only output operation suggestions for segmentation, reallocation, or order swapping, and prohibits adding, deleting, or rewriting text content.

6. The speech multi-role separation method based on a large model according to claim 1, characterized in that, The optimization algorithm is a dynamic programming algorithm or an integer linear programming algorithm; the objective function of the optimization algorithm is used to maximize acoustic similarity and semantic coherence, minimize unreasonable speaker switching frequency, and introduce a reward term for satisfying the operation suggestions of the large language model.

7. The speech multi-role separation method based on a large model according to claim 2, characterized in that, The method further includes: performing overlapping speech detection on the audio segments in the audio segment sequence; when overlapping speech is detected, performing source separation or parallel annotation operation; generating an audit log, which records the modification process, modification type and evidence used from the initial attribution result to the final speaker attribution result.

8. A speech multi-role separation device based on a large model, characterized in that, Includes a unit for performing the method as described in any one of claims 1-7.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-7.

Images