Speaker diary task optimization method based on large language model semantic capability

Abstract

The present application relates to a speaker diary task optimization method based on the semantic ability of a large language model, belonging to the technical field of artificial intelligence. The present application comprises the following steps: generating timestamped speech transcription text through a voice activity detection and automatic speech recognition module, integrating the generated timestamp with the transcription text to form a timestamp text stream; analyzing the timestamp text stream using a prompt constructor to generate prompt words that match the speaker diary task; inputting the generated prompt words and timestamp text stream into a large language model to analyze the timestamp and text content, generating a preliminary speaker diary result containing a timestamp, a sentence, and a speaker label; post-processing the preliminary speaker diary result to output an accurate speaker diary result, and the error rate of the present application is significantly reduced.

Description

TECHNICAL FIELD

[0001] The application relates to a speaker diarization task optimization method based on semantic capabilities of a large language model, and belongs to the technical field of artificial intelligence. BACKGROUND

[0002] With the rapid development of speech processing technology, speaker diarization, as an important part of speech recognition technology, has attracted more and more attention. The goal of the speaker diarization task is to identify and label the speaker identity of each segment of speech in an audio stream through time indexing. In simple terms, it is to solve the problem of "who said what at what time". This task has wide application value in media broadcasting, conference recording, social media conversation, court recording and business conference. Traditional speaker diarization methods are mainly divided into two categories: modular methods based on clustering and end-to-end systems. Among them, the method based on clustering mainly relies on speech activity detection, speaker embedding extraction and clustering algorithm, and has good flexibility and scalability. However, these methods are highly dependent on the quality of feature extraction, and the optimization objectives of each module are inconsistent, making it difficult to achieve global optimization. The end-to-end method is to complete the speaker diarization task through a unified neural network architecture, and uses a permutation invariant loss function to solve the speaker overlap problem. This method simplifies the process while improving performance in complex scenarios, but it is highly dependent on large-scale labeled data, and its robustness in a multi-noise environment still needs to be improved. SUMMARY

[0003] The technical problem to be solved by the present application is that the present application proposes a speaker diarization task optimization method based on semantic capabilities of a large language model. The method solves the problem of high dependence on large-scale labeled data and poor robustness in a multi-noise environment. The error rate is reduced, and the accuracy and readability of the speaker diarization result are improved.

[0004] The technical solution of the present application is: a speaker diarization task optimization method based on semantic capabilities of a large language model, the method comprising:

[0005] Step 1, generating a timestamped speech transcription text through a speech activity detection and automatic speech recognition module, integrating the generated timestamp with the transcription text to form a timestamp text stream;

[0006] Step 2, analyzing the timestamp text stream using a prompt constructor to generate prompt words matching the speaker diarization task;

[0007] Step 3, inputting the generated prompt words and timestamp text stream into a large language model, parsing the timestamp and text content to generate a preliminary speaker diarization result containing the timestamp, sentence and speaker label;

[0008] Step4、post-processing the preliminary speaker diary result to output an accurate speaker diary result.

[0009] Further, the Step1 comprises: the voice activity detection module adopts FSMN single-channel large language model.

[0010] The voice activity detection module first detects the valid voice segments in the audio and labels the start and end time stamps of these voice segments; then the automatic speech recognition module transcribes these voice segments to generate corresponding texts; finally, the time stamps generated by the voice activity detection module and the texts generated by the automatic speech recognition module are integrated into a timestamp text stream.

[0011] Further, the Step2 comprises:

[0012] First, the timestamp text stream is analyzed by the prompt constructor to extract semantic information, context association and time characteristics in the text, and combined with the characteristics of the dialogue structure, the potential speaker switching points are identified;

[0013] Then, the prompt word task is constructed; the prompt word contains semantic content, time stamp and context information.

[0014] Further, in the Step3, by inputting the constructed prompt word into the large language model, the semantic understanding ability of the model and the Few-shot prompt strategy are used to generate the preliminary speaker diary result containing time stamp, sentence and speaker label.

[0015] Further, the Step3 comprises:

[0016] Step3.1, the large language model with input prompt word is combined with the input timestamp text stream, and the Few-shot prompt strategy is used to analyze the semantic content, identify the speech area and switching point of different speakers, and show the typical dialogue structure to the model through Few-shot to guide the model to accurately distinguish the speech of different speakers; at the same time, the model judges the speaker identity according to the semantic characteristics, and generates the preliminary speaker diary result containing time stamp, sentence and speaker label.

[0017] Step3.2, then, multiple rounds of dialogue input and multiple generation of preliminary speaker diary result: if some speeches in the generated diary cannot mark the speaker identity, it is marked as "speaker 1" by default; and the content of interjection or semantic turning point is assigned to other speakers according to the semantic characteristics.

[0018] Further, in the Step4, the post-processing of the preliminary speaker diary result comprises:

[0019] Step 4.1: Merge consecutive speeches by the same speaker, adjust the start and end positions of the timestamps, and align the timestamps.

[0020] Step 4.2. Eliminate redundancy and delete repeated statements or invalid speeches.

[0021] The present invention provides a speaker log task optimization system based on the semantic capability of a large language model. The system comprises: a module for executing the speaker log task optimization method based on the semantic capability of a large language model.

[0022] The present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the program, the method for optimizing a speaker log task based on the semantic capability of a large language model is implemented.

[0023] The present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for optimizing a speaker log task based on the semantic capability of a large language model is implemented.

[0024] The present invention provides a computer program product, comprising a computer program, which implements the speaker log task optimization method based on the semantic ability of a large language model when the computer program is executed by a processor.

[0025] The beneficial effects of the present invention are:

[0026] 1. The method of the present invention constructs specialized prompts for the speaker log task, using a large language model to directly generate speaker log results, thereby completing the task semantically. The process mainly includes generating a timestamped transcript through voice activity detection and an automatic speech recognition system, and then constructing prompt words to generate task-specific prompts. These prompt words are processed by the large language model to generate the final speaker log results.

[0027] 2. The proposed method uses clustered speaker log error rate as an evaluation metric. Experimental results on the MagicData-RAMC dataset show that the error rate of the model is significantly reduced. Compared with the baseline system, the performance of the speaker log task is improved by 30% using this method.

[0028] 3. The log results generated by the method of the present invention have accurate timestamp calibration, clear semantic logic, and good adaptability to complex interruption scenarios, thereby improving the stability of the system. The present invention can be applied to complex conversations involving interruptions, accurately identifying the speaker through semantic information and assigning timestamps.

[0029] 4. The present invention quickly captures task characteristics and generates high-quality speaker log results through a small number of examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a comparison chart of the effects of using semantic information and embedded information in the speaker log task proposed by the present invention;

[0031] Figure 2 This is a flowchart of semantic speaker log processing based on a large language model driven by the present invention. DETAILED DESCRIPTION

[0032] Example 1: Figure 1-Figure 2 As shown, a speaker log task optimization method based on the semantic capability of a large language model includes:

[0033] Step 1: Generate a speech transcript with a timestamp through the voice activity detection (VAD) and automatic speech recognition (ASR) modules, and integrate the generated timestamp with the transcript to form a timestamp text stream;

[0034] Furthermore, Step 1 includes: the voice activity detection module uses the FSMN monophonic large language model to efficiently detect the start and end time of the voice segment;

[0035] The voice activity detection module first detects valid speech segments in the audio and marks the start and end timestamps of these speech segments; then the automatic speech recognition module transcribes these speech segments to generate corresponding text; finally, the timestamps generated by the voice activity detection module and the text generated by the automatic speech recognition module are integrated into a timestamp text stream.

[0036] The present invention integrates time-stamped text data to form a time-stamped text stream. This process is achieved by integrating the time stamp information generated by the voice activity detection and automatic speech recognition modules with the transcribed text data. The start and end timestamps of each speech segment are paired with the corresponding transcribed text to form a time-stamped text stream. This text stream is then used as input for subsequent steps, and the output of each transcribed text is closely related to the corresponding time stamp information, forming a time-series text record.

[0037] This process essentially combines the start and end time of each speech segment with the corresponding transcript to form a timestamp and text pair.

[0038] The voice activity detection module is primarily responsible for isolating valid speech segments and annotating their start and end times, while the automatic speech recognition module is primarily responsible for transcribing the speech segments into time-stamped text data. The voice activity detection module first processes the input audio signal, identifying segments with speech activity and ignoring segments of silence or noise. For each valid speech segment, the voice activity detection module annotates the start and end times, effectively creating the timestamp for each valid speech segment. After the voice activity detection module completes, the automatic speech recognition module transcribes each speech segment, converting the speech content into text. The automatic speech recognition module not only transcribes each speech segment into text but also assigns a timestamp to each transcribed text, marking the start and end times for each text unit, such as a word or sentence.

[0039] After passing through the voice activity detection and automatic speech recognition modules, the final output is a sequence of multiple speech segments, their corresponding transcribed text, and timestamps, namely a timestamp text stream. This stream contains not only the text content of each speech segment, but also the time position of this content in the conversation.

[0040] In this process, there is no additional integration process, that is, the timestamps and texts generated by the voice activity detection module and the automatic speech recognition module are directly combined to form a timestamp text stream, and their outputs can be directly combined to form a complete timestamp and transcript text pair.

[0041] Step 2: Use the prompt builder to analyze the timestamp text stream and generate prompt words that match the speaker log task;

[0042] Furthermore, the Step 2 includes:

[0043] The construction of prompt words is a key step in achieving speaker log generation. First, the prompt builder analyzes the timestamp text stream to extract semantic information, contextual associations, and temporal features from the text. It then combines the characteristics of the dialogue structure to identify potential speaker transition points.

[0044] Then, the prompt word task is constructed, such as providing a two-person conversation example in a standard format, including timestamps, text, and speaker labels, to guide the large language model to understand the task requirements; the prompt word contains semantic content, timestamps, and contextual information.

[0045] The process of "generating prompt words that match the speaker log task" in this invention first requires a prompt builder to analyze the timestamped text stream. This prompt builder extracts key information from the timestamped transcripts generated by the voice activity detection and automatic speech recognition modules. For example, it identifies key content and semantic relationships within the text, such as the topic of each paragraph and the speaker's speech content. Furthermore, this builder understands the relationship between the preceding and following context, ensuring the coherence and logic of the conversation, helping the model better capture changes in context and identify the order of speech.

[0046] Second, the prompt generator identifies potential speaker transitions based on the semantic and temporal characteristics of the conversation. For example, it identifies whether a particular statement is a response, supplement, or rebuttal to the previous one. Long pauses or changes in speech rate may also indicate a speaker switch.

[0047] Finally, after completing the analysis, the prompt builder generates prompts that match the speaker's log task based on the extracted information. For example, the prompt builder can describe the core information of each speech based on its semantic content and provide corresponding time information for each speech to ensure the order and timing of the speeches. It also uses contextual information to supplement the context of the conversation, helping the model to more accurately understand the context of the speaker's speech.

[0048] Step 3: Input the generated prompt words and timestamp text stream into the Large Language Model (LLM), parse the timestamp and text content, and generate preliminary speaker log results containing timestamps, sentences, and speaker labels;

[0049] Furthermore, in Step 3, by inputting the constructed prompt words into the large language model, the semantic understanding ability of the model and the few-shot prompt strategy are utilized to generate a preliminary speaker log result containing timestamps, sentences and speaker labels.

[0050] Furthermore, the Step 3 includes:

[0051] Step 3.1. Combine the large language model of the input prompt word with the input timestamp text stream, analyze the semantic content through the few-shot prompt strategy, identify the speaking areas and transition points of different speakers, and present the typical dialogue structure to the model through few-shot, guiding the model to accurately distinguish the speeches of different speakers; at the same time, the model determines the speaker identity based on semantic features and generates preliminary speaker log results containing timestamps, sentences and speaker labels.

[0052] Step 3.2: Multiple rounds of dialogue input are then performed, and preliminary speaker logs are generated multiple times. If some statements in the generated logs are not labeled with the speaker's identity, they are defaulted to "Speaker 1." Interruptions or semantic transitions are assigned to other speakers based on semantic features, such as "Speaker 2." The generated logs have accurate timestamp calibration, clear semantic logic, and good adaptability to complex interruption scenarios, thereby improving system stability.

[0053] Step 4: Post-process the preliminary speaker log results and output accurate speaker log results.

[0054] Furthermore, in Step 4, the initially generated log results are post-processed to optimize the accuracy of timestamps and speaker assignments. The post-processing of the initial speaker log results includes:

[0055] Step 4.1: Merge consecutive speeches by the same speaker, adjust the start and end positions of the timestamps, and align the timestamps to make the results more accurate.

[0056] Step 4.2. Eliminate redundancy, delete repeated statements or invalid statements, and ensure the conciseness and readability of the log results.

[0057] The present invention provides a speaker log task optimization system based on the semantic capability of a large language model, the system comprising:

[0058] A timestamp text stream acquisition module is used to generate a speech transcription text with a timestamp through a voice activity detection and automatic speech recognition module, and integrate the generated timestamp with the transcription text to form a timestamp text stream;

[0059] The prompt word construction module is used to analyze the timestamp text stream using the prompt builder and generate prompt words that match the speaker log task;

[0060] A preliminary speaker log result generation module is used to input the generated prompt words and timestamp text stream into the large language model, parse the timestamp and text content, and generate a preliminary speaker log result containing timestamps, sentences, and speaker labels;

[0061] The post-processing module is used to post-process the preliminary speaker log results and output accurate speaker log results.

[0062] The present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the program, the method for optimizing a speaker log task based on the semantic capability of a large language model is implemented.

[0063] The present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for optimizing a speaker log task based on the semantic capability of a large language model is implemented.

[0064] The present invention provides a computer program product, comprising a computer program, which implements the speaker log task optimization method based on the semantic ability of a large language model when the computer program is executed by a processor.

[0065] This paper uses the Clustered Speaker Log Error Rate (CDER) as an evaluation metric to ensure the consistency of timestamps and semantics in the log results. The model's performance is evaluated by merging consecutive utterances from the same speaker and comparing the generated results with a reference. Experiments on the MagicData-RAMC dataset show that the model's error rate is significantly reduced to 0.244, improving the performance of the speaker log task by 30%.

[0066] To verify the effectiveness of the proposed method, the FSMN-Monophone VAD was used as the baseline large language model for the speaker log generation method based on a large language model. The advantages of this large language model are that it considers contextual information during structural modeling, providing fast training and inference speeds, controllable latency, enhanced abstract learning capabilities, and improved discriminative capabilities. The FSMN model consists of a 24-layer Transformer encoder and decoder, with a hidden layer dimension of 1024 and a forward propagation layer dimension of 4096, enhancing the model's nonlinear modeling capabilities. The length predictor consists of five layers of one-dimensional convolutional networks and two linear layers, with a hidden layer dimension of 512. Each convolutional layer uses a Reluctant Unit (ReLU) activation function to introduce nonlinearity and enhance the model's feature learning capabilities.

[0067] This paper uses the General Language Model (GLM) as the core large language model to perform speaker logging tasks. The model is based on a 24-layer Transformer architecture, with a hidden layer dimension of 1024 and a forward propagation layer dimension of 4096. It uses a multi-head self-attention mechanism and GELU activation function, and has powerful semantic modeling capabilities. The model uses a few-shot prompt design, combining timestamps, semantic information, and context to achieve high-precision speaker log generation. At the same time, the large language model introduces layer normalization and dropout regularization techniques, and adopts a multi-round generation strategy to reduce uncertainty.

[0068] Compared with the baseline model, the MagicData-RAMC dataset corpus of the present invention is combined with the speaker log task. The specific experimental results are shown in Table 1 below:

[0069] Table 1 shows the performance comparison of large language models under different prompt formats.

[0070]

[0071] The comparison results in Table 1 show that GLM performs best in the few-shot prompt design, with a CDER of 0.244, significantly outperforming the Baichuan model and the Yi model. Under the chain-of-thought prompt, GLM's performance drops to 0.498, but it still outperforms the other two large language models. Despite a slight performance drop under the chain-of-thought and CRISPE prompts, GLM still outperforms the other models. This shows that GLM is most adaptable to few-shot prompts, able to quickly capture task characteristics and generate high-quality speaker log results with a small number of examples. In contrast, the chain-of-thought prompt may introduce additional generation biases due to the need for more complex reasoning capabilities of the model. While the CRISPE format prompt has certain requirements for task parsing capabilities, GLM still demonstrates strong stability and generation capabilities.

[0072] Table 2 shows the comparative experiments of different numbers of examples in Few-shot prompts.

[0073]

[0074] The comparative experimental results in Table 2 show that when the few-shot approach is 2-shot, the model achieves the lowest clustered speaker log error rate (CDER) of 0.244, achieving optimal performance. However, due to insufficient examples in the 1-shot approach, the model's ability to understand the task is limited, resulting in a higher CDER of 0.350. While the 3-shot approach provides more information, its increased complexity may interfere with model performance, with the CDER rising to 0.313. Experimental data demonstrates that using 2-shot achieves optimal performance on the speaker log task, reducing the CDER. It also helps the GLM balance task parsing and generation capabilities, providing sufficient task guidance while avoiding performance degradation caused by redundant information. Too few or too many examples in the prompt design can affect the model's task understanding and output accuracy, indicating that the design of few-shot prompts requires careful consideration of the number of examples to maximize the model's performance.

[0075] The specific embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. A speaker log task optimization method based on the semantic capabilities of a large language model, characterized by: The method comprises: Step 1: Generate a speech transcript with a timestamp through voice activity detection and automatic speech recognition modules, and integrate the generated timestamp with the transcript to form a timestamp text stream; Step 2: Use the prompt builder to analyze the timestamp text stream and generate prompt words that match the speaker log task; Step 3: Input the generated prompt words and timestamp text stream into the large language model, parse the timestamp and text content, and generate a preliminary speaker log result containing timestamps, sentences, and speaker labels; Step 4: Post-process the preliminary speaker log results to output accurate speaker log results; Step 1 includes: the voice activity detection module adopts the FSMN mono large language model; The voice activity detection module first detects valid speech segments in the audio and annotates the start and end timestamps of these segments. The automatic speech recognition module then transcribes these segments to generate corresponding text. Finally, the timestamps generated by the voice activity detection module and the text generated by the automatic speech recognition module are combined into a timestamp text stream. Step 3 includes: Step 3.1: Combine the large language model with the input prompt word and the input timestamp text stream. Analyze the semantic content using a few-shot prompt strategy to identify the speech areas and transition points of different speakers. The model is presented with typical conversation structures using a few-shot approach, guiding it to accurately distinguish between different speakers. Simultaneously, the model determines the speaker identity based on semantic features and generates a preliminary speaker log containing timestamps, sentences, and speaker labels. Step 3.2: After that, multiple rounds of dialogue input are performed and preliminary speaker log results are generated multiple times. If some speeches in the generated log fail to be labeled with the speaker identity, they are labeled as "Speaker 1" by default; and interjections or semantic transitions are assigned to other speakers based on semantic features.

2. The speaker log task optimization method based on the semantic capability of a large language model according to claim 1 is characterized by: Step 2 includes: First, a prompt builder is used to analyze the timestamp text stream to extract semantic information, contextual associations, and temporal features from the text. This is combined with the characteristics of the dialogue structure to identify potential speaker transition points. Then, the prompt word task is constructed; the prompt word contains semantic content, timestamp and context information.

3. The speaker log task optimization method based on the semantic capability of a large language model according to claim 1, characterized in that: In Step 4, the post-processing of the preliminary speaker log results includes: Step 4.1: Merge consecutive speeches by the same speaker, adjust the start and end positions of the timestamps, and align the timestamps. Step 4.

2. Eliminate redundancy and delete repeated statements or invalid speeches.

4. A speaker log task optimization system based on the semantic capabilities of a large language model, characterized by: The system includes: a module for executing the speaker log task optimization method based on the semantic capability of a large language model according to any one of claims 1 to 3.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the speaker log task optimization method based on the semantic capability of a large language model is implemented as described in any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for optimizing the speaker log task based on the semantic capability of a large language model is implemented.

7. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for optimizing the speaker log task based on the semantic capability of a large language model is implemented.

Images